JeVoisBase  1.6
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoSaliency.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
26 
27 #include <opencv2/core/core.hpp>
28 #include <opencv2/imgproc/imgproc.hpp>
29 
30 #include <future>
31 #include <linux/videodev2.h> // for v4l2 pixel types
32 
33 // icon by Freepik in other at flaticon
34 
35 //! Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998
36 /*! Visual saliency algorithm as described at http://ilab.usc.edu/bu/
37 
38  This algorithm finds the location in the camera's view that is the most attention-grabbing, conspicuous, or
39  so-called salient. This location is marked on every video frame by the pink square. Salient locations detected on
40  each frame are smoothed over time using a Kalman filter. The smoothed attention trajectory is shown with a green
41  circle.
42 
43  For an introduction to visual saliency computation, see http://ilab.usc.edu/bu/
44 
45  Serial Messages
46  ---------------
47 
48  This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
49  sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
50  smoothed attended (most salient) location (green circle in the video display):
51 
52  - Serial message type: \b 2D
53  - `id`: always \b sm (shorthand for saliency map)
54  - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
55  - `w`, `h`: standardized size of the pink square box around each attended point
56  - `extra`: none (empty string)
57 
58 
59  @author Laurent Itti
60 
61  @videomapping YUYV 176 90 120.0 YUYV 88 72 120.0 JeVois DemoSaliency
62  @videomapping YUYV 320 150 60.0 YUYV 160 120 60.0 JeVois DemoSaliency
63  @videomapping YUYV 352 180 120.0 YUYV 176 144 120.0 JeVois DemoSaliency
64  @videomapping YUYV 352 180 100.0 YUYV 176 144 100.0 JeVois DemoSaliency
65  @videomapping YUYV 640 300 60.0 YUYV 320 240 60.0 JeVois DemoSaliency
66  @videomapping YUYV 704 360 30.0 YUYV 352 288 30.0 JeVois DemoSaliency
67  @videomapping YUYV 1280 600 15.0 YUYV 640 480 15.0 JeVois DemoSaliency
68  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
69  @email itti\@usc.edu
70  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
71  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
72  @mainurl http://jevois.org
73  @supporturl http://jevois.org/doc
74  @otherurl http://iLab.usc.edu
75  @license GPL v3
76  @distribution Unrestricted
77  @restrictions None
78  \ingroup modules */
80 {
81  public:
82  //! Constructor
83  DemoSaliency(std::string const & instance) : jevois::StdModule(instance), itsTimer("DemoSaliency")
84  {
85  itsSaliency = addSubComponent<Saliency>("saliency");
86  itsKF = addSubComponent<Kalman2D>("kalman");
87  }
88 
89  //! Virtual destructor for safe inheritance
90  virtual ~DemoSaliency() { }
91 
92  //! Processing function
93  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
94  {
95  // Wait for next available camera image:
96  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
97  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
98 
99  itsTimer.start();
100 
101  // Check whether the input image size is small, in which case we will scale the maps up one notch for the purposes
102  // of this demo:
103  if (w < 170) { itsSaliency->centermin::set(1); itsSaliency->smscale::set(3); }
104  else { itsSaliency->centermin::set(2); itsSaliency->smscale::set(4); }
105 
106  // Launch the saliency computation in a thread:
107  auto sal_fut = std::async(std::launch::async, [&](){ itsSaliency->process(inimg, true); });
108 
109  // While computing, wait for an image from our gadget driver into which we will put our results:
110  jevois::RawImage outimg = outframe.get();
111 
112  // Paste the original image to the top-left corner of the display:
113  jevois::rawimage::paste(inimg, outimg, 0, 0);
114  jevois::rawimage::writeText(outimg, "JeVois Saliency + Gist Demo", 3, 3, jevois::yuyv::White);
115 
116  // Once saliency is done using the input image, let camera know we are done with it:
117  itsSaliency->waitUntilDoneWithInput();
118  inframe.done();
119 
120  // Wait until saliency computation is complete:
121  sal_fut.get();
122 
123  // Get some info from the saliency computation:
124  int const smlev = itsSaliency->smscale::get();
125  int const smfac = (1 << smlev);
126  int const roihw = (smfac * 3) / 2; // roi half width and height
127  int const mapdrawfac = smfac / 4; // factor by which we enlarge the feature maps for drawing
128  int const mapdw = (w >> smlev) * mapdrawfac; // width of the drawn feature maps
129  int const mapdh = (h >> smlev) * mapdrawfac; // height of the drawn feature maps
130 
131  // Enforce the correct output image size and format:
132  outimg.require("output", w + (w & ~(smfac-1)), h + mapdh, V4L2_PIX_FMT_YUYV);
133 
134  // Find most salient point:
135  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
136 
137  // Compute attended ROI (note: coords must be even to avoid flipping U/V when we later paste):
138  unsigned int const dmx = (mx << smlev) + (smfac >> 2);
139  unsigned int const dmy = (my << smlev) + (smfac >> 2);
140  int rx = std::min(int(w) - roihw, std::max(roihw, int(dmx + 1 + smfac/4)));
141  int ry = std::min(int(h) - roihw, std::max(roihw, int(dmy + 1 + smfac/4)));
142 
143  // Asynchronously launch a bunch of saliency drawings and filter the attended locations
144  auto draw_fut =
145  std::async(std::launch::async, [&]() {
146  // Filter the attended locations:
147  itsKF->set(dmx, dmy, w, h);
148  float kfxraw, kfyraw, kfximg, kfyimg;
149  itsKF->get(kfxraw, kfyraw, kfximg, kfyimg, inimg.width, inimg.height, 1.0F, 1.0F);
150 
151  // Draw a circle around the kalman-filtered attended location:
152  jevois::rawimage::drawCircle(outimg, int(kfximg), int(kfyimg), 20, 1, jevois::yuyv::LightGreen);
153 
154  // Send kalman-filtered most-salient-point coords to serial port (for arduino, etc):
155  sendSerialImg2D(inimg.width, inimg.height, kfximg, kfyimg, roihw * 2, roihw * 2, "sm");
156 
157  // Paste the saliency map:
158  drawMap(outimg, &itsSaliency->salmap, w, 0, smfac, 20);
159  jevois::rawimage::writeText(outimg, "Saliency Map", w*2 - 12*6-4, 3, jevois::yuyv::White);
160  });
161 
162  // Paste the feature maps:
163  unsigned int dx = 0; // drawing x offset for each feature map
164  drawMap(outimg, &itsSaliency->color, dx, h, mapdrawfac, 18);
165  jevois::rawimage::writeText(outimg, "Color", dx+3, h+3, jevois::yuyv::White);
166  dx += mapdw;
167 
168  drawMap(outimg, &itsSaliency->intens, dx, h, mapdrawfac, 18);
169  jevois::rawimage::writeText(outimg, "Intensity", dx+3, h+3, jevois::yuyv::White);
170  dx += mapdw;
171 
172  drawMap(outimg, &itsSaliency->ori, dx, h, mapdrawfac, 18);
173  jevois::rawimage::writeText(outimg, "Orientation", dx+3, h+3, jevois::yuyv::White);
174  dx += mapdw;
175 
176  drawMap(outimg, &itsSaliency->flicker, dx, h, mapdrawfac, 18);
177  jevois::rawimage::writeText(outimg, "Flicker", dx+3, h+3, jevois::yuyv::White);
178  dx += mapdw;
179 
180  drawMap(outimg, &itsSaliency->motion, dx, h, mapdrawfac, 18);
181  jevois::rawimage::writeText(outimg, "Motion", dx+3, h+3, jevois::yuyv::White);
182  dx += mapdw;
183 
184  // Blank out free space in bottom-right corner, we will then draw the gist (which may only partially occupy that
185  // available space):
186  unsigned int const gw = outimg.width - dx, gh = outimg.height - h;
187  jevois::rawimage::drawFilledRect(outimg, dx, h, gw, gh, 0x8000);
188 
189  // Draw the gist vector, picking a zoom factor to maximize the area filled:
190  unsigned int const gscale = int(sqrt((gw * gh) / itsSaliency->gist_size));
191 
192  drawGist(outimg, itsSaliency->gist, itsSaliency->gist_size, dx, h, gw / gscale, gscale);
193  jevois::rawimage::drawRect(outimg, dx, h, gw, gh, 0x80a0);
194  jevois::rawimage::writeText(outimg, "Gist", dx+3, h+3, jevois::yuyv::White);
195 
196  // Wait for all drawings to complete:
197  draw_fut.get();
198 
199  // Draw a small square at most salient location in image and in saliency map:
200  jevois::rawimage::drawFilledRect(outimg, dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
201  jevois::rawimage::drawFilledRect(outimg, w + dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
202 
203  // Draw an ROI box around the most salient point:
204  jevois::rawimage::drawRect(outimg, rx - roihw, ry - roihw, roihw*2, roihw*2, 0xf0f0);
205  jevois::rawimage::drawRect(outimg, rx - roihw + 1, ry - roihw + 1, roihw*2-2, roihw*2-2, 0xf0f0);
206 
207  // Show processing fps:
208  std::string const & fpscpu = itsTimer.stop();
209  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
210 
211  // Send the output image with our processing results to the host over USB:
212  outframe.send();
213  }
214 
215  protected:
216  std::shared_ptr<Saliency> itsSaliency;
217  std::shared_ptr<Kalman2D> itsKF;
219 };
220 
221 // Allow the module to be loaded as a shared object (.so) file:
std::shared_ptr< Saliency > itsSaliency
Definition: DemoSaliency.C:216
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
JEVOIS_REGISTER_MODULE(DemoSaliency)
void drawGist(jevois::RawImage &img, unsigned char const *gist, size_t gistsize, unsigned int xoff, unsigned int yoff, unsigned int width, unsigned int scale)
Definition: Saliency.C:771
Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998.
Definition: DemoSaliency.C:79
StdModule(std::string const &instance)
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoSaliency.C:93
void drawMap(jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale)
Definition: Saliency.C:709
void sendSerialImg2D(unsigned int camw, unsigned int camh, float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
DemoSaliency(std::string const &instance)
Constructor.
Definition: DemoSaliency.C:83
std::string const & stop()
jevois::Timer itsTimer
Definition: DemoSaliency.C:218
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
virtual ~DemoSaliency()
Virtual destructor for safe inheritance.
Definition: DemoSaliency.C:90
std::shared_ptr< Kalman2D > itsKF
Definition: DemoSaliency.C:217
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
unsigned int width
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const