JeVoisBase  1.3
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoSaliency.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
26 
27 #include <opencv2/core/core.hpp>
28 #include <opencv2/imgproc/imgproc.hpp>
29 
30 #include <future>
31 #include <linux/videodev2.h> // for v4l2 pixel types
32 
33 // icon by Freepik in other at flaticon
34 
35 //! Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998
36 /*! Visual saliency algorithm as described at http://ilab.usc.edu/bu/
37 
38  This algorithm finds the loaction in the camera's view that is the most attention-grabbing, conspicuous, or
39  so-called salient. This location is marked on every video frame by the pink square. Locations are smoothed over
40  frames using a Kalman filter. The smoothed locations are shown with a green circle.
41 
42  For an introduction to visual saliency, see http://ilab.usc.edu/bu/
43 
44  Serial Messages
45  ---------------
46 
47  This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
48  sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
49  filtered attended (most salient) location (green circle in the video display):
50 
51  - Serial message type: \b 2D
52  - `id`: always \b sm (shorthand for saliency map)
53  - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
54  - `w`, `h`: standardized size of the pink square box around each attended point
55  - `extra`: none (empty string)
56 
57  @author Laurent Itti
58 
59  @videomapping YUYV 176 90 120.0 YUYV 88 72 120.0 JeVois DemoSaliency
60  @videomapping YUYV 320 150 60.0 YUYV 160 120 60.0 JeVois DemoSaliency
61  @videomapping YUYV 352 180 120.0 YUYV 176 144 120.0 JeVois DemoSaliency
62  @videomapping YUYV 352 180 100.0 YUYV 176 144 100.0 JeVois DemoSaliency
63  @videomapping YUYV 640 300 60.0 YUYV 320 240 60.0 JeVois DemoSaliency
64  @videomapping YUYV 704 360 30.0 YUYV 352 288 30.0 JeVois DemoSaliency
65  @videomapping YUYV 1280 600 15.0 YUYV 640 480 15.0 JeVois DemoSaliency
66  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
67  @email itti\@usc.edu
68  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
69  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
70  @mainurl http://jevois.org
71  @supporturl http://jevois.org/doc
72  @otherurl http://iLab.usc.edu
73  @license GPL v3
74  @distribution Unrestricted
75  @restrictions None
76  \ingroup modules */
78 {
79  public:
80  //! Constructor
81  DemoSaliency(std::string const & instance) : jevois::StdModule(instance), itsTimer("DemoSaliency")
82  {
83  itsSaliency = addSubComponent<Saliency>("saliency");
84  itsKF = addSubComponent<Kalman2D>("kalman");
85  }
86 
87  //! Virtual destructor for safe inheritance
88  virtual ~DemoSaliency() { }
89 
90  //! Processing function
91  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
92  {
93  // Wait for next available camera image:
94  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
95  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
96 
97  itsTimer.start();
98 
99  // Check whether the input image size is small, in which case we will scale the maps up one notch for the purposes
100  // of this demo:
101  if (w < 170) { itsSaliency->centermin::set(1); itsSaliency->smscale::set(3); }
102  else { itsSaliency->centermin::set(2); itsSaliency->smscale::set(4); }
103 
104  // Launch the saliency computation in a thread:
105  auto sal_fut = std::async(std::launch::async, [&](){ itsSaliency->process(inimg, true); });
106 
107  // While computing, wait for an image from our gadget driver into which we will put our results:
108  jevois::RawImage outimg = outframe.get();
109 
110  // Paste the original image to the top-left corner of the display:
111  jevois::rawimage::paste(inimg, outimg, 0, 0);
112  jevois::rawimage::writeText(outimg, "JeVois Saliency + Gist Demo", 3, 3, jevois::yuyv::White);
113 
114  // Once saliency is done using the input image, let camera know we are done with it:
115  itsSaliency->waitUntilDoneWithInput();
116  inframe.done();
117 
118  // Wait until saliency computation is complete:
119  sal_fut.get();
120 
121  // Get some info from the saliency computation:
122  int const smlev = itsSaliency->smscale::get();
123  int const smfac = (1 << smlev);
124  int const roihw = (smfac * 3) / 2; // roi half width and height
125  int const mapdrawfac = smfac / 4; // factor by which we enlarge the feature maps for drawing
126  int const mapdw = (w >> smlev) * mapdrawfac; // width of the drawn feature maps
127  int const mapdh = (h >> smlev) * mapdrawfac; // height of the drawn feature maps
128 
129  // Enforce the correct output image size and format:
130  outimg.require("output", w + (w & ~(smfac-1)), h + mapdh, V4L2_PIX_FMT_YUYV);
131 
132  // Find most salient point:
133  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
134 
135  // Compute attended ROI (note: coords must be even to avoid flipping U/V when we later paste):
136  unsigned int const dmx = (mx << smlev) + (smfac >> 2);
137  unsigned int const dmy = (my << smlev) + (smfac >> 2);
138  int rx = std::min(int(w) - roihw, std::max(roihw, int(dmx + 1 + smfac/4)));
139  int ry = std::min(int(h) - roihw, std::max(roihw, int(dmy + 1 + smfac/4)));
140 
141  // Asynchronously launch a bunch of saliency drawings and filter the attended locations
142  auto draw_fut =
143  std::async(std::launch::async, [&]() {
144  // Filter the attended locations:
145  itsKF->set(dmx, dmy, w, h);
146  float kfxraw, kfyraw, kfximg, kfyimg;
147  itsKF->get(kfxraw, kfyraw, kfximg, kfyimg, inimg.width, inimg.height, 1.0F, 1.0F);
148 
149  // Draw a circle around the kalman-filtered attended location:
150  jevois::rawimage::drawCircle(outimg, int(kfximg), int(kfyimg), 20, 1, jevois::yuyv::LightGreen);
151 
152  // Send kalman-filtered most-salient-point coords to serial port (for arduino, etc):
153  sendSerialImg2D(inimg.width, inimg.height, kfximg, kfyimg, roihw * 2, roihw * 2, "sm");
154 
155  // Paste the saliency map:
156  drawMap(outimg, &itsSaliency->salmap, w, 0, smfac, 20);
157  jevois::rawimage::writeText(outimg, "Saliency Map", w*2 - 12*6-4, 3, jevois::yuyv::White);
158  });
159 
160  // Paste the feature maps:
161  unsigned int dx = 0; // drawing x offset for each feature map
162  drawMap(outimg, &itsSaliency->color, dx, h, mapdrawfac, 18);
163  jevois::rawimage::writeText(outimg, "Color", dx+3, h+3, jevois::yuyv::White);
164  dx += mapdw;
165 
166  drawMap(outimg, &itsSaliency->intens, dx, h, mapdrawfac, 18);
167  jevois::rawimage::writeText(outimg, "Intensity", dx+3, h+3, jevois::yuyv::White);
168  dx += mapdw;
169 
170  drawMap(outimg, &itsSaliency->ori, dx, h, mapdrawfac, 18);
171  jevois::rawimage::writeText(outimg, "Orientation", dx+3, h+3, jevois::yuyv::White);
172  dx += mapdw;
173 
174  drawMap(outimg, &itsSaliency->flicker, dx, h, mapdrawfac, 18);
175  jevois::rawimage::writeText(outimg, "Flicker", dx+3, h+3, jevois::yuyv::White);
176  dx += mapdw;
177 
178  drawMap(outimg, &itsSaliency->motion, dx, h, mapdrawfac, 18);
179  jevois::rawimage::writeText(outimg, "Motion", dx+3, h+3, jevois::yuyv::White);
180  dx += mapdw;
181 
182  // Blank out free space in bottom-right corner, we will then draw the gist (which may only partially occupy that
183  // available space):
184  unsigned int const gw = outimg.width - dx, gh = outimg.height - h;
185  jevois::rawimage::drawFilledRect(outimg, dx, h, gw, gh, 0x8000);
186 
187  // Draw the gist vector, picking a zoom factor to maximize the area filled:
188  unsigned int const gscale = int(sqrt((gw * gh) / itsSaliency->gist_size));
189 
190  drawGist(outimg, itsSaliency->gist, itsSaliency->gist_size, dx, h, gw / gscale, gscale);
191  jevois::rawimage::drawRect(outimg, dx, h, gw, gh, 0x80a0);
192  jevois::rawimage::writeText(outimg, "Gist", dx+3, h+3, jevois::yuyv::White);
193 
194  // Wait for all drawings to complete:
195  draw_fut.get();
196 
197  // Draw a small square at most salient location in image and in saliency map:
198  jevois::rawimage::drawFilledRect(outimg, dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
199  jevois::rawimage::drawFilledRect(outimg, w + dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
200 
201  // Draw an ROI box around the most salient point:
202  jevois::rawimage::drawRect(outimg, rx - roihw, ry - roihw, roihw*2, roihw*2, 0xf0f0);
203  jevois::rawimage::drawRect(outimg, rx - roihw + 1, ry - roihw + 1, roihw*2-2, roihw*2-2, 0xf0f0);
204 
205  // Show processing fps:
206  std::string const & fpscpu = itsTimer.stop();
207  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
208 
209  // Send the output image with our processing results to the host over USB:
210  outframe.send();
211  }
212 
213  protected:
214  std::shared_ptr<Saliency> itsSaliency;
215  std::shared_ptr<Kalman2D> itsKF;
217 };
218 
219 // Allow the module to be loaded as a shared object (.so) file:
std::shared_ptr< Saliency > itsSaliency
Definition: DemoSaliency.C:214
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
unsigned int height
JEVOIS_REGISTER_MODULE(DemoSaliency)
void drawGist(jevois::RawImage &img, unsigned char const *gist, size_t gistsize, unsigned int xoff, unsigned int yoff, unsigned int width, unsigned int scale)
Definition: Saliency.C:771
Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998.
Definition: DemoSaliency.C:77
StdModule(std::string const &instance)
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoSaliency.C:91
void drawMap(jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale)
Definition: Saliency.C:709
void sendSerialImg2D(unsigned int camw, unsigned int camh, float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
DemoSaliency(std::string const &instance)
Constructor.
Definition: DemoSaliency.C:81
jevois::Timer itsTimer
Definition: DemoSaliency.C:216
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
virtual ~DemoSaliency()
Virtual destructor for safe inheritance.
Definition: DemoSaliency.C:88
std::shared_ptr< Kalman2D > itsKF
Definition: DemoSaliency.C:215
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
unsigned int width
std::string const & stop()
void paste(RawImage const &src, RawImage &dest, int dx, int dy)