JeVoisBase  1.15
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoSaliency.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
26 
27 #include <opencv2/core/core.hpp>
28 #include <opencv2/imgproc/imgproc.hpp>
29 
30 #include <future>
31 #include <linux/videodev2.h> // for v4l2 pixel types
32 
33 // icon by Freepik in other at flaticon
34 
35 //! Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998
36 /*! Visual saliency algorithm as described at http://ilab.usc.edu/bu/
37 
38  This algorithm finds the location in the camera's view that is the most attention-grabbing, conspicuous, or
39  so-called salient. This location is marked on every video frame by the pink square. Salient locations detected on
40  each frame are smoothed over time using a Kalman filter. The smoothed attention trajectory is shown with a green
41  circle.
42 
43  For an introduction to visual saliency computation, see http://ilab.usc.edu/bu/
44 
45  Serial Messages
46  ---------------
47 
48  This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
49  sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
50  smoothed attended (most salient) location (green circle in the video display):
51 
52  - Serial message type: \b 2D
53  - `id`: always \b sm (shorthand for saliency map)
54  - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
55  - `w`, `h`: standardized size of the pink square box around each attended point
56  - `extra`: none (empty string)
57 
58  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
59  standardized coordinates.
60 
61 
62  @author Laurent Itti
63 
64  @videomapping YUYV 176 90 120.0 YUYV 88 72 120.0 JeVois DemoSaliency
65  @videomapping YUYV 320 150 60.0 YUYV 160 120 60.0 JeVois DemoSaliency
66  @videomapping YUYV 352 180 120.0 YUYV 176 144 120.0 JeVois DemoSaliency
67  @videomapping YUYV 352 180 100.0 YUYV 176 144 100.0 JeVois DemoSaliency
68  @videomapping YUYV 640 300 60.0 YUYV 320 240 60.0 JeVois DemoSaliency
69  @videomapping YUYV 704 360 30.0 YUYV 352 288 30.0 JeVois DemoSaliency
70  @videomapping YUYV 1280 600 15.0 YUYV 640 480 15.0 JeVois DemoSaliency
71  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
72  @email itti\@usc.edu
73  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
74  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
75  @mainurl http://jevois.org
76  @supporturl http://jevois.org/doc
77  @otherurl http://iLab.usc.edu
78  @license GPL v3
79  @distribution Unrestricted
80  @restrictions None
81  \ingroup modules */
83 {
84  public:
85  //! Constructor
86  DemoSaliency(std::string const & instance) : jevois::StdModule(instance), itsTimer("DemoSaliency")
87  {
88  itsSaliency = addSubComponent<Saliency>("saliency");
89  itsKF = addSubComponent<Kalman2D>("kalman");
90  }
91 
92  //! Virtual destructor for safe inheritance
93  virtual ~DemoSaliency() { }
94 
95  //! Processing function
96  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
97  {
98  // Wait for next available camera image:
99  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
100  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
101 
102  itsTimer.start();
103 
104  // Check whether the input image size is small, in which case we will scale the maps up one notch for the purposes
105  // of this demo:
106  if (w < 170) { itsSaliency->centermin::set(1); itsSaliency->smscale::set(3); }
107  else { itsSaliency->centermin::set(2); itsSaliency->smscale::set(4); }
108 
109  // Launch the saliency computation in a thread:
110  auto sal_fut = std::async(std::launch::async, [&](){ itsSaliency->process(inimg, true); });
111 
112  // While computing, wait for an image from our gadget driver into which we will put our results:
113  jevois::RawImage outimg = outframe.get();
114 
115  // Paste the original image to the top-left corner of the display:
116  jevois::rawimage::paste(inimg, outimg, 0, 0);
117  jevois::rawimage::writeText(outimg, "JeVois Saliency + Gist Demo", 3, 3, jevois::yuyv::White);
118 
119  // Once saliency is done using the input image, let camera know we are done with it:
120  itsSaliency->waitUntilDoneWithInput();
121  inframe.done();
122 
123  // Wait until saliency computation is complete:
124  sal_fut.get();
125 
126  // Get some info from the saliency computation:
127  int const smlev = itsSaliency->smscale::get();
128  int const smfac = (1 << smlev);
129  int const roihw = (smfac * 3) / 2; // roi half width and height
130  int const mapdrawfac = smfac / 4; // factor by which we enlarge the feature maps for drawing
131  int const mapdw = (w >> smlev) * mapdrawfac; // width of the drawn feature maps
132  int const mapdh = (h >> smlev) * mapdrawfac; // height of the drawn feature maps
133 
134  // Enforce the correct output image size and format:
135  outimg.require("output", w + (w & ~(smfac-1)), h + mapdh, V4L2_PIX_FMT_YUYV);
136 
137  // Find most salient point:
138  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
139 
140  // Compute attended ROI (note: coords must be even to avoid flipping U/V when we later paste):
141  unsigned int const dmx = (mx << smlev) + (smfac >> 2);
142  unsigned int const dmy = (my << smlev) + (smfac >> 2);
143  int rx = std::min(int(w) - roihw, std::max(roihw, int(dmx + 1 + smfac/4)));
144  int ry = std::min(int(h) - roihw, std::max(roihw, int(dmy + 1 + smfac/4)));
145 
146  // Asynchronously launch a bunch of saliency drawings and filter the attended locations
147  auto draw_fut =
148  std::async(std::launch::async, [&]() {
149  // Filter the attended locations:
150  itsKF->set(dmx, dmy, w, h);
151  float kfxraw, kfyraw, kfximg, kfyimg;
152  itsKF->get(kfxraw, kfyraw, kfximg, kfyimg, inimg.width, inimg.height, 1.0F, 1.0F);
153 
154  // Draw a circle around the kalman-filtered attended location:
155  jevois::rawimage::drawCircle(outimg, int(kfximg), int(kfyimg), 20, 1, jevois::yuyv::LightGreen);
156 
157  // Send kalman-filtered most-salient-point coords to serial port (for arduino, etc):
158  sendSerialImg2D(inimg.width, inimg.height, kfximg, kfyimg, roihw * 2, roihw * 2, "sm");
159 
160  // Paste the saliency map:
161  drawMap(outimg, &itsSaliency->salmap, w, 0, smfac, 20);
162  jevois::rawimage::writeText(outimg, "Saliency Map", w*2 - 12*6-4, 3, jevois::yuyv::White);
163  });
164 
165  // Paste the feature maps:
166  unsigned int dx = 0; // drawing x offset for each feature map
167  drawMap(outimg, &itsSaliency->color, dx, h, mapdrawfac, 18);
168  jevois::rawimage::writeText(outimg, "Color", dx+3, h+3, jevois::yuyv::White);
169  dx += mapdw;
170 
171  drawMap(outimg, &itsSaliency->intens, dx, h, mapdrawfac, 18);
172  jevois::rawimage::writeText(outimg, "Intensity", dx+3, h+3, jevois::yuyv::White);
173  dx += mapdw;
174 
175  drawMap(outimg, &itsSaliency->ori, dx, h, mapdrawfac, 18);
176  jevois::rawimage::writeText(outimg, "Orientation", dx+3, h+3, jevois::yuyv::White);
177  dx += mapdw;
178 
179  drawMap(outimg, &itsSaliency->flicker, dx, h, mapdrawfac, 18);
180  jevois::rawimage::writeText(outimg, "Flicker", dx+3, h+3, jevois::yuyv::White);
181  dx += mapdw;
182 
183  drawMap(outimg, &itsSaliency->motion, dx, h, mapdrawfac, 18);
184  jevois::rawimage::writeText(outimg, "Motion", dx+3, h+3, jevois::yuyv::White);
185  dx += mapdw;
186 
187  // Blank out free space in bottom-right corner, we will then draw the gist (which may only partially occupy that
188  // available space):
189  unsigned int const gw = outimg.width - dx, gh = outimg.height - h;
190  jevois::rawimage::drawFilledRect(outimg, dx, h, gw, gh, 0x8000);
191 
192  // Draw the gist vector, picking a zoom factor to maximize the area filled:
193  unsigned int const gscale = int(sqrt((gw * gh) / itsSaliency->gist_size));
194 
195  drawGist(outimg, itsSaliency->gist, itsSaliency->gist_size, dx, h, gw / gscale, gscale);
196  jevois::rawimage::drawRect(outimg, dx, h, gw, gh, 0x80a0);
197  jevois::rawimage::writeText(outimg, "Gist", dx+3, h+3, jevois::yuyv::White);
198 
199  // Wait for all drawings to complete:
200  draw_fut.get();
201 
202  // Draw a small square at most salient location in image and in saliency map:
203  jevois::rawimage::drawFilledRect(outimg, dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
204  jevois::rawimage::drawFilledRect(outimg, w + dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
205 
206  // Draw an ROI box around the most salient point:
207  jevois::rawimage::drawRect(outimg, rx - roihw, ry - roihw, roihw*2, roihw*2, 0xf0f0);
208  jevois::rawimage::drawRect(outimg, rx - roihw + 1, ry - roihw + 1, roihw*2-2, roihw*2-2, 0xf0f0);
209 
210  // Show processing fps:
211  std::string const & fpscpu = itsTimer.stop();
212  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
213 
214  // Send the output image with our processing results to the host over USB:
215  outframe.send();
216  }
217 
218  protected:
219  std::shared_ptr<Saliency> itsSaliency;
220  std::shared_ptr<Kalman2D> itsKF;
222 };
223 
224 // Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DemoSaliency)
drawGist
void drawGist(jevois::RawImage &img, unsigned char const *gist, size_t gistsize, unsigned int xoff, unsigned int yoff, unsigned int width, unsigned int scale)
Definition: Saliency.C:771
jevois::OutputFrame
Timer.H
Module.H
DemoSaliency
Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998.
Definition: DemoSaliency.C:82
jevois::StdModule::sendSerialImg2D
void sendSerialImg2D(unsigned int camw, unsigned int camh, float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
Log.H
jevois::RawImage
jevois::rawimage::drawCircle
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
jevois::Timer::start
void start()
drawMap
void drawMap(jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale)
Definition: Saliency.C:709
DemoSaliency::itsKF
std::shared_ptr< Kalman2D > itsKF
Definition: DemoSaliency.C:220
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::RawImage::width
unsigned int width
jevois::Timer::stop
const std::string & stop()
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
Kalman2D.H
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::StdModule::StdModule
StdModule(std::string const &instance)
RawImageOps.H
jevois::RawImage::height
unsigned int height
jevois::InputFrame
DemoSaliency::~DemoSaliency
virtual ~DemoSaliency()
Virtual destructor for safe inheritance.
Definition: DemoSaliency.C:93
DemoSaliency::itsSaliency
std::shared_ptr< Saliency > itsSaliency
Definition: DemoSaliency.C:219
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::rawimage::drawRect
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
DemoSaliency::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoSaliency.C:96
DemoSaliency::DemoSaliency
DemoSaliency(std::string const &instance)
Constructor.
Definition: DemoSaliency.C:86
DemoSaliency::itsTimer
jevois::Timer itsTimer
Definition: DemoSaliency.C:221
jevois::StdModule
Saliency.H
ColorConversion.h
intg32
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
jevois::Timer