JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoSaliency.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
26 
27 #include <opencv2/core/core.hpp>
28 #include <opencv2/imgproc/imgproc.hpp>
29 
30 #include <future>
31 #include <linux/videodev2.h> // for v4l2 pixel types
32 
33 // icon by Freepik in other at flaticon
34 
35 //! Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998
36 /*! Visual saliency algorithm as described at http://ilab.usc.edu/bu/
37 
38  This algorithm finds the location in the camera's view that is the most attention-grabbing, conspicuous, or
39  so-called salient. This location is marked on every video frame by the pink square. Salient locations detected on
40  each frame are smoothed over time using a Kalman filter. The smoothed attention trajectory is shown with a green
41  circle.
42 
43  For an introduction to visual saliency computation, see http://ilab.usc.edu/bu/
44 
45  Serial Messages
46  ---------------
47 
48  This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
49  sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
50  smoothed attended (most salient) location (green circle in the video display):
51 
52  - Serial message type: \b 2D
53  - `id`: always \b sm (shorthand for saliency map)
54  - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
55  - `w`, `h`: standardized size of the pink square box around each attended point
56  - `extra`: none (empty string)
57 
58  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
59  standardized coordinates.
60 
61 
62  @author Laurent Itti
63 
64  @videomapping YUYV 176 90 120.0 YUYV 88 72 120.0 JeVois DemoSaliency
65  @videomapping YUYV 320 150 60.0 YUYV 160 120 60.0 JeVois DemoSaliency
66  @videomapping YUYV 352 180 120.0 YUYV 176 144 120.0 JeVois DemoSaliency
67  @videomapping YUYV 352 180 100.0 YUYV 176 144 100.0 JeVois DemoSaliency
68  @videomapping YUYV 640 300 60.0 YUYV 320 240 60.0 JeVois DemoSaliency
69  @videomapping YUYV 704 360 30.0 YUYV 352 288 30.0 JeVois DemoSaliency
70  @videomapping YUYV 1280 600 15.0 YUYV 640 480 15.0 JeVois DemoSaliency
71  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
72  @email itti\@usc.edu
73  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
74  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
75  @mainurl http://jevois.org
76  @supporturl http://jevois.org/doc
77  @otherurl http://iLab.usc.edu
78  @license GPL v3
79  @distribution Unrestricted
80  @restrictions None
81  \ingroup modules */
83 {
84  public:
85  //! Constructor
86  DemoSaliency(std::string const & instance) : jevois::StdModule(instance), itsTimer("DemoSaliency")
87  {
88  itsSaliency = addSubComponent<Saliency>("saliency");
89  itsKF = addSubComponent<Kalman2D>("kalman");
90  }
91 
92  //! Virtual destructor for safe inheritance
93  virtual ~DemoSaliency() { }
94 
95  //! Processing function
96  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
97  {
98  // Wait for next available camera image:
99  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
100  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
101 
102  itsTimer.start();
103 
104  // Check whether the input image size is small, in which case we will scale the maps up one notch for the purposes
105  // of this demo:
106  if (w < 170) { itsSaliency->centermin::set(1); itsSaliency->smscale::set(3); }
107  else if (w <= 800) { itsSaliency->centermin::set(2); itsSaliency->smscale::set(4); }
108  else { itsSaliency->centermin::set(3); itsSaliency->smscale::set(4); }
109 
110  // Launch the saliency computation in a thread:
111  auto sal_fut = jevois::async([&](){ itsSaliency->process(inimg, true); });
112 
113  // While computing, wait for an image from our gadget driver into which we will put our results:
114  jevois::RawImage outimg = outframe.get();
115 
116  // Paste the original image to the top-left corner of the display:
117  jevois::rawimage::paste(inimg, outimg, 0, 0);
118  jevois::rawimage::writeText(outimg, "JeVois Saliency + Gist Demo", 3, 3, jevois::yuyv::White);
119 
120  // Once saliency is done using the input image, let camera know we are done with it:
121  itsSaliency->waitUntilDoneWithInput();
122  inframe.done();
123 
124  // Wait until saliency computation is complete:
125  sal_fut.get();
126 
127  // Get some info from the saliency computation:
128  int const smlev = itsSaliency->smscale::get();
129  int const smfac = (1 << smlev);
130  int const roihw = (smfac * 3) / 2; // roi half width and height
131  int const mapdrawfac = smfac / 4; // factor by which we enlarge the feature maps for drawing
132  int const mapdw = (w >> smlev) * mapdrawfac; // width of the drawn feature maps
133  int const mapdh = (h >> smlev) * mapdrawfac; // height of the drawn feature maps
134 
135  // Enforce the correct output image size and format:
136  outimg.require("output", w + (w & ~(smfac-1)), h + mapdh, V4L2_PIX_FMT_YUYV);
137 
138  // Find most salient point:
139  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
140 
141  // Compute attended ROI (note: coords must be even to avoid flipping U/V when we later paste):
142  unsigned int const dmx = (mx << smlev) + (smfac >> 2);
143  unsigned int const dmy = (my << smlev) + (smfac >> 2);
144  int rx = std::min(int(w) - roihw, std::max(roihw, int(dmx + 1 + smfac/4)));
145  int ry = std::min(int(h) - roihw, std::max(roihw, int(dmy + 1 + smfac/4)));
146 
147  // Asynchronously launch a bunch of saliency drawings and filter the attended locations
148  auto draw_fut =
149  jevois::async([&]() {
150  // Filter the attended locations:
151  itsKF->set(dmx, dmy, w, h);
152  float kfxraw, kfyraw, kfximg, kfyimg;
153  itsKF->get(kfxraw, kfyraw, kfximg, kfyimg, inimg.width, inimg.height, 1.0F, 1.0F);
154 
155  // Draw a circle around the kalman-filtered attended location:
156  jevois::rawimage::drawCircle(outimg, int(kfximg), int(kfyimg), 20, 1, jevois::yuyv::LightGreen);
157 
158  // Send kalman-filtered most-salient-point coords to serial port (for arduino, etc):
159  sendSerialImg2D(inimg.width, inimg.height, kfximg, kfyimg, roihw * 2, roihw * 2, "sm");
160 
161  // Paste the saliency map:
162  drawMap(outimg, &itsSaliency->salmap, w, 0, smfac, 20);
163  jevois::rawimage::writeText(outimg, "Saliency Map", w*2 - 12*6-4, 3, jevois::yuyv::White);
164  });
165 
166  // Paste the feature maps:
167  unsigned int dx = 0; // drawing x offset for each feature map
168  drawMap(outimg, &itsSaliency->color, dx, h, mapdrawfac, 18);
169  jevois::rawimage::writeText(outimg, "Color", dx+3, h+3, jevois::yuyv::White);
170  dx += mapdw;
171 
172  drawMap(outimg, &itsSaliency->intens, dx, h, mapdrawfac, 18);
173  jevois::rawimage::writeText(outimg, "Intensity", dx+3, h+3, jevois::yuyv::White);
174  dx += mapdw;
175 
176  drawMap(outimg, &itsSaliency->ori, dx, h, mapdrawfac, 18);
177  jevois::rawimage::writeText(outimg, "Orientation", dx+3, h+3, jevois::yuyv::White);
178  dx += mapdw;
179 
180  drawMap(outimg, &itsSaliency->flicker, dx, h, mapdrawfac, 18);
181  jevois::rawimage::writeText(outimg, "Flicker", dx+3, h+3, jevois::yuyv::White);
182  dx += mapdw;
183 
184  drawMap(outimg, &itsSaliency->motion, dx, h, mapdrawfac, 18);
185  jevois::rawimage::writeText(outimg, "Motion", dx+3, h+3, jevois::yuyv::White);
186  dx += mapdw;
187 
188  // Blank out free space in bottom-right corner, we will then draw the gist (which may only partially occupy that
189  // available space):
190  unsigned int const gw = outimg.width - dx, gh = outimg.height - h;
191  jevois::rawimage::drawFilledRect(outimg, dx, h, gw, gh, 0x8000);
192 
193  // Draw the gist vector, picking a zoom factor to maximize the area filled:
194  unsigned int const gscale = int(sqrt((gw * gh) / itsSaliency->gist_size));
195 
196  drawGist(outimg, itsSaliency->gist, itsSaliency->gist_size, dx, h, gw / gscale, gscale);
197  jevois::rawimage::drawRect(outimg, dx, h, gw, gh, 0x80a0);
198  jevois::rawimage::writeText(outimg, "Gist", dx+3, h+3, jevois::yuyv::White);
199 
200  // Wait for all drawings to complete:
201  draw_fut.get();
202 
203  // Draw a small square at most salient location in image and in saliency map:
204  jevois::rawimage::drawFilledRect(outimg, dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
205  jevois::rawimage::drawFilledRect(outimg, w + dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
206 
207  // Draw an ROI box around the most salient point:
208  jevois::rawimage::drawRect(outimg, rx - roihw, ry - roihw, roihw*2, roihw*2, 0xf0f0);
209  jevois::rawimage::drawRect(outimg, rx - roihw + 1, ry - roihw + 1, roihw*2-2, roihw*2-2, 0xf0f0);
210 
211  // Show processing fps:
212  std::string const & fpscpu = itsTimer.stop();
213  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
214 
215  // Send the output image with our processing results to the host over USB:
216  outframe.send();
217  }
218 
219  protected:
220  std::shared_ptr<Saliency> itsSaliency;
221  std::shared_ptr<Kalman2D> itsKF;
223 };
224 
225 // Allow the module to be loaded as a shared object (.so) file:
drawGist
void drawGist(jevois::RawImage &img, unsigned char const *gist, size_t gistsize, unsigned int xoff, unsigned int yoff, unsigned int width, unsigned int scale)
Definition: Saliency.C:771
jevois::OutputFrame
Timer.H
Module.H
DemoSaliency
Simple demo of the visual saliency algorithm of Itti et al., IEEE PAMI, 1998.
Definition: DemoSaliency.C:82
jevois::StdModule::sendSerialImg2D
void sendSerialImg2D(unsigned int camw, unsigned int camh, float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
Log.H
jevois::RawImage
jevois::rawimage::drawCircle
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
jevois::Timer::start
void start()
drawMap
void drawMap(jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale)
Definition: Saliency.C:709
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DemoSaliency)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
DemoSaliency::itsSaliency
std::shared_ptr< Saliency > itsSaliency
Definition: DemoSaliency.C:220
jevois::RawImage::width
unsigned int width
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
DemoSaliency::itsKF
std::shared_ptr< Kalman2D > itsKF
Definition: DemoSaliency.C:221
jevois::Timer::stop
const std::string & stop(double *seconds)
Kalman2D.H
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::StdModule::StdModule
StdModule(std::string const &instance)
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
RawImageOps.H
jevois::RawImage::height
unsigned int height
jevois::InputFrame
DemoSaliency::~DemoSaliency
virtual ~DemoSaliency()
Virtual destructor for safe inheritance.
Definition: DemoSaliency.C:93
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::rawimage::drawRect
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
DemoSaliency::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoSaliency.C:96
h
int h
DemoSaliency::DemoSaliency
DemoSaliency(std::string const &instance)
Constructor.
Definition: DemoSaliency.C:86
DemoSaliency::itsTimer
jevois::Timer itsTimer
Definition: DemoSaliency.C:222
jevois::StdModule
Saliency.H
ColorConversion.h
intg32
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
jevois::Timer