JeVoisBase  1.20
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
SaliencySURF.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 
19 #include <jevois/Core/Module.H>
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Util/Utils.H>
23 #include <jevois/Debug/Timer.H>
25 
26 #include <linux/videodev2.h>
29 #include <opencv2/opencv.hpp>
30 
31 // icon by Freepik in people at flaticon
32 
33 // Module parameters:
34 #define PATHPREFIX "/jevois/data/saliencysurf/"
35 static jevois::ParameterCategory const ParamCateg("Salient Regions Options");
36 
37 //! Parameter \relates SaliencySURF
38 JEVOIS_DECLARE_PARAMETER(inhsigma, float, "Sigma (pixels) used for inhibition of return", 32.0F, ParamCateg);
39 
40 //! Parameter \relates SaliencySURF
41 JEVOIS_DECLARE_PARAMETER(regions, size_t, "Number of salient regions", 2, ParamCateg);
42 
43 //! Parameter \relates SaliencySURF
44 JEVOIS_DECLARE_PARAMETER(rsiz, size_t, "Width and height (pixels) of salient regions", 64, ParamCateg);
45 
46 //! Parameter \relates SaliencySURF
47 JEVOIS_DECLARE_PARAMETER(save, bool, "Save regions when true, useful to create a training set. They will be saved to "
48  PATHPREFIX, false, ParamCateg);
49 
50 //! Simple salient region detection and identification using keypoint matching
51 /*! This module finds objects by matching keypoint descriptors between a current set of salient regions and a set of
52  training images.
53 
54  Here we use SURF keypoints and descriptors as provided by OpenCV. The algorithm is quite slow and consists of 3
55  phases:
56  - detect keypoint locations,
57  - compute keypoint descriptors,
58  - and match descriptors from current image to training image descriptors.
59 
60  Here, we alternate between computing keypoints and descriptors on one frame (or more, depending on how slow that
61  gets), and doing the matching on the next frame. This module also provides an example of letting some computation
62  happen even after we exit the \c process() function. Here, we keep detecting keypoints and computing descriptors
63  even outside \c process().
64 
65  Also see the \jvmod{ObjectDetect} module for a related algorithm (without attention).
66 
67  Training
68  --------
69 
70  Simply add images of the objects you want to detect into <b>JEVOIS:/modules/JeVois/SaliencySURF/images/</b> on your
71  JeVois microSD card.
72 
73  Those will be processed when the module starts.
74 
75  The names of recognized objects returned by this module are simply the file names of the pictures you have added in
76  that directory. No additional training procedure is needed.
77 
78  Beware that the more images you add, the slower the algorithm will run, and the higher your chances of confusions
79  among several of your objects.
80 
81  This module provides parameters that allow you to determine how strict a match should be. With stricter matching,
82  you may sometimes miss an object (i.e., it was there, but was not detected by the algorithm). With looser matching,
83  you may get more false alarms (i.e., there was something else in the camera's view, but it was matched as one of
84  your objects). If you are experiencing difficulties getting any matches, try to loosen the settings, for example:
85 
86  \verbatim
87  setpar goodpts 5 ... 100
88  setpar distthresh 0.5
89  \endverbatim
90 
91  @author Laurent Itti
92 
93  @displayname Saliency SURF
94  @videomapping YUYV 320 288 30.0 YUYV 320 240 30.0 JeVois SaliencySURF
95  @email itti\@usc.edu
96  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
97  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
98  @mainurl http://jevois.org
99  @supporturl http://jevois.org/doc
100  @otherurl http://iLab.usc.edu
101  @license GPL v3
102  @distribution Unrestricted
103  @restrictions None
104  \ingroup modules */
105 class SaliencySURF : public jevois::Module, public jevois::Parameter<inhsigma, regions, rsiz, save>
106 {
107  public:
108  // ####################################################################################################
109  //! Constructor
110  // ####################################################################################################
111  SaliencySURF(std::string const & instance) : jevois::Module(instance), itsBuf(1000)
112  {
113  itsSaliency = addSubComponent<Saliency>("saliency");
114  itsMatcher = addSubComponent<ObjectMatcher>("surf");
115  }
116 
117  // ####################################################################################################
118  //! Virtual destructor for safe inheritance
119  // ####################################################################################################
120  virtual ~SaliencySURF()
121  { }
122 
123  // ####################################################################################################
124  //! Get started
125  // ####################################################################################################
126  void postInit() override
127  {
128  // Get our run() thread going, it is in charge of compresing and saving frames:
129  itsRunFut = jevois::async(std::bind(&SaliencySURF::run, this));
130 
131  LINFO("Using " << itsMatcher->numtrain() << " Training Images.");
132  }
133 
134  // ####################################################################################################
135  //! Get stopped
136  // ####################################################################################################
137  void postUninit() override
138  {
139  // Push an empty frame into our buffer to signal the end of video to our thread:
140  itsBuf.push(cv::Mat());
141 
142  // Wait for the thread to complete:
143  LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
144  try { itsRunFut.get(); } catch (...) { jevois::warnAndIgnoreException(); }
145  LINFO("Writer thread completed. Syncing disk...");
146  if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
147  }
148 
149  // ####################################################################################################
150  //! Processing function
151  // ####################################################################################################
152  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
153  {
154  static jevois::Timer timer("processing", 30, LOG_DEBUG);
155 
156  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
157  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
158  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
159 
160  timer.start();
161 
162  // While we process it, start a thread to wait for output frame and paste the input image into it:
163  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
164  cv::Mat grayimg;
165 
166  auto paste_fut = jevois::async([&]() {
167  // Convert input image to greyscale:
168  grayimg = jevois::rawimage::convertToCvGray(inimg);
169 
170  // Wait output frame and paste input into it:
171  outimg = outframe.get();
172  outimg.require("output", w, h + 4*12, inimg.fmt);
173  jevois::rawimage::paste(inimg, outimg, 0, 0);
174  jevois::rawimage::writeText(outimg, "JeVois Saliency + SURF Demo", 3, 3, jevois::yuyv::White);
175  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
176  });
177 
178  // Compute the saliency map, no gist:
179  itsSaliency->process(inimg, false);
180 
181  // Get some info from the saliency computation:
182  int const smlev = itsSaliency->smscale::get();
183  int const smfac = (1 << smlev);
184  int const rwh = rsiz::get();
185 
186  // We need the grayscale and output images to proceed:
187  paste_fut.get();
188 
189  // Process each region:
190  int k = 0;
191  for (size_t i = 0; i < regions::get(); ++i)
192  {
193  // Find most salient point:
194  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
195 
196  // Compute attended ROI (note: coords must be even to avoid flipping U/V when we later paste):
197  unsigned int const dmx = (mx << smlev) + (smfac >> 2);
198  unsigned int const dmy = (my << smlev) + (smfac >> 2);
199  int rx = (std::min(int(w) - rwh/2, std::max(rwh/2, int(dmx + 1 + (smfac >> 2))))) & (~1);
200  int ry = (std::min(int(h) - rwh/2, std::max(rwh/2, int(dmy + 1 + (smfac >> 2))))) & (~1);
201  unsigned short col = jevois::yuyv::White;
202 
203  // Grab the ROI:
204  cv::Mat roi = grayimg(cv::Rect(rx - rwh/2, ry - rwh/2, rwh, rwh));
205 
206  // Save it if desired:
207  if (save::get()) itsBuf.push(roi);
208 
209  // Process it through our matcher:
210  size_t trainidx;
211  double dist = itsMatcher->process(roi, trainidx);
212  if (dist < 100.0)
213  {
214  jevois::rawimage::writeText(outimg, std::string("Detected: ") + itsMatcher->traindata(trainidx).name +
215  " avg distance " + std::to_string(dist), 3, h + k*12 + 2, jevois::yuyv::White);
216  col = jevois::yuyv::LightGreen;
217  ++k;
218  }
219 
220  // Draw the ROI:
221  jevois::rawimage::drawRect(outimg, rx - rwh/2, ry - rwh/2, rwh, rwh, 1, col);
222 
223  // Inhibit this salient location so we move to the next one:
224  itsSaliency->inhibitionOfReturn(mx, my, inhsigma::get() / smfac);
225  }
226 
227  // Show processing fps:
228  std::string const & fpscpu = timer.stop();
229  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
230 
231  // Send the output image with our processing results to the host over USB:
232  outframe.send();
233  }
234 
235  private:
236  // ####################################################################################################
237  void run() // Runs in a thread to save regions as images, for training
238  {
239  size_t frame = 0; char tmp[2048];
240 
241  // Create directory just in case it does not exist:
242  std::string const cmd = "/bin/mkdir -p " PATHPREFIX;
243  if (std::system(cmd.c_str())) LERROR("Error running [" << cmd << "] -- IGNORED");
244 
245  while (true)
246  {
247  // Get next frame from the buffer:
248  cv::Mat im = itsBuf.pop();
249 
250  // An empty image will be pushed when we are ready to unload the module:
251  if (im.empty()) break;
252 
253  // Write the frame:
254  std::snprintf(tmp, 2047, "%s/frame%06zu.png", PATHPREFIX, frame);
255  cv::imwrite(tmp, im);
256 
257  // Report what is going on once in a while:
258  if ((++frame % 100) == 0) LINFO("Saved " << frame << " salient regions.");
259  }
260  }
261 
262  std::shared_ptr<ObjectMatcher> itsMatcher;
263  std::shared_ptr<Saliency> itsSaliency;
264  std::future<void> itsRunFut;
266 };
267 
268 // Allow the module to be loaded as a shared object (.so) file:
SaliencySURF::SaliencySURF
SaliencySURF(std::string const &instance)
Constructor.
Definition: SaliencySURF.C:111
jevois::OutputFrame
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
Timer.H
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(SaliencySURF)
Module.H
demo.roi
roi
Definition: demo.py:75
Log.H
jevois::BoundedBuffer< cv::Mat, jevois::BlockingBehavior::Block, jevois::BlockingBehavior::Block >
jevois::RawImage
jevois::Timer::start
void start()
SaliencySURF::~SaliencySURF
virtual ~SaliencySURF()
Virtual destructor for safe inheritance.
Definition: SaliencySURF.C:120
jevois::ParameterCategory
LERROR
#define LERROR(msg)
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::RawImage::width
unsigned int width
SaliencySURF::postInit
void postInit() override
Get started.
Definition: SaliencySURF.C:126
PATHPREFIX
#define PATHPREFIX
Definition: SaliencySURF.C:34
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
F
float F
system
std::string system(std::string const &cmd, bool errtoo=true)
jevois::BoundedBuffer::pop
T pop()
jevois::BoundedBuffer::push
void push(T const &val)
SaliencySURF
Simple salient region detection and identification using keypoint matching.
Definition: SaliencySURF.C:105
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::warnAndIgnoreException
std::string warnAndIgnoreException(std::string const &prefix="")
jevois::BoundedBuffer::filled_size
size_t filled_size() const
jevois::Module
RawImageOps.H
jevois::RawImage::height
unsigned int height
to_string
std::string to_string(T const &val)
SaliencySURF::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: SaliencySURF.C:152
jevois::InputFrame
demo.frame
frame
Definition: demo.py:120
Utils.H
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::rawimage::drawRect
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
jevois::RawImage::fmt
unsigned int fmt
ObjectMatcher.H
h
int h
BoundedBuffer.H
Saliency.H
ARtoolkit::JEVOIS_DECLARE_PARAMETER
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .dat extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory (if relative stem) or " "from the specified absolute location (if absolute stem).", JEVOIS_SHARE_PATH "/camera/camera_para", ParamCateg)
Parameter.
LINFO
#define LINFO(msg)
SaliencySURF::postUninit
void postUninit() override
Get stopped.
Definition: SaliencySURF.C:137
demo.w
w
Definition: demo.py:85
intg32
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
jevois::Timer
jevois::Component::Module
friend friend class Module