JeVoisBase  1.11
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ObjectDetect.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
20 #include <jevois/Util/Utils.H>
22 #include <jevois/Debug/Timer.H>
23 
24 #include <linux/videodev2.h>
26 #include <opencv2/imgcodecs.hpp>
27 
28 #include <cstdio> // for std::remove
29 
30 // icon by Vectors Market in arrows at flaticon
31 
32 static jevois::ParameterCategory const ParamCateg("Object Detection Options");
33 
34 //! Define a pair of floats, to avoid macro parsing problems when used as a parameter:
35 typedef std::pair<float, float> floatpair;
36 
37 //! Parameter \relates ObjectDetect
39  "Width and height (in percent of image size, with valid percentages between "
40  "10.0 and 100.0) of the window used to interactively save objects",
41  floatpair(50.0F, 50.0F), ParamCateg);
42 
43 //! Parameter \relates ObjectDetect
44 JEVOIS_DECLARE_PARAMETER(showwin, bool, "Show the interactive image capture window when true", false, ParamCateg);
45 
46 //! Simple object detection using keypoint matching
47 /*! This module finds objects by matching keypoint descriptors between the current image and a set of training
48  images. Here we use SURF keypoints and descriptors as provided by OpenCV.
49 
50  The algorithm consists of 4 phases:
51  - detect keypoint locations, typically corners or other distinctive texture elements or markings;
52  - compute keypoint descriptors, which are summary representations of the image neighborhood around each keypoint;
53  - match descriptors from current image to descriptors previously extracted from training images;
54  - if enough matches are found between the current image and a given training image, and they are of good enough
55  quality, compute the homography (geometric transformation) between keypoint locations in that training image and
56  locations of the matching keypoints in the current image. If it is well conditioned (i.e., a 3D viewpoint change
57  could well explain how the keypoints moved between the training and current images), declare that a match was
58  found, and draw a green rectangle around the detected object.
59 
60  The algorithm comes by default with one training image, for the Priority Mail logo of the U.S. Postal
61  Service. Search for "USPS priority mail" on the web and point JeVois to a picture of the logo on your screen to
62  recognize it. See the screenshots of this module for examples of how that logo looks.
63 
64  Offline training
65  ----------------
66 
67  Simply add images of the objects you want to detect in <b>JEVOIS:/modules/JeVois/ObjectDetect/images/</b> on your
68  JeVois microSD card. Those will be processed when the module starts. The names of recognized objects returned by
69  this module are simply the file names of the pictures you have added in that directory. No additional training
70  procedure is needed. Beware that the more images you add, the slower the algorithm will run, and the higher your
71  chances of confusions among several of your objects.
72 
73  With \jvversion{1.1} or later, you do not need to eject the microSD from JeVois, and you can instead add images live
74  by exporting the microSD inside JeVois using the \c usbsd command. See \ref MicroSD (last section) for details. When
75  you are done adding new images or deleting unwanted ones, properly eject the virtual USB flash drive, and JeVois
76  will restart and load the new training data.
77 
78  Live training
79  -------------
80 
81  With \jvversion{1.2} and later you can train this algorithm live by telling JeVois to capture and save an image of
82  an object, which can be used later to identify this object again.
83 
84  First, enable display of a training window using:
85  \verbatim
86  setpar showwin true
87  \endverbatim
88 
89  You should now see a gray rectangle. You can adjust the window size and aspect ratio using the \p win parameter. By
90  default, the algorithm will train new objects that occupy half the width and height of the camera image.
91 
92  Point your JeVois camera to a clean view of an object you want to learn (if possible, with a blank, featureless
93  background, as this algorithm does not attempt to segment objects and would otherwise also learn features of the
94  background as part of the object). Make sure the objects fits inside the gray rectangle and fills as much of it as
95  possible. You should adjust the distance between the object and the camera, and the grey rectangle, to roughly match
96  the distance at which you want to detect that object in the future. Then issue the command:
97 
98  \verbatim
99  save somename
100  \endverbatim
101 
102  over a serial connection to JeVois, where \a somename is the name you want to give to this object. This will grab
103  the current camera image, crop it using the gray rectangle, and save the crop as a new training image
104  <b>somename.png</b> for immediate use. The algorithm will immediately re-train on all objects, including the new
105  one. You should see the object being detected shortly after you send your save command. Note that we save the image
106  as grayscale since this algorithm does not use color anyway.
107 
108  You can see the list of current images by using command:
109 
110  \verbatim
111  list
112  \endverbatim
113 
114  Finally, you can delete an image using command:
115 
116  \verbatim
117  del somename
118  \endverbatim
119 
120  where \a somename is the object name without extension, and a .png extension will be added. The image will
121  immediately be deleted and that object will not be recognized anymore.
122 
123  For more information, see JeVois tutorial [Live training of the Object Detection
124  module](http://jevois.org/tutorials/UserObjectDetect.html) and the associated video:
125 
126  \youtube{qwJOcsbkZLE}
127 
128  Serial Messages
129  ---------------
130 
131  This module can send standardized serial messages as described in \ref UserSerialStyle. One message is issued on
132  every video frame for the best detected object (highest score).
133 
134  - Serial message type: \b 2D
135  - `id`: filename of the recognized object
136  - `x`, `y`: standardized 2D coordinates of the object center
137  - `w`, `h`, or vertices: Standardized bounding box around the object
138  - `extra`: none (empty string)
139 
140  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
141  standardized coordinates.
142 
143  Programmer notes
144  ----------------
145 
146  This algorithm is quite slow. So, here, we alternate between computing keypoints and descriptors on one frame (or
147  more, depending on how slow that gets), and doing the matching on the next frame. This module also provides an
148  example of letting some computation happen even after we exit the `process()` function. Here, we keep detecting
149  keypoints and computing descriptors even outside `process()`. The itsKPfut future is our handle to that thread, and
150  we also use it to alternate between detection and matching on alternating frames.
151 
152 
153  @author Laurent Itti
154 
155  @videomapping YUYV 320 252 30.0 YUYV 320 240 30.0 JeVois ObjectDetect
156  @modulecommand list - show current list of training images
157  @modulecommand save somename - grab current frame and save as new training image somename.png
158  @modulecommand del somename - delete training image somename.png
159  @email itti\@usc.edu
160  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
161  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
162  @mainurl http://jevois.org
163  @supporturl http://jevois.org/doc
164  @otherurl http://iLab.usc.edu
165  @license GPL v3
166  @distribution Unrestricted
167  @restrictions None
168  \ingroup modules */
170  public jevois::Parameter<win, showwin>
171 {
172  public:
173  // ####################################################################################################
174  //! Constructor
175  // ####################################################################################################
176  ObjectDetect(std::string const & instance) : jevois::StdModule(instance), itsDist(1.0e30)
177  { itsMatcher = addSubComponent<ObjectMatcher>("surf"); }
178 
179  // ####################################################################################################
180  //! Virtual destructor for safe inheritance
181  // ####################################################################################################
182  virtual ~ObjectDetect() { }
183 
184  // ####################################################################################################
185  //! Parameter callback
186  // ####################################################################################################
187  void onParamChange(win const & JEVOIS_UNUSED_PARAM(param), floatpair const & newval)
188  {
189  // Just check that the values are valid here. They will get stored in our param and used later:
190  if (newval.first < 10.0F || newval.first > 100.0F || newval.second < 10.0F || newval.second > 100.0F)
191  throw std::range_error("Invalid window percentage values, must be between 10.0 and 100.0");
192  }
193 
194  // ####################################################################################################
195  //! Processing function with no USB output
196  // ####################################################################################################
197  virtual void process(jevois::InputFrame && inframe) override
198  {
199  static jevois::Timer timer("processing", 100, LOG_DEBUG);
200 
201  // Wait for next available camera image. Any resolution and format ok, we just convert to grayscale:
202  itsGrayImg = inframe.getCvGRAY();
203 
204  timer.start();
205 
206  // Compute keypoints and descriptors, then match descriptors to our training images:
207  itsDist = itsMatcher->process(itsGrayImg, itsTrainIdx, itsCorners);
208 
209  // Send message about object if a good one was found:
210  if (itsDist < 100.0 && itsCorners.size() == 4)
211  sendSerialContour2D(itsGrayImg.cols, itsGrayImg.rows, itsCorners, itsMatcher->traindata(itsTrainIdx).name);
212 
213  // Show processing fps to log:
214  timer.stop();
215  }
216 
217  // ####################################################################################################
218  //! Processing function with USB output
219  // ####################################################################################################
220  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
221  {
222  static jevois::Timer timer("processing", 100, LOG_DEBUG);
223 
224  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
225  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
226  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
227 
228  timer.start();
229 
230  // While we process it, start a thread to wait for output frame and paste the input image into it:
231  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
232  auto paste_fut = std::async(std::launch::async, [&]() {
233  outimg = outframe.get();
234  outimg.require("output", w, h + 12, inimg.fmt);
235  jevois::rawimage::paste(inimg, outimg, 0, 0);
236  jevois::rawimage::writeText(outimg, "JeVois SURF Object Detection Demo", 3, 3, jevois::yuyv::White);
237  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
238  });
239 
240  // Decide what to do on this frame depending on itsKPfut: if it is valid, we have been computing some new
241  // keypoints and descriptors and we should match them now if that computation is finished. If it is not finished,
242  // we will just skip this frame and only do some drawings on it while we wait some more. If we have not been
243  // computing keypoints and descriptors, that means we did some matching on the last frame, so start computing a
244  // new set of keypoints and descriptors now:
245  if (itsKPfut.valid())
246  {
247  // Are we finished yet with computing the keypoints and descriptors?
248  if (itsKPfut.wait_for(std::chrono::milliseconds(2)) == std::future_status::ready)
249  {
250  // Do a get() on our future to free up the async thread and get any exception it might have thrown:
251  itsKPfut.get();
252 
253  // Match descriptors to our training images:
254  itsDist = itsMatcher->match(itsKeypoints, itsDescriptors, itsTrainIdx, itsCorners);
255  }
256 
257  // Future is not ready, do nothing except drawings on this frame and we will try again on the next one...
258  }
259  else
260  {
261  // Convert input image to greyscale:
262  itsGrayImg = jevois::rawimage::convertToCvGray(inimg);
263 
264  // Start a thread that will compute keypoints and descriptors:
265  itsKPfut = std::async(std::launch::async, [&]() {
266  itsMatcher->detect(itsGrayImg, itsKeypoints);
267  itsMatcher->compute(itsGrayImg, itsKeypoints, itsDescriptors);
268  });
269  }
270 
271  // Wait for paste to finish up:
272  paste_fut.get();
273 
274  // Let camera know we are done processing the input image:
275  inframe.done();
276 
277  // Draw object if one was found (note: given the flip-flop above, drawing locations only get updated at half the
278  // frame rate, i.e., we draw the same thing on two successive video frames):
279  if (itsDist < 100.0 && itsCorners.size() == 4)
280  {
281  jevois::rawimage::drawLine(outimg, int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F),
282  int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
283  2, jevois::yuyv::LightGreen);
284  jevois::rawimage::drawLine(outimg, int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
285  int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F), 2,
286  jevois::yuyv::LightGreen);
287  jevois::rawimage::drawLine(outimg, int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F),
288  int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F), 2,
289  jevois::yuyv::LightGreen);
290  jevois::rawimage::drawLine(outimg, int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F),
291  int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F), 2,
292  jevois::yuyv::LightGreen);
293  jevois::rawimage::writeText(outimg, std::string("Detected: ") + itsMatcher->traindata(itsTrainIdx).name +
294  " avg distance " + std::to_string(itsDist), 3, h + 1, jevois::yuyv::White);
295 
296  sendSerialContour2D(w, h, itsCorners, itsMatcher->traindata(itsTrainIdx).name);
297  }
298 
299  // Show capture window if desired:
300  if (showwin::get())
301  {
302  floatpair const wi = win::get();
303  int const ww = (wi.first * 0.01F) * w, wh = (wi.second * 0.01F) * h;
304  jevois::rawimage::drawRect(outimg, (w - ww) / 2, (h - wh) / 2, ww, wh, 1, jevois::yuyv::MedGrey);
305  }
306 
307  // Show processing fps:
308  std::string const & fpscpu = timer.stop();
309  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
310 
311  // Send the output image with our processing results to the host over USB:
312  outframe.send();
313  }
314 
315  // ####################################################################################################
316  //! Receive a string from a serial port which contains a user command
317  // ####################################################################################################
318  void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> s) override
319  {
320  std::vector<std::string> tok = jevois::split(str);
321  if (tok.empty()) throw std::runtime_error("Unsupported empty module command");
322  std::string const dirname = absolutePath(itsMatcher->traindir::get());
323 
324  if (tok[0] == "save")
325  {
326  if (tok.size() == 1) throw std::runtime_error("save command requires one <name> argument");
327 
328  // Crop itsGrayImg using the desired window:
329  floatpair const wi = win::get();
330  int ww = (wi.first * 0.01F) * itsGrayImg.cols;
331  int wh = (wi.second * 0.01F) * itsGrayImg.rows;
332  cv::Rect cr( (itsGrayImg.cols - ww) / 2, (itsGrayImg.rows - wh) / 2, ww, wh);
333 
334  // Save it:
335  cv::imwrite(dirname + '/' + tok[1] + ".png", itsGrayImg(cr));
336  s->writeString(tok[1] + ".png saved and trained.");
337  }
338  else if (tok[0] == "del")
339  {
340  if (tok.size() == 1) throw std::runtime_error("del command requires one <name> argument");
341  if (std::remove((dirname + '/' + tok[1] + ".png").c_str()))
342  throw std::runtime_error("Failed to delete " + tok[1] + ".png");
343  s->writeString(tok[1] + ".png deleted and forgotten.");
344  }
345  else if (tok[0] == "list")
346  {
347  std::string lst = jevois::system("/bin/ls \"" + dirname + '\"');
348  std::vector<std::string> files = jevois::split(lst, "\\n");
349  for (std::string const & f : files) s->writeString(f);
350  return;
351  }
352  else throw std::runtime_error("Unsupported module command [" + str + ']');
353 
354  // If we get here, we had a successful save or del. We need to nuke our matcher and re-load it to retrain:
355  // First, wait until our component is not computing anymore:
356  try { if (itsKPfut.valid()) itsKPfut.get(); } catch (...) { }
357 
358  // Detach the sub:
359  removeSubComponent(itsMatcher);
360 
361  // Nuke it:
362  itsMatcher.reset();
363 
364  // Nuke any other old data:
365  itsKeypoints.clear(); itsDescriptors = cv::Mat(); itsDist = 1.0e30; itsCorners.clear();
366 
367  // Instantiate a new one, it will load the training data:
368  itsMatcher = addSubComponent<ObjectMatcher>("surf");
369  }
370 
371  // ####################################################################################################
372  //! Human-readable description of this Module's supported custom commands
373  // ####################################################################################################
374  void supportedCommands(std::ostream & os) override
375  {
376  os << "list - show current list of training images" << std::endl;
377  os << "save <somename> - grab current frame and save as new training image <somename>.png" << std::endl;
378  os << "del <somename> - delete training image <somename>.png" << std::endl;
379  }
380 
381  private:
382  std::shared_ptr<ObjectMatcher> itsMatcher;
383  std::future<void> itsKPfut;
384  cv::Mat itsGrayImg;
385  std::vector<cv::KeyPoint> itsKeypoints;
386  cv::Mat itsDescriptors;
387  size_t itsTrainIdx;
388  double itsDist;
389  std::vector<cv::Point2f> itsCorners;
390 };
391 
392 // Allow the module to be loaded as a shared object (.so) file:
Simple object detection using keypoint matching.
Definition: ObjectDetect.C:169
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
virtual void process(jevois::InputFrame &&inframe) override
Processing function with no USB output.
Definition: ObjectDetect.C:197
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface > s) override
Receive a string from a serial port which contains a user command.
Definition: ObjectDetect.C:318
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
void drawLine(RawImage &img, int x1, int y1, int x2, int y2, unsigned int thick, unsigned int col)
JEVOIS_REGISTER_MODULE(ObjectDetect)
unsigned int fmt
void onParamChange(win const &JEVOIS_UNUSED_PARAM(param), floatpair const &newval)
Parameter callback.
Definition: ObjectDetect.C:187
JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(dataroot, std::string, "Root path for data, config, and weight files. " "If empty, use the module's path.", JEVOIS_SHARE_PATH "/darknet/single", ParamCateg)
Parameter.
StdModule(std::string const &instance)
void supportedCommands(std::ostream &os) override
Human-readable description of this Module&#39;s supported custom commands.
Definition: ObjectDetect.C:374
std::string system(std::string const &cmd, bool errtoo=true)
cv::Mat convertToCvGray(RawImage const &src)
std::string const & stop()
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with USB output.
Definition: ObjectDetect.C:220
void removeSubComponent(std::shared_ptr< Comp > &component)
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .cfg extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory.", "camera_para", ParamCateg)
Parameter.
ObjectDetect(std::string const &instance)
Constructor.
Definition: ObjectDetect.C:176
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::string to_string(T const &val)
std::pair< float, float > floatpair
Define a pair of floats, to avoid macro parsing problems when used as a parameter: ...
Definition: ObjectDetect.C:35
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
unsigned int width
virtual ~ObjectDetect()
Virtual destructor for safe inheritance.
Definition: ObjectDetect.C:182
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
std::string absolutePath(std::string const &path="")
std::vector< std::string > split(std::string const &input, std::string const &regex="\")
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const