JeVoisBase  1.8
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ObjectDetect.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
20 #include <jevois/Util/Utils.H>
22 #include <jevois/Debug/Timer.H>
23 
24 #include <linux/videodev2.h>
26 #include <opencv2/imgcodecs.hpp>
27 
28 #include <cstdio> // for std::remove
29 
30 // icon by Vectors Market in arrows at flaticon
31 
32 static jevois::ParameterCategory const ParamCateg("Object Detection Options");
33 
34 //! Define a pair of floats, to avoid macro parsing problems when used as a parameter:
35 typedef std::pair<float, float> floatpair;
36 
37 //! Parameter \relates ObjectDetect
39  "Width and height (in percent of image size, with valid percentages between "
40  "10.0 and 100.0) of the window used to interactively save objects",
41  floatpair(50.0F, 50.0F), ParamCateg);
42 
43 //! Parameter \relates ObjectDetect
44 JEVOIS_DECLARE_PARAMETER(showwin, bool, "Show the interactive image capture window when true", false, ParamCateg);
45 
46 //! Simple object detection using keypoint matching
47 /*! This module finds objects by matching keypoint descriptors between the current image and a set of training
48  images. Here we use SURF keypoints and descriptors as provided by OpenCV.
49 
50  The algorithm consists of 4 phases:
51  - detect keypoint locations, typically corners or other distinctive texture elements or markings;
52  - compute keypoint descriptors, which are summary representations of the image neighborhood around each keypoint;
53  - match descriptors from current image to descriptors previously extracted from training images;
54  - if enough matches are found between the current image and a given training image, and they are of good enough
55  quality, compute the homography (geometric transformation) between keypoint locations in that training image and
56  locations of the matching keypoints in the current image. If it is well conditioned (i.e., a 3D viewpoint change
57  could well explain how the keypoints moved between the training and current images), declare that a match was
58  found, and draw a green rectangle around the detected object.
59 
60  The algorithm comes by default with one training image, for the Priority Mail logo of the U.S. Postal
61  Service. Search for "USPS priority mail" on the web and point JeVois to a picture of the logo on your screen to
62  recognize it. See the screenshots of this module for examples of how that logo looks.
63 
64  Offline training
65  ----------------
66 
67  Simply add images of the objects you want to detect in <b>JEVOIS:/modules/JeVois/ObjectDetect/images/</b> on your
68  JeVois microSD card. Those will be processed when the module starts. The names of recognized objects returned by
69  this module are simply the file names of the pictures you have added in that directory. No additional training
70  procedure is needed. Beware that the more images you add, the slower the algorithm will run, and the higher your
71  chances of confusions among several of your objects.
72 
73  With \jvversion{1.1} or later, you do not need to eject the microSD from JeVois, and you can instead add images live
74  by exporting the microSD inside JeVois using the \c usbsd command. See \ref MicroSD (last section) for details. When
75  you are done adding new images or deleting unwanted ones, properly eject the virtual USB flash drive, and JeVois
76  will restart and load the new training data.
77 
78  Live training
79  -------------
80 
81  With \jvversion{1.2} and later you can train this algorithm live by telling JeVois to capture and save an image of
82  an object, which can be used later to identify this object again.
83 
84  First, enable display of a training window using:
85  \verbatim
86  setpar showwin true
87  \endverbatim
88 
89  You should now see a gray rectangle. You can adjust the window size and aspect ratio using the \p win parameter. By
90  default, the algorithm will train new objects that occupy half the width and height of the camera image.
91 
92  Point your JeVois camera to a clean view of an object you want to learn (if possible, with a blank, featureless
93  background, as this algorithm does not attempt to segment objects and would otherwise also learn features of the
94  background as part of the object). Make sure the objects fits inside the gray rectangle and fills as much of it as
95  possible. You should adjust the distance between the object and the camera, and the grey rectangle, to roughly match
96  the distance at which you want to detect that object in the future. Then issue the command:
97 
98  \verbatim
99  save somename
100  \endverbatim
101 
102  over a serial connection to JeVois, where \a somename is the name you want to give to this object. This will grab
103  the current camera image, crop it using the gray rectangle, and save the crop as a new training image
104  <b>somename.png</b> for immediate use. The algorithm will immediately re-train on all objects, including the new
105  one. You should see the object being detected shortly after you send your save command. Note that we save the image
106  as grayscale since this algorithm does not use color anyway.
107 
108  You can see the list of current images by using command:
109 
110  \verbatim
111  list
112  \endverbatim
113 
114  Finally, you can delete an image using command:
115 
116  \verbatim
117  del somename
118  \endverbatim
119 
120  where \a somename is the object name without extension, and a .png extension will be added. The image will
121  immediately be deleted and that object will not be recognized anymore.
122 
123  For more information, see JeVois tutorial [Live training of the Object Detection
124  module](http://jevois.org/tutorials/UserObjectDetect.html) and the associated video:
125 
126  \youtube{qwJOcsbkZLE}
127 
128  Serial Messages
129  ---------------
130 
131  This module can send standardized serial messages as described in \ref UserSerialStyle. One message is issued on
132  every video frame for the best detected object (highest score).
133 
134  - Serial message type: \b 2D
135  - `id`: filename of the recognized object
136  - `x`, `y`: standardized 2D coordinates of the object center
137  - `w`, `h`, or vertices: Standardized bounding box around the object
138  - `extra`: none (empty string)
139 
140  Programmer notes
141  ----------------
142 
143  This algorithm is quite slow. So, here, we alternate between computing keypoints and descriptors on one frame (or
144  more, depending on how slow that gets), and doing the matching on the next frame. This module also provides an
145  example of letting some computation happen even after we exit the `process()` function. Here, we keep detecting
146  keypoints and computing descriptors even outside `process()`. The itsKPfut future is our handle to that thread, and
147  we also use it to alternate between detection and matching on alternating frames.
148 
149 
150  @author Laurent Itti
151 
152  @videomapping YUYV 320 252 30.0 YUYV 320 240 30.0 JeVois ObjectDetect
153  @modulecommand list - show current list of training images
154  @modulecommand save somename - grab current frame and save as new training image somename.png
155  @modulecommand del somename - delete training image somename.png
156  @email itti\@usc.edu
157  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
158  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
159  @mainurl http://jevois.org
160  @supporturl http://jevois.org/doc
161  @otherurl http://iLab.usc.edu
162  @license GPL v3
163  @distribution Unrestricted
164  @restrictions None
165  \ingroup modules */
167  public jevois::Parameter<win, showwin>
168 {
169  public:
170  // ####################################################################################################
171  //! Constructor
172  // ####################################################################################################
173  ObjectDetect(std::string const & instance) : jevois::StdModule(instance), itsDist(1.0e30)
174  { itsMatcher = addSubComponent<ObjectMatcher>("surf"); }
175 
176  // ####################################################################################################
177  //! Virtual destructor for safe inheritance
178  // ####################################################################################################
179  virtual ~ObjectDetect() { }
180 
181  // ####################################################################################################
182  //! Parameter callback
183  // ####################################################################################################
184  void onParamChange(win const & JEVOIS_UNUSED_PARAM(param), floatpair const & newval)
185  {
186  // Just check that the values are valid here. They will get stored in our param and used later:
187  if (newval.first < 10.0F || newval.first > 100.0F || newval.second < 10.0F || newval.second > 100.0F)
188  throw std::range_error("Invalid window percentage values, must be between 10.0 and 100.0");
189  }
190 
191  // ####################################################################################################
192  //! Processing function with no USB output
193  // ####################################################################################################
194  virtual void process(jevois::InputFrame && inframe) override
195  {
196  static jevois::Timer timer("processing", 100, LOG_DEBUG);
197 
198  // Wait for next available camera image. Any resolution and format ok, we just convert to grayscale:
199  itsGrayImg = inframe.getCvGRAY();
200 
201  timer.start();
202 
203  // Compute keypoints and descriptors, then match descriptors to our training images:
204  itsDist = itsMatcher->process(itsGrayImg, itsTrainIdx, itsCorners);
205 
206  // Send message about object if a good one was found:
207  if (itsDist < 100.0 && itsCorners.size() == 4)
208  sendSerialContour2D(itsGrayImg.cols, itsGrayImg.rows, itsCorners, itsMatcher->traindata(itsTrainIdx).name);
209 
210  // Show processing fps to log:
211  timer.stop();
212  }
213 
214  // ####################################################################################################
215  //! Processing function with USB output
216  // ####################################################################################################
217  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
218  {
219  static jevois::Timer timer("processing", 100, LOG_DEBUG);
220 
221  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
222  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
223  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
224 
225  timer.start();
226 
227  // While we process it, start a thread to wait for output frame and paste the input image into it:
228  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
229  auto paste_fut = std::async(std::launch::async, [&]() {
230  outimg = outframe.get();
231  outimg.require("output", w, h + 12, inimg.fmt);
232  jevois::rawimage::paste(inimg, outimg, 0, 0);
233  jevois::rawimage::writeText(outimg, "JeVois SURF Object Detection Demo", 3, 3, jevois::yuyv::White);
234  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
235  });
236 
237  // Decide what to do on this frame depending on itsKPfut: if it is valid, we have been computing some new
238  // keypoints and descriptors and we should match them now if that computation is finished. If it is not finished,
239  // we will just skip this frame and only do some drawings on it while we wait some more. If we have not been
240  // computing keypoints and descriptors, that means we did some matching on the last frame, so start computing a
241  // new set of keypoints and descriptors now:
242  if (itsKPfut.valid())
243  {
244  // Are we finished yet with computing the keypoints and descriptors?
245  if (itsKPfut.wait_for(std::chrono::milliseconds(2)) == std::future_status::ready)
246  {
247  // Do a get() on our future to free up the async thread and get any exception it might have thrown:
248  itsKPfut.get();
249 
250  // Match descriptors to our training images:
251  itsDist = itsMatcher->match(itsKeypoints, itsDescriptors, itsTrainIdx, itsCorners);
252  }
253 
254  // Future is not ready, do nothing except drawings on this frame and we will try again on the next one...
255  }
256  else
257  {
258  // Convert input image to greyscale:
259  itsGrayImg = jevois::rawimage::convertToCvGray(inimg);
260 
261  // Start a thread that will compute keypoints and descriptors:
262  itsKPfut = std::async(std::launch::async, [&]() {
263  itsMatcher->detect(itsGrayImg, itsKeypoints);
264  itsMatcher->compute(itsGrayImg, itsKeypoints, itsDescriptors);
265  });
266  }
267 
268  // Wait for paste to finish up:
269  paste_fut.get();
270 
271  // Let camera know we are done processing the input image:
272  inframe.done();
273 
274  // Draw object if one was found (note: given the flip-flop above, drawing locations only get updated at half the
275  // frame rate, i.e., we draw the same thing on two successive video frames):
276  if (itsDist < 100.0 && itsCorners.size() == 4)
277  {
278  jevois::rawimage::drawLine(outimg, int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F),
279  int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
280  2, jevois::yuyv::LightGreen);
281  jevois::rawimage::drawLine(outimg, int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
282  int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F), 2,
283  jevois::yuyv::LightGreen);
284  jevois::rawimage::drawLine(outimg, int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F),
285  int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F), 2,
286  jevois::yuyv::LightGreen);
287  jevois::rawimage::drawLine(outimg, int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F),
288  int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F), 2,
289  jevois::yuyv::LightGreen);
290  jevois::rawimage::writeText(outimg, std::string("Detected: ") + itsMatcher->traindata(itsTrainIdx).name +
291  " avg distance " + std::to_string(itsDist), 3, h + 1, jevois::yuyv::White);
292 
293  sendSerialContour2D(w, h, itsCorners, itsMatcher->traindata(itsTrainIdx).name);
294  }
295 
296  // Show capture window if desired:
297  if (showwin::get())
298  {
299  floatpair const wi = win::get();
300  int const ww = (wi.first * 0.01F) * w, wh = (wi.second * 0.01F) * h;
301  jevois::rawimage::drawRect(outimg, (w - ww) / 2, (h - wh) / 2, ww, wh, 1, jevois::yuyv::MedGrey);
302  }
303 
304  // Show processing fps:
305  std::string const & fpscpu = timer.stop();
306  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
307 
308  // Send the output image with our processing results to the host over USB:
309  outframe.send();
310  }
311 
312  // ####################################################################################################
313  //! Receive a string from a serial port which contains a user command
314  // ####################################################################################################
315  void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> s) override
316  {
317  std::vector<std::string> tok = jevois::split(str);
318  if (tok.empty()) throw std::runtime_error("Unsupported empty module command");
319  std::string const dirname = absolutePath(itsMatcher->traindir::get());
320 
321  if (tok[0] == "save")
322  {
323  if (tok.size() == 1) throw std::runtime_error("save command requires one <name> argument");
324 
325  // Crop itsGrayImg using the desired window:
326  floatpair const wi = win::get();
327  int ww = (wi.first * 0.01F) * itsGrayImg.cols;
328  int wh = (wi.second * 0.01F) * itsGrayImg.rows;
329  cv::Rect cr( (itsGrayImg.cols - ww) / 2, (itsGrayImg.rows - wh) / 2, ww, wh);
330 
331  // Save it:
332  cv::imwrite(dirname + '/' + tok[1] + ".png", itsGrayImg(cr));
333  s->writeString(tok[1] + ".png saved and trained.");
334  }
335  else if (tok[0] == "del")
336  {
337  if (tok.size() == 1) throw std::runtime_error("del command requires one <name> argument");
338  if (std::remove((dirname + '/' + tok[1] + ".png").c_str()))
339  throw std::runtime_error("Failed to delete " + tok[1] + ".png");
340  s->writeString(tok[1] + ".png deleted and forgotten.");
341  }
342  else if (tok[0] == "list")
343  {
344  std::string lst = jevois::system("/bin/ls \"" + dirname + '\"');
345  std::vector<std::string> files = jevois::split(lst, "\\n");
346  for (std::string const & f : files) s->writeString(f);
347  return;
348  }
349  else throw std::runtime_error("Unsupported module command [" + str + ']');
350 
351  // If we get here, we had a successful save or del. We need to nuke our matcher and re-load it to retrain:
352  // First, wait until our component is not computing anymore:
353  try { if (itsKPfut.valid()) itsKPfut.get(); } catch (...) { }
354 
355  // Detach the sub:
356  removeSubComponent(itsMatcher);
357 
358  // Nuke it:
359  itsMatcher.reset();
360 
361  // Nuke any other old data:
362  itsKeypoints.clear(); itsDescriptors = cv::Mat(); itsDist = 1.0e30; itsCorners.clear();
363 
364  // Instantiate a new one, it will load the training data:
365  itsMatcher = addSubComponent<ObjectMatcher>("surf");
366  }
367 
368  // ####################################################################################################
369  //! Human-readable description of this Module's supported custom commands
370  // ####################################################################################################
371  void supportedCommands(std::ostream & os) override
372  {
373  os << "list - show current list of training images" << std::endl;
374  os << "save <somename> - grab current frame and save as new training image <somename>.png" << std::endl;
375  os << "del <somename> - delete training image <somename>.png" << std::endl;
376  }
377 
378  private:
379  std::shared_ptr<ObjectMatcher> itsMatcher;
380  std::future<void> itsKPfut;
381  cv::Mat itsGrayImg;
382  std::vector<cv::KeyPoint> itsKeypoints;
383  cv::Mat itsDescriptors;
384  size_t itsTrainIdx;
385  double itsDist;
386  std::vector<cv::Point2f> itsCorners;
387 };
388 
389 // Allow the module to be loaded as a shared object (.so) file:
Simple object detection using keypoint matching.
Definition: ObjectDetect.C:166
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
virtual void process(jevois::InputFrame &&inframe) override
Processing function with no USB output.
Definition: ObjectDetect.C:194
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface > s) override
Receive a string from a serial port which contains a user command.
Definition: ObjectDetect.C:315
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
void drawLine(RawImage &img, int x1, int y1, int x2, int y2, unsigned int thick, unsigned int col)
JEVOIS_REGISTER_MODULE(ObjectDetect)
unsigned int fmt
void onParamChange(win const &JEVOIS_UNUSED_PARAM(param), floatpair const &newval)
Parameter callback.
Definition: ObjectDetect.C:184
JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(dataroot, std::string, "Root path for data, config, and weight files. " "If empty, use the module's path.", JEVOIS_SHARE_PATH "/darknet/single", ParamCateg)
Parameter.
StdModule(std::string const &instance)
void supportedCommands(std::ostream &os) override
Human-readable description of this Module&#39;s supported custom commands.
Definition: ObjectDetect.C:371
cv::Mat convertToCvGray(RawImage const &src)
std::string const & stop()
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with USB output.
Definition: ObjectDetect.C:217
void removeSubComponent(std::shared_ptr< Comp > &component)
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .cfg extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory.", "camera_para", ParamCateg)
Parameter.
ObjectDetect(std::string const &instance)
Constructor.
Definition: ObjectDetect.C:173
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::string to_string(T const &val)
std::string system(std::string const &cmd)
std::pair< float, float > floatpair
Define a pair of floats, to avoid macro parsing problems when used as a parameter: ...
Definition: ObjectDetect.C:35
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
unsigned int width
virtual ~ObjectDetect()
Virtual destructor for safe inheritance.
Definition: ObjectDetect.C:179
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
std::string absolutePath(std::string const &path="")
std::vector< std::string > split(std::string const &input, std::string const &regex="\")
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const