JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ObjectDetect.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
20 #include <jevois/Util/Utils.H>
22 #include <jevois/Debug/Timer.H>
23 
24 #include <linux/videodev2.h>
26 #include <opencv2/imgcodecs.hpp>
27 
28 #include <cstdio> // for std::remove
29 
30 // icon by Vectors Market in arrows at flaticon
31 
32 static jevois::ParameterCategory const ParamCateg("Object Detection Options");
33 
34 //! Define a pair of floats, to avoid macro parsing problems when used as a parameter:
35 typedef std::pair<float, float> floatpair;
36 
37 //! Parameter \relates ObjectDetect
39  "Width and height (in percent of image size, with valid percentages between "
40  "10.0 and 100.0) of the window used to interactively save objects",
41  floatpair(50.0F, 50.0F), ParamCateg);
42 
43 //! Parameter \relates ObjectDetect
44 JEVOIS_DECLARE_PARAMETER(showwin, bool, "Show the interactive image capture window when true", false, ParamCateg);
45 
46 //! Simple object detection using keypoint matching
47 /*! This module finds objects by matching keypoint descriptors between the current image and a set of training
48  images. Here we use SURF keypoints and descriptors as provided by OpenCV.
49 
50  The algorithm consists of 4 phases:
51  - detect keypoint locations, typically corners or other distinctive texture elements or markings;
52  - compute keypoint descriptors, which are summary representations of the image neighborhood around each keypoint;
53  - match descriptors from current image to descriptors previously extracted from training images;
54  - if enough matches are found between the current image and a given training image, and they are of good enough
55  quality, compute the homography (geometric transformation) between keypoint locations in that training image and
56  locations of the matching keypoints in the current image. If it is well conditioned (i.e., a 3D viewpoint change
57  could well explain how the keypoints moved between the training and current images), declare that a match was
58  found, and draw a green rectangle around the detected object.
59 
60  The algorithm comes by default with one training image, for the Priority Mail logo of the U.S. Postal
61  Service. Search for "USPS priority mail" on the web and point JeVois to a picture of the logo on your screen to
62  recognize it. See the screenshots of this module for examples of how that logo looks.
63 
64  Offline training
65  ----------------
66 
67  Simply add images of the objects you want to detect in <b>JEVOIS:/modules/JeVois/ObjectDetect/images/</b> on your
68  JeVois microSD card. Those will be processed when the module starts. The names of recognized objects returned by
69  this module are simply the file names of the pictures you have added in that directory. No additional training
70  procedure is needed. Beware that the more images you add, the slower the algorithm will run, and the higher your
71  chances of confusions among several of your objects.
72 
73  With \jvversion{1.1} or later, you do not need to eject the microSD from JeVois, and you can instead add images live
74  by exporting the microSD inside JeVois using the \c usbsd command. See \ref MicroSD (last section) for details. When
75  you are done adding new images or deleting unwanted ones, properly eject the virtual USB flash drive, and JeVois
76  will restart and load the new training data.
77 
78  Live training
79  -------------
80 
81  With \jvversion{1.2} and later you can train this algorithm live bu telling JeVois to capture and save an image of
82  an object, which can be used later to identify this object again.
83 
84  First, enable display of a training window using:
85  \verbatim
86  setpar showwin true
87  \endverbatim
88 
89  You should now see a gray rectangle. You can adjust the window size and aspect ratio using the \p win parameter. By
90  default, the algorithm will train new objects that occupy half the width and height of the camera image.
91 
92  Point your JeVois camera to a clean view of an object you want to learn (if possible, with a blank, featureless
93  background, as this algorithm does not attempt to segment objects and would otherwise also learn features of the
94  background as part of the object). Make sure the objects fits inside the gray rectangle and fills as much of it as
95  possible. You should adjust the distance between the object and the camera, and the grey rectangle, to roughly match
96  the distance at which you want to detect that object in the future. Then issue the command:
97 
98  \verbatim
99  save somename
100  \endverbatim
101 
102  over a serial connection to JeVois, where \a somename is the name you want to give to this object. This will grab
103  the current camera image, crop it using the gray rectangle, and save the crop as a new training image
104  <b>somename.png</b> for immediate use. The algorithm will immediately re-train on all objects, including the new
105  one. You should see the object being detected shortly after you send your save command. Note that we save the image
106  as grayscale since this algorithm does not use color anyway.
107 
108  You can see the list of current images by using command:
109 
110  \verbatim
111  list
112  \endverbatim
113 
114  Finally, you can delete an image using command:
115 
116  \verbatim
117  del somename
118  \endverbatim
119 
120  where \a somename is the object name without extension, and a .png extension will be added. The image will
121  immediately be deleted and that object will not be recognized anymore.
122 
123  For more information, see JeVois tutorial [Live training of the Object Detection
124  module](http://jevois.org/tutorials/UserObjectDetect.html} and the associated video:
125 
126  \youtube{qwJOcsbkZLE}
127 
128  Serial Messages
129  ---------------
130 
131  This module can send standardized serial messages as described in \ref UserSerialStyle. One message is issued on
132  every video frame for the best detected object (highest score).
133 
134  - Serial message type: \b 2D
135  - `id`: filename of the recognized object
136  - `x`, `y`: standardized 2D coordinates of the object center
137  - `w`, `h`, or vertices: Standardized bounding box around the object
138  - `extra`: none (empty string)
139 
140  Programmer notes
141  ----------------
142 
143  This algorithm is quite slow. So, here, we alternate between computing keypoints and descriptors on one frame (or
144  more, depending on how slow that gets), and doing the matching on the next frame. This module also provides an
145  example of letting some computation happen even after we exit the `process()` function. Here, we keep detecting
146  keypoints and computing descriptors even outside `process()`. The itsKPfut future is our handle to that thread, and
147  we also use it to alternate between detection and matching on alternating frames.
148 
149 
150  @author Laurent Itti
151 
152  @videomapping YUYV 320 252 30.0 YUYV 320 240 30.0 JeVois ObjectDetect
153  @modulecommand list - show current list of training images
154  @modulecommand save somename - grab current frame and save as new training image somename.png
155  @modulecommand del somename - delete training image somename.png
156  @email itti\@usc.edu
157  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
158  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
159  @mainurl http://jevois.org
160  @supporturl http://jevois.org/doc
161  @otherurl http://iLab.usc.edu
162  @license GPL v3
163  @distribution Unrestricted
164  @restrictions None
165  \ingroup modules */
167  public jevois::Parameter<win, showwin>
168 {
169  public:
170  // ####################################################################################################
171  //! Constructor
172  // ####################################################################################################
173  ObjectDetect(std::string const & instance) : jevois::StdModule(instance), itsDist(1.0e30)
174  { itsMatcher = addSubComponent<ObjectMatcher>("surf"); }
175 
176  // ####################################################################################################
177  //! Virtual destructor for safe inheritance
178  // ####################################################################################################
179  virtual ~ObjectDetect() { }
180 
181  // ####################################################################################################
182  //! Parameter callback
183  // ####################################################################################################
184  void onParamChange(win const & JEVOIS_UNUSED_PARAM(param), floatpair const & newval)
185  {
186  // Just check that the values are valid here. They will get stored in our param and used later:
187  if (newval.first < 10.0F || newval.first > 100.0F || newval.second < 10.0F || newval.second > 100.0F)
188  throw std::range_error("Invalid window percentage values, must be between 10.0 and 100.0");
189  }
190 
191  // ####################################################################################################
192  //! Processing function with USB output
193  // ####################################################################################################
194  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
195  {
196  static jevois::Timer timer("processing", 100, LOG_DEBUG);
197 
198  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
199  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
200  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
201 
202  timer.start();
203 
204  // While we process it, start a thread to wait for output frame and paste the input image into it:
205  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
206  auto paste_fut = std::async(std::launch::async, [&]() {
207  outimg = outframe.get();
208  outimg.require("output", w, h + 12, inimg.fmt);
209  jevois::rawimage::paste(inimg, outimg, 0, 0);
210  jevois::rawimage::writeText(outimg, "JeVois SURF Object Detection Demo", 3, 3, jevois::yuyv::White);
211  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
212  });
213 
214  // Decide what to do on this frame depending on itsKPfut: if it is valid, we have been computing some new
215  // keypoints and descriptors and we should match them now if that computation is finished. If it is not finished,
216  // we will just skip this frame and only do some drawings on it while we wait some more. If we have not been
217  // computing keypoints and descriptors, that means we did some matching on the last frame, so start computing a
218  // new set of keypoints and descriptors now:
219  if (itsKPfut.valid())
220  {
221  // Are we finished yet with computing the keypoints and descriptors?
222  if (itsKPfut.wait_for(std::chrono::milliseconds(2)) == std::future_status::ready)
223  {
224  // Do a get() on our future to free up the async thread and get any exception it might have thrown:
225  itsKPfut.get();
226 
227  // Match descriptors to our training images:
228  itsDist = itsMatcher->match(itsKeypoints, itsDescriptors, itsTrainIdx, itsCorners);
229  }
230 
231  // Future is not ready, do nothing except drawings on this frame and we will try again on the next one...
232  }
233  else
234  {
235  // Convert input image to greyscale:
236  itsGrayImg = jevois::rawimage::convertToCvGray(inimg);
237 
238  // Start a thread that will compute keypoints and descriptors:
239  itsKPfut = std::async(std::launch::async, [&]() {
240  itsMatcher->detect(itsGrayImg, itsKeypoints);
241  itsMatcher->compute(itsGrayImg, itsKeypoints, itsDescriptors);
242  });
243  }
244 
245  // Wait for paste to finish up:
246  paste_fut.get();
247 
248  // Let camera know we are done processing the input image:
249  inframe.done();
250 
251  // Draw object if one was found (note: given the flip-flop above, drawing locations only get updated at half the
252  // frame rate, i.e., we draw the same thing on two successive video frames):
253  if (itsDist < 100.0 && itsCorners.size() == 4)
254  {
255  jevois::rawimage::drawLine(outimg, int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F),
256  int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
257  2, jevois::yuyv::LightGreen);
258  jevois::rawimage::drawLine(outimg, int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
259  int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F), 2,
260  jevois::yuyv::LightGreen);
261  jevois::rawimage::drawLine(outimg, int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F),
262  int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F), 2,
263  jevois::yuyv::LightGreen);
264  jevois::rawimage::drawLine(outimg, int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F),
265  int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F), 2,
266  jevois::yuyv::LightGreen);
267  jevois::rawimage::writeText(outimg, std::string("Detected: ") + itsMatcher->traindata(itsTrainIdx).name +
268  " avg distance " + std::to_string(itsDist), 3, h + 1, jevois::yuyv::White);
269 
270  sendSerialContour2D(w, h, itsCorners, itsMatcher->traindata(itsTrainIdx).name);
271  }
272 
273  // Show capture window if desired:
274  if (showwin::get())
275  {
276  floatpair const wi = win::get();
277  int const ww = (wi.first * 0.01F) * w, wh = (wi.second * 0.01F) * h;
278  jevois::rawimage::drawRect(outimg, (w - ww) / 2, (h - wh) / 2, ww, wh, 1, jevois::yuyv::MedGrey);
279  }
280 
281  // Show processing fps:
282  std::string const & fpscpu = timer.stop();
283  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
284 
285  // Send the output image with our processing results to the host over USB:
286  outframe.send();
287  }
288 
289  // ####################################################################################################
290  //! Receive a string from a serial port which contains a user command
291  // ####################################################################################################
292  void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> s) override
293  {
294  std::vector<std::string> tok = jevois::split(str);
295  if (tok.empty()) throw std::runtime_error("Unsupported empty module command");
296  std::string const dirname = absolutePath(itsMatcher->traindir::get());
297 
298  if (tok[0] == "save")
299  {
300  if (tok.size() == 1) throw std::runtime_error("save command requires one <name> argument");
301 
302  // Crop itsGrayImg using the desired window:
303  floatpair const wi = win::get();
304  int ww = (wi.first * 0.01F) * itsGrayImg.cols;
305  int wh = (wi.second * 0.01F) * itsGrayImg.rows;
306  cv::Rect cr( (itsGrayImg.cols - ww) / 2, (itsGrayImg.rows - wh) / 2, ww, wh);
307 
308  // Save it:
309  cv::imwrite(dirname + '/' + tok[1] + ".png", itsGrayImg(cr));
310  s->writeString(tok[1] + ".png saved and trained.");
311  }
312  else if (tok[0] == "del")
313  {
314  if (tok.size() == 1) throw std::runtime_error("del command requires one <name> argument");
315  if (std::remove((dirname + '/' + tok[1] + ".png").c_str()))
316  throw std::runtime_error("Failed to delete " + tok[1] + ".png");
317  s->writeString(tok[1] + ".png deleted and forgotten.");
318  }
319  else if (tok[0] == "list")
320  {
321  std::string lst = jevois::system("/bin/ls \"" + dirname + '\"');
322  std::vector<std::string> files = jevois::split(lst, "\\n");
323  for (std::string const & f : files) s->writeString(f);
324  return;
325  }
326  else throw std::runtime_error("Unsupported module command [" + str + ']');
327 
328  // If we get here, we had a successful save or del. We need to nuke our matcher and re-load it to retrain:
329  // First, wait until our component is not computing anymore:
330  try { if (itsKPfut.valid()) itsKPfut.get(); } catch (...) { }
331 
332  // Detach the sub:
333  removeSubComponent(itsMatcher);
334 
335  // Nuke it:
336  itsMatcher.reset();
337 
338  // Nuke any other old data:
339  itsKeypoints.clear(); itsDescriptors = cv::Mat(); itsDist = 1.0e30; itsCorners.clear();
340 
341  // Instantiate a new one, it will load the training data:
342  itsMatcher = addSubComponent<ObjectMatcher>("surf");
343  }
344 
345  // ####################################################################################################
346  //! Human-readable description of this Module's supported custom commands
347  // ####################################################################################################
348  void supportedCommands(std::ostream & os) override
349  {
350  os << "list - show current list of training images" << std::endl;
351  os << "save <somename> - grab current frame and save as new training image <somename>.png" << std::endl;
352  os << "del <somename> - delete training image <somename>.png" << std::endl;
353  }
354 
355  private:
356  std::shared_ptr<ObjectMatcher> itsMatcher;
357  std::future<void> itsKPfut;
358  cv::Mat itsGrayImg;
359  std::vector<cv::KeyPoint> itsKeypoints;
360  cv::Mat itsDescriptors;
361  size_t itsTrainIdx;
362  double itsDist;
363  std::vector<cv::Point2f> itsCorners;
364 };
365 
366 // Allow the module to be loaded as a shared object (.so) file:
Simple object detection using keypoint matching.
Definition: ObjectDetect.C:166
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface > s) override
Receive a string from a serial port which contains a user command.
Definition: ObjectDetect.C:292
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
void drawLine(RawImage &img, int x1, int y1, int x2, int y2, unsigned int thick, unsigned int col)
JEVOIS_REGISTER_MODULE(ObjectDetect)
unsigned int fmt
void onParamChange(win const &JEVOIS_UNUSED_PARAM(param), floatpair const &newval)
Parameter callback.
Definition: ObjectDetect.C:184
JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(dataroot, std::string, "Root path for data, config, and weight files. " "If empty, use the module's path.", JEVOIS_SHARE_PATH "/darknet/single", ParamCateg)
Parameter.
StdModule(std::string const &instance)
void supportedCommands(std::ostream &os) override
Human-readable description of this Module&#39;s supported custom commands.
Definition: ObjectDetect.C:348
cv::Mat convertToCvGray(RawImage const &src)
std::string const & stop()
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with USB output.
Definition: ObjectDetect.C:194
void removeSubComponent(std::shared_ptr< Comp > &component)
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .cfg extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory.", "camera_para", ParamCateg)
Parameter.
ObjectDetect(std::string const &instance)
Constructor.
Definition: ObjectDetect.C:173
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::string to_string(T const &val)
std::string system(std::string const &cmd)
std::pair< float, float > floatpair
Define a pair of floats, to avoid macro parsing problems when used as a parameter: ...
Definition: ObjectDetect.C:35
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
unsigned int width
virtual ~ObjectDetect()
Virtual destructor for safe inheritance.
Definition: ObjectDetect.C:179
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
std::string absolutePath(std::string const &path="")
std::vector< std::string > split(std::string const &input, std::string const &regex="\")
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const