JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ObjectDetect.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
20 #include <jevois/Util/Utils.H>
22 #include <jevois/Debug/Timer.H>
23 
24 #include <linux/videodev2.h>
26 #include <opencv2/imgcodecs.hpp>
27 
28 #include <cstdio> // for std::remove
29 
30 // icon by Vectors Market in arrows at flaticon
31 
32 static jevois::ParameterCategory const ParamCateg("Object Detection Options");
33 
34 //! Define a pair of floats, to avoid macro parsing problems when used as a parameter:
35 typedef std::pair<float, float> floatpair;
36 
37 //! Parameter \relates ObjectDetect
39  "Width and height (in percent of image size, with valid percentages between "
40  "10.0 and 100.0) of the window used to interactively save objects",
41  floatpair(50.0F, 50.0F), ParamCateg);
42 
43 //! Parameter \relates ObjectDetect
44 JEVOIS_DECLARE_PARAMETER(showwin, bool, "Show the interactive image capture window when true", false, ParamCateg);
45 
46 //! Simple object detection using keypoint matching
47 /*! This module finds objects by matching keypoint descriptors between the current image and a set of training
48  images. Here we use SURF keypoints and descriptors as provided by OpenCV.
49 
50  The algorithm consists of 4 phases:
51  - detect keypoint locations, typically corners or other distinctive texture elements or markings;
52  - compute keypoint descriptors, which are summary representations of the image neighborhood around each keypoint;
53  - match descriptors from current image to descriptors previously extracted from training images;
54  - if enough matches are found between the current image and a given training image, and they are of good enough
55  quality, compute the homography (geometric transformation) between keypoint locations in that training image and
56  locations of the matching keypoints in the current image. If it is well conditioned (i.e., a 3D viewpoint change
57  could well explain how the keypoints moved between the training and current images), declare that a match was
58  found, and draw a green rectangle around the detected object.
59 
60  The algorithm comes by default with one training image, for the Priority Mail logo of the U.S. Postal
61  Service. Search for "USPS priority mail" on the web and point JeVois to a picture of the logo on your screen to
62  recognize it. See the screenshots of this module for examples of how that logo looks.
63 
64  Offline training
65  ----------------
66 
67  Simply add images of the objects you want to detect in <b>JEVOIS:/modules/JeVois/ObjectDetect/images/</b> on your
68  JeVois microSD card. Those will be processed when the module starts. The names of recognized objects returned by
69  this module are simply the file names of the pictures you have added in that directory. No additional training
70  procedure is needed. Beware that the more images you add, the slower the algorithm will run, and the higher your
71  chances of confusions among several of your objects.
72 
73  With \jvversion{1.1} or later, you do not need to eject the microSD from JeVois, and you can instead add images live
74  by exporting the microSD inside JeVois using the \c usbsd command. See \ref MicroSD (last section) for details. When
75  you are done addingnew images or deleting unwanted ones, properly eject the virtual USB flash drive, and JeVois will
76  restart and load the new training data.
77 
78  Live training
79  -------------
80 
81  With \jvversion{1.2} and later you can train this algorithm live.
82 
83  First, enable display of a training window using:
84  \verbatim
85  setpar showwin true
86  \endverbatim
87 
88  You should now see a grey rectangle. You can adjust the window size and aspect ratio using the \p win parameter. By
89  default, the algorithm will train new objects that occupy half width and height of the camera image.
90 
91  Point your JeVois camera to a clean view of an object you want to learn (if possible, with a blank, featureless
92  background, as this algorithm does not attempt to segment objects and would otherwise also learn features of the
93  backgrouns as part of the object). Make sure the objects fits inside the grey rectangle and fills as much of it as
94  possible. You should adjust the distance between th eobject and the camera, and the grey rectangle, to roughly match
95  the distance at which you want to detect that object in the future. Then issue the command:
96 
97  \verbatim
98  save somename
99  \endverbatim
100 
101  over a serial connection to JeVois, where <em>somename</em> is the name you want to give to this object. This will
102  grab the current camera image, crop it using the grey rectangle, and save the crop as a new training image
103  <b>somename.png</b> for immediate use. The algorithm will immediately re-train on all objects, including the new
104  one. You should see the object being detected shortly after you send your save command. Note that we save the image
105  as grayscale since this algorithm does not use color anyway.
106 
107  You can see the list of current images by using command:
108  \verbatim
109  list
110  \endverbatim
111 
112  Finally, you can delete an image using command:
113 
114  \verbatim
115  del somename
116  \endverbatim
117  where <em>somename</em> is the object name without extension, and a .png extension will be added. The image will
118  immediately be deleted and that object will not be recognized anymore.
119 
120  Serial Messages
121  ---------------
122 
123  This module can send standardized serial messages as described in \ref UserSerialStyle. One message is issued on
124  every video frame for the best detected object (highest score).
125 
126  - Serial message type: \b 2D
127  - `id`: filename of the recognized object
128  - `x`, `y`: standardized 2D coordinates of the object center
129  - `w`, `h`, or vertices: Standardized bounding box around the object
130  - `extra`: none (empty string)
131 
132  Programmer notes
133  ----------------
134 
135  This algorithm is quite slow. So, here, we alternate between computing keypoints and descriptors on
136  one frame (or more, depending on how slow that gets), and doing the matching on the next frame. This module also
137  provides an example of letting some computation happen even after we exit the process() function. Here, we keep
138  detecting keypoints and computing descriptors even outside process(). The itsKPfut future is our handle to that
139  thread, and we also use it to alternate between detection and matching on alternating frames.
140 
141 
142  @author Laurent Itti
143 
144  @videomapping YUYV 320 252 30.0 YUYV 320 240 30.0 JeVois ObjectDetect
145  @modulecommand list - show current list of training images
146  @modulecommand save somename - grab current frame and save as new training image somename.png
147  @modulecommand del somename - delete training image somename.png
148  @email itti\@usc.edu
149  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
150  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
151  @mainurl http://jevois.org
152  @supporturl http://jevois.org/doc
153  @otherurl http://iLab.usc.edu
154  @license GPL v3
155  @distribution Unrestricted
156  @restrictions None
157  \ingroup modules */
159  public jevois::Parameter<win, showwin>
160 {
161  public:
162  // ####################################################################################################
163  //! Constructor
164  // ####################################################################################################
165  ObjectDetect(std::string const & instance) : jevois::StdModule(instance), itsDist(1.0e30)
166  { itsMatcher = addSubComponent<ObjectMatcher>("surf"); }
167 
168  // ####################################################################################################
169  //! Virtual destructor for safe inheritance
170  // ####################################################################################################
171  virtual ~ObjectDetect() { }
172 
173  // ####################################################################################################
174  //! Parameter callback
175  // ####################################################################################################
176  void onParamChange(win const & JEVOIS_UNUSED_PARAM(param), floatpair const & newval)
177  {
178  // Just check that the values are valid here. They will get stored in our param and used later:
179  if (newval.first < 10.0F || newval.first > 100.0F || newval.second < 10.0F || newval.second > 100.0F)
180  throw std::range_error("Invalid window percentage values, must be between 10.0 and 100.0");
181  }
182 
183  // ####################################################################################################
184  //! Processing function with USB output
185  // ####################################################################################################
186  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
187  {
188  static jevois::Timer timer("processing", 100, LOG_DEBUG);
189 
190  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
191  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
192  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
193 
194  timer.start();
195 
196  // While we process it, start a thread to wait for output frame and paste the input image into it:
197  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
198  auto paste_fut = std::async(std::launch::async, [&]() {
199  outimg = outframe.get();
200  outimg.require("output", w, h + 12, inimg.fmt);
201  jevois::rawimage::paste(inimg, outimg, 0, 0);
202  jevois::rawimage::writeText(outimg, "JeVois SURF Object Detection Demo", 3, 3, jevois::yuyv::White);
203  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
204  });
205 
206  // Decide what to do on this frame depending on itsKPfut: if it is valid, we have been computing some new
207  // keypoints and descriptors and we should match them now if that computation is finished. If it is not finished,
208  // we will just skip this frame and only do some drawings on it while we wait some more. If we have not been
209  // computing keypoints and descriptors, that means we did some matching on the last frame, so start computing a
210  // new set of keypoints and descriptors now:
211  if (itsKPfut.valid())
212  {
213  // Are we finished yet with computing the keypoints and descriptors?
214  if (itsKPfut.wait_for(std::chrono::milliseconds(2)) == std::future_status::ready)
215  {
216  // Do a get() on our future to free up the async thread and get any exception it might have thrown:
217  itsKPfut.get();
218 
219  // Match descriptors to our training images:
220  itsDist = itsMatcher->match(itsKeypoints, itsDescriptors, itsTrainIdx, itsCorners);
221  }
222 
223  // Future is not ready, do nothing except drawings on this frame and we will try again on the next one...
224  }
225  else
226  {
227  // Convert input image to greyscale:
228  itsGrayImg = jevois::rawimage::convertToCvGray(inimg);
229 
230  // Start a thread that will compute keypoints and descriptors:
231  itsKPfut = std::async(std::launch::async, [&]() {
232  itsMatcher->detect(itsGrayImg, itsKeypoints);
233  itsMatcher->compute(itsGrayImg, itsKeypoints, itsDescriptors);
234  });
235  }
236 
237  // Wait for paste to finish up:
238  paste_fut.get();
239 
240  // Let camera know we are done processing the input image:
241  inframe.done();
242 
243  // Draw object if one was found (note: given the flip-flop above, drawing locations only get updated at half the
244  // frame rate, i.e., we draw the same thing on two successive video frames):
245  if (itsDist < 100.0 && itsCorners.size() == 4)
246  {
247  jevois::rawimage::drawLine(outimg, int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F),
248  int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
249  2, jevois::yuyv::LightGreen);
250  jevois::rawimage::drawLine(outimg, int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
251  int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F), 2,
252  jevois::yuyv::LightGreen);
253  jevois::rawimage::drawLine(outimg, int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F),
254  int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F), 2,
255  jevois::yuyv::LightGreen);
256  jevois::rawimage::drawLine(outimg, int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F),
257  int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F), 2,
258  jevois::yuyv::LightGreen);
259  jevois::rawimage::writeText(outimg, std::string("Detected: ") + itsMatcher->traindata(itsTrainIdx).name +
260  " avg distance " + std::to_string(itsDist), 3, h + 1, jevois::yuyv::White);
261 
262  sendSerialContour2D(w, h, itsCorners, itsMatcher->traindata(itsTrainIdx).name);
263  }
264 
265  // Show capture window if desired:
266  if (showwin::get())
267  {
268  floatpair const wi = win::get();
269  int const ww = (wi.first * 0.01F) * w, wh = (wi.second * 0.01F) * h;
270  jevois::rawimage::drawRect(outimg, (w - ww) / 2, (h - wh) / 2, ww, wh, 1, jevois::yuyv::MedGrey);
271  }
272 
273  // Show processing fps:
274  std::string const & fpscpu = timer.stop();
275  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
276 
277  // Send the output image with our processing results to the host over USB:
278  outframe.send();
279  }
280 
281  // ####################################################################################################
282  //! Receive a string from a serial port which contains a user command
283  // ####################################################################################################
284  void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> s) override
285  {
286  std::vector<std::string> tok = jevois::split(str);
287  if (tok.empty()) throw std::runtime_error("Unsupported empty module command");
288  std::string const dirname = absolutePath(itsMatcher->traindir::get());
289 
290  if (tok[0] == "save")
291  {
292  if (tok.size() == 1) throw std::runtime_error("save command requires one <name> argument");
293 
294  // Crop itsGrayImg using the desired window:
295  floatpair const wi = win::get();
296  int ww = (wi.first * 0.01F) * itsGrayImg.cols;
297  int wh = (wi.second * 0.01F) * itsGrayImg.rows;
298  cv::Rect cr( (itsGrayImg.cols - ww) / 2, (itsGrayImg.rows - wh) / 2, ww, wh);
299 
300  // Save it:
301  cv::imwrite(dirname + '/' + tok[1] + ".png", itsGrayImg(cr));
302  s->writeString(tok[1] + ".png saved and trained.");
303  }
304  else if (tok[0] == "del")
305  {
306  if (tok.size() == 1) throw std::runtime_error("del command requires one <name> argument");
307  if (std::remove((dirname + '/' + tok[1] + ".png").c_str()))
308  throw std::runtime_error("Failed to delete " + tok[1] + ".png");
309  s->writeString(tok[1] + ".png deleted and forgotten.");
310  }
311  else if (tok[0] == "list")
312  {
313  std::string lst = jevois::system("/bin/ls \"" + dirname + '\"');
314  std::vector<std::string> files = jevois::split(lst, "\\n");
315  for (std::string const & f : files) s->writeString(f);
316  return;
317  }
318  else throw std::runtime_error("Unsupported module command [" + str + ']');
319 
320  // If we get here, we had a successful save or del. We need to nuke our matcher and re-load it to retrain:
321  // First, wait until our component is not computing anymore:
322  try { if (itsKPfut.valid()) itsKPfut.get(); } catch (...) { }
323 
324  // Detach the sub:
325  removeSubComponent(itsMatcher);
326 
327  // Nuke it:
328  itsMatcher.reset();
329 
330  // Nuke any other old data:
331  itsKeypoints.clear(); itsDescriptors = cv::Mat(); itsDist = 1.0e30; itsCorners.clear();
332 
333  // Instantiate a new one, it will load the training data:
334  itsMatcher = addSubComponent<ObjectMatcher>("surf");
335  }
336 
337  // ####################################################################################################
338  //! Human-readable description of this Module's supported custom commands
339  // ####################################################################################################
340  void supportedCommands(std::ostream & os) override
341  {
342  os << "list - show current list of training images" << std::endl;
343  os << "save <somename> - grab current frame and save as new training image <somename>.png" << std::endl;
344  os << "del <somename> - delete training image <somename>.png" << std::endl;
345  }
346 
347  private:
348  std::shared_ptr<ObjectMatcher> itsMatcher;
349  std::future<void> itsKPfut;
350  cv::Mat itsGrayImg;
351  std::vector<cv::KeyPoint> itsKeypoints;
352  cv::Mat itsDescriptors;
353  size_t itsTrainIdx;
354  double itsDist;
355  std::vector<cv::Point2f> itsCorners;
356 };
357 
358 // Allow the module to be loaded as a shared object (.so) file:
Simple object detection using keypoint matching.
Definition: ObjectDetect.C:158
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface > s) override
Receive a string from a serial port which contains a user command.
Definition: ObjectDetect.C:284
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
void drawLine(RawImage &img, int x1, int y1, int x2, int y2, unsigned int thick, unsigned int col)
JEVOIS_REGISTER_MODULE(ObjectDetect)
unsigned int fmt
void onParamChange(win const &JEVOIS_UNUSED_PARAM(param), floatpair const &newval)
Parameter callback.
Definition: ObjectDetect.C:176
JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(dataroot, std::string, "Root path for data, config, and weight files. " "If empty, use the module's path.", JEVOIS_SHARE_PATH "/darknet/single", ParamCateg)
Parameter.
StdModule(std::string const &instance)
void supportedCommands(std::ostream &os) override
Human-readable description of this Module&#39;s supported custom commands.
Definition: ObjectDetect.C:340
cv::Mat convertToCvGray(RawImage const &src)
std::string const & stop()
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with USB output.
Definition: ObjectDetect.C:186
void removeSubComponent(std::shared_ptr< Comp > &component)
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .cfg extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory.", "camera_para", ParamCateg)
Parameter.
ObjectDetect(std::string const &instance)
Constructor.
Definition: ObjectDetect.C:165
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::string to_string(T const &val)
std::string system(std::string const &cmd)
std::pair< float, float > floatpair
Define a pair of floats, to avoid macro parsing problems when used as a parameter: ...
Definition: ObjectDetect.C:35
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
unsigned int width
virtual ~ObjectDetect()
Virtual destructor for safe inheritance.
Definition: ObjectDetect.C:171
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
std::string absolutePath(std::string const &path="")
std::vector< std::string > split(std::string const &input, std::string const &regex="\")
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const