JeVoisBase  1.0
JeVois Smart Embedded Machine Vision Toolkit Base Modules
ObjectDetect.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
20 #include <jevois/Util/Utils.H>
22 #include <jevois/Debug/Timer.H>
23 
24 #include <linux/videodev2.h>
26 
27 
28 // icon by Vectors Market in arrows at flaticon
29 
30 //! Simple object detection using keypoint matching
31 /*! This module finds objects by matching keypoint descriptors between the current image and a set of training
32  images. Here we use SURF keypoints and descriptors as provided by OpenCV. The algorithm is quite slow and consists
33  of 3 phases: detect keypoint locations, compute keypoint descriptors, and match descriptors from current image to
34  training image descriptors. Here, we alternate between computing keypoints and descriptors on one frame (or more,
35  depending on how slow that gets), and doing the matching on the next frame. This module also provides an example of
36  letting some computation happen even after we exit the process() function. Here, we keep detecting keypoints and
37  computing descriptors even outside process(). The itsKPfut future is our handle to that thread, and we also use it
38  to alternate between detection and matching on alternating frames.
39 
40  Training
41  --------
42 
43  Simply add images of the objects you want to detect in JEVOIS:/modules/JeVois/ObjectDetect/images/ on your JeVois
44  microSD card. Those will be processed when the module starts. The names of recognized objects returned by this
45  module are simply the file names of the pictures you have added in that directory. No additional trainign procedure
46  is needed. Beware that the more images you add, the slower the algorithm will run, and the higher your chances of
47  confusions among several of your objects.
48 
49  @author Laurent Itti
50 
51  @videomapping YUYV 320 252 30.0 YUYV 320 240 30.0 JeVois ObjectDetect
52  @email itti\@usc.edu
53  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
54  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
55  @mainurl http://jevois.org
56  @supporturl http://jevois.org/doc
57  @otherurl http://iLab.usc.edu
58  @license GPL v3
59  @distribution Unrestricted
60  @restrictions None
61  \ingroup modules */
63 {
64  public:
65  //! Constructor
66  ObjectDetect(std::string const & instance) : jevois::Module(instance), itsDist(1.0e30)
67  { itsMatcher = addSubComponent<ObjectMatcher>("surf"); }
68 
69  //! Virtual destructor for safe inheritance
70  virtual ~ObjectDetect() { }
71 
72  //! Processing function
73  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
74  {
75  static jevois::Timer timer("processing", 100, LOG_DEBUG);
76 
77  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
78  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
79  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
80 
81  timer.start();
82 
83  // While we process it, start a thread to wait for output frame and paste the input image into it:
84  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
85  auto paste_fut = std::async(std::launch::async, [&]() {
86  outimg = outframe.get();
87  outimg.require("output", w, h + 12, inimg.fmt);
88  jevois::rawimage::paste(inimg, outimg, 0, 0);
89  jevois::rawimage::writeText(outimg, "JeVois SURF Object Detection Demo", 3, 3, jevois::yuyv::White);
90  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
91  });
92 
93  // Decide what to do on this frame depending on itsKPfut: if it is valid, we have been computing some new
94  // keypoints and descriptors and we should match them now if that computation is finished. If it is not finished,
95  // we will just skip this frame and only do some drawings on it while we wait some more. If we have not been
96  // computing keypoints and descriptors, that means we did some matching on the last frame, so start computing a
97  // new set of keypoints and descriptors now:
98  if (itsKPfut.valid())
99  {
100  // Are we finished yet with computing the keypoints and descriptors?
101  if (itsKPfut.wait_for(std::chrono::milliseconds(2)) == std::future_status::ready)
102  {
103  // Do a get() on our future to free up the async thread and get any exception it might have thrown:
104  itsKPfut.get();
105 
106  // Match descriptors to our training images:
107  itsDist = itsMatcher->match(itsKeypoints, itsDescriptors, itsTrainIdx, itsCorners);
108  }
109 
110  // Future is not ready, do nothing except drawings on this frame and we will try again on the next one...
111  }
112  else
113  {
114  // Convert input image to greyscale:
115  itsGrayImg = jevois::rawimage::convertToCvGray(inimg);
116 
117  // Start a thread that will compute keypoints and descriptors:
118  itsKPfut = std::async(std::launch::async, [&]() {
119  itsMatcher->detect(itsGrayImg, itsKeypoints);
120  itsMatcher->compute(itsGrayImg, itsKeypoints, itsDescriptors);
121  });
122  }
123 
124  // Wait for paste to finish up:
125  paste_fut.get();
126 
127  // Let camera know we are done processing the input image:
128  inframe.done();
129 
130  // Draw object if one was found (note: given the flip-flop above, drawing locations only get updated at half the
131  // frame rate, i.e., we draw the same thing on two successive video frames):
132  if (itsDist < 100.0 && itsCorners.size() == 4)
133  {
134  jevois::rawimage::drawLine(outimg, int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F),
135  int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
136  2, jevois::yuyv::LightGreen);
137  jevois::rawimage::drawLine(outimg, int(itsCorners[1].x + 0.499F), int(itsCorners[1].y + 0.499F),
138  int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F), 2,
139  jevois::yuyv::LightGreen);
140  jevois::rawimage::drawLine(outimg, int(itsCorners[2].x + 0.499F), int(itsCorners[2].y + 0.499F),
141  int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F), 2,
142  jevois::yuyv::LightGreen);
143  jevois::rawimage::drawLine(outimg, int(itsCorners[3].x + 0.499F), int(itsCorners[3].y + 0.499F),
144  int(itsCorners[0].x + 0.499F), int(itsCorners[0].y + 0.499F), 2,
145  jevois::yuyv::LightGreen);
146  jevois::rawimage::writeText(outimg, std::string("Detected: ") + itsMatcher->traindata(itsTrainIdx).name +
147  " avg distance " + std::to_string(itsDist), 3, h + 1, jevois::yuyv::White);
148  }
149 
150  // Show processing fps:
151  std::string const & fpscpu = timer.stop();
152  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
153 
154  // Send the output image with our processing results to the host over USB:
155  outframe.send();
156  }
157 
158  private:
159  std::shared_ptr<ObjectMatcher> itsMatcher;
160  std::future<void> itsKPfut;
161  cv::Mat itsGrayImg;
162  std::vector<cv::KeyPoint> itsKeypoints;
163  cv::Mat itsDescriptors;
164  size_t itsTrainIdx;
165  double itsDist;
166  std::vector<cv::Point2f> itsCorners;
167 };
168 
169 // Allow the module to be loaded as a shared object (.so) file:
Simple object detection using keypoint matching.
Definition: ObjectDetect.C:62
friend friend class Module
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
unsigned int height
void drawLine(RawImage &img, int x1, int y1, int x2, int y2, unsigned int thick, unsigned int col)
JEVOIS_REGISTER_MODULE(ObjectDetect)
unsigned int fmt
cv::Mat convertToCvGray(RawImage const &src)
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: ObjectDetect.C:73
ObjectDetect(std::string const &instance)
Constructor.
Definition: ObjectDetect.C:66
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::string to_string(T const &val)
unsigned int width
std::string const & stop()
virtual ~ObjectDetect()
Virtual destructor for safe inheritance.
Definition: ObjectDetect.C:70
void paste(RawImage const &src, RawImage &dest, int dx, int dy)