JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ArUcoBlob.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2018 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 #include <jevois/Core/Module.H>
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Util/Utils.H>
23 #include <jevois/Debug/Timer.H>
25 
26 #include <opencv2/imgproc/imgproc.hpp>
27 
28 #include <map>
29 
30 static jevois::ParameterCategory const ParamCateg("ArUcoBlob Options");
31 
32 //! Parameter \relates ArUcoBlob
33 JEVOIS_DECLARE_PARAMETER(numtrack, size_t, "Number of parallel blob trackers to run. They will be named blob0, "
34  "blob1, etc for parameters and serial messages",
35  3, ParamCateg);
36 
37 //! Combined ArUco marker + multiple color-based object detection
38 /*! This modules 1) detects ArUco markers (small black-and-white geometric patterns which can be used as tags for some
39  objects), and, in parallel, 2) isolates pixels within multiple given HSV ranges (hue, saturation, and value of color
40  pixels), does some cleanups, and extracts object contours. It sends information about detected ArUco tags and color
41  objects over serial.
42 
43  This module was developed to allow students to easily develop visually-guided robots that can at the same time
44  detect ArUco markers placed in the environment to signal certain key objects (e.g., charging station, home base) and
45  colored objects of different kinds (e.g., blue people, green trees, and yellow fires).
46 
47  This module usually works best with the camera sensor set to manual exposure, manual gain, manual color balance, etc
48  so that HSV color values are reliable. See the file \b script.cfg file in this module's directory for an example of
49  how to set the camera settings each time this module is loaded.
50 
51  Since this is a combination module, refer to:
52 
53  - \jvmod{DemoArUco} for the ArUco algorithm and messages
54  - \jvmod{ObjectTracker} for the blob detection algorithm and messages
55 
56  The number of parallel blob trackers is determined by parameter \p numtrack, which should be set before the module
57  is initialized, i.e., in the module's \b params.cfg file. It cannot be changed while the module is running.
58 
59  The module runs at about 50 frames/s with 3 parallel blob detectors plus ArUco, at 320x240 camera sensor
60  resolution. Increasing to 10 parallel blob detectors will still get you about 25 frames/s (but finding robust
61  non-overlapping HSV ranges for all those detectors will become challenging!)
62 
63  To configure parameters \p hrange, \p srange, and \p vrange for each detector in the module's \b scrip.cfg, we
64  recommend that you do it one by one for each kind of colored object you want, using the \jvmod{ObjectTracker} module
65  (which shares the same color blob detection code, just for one HSV range) and the tutorial on <A
66  HREF="http://jevois.org/tutorials/UserColorTracking.html">Tuning the color-based object tracker using a python
67  graphical interface</A>, or the sliders in JeVois Inventor. Just make sure that both modules have the same camera
68  settings in their respective \b script.cfg files.
69 
70  Using the serial outputs
71  ------------------------
72 
73  We recommend the following settings (to apply after you load the module, for example in the module's \b script.cfg
74  file):
75  \code{.py}
76  setpar serout USB # to send to serial-over-USB, or use Hard to send to 4-pin serial port
77  setpar serstyle Normal # to get ID, center location, size for every detected color blob and ArUco tag
78  setpar serstamp Frame # to add video frame number to all messages
79  \endcode
80 
81  With a scene as shown in this module's screenshots, you would then get outputs like:
82 
83  \verbatim
84  ...
85  1557 N2 U42 -328 -9 706 569
86  1557 N2 U18 338 -241 613 444
87  1557 N2 blob0 616 91 406 244
88  1557 N2 blob1 28 584 881 331
89  1557 N2 blob2 47 -553 469 206
90  1558 N2 U42 -328 -9 706 569
91  1558 N2 U18 338 -241 613 444
92  1558 N2 blob0 547 113 519 275
93  1558 N2 blob1 28 581 881 338
94  1558 N2 blob2 47 -553 469 206
95  1559 N2 U42 -331 -13 700 563
96  1559 N2 U18 338 -244 613 450
97  1559 N2 blob0 369 153 200 194
98  1559 N2 blob0 616 94 381 250
99  1559 N2 blob1 28 581 881 338
100  1559 N2 blob2 47 -553 469 206
101  ...
102  \endverbatim
103 
104  which basically means that, on frame 1557, ArUco markers U42 and U18 were detected, then blob detector named "blob0"
105  (configured for light blue objects in \b script.cfg) detected one blob, then "blob1" (configured for yellow) also
106  detected one, and finally "blob2" (configured for green) found one too. That was all for frame 1157, and we then
107  switch to frame 1158 (with essentially the same detections), then frame 1159 (note how blob0 detected 2 different
108  blobs on that frame), and so on. See \ref UserSerialStyle for more info about these messages.
109 
110  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
111  standardized coordinates.
112 
113  Running with no video output (standalone mode)
114  ----------------------------------------------
115 
116  Try these settings in the global initialization script file of JeVois, which is executed when JeVois powers up, in
117  <b>JEVOIS:/config/initscript.cfg</b>:
118 
119  \code{.py}
120  setmapping2 YUYV 320 240 45.0 JeVois ArUcoBlob # to select this module upon power up
121  setpar serout Hard # to send detection messages to 4-pin serial port
122  setpar serstyle Normal # to get ID, center location, size
123  setpar serstamp Frame # to add video frame number to all messages
124  streamon # start capturing and processing camera sensor data
125  \endcode
126 
127  Make sure you do not have conflicting settings in the module's \b params.cfg or \b script.cfg file; as a reminder,
128  the order of execution is: 1) \b initscript.cfg runs, which loads the module through the `setmapping2` command; 2)
129  as part of the loading process and before the module is initialized, settings in \b params.cfg are applied; 3) the
130  module is then initialized and commands in \b script.cfg are run; 4) the additional commands following `setmapping2`
131  in \b initscript.cfg are finally run. Next time JeVois starts up, it will automatically load this module and start
132  sending messages to the hardware 4-pin serial port, which you should then connect to an Arduino or other robot
133  controller.
134 
135  @author Laurent Itti
136 
137  @displayname ArUco Blob
138  @videomapping YUYV 320 266 30.0 YUYV 320 240 30.0 JeVois ArUcoBlob
139  @videomapping NONE 0 0 0.0 YUYV 320 240 30.0 JeVois ArUcoBlob
140  @email itti\@usc.edu
141  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
142  @copyright Copyright (C) 2018 by Laurent Itti, iLab and the University of Southern California
143  @mainurl http://jevois.org
144  @supporturl http://jevois.org/doc
145  @otherurl http://iLab.usc.edu
146  @license GPL v3
147  @distribution Unrestricted
148  @restrictions None
149  \ingroup modules */
151  public jevois::Parameter<numtrack>
152 {
153  public:
154  // ####################################################################################################
155  //! Constructor
156  // ####################################################################################################
157  ArUcoBlob(std::string const & instance) :
158  jevois::StdModule(instance)
159  {
160  itsArUco = addSubComponent<ArUco>("aruco");
161  // We instantiate the blob detectors in postInit() once their number is finalized
162  }
163 
164  // ####################################################################################################
165  //! Post-init: instantiate the blob detectors
166  // ####################################################################################################
167  void postInit() override
168  {
170 
171  for (int i = 0; i < numtrack::get(); ++i)
172  itsBlobs.push_back(addSubComponent<BlobDetector>("blob" + std::to_string(i)));
173  }
174 
175  // ####################################################################################################
176  // Pre-unInit: release the blob detectors
177  // ####################################################################################################
178  void preUninit() override
179  {
180  for (auto & b : itsBlobs) removeSubComponent(b);
181  itsBlobs.clear();
182  }
183 
184  // ####################################################################################################
185  //! Virtual destructor for safe inheritance
186  // ####################################################################################################
187  virtual ~ArUcoBlob() { }
188 
189  // ####################################################################################################
190  //! Detect blobs in parallel threads
191  // ####################################################################################################
192  void detectBlobs(jevois::RawImage * outimg = nullptr)
193  {
194  itsContours.clear();
195 
196  // In a bunch of threads, detect blobs and get the contours:
197  for (auto & b : itsBlobs)
198  itsBlobFuts.push_back(jevois::async([this](std::shared_ptr<BlobDetector> b)
199  {
200  auto c = b->detect(itsImgHsv);
201  std::lock_guard<std::mutex> _(itsBlobMtx);
202  itsContours[b->instanceName()] = std::move(c);
203  }, b));
204  }
205 
206  // ####################################################################################################
207  //! Gather our blob threads and send/draw the results
208  // ####################################################################################################
209  void sendBlobs(unsigned int w, unsigned int h, jevois::RawImage * outimg = nullptr)
210  {
211  for (auto & f : itsBlobFuts)
212  try { f.get(); } catch (...) { LERROR("Ooops, some blob detector threw -- IGNORED"); }
213  itsBlobFuts.clear();
214 
215  // Send a serial message for each detected blob:
216  for (auto const & cc : itsContours)
217  for (auto const & c : cc.second)
218  sendSerialContour2D(w, h, c, cc.first);
219  }
220 
221  // ####################################################################################################
222  //! Detect ArUcos
223  // ####################################################################################################
224  void detectArUco(cv::Mat cvimg, std::vector<int> & ids, std::vector<std::vector<cv::Point2f>> & corners,
225  std::vector<cv::Vec3d> & rvecs, std::vector<cv::Vec3d> & tvecs,
226  unsigned int h, jevois::RawImage * outimg = nullptr)
227  {
228  itsArUco->detectMarkers(cvimg, ids, corners);
229 
230  if (itsArUco->dopose::get() && ids.empty() == false)
231  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
232 
233  // Show all the results:
234  if (outimg) itsArUco->drawDetections(*outimg, 3, h+2, ids, corners, rvecs, tvecs);
235  }
236 
237  // ####################################################################################################
238  //! Processing function, no USB video output
239  // ####################################################################################################
240  virtual void process(jevois::InputFrame && inframe) override
241  {
242  // Wait for next available camera image. Any resolution and format ok:
243  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
244 
245  // Convert input image to BGR24, then to HSV:
246  cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
247  cv::cvtColor(imgbgr, itsImgHsv, cv::COLOR_BGR2HSV);
248 
249  // Detect blobs in parallel threads:
250  detectBlobs();
251 
252  // In our thread, detect ArUcos; first convert to gray:
253  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
254 
255  // Let camera know we are done processing the input image:
256  inframe.done();
257 
258  // Detect ArUcos:
259  std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
260  detectArUco(cvimg, ids, corners, rvecs, tvecs, h);
261 
262  // Send ArUco serial output:
263  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
264 
265  // Done with ArUco, gather the blobs and send the serial messages:
266  sendBlobs(w, h);
267  }
268 
269  // ####################################################################################################
270  //! Processing function, with USB video output
271  // ####################################################################################################
272  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
273  {
274  static jevois::Timer timer("processing");
275 
276  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
277  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
278  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
279 
280  timer.start();
281 
282  // While we process it, start a thread to wait for output frame and paste the input image into it:
283  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
284  auto paste_fut = jevois::async([&]() {
285  outimg = outframe.get();
286  outimg.require("output", w, h + 26, inimg.fmt);
287  jevois::rawimage::paste(inimg, outimg, 0, 0);
288  jevois::rawimage::writeText(outimg, "JeVois ArUco + Color Blobs", 3, 3, jevois::yuyv::White);
289  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
290  });
291 
292  // Convert input image to BGR24, then to HSV:
293  cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
294  cv::cvtColor(imgbgr, itsImgHsv, cv::COLOR_BGR2HSV);
295 
296  // Detect blobs in parallel threads:
297  detectBlobs(&outimg);
298 
299  // In our thread, detect ArUcos; first convert to gray:
300  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
301 
302  // Let camera know we are done processing the input image:
303  inframe.done();
304 
305  // Wait for paste to finish up:
306  paste_fut.get();
307 
308  // Detect ArUcos:
309  std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
310  detectArUco(cvimg, ids, corners, rvecs, tvecs, h, &outimg);
311 
312  // Send ArUco serial output:
313  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
314 
315  // Done with ArUco, gather the blobs and send the serial messages:
316  sendBlobs(w, h);
317 
318  // Draw all detected contours in a thread:
319  std::future<void> draw_fut = jevois::async([&]() {
320  // We reinterpret the top portion of our YUYV output image as an opencv 8UC2 image:
321  cv::Mat outuc2 = jevois::rawimage::cvImage(outimg); // pixel data shared
322  for (auto const & cc : itsContours)
323  {
324  int color = (cc.first.back() - '0') * 123;
325  cv::drawContours(outuc2, cc.second, -1, color, 2, 8);
326  for (auto const & cont : cc.second)
327  {
328  cv::Moments moment = cv::moments(cont);
329  double const area = moment.m00;
330  int const x = int(moment.m10 / area + 0.4999);
331  int const y = int(moment.m01 / area + 0.4999);
332  jevois::rawimage::drawCircle(outimg, x, y, 20, 1, color);
333  }
334  }
335  });
336 
337  // Show number of detected objects:
338  std::string str = "Detected ";
339  for (auto const & cc : itsContours) str += std::to_string(cc.second.size()) + ' ';
340  jevois::rawimage::writeText(outimg, str + "blobs.", 3, h + 14, jevois::yuyv::White);
341 
342  // Show processing fps:
343  std::string const & fpscpu = timer.stop();
344  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
345 
346  // Wait until all contours are drawn, if they had been requested:
347  draw_fut.get();
348 
349  // Send the output image with our processing results to the host over USB:
350  outframe.send();
351  }
352 
353  // ####################################################################################################
354  protected:
355  std::shared_ptr<ArUco> itsArUco;
356  std::vector<std::shared_ptr<BlobDetector> > itsBlobs;
357  cv::Mat itsImgHsv;
358  std::map<std::string, std::vector<std::vector<cv::Point>>> itsContours;
359  std::vector<std::future<void>> itsBlobFuts;
360  std::mutex itsBlobMtx;
361 };
362 
363 // Allow the module to be loaded as a shared object (.so) file:
jevois::Component::instanceName
const std::string & instanceName() const
jevois::StdModule::sendSerialContour2D
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
jevois::OutputFrame
BlobDetector::detect
std::vector< std::vector< cv::Point > > detect(cv::Mat const &imghsv)
Detect blobs, each is represented as a contour (vector of (x,y) coordinates)
Definition: BlobDetector.C:29
Timer.H
ArUcoBlob::itsBlobMtx
std::mutex itsBlobMtx
Definition: ArUcoBlob.C:360
ArUcoBlob::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no USB video output.
Definition: ArUcoBlob.C:240
Module.H
ArUcoBlob::itsBlobs
std::vector< std::shared_ptr< BlobDetector > > itsBlobs
Definition: ArUcoBlob.C:356
BlobDetector.H
Coordinates.H
Log.H
jevois::RawImage
ArUcoBlob::postInit
void postInit() override
Post-init: instantiate the blob detectors.
Definition: ArUcoBlob.C:167
jevois::rawimage::drawCircle
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
jevois::Timer::start
void start()
ArUcoBlob::itsArUco
std::shared_ptr< ArUco > itsArUco
Definition: ArUcoBlob.C:355
jevois::Component::removeSubComponent
void removeSubComponent(std::shared_ptr< Comp > &component)
jevois::rawimage::convertToCvBGR
cv::Mat convertToCvBGR(RawImage const &src)
jevois::ParameterCategory
LERROR
#define LERROR(msg)
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
ArUcoBlob::~ArUcoBlob
virtual ~ArUcoBlob()
Virtual destructor for safe inheritance.
Definition: ArUcoBlob.C:187
ArUcoBlob::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, with USB video output.
Definition: ArUcoBlob.C:272
jevois::RawImage::width
unsigned int width
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
ArUcoBlob::itsBlobFuts
std::vector< std::future< void > > itsBlobFuts
Definition: ArUcoBlob.C:359
area
double area(const std::vector< Point2D< T > > &polygon, const bool getsigned=false)
What is the area of a polygon?
Definition: Point2D.H:422
ArUcoBlob
Combined ArUco marker + multiple color-based object detection.
Definition: ArUcoBlob.C:150
jevois::Timer::stop
const std::string & stop(double *seconds)
ArUcoBlob::ArUcoBlob
ArUcoBlob(std::string const &instance)
Constructor.
Definition: ArUcoBlob.C:157
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::StdModule::StdModule
StdModule(std::string const &instance)
ArUcoBlob::itsImgHsv
cv::Mat itsImgHsv
Definition: ArUcoBlob.C:357
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
RawImageOps.H
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(ArUcoBlob)
jevois::RawImage::height
unsigned int height
to_string
std::string to_string(T const &val)
jevois::InputFrame
jevois::rawimage::cvImage
cv::Mat cvImage(RawImage const &src)
ArUcoBlob::detectArUco
void detectArUco(cv::Mat cvimg, std::vector< int > &ids, std::vector< std::vector< cv::Point2f >> &corners, std::vector< cv::Vec3d > &rvecs, std::vector< cv::Vec3d > &tvecs, unsigned int h, jevois::RawImage *outimg=nullptr)
Detect ArUcos.
Definition: ArUcoBlob.C:224
Utils.H
ArUcoBlob::sendBlobs
void sendBlobs(unsigned int w, unsigned int h, jevois::RawImage *outimg=nullptr)
Gather our blob threads and send/draw the results.
Definition: ArUcoBlob.C:209
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
ArUcoBlob::itsContours
std::map< std::string, std::vector< std::vector< cv::Point > > > itsContours
Definition: ArUcoBlob.C:358
ArUco.H
test-model.color
color
Definition: test-model.py:38
h
int h
ArUcoBlob::preUninit
void preUninit() override
Definition: ArUcoBlob.C:178
freeze
void freeze()
jevois::StdModule
ARtoolkit::JEVOIS_DECLARE_PARAMETER
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .dat extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory (if relative stem) or " "from the specified absolute location (if absolute stem).", JEVOIS_SHARE_PATH "/camera/camera_para", ParamCateg)
Parameter.
ArUcoBlob::detectBlobs
void detectBlobs(jevois::RawImage *outimg=nullptr)
Detect blobs in parallel threads.
Definition: ArUcoBlob.C:192
jevois::Timer