JeVoisBase  1.22
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
ArUcoBlob.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2018 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17#include <jevois/Core/Module.H>
20#include <jevois/Debug/Log.H>
21#include <jevois/Util/Utils.H>
23#include <jevois/Debug/Timer.H>
25
26#include <opencv2/imgproc/imgproc.hpp>
27
28#include <map>
29
30static jevois::ParameterCategory const ParamCateg("ArUcoBlob Options");
31
32//! Parameter \relates ArUcoBlob
33JEVOIS_DECLARE_PARAMETER(numtrack, size_t, "Number of parallel blob trackers to run. They will be named blob0, "
34 "blob1, etc for parameters and serial messages",
35 3, ParamCateg);
36
37//! Combined ArUco marker + multiple color-based object detection
38/*! This modules 1) detects ArUco markers (small black-and-white geometric patterns which can be used as tags for some
39 objects), and, in parallel, 2) isolates pixels within multiple given HSV ranges (hue, saturation, and value of color
40 pixels), does some cleanups, and extracts object contours. It sends information about detected ArUco tags and color
41 objects over serial.
42
43 This module was developed to allow students to easily develop visually-guided robots that can at the same time
44 detect ArUco markers placed in the environment to signal certain key objects (e.g., charging station, home base) and
45 colored objects of different kinds (e.g., blue people, green trees, and yellow fires).
46
47 This module usually works best with the camera sensor set to manual exposure, manual gain, manual color balance, etc
48 so that HSV color values are reliable. See the file \b script.cfg file in this module's directory for an example of
49 how to set the camera settings each time this module is loaded.
50
51 Since this is a combination module, refer to:
52
53 - \jvmod{DemoArUco} for the ArUco algorithm and messages
54 - \jvmod{ObjectTracker} for the blob detection algorithm and messages
55
56 The number of parallel blob trackers is determined by parameter \p numtrack, which should be set before the module
57 is initialized, i.e., in the module's \b params.cfg file. It cannot be changed while the module is running.
58
59 The module runs at about 50 frames/s with 3 parallel blob detectors plus ArUco, at 320x240 camera sensor
60 resolution. Increasing to 10 parallel blob detectors will still get you about 25 frames/s (but finding robust
61 non-overlapping HSV ranges for all those detectors will become challenging!)
62
63 To configure parameters \p hrange, \p srange, and \p vrange for each detector in the module's \b scrip.cfg, we
64 recommend that you do it one by one for each kind of colored object you want, using the \jvmod{ObjectTracker} module
65 (which shares the same color blob detection code, just for one HSV range) and the tutorial on <A
66 HREF="http://jevois.org/tutorials/UserColorTracking.html">Tuning the color-based object tracker using a python
67 graphical interface</A>, or the sliders in JeVois Inventor. Just make sure that both modules have the same camera
68 settings in their respective \b script.cfg files.
69
70 Using the serial outputs
71 ------------------------
72
73 We recommend the following settings (to apply after you load the module, for example in the module's \b script.cfg
74 file):
75 \code{.py}
76 setpar serout USB # to send to serial-over-USB, or use Hard to send to 4-pin serial port
77 setpar serstyle Normal # to get ID, center location, size for every detected color blob and ArUco tag
78 setpar serstamp Frame # to add video frame number to all messages
79 \endcode
80
81 With a scene as shown in this module's screenshots, you would then get outputs like:
82
83 \verbatim
84 ...
85 1557 N2 U42 -328 -9 706 569
86 1557 N2 U18 338 -241 613 444
87 1557 N2 blob0 616 91 406 244
88 1557 N2 blob1 28 584 881 331
89 1557 N2 blob2 47 -553 469 206
90 1558 N2 U42 -328 -9 706 569
91 1558 N2 U18 338 -241 613 444
92 1558 N2 blob0 547 113 519 275
93 1558 N2 blob1 28 581 881 338
94 1558 N2 blob2 47 -553 469 206
95 1559 N2 U42 -331 -13 700 563
96 1559 N2 U18 338 -244 613 450
97 1559 N2 blob0 369 153 200 194
98 1559 N2 blob0 616 94 381 250
99 1559 N2 blob1 28 581 881 338
100 1559 N2 blob2 47 -553 469 206
101 ...
102 \endverbatim
103
104 which basically means that, on frame 1557, ArUco markers U42 and U18 were detected, then blob detector named "blob0"
105 (configured for light blue objects in \b script.cfg) detected one blob, then "blob1" (configured for yellow) also
106 detected one, and finally "blob2" (configured for green) found one too. That was all for frame 1157, and we then
107 switch to frame 1158 (with essentially the same detections), then frame 1159 (note how blob0 detected 2 different
108 blobs on that frame), and so on. See \ref UserSerialStyle for more info about these messages.
109
110 See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
111 standardized coordinates.
112
113 Running with no video output (standalone mode)
114 ----------------------------------------------
115
116 Try these settings in the global initialization script file of JeVois, which is executed when JeVois powers up, in
117 <b>JEVOIS:/config/initscript.cfg</b>:
118
119 \code{.py}
120 setmapping2 YUYV 320 240 45.0 JeVois ArUcoBlob # to select this module upon power up
121 setpar serout Hard # to send detection messages to 4-pin serial port
122 setpar serstyle Normal # to get ID, center location, size
123 setpar serstamp Frame # to add video frame number to all messages
124 streamon # start capturing and processing camera sensor data
125 \endcode
126
127 Make sure you do not have conflicting settings in the module's \b params.cfg or \b script.cfg file; as a reminder,
128 the order of execution is: 1) \b initscript.cfg runs, which loads the module through the `setmapping2` command; 2)
129 as part of the loading process and before the module is initialized, settings in \b params.cfg are applied; 3) the
130 module is then initialized and commands in \b script.cfg are run; 4) the additional commands following `setmapping2`
131 in \b initscript.cfg are finally run. Next time JeVois starts up, it will automatically load this module and start
132 sending messages to the hardware 4-pin serial port, which you should then connect to an Arduino or other robot
133 controller.
134
135 @author Laurent Itti
136
137 @displayname ArUco Blob
138 @videomapping YUYV 320 266 30.0 YUYV 320 240 30.0 JeVois ArUcoBlob
139 @videomapping NONE 0 0 0.0 YUYV 320 240 30.0 JeVois ArUcoBlob
140 @email itti\@usc.edu
141 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
142 @copyright Copyright (C) 2018 by Laurent Itti, iLab and the University of Southern California
143 @mainurl http://jevois.org
144 @supporturl http://jevois.org/doc
145 @otherurl http://iLab.usc.edu
146 @license GPL v3
147 @distribution Unrestricted
148 @restrictions None
149 \ingroup modules */
151 public jevois::Parameter<numtrack>
152{
153 public:
154 // ####################################################################################################
155 //! Constructor
156 // ####################################################################################################
157 ArUcoBlob(std::string const & instance) :
158 jevois::StdModule(instance)
159 {
160 itsArUco = addSubComponent<ArUco>("aruco");
161 // We instantiate the blob detectors in postInit() once their number is finalized
162 }
163
164 // ####################################################################################################
165 //! Post-init: instantiate the blob detectors
166 // ####################################################################################################
167 void postInit() override
168 {
169 numtrack::freeze(true);
170
171 for (size_t i = 0; i < numtrack::get(); ++i)
172 itsBlobs.push_back(addSubComponent<BlobDetector>("blob" + std::to_string(i)));
173 }
174
175 // ####################################################################################################
176 // Pre-unInit: release the blob detectors
177 // ####################################################################################################
178 void preUninit() override
179 {
180 for (auto & b : itsBlobs) removeSubComponent(b);
181 itsBlobs.clear();
182 }
183
184 // ####################################################################################################
185 //! Virtual destructor for safe inheritance
186 // ####################################################################################################
187 virtual ~ArUcoBlob() { }
188
189 // ####################################################################################################
190 //! Detect blobs in parallel threads
191 // ####################################################################################################
193 {
194 itsContours.clear();
195
196 // In a bunch of threads, detect blobs and get the contours:
197 for (auto & b : itsBlobs)
198 itsBlobFuts.push_back(jevois::async([this](std::shared_ptr<BlobDetector> b)
199 {
200 auto c = b->detect(itsImgHsv);
201 std::lock_guard<std::mutex> _(itsBlobMtx);
202 itsContours[b->instanceName()] = std::move(c);
203 }, b));
204 }
205
206 // ####################################################################################################
207 //! Gather our blob threads and send/draw the results
208 // ####################################################################################################
209 void sendBlobs(unsigned int w, unsigned int h)
210 {
211 for (auto & f : itsBlobFuts)
212 try { f.get(); } catch (...) { LERROR("Ooops, some blob detector threw -- IGNORED"); }
213 itsBlobFuts.clear();
214
215 // Send a serial message for each detected blob:
216 for (auto const & cc : itsContours)
217 for (auto const & c : cc.second)
218 sendSerialContour2D(w, h, c, cc.first);
219 }
220
221 // ####################################################################################################
222 //! Detect ArUcos
223 // ####################################################################################################
224 void detectArUco(cv::Mat cvimg, std::vector<int> & ids, std::vector<std::vector<cv::Point2f>> & corners,
225 std::vector<cv::Vec3d> & rvecs, std::vector<cv::Vec3d> & tvecs,
226 unsigned int h, jevois::RawImage * outimg = nullptr)
227 {
228 itsArUco->detectMarkers(cvimg, ids, corners);
229
230 if (itsArUco->dopose::get() && ids.empty() == false)
231 itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
232
233 // Show all the results:
234 if (outimg) itsArUco->drawDetections(*outimg, 3, h+2, ids, corners, rvecs, tvecs);
235 }
236
237 // ####################################################################################################
238 //! Processing function, no USB video output
239 // ####################################################################################################
240 virtual void process(jevois::InputFrame && inframe) override
241 {
242 // Wait for next available camera image. Any resolution and format ok:
243 jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
244
245 // Convert input image to BGR24, then to HSV:
246 cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
247 cv::cvtColor(imgbgr, itsImgHsv, cv::COLOR_BGR2HSV);
248
249 // Detect blobs in parallel threads:
250 detectBlobs();
251
252 // In our thread, detect ArUcos; first convert to gray:
253 cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
254
255 // Let camera know we are done processing the input image:
256 inframe.done();
257
258 // Detect ArUcos:
259 std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
260 detectArUco(cvimg, ids, corners, rvecs, tvecs, h);
261
262 // Send ArUco serial output:
263 itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
264
265 // Done with ArUco, gather the blobs and send the serial messages:
266 sendBlobs(w, h);
267 }
268
269 // ####################################################################################################
270 //! Processing function, with USB video output
271 // ####################################################################################################
272 virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
273 {
274 static jevois::Timer timer("processing");
275
276 // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
277 jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
278 inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
279
280 timer.start();
281
282 // While we process it, start a thread to wait for output frame and paste the input image into it:
283 jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
284 auto paste_fut = jevois::async([&]() {
285 outimg = outframe.get();
286 outimg.require("output", w, h + 26, inimg.fmt);
287 jevois::rawimage::paste(inimg, outimg, 0, 0);
288 jevois::rawimage::writeText(outimg, "JeVois ArUco + Color Blobs", 3, 3, jevois::yuyv::White);
289 jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
290 });
291
292 // Convert input image to BGR24, then to HSV:
293 cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
294 cv::cvtColor(imgbgr, itsImgHsv, cv::COLOR_BGR2HSV);
295
296 // Detect blobs in parallel threads:
297 detectBlobs();
298
299 // In our thread, detect ArUcos; first convert to gray:
300 cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
301
302 // Let camera know we are done processing the input image:
303 inframe.done();
304
305 // Wait for paste to finish up:
306 paste_fut.get();
307
308 // Detect ArUcos:
309 std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
310 detectArUco(cvimg, ids, corners, rvecs, tvecs, h, &outimg);
311
312 // Send ArUco serial output:
313 itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
314
315 // Done with ArUco, gather the blobs and send the serial messages:
316 sendBlobs(w, h);
317
318 // Draw all detected contours in a thread:
319 std::future<void> draw_fut = jevois::async([&]() {
320 // We reinterpret the top portion of our YUYV output image as an opencv 8UC2 image:
321 cv::Mat outuc2 = jevois::rawimage::cvImage(outimg); // pixel data shared
322 for (auto const & cc : itsContours)
323 {
324 int color = (cc.first.back() - '0') * 123;
325 cv::drawContours(outuc2, cc.second, -1, color, 2, 8);
326 for (auto const & cont : cc.second)
327 {
328 cv::Moments moment = cv::moments(cont);
329 double const area = moment.m00;
330 int const x = int(moment.m10 / area + 0.4999);
331 int const y = int(moment.m01 / area + 0.4999);
332 jevois::rawimage::drawCircle(outimg, x, y, 20, 1, color);
333 }
334 }
335 });
336
337 // Show number of detected objects:
338 std::string str = "Detected ";
339 for (auto const & cc : itsContours) str += std::to_string(cc.second.size()) + ' ';
340 jevois::rawimage::writeText(outimg, str + "blobs.", 3, h + 14, jevois::yuyv::White);
341
342 // Show processing fps:
343 std::string const & fpscpu = timer.stop();
344 jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
345
346 // Wait until all contours are drawn, if they had been requested:
347 draw_fut.get();
348
349 // Send the output image with our processing results to the host over USB:
350 outframe.send();
351 }
352
353 // ####################################################################################################
354 protected:
355 std::shared_ptr<ArUco> itsArUco;
356 std::vector<std::shared_ptr<BlobDetector> > itsBlobs;
357 cv::Mat itsImgHsv;
358 std::map<std::string, std::vector<std::vector<cv::Point>>> itsContours;
359 std::vector<std::future<void>> itsBlobFuts;
360 std::mutex itsBlobMtx;
361};
362
363// Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE(ArUcoBlob)
int h
double area(const std::vector< Point2D< T > > &polygon, const bool getsigned=false)
What is the area of a polygon?
Definition Point2D.H:422
Combined ArUco marker + multiple color-based object detection.
Definition ArUcoBlob.C:152
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no USB video output.
Definition ArUcoBlob.C:240
std::vector< std::future< void > > itsBlobFuts
Definition ArUcoBlob.C:359
virtual ~ArUcoBlob()
Virtual destructor for safe inheritance.
Definition ArUcoBlob.C:187
void postInit() override
Post-init: instantiate the blob detectors.
Definition ArUcoBlob.C:167
std::map< std::string, std::vector< std::vector< cv::Point > > > itsContours
Definition ArUcoBlob.C:358
void sendBlobs(unsigned int w, unsigned int h)
Gather our blob threads and send/draw the results.
Definition ArUcoBlob.C:209
ArUcoBlob(std::string const &instance)
Constructor.
Definition ArUcoBlob.C:157
std::shared_ptr< ArUco > itsArUco
Definition ArUcoBlob.C:355
void detectBlobs()
Detect blobs in parallel threads.
Definition ArUcoBlob.C:192
JEVOIS_DECLARE_PARAMETER(numtrack, size_t, "Number of parallel blob trackers to run. They will be named blob0, " "blob1, etc for parameters and serial messages", 3, ParamCateg)
Parameter.
cv::Mat itsImgHsv
Definition ArUcoBlob.C:357
std::vector< std::shared_ptr< BlobDetector > > itsBlobs
Definition ArUcoBlob.C:356
void preUninit() override
Definition ArUcoBlob.C:178
void detectArUco(cv::Mat cvimg, std::vector< int > &ids, std::vector< std::vector< cv::Point2f > > &corners, std::vector< cv::Vec3d > &rvecs, std::vector< cv::Vec3d > &tvecs, unsigned int h, jevois::RawImage *outimg=nullptr)
Detect ArUcos.
Definition ArUcoBlob.C:224
std::mutex itsBlobMtx
Definition ArUcoBlob.C:360
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, with USB video output.
Definition ArUcoBlob.C:272
void removeSubComponent(std::shared_ptr< Comp > &component)
unsigned int fmt
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
StdModule(std::string const &instance)
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
std::string const & stop(double *seconds)
#define LERROR(msg)
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
cv::Mat cvImage(RawImage const &src)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
cv::Mat convertToCvGray(RawImage const &src)
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
cv::Mat convertToCvBGR(RawImage const &src)
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
unsigned short constexpr White