JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
MarkersCombo.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, BA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
24 #include <opencv2/core/core.hpp>
25 
26 //! Simple demo of QRcode + ARtoolkit + ArUco markers detection and decoding
27 /*! Detect and decode 3 kinds of coded patterns which can be useful to a robot:
28 
29  - QR-codes as in \jvmod{DemoQRcode}
30  - AR-toolkit markers as in \jvmod{DemoARtoolkit}
31  - ArUco markers as in \jvmod{DemoArUco}
32 
33  The three algorithms run in parallel. You should be able to sustain 50 frames/s at 320x240 camera resolution, and 20
34  frames/s at 640x480 camera resolution.
35 
36  Serial Messages
37  ---------------
38 
39  This module can send standardized serial messages as described in \ref UserSerialStyle.
40 
41  When \p dopose is turned on, 3D messages will be sent, otherwise 2D messages.
42 
43  One message is issued for every detected marker, on every video frame.
44 
45  2D messages when \p msg3d and \p dopose are off:
46 
47  - Serial message type: \b 2D
48  - `id`: decoded marker ID (with prefix 'U' for ArUco, or 'A' for ARtoolkit),
49  or type of symbol (e.g., \b QR-Code, \b ISBN13, etc).
50  - `x`, `y`, or vertices: standardized 2D coordinates of marker center or corners
51  - `w`, `h`: standardized marker size
52  - `extra`: none (empty string) for ArUco and ARtoolkit markers, otherwise decoded barcode or QRcode content.
53 
54  3D messages when \p msg3d and \p dopose are on:
55 
56  - Serial message type: \b 3D
57  - `id`: decoded marker ID (with prefix 'U' for ArUco, or 'A' for ARtoolkit),
58  or type of symbol (e.g., \b QR-Code, \b ISBN13, etc).
59  - `x`, `y`, `z`, or vertices: 3D coordinates in millimeters of marker center or corners
60  - `w`, `h`, `d`: marker size in millimeters, a depth of 1mm is always used
61  - `extra`: none (empty string) for ArUco and ARtoolkit markers, otherwise decoded barcode or QRcode content.
62 
63  If you will use the quaternion data (Detail message style; see \ref UserSerialStyle), you should probably set the \p
64  serprec parameter to something non-zero to get enough accuracy in the quaternion values.
65 
66  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
67  standardized coordinates.
68 
69 
70  @author Laurent Itti
71 
72  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois MarkersCombo
73  @videomapping YUYV 320 306 50.0 YUYV 320 240 50.0 JeVois MarkersCombo
74  @videomapping YUYV 640 546 20.0 YUYV 640 480 20.0 JeVois MarkersCombo
75  @email itti\@usc.edu
76  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
77  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
78  @mainurl http://jevois.org
79  @supporturl http://jevois.org/doc
80  @otherurl http://iLab.usc.edu
81  @license GPL v3
82  @distribution Unrestricted
83  @restrictions None
84  \ingroup modules */
86 {
87  public:
88  // ####################################################################################################
89  //! Constructor
90  // ####################################################################################################
91  MarkersCombo(std::string const & instance) : jevois::StdModule(instance)
92  {
93  itsArUco = addSubComponent<ArUco>("aruco");
94  itsQRcode = addSubComponent<QRcode>("qrcode");
95  itsARtoolkit = addSubComponent<ARtoolkit>("artoolkit");
96  }
97 
98  // ####################################################################################################
99  //! Virtual destructor for safe inheritance
100  // ####################################################################################################
101  virtual ~MarkersCombo()
102  { }
103 
104  // ####################################################################################################
105  //! Processing function, no video output
106  // ####################################################################################################
107  virtual void process(jevois::InputFrame && inframe) override
108  {
109  // Wait for next available camera image as grayscale:
110  cv::Mat cvimg = inframe.getCvGRAY();
111 
112  // Launch QRcode:
113  auto qr_fut = jevois::async([&]() {
114  zbar::Image zgray(cvimg.cols, cvimg.rows, "Y800", cvimg.data, cvimg.total());
115  itsQRcode->process(zgray);
116  itsQRcode->sendSerial(this, zgray, cvimg.cols, cvimg.rows);
117  zgray.set_data(nullptr, 0);
118  });
119 
120  // Launch AR toolkit:
121  auto ar_fut = jevois::async([&]() {
122  itsARtoolkit->detectMarkers(cvimg);
123  itsARtoolkit->sendSerial(this);
124  });
125 
126  // Process ArUco in the main thread:
127  std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
128  itsArUco->detectMarkers(cvimg, ids, corners);
129  if (itsArUco->dopose::get() && ids.empty() == false) itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
130  itsArUco->sendSerial(this, ids, corners, cvimg.cols, cvimg.rows, rvecs, tvecs);
131  }
132 
133  // ####################################################################################################
134  //! Processing function with video output to USB
135  // ####################################################################################################
136  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
137  {
138  static jevois::Timer timer("processing", 100, LOG_DEBUG);
139 
140  // Wait for next available camera image:
141  jevois::RawImage const inimg = inframe.get();
142 
143  timer.start();
144 
145  // We only handle one specific pixel format, any size in this demo:
146  unsigned int const w = inimg.width, h = inimg.height;
147  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
148 
149  // While we process it, start a thread to wait for out frame and paste the input into it. It will hold a unique
150  // lock onto mtx. Other threads should attempt a shared_lock onto mtx before they draw into outimg:
151  jevois::RawImage outimg; //boost::shared_mutex mtx; boost::unique_lock<boost::shared_mutex> ulck(mtx);
152  auto paste_fut = jevois::async([&]() {
153  outimg = outframe.get();
154  outimg.require("output", w, h + 66, inimg.fmt);
155  jevois::rawimage::paste(inimg, outimg, 0, 0);
156  //ulck.unlock();
157  jevois::rawimage::writeText(outimg, "JeVois Makers Combo", 3, 3, jevois::yuyv::White);
158  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, jevois::yuyv::Black);
159  inframe.done();
160  });
161 
162  // Convert the image to grayscale:
163  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
164 
165  // Launch QRcode:
166  auto qr_fut = jevois::async([&]() {
167  zbar::Image zgray(cvimg.cols, cvimg.rows, "Y800", cvimg.data, cvimg.total());
168  itsQRcode->process(zgray);
169  itsQRcode->sendSerial(this, zgray, w, h);
170  //boost::shared_lock<boost::shared_mutex> _(mtx); // wait until paste is complete
171  itsQRcode->drawDetections(outimg, 3, h + 23, zgray, w, h, 3);
172  zgray.set_data(nullptr, 0);
173  });
174 
175  // Launch AR toolkit:
176  auto ar_fut = jevois::async([&]() {
177  itsARtoolkit->detectMarkers(cvimg);
178  itsARtoolkit->sendSerial(this);
179  //boost::shared_lock<boost::shared_mutex> _(mtx);
180  itsARtoolkit->drawDetections(outimg, 3, h + 13); // wait until paste is complete
181  });
182 
183  // Process ArUco in the main thread:
184  std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
185  itsArUco->detectMarkers(cvimg, ids, corners);
186  if (itsArUco->dopose::get() && ids.empty() == false) itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
187 
188  // Show aruco results and send serial:
189  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
190  //boost::shared_lock<boost::shared_mutex> _(mtx);
191  itsArUco->drawDetections(outimg, 3, h + 3, ids, corners, rvecs, tvecs);
192 
193  // Show processing fps:
194  std::string const & fpscpu = timer.stop();
195  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
196 
197  // Wait for all threads to finish up:
198  qr_fut.get();
199  ar_fut.get();
200  paste_fut.get();
201 
202  // Send the output image with our processing results to the host over USB:
203  outframe.send();
204  }
205 
206 #ifdef JEVOIS_PRO
207  // ####################################################################################################
208  //! Processing function with GUI output
209  // ####################################################################################################
210  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
211  {
212  static jevois::Timer timer("processing", 100, LOG_DEBUG);
213 
214  // Start the GUI frame:
215  unsigned short winw, winh;
216  bool idle = helper.startFrame(winw, winh);
217 
218  // Draw the camera frame:
219  int x = 0, y = 0; unsigned short iw = 0, ih = 0;
220  helper.drawInputFrame("camera", inframe, x, y, iw, ih);
221 
222  // Wait for next available camera image:
223  jevois::RawImage const inimg = inframe.getp();
224  unsigned int const w = inimg.width, h = inimg.height;
225  helper.itext("JeVois-Pro ArUco + ARtoolkit + QRcode/Barcode Detection");
226 
227  timer.start();
228 
229  // Convert the image to grayscale:
230  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
231 
232  // Launch QRcode:
233  zbar::Image zgray(cvimg.cols, cvimg.rows, "Y800", cvimg.data, cvimg.total());
234  auto qr_fut = jevois::async([&]() {
235  itsQRcode->process(zgray);
236  itsQRcode->sendSerial(this, zgray, w, h);
237  });
238 
239  // Launch AR toolkit:
240  auto ar_fut = jevois::async([&]() {
241  itsARtoolkit->detectMarkers(cvimg);
242  itsARtoolkit->sendSerial(this);
243  });
244 
245  // Process ArUco in the main thread:
246  std::vector<int> ids; std::vector<std::vector<cv::Point2f> > corners; std::vector<cv::Vec3d> rvecs, tvecs;
247  itsArUco->detectMarkers(cvimg, ids, corners);
248  if (itsArUco->dopose::get() && ids.empty() == false) itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
249 
250  // Show aruco results and send serial:
251  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
252  itsArUco->drawDetections(helper, ids, corners, rvecs, tvecs);
253 
254  // Wait for all threads to finish up, draw results:
255  ar_fut.get();
256  itsARtoolkit->drawDetections(helper);
257 
258  qr_fut.get();
259  itsQRcode->drawDetections(helper, zgray, w, h);
260  zgray.set_data(nullptr, 0);
261 
262  // Let camera know we are done processing the input image:
263  inframe.done();
264 
265  // Show processing fps:
266  std::string const & fpscpu = timer.stop();
267  helper.iinfo(inframe, fpscpu, winw, winh);
268 
269  // Render the image and GUI:
270  helper.endFrame();
271  }
272 #endif
273 
274  // ####################################################################################################
275  protected:
276  std::shared_ptr<ArUco> itsArUco;
277  std::shared_ptr<QRcode> itsQRcode;
278  std::shared_ptr<ARtoolkit> itsARtoolkit;
279 };
280 
281 // Allow the module to be loaded as a shared object (.so) file:
jevois::OutputFrame
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
Timer.H
jevois::GUIhelper::itext
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
Module.H
jevois::GUIhelper
MarkersCombo::MarkersCombo
MarkersCombo(std::string const &instance)
Constructor.
Definition: MarkersCombo.C:91
jevois::GUIhelper::endFrame
void endFrame()
MarkersCombo::itsQRcode
std::shared_ptr< QRcode > itsQRcode
Definition: MarkersCombo.C:277
jevois::RawImage
jevois::Timer::start
void start()
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
MarkersCombo::~MarkersCombo
virtual ~MarkersCombo()
Virtual destructor for safe inheritance.
Definition: MarkersCombo.C:101
MarkersCombo::itsArUco
std::shared_ptr< ArUco > itsArUco
Definition: MarkersCombo.C:276
jevois::RawImage::width
unsigned int width
MarkersCombo::itsARtoolkit
std::shared_ptr< ARtoolkit > itsARtoolkit
Definition: MarkersCombo.C:278
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::StdModule::StdModule
StdModule(std::string const &instance)
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
MarkersCombo
Simple demo of QRcode + ARtoolkit + ArUco markers detection and decoding.
Definition: MarkersCombo.C:85
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
jevois::InputFrame
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
MarkersCombo::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: MarkersCombo.C:136
jevois::RawImage::fmt
unsigned int fmt
ArUco.H
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(MarkersCombo)
h
int h
jevois::StdModule
QRcode.H
MarkersCombo::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition: MarkersCombo.C:107
ARtoolkit.H
jevois::Timer