JeVoisBase  1.18
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoArUco.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, BA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
22 #include <opencv2/core/core.hpp>
23 
24 //! Simple demo of ArUco augmented reality markers detection and decoding
25 /*! Detect and decode patterns known as ArUco markers, which are small 2D barcodes often used in augmented
26  reality and robotics.
27 
28  ArUco markers are small 2D barcodes. Each ArUco marker corresponds to a number, encoded into a small grid of black
29  and white pixels. The ArUco decoding algorithm is capable of locating, decoding, and of estimating the pose
30  (location and orientation in space) of any ArUco markers in the camera's field of view.
31 
32  ArUco markers are very useful as tags for many robotics and augmented reality applications. For example, one may
33  place an ArUco next to a robot's charging station, an elevator button, or an object that a robot should manipulate.
34 
35  The implementation of ArUco used by JeVois is the one of OpenCV-Contrib, documented here:
36  https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html
37 
38  ArUco markers can be created with several standard dictionaries. Different dictionaries give rise to different
39  numbers of pixels in the markers, and to different numbers of possible symbols that can be created using the
40  dictionary. The default dictionary used by JeVois is 4x4 with 50 symbols. Other dictionaries are also supported by
41  setting the parameter \p dictionary over serial port or in a config file, up to 7x7 with 1000 symbols.
42 
43  Creating and printing markers
44  -----------------------------
45 
46  We have created the 50 markers available in the default dictionary (4x4_50) as PNG images that you can download and
47  print, at http://jevois.org/data/ArUco.zip
48 
49  To make your own, for example, using another dictionary, see the documentation of the \ref ArUco component of
50  JeVoisBase. Some utilities are provided with the component.
51 
52  Serial Messages
53  ---------------
54 
55  This module can send standardized serial messages as described in \ref UserSerialStyle.
56 
57  When \p dopose is turned on, 3D messages will be sent, otherwise 2D messages.
58 
59  One message is issued for every detected ArUco, on every video frame.
60 
61  2D messages when \p dopose is off:
62 
63  - Serial message type: \b 2D
64  - `id`: decoded ArUco marker ID, with a prefix 'U'
65  - `x`, `y`, or vertices: standardized 2D coordinates of marker center or corners
66  - `w`, `h`: standardized marker size
67  - `extra`: none (empty string)
68 
69  3D messages when \p dopose is on:
70 
71  - Serial message type: \b 3D
72  - `id`: decoded ArUco marker ID, with a prefix 'U'
73  - `x`, `y`, `z`, or vertices: 3D coordinates in millimeters of marker center or corners
74  - `w`, `h`, `d`: marker size in millimeters, a depth of 1mm is always used
75  - `extra`: none (empty string)
76 
77  If you will use the quaternion data (Detail message style; see \ref UserSerialStyle), you should probably set the \p
78  serprec parameter to something non-zero to get enough accuracy in the quaternion values.
79 
80  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
81  standardized coordinates.
82 
83  Things to try
84  -------------
85 
86  - First, use a video viewer software on a host computer and select one of the video modes with video output over
87  USB. Point your JeVois camera towards one of the screenshots provided with this module, or towards some ArUco
88  markers that you find on the web or that you have printed from the collection above (note: the default dictionary
89  is 4x4_50, see parameter \p dictionary).
90 
91  - Then try it with no video output, as it would be used by a robot. Connect to the command-line interface of your
92  JeVois camera through the serial-over-USB connection (see \ref UserCli; on Linux, you would use <b>sudo screen
93  /dev/ttyACM0 115200</b>) and try:
94  \verbatim
95  setpar serout USB
96  setmapping2 YUYV 320 240 30.0 JeVois DemoArUco
97  streamon
98  \endverbatim
99  and point the camera to some markers; the camera should issue messages about all the markers it identifies.
100 
101  Computing and showing 3D pose
102  -----------------------------
103 
104  The OpenCV ArUco module can also compute the 3D location and orientation of each marker in the world when \p dopose
105  is true. The requires that the camera be calibrated, see the documentation of the \ref ArUco component in
106  JeVoisBase. A generic calibration that is for a JeVois camera with standard lens is included in files \b
107  calibration640x480.yaml, \b calibration352x288.yaml, etc in the jevoisbase share directory (on the MicroSD, this is
108  in <b>JEVOIS:/share/camera/</b>).
109 
110  When doing pose estimation, you should set the \p markerlen parameter to the size (width) in millimeters of your
111  actual physical markers. Knowing that size will allow the pose estimation algorithm to know where in the world your
112  detected markers are.
113 
114  For more about camera calibration, see [this tutorial](http://jevois.org/tutorials/UserArUcoCalib.html) and
115  http://jevois.org/basedoc/ArUco_8H_source.html
116 
117  Tutorial and video
118  ------------------
119 
120  Check out this tutorial on how to [build a simple visually-guided toy robot car for under $100 with
121  JeVois](http://jevois.org/tutorials/UserRobotCar.html), which uses ArUco at its core. A demo video is here:
122 
123  \youtube{7cMtD-ef83E}
124 
125 
126  @author Laurent Itti
127 
128  @displayname Demo ArUco
129  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois DemoArUco
130  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
131  @videomapping YUYV 640 500 20.0 YUYV 640 480 20.0 JeVois DemoArUco
132  @email itti\@usc.edu
133  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
134  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
135  @mainurl http://jevois.org
136  @supporturl http://jevois.org/doc
137  @otherurl http://iLab.usc.edu
138  @license GPL v3
139  @distribution Unrestricted
140  @restrictions None
141  \ingroup modules */
143 {
144  public:
145  // ####################################################################################################
146  //! Constructor
147  // ####################################################################################################
148  DemoArUco(std::string const & instance) : jevois::StdModule(instance)
149  {
150  itsArUco = addSubComponent<ArUco>("aruco");
151  }
152 
153  // ####################################################################################################
154  //! Virtual destructor for safe inheritance
155  // ####################################################################################################
156  virtual ~DemoArUco()
157  { }
158 
159  // ####################################################################################################
160  //! Processing function, no video output
161  // ####################################################################################################
162  virtual void process(jevois::InputFrame && inframe) override
163  {
164  // Wait for next available camera image, any format and resolution ok here:
165  jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
166 
167  // Convert the image to grayscale and process:
168  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
169  std::vector<int> ids;
170  std::vector<std::vector<cv::Point2f> > corners;
171  itsArUco->detectMarkers(cvimg, ids, corners);
172 
173  // Do pose computation if desired:
174  std::vector<cv::Vec3d> rvecs, tvecs;
175  if (itsArUco->dopose::get() && ids.empty() == false)
176  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
177 
178  // Let camera know we are done processing the input image:
179  inframe.done();
180 
181  // Send serial output:
182  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
183  }
184 
185  // ####################################################################################################
186  //! Processing function with video output to USB
187  // ####################################################################################################
188  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
189  {
190  static jevois::Timer timer("processing", 100, LOG_DEBUG);
191 
192  // Wait for next available camera image:
193  jevois::RawImage const inimg = inframe.get();
194 
195  timer.start();
196 
197  // We only handle one specific pixel format, any size in this demo:
198  unsigned int const w = inimg.width, h = inimg.height;
199  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
200 
201  // While we process it, start a thread to wait for out frame and paste the input into it:
202  jevois::RawImage outimg;
203  auto paste_fut = jevois::async([&]() {
204  outimg = outframe.get();
205  outimg.require("output", w, h + 20, inimg.fmt);
206  jevois::rawimage::paste(inimg, outimg, 0, 0);
207  jevois::rawimage::writeText(outimg, "JeVois ArUco Demo", 3, 3, jevois::yuyv::White);
208  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, jevois::yuyv::Black);
209  });
210 
211  // Convert the image to grayscale and process:
212  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
213  std::vector<int> ids;
214  std::vector<std::vector<cv::Point2f> > corners;
215  std::vector<cv::Vec3d> rvecs, tvecs;
216  itsArUco->detectMarkers(cvimg, ids, corners);
217 
218  if (itsArUco->dopose::get() && ids.empty() == false)
219  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
220 
221  // Wait for paste to finish up:
222  paste_fut.get();
223 
224  // Let camera know we are done processing the input image:
225  inframe.done();
226 
227  // Show all the results:
228  itsArUco->drawDetections(outimg, 3, h+5, ids, corners, rvecs, tvecs);
229 
230  // Send serial output:
231  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
232 
233  // Show processing fps:
234  std::string const & fpscpu = timer.stop();
235  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
236 
237  // Send the output image with our processing results to the host over USB:
238  outframe.send();
239  }
240 
241 #ifdef JEVOIS_PRO
242  // ####################################################################################################
243  //! Processing function with zero-copy and GUI on JeVois-Pro
244  // ####################################################################################################
245  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
246  {
247  static jevois::Timer timer("processing", 100, LOG_DEBUG);
248 
249  // Start the GUI frame:
250  unsigned short winw, winh;
251  bool idle = helper.startFrame(winw, winh);
252 
253  // Draw the camera frame:
254  int x = 0, y = 0; unsigned short iw = 0, ih = 0;
255  helper.drawInputFrame("camera", inframe, x, y, iw, ih);
256 
257  // Wait for next available camera image:
258  jevois::RawImage const inimg = inframe.getp();
259  unsigned int const w = inimg.width, h = inimg.height;
260  helper.itext("JeVois-Pro ArUco Marker Detection");
261 
262  timer.start();
263 
264  // Convert the image to grayscale and process:
265  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
266  std::vector<int> ids;
267  std::vector<std::vector<cv::Point2f> > corners;
268  std::vector<cv::Vec3d> rvecs, tvecs;
269  itsArUco->detectMarkers(cvimg, ids, corners);
270 
271  if (itsArUco->dopose::get() && ids.empty() == false)
272  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
273 
274  // Let camera know we are done processing the input image:
275  inframe.done();
276 
277  // Show all the results:
278  itsArUco->drawDetections(helper, ids, corners, rvecs, tvecs);
279 
280  // Send serial output:
281  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
282 
283  // Show processing fps:
284  std::string const & fpscpu = timer.stop();
285  helper.iinfo(inframe, fpscpu, winw, winh);
286 
287  // Render the image and GUI:
288  helper.endFrame();
289  }
290 #endif
291 
292  // ####################################################################################################
293  protected:
294  std::shared_ptr<ArUco> itsArUco;
295 };
296 
297 // Allow the module to be loaded as a shared object (.so) file:
jevois::GUIhelper
jevois::OutputFrame
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
DemoArUco::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition: DemoArUco.C:162
Timer.H
jevois::GUIhelper::itext
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
Module.H
jevois::GUIhelper::endFrame
void endFrame()
jevois::RawImage
jevois::Timer::start
void start()
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
DemoArUco
Simple demo of ArUco augmented reality markers detection and decoding.
Definition: DemoArUco.C:142
DemoArUco::DemoArUco
DemoArUco(std::string const &instance)
Constructor.
Definition: DemoArUco.C:148
jevois::RawImage::width
unsigned int width
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
DemoArUco::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: DemoArUco.C:188
jevois::StdModule::StdModule
StdModule(std::string const &instance)
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
jevois::InputFrame
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
ArUco.H
h
int h
DemoArUco::~DemoArUco
virtual ~DemoArUco()
Virtual destructor for safe inheritance.
Definition: DemoArUco.C:156
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DemoArUco)
jevois::StdModule
DemoArUco::itsArUco
std::shared_ptr< ArUco > itsArUco
Definition: DemoArUco.C:294
demo.w
w
Definition: demo.py:85
jevois::Timer