JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoArUco.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, BA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
22 #include <opencv2/core/core.hpp>
23 
24 //! Simple demo of ArUco augmented reality markers detection and decoding
25 /*! Detect and decode patterns known as ArUco markers, which are small 2D barcodes often used in augmented
26  reality and robotics.
27 
28  ArUco markers are small 2D barcodes. Each ArUco marker corresponds to a number, encoded into a small grid of black
29  and white pixels. The ArUco decoding algorithm is capable of locating, decoding, and of estimating the pose
30  (location and orientation in space) of any ArUco markers in the camera's field of view.
31 
32  ArUco markers are very useful as tags for many robotics and augmented reality applications. For example, one may
33  place an ArUco next to a robot's charging station, an elevator button, or an object that a robot should manipulate.
34 
35  For more information about ArUco, see https://www.uco.es/investiga/grupos/ava/node/26
36 
37  The implementation of ArUco used by JeVois is the one of OpenCV-Contrib, documented here:
38  http://docs.opencv.org/3.2.0/d5/dae/tutorial_aruco_detection.html
39 
40  ArUco markers can be created with several standard dictionaries. Different dictionaries give rise to different
41  numbers of pixels in the markers, and to different numbers of possible symbols that can be created using the
42  dictionary. The default dictionary used by JeVois is 4x4 with 50 symbols. Other dictionaries are also supported by
43  setting the parameter \p dictionary over serial port or in a config file, up to 7x7 with 1000 symbols.
44 
45  Creating and printing markers
46  -----------------------------
47 
48  We have created the 50 markers available in the default dictionary (4x4_50) as PNG images that you can download and
49  print, at http://jevois.org/data/ArUco.zip
50 
51  To make your own, for example, using another dictionary, see the documentation of the \ref ArUco component of
52  JeVoisBase. Some utilities are provided with the component.
53 
54  Serial Messages
55  ---------------
56 
57  This module can send standardized serial messages as described in \ref UserSerialStyle.
58 
59  When \p dopose is turned on, 3D messages will be sent, otherwise 2D messages.
60 
61  One message is issued for every detected ArUco, on every video frame.
62 
63  2D messages when \p dopose is off:
64 
65  - Serial message type: \b 2D
66  - `id`: decoded ArUco marker ID, with a prefix 'U'
67  - `x`, `y`, or vertices: standardized 2D coordinates of marker center or corners
68  - `w`, `h`: standardized marker size
69  - `extra`: none (empty string)
70 
71  3D messages when \p dopose is on:
72 
73  - Serial message type: \b 3D
74  - `id`: decoded ArUco marker ID, with a prefix 'U'
75  - `x`, `y`, `z`, or vertices: 3D coordinates in millimeters of marker center or corners
76  - `w`, `h`, `d`: marker size in millimeters, a depth of 1mm is always used
77  - `extra`: none (empty string)
78 
79  If you will use the quaternion data (Detail message style; see \ref UserSerialStyle), you should probably set the \p
80  serprec parameter to something non-zero to get enough accuracy in the quaternion values.
81 
82  Things to try
83  -------------
84 
85  - First, use a video viewer software on a host computer and select one of the video modes with video output over
86  USB. Point your JeVois camera towards one of the screenshots provided with this module, or towards some ArUco
87  markers that you find on the web or that you have printed from the collection above (note: the default dictionary
88  is 4x4_50, see parameter \p dictionary).
89 
90  - Then try it with no video output, as it would be used by a robot. Connect to the command-line interface of your
91  JeVois camera through the serial-over-USB connection (see \ref UserCli; on Linux, you would use <b>sudo screen
92  /dev/ttyACM0 115200</b>) and try:
93  \verbatim
94  setpar serout USB
95  setmapping2 YUYV 320 240 30.0 JeVois DemoArUco
96  streamon
97  \endverbatim
98  and point the camera to some markers; the camera should issue messages about all the markers it identifies.
99 
100  Computing and showing 3D pose
101  -----------------------------
102 
103  The OpenCV ArUco module can also compute the 3D location and orientation of each marker in the world when \p dopose
104  is true. The requires that the camera be calibrated, see the documentation of the \ref ArUco component in
105  JeVoisBase. A generic calibration that is for a JeVois camera with standard lens is included in files \b
106  calibration640x480.yaml, \b calibration352x288.yaml, etc in the module's directory (on the MicroSD, this is in
107  <b>JEVOIS:/modules/JeVois/DemoArUco/</b>).
108 
109  When doing pose estimation, you should set the \p markerlen parameter to the size (width) in millimeters of your
110  actual physical markers. Knowing that size will allow the pose estimation algorithm to know where in the world your
111  detected markers are.
112 
113  Tutorial and video
114  ------------------
115 
116  Check out this tutorial on how to [build a simple visually-guided toy robot car for under $100 with
117  JeVois](http://jevois.org/tutorials/UserRobotCar.html), which uses ArUco at its core. A demo video is here:
118 
119  \youtube{7cMtD-ef83E}
120 
121 
122  @author Laurent Itti
123 
124  @displayname Demo ArUco
125  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois DemoArUco
126  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
127  @videomapping YUYV 640 500 20.0 YUYV 640 480 20.0 JeVois DemoArUco
128  @email itti\@usc.edu
129  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
130  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
131  @mainurl http://jevois.org
132  @supporturl http://jevois.org/doc
133  @otherurl http://iLab.usc.edu
134  @license GPL v3
135  @distribution Unrestricted
136  @restrictions None
137  \ingroup modules */
139 {
140  public:
141  // ####################################################################################################
142  //! Constructor
143  // ####################################################################################################
144  DemoArUco(std::string const & instance) : jevois::StdModule(instance)
145  {
146  itsArUco = addSubComponent<ArUco>("aruco");
147  }
148 
149  // ####################################################################################################
150  //! Virtual destructor for safe inheritance
151  // ####################################################################################################
152  virtual ~DemoArUco()
153  { }
154 
155  // ####################################################################################################
156  //! Processing function, no video output
157  // ####################################################################################################
158  virtual void process(jevois::InputFrame && inframe) override
159  {
160  // Wait for next available camera image, any format and resolution ok here:
161  jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
162 
163  // Convert the image to grayscale and process:
164  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
165  std::vector<int> ids;
166  std::vector<std::vector<cv::Point2f> > corners;
167  itsArUco->detectMarkers(cvimg, ids, corners);
168 
169  // Do pose computation if desired:
170  std::vector<cv::Vec3d> rvecs, tvecs;
171  if (itsArUco->dopose::get() && ids.empty() == false)
172  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
173 
174  // Let camera know we are done processing the input image:
175  inframe.done();
176 
177  // Send serial output:
178  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
179  }
180 
181  // ####################################################################################################
182  //! Processing function with video output to USB
183  // ####################################################################################################
184  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
185  {
186  static jevois::Timer timer("processing", 100, LOG_DEBUG);
187 
188  // Wait for next available camera image:
189  jevois::RawImage const inimg = inframe.get();
190 
191  timer.start();
192 
193  // We only handle one specific pixel format, any size in this demo:
194  unsigned int const w = inimg.width, h = inimg.height;
195  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
196 
197  // While we process it, start a thread to wait for out frame and paste the input into it:
198  jevois::RawImage outimg;
199  auto paste_fut = std::async(std::launch::async, [&]() {
200  outimg = outframe.get();
201  outimg.require("output", w, h + 20, inimg.fmt);
202  jevois::rawimage::paste(inimg, outimg, 0, 0);
203  jevois::rawimage::writeText(outimg, "JeVois ArUco Demo", 3, 3, jevois::yuyv::White);
204  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, jevois::yuyv::Black);
205  });
206 
207  // Convert the image to grayscale and process:
208  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
209  std::vector<int> ids;
210  std::vector<std::vector<cv::Point2f> > corners;
211  std::vector<cv::Vec3d> rvecs, tvecs;
212  itsArUco->detectMarkers(cvimg, ids, corners);
213 
214  if (itsArUco->dopose::get() && ids.empty() == false)
215  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
216 
217  // Wait for paste to finish up:
218  paste_fut.get();
219 
220  // Let camera know we are done processing the input image:
221  inframe.done();
222 
223  // Show all the results:
224  itsArUco->drawDetections(outimg, 3, h+5, ids, corners, rvecs, tvecs);
225 
226  // Send serial output:
227  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
228 
229  // Show processing fps:
230  std::string const & fpscpu = timer.stop();
231  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
232 
233  // Send the output image with our processing results to the host over USB:
234  outframe.send();
235  }
236 
237  // ####################################################################################################
238  protected:
239  std::shared_ptr<ArUco> itsArUco;
240 };
241 
242 // Allow the module to be loaded as a shared object (.so) file:
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition: DemoArUco.C:158
JEVOIS_REGISTER_MODULE(DemoArUco)
unsigned int fmt
DemoArUco(std::string const &instance)
Constructor.
Definition: DemoArUco.C:144
Simple demo of ArUco augmented reality markers detection and decoding.
Definition: DemoArUco.C:138
StdModule(std::string const &instance)
cv::Mat convertToCvGray(RawImage const &src)
std::string const & stop()
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::shared_ptr< ArUco > itsArUco
Definition: DemoArUco.C:239
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: DemoArUco.C:184
unsigned int width
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
virtual ~DemoArUco()
Virtual destructor for safe inheritance.
Definition: DemoArUco.C:152
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const