JeVoisBase  1.3
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoArUco.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, BA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
22 #include <opencv2/core/core.hpp>
23 
24 //! Simple demo of ArUco augmented reality markers detection and decoding
25 /*! Detect and decode patterns known as ArUco markers, which are small 2D barcodes often used in augmented
26  reality and robotics.
27 
28  ArUco markers are small 2D barcodes. Each ArUco marker corresponds to a number, encoded into a small grid of black
29  and white pixels. The ArUco decoding algorithm is capable of locating, decoding, and of estimating the pose
30  (location and orientation in space) of any ArUco markers in the camera's field of view.
31 
32  ArUcos are very useful as tags for many robotics and augmented reality applications. For example, one may place an
33  ArUco next to a robot's charging station, an elevator button, or an object that a robot should manipulate.
34 
35  For more information about ArUco, see https://www.uco.es/investiga/grupos/ava/node/26
36 
37  The implementation of ArUco used by JeVois is the one of OpenCV-Contrib, documented here:
38  http://docs.opencv.org/3.2.0/d5/dae/tutorial_aruco_detection.html
39 
40  ArUco markers can be created with several standard dictionaries. Different dictionaries give rise to different
41  numbers of pixels in the markers, and to different numbers of possible symbols that can be created using the
42  dictionary. The default dictionary used by JeVois is 4x4 with 50 symbols. Other dictionaries are also supported by
43  setting the appropriate parameter over serial port or in a config file, up to 7x7 with 1000 symbols.
44 
45  Creating and printing markers
46  -----------------------------
47 
48  We have created the 50 markers available in the default dictionary (4x4_50) as PNG images that you can download and
49  print, at http://jevois.org/data/ArUco.zip
50 
51  To make your own, for example, using another dictionary, see the documentation of the ArUco component of
52  JeVoisBase. Some utilities are provided with the component.
53 
54  Serial Messages
55  ---------------
56 
57  This module can send standardized serial messages as described in \ref UserSerialStyle.
58 
59  When \p dopose is turned on, 3D messages will be sent, otherwise 2D messages.
60 
61  One message is issued for every detected ArUco, on every video frame.
62 
63  2D messages when \p dopose is off:
64 
65  - Serial message type: \b 2D
66  - `id`: decoded ArUco marker ID, with a prefix 'U'
67  - `x`, `y`, or vertices: standardized 2D coordinates of marker center or corners
68  - `w`, `h`: standardized marker size
69  - `extra`: none (empty string)
70 
71  3D messages when \p dopose is on:
72 
73  - Serial message type: \b 3D
74  - `id`: decoded ArUco marker ID, with a prefix 'U'
75  - `x`, `y`, `z`, or vertices: 3D coordinates in millimeters of marker center or corners
76  - `w`, `h`, `d`: marker size in millimeters, a depth of 1mm is always used
77  - `extra`: none (empty string)
78 
79  If you will use the quaternion data (Detail message style; see \ref UserSerialStyle), you should probably set the \p
80  serprec parameter to something non-zero to get enough accuracy in the quaternion values.
81 
82  Things to try
83  -------------
84 
85  - First, use a video viewer software on a host computer and select one of the video modes with video output over
86  USB. Point your JeVois camera towards one of the screenshots provided with this module, or towards some ArUco
87  markers that you find on the web or that you have printed from the collection above (note: the default dictionary
88  is 4x4_50, see parameter \p dictionary).
89 
90  - Then try it with no video output, as it would be used by a robot. Connect to the command-line interface of your
91  JeVois camera through the serial-over-USB connection (see \ref UserCli; on Linux, you would use <b>sudo screen
92  /dev/ttyACM0 115200</b>) and try:
93  \verbatim
94  setpar serout USB
95  setmapping2 YUYV 320 240 30.0 JeVois DemoArUco
96  streamon
97  \endverbatim
98  and point the camera to some markers; the camera should issue messages about all the markers it identifies.
99 
100  Computing and showing 3D pose
101  -----------------------------
102 
103  The OpenCV ArUco module can also compute the 3D location and orientation of each marker in the world when \p dopose
104  is true. The requires that the camera be calibrated, see the documentation of the \ref ArUco component in
105  JeVoisBase. A generic calibration that is for a JeVois camera with standard lens is included in files \b
106  calibration640x480.yaml, \b calibration352x288.yaml, etc in the module's directory (on the MicroSD, this is in
107  <b>JEVOIS:/modules/JeVois/DemoArUco/</b>).
108 
109  When doing pose estimation, you should set the \p markerlen parameter to the size (width) in millimeters of your
110  actual physical markers. Knowing that size will allow the pose estimation algorithm to know where in the world your
111  detected markers are.
112 
113 
114  @author Laurent Itti
115 
116  @displayname Demo ArUco
117  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois DemoArUco
118  @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
119  @videomapping YUYV 640 500 20.0 YUYV 640 480 20.0 JeVois DemoArUco
120  @email itti\@usc.edu
121  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
122  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
123  @mainurl http://jevois.org
124  @supporturl http://jevois.org/doc
125  @otherurl http://iLab.usc.edu
126  @license GPL v3
127  @distribution Unrestricted
128  @restrictions None
129  \ingroup modules */
131 {
132  public:
133  // ####################################################################################################
134  //! Constructor
135  // ####################################################################################################
136  DemoArUco(std::string const & instance) : jevois::StdModule(instance)
137  {
138  itsArUco = addSubComponent<ArUco>("aruco");
139  }
140 
141  // ####################################################################################################
142  //! Virtual destructor for safe inheritance
143  // ####################################################################################################
144  virtual ~DemoArUco()
145  { }
146 
147  // ####################################################################################################
148  //! Processing function, no video output
149  // ####################################################################################################
150  virtual void process(jevois::InputFrame && inframe) override
151  {
152  // Wait for next available camera image, any format and resolution ok here:
153  jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
154 
155  // Convert the image to grayscale and process:
156  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
157  std::vector<int> ids;
158  std::vector<std::vector<cv::Point2f> > corners;
159  itsArUco->detectMarkers(cvimg, ids, corners);
160 
161  // Do pose computation if desired:
162  std::vector<cv::Vec3d> rvecs, tvecs;
163  if (itsArUco->dopose::get() && ids.empty() == false)
164  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
165 
166  // Let camera know we are done processing the input image:
167  inframe.done();
168 
169  // Send serial output:
170  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
171  }
172 
173  // ####################################################################################################
174  //! Processing function with video output to USB
175  // ####################################################################################################
176  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
177  {
178  static jevois::Timer timer("processing", 100, LOG_DEBUG);
179 
180  // Wait for next available camera image:
181  jevois::RawImage const inimg = inframe.get();
182 
183  timer.start();
184 
185  // We only handle one specific pixel format, any size in this demo:
186  unsigned int const w = inimg.width, h = inimg.height;
187  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
188 
189  // While we process it, start a thread to wait for out frame and paste the input into it:
190  jevois::RawImage outimg;
191  auto paste_fut = std::async(std::launch::async, [&]() {
192  outimg = outframe.get();
193  outimg.require("output", w, h + 20, inimg.fmt);
194  jevois::rawimage::paste(inimg, outimg, 0, 0);
195  jevois::rawimage::writeText(outimg, "JeVois ArUco Demo", 3, 3, jevois::yuyv::White);
196  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, jevois::yuyv::Black);
197  });
198 
199  // Convert the image to grayscale and process:
200  cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
201  std::vector<int> ids;
202  std::vector<std::vector<cv::Point2f> > corners;
203  std::vector<cv::Vec3d> rvecs, tvecs;
204  itsArUco->detectMarkers(cvimg, ids, corners);
205 
206  if (itsArUco->dopose::get() && ids.empty() == false)
207  itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
208 
209  // Wait for paste to finish up:
210  paste_fut.get();
211 
212  // Let camera know we are done processing the input image:
213  inframe.done();
214 
215  // Show all the results:
216  itsArUco->drawDetections(outimg, 3, h+5, ids, corners, rvecs, tvecs);
217 
218  // Send serial output:
219  itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
220 
221  // Show processing fps:
222  std::string const & fpscpu = timer.stop();
223  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
224 
225  // Send the output image with our processing results to the host over USB:
226  outframe.send();
227  }
228 
229  // ####################################################################################################
230  protected:
231  std::shared_ptr<ArUco> itsArUco;
232 };
233 
234 // Allow the module to be loaded as a shared object (.so) file:
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
unsigned int height
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition: DemoArUco.C:150
JEVOIS_REGISTER_MODULE(DemoArUco)
unsigned int fmt
DemoArUco(std::string const &instance)
Constructor.
Definition: DemoArUco.C:136
Simple demo of ArUco augmented reality markers detection and decoding.
Definition: DemoArUco.C:130
StdModule(std::string const &instance)
cv::Mat convertToCvGray(RawImage const &src)
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::shared_ptr< ArUco > itsArUco
Definition: DemoArUco.C:231
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: DemoArUco.C:176
unsigned int width
std::string const & stop()
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
virtual ~DemoArUco()
Virtual destructor for safe inheritance.
Definition: DemoArUco.C:144