JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoARtoolkit.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Shixian Wen - 3641 Watt Way, HNB-10A - Los Angeles, BA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - shixianw@usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
23 #include <opencv2/core/core.hpp>
24 #include <opencv2/imgproc/imgproc.hpp>
25 
26 //! Augmented reality markers using ARtoolkit
27 /*! Detect and decode patterns known as ARtoolkit markers, which are small 2D barcodes often used in augmented
28  reality and robotics. See https://archive.artoolkit.org/ - This module uses version 5.x for now.
29 
30  This algorithm is very fast. It runs at 400+ frames/s at 320x240 and can easily sustain the maximum 15 frames/s of
31  the JeVois camera sensor chip at 1280x1024. It may be interesting to combine it with other algorithms running on the
32  quad-core JeVois processor (see for example the \jvmod{MarkersCombo} module).
33 
34  Sample patterns
35  ---------------
36 
37  This module uses by default the dictionary of 3x3 patterns with parity (32 different patterns). You can download
38  them from http://jevois.org/data/ARtoolkit3x3par.zip
39 
40  Also see the files in <b>jevoisbase/Contrib/ARToolKit5arm/doc/patterns/</b> for more.
41 
42  Serial Messages
43  ---------------
44 
45  This module can send standardized serial messages as described in \ref UserSerialStyle.
46 
47  When \p msg3d is turned on, 3D messages will be sent, otherwise 2D messages.
48 
49  One message is issued for every detected marker, on every video frame.
50 
51  2D messages when \p msg3d is off:
52 
53  - Serial message type: \b 2D
54  - `id`: decoded ARtoolkit marker ID, with a prefix 'A'
55  - `x`, `y`, or vertices: standardized 2D coordinates of marker center or corners
56  - `w`, `h`: standardized marker size
57  - `extra`: none (empty string)
58 
59  3D messages when \p msg3d is on:
60 
61  - Serial message type: \b 3D
62  - `id`: decoded ARtoolkit marker ID, with a prefix 'A'
63  - `x`, `y`, `z`, or vertices: 3D coordinates in millimeters of marker center or corners
64  - `w`, `h`, `d`: marker size in millimeters, a depth of 1mm is always used
65  - `extra`: none (empty string)
66 
67  If you will use the quaternion data (Detail message style; see \ref UserSerialStyle), you should probably set the \p
68  serprec parameter to something non-zero to get enough accuracy in the quaternion values.
69 
70  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
71  standardized coordinates.
72 
73  For information about camera calibration and obtaining precise location, distance and pose estimates, see [this
74  tutorial](http://jevois.org/tutorials/UserARcalib.html).
75 
76  @author Shixian Wen
77 
78  @displayname Demo ARtoolkit
79  @videomapping NONE 0 0 0 YUYV 320 240 60.0 JeVois DemoARtoolkit
80  @videomapping NONE 0 0 0 YUYV 640 480 30.0 JeVois DemoARtoolkit
81  @videomapping NONE 0 0 0 YUYV 1280 1024 15.0 JeVois DemoARtoolkit
82  @videomapping YUYV 320 258 60.0 YUYV 320 240 60.0 JeVois DemoARtoolkit
83  @videomapping YUYV 640 498 30.0 YUYV 640 480 30.0 JeVois DemoARtoolkit
84  @email shixianw\@usc.edu
85  @address University of Southern California, HNB-10A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
86  @copyright Copyright (C) 2017 by Shixian Wen, iLab and the University of Southern California
87  @mainurl http://jevois.org
88  @supporturl http://jevois.org/doc
89  @otherurl http://iLab.usc.edu
90  @license GPL v3
91  @distribution Unrestricted
92  @restrictions None
93  \ingroup modules */
95 {
96  public:
97  // ####################################################################################################
98  DemoARtoolkit(std::string const & instance) : jevois::StdModule(instance)
99  {
100  itsARtoolkit = addSubComponent<ARtoolkit>("artoolkit");
101  }
102 
103  // ####################################################################################################
104  virtual ~DemoARtoolkit()
105  { }
106 
107  // ####################################################################################################
108  virtual void process(jevois::InputFrame && inframe) override
109  {
110  static jevois::Timer timer("processing");
111 
112  // Wait for next available camera image:
113  jevois::RawImage const inimg = inframe.get();
114  timer.start();
115 
116  // ARtoolkit component can process YUYV images directly with no conversion required:
117  if (inimg.fmt == V4L2_PIX_FMT_YUYV)
118  itsARtoolkit->detectMarkers(inimg);
119  else
120  {
121  // Otherwise, convert to gray:
122  cv::Mat incv = jevois::rawimage::convertToCvGray(inimg);
123  itsARtoolkit->detectMarkers(incv);
124  }
125 
126  // We are done with the input frame:
127  inframe.done();
128 
129  // Send serial messages:
130  itsARtoolkit->sendSerial(this);
131  timer.stop();
132  }
133 
134  // ####################################################################################################
135  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
136  {
137  static jevois::Timer timer("processing", 100, LOG_DEBUG);
138 
139  // Wait for next available camera image:
140  jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
141  timer.start();
142 
143  // While we process it, start a thread to wait for out frame and paste the input into it:
144  jevois::RawImage outimg;
145  auto paste_fut = jevois::async([&]() {
146  outimg = outframe.get();
147  outimg.require("output", w, h + 18, inimg.fmt);
148  jevois::rawimage::paste(inimg, outimg, 0, 0);
149  jevois::rawimage::writeText(outimg, "JeVois ARtoolkit Demo", 3, 3, jevois::yuyv::White);
150  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height - h, jevois::yuyv::Black);
151  });
152 
153  // ARtoolkit component can process YUYV images directly with no conversion required:
154  itsARtoolkit->detectMarkers(inimg);
155 
156  // Wait for paste to finish up:
157  paste_fut.get();
158 
159  // We are done with the input frame:
160  inframe.done();
161 
162  // Draw the detections & send serial messages:
163  itsARtoolkit->drawDetections(outimg, 3, h + 3);
164  itsARtoolkit->sendSerial(this);
165 
166  // Show processing fps:
167  std::string const & fpscpu = timer.stop();
168  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
169 
170  // Send the output image with our processing results to the host over USB:
171  outframe.send();
172  }
173 
174 #ifdef JEVOIS_PRO
175  // ####################################################################################################
176  //! Processing function with zero-copy and GUI on JeVois-Pro
177  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
178  {
179  static jevois::Timer timer("processing", 100, LOG_DEBUG);
180 
181  // Start the GUI frame:
182  unsigned short winw, winh;
183  bool idle = helper.startFrame(winw, winh);
184 
185  // Draw the camera frame:
186  int x = 0, y = 0; unsigned short w = 0, h = 0;
187  helper.drawInputFrame("camera", inframe, x, y, w, h);
188 
189  // Get the camera image so we can process it:
190  jevois::RawImage const inimg = inframe.get();
191  helper.itext("JeVois-Pro ARtoolkit Marker Detection");
192 
193  // ARtoolkit component can process YUYV images directly with no conversion required:
194  timer.start();
195  itsARtoolkit->detectMarkers(inimg);
196 
197  // We are done with the input frame:
198  inframe.done();
199 
200  // Draw the detections & send serial messages:
201  itsARtoolkit->drawDetections(helper);
202  itsARtoolkit->sendSerial(this);
203 
204  // Show processing fps:
205  std::string const & fpscpu = timer.stop();
206  helper.iinfo(inframe, fpscpu, winw, winh);
207 
208  // Render the image and GUI:
209  helper.endFrame();
210  }
211 
212 #endif // JEVOIS_PRO
213 
214  // ####################################################################################################
215  protected:
216  std::shared_ptr<ARtoolkit> itsARtoolkit;
217 };
218 
219 // Allow the module to be loaded as a shared object (.so) file:
DemoARtoolkit
Augmented reality markers using ARtoolkit.
Definition: DemoARtoolkit.C:94
DemoARtoolkit::process
virtual void process(jevois::InputFrame &&inframe) override
Definition: DemoARtoolkit.C:108
jevois::OutputFrame
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
Timer.H
jevois::GUIhelper::itext
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
Module.H
jevois::GUIhelper
jevois::GUIhelper::endFrame
void endFrame()
Coordinates.H
jevois::RawImage
jevois::Timer::start
void start()
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::RawImage::width
unsigned int width
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
DemoARtoolkit::DemoARtoolkit
DemoARtoolkit(std::string const &instance)
Definition: DemoARtoolkit.C:98
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
DemoARtoolkit::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Definition: DemoARtoolkit.C:135
DemoARtoolkit::itsARtoolkit
std::shared_ptr< ARtoolkit > itsARtoolkit
Definition: DemoARtoolkit.C:216
jevois::StdModule::StdModule
StdModule(std::string const &instance)
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
jevois::InputFrame
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DemoARtoolkit)
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
DemoARtoolkit::~DemoARtoolkit
virtual ~DemoARtoolkit()
Definition: DemoARtoolkit.C:104
h
int h
jevois::StdModule
ARtoolkit.H
jevois::Timer