JeVoisBase  1.18
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
RoadNavigation.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
24 
25 #include <opencv2/core/core.hpp>
26 #include <opencv2/imgproc/imgproc.hpp>
27 
28 #include <future>
29 #include <linux/videodev2.h> // for v4l2 pixel types
30 
31 // icon by Dave Gandy in transport at flaticon
32 
33 //! Parameter \relates RoadNavigation
34 JEVOIS_DECLARE_PARAMETER(vpconf, float, "Minimum vanishing point confidence required to send a serial message. "
35  "Note that currently, confidence values are binary (only 0 or 1), and quite conservative "
36  "(many good detections may get a confidence of 0 if they are not very clean).",
37  0.0F, roadfinder::ParamCateg);
38 
39 //! Road finder demo
40 /*! This algorithm detects road using a compination of edge detection and tracking, and texture analysis. The algorithm
41  is an implementation of Chang, Siagian and Itti, IROS 2012, available at
42  http://ilab.usc.edu/publications/doc/Chang_etal12iros.pdf
43 
44  The algorithms combines detection and tracking of line segments at the edges of the road or on the road (e.g., lane
45  dividers), and texture analysis to distinguish the road region from its surroundings. One strength of this algorithm
46  is that it requires no training. No need to drive your robot on many roads to learn about the appearance of
47  roads. The flipside of this, of course, is that driving on roads which do not look like standard roads or trails may
48  fail.
49 
50  The algorithm outputs the horizontal coordinate of the vanishing point of the road, which usually is a good
51  indication of the road heading (except in very tight bends or corners).
52 
53  Demo display outputs
54  --------------------
55 
56  Detected line segments are shown in black and white, while segments that have been reliably tracked over multiple
57  frames are shown in thick purple. Estimated vanishing point location and confidence is shown as a big green disk on
58  the horizon line.
59 
60  Note that the demo display runs much slower than the algorithm with no video output (which should be used on
61  autonomous cars), simply because drawing all the detected line segments takes a lot of time.
62 
63  Serial Messages
64  ---------------
65 
66  This module can send standardized serial messages as described in \ref UserSerialStyle. One 1D message is issued on
67  every video frame for the vanishing point's horizontal location.
68 
69  - Serial message type: \b 1D
70  - `id`: always \b vp (shorthand for vanishing point)
71  - `x`: standardized 2D horizontal coordinate of the vanishing point
72  - `w`: always 0.
73  - `extra`: none (empty string)
74 
75  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
76  standardized coordinates.
77 
78  Trying it out
79  -------------
80 
81  To casually try out this module, just search the web for pictures of roads and point the JeVois camera to one of
82  them. Make sure that you align the horizon line of the algorithm (which has a number of purple and green disks)
83  roughly with the horizon line in your picture. As you move the camera left and right, the location of the large
84  green disk that marks the detected vanishing point should move left and right, and should point to the vanishing
85  point of the road in your image.
86 
87  When using on a mobile robot in the real world, setting the proper horizon line is essential for good operation of
88  the algorithm. This is determined by parameter \c horizon, which should be tuned according to the height and
89  tilt angle of the JeVois camera on your vehicle.
90 
91 
92  @author Laurent Itti
93 
94  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois RoadNavigation
95  @videomapping NONE 0 0 0 YUYV 176 144 120.0 JeVois RoadNavigation
96  @videomapping YUYV 320 256 30.0 YUYV 320 240 30.0 JeVois RoadNavigation
97  @videomapping YUYV 176 160 120.0 YUYV 176 144 120.0 JeVois RoadNavigation
98  @email itti\@usc.edu
99  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
100  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
101  @mainurl http://jevois.org
102  @supporturl http://jevois.org/doc
103  @otherurl http://iLab.usc.edu
104  @license GPL v3
105  @distribution Unrestricted
106  @restrictions None
107  \ingroup modules */
108 class RoadNavigation : public jevois::StdModule, jevois::Parameter<vpconf>
109 {
110  public:
111  // ####################################################################################################
112  //! Constructor
113  // ####################################################################################################
114  RoadNavigation(std::string const & instance) :
115  jevois::StdModule(instance), itsProcessingTimer("Processing", 30, LOG_DEBUG)
116  {
117  itsRoadFinder = addSubComponent<RoadFinder>("roadfinder");
118  }
119 
120  // ####################################################################################################
121  //! Virtual destructor for safe inheritance
122  // ####################################################################################################
123  virtual ~RoadNavigation() { }
124 
125  // ####################################################################################################
126  //! Processing function, no video out
127  // ####################################################################################################
128  virtual void process(jevois::InputFrame && inframe) override
129  {
130  // Wait for next available camera image, convert it to gray, and release it:
131  cv::Mat imggray = inframe.getCvGRAY();
132 
133  // Compute the vanishing point, with no drawings:
134  jevois::RawImage visual; // unallocated pixels, will not draw anything
135  itsRoadFinder->process(imggray, visual);
136 
137  // Get the vanishing point and send to serial:
138  int const w = imggray.cols;
139  std::pair<Point2D<int>, float> vp = itsRoadFinder->getCurrVanishingPoint();
140  if (vp.second >= vpconf::get()) sendSerialImg1Dx(w, vp.first.i, 0.0F, "vp");
141  }
142 
143  // ####################################################################################################
144  //! Processing function with USB video out
145  // ####################################################################################################
146  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
147  {
148  // Wait for next available camera image:
149  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
150  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
151 
153 
154  // Convert it to gray:
155  cv::Mat imggray = jevois::rawimage::convertToCvGray(inimg);
156 
157  // Compute the vanishing point. Note: the results will be drawn into inimg, so that we don't have to wait for
158  // outimg to be ready. It's ok to modify the input image, its buffer will be sent back to the camera driver for
159  // capture once we are done here, and will be overwritten anyway:
160  itsRoadFinder->process(imggray, inimg);
161 
162  // Wait for an image from our gadget driver into which we will put our results:
163  jevois::RawImage outimg = outframe.get();
164 
165  // Enforce the correct output image size and format:
166  outimg.require("output", w, h + 16, V4L2_PIX_FMT_YUYV);
167 
168  // Paste the original image + drawings to the top-left corner of the display:
169  unsigned short const txtcol = 0xa0ff; //WHITE: 0x80ff;
170  jevois::rawimage::paste(inimg, outimg, 0, 0);
171  jevois::rawimage::writeText(outimg, "JeVois Road Navigation Demo", 3, 3, txtcol);
172 
173  // Let camera know we are done processing the raw YUV input image:
174  inframe.done();
175 
176  // Clear out the bottom section of the image:
177  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, jevois::yuyv::Black);
178 
179  // Get the vanishing point and send to serial:
180  std::pair<Point2D<int>, float> vp = itsRoadFinder->getCurrVanishingPoint();
181  if (vp.second >= vpconf::get()) sendSerialImg1Dx(w, vp.first.i, 0.0F, "vp");
182 
183  // Write some extra info about the vp:
184  std::ostringstream otxt; otxt << std::fixed << std::setprecision(3);
185  otxt << "VP x=" << vp.first.i << " (" << vp.second << ") CTR=" << std::setprecision(1);
186  auto cp = itsRoadFinder->getCurrCenterPoint();
187  auto tp = itsRoadFinder->getCurrTargetPoint();
188  float const tpx = itsRoadFinder->getFilteredTargetX();
189  otxt << cp.i << " TGT=" << tp.i << " fTPX=" << tpx;
190  jevois::rawimage::writeText(outimg, otxt.str().c_str(), 3, h + 3, jevois::yuyv::White);
191 
192  // Show processing fps:
193  std::string const & fpscpu = itsProcessingTimer.stop();
194  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
195 
196  // Send the output image with our processing results to the host over USB:
197  outframe.send();
198  }
199 
200  // ####################################################################################################
201  //! Module internals
202  // ####################################################################################################
203  protected:
205  std::shared_ptr<RoadFinder> itsRoadFinder;
206 };
207 
208 // Allow the module to be loaded as a shared object (.so) file:
RoadNavigation::~RoadNavigation
virtual ~RoadNavigation()
Virtual destructor for safe inheritance.
Definition: RoadNavigation.C:123
RoadNavigation::itsProcessingTimer
jevois::Timer itsProcessingTimer
Module internals.
Definition: RoadNavigation.C:204
jevois::OutputFrame
RoadFinder.H
Timer.H
Module.H
Log.H
jevois::StdModule::sendSerialImg1Dx
void sendSerialImg1Dx(unsigned int camw, float x, float size=0.0F, std::string const &id="", std::string const &extra="")
jevois::RawImage
jevois::Timer::start
void start()
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
RoadNavigation::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with USB video out.
Definition: RoadNavigation.C:146
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
RoadNavigation::itsRoadFinder
std::shared_ptr< RoadFinder > itsRoadFinder
Definition: RoadNavigation.C:205
jevois::RawImage::width
unsigned int width
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
F
float F
RoadNavigation::RoadNavigation
RoadNavigation(std::string const &instance)
Constructor.
Definition: RoadNavigation.C:114
jevois::Timer::stop
const std::string & stop(double *seconds)
RoadNavigation
Road finder demo.
Definition: RoadNavigation.C:108
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::StdModule::StdModule
StdModule(std::string const &instance)
RawImageOps.H
jevois::RawImage::height
unsigned int height
jevois::InputFrame
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
h
int h
RoadNavigation::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video out.
Definition: RoadNavigation.C:128
jevois::StdModule
ARtoolkit::JEVOIS_DECLARE_PARAMETER
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .dat extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory (if relative stem) or " "from the specified absolute location (if absolute stem).", JEVOIS_SHARE_PATH "/camera/camera_para", ParamCateg)
Parameter.
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(RoadNavigation)
demo.w
w
Definition: demo.py:85
jevois::Timer