JeVoisBase  1.20
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ObjectTracker.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Util/Utils.H>
23 #include <jevois/Debug/Timer.H>
25 
26 #include <linux/videodev2.h>
27 #include <opencv2/imgproc/imgproc.hpp>
28 #include <string.h>
29 
30 // This code was loosely inspired by:
31 
32 // https://raw.githubusercontent.com/kylehounslow/opencv-tuts/master/object-tracking-tut/objectTrackingTut.cpp
33 
34 // Written by Kyle Hounslow 2013
35 
36 // Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
37 // documentation files (the "Software") , to deal in the Software without restriction, including without limitation the
38 // rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
39 // permit persons to whom the Software is furnished to do so, subject to the following conditions:
40 //
41 // The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
42 // Software.
43 //
44 // THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
45 // WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
46 // COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
47 // OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
48 
49 // icon by Roundicons in miscellaneous at flaticon
50 
51 //! Simple color-based object detection/tracking
52 /*! This modules isolates pixels within a given HSV range (hue, saturation, and value of color pixels), does some
53  cleanups, and extracts object contours. It sends information about object centers over serial.
54 
55  This module usually works best with the camera sensor set to manual exposure, manual gain, manual color balance, etc
56  so that HSV color values are reliable. See the file \b script.cfg file in this module's directory for an example of
57  how to set the camera settings each time this module is loaded.
58 
59  This code was loosely inspired by:
60  https://raw.githubusercontent.com/kylehounslow/opencv-tuts/master/object-tracking-tut/objectTrackingTut.cpp written
61  by Kyle Hounslow, 2013.
62 
63  Serial Messages
64  ---------------
65 
66  This module can send standardized serial messages as described in \ref UserSerialStyle. One message is issued on
67  every video frame for each detected and good object (good objects have a pixel area within the range specified by \p
68  objectarea, and are only reported when the image is clean enough according to \p maxnumobj). The \p id field in the
69  messages simply is \b blob for all messages.
70 
71  - Serial message type: \b 2D
72  - `id`: always \b blob
73  - `x`, `y`, or vertices: standardized 2D coordinates of blob center or of corners of bounding box
74  (depending on \p serstyle)
75  - `w`, `h`: standardized object size
76  - `extra`: none (empty string)
77 
78  \jvversion{1.8.1}: Note that this module sends one message per detected object (blob) and per frame. You may want to
79  specify `setpar serstamp Frame` if you need to get all the objects on one given frame (they will have the same frame
80  number prefix). Here is an example output with `setpar serstyle Detail` and `setpar serstamp Frame` (see \ref
81  UserSerialStyle for more info about \p serstamp and \p serstyle):
82 
83  \verbatim
84  2940 D2 blob 4 -238 237 -238 19 13 19 13 237
85  2940 D2 blob 4 266 209 194 206 213 -277 285 -274
86  2940 D2 blob 4 575 -243 499 -511 613 -544 689 -275
87  2940 D2 blob 4 -150 -413 -150 -613 50 -613 50 -413
88  2940 D2 blob 4 -913 -306 -1000 -306 -1000 -750 -913 -750
89  2941 D2 blob 4 -313 119 -313 -106 -63 -106 -63 119
90  2941 D2 blob 4 189 105 115 99 157 -385 231 -379
91  2941 D2 blob 4 469 -275 469 -500 575 -500 575 -275
92  2941 D2 blob 4 -200 -531 -200 -744 -13 -744 -13 -531
93  2942 D2 blob 4 -381 -13 -381 -244 -113 -244 -113 -13
94  2942 D2 blob 4 478 -213 310 -402 414 -494 582 -305
95  2942 D2 blob 4 144 -6 53 -15 114 -668 206 -660
96  \endverbatim
97  In this example, we detected 5 blobs on frame 2940, then 4 blobs on frame 2941, then 3 blobs on frame 2942.
98 
99  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
100  standardized coordinates.
101 
102  Trying it out
103  -------------
104 
105  The default parameter settings (which are set in \b script.cfg explained below) attempt to detect light blue
106  objects. Present a light blue object to the JeVois camera and see whether it is detected. When detected and good
107  enough according to \p objectarea and \p maxnumobj, a green circle will be drawn at the center of each good object.
108 
109  For further use of this module, you may want to check out the following tutorials:
110 
111  - [Tuning the color-based object tracker using a python graphical
112  interface](http://jevois.org/tutorials/UserColorTracking.html)
113  - [Making a motorized pan-tilt head for JeVois and tracking
114  objects](http://jevois.org/tutorials/UserPanTilt.html)
115  - \ref ArduinoTutorial
116 
117  Tuning
118  ------
119 
120  You should adjust parameters \p hrange, \p srange, and \p vrange to isolate the range of Hue, Saturation, and Value
121  (respectively) that correspond to the objects you want to detect. Note that there is a \b script.cfg file in this
122  module's directory that provides a range tuned to a lighgt blue object, as shown in the demo screenshot.
123 
124  Tuning the parameters is best done interactively by connecting to your JeVois camera while it is looking at some
125  object of the desired color. Once you have achieved a tuning, you may want to set the hrange, srange, and vrange
126  parameters in your \b script.cfg file for this module on the microSD card (see below).
127 
128  Typically, you would start by narrowing down on the hue, then the value, and finally the saturation. Make sure you
129  also move your camera around and show it typical background clutter so check for false positives (detections of
130  things which you are not interested, which can happen if your ranges are too wide).
131 
132  Config file
133  -----------
134 
135  JeVois allows you to store parameter settings and commands in a file named \b script.cfg stored in the directory of
136  a module. The file \b script.cfg may contain any sequence of commands as you would type them interactively in the
137  JeVois command-line interface. For the \jvmod{ObjectTracker} module, a default script is provided that sets the
138  camera to manual color, gain, and exposure mode (for more reliable color values), and to setup communication with a
139  pan/tilt head as described in \ref ArduinoTutorial.
140 
141  The \b script.cfg file for \jvmod{ObjectTracker} is stored on your microSD at
142  <b>JEVOIS:/modules/JeVois/ObjectTracker/script.cfg</b> and is shown in \ref ArduinoTutorial as an example.
143 
144 
145  @author Laurent Itti
146 
147  @videomapping YUYV 320 254 60.0 YUYV 320 240 60.0 JeVois ObjectTracker
148  @videomapping NONE 0 0 0.0 YUYV 320 240 60.0 JeVois ObjectTracker
149  @email itti\@usc.edu
150  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
151  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
152  @mainurl http://jevois.org
153  @supporturl http://jevois.org/doc
154  @otherurl http://iLab.usc.edu
155  @license GPL v3
156  @distribution Unrestricted
157  @restrictions None
158  \ingroup modules */
160 {
161  public:
162  //! Constructor
163  ObjectTracker(std::string const & instance) :
164  jevois::StdModule(instance)
165  {
166  itsDetector = addSubComponent<BlobDetector>("detector");
167  }
168 
169  //! Virtual destructor for safe inheritance
170  virtual ~ObjectTracker() { }
171 
172  //! Processing function, no USB video output
173  virtual void process(jevois::InputFrame && inframe) override
174  {
175  // Wait for next available camera image. Any resolution and format ok:
176  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
177 
178  // Convert input image to BGR24, then to HSV:
179  cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
180  cv::Mat imghsv; cv::cvtColor(imgbgr, imghsv, cv::COLOR_BGR2HSV);
181 
182  // Let camera know we are done processing the input image:
183  inframe.done();
184 
185  // Detect blobs and get their contours:
186  auto contours = itsDetector->detect(imghsv);
187 
188  // Send a serial message for each detected blob:
189  for (auto const & c : contours) sendSerialContour2D(w, h, c, "blob");
190  }
191 
192  //! Processing function, with USB video output
193  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
194  {
195  static jevois::Timer timer("processing");
196 
197  // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
198  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
199  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
200 
201  timer.start();
202 
203  // While we process it, start a thread to wait for output frame and paste the input image into it:
204  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
205  auto paste_fut = jevois::async([&]() {
206  outimg = outframe.get();
207  outimg.require("output", w, h + 14, inimg.fmt);
208  jevois::rawimage::paste(inimg, outimg, 0, 0);
209  jevois::rawimage::writeText(outimg, "JeVois Color Object Tracker", 3, 3, jevois::yuyv::White);
210  jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
211  });
212 
213  // Convert input image to BGR24, then to HSV:
214  cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
215  cv::Mat imghsv; cv::cvtColor(imgbgr, imghsv, cv::COLOR_BGR2HSV);
216 
217  // Detect blobs and get their contours:
218  auto contours = itsDetector->detect(imghsv);
219 
220  // Wait for paste to finish up:
221  paste_fut.get();
222 
223  // Let camera know we are done processing the input image:
224  inframe.done();
225 
226  // Draw all detected contours in a thread:
227  std::future<void> draw_fut = jevois::async([&]() {
228  // We reinterpret the top portion of our YUYV output image as an opencv 8UC2 image:
229  cv::Mat outuc2 = jevois::rawimage::cvImage(outimg); // pixel data shared
230  cv::drawContours(outuc2, contours, -1, jevois::yuyv::LightPink, 2, 8);
231  });
232 
233  // Send a serial message and draw a circle for each detected blob:
234  for (auto const & c : contours)
235  {
236  sendSerialContour2D(w, h, c, "blob");
237 
238  cv::Moments moment = cv::moments(c);
239  double const area = moment.m00;
240  int const x = int(moment.m10 / area + 0.4999);
241  int const y = int(moment.m01 / area + 0.4999);
242  jevois::rawimage::drawCircle(outimg, x, y, 20, 1, jevois::yuyv::LightGreen);
243  }
244 
245  // Show number of detected objects:
246  jevois::rawimage::writeText(outimg, "Detected " + std::to_string(contours.size()) + " objects.",
247  3, h + 2, jevois::yuyv::White);
248 
249  // Show processing fps:
250  std::string const & fpscpu = timer.stop();
251  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
252 
253  // Wait until all contours are drawn, if they had been requested:
254  draw_fut.get();
255 
256  // Send the output image with our processing results to the host over USB:
257  outframe.send();
258  }
259 
260  private:
261  std::shared_ptr<BlobDetector> itsDetector;
262 };
263 
264 // Allow the module to be loaded as a shared object (.so) file:
ObjectTracker::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no USB video output.
Definition: ObjectTracker.C:173
jevois::StdModule::sendSerialContour2D
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
jevois::OutputFrame
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
demo.int
int
Definition: demo.py:37
Timer.H
Module.H
BlobDetector.H
Coordinates.H
Log.H
jevois::RawImage
jevois::rawimage::drawCircle
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
jevois::Timer::start
void start()
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(ObjectTracker)
jevois::rawimage::convertToCvBGR
cv::Mat convertToCvBGR(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::RawImage::width
unsigned int width
ObjectTracker
Simple color-based object detection/tracking.
Definition: ObjectTracker.C:159
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
area
double area(const std::vector< Point2D< T > > &polygon, const bool getsigned=false)
What is the area of a polygon?
Definition: Point2D.H:422
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::rawimage::drawFilledRect
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
jevois::StdModule::StdModule
StdModule(std::string const &instance)
RawImageOps.H
jevois::RawImage::height
unsigned int height
to_string
std::string to_string(T const &val)
jevois::InputFrame
jevois::rawimage::cvImage
cv::Mat cvImage(RawImage const &src)
ObjectTracker::ObjectTracker
ObjectTracker(std::string const &instance)
Constructor.
Definition: ObjectTracker.C:163
Utils.H
ObjectTracker::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, with USB video output.
Definition: ObjectTracker.C:193
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
h
int h
jevois::StdModule
ObjectTracker::~ObjectTracker
virtual ~ObjectTracker()
Virtual destructor for safe inheritance.
Definition: ObjectTracker.C:170
demo.w
w
Definition: demo.py:85
jevois::Timer