JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoBackgroundSubtract.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
23 
24 #include <opencv2/core/core.hpp>
25 #include <opencv2/imgproc/imgproc.hpp>
26 #include <opencv2/video/background_segm.hpp>
27 
28 #include <future>
29 #include <linux/videodev2.h> // for v4l2 pixel types
30 
31 // icon by by Freepik in arrows at flaticon
32 
33 //! Background subtraction to detect moving objects
34 /*! This background subtraction algorithm learns, over time, a statistical model of the appearance of a scene when the
35  camera is not moving. Any moving object entering the field of view will then be detected as significantly different
36  from the learned pixel model.
37 
38  You can thus use this module to detect moving objects with a static JeVois camera. This module is quite robust to
39  image noise since it learns a statistical model of the distribution of "normal" values a pixel may take over time,
40  as opposed to more simply subtracting one camera image from the previous one (which would be much noisier).
41 
42  Trying it out
43  -------------
44 
45  - Make sure that your JeVois camera is stable and not moving (nor vibrating because of its fan); you may want to
46  mount it securely using screws, zip ties, etc
47 
48  - When nothing moves, you should see the right side of the video progressively fade to black. This is the result of
49  this module learning the normal range of RGB values that each pixel may take over time, given sensor noise, small
50  variations in lighting, and other environmental factors.
51 
52  - Now move your hand or some other object in front of JeVois. It should be detected on the right side of the display
53  and should show up in white color.
54 
55  - Note that the autogain and autoexposure features of the JeVois camera sensor may trip this model. For example if
56  you suddenly present a large white object in front of the JeVois camera that had been looking at an overall dark
57  scene for a while, that bright white object might trigger an automatic reduction in exposure, which will in turn
58  brighten the whole image and make it appear much different from what it used to be (so, everything might briefly
59  turn white color in the right panel of the video display). To avoid this, you may want to use this module with
60  manual settings for gain, white balance, and exposure.
61 
62  Note that this class has internal state (it learns the statistics of the background over time).
63 
64 
65  @author Laurent Itti
66 
67  @videomapping YUYV 640 240 15.0 YUYV 320 240 15.0 JeVois DemoBackgroundSubtract
68  @videomapping YUYV 320 120 30.0 YUYV 160 120 30.0 JeVois DemoBackgroundSubtract
69  @email itti\@usc.edu
70  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
71  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
72  @mainurl http://jevois.org
73  @supporturl http://jevois.org/doc
74  @otherurl http://iLab.usc.edu
75  @license GPL v3
76  @distribution Unrestricted
77  @restrictions None
78  \ingroup modules */
80 {
81  public:
82  //! Constructor
83  DemoBackgroundSubtract(std::string const & instance) :
84  jevois::Module(instance), itsProcessingTimer("Processing"), pMOG2(cv::createBackgroundSubtractorMOG2())
85  { }
86 
87  //! Virtual destructor for safe inheritance
89 
90  //! Processing function
91  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
92  {
93  // Wait for next available camera image:
94  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
95  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
96 
98 
99  // Convert it to BGR:
100  cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
101 
102  // Compute the foreground mask in a thread:
103  cv::Mat fgmask;
104  auto fg_fut = std::async(std::launch::async, [&]() { pMOG2->apply(imgbgr, fgmask); });
105 
106  // While computing, wait for an image from our gadget driver into which we will put our results:
107  jevois::RawImage outimg = outframe.get();
108 
109  // Enforce the correct output image size and format:
110  outimg.require("output", w * 2, h, V4L2_PIX_FMT_YUYV);
111 
112  // Paste the original image to the top-left corner of the display:
113  jevois::rawimage::paste(inimg, outimg, 0, 0);
114  jevois::rawimage::writeText(outimg, "JeVois Background Subtraction Demo", 3, 3, jevois::yuyv::White);
115 
116  // Let camera know we are done processing the raw input image:
117  inframe.done();
118 
119  // Wait for the processing results:
120  fg_fut.get();
121 
122  // Paste the results into the output image:
123  jevois::rawimage::pasteGreyToYUYV(fgmask, outimg, w, 0);
124  jevois::rawimage::writeText(outimg, "Foreground Mask", w+3, 3, jevois::yuyv::White);
125 
126  // Show processing fps:
127  std::string const & fpscpu = itsProcessingTimer.stop();
128  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
129 
130  // Send the output image with our processing results to the host over USB:
131  outframe.send();
132  }
133 
134  protected:
136  cv::Ptr<cv::BackgroundSubtractor> pMOG2;
137 };
138 
139 // Allow the module to be loaded as a shared object (.so) file:
DemoBackgroundSubtract(std::string const &instance)
Constructor.
friend friend class Module
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
cv::Mat convertToCvBGR(RawImage const &src)
unsigned int height
JEVOIS_REGISTER_MODULE(DemoBackgroundSubtract)
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
std::string const & stop()
cv::Ptr< cv::BackgroundSubtractor > pMOG2
virtual ~DemoBackgroundSubtract()
Virtual destructor for safe inheritance.
void pasteGreyToYUYV(cv::Mat const &src, RawImage &dest, int dx, int dy)
Background subtraction to detect moving objects.
unsigned int width
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const