JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoBackgroundSubtract.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
23 
24 #include <opencv2/core/core.hpp>
25 #include <opencv2/imgproc/imgproc.hpp>
26 #include <opencv2/video/background_segm.hpp>
27 
28 #include <future>
29 #include <linux/videodev2.h> // for v4l2 pixel types
30 
31 // icon by by Freepik in arrows at flaticon
32 
33 //! Background subtraction to detect moving objects
34 /*! This background subtraction algorithm learns, over time, a statistical model of the appearance of a scene when the
35  camera is not moving. Any moving object entering the field of view will then be detected as significantly different
36  from the learned pixel model.
37 
38  You can thus use this module to detect moving objects with a static JeVois camera. This module is quite robust to
39  image noise since it learns a statistical model of the distribution of "normal" values a pixel may take over time,
40  as opposed to more simply subtracting one camera image from the previous one (which would be much noisier).
41 
42  Trying it out
43  -------------
44 
45  - Make sure that your JeVois camera is stable and not moving (nor vibrating because of its fan); you may want to
46  mount it securely using screws, zip ties, etc
47 
48  - When nothing moves, you should see the right side of the video progressively fade to black. This is the result of
49  this module learning the normal range of RGB values that each pixel may take over time, given sensor noise, small
50  variations in lighting, and other environmental factors.
51 
52  - Now move your hand or some other object in front of JeVois. It should be detected on the right side of the display
53  and should show up in white color.
54 
55  - Note that the autogain and autoexposure features of the JeVois camera sensor may trip this model. For example if
56  you suddenly present a large white object in front of the JeVois camera that had been looking at an overall dark
57  scene for a while, that bright white object might trigger an automatic reduction in exposure, which will in turn
58  brighten the whole image and make it appear much different from what it used to be (so, everything might briefly
59  turn white color in the right panel of the video display). To avoid this, you may want to use this module with
60  manual settings for gain, white balance, and exposure.
61 
62  Note that this class has internal state (it learns the statistics of the background over time).
63 
64 
65  @author Laurent Itti
66 
67  @videomapping YUYV 640 240 15.0 YUYV 320 240 15.0 JeVois DemoBackgroundSubtract
68  @videomapping YUYV 320 120 30.0 YUYV 160 120 30.0 JeVois DemoBackgroundSubtract
69  @email itti\@usc.edu
70  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
71  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
72  @mainurl http://jevois.org
73  @supporturl http://jevois.org/doc
74  @otherurl http://iLab.usc.edu
75  @license GPL v3
76  @distribution Unrestricted
77  @restrictions None
78  \ingroup modules */
80 {
81  public:
82  //! Constructor
83  DemoBackgroundSubtract(std::string const & instance) :
84  jevois::Module(instance), itsProcessingTimer("Processing"), bgs(cv::createBackgroundSubtractorMOG2())
85  { }
86 
87  //! Virtual destructor for safe inheritance
89 
90  //! Processing function
91  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
92  {
93  // Wait for next available camera image:
94  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
95  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
96 
98 
99  // Convert it to BGR:
100  cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
101 
102  // Paste the original image to the top-left corner of the display in a thread:
103  jevois::RawImage outimg;
104  auto paste_fut =
105  jevois::async([&]()
106  {
107  outimg = outframe.get();
108  outimg.require("output", w * 2, h, V4L2_PIX_FMT_YUYV);
109  jevois::rawimage::paste(inimg, outimg, 0, 0);
110  jevois::rawimage::writeText(outimg, "JeVois Background Subtraction", 3, 3,
111  jevois::yuyv::White);
112  });
113 
114  // Compute the foreground mask in a thread:
115  cv::Mat fgmask;
116  bgs->apply(imgbgr, fgmask);
117 
118  // Wait for paste to be done:
119  paste_fut.get();
120 
121  // Let camera know we are done processing the raw input image:
122  inframe.done();
123 
124  // Paste the results into the output image:
125  jevois::rawimage::pasteGreyToYUYV(fgmask, outimg, w, 0);
126  jevois::rawimage::writeText(outimg, "Foreground Mask", w+3, 3, jevois::yuyv::White);
127 
128  // Show processing fps:
129  std::string const & fpscpu = itsProcessingTimer.stop();
130  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
131 
132  // Send the output image with our processing results to the host over USB:
133  outframe.send();
134  }
135 
136 #ifdef JEVOIS_PRO
137  // ####################################################################################################
138  //! Processing function with zero-copy and GUI on JeVois-Pro
139  // ####################################################################################################
140  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
141  {
142  static jevois::Timer timer("processing", 100, LOG_DEBUG);
143 
144  // Start the GUI frame:
145  unsigned short winw, winh;
146  bool idle = helper.startFrame(winw, winh);
147 
148  // Draw the camera frame:
149  int x = 0, y = 0; unsigned short iw = 0, ih = 0;
150  helper.drawInputFrame("camera", inframe, x, y, iw, ih);
151  helper.itext("JeVois-Pro Background Subtraction");
152 
153  // Wait for next available camera image:
154  cv::Mat imgbgr = inframe.getCvRGBp();
155 
156  timer.start();
157 
158  // Compute the foreground mask:
159  cv::Mat fgmask;
160  bgs->apply(imgbgr, fgmask);
161 
162  // Convert to RGBA, with zero alpha (transparent) in background and full alpha on foreground:
163  cv::Mat chans[4] { fgmask, fgmask, fgmask, fgmask };
164  cv::Mat mask; cv::merge(chans, 4, mask);
165 
166  // Draw overlay:
167  helper.drawImage("fgmask", mask, true, x, y, iw, ih, false /* noalias */, true /* isoverlay */);
168 
169  // Show processing fps:
170  std::string const & fpscpu = timer.stop();
171  helper.iinfo(inframe, fpscpu, winw, winh);
172 
173  // Render the image and GUI:
174  helper.endFrame();
175  }
176 #endif
177 
178  protected:
180  cv::Ptr<cv::BackgroundSubtractor> bgs;
181 };
182 
183 // Allow the module to be loaded as a shared object (.so) file:
jevois::GUIhelper::drawImage
void drawImage(char const *name, RawImage const &img, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool isoverlay=false)
jevois::OutputFrame
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
DemoBackgroundSubtract::itsProcessingTimer
jevois::Timer itsProcessingTimer
Definition: DemoBackgroundSubtract.C:179
DemoBackgroundSubtract::bgs
cv::Ptr< cv::BackgroundSubtractor > bgs
Definition: DemoBackgroundSubtract.C:180
Timer.H
jevois::GUIhelper::itext
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
Module.H
jevois::GUIhelper
jevois::GUIhelper::endFrame
void endFrame()
Log.H
jevois::RawImage
jevois::Timer::start
void start()
jevois::rawimage::convertToCvBGR
cv::Mat convertToCvBGR(RawImage const &src)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
DemoBackgroundSubtract::DemoBackgroundSubtract
DemoBackgroundSubtract(std::string const &instance)
Constructor.
Definition: DemoBackgroundSubtract.C:83
DemoBackgroundSubtract::~DemoBackgroundSubtract
virtual ~DemoBackgroundSubtract()
Virtual destructor for safe inheritance.
Definition: DemoBackgroundSubtract.C:88
jevois::RawImage::width
unsigned int width
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DemoBackgroundSubtract)
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::Module
DemoBackgroundSubtract
Background subtraction to detect moving objects.
Definition: DemoBackgroundSubtract.C:79
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
DemoBackgroundSubtract::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoBackgroundSubtract.C:91
jevois::InputFrame
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
h
int h
jevois::rawimage::pasteGreyToYUYV
void pasteGreyToYUYV(cv::Mat const &src, RawImage &dest, int dx, int dy)
jevois::Timer
jevois::Component::Module
friend friend class Module