JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
SurpriseRecorder.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
22 #include <opencv2/videoio.hpp> // for cv::VideoCapture
23 #include <opencv2/imgproc.hpp> // for cv::rectangle()
24 #include <linux/videodev2.h> // for v4l2 pixel types
25 #include <fstream>
26 #include <jevois/Debug/Profiler.H>
27 
28 static jevois::ParameterCategory const ParamCateg("Surprise Recording Options");
29 
30 #define PATHPREFIX "/jevois/data/surpriserecorder/"
31 
32 //! Parameter \relates SurpriseRecorder
33 JEVOIS_DECLARE_PARAMETER(filename, std::string, "Name of the video file to write. If path is not absolute, "
34  PATHPREFIX " will be prepended to it. Name should contain a printf-like directive for "
35  "one int argument, which will start at 0 and be incremented on each streamoff command.",
36  "video%06d.avi", ParamCateg);
37 
38 //! Parameter \relates SurpriseRecorder
39 JEVOIS_DECLARE_PARAMETER(fourcc, std::string, "FourCC of the codec to use. The OpenCV VideoWriter doc is unclear "
40  "as to which codecs are supported. Presumably, the ffmpeg library is used inside OpenCV. "
41  "Hence any video encoder supported by ffmpeg should work. Tested codecs include: MJPG, "
42  "MP4V, AVC1. Make sure you also pick the right filename extension (e.g., .avi for MJPG, "
43  ".mp4 for MP4V, etc)",
44  "MJPG", boost::regex("^\\w{4}$"), ParamCateg);
45 
46 //! Parameter \relates SurpriseRecorder
47 JEVOIS_DECLARE_PARAMETER(fps, double, "Video frames/sec as stored in the file and to be used both for recording and "
48  "playback. Beware that the video writer will drop frames if you are capturing faster than "
49  "the frame rate specified here. For example, if capturing at 120fps, be sure to set this "
50  "parameter to 120, otherwise by default the saved video will be at 30fps even though capture "
51  "was running at 120fps.",
52  15.0, ParamCateg);
53 
54 //! Parameter \relates SurpriseRecorder
55 JEVOIS_DECLARE_PARAMETER(thresh, double, "Surprise threshold. Lower values will record more events.",
56  1.0e7, ParamCateg);
57 
58 //! Parameter \relates SurpriseRecorder
59 JEVOIS_DECLARE_PARAMETER(ctxframes, unsigned int, "Number of context video frames recorded before and after "
60  "each surprising event.",
61  150, ParamCateg);
62 
63 
64 //! Surprise-based recording of events
65 /*! This module detects surprising events in the live video feed from the camera, and records short video clips of each
66  detected event.
67 
68  Surprising is here defined according to Itti and Baldi's mathematical theory of surprise (see, e.g.,
69  http://ilab.usc.edu/surprise/) which is applied to monitoring live video streams. When a surprising event is
70  detected, a short video clip of that event is saved to the microSD card inside JeVois, for later review.
71 
72  It was created in this JeVois tutorial: http://jevois.org/tutorials/ProgrammerSurprise.html
73 
74  Using this module
75  -----------------
76 
77  This module does not send any video output to USB. Rather, it just saves surprising events to microSD for later
78  review. Hence, you may want to try the following:
79 
80  - mount the JeVois camera where you want it to detect surprising events
81 
82  - run it connected to a laptop computer, using any mode which does have some video output over USB (e.g., 640x500
83  YUYV). Adjust the camera orientation to best fit your needs.
84 
85  - edit <b>JEVOIS:/config/initscript.cfg</b> to contain:
86  \verbatim
87  setmapping2 YUYV 640 480 15.0 JeVois SurpriseRecorder
88  setpar thresh 1e7
89  setpar channels S
90  streamon
91  \endverbatim
92  and see the above tutorial for more details. Next time you power JeVois, it will immediately start detecting and
93  recording surprising events in its view.
94 
95  Example
96  -------
97 
98  Here is one hour of video surveillance footage. It is very boring overall. Except that a few brief surprising things
99  occur (a few seconds each). Can you find them?
100 
101  \youtube{aSKncW7Jxrs}
102 
103  Here is what the SurpriseRecorder module found (4 true events plus 2 false alarms):
104 
105  \youtube{zIslIsHBfYw}
106 
107  With only 6 surprising events, and assuming +/- 10 seconds of context frames around each event, we have achieved a
108  compression of the surveillance footage from 60 minutes to 2 minutes (a factor 30x).
109 
110 
111  @author Laurent Itti
112 
113  @videomapping NONE 0 0 0 YUYV 640 480 15.0 JeVois SurpriseRecorder
114  @email itti\@usc.edu
115  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
116  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
117  @mainurl http://jevois.org
118  @supporturl http://jevois.org/doc
119  @otherurl http://iLab.usc.edu
120  @license GPL v3
121  @distribution Unrestricted
122  @restrictions None
123  \ingroup modules */
125  public jevois::Parameter<filename, fourcc, fps, thresh, ctxframes>
126 
127 {
128  public:
129  // ####################################################################################################
130  //! Constructor
131  // ####################################################################################################
132  SurpriseRecorder(std::string const & instance) : jevois::Module(instance), itsBuf(1000), itsToSave(0),
133  itsFileNum(0), itsRunning(false)
134  { itsSurprise = addSubComponent<Surprise>("surprise"); }
135 
136  // ####################################################################################################
137  //! Virtual destructor for safe inheritance
138  // ####################################################################################################
140  { }
141 
142  // ####################################################################################################
143  //! Get started
144  // ####################################################################################################
145  void postInit() override
146  {
147  itsRunning.store(true);
148 
149  // Get our run() thread going, it is in charge of compressing and saving frames:
150  itsRunFut = jevois::async(std::bind(&SurpriseRecorder::run, this));
151  }
152 
153  // ####################################################################################################
154  //! Get stopped
155  // ####################################################################################################
156  void postUninit() override
157  {
158  // Signal end of run:
159  itsRunning.store(false);
160 
161  // Push an empty frame into our buffer to signal the end of video to our thread:
162  itsBuf.push(cv::Mat());
163 
164  // Wait for the thread to complete:
165  LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
166  try { itsRunFut.get(); } catch (...) { jevois::warnAndIgnoreException(); }
167  LINFO("Writer thread completed. Syncing disk...");
168  if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
169  LINFO("Video " << itsFilename << " saved.");
170  }
171 
172  // ####################################################################################################
173  //! Processing function, version with no video output
174  // ####################################################################################################
175  void process(jevois::InputFrame && inframe) override
176  {
177  static jevois::Profiler prof("surpriserecorder");
178 
179  // Wait for next available camera image:
180  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
181  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
182 
183  prof.start();
184 
185  // Compute surprise in a thread:
186  std::future<double> itsSurpFut =
187  jevois::async([&]() { return itsSurprise->process(inimg); } );
188 
189  prof.checkpoint("surprise launched");
190 
191  // Convert the image to OpenCV BGR and push into our context buffer:
192  cv::Mat cvimg = jevois::rawimage::convertToCvBGR(inimg);
193  itsCtxBuf.push_back(cvimg);
194  if (itsCtxBuf.size() > ctxframes::get()) itsCtxBuf.pop_front();
195 
196  prof.checkpoint("image pushed");
197 
198  // Wait until our surprise thread is done:
199  double surprise = itsSurpFut.get(); // this could throw and that is ok
200  //LINFO("surprise = " << surprise << " itsToSave = " << itsToSave);
201 
202  prof.checkpoint("surprise done");
203 
204  // Let camera know we are done processing the raw input image:
205  inframe.done();
206 
207  // If the current frame is surprising, check whether we are already saving. If so, just push the current frame for
208  // saving and reset itsToSave to full context length (after the event). Otherwise, keep saving until the context
209  // after the event is exhausted:
210  if (surprise >= thresh::get())
211  {
212  // Draw a rectangle on surprising frames. Note that we draw it in cvimg but, since the pixel memory is shared
213  // with the copy of it we just pushed into itsCtxBuf, the rectangle will get drawn in there too:
214  cv::rectangle(cvimg, cv::Point(3, 3), cv::Point(w-4, h-4), cv::Scalar(0,0,255), 7);
215 
216  if (itsToSave)
217  {
218  // we are still saving the context after the previous event, just add our new one:
219  itsBuf.push(cvimg);
220 
221  // Reset the number of frames we will save after the end of the event:
222  itsToSave = ctxframes::get();
223  }
224  else
225  {
226  // Start of a new event. Dump the whole itsCtxBuf to the writer:
227  for (cv::Mat const & im : itsCtxBuf) itsBuf.push(im);
228 
229  // Initialize the number of frames we will save after the end of the event:
230  itsToSave = ctxframes::get();
231  }
232  }
233  else if (itsToSave)
234  {
235  // No more surprising event, but we are still saving the context after the last one:
236  itsBuf.push(cvimg);
237 
238  // One more context frame after the last event was saved:
239  --itsToSave;
240 
241  // Last context frame after the event was just pushed? If so, push an empty frame as well to close the current
242  // video file. We will open a new file on the next surprising event:
243  if (itsToSave == 0) itsBuf.push(cv::Mat());
244  }
245 
246  prof.stop();
247  }
248 
249  // ####################################################################################################
250  protected:
251  std::shared_ptr<Surprise> itsSurprise;
252 
253  // ####################################################################################################
254  //! Video writer thread
255  // ####################################################################################################
256  void run() // Runs in a thread
257  {
258  while (itsRunning.load())
259  {
260  // Create a VideoWriter here, since it has no close() function, this will ensure it gets destroyed and closes
261  // the movie once we stop the recording:
262  cv::VideoWriter writer;
263  int frame = 0;
264 
265  while (true)
266  {
267  // Get next frame from the buffer:
268  cv::Mat im = itsBuf.pop();
269 
270  // An empty image will be pushed when we are ready to close the video file:
271  if (im.empty()) break;
272 
273  // Start the encoder if it is not yet running:
274  if (writer.isOpened() == false)
275  {
276  // Parse the fourcc, regex in our param definition enforces 4 alphanumeric chars:
277  std::string const fcc = fourcc::get();
278  int const cvfcc = cv::VideoWriter::fourcc(fcc[0], fcc[1], fcc[2], fcc[3]);
279 
280  // Add path prefix if given filename is relative:
281  std::string fn = filename::get();
282  if (fn.empty()) LFATAL("Cannot save to an empty filename");
283  if (fn[0] != '/') fn = PATHPREFIX + fn;
284 
285  // Create directory just in case it does not exist:
286  std::string const cmd = "/bin/mkdir -p " + fn.substr(0, fn.rfind('/'));
287  if (std::system(cmd.c_str())) LERROR("Error running [" << cmd << "] -- IGNORED");
288 
289  // Fill in the file number; be nice and do not overwrite existing files:
290  while (true)
291  {
292  char tmp[2048];
293  std::snprintf(tmp, 2047, fn.c_str(), itsFileNum);
294  std::ifstream ifs(tmp);
295  if (ifs.is_open() == false) { itsFilename = tmp; break; }
296  ++itsFileNum;
297  }
298 
299  // Open the writer:
300  if (writer.open(itsFilename, cvfcc, fps::get(), im.size(), true) == false)
301  LFATAL("Failed to open video encoder for file [" << itsFilename << ']');
302 
303  sendSerial("SAVETO " + itsFilename);
304  }
305 
306  // Write the frame:
307  writer << im;
308 
309  // Report what is going on once in a while:
310  if ((++frame % 100) == 0) sendSerial("SAVEDNUM " + std::to_string(frame));
311  }
312 
313  sendSerial("SAVEDONE " + itsFilename);
314 
315  // Our writer runs out of scope and closes the file here.
316  ++itsFileNum;
317  }
318  }
319 
320  std::future<void> itsRunFut; //!< Future for our run() thread
321  std::deque<cv::Mat> itsCtxBuf; //!< Buffer for context frames before event start
323  jevois::BlockingBehavior::Block> itsBuf; //!< Buffer for frames to save
324  int itsToSave; //!< Number of context frames after end of event that remain to be saved
325  int itsFileNum; //!< Video file number
326  std::atomic<bool> itsRunning; //!< Flag to let run thread when to quit
327  std::string itsFilename; //!< Current video file name
328 };
329 
330 // Allow the module to be loaded as a shared object (.so) file:
SurpriseRecorder::SurpriseRecorder
SurpriseRecorder(std::string const &instance)
Constructor.
Definition: SurpriseRecorder.C:132
Profiler.H
SurpriseRecorder::itsRunning
std::atomic< bool > itsRunning
Flag to let run thread when to quit.
Definition: SurpriseRecorder.C:326
SurpriseRecorder
Surprise-based recording of events.
Definition: SurpriseRecorder.C:124
Module.H
jevois::Profiler::checkpoint
void checkpoint(char const *description)
SurpriseRecorder::itsSurprise
std::shared_ptr< Surprise > itsSurprise
Definition: SurpriseRecorder.C:251
jevois::BoundedBuffer
jevois::RawImage
Surprise.H
jevois::rawimage::convertToCvBGR
cv::Mat convertToCvBGR(RawImage const &src)
jevois::ParameterCategory
LERROR
#define LERROR(msg)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
SurpriseRecorder::~SurpriseRecorder
virtual ~SurpriseRecorder()
Virtual destructor for safe inheritance.
Definition: SurpriseRecorder.C:139
jevois::RawImage::width
unsigned int width
jevois
SurpriseRecorder::postUninit
void postUninit() override
Get stopped.
Definition: SurpriseRecorder.C:156
system
std::string system(std::string const &cmd, bool errtoo=true)
surprise
Definition: Surprise.H:22
jevois::BoundedBuffer::pop
T pop()
SurpriseRecorder::itsFileNum
int itsFileNum
Video file number.
Definition: SurpriseRecorder.C:325
SurpriseRecorder::postInit
void postInit() override
Get started.
Definition: SurpriseRecorder.C:145
jevois::BoundedBuffer::push
void push(T const &val)
SurpriseRecorder::itsFilename
std::string itsFilename
Current video file name.
Definition: SurpriseRecorder.C:327
jevois::Profiler
SurpriseRecorder::itsRunFut
std::future< void > itsRunFut
Future for our run() thread.
Definition: SurpriseRecorder.C:320
SurpriseRecorder::itsCtxBuf
std::deque< cv::Mat > itsCtxBuf
Buffer for context frames before event start.
Definition: SurpriseRecorder.C:321
jevois::warnAndIgnoreException
std::string warnAndIgnoreException(std::string const &prefix="")
SurpriseRecorder::itsBuf
jevois::BoundedBuffer< cv::Mat, jevois::BlockingBehavior::Block, jevois::BlockingBehavior::Block > itsBuf
Buffer for frames to save.
Definition: SurpriseRecorder.C:323
jevois::BlockingBehavior::Block
@ Block
jevois::BoundedBuffer::filled_size
size_t filled_size() const
LFATAL
#define LFATAL(msg)
jevois::Module
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
RawImageOps.H
jevois::RawImage::height
unsigned int height
jevois::Profiler::start
void start()
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(SurpriseRecorder)
to_string
std::string to_string(T const &val)
jevois::InputFrame
test-model.frame
frame
Definition: test-model.py:22
SurpriseRecorder::process
void process(jevois::InputFrame &&inframe) override
Processing function, version with no video output.
Definition: SurpriseRecorder.C:175
jevois::Profiler::stop
void stop()
PATHPREFIX
#define PATHPREFIX
Definition: SurpriseRecorder.C:30
h
int h
jevois::Module::sendSerial
virtual void sendSerial(std::string const &str)
BoundedBuffer.H
ARtoolkit::JEVOIS_DECLARE_PARAMETER
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .dat extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory (if relative stem) or " "from the specified absolute location (if absolute stem).", JEVOIS_SHARE_PATH "/camera/camera_para", ParamCateg)
Parameter.
LINFO
#define LINFO(msg)
SurpriseRecorder::run
void run()
Video writer thread.
Definition: SurpriseRecorder.C:256
jevois::Component::Module
friend friend class Module
SurpriseRecorder::itsToSave
int itsToSave
Number of context frames after end of event that remain to be saved.
Definition: SurpriseRecorder.C:324