JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
SaveVideo.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
22 
23 #include <opencv2/core/version.hpp>
24 
25 #include <opencv2/videoio.hpp> // for cv::VideoCapture
26 #include <opencv2/imgproc/imgproc.hpp>
27 
28 #include <future>
29 #include <linux/videodev2.h> // for v4l2 pixel types
30 #include <cstdlib> // for std::system()
31 #include <cstdio> // for snprintf()
32 #include <fstream>
33 
34 // icon by Madebyoliver in multimedia at flaticon
35 
36 static jevois::ParameterCategory const ParamCateg("Video Saving Options");
37 
38 #define PATHPREFIX JEVOIS_ROOT_PATH "/data/savevideo/"
39 
40 //! Parameter \relates SaveVideo
41 JEVOIS_DECLARE_PARAMETER(filename, std::string, "Name of the video file to write. If path is not absolute, "
42  PATHPREFIX " will be prepended to it. Name should contain a printf-like directive for "
43  "one int argument, which will start at 0 and be incremented on each streamoff command.",
44  "video%06d.avi", ParamCateg);
45 
46 //! Parameter \relates SaveVideo
47 JEVOIS_DECLARE_PARAMETER(fourcc, std::string, "FourCC of the codec to use. The OpenCV VideoWriter doc is unclear "
48  "as to which codecs are supported. Presumably, the ffmpeg library is used inside OpenCV. "
49  "Hence any video encoder supported by ffmpeg should work. Tested codecs include: MJPG, "
50  "MP4V, AVC1. Make sure you also pick the right filename extension (e.g., .avi for MJPG, "
51  ".mp4 for MP4V, etc)",
52  "MJPG", boost::regex("^\\w{4}$"), ParamCateg);
53 
54 //! Parameter \relates SaveVideo
55 JEVOIS_DECLARE_PARAMETER(fps, double, "Video frames/sec as stored in the file and to be used both for recording and "
56  "playback. Beware that the video writer will drop frames if you are capturing faster than "
57  "the frame rate specified here. For example, if capturing at 120fps, be sure to set this "
58  "parameter to 120, otherwise by default the saved video will be at 30fps even though capture "
59  "was running at 120fps.",
60  30.0, ParamCateg);
61 
62 //! Save captured camera frames into a video file
63 /*! This module records video and saves it to the MicroSD card inside the JeVois smart camera. It is useful, for
64  example, to record footage that will be used to train some machine vision algorithm.
65 
66  Issue the command \c start over the command-line interface to start saving video frames, and \c stop to stop
67  saving. Successive start/stop commands will increment the file number (%d argument in the \p filename
68  parameter. Before a file is written, the module checks whether it already exists, and, if so, skips over it by
69  incrementing the file number. See \ref UserCli for details on how to use the command-line interface.
70 
71  This module works with any video resolution and pixel format supported by the camera sensor. Thus, additional video
72  mappings are possible beyond the ones listed here.
73 
74  See \ref PixelFormats for information about pixel formats; with this module you can use the formats supported by the
75  camera sensor: YUYV, BAYER, RGB565
76 
77  This module accepts any resolution supported by the JeVois camera sensor:
78 
79  - SXGA (1280 x 1024): up to 15 fps
80  - VGA (640 x 480): up to 30 fps
81  - CIF (352 x 288): up to 60 fps
82  - QVGA (320 x 240): up to 60 fps
83  - QCIF (176 x 144): up to 120 fps
84  - QQVGA (160 x 120): up to 60 fps
85  - QQCIF (88 x 72): up to 120 fps
86 
87  This module can operate both with USB video output, or no USB video output.
88 
89  When using with no USB output (NONE output format), you should first issue a \c streamon command to start video
90  streaming by the camera sensor chip, then issue a \c start when you are ready to start recording. The \c streamon is
91  not necessary when using a video mapping with USB video output, as the host computer over USB triggers video
92  streaming when it starts grabbing frames from the JeVois camera.
93 
94  This module internally uses the OpenCV VideoWriter class to compress and write the video file. See the OpenCV
95  documentation for which video formats are supported.
96 
97  You should be aware of two things when attempting video recording at high frame rates:
98 
99  - If using a video mapping with USB output, the frame rate may be limited by the maximum rate at which the host
100  computer can grab and display. On many host computers, for example, the display refresh rate might be 60Hz. Using
101  a video capture software on these computers is likely going to limit the maximum display rate to 60 frames/s and
102  that will in turn limit the capture and saving rate. This is not an issue when using a video mapping with no USB
103  output.
104 
105  - The \p fps parameter should be set to the rate at which you want to save video. If you capture at 60 frames/s
106  according to your video mapping but \p fps is set to 30, saving will be at 30 fps. This limitation of rate is done
107  internally by the OpenCV VideoWriter. So just make sure that you set the \p fps parameter to the rate at which you
108  want to save.
109 
110  Note that this module may suffer from DMA coherency artifacts if the \c camturbo parameter of the jevois::Engine is
111  turned on, which it is by default. The \c camturbo parameter relaxes some of the cache coherency constraints on the
112  video buffers captured by the camera sensor, which allows the JeVois processor to access video pixel data from
113  memory faster. But with modules that do not do much processing, sometimes this yields video artifacts, we presume
114  because some of the video data from previous frames still is in the CPU cache and hence is not again fetched from
115  main memory by the CPU. If you see short stripes of what appears to be wrong pixel colors in the video, try to
116  disable \c camturbo: edit JEVOIS:/config/params.cfg on your MicroSD card and in there turn \c camturbo to false.
117 
118 
119  Example use
120  -----------
121 
122  Connect to the JeVois command-line interface (see \ref UserCli), and issue
123 
124  \verbatim
125  setmapping2 YUYV 640 480 30.0 JeVois SaveVideo
126  streamon
127  start
128  \endverbatim
129 
130  and you will be saving compressed video with 640x480 at 30fps to your microSD card inside JeVois. When you are done,
131  just issue:
132 
133  \verbatim
134  stop
135  \endverbatim
136 
137  and the video file will be finalized and closed. If you want to access it directly by exporting the microSD inside
138  JeVois as a virtual USB flash drive, issue (with \jvversion{1.3} and later):
139 
140  \verbatim
141  streamoff
142  \endverbatim
143 
144  to stop camera sensor streaming, and
145 
146  \verbatim
147  usbsd
148  \endverbatim
149 
150  to export the microSD as a virtual USB flash drive. The video file(s) will be in the <b>JEVOIS:/data/savevideo/</b>
151  folder. You can then use the recorded video to test and debug a new algorithm using your host computer, by running
152  \c jevois-daemon on the host computer with a video file input as opposed to a live camera input, as follows:
153 
154  \verbatim
155  jevois-daemon --cameradev=myvideo.avi --videomapping=num
156  \endverbatim
157 
158  where you replace \a num above by the video mapping you want to use. See \ref JeVoisDaemon for more info about the
159  \p cameradev parameter.
160 
161  @author Laurent Itti
162 
163  @videomapping YUYV 320 240 60.0 YUYV 320 240 60.0 JeVois SaveVideo
164  @videomapping YUYV 320 240 30.0 YUYV 320 240 30.0 JeVois SaveVideo
165  @videomapping NONE 0 0 0 YUYV 320 240 60.0 JeVois SaveVideo
166  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois SaveVideo
167  @videomapping NONE 0 0 0 YUYV 176 144 120.0 JeVois SaveVideo
168  @modulecommand start - start saving video
169  @modulecommand stop - stop saving video and increment video file number
170  @email itti\@usc.edu
171  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
172  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
173  @mainurl http://jevois.org
174  @supporturl http://jevois.org/doc
175  @otherurl http://iLab.usc.edu
176  @license GPL v3
177  @distribution Unrestricted
178  @restrictions None
179  \ingroup modules */
180 class SaveVideo : public jevois::Module,
181  public jevois::Parameter<filename, fourcc, fps>
182 {
183  public:
184  // ####################################################################################################
185  //! Constructor
186  // ####################################################################################################
187  SaveVideo(std::string const & instance) : jevois::Module(instance), itsBuf(1000), itsSaving(false),
188  itsFileNum(0), itsRunning(false)
189  { }
190 
191  // ####################################################################################################
192  //! Get started
193  // ####################################################################################################
194  void postInit() override
195  {
196  itsRunning.store(true);
197 
198  // Get our run() thread going, it is in charge of compresing and saving frames:
199  itsRunFut = std::async(std::launch::async, &SaveVideo::run, this);
200  }
201 
202  // ####################################################################################################
203  //! Get stopped
204  // ####################################################################################################
205  void postUninit() override
206  {
207  // Signal end of run:
208  itsRunning.store(false);
209 
210  // Push an empty frame into our buffer to signal the end of video to our thread:
211  itsBuf.push(cv::Mat());
212 
213  // Wait for the thread to complete:
214  LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
215  try { itsRunFut.get(); } catch (...) { jevois::warnAndIgnoreException(); }
216  LINFO("Writer thread completed. Syncing disk...");
217  if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
218  LINFO("Video " << itsFilename << " saved.");
219  }
220 
221  // ####################################################################################################
222  //! Virtual destructor for safe inheritance
223  // ####################################################################################################
224  virtual ~SaveVideo()
225  { }
226 
227  // ####################################################################################################
228  //! Processing function, version that also shows a debug video output
229  // ####################################################################################################
230  void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
231  {
232  // Wait for next available camera image:
233  jevois::RawImage inimg = inframe.get(true); unsigned int const w = inimg.width, h = inimg.height;
234  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
235 
236  if (itsSaving.load())
237  {
238  // Convert image to BGR and push to our writer thread:
239  if (itsBuf.filled_size() > 1000) LERROR("Image queue too large, video writer cannot keep up - DROPPING FRAME");
241  }
242 
243  // Copy the input image to output:
244  jevois::RawImage outimg = outframe.get();
245  outimg.require("output", w, h, V4L2_PIX_FMT_YUYV);
246 
247  jevois::rawimage::paste(inimg, outimg, 0, 0);
248 
249  // Let camera know we are done processing the raw YUV input image:
250  inframe.done();
251 
252  // Show some text messages:
253  std::string txt = "SaveVideo: "; if (itsSaving.load()) txt += "RECORDING"; else txt += "not recording";
254  jevois::rawimage::writeText(outimg, txt.c_str(), 3, 3, jevois::yuyv::White);
255  jevois::rawimage::writeText(outimg, itsFilename.c_str(), 3, h - 13, jevois::yuyv::White);
256 
257  // Send output frame over USB:
258  outframe.send();
259  }
260 
261  // ####################################################################################################
262  //! Processing function, version with no video output
263  // ####################################################################################################
264  void process(jevois::InputFrame && inframe) override
265  {
266  // Wait for next available camera image:
267  jevois::RawImage inimg = inframe.get(true);
268 
269  if (itsSaving.load())
270  {
271  // Convert image to BGR and push to our writer thread:
272  if (itsBuf.filled_size() > 1000) LERROR("Image queue too large, video writer cannot keep up - DROPPING FRAME");
274  }
275 
276  // Let camera know we are done processing the raw YUV input image:
277  inframe.done();
278  }
279 
280  // ####################################################################################################
281  //! Receive a string from a serial port which contains a user command
282  // ####################################################################################################
283  void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> s) override
284  {
285  if (str == "start")
286  {
287  itsSaving.store(true);
288  sendSerial("SAVESTART");
289  }
290  else if (str == "stop")
291  {
292  itsSaving.store(false);
293  sendSerial("SAVESTOP");
294 
295  // Push an empty frame into our buffer to signal the end of video to our thread:
296  itsBuf.push(cv::Mat());
297 
298  // Wait for the thread to empty our image buffer:
299  while (itsBuf.filled_size())
300  {
301  LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
302  std::this_thread::sleep_for(std::chrono::milliseconds(200));
303  }
304  LINFO("Writer thread completed. Syncing disk...");
305  if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
306  LINFO("Video " << itsFilename << " saved.");
307  }
308  else throw std::runtime_error("Unsupported module command");
309  }
310 
311  // ####################################################################################################
312  //! Human-readable description of this Module's supported custom commands
313  // ####################################################################################################
314  void supportedCommands(std::ostream & os) override
315  {
316  os << "start - start saving video" << std::endl;
317  os << "stop - stop saving video and increment video file number" << std::endl;
318  }
319 
320  protected:
321  void run() // Runs in a thread
322  {
323  while (itsRunning.load())
324  {
325  // Create a VideoWriter here, since it has no close() function, this will ensure it gets destroyed and closes
326  // the movie once we stop the recording:
327  cv::VideoWriter writer;
328  int frame = 0;
329 
330  while (true)
331  {
332  // Get next frame from the buffer:
333  cv::Mat im = itsBuf.pop();
334 
335  // An empty image will be pushed when we are ready to close the video file:
336  if (im.empty()) break;
337 
338  // Start the encoder if it is not yet running:
339  if (writer.isOpened() == false)
340  {
341  // Parse the fourcc, regex in our param definition enforces 4 alphanumeric chars:
342  std::string const fcc = fourcc::get();
343  int const cvfcc = cv::VideoWriter::fourcc(fcc[0], fcc[1], fcc[2], fcc[3]);
344 
345  // Add path prefix if given filename is relative:
346  std::string fn = filename::get();
347  if (fn.empty()) LFATAL("Cannot save to an empty filename");
348  if (fn[0] != '/') fn = PATHPREFIX + fn;
349 
350  // Create directory just in case it does not exist:
351  std::string const cmd = "/bin/mkdir -p " + fn.substr(0, fn.rfind('/'));
352  if (std::system(cmd.c_str())) LERROR("Error running [" << cmd << "] -- IGNORED");
353 
354  // Fill in the file number; be nice and do not overwrite existing files:
355  while (true)
356  {
357  char tmp[2048];
358  std::snprintf(tmp, 2047, fn.c_str(), itsFileNum);
359  std::ifstream ifs(tmp);
360  if (ifs.is_open() == false) { itsFilename = tmp; break; }
361  ++itsFileNum;
362  }
363 
364  // Open the writer:
365  if (writer.open(itsFilename, cvfcc, fps::get(), im.size(), true) == false)
366  LFATAL("Failed to open video encoder for file [" << itsFilename << ']');
367 
368  sendSerial("SAVETO " + itsFilename);
369  }
370 
371  // Write the frame:
372  writer << im;
373 
374  // Report what is going on once in a while:
375  if ((++frame % 100) == 0) sendSerial("SAVEDNUM " + std::to_string(frame));
376  }
377 
378  // Our writer runs out of scope and closes the file here.
379  ++itsFileNum;
380  }
381  }
382 
383  std::future<void> itsRunFut;
385  std::atomic<bool> itsSaving;
387  std::atomic<bool> itsRunning;
388  std::string itsFilename;
389 };
390 
391 // Allow the module to be loaded as a shared object (.so) file:
Save captured camera frames into a video file.
Definition: SaveVideo.C:180
std::string warnAndIgnoreException()
friend friend class Module
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
cv::Mat convertToCvBGR(RawImage const &src)
unsigned int height
void push(T const &val)
virtual ~SaveVideo()
Virtual destructor for safe inheritance.
Definition: SaveVideo.C:224
size_t filled_size() const
void postUninit() override
Get stopped.
Definition: SaveVideo.C:205
void supportedCommands(std::ostream &os) override
Human-readable description of this Module&#39;s supported custom commands.
Definition: SaveVideo.C:314
void run()
Definition: SaveVideo.C:321
std::string itsFilename
Definition: SaveVideo.C:388
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface > s) override
Receive a string from a serial port which contains a user command.
Definition: SaveVideo.C:283
void postInit() override
Get started.
Definition: SaveVideo.C:194
#define LERROR(msg)
void process(jevois::InputFrame &&inframe) override
Processing function, version with no video output.
Definition: SaveVideo.C:264
JEVOIS_REGISTER_MODULE(SaveVideo)
int itsFileNum
Definition: SaveVideo.C:386
virtual void sendSerial(std::string const &str)
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .cfg extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory.", "camera_para", ParamCateg)
Parameter.
#define LFATAL(msg)
std::future< void > itsRunFut
Definition: SaveVideo.C:383
#define PATHPREFIX
Definition: SaveVideo.C:38
std::string to_string(T const &val)
jevois::BoundedBuffer< cv::Mat, jevois::BlockingBehavior::Block, jevois::BlockingBehavior::Block > itsBuf
Definition: SaveVideo.C:384
std::string system(std::string const &cmd)
#define LINFO(msg)
SaveVideo(std::string const &instance)
Constructor.
Definition: SaveVideo.C:187
std::atomic< bool > itsRunning
Definition: SaveVideo.C:387
unsigned int width
void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, version that also shows a debug video output.
Definition: SaveVideo.C:230
std::atomic< bool > itsSaving
Definition: SaveVideo.C:385
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const