JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
SaveVideo.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Log.H>
20 #include <jevois/Debug/Timer.H>
21 #include <jevois/Util/Utils.H>
24 
25 #include <opencv2/core/version.hpp>
26 
27 #include <opencv2/videoio.hpp> // for cv::VideoCapture
28 #include <opencv2/imgproc/imgproc.hpp>
29 
30 #include <future>
31 #include <linux/videodev2.h> // for v4l2 pixel types
32 #include <cstdlib> // for std::system()
33 #include <cstdio> // for snprintf()
34 #include <fstream>
35 
36 // icon by Madebyoliver in multimedia at flaticon
37 
38 static jevois::ParameterCategory const ParamCateg("Video Saving Options");
39 
40 #define PATHPREFIX JEVOIS_ROOT_PATH "/data/savevideo/"
41 
42 //! Parameter \relates SaveVideo
43 JEVOIS_DECLARE_PARAMETER(filename, std::string, "Name of the video file to write. If path is not absolute, "
44  PATHPREFIX " will be prepended to it. Name should contain a printf-like directive for "
45  "one int argument, which will start at 0 and be incremented on each streamoff command.",
46  "video%06d.avi", ParamCateg);
47 
48 //! Parameter \relates SaveVideo
49 JEVOIS_DECLARE_PARAMETER(fourcc, std::string, "FourCC of the codec to use. The OpenCV VideoWriter doc is unclear "
50  "as to which codecs are supported. Presumably, the ffmpeg library is used inside OpenCV. "
51  "Hence any video encoder supported by ffmpeg should work. Tested codecs include: MJPG, "
52  "MP4V, AVC1. Make sure you also pick the right filename extension (e.g., .avi for MJPG, "
53  ".mp4 for MP4V, etc)",
54 #ifdef JEVOIS_PRO
55  "X264",
56 #else
57  "MJPG",
58 #endif
59  boost::regex("^\\w{4}$"), ParamCateg);
60 
61 //! Parameter \relates SaveVideo
62 JEVOIS_DECLARE_PARAMETER(fps, double, "Video frames/sec as stored in the file and to be used both for recording and "
63  "playback. Beware that the video writer will drop frames if you are capturing faster than "
64  "the frame rate specified here. For example, if capturing at 120fps, be sure to set this "
65  "parameter to 120, otherwise by default the saved video will be at 30fps even though capture "
66  "was running at 120fps.",
67  30.0, ParamCateg);
68 
69 //! Save captured camera frames into a video file
70 /*! This module records video and saves it to the MicroSD card inside the JeVois smart camera. It is useful, for
71  example, to record footage that will be used to train some machine vision algorithm.
72 
73  Issue the command \c start over the command-line interface to start saving video frames, and \c stop to stop
74  saving. Successive start/stop commands will increment the file number (%d argument in the \p filename
75  parameter. Before a file is written, the module checks whether it already exists, and, if so, skips over it by
76  incrementing the file number. See \ref UserCli for details on how to use the command-line interface.
77 
78  This module works with any video resolution and pixel format supported by the camera sensor. Thus, additional video
79  mappings are possible beyond the ones listed here.
80 
81  See \ref PixelFormats for information about pixel formats; with this module you can use the formats supported by the
82  camera sensor.
83 
84  This module accepts any resolution supported by the JeVois camera sensor.
85 
86  This module can operate both with USB/GUI video output, or no USB video output.
87 
88  When using with no USB/GUI output (NONE output format), you should first issue a \c streamon command to start video
89  streaming by the camera sensor chip, then issue a \c start when you are ready to start recording. The \c streamon is
90  not necessary when using a video mapping with USB video output, as the host computer over USB triggers video
91  streaming when it starts grabbing frames from the JeVois camera.
92 
93  This module internally uses the OpenCV VideoWriter class to compress and write the video file. See the OpenCV
94  documentation for which video formats are supported.
95 
96  You should be aware of two things when attempting video recording at high frame rates:
97 
98  - If using a video mapping with USB output, the frame rate may be limited by the maximum rate at which the host
99  computer can grab and display. On many host computers, for example, the display refresh rate might be 60Hz. Using
100  a video capture software on these computers is likely going to limit the maximum display rate to 60 frames/s and
101  that will in turn limit the capture and saving rate. This is not an issue when using a video mapping with no USB
102  output.
103 
104  - The \p fps parameter should be set to the rate at which you want to save video. If you capture at 60 frames/s
105  according to your video mapping but \p fps is set to 30, saving will be at 30 fps. This limitation of rate is done
106  internally by the OpenCV VideoWriter. So just make sure that you set the \p fps parameter to the rate at which you
107  want to save.
108 
109  Note that this module may suffer from DMA coherency artifacts if the \c camturbo parameter of the jevois::Engine is
110  turned on, which it is by default. The \c camturbo parameter relaxes some of the cache coherency constraints on the
111  video buffers captured by the camera sensor, which allows the JeVois processor to access video pixel data from
112  memory faster. But with modules that do not do much processing, sometimes this yields video artifacts, we presume
113  because some of the video data from previous frames still is in the CPU cache and hence is not again fetched from
114  main memory by the CPU. If you see short stripes of what appears to be wrong pixel colors in the video, try to
115  disable \c camturbo: edit JEVOIS:/config/params.cfg on your MicroSD card and in there turn \c camturbo to false.
116 
117 
118  Example use
119  -----------
120 
121  Connect to the JeVois command-line interface (see \ref UserCli), and issue
122 
123  \verbatim
124  setmapping2 YUYV 640 480 30.0 JeVois SaveVideo
125  streamon
126  start
127  \endverbatim
128 
129  and you will be saving compressed video with 640x480 at 30fps to your microSD card inside JeVois. When you are done,
130  just issue:
131 
132  \verbatim
133  stop
134  \endverbatim
135 
136  and the video file will be finalized and closed. If you want to access it directly by exporting the microSD inside
137  JeVois as a virtual USB flash drive, issue (with \jvversion{1.3} and later):
138 
139  \verbatim
140  streamoff
141  \endverbatim
142 
143  to stop camera sensor streaming, and
144 
145  \verbatim
146  usbsd
147  \endverbatim
148 
149  to export the microSD as a virtual USB flash drive. The video file(s) will be in the <b>JEVOIS:/data/savevideo/</b>
150  folder. You can then use the recorded video to test and debug a new algorithm using your host computer, by running
151  \c jevois-daemon on the host computer with a video file input as opposed to a live camera input, as follows:
152 
153  \verbatim
154  jevois-daemon --cameradev=myvideo.avi --videomapping=num
155  \endverbatim
156 
157  where you replace \a num above by the video mapping you want to use. See \ref JeVoisDaemon for more info about the
158  \p cameradev parameter.
159 
160  @author Laurent Itti
161 
162  @videomapping YUYV 320 240 60.0 YUYV 320 240 60.0 JeVois SaveVideo
163  @videomapping YUYV 320 240 30.0 YUYV 320 240 30.0 JeVois SaveVideo
164  @videomapping NONE 0 0 0 YUYV 320 240 60.0 JeVois SaveVideo
165  @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois SaveVideo
166  @videomapping NONE 0 0 0 YUYV 176 144 120.0 JeVois SaveVideo
167  @modulecommand start - start saving video
168  @modulecommand stop - stop saving video and increment video file number
169  @email itti\@usc.edu
170  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
171  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
172  @mainurl http://jevois.org
173  @supporturl http://jevois.org/doc
174  @otherurl http://iLab.usc.edu
175  @license GPL v3
176  @distribution Unrestricted
177  @restrictions None
178  \ingroup modules */
179 class SaveVideo : public jevois::Module,
180  public jevois::Parameter<filename, fourcc, fps>
181 {
182  public:
183  // ####################################################################################################
184  //! Constructor
185  // ####################################################################################################
186  SaveVideo(std::string const & instance) : jevois::Module(instance), itsBuf(1000), itsSaving(false),
187  itsFileNum(0), itsRunning(false)
188  { }
189 
190  // ####################################################################################################
191  //! Get started
192  // ####################################################################################################
193  void postInit() override
194  {
195  itsRunning.store(true);
196 
197  // Get our run() thread going, it is in charge of compresing and saving frames:
198  itsRunFut = jevois::async(std::bind(&SaveVideo::run, this));
199  }
200 
201  // ####################################################################################################
202  //! Get stopped
203  // ####################################################################################################
204  void postUninit() override
205  {
206  // Signal end of run:
207  itsRunning.store(false);
208 
209  // Push an empty frame into our buffer to signal the end of video to our thread:
210  itsBuf.push(cv::Mat());
211 
212  // Wait for the thread to complete:
213  LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
214  if (itsRunFut.valid()) try { itsRunFut.get(); } catch (...) { jevois::warnAndIgnoreException(); }
215  LINFO("Writer thread completed. Syncing disk...");
216  if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
217  LINFO("Video " << itsFilename << " saved.");
218  }
219 
220  // ####################################################################################################
221  //! Virtual destructor for safe inheritance
222  // ####################################################################################################
223  virtual ~SaveVideo()
224  { }
225 
226  // ####################################################################################################
227  //! Processing function, version that also shows a debug video output
228  // ####################################################################################################
229  void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
230  {
231  // Wait for next available camera image:
232  jevois::RawImage inimg = inframe.get(true); unsigned int const w = inimg.width, h = inimg.height;
233  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
234 
235  if (itsSaving.load())
236  {
237  // Convert image to BGR and push to our writer thread:
238  if (itsBuf.filled_size() > 1000) LERROR("Image queue too large, video writer cannot keep up - DROPPING FRAME");
240  }
241 
242  // Copy the input image to output:
243  jevois::RawImage outimg = outframe.get();
244  outimg.require("output", w, h, V4L2_PIX_FMT_YUYV);
245 
246  jevois::rawimage::paste(inimg, outimg, 0, 0);
247 
248  // Let camera know we are done processing the raw YUV input image:
249  inframe.done();
250 
251  // Show some text messages:
252  std::string txt = "SaveVideo: "; if (itsSaving.load()) txt += "RECORDING"; else txt += "not recording";
253  jevois::rawimage::writeText(outimg, txt.c_str(), 3, 3, jevois::yuyv::White);
254  jevois::rawimage::writeText(outimg, itsFilename.c_str(), 3, h - 13, jevois::yuyv::White);
255 
256  // Send output frame over USB:
257  outframe.send();
258  }
259 
260  // ####################################################################################################
261  //! Processing function, version with no video output
262  // ####################################################################################################
263  void process(jevois::InputFrame && inframe) override
264  {
265  // Wait for next available camera image:
266  jevois::RawImage inimg = inframe.get(true);
267 
268  if (itsSaving.load())
269  {
270  // Convert image to BGR and push to our writer thread:
271  if (itsBuf.filled_size() > 1000) LERROR("Image queue too large, video writer cannot keep up - DROPPING FRAME");
273  }
274 
275  // Let camera know we are done processing the raw YUV input image:
276  inframe.done();
277  }
278 
279 #ifdef JEVOIS_PRO
280  // ####################################################################################################
281  //! Processing function with zero-copy and GUI on JeVois-Pro
282  // ####################################################################################################
283  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
284  {
285  static jevois::Timer timer("processing", 100, LOG_DEBUG);
286 
287  // Start the GUI frame:
288  unsigned short winw, winh;
289  bool idle = helper.startFrame(winw, winh);
290 
291  // Draw the camera frame:
292  int x = 0, y = 0; unsigned short iw = 0, ih = 0;
293  helper.drawInputFrame("camera", inframe, x, y, iw, ih);
294 
295  // Wait for next available camera image:
296  jevois::RawImage const inimg = inframe.getp();
297  unsigned int const w = inimg.width, h = inimg.height;
298  helper.itext(std::string("JeVois-Pro Save Video: ") + (itsSaving.load() ? "RECORDING" : "not recording"));
299  if (itsFilename.empty() == false) helper.itext("Saving to " + itsFilename);
300  size_t const n = itsBuf.filled_size();
301  if (n) helper.itext(std::to_string(n) + " queued frames waiting to save...");
302 
303  // Draw a start/stop button:
304  if (ImGui::Begin("SaveVideo Control"))
305  {
306  ImGui::Text("Click Record to start saving,");
307  ImGui::Text("click it again to stop.");
308  ImGui::Separator();
309  bool save = itsSaving.load(); bool wassaving = save; helper.toggleButton("Record", &save);
310  if (wassaving && !save) itsBuf.push(cv::Mat()); // Let writer thread know to close this file
311  itsSaving.store(save);
312  ImGui::Separator();
313  if (itsSaving.load()) ImGui::Text("Recording...");
314  else if (n) ImGui::Text("Saving queued frames...");
315  else ImGui::Text("Ready to record");
316  }
317  ImGui::End();
318 
319  timer.start();
320 
321  if (itsSaving.load())
322  {
323  // Convert image to BGR and push to our writer thread:
324  if (itsBuf.filled_size() > 1000)
325  helper.reportError("Image queue too large, video writer cannot keep up - DROPPING FRAME");
326  else
328  }
329 
330  // Let camera know we are done processing the input image:
331  inframe.done();
332 
333  // Show processing fps:
334  std::string const & fpscpu = timer.stop();
335  helper.iinfo(inframe, fpscpu, winw, winh);
336 
337  // Render the image and GUI:
338  helper.endFrame();
339  }
340 #endif
341 
342  // ####################################################################################################
343  //! Receive a string from a serial port which contains a user command
344  // ####################################################################################################
345  void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> s) override
346  {
347  if (str == "start")
348  {
349  itsSaving.store(true);
350  sendSerial("SAVESTART");
351  }
352  else if (str == "stop")
353  {
354  itsSaving.store(false);
355  sendSerial("SAVESTOP");
356 
357  // Push an empty frame into our buffer to signal the end of video to our thread:
358  itsBuf.push(cv::Mat());
359 
360 #ifndef JEVOIS_PRO
361  // Wait for the thread to empty our image buffer:
362  while (itsBuf.filled_size())
363  {
364  LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
365  std::this_thread::sleep_for(std::chrono::milliseconds(200));
366  }
367  LINFO("Writer thread completed. Syncing disk...");
368  if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
369  LINFO("Video saved.");
370 #endif
371  }
372  else throw std::runtime_error("Unsupported module command");
373  }
374 
375  // ####################################################################################################
376  //! Human-readable description of this Module's supported custom commands
377  // ####################################################################################################
378  void supportedCommands(std::ostream & os) override
379  {
380  os << "start - start saving video" << std::endl;
381  os << "stop - stop saving video and increment video file number" << std::endl;
382  }
383 
384  protected:
385  void run() // Runs in a thread
386  {
387  while (itsRunning.load())
388  {
389  // Create a VideoWriter here, since it has no close() function, this will ensure it gets destroyed and closes
390  // the movie once we stop the recording:
391  cv::VideoWriter writer;
392  int frame = 0;
393 
394  while (true)
395  {
396  // Get next frame from the buffer:
397  cv::Mat im = itsBuf.pop();
398 
399  // An empty image will be pushed when we are ready to close the video file:
400  if (im.empty()) break;
401 
402  // Start the encoder if it is not yet running:
403  if (writer.isOpened() == false)
404  {
405  // Parse the fourcc, regex in our param definition enforces 4 alphanumeric chars:
406  std::string const fcc = fourcc::get();
407  int const cvfcc = cv::VideoWriter::fourcc(fcc[0], fcc[1], fcc[2], fcc[3]);
408 
409  // Add path prefix if given filename is relative:
410  std::string fn = filename::get();
411  if (fn.empty()) LFATAL("Cannot save to an empty filename");
412  if (fn[0] != '/') fn = PATHPREFIX + fn;
413 
414  // Create directory just in case it does not exist:
415  std::string const cmd = "/bin/mkdir -p " + fn.substr(0, fn.rfind('/'));
416  if (std::system(cmd.c_str())) LERROR("Error running [" << cmd << "] -- IGNORED");
417 
418  // Fill in the file number; be nice and do not overwrite existing files:
419  while (true)
420  {
421  std::string tmp = jevois::sformat(fn.c_str(), itsFileNum);
422  std::ifstream ifs(tmp);
423  if (ifs.is_open() == false) { itsFilename = tmp; break; }
424  ++itsFileNum;
425  }
426 
427  // Open the writer:
428  if (writer.open(itsFilename, cvfcc, fps::get(), im.size(), true) == false)
429  LFATAL("Failed to open video encoder for file [" << itsFilename << ']');
430 
431  sendSerial("SAVETO " + itsFilename);
432  }
433 
434  // Write the frame:
435  writer << im;
436 
437  // Report what is going on once in a while:
438  if ((++frame % 100) == 0) sendSerial("SAVEDNUM " + std::to_string(frame));
439  }
440 
441  // Our writer runs out of scope and closes the file here.
442  LINFO("Closing " + itsFilename);
443  ++itsFileNum;
444  itsFilename.clear();
445  }
446  }
447 
448  std::future<void> itsRunFut;
450  std::atomic<bool> itsSaving;
452  std::atomic<bool> itsRunning;
453  std::string itsFilename;
454 };
455 
456 // Allow the module to be loaded as a shared object (.so) file:
jevois::OutputFrame
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
SaveVideo::process
void process(jevois::InputFrame &&inframe) override
Processing function, version with no video output.
Definition: SaveVideo.C:263
Timer.H
jevois::GUIhelper::itext
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
SaveVideo::parseSerial
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface > s) override
Receive a string from a serial port which contains a user command.
Definition: SaveVideo.C:345
Module.H
jevois::sformat
std::string sformat(char const *fmt,...) __attribute__((format(__printf__
jevois::GUIhelper
SaveVideo::supportedCommands
void supportedCommands(std::ostream &os) override
Human-readable description of this Module's supported custom commands.
Definition: SaveVideo.C:378
jevois::GUIhelper::endFrame
void endFrame()
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(SaveVideo)
Log.H
jevois::BoundedBuffer< cv::Mat, jevois::BlockingBehavior::Block, jevois::BlockingBehavior::Block >
jevois::RawImage
jevois::rawimage::convertToCvBGR
cv::Mat convertToCvBGR(RawImage const &src)
jevois::ParameterCategory
SaveVideo::run
void run()
Definition: SaveVideo.C:385
SaveVideo::~SaveVideo
virtual ~SaveVideo()
Virtual destructor for safe inheritance.
Definition: SaveVideo.C:223
LERROR
#define LERROR(msg)
SaveVideo::SaveVideo
SaveVideo(std::string const &instance)
Constructor.
Definition: SaveVideo.C:186
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
SaveVideo::itsRunFut
std::future< void > itsRunFut
Definition: SaveVideo.C:448
jevois::RawImage::width
unsigned int width
SaveVideo::itsFileNum
int itsFileNum
Definition: SaveVideo.C:451
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
jevois::GUIhelper::reportError
void reportError(std::string const &err)
jevois
system
std::string system(std::string const &cmd, bool errtoo=true)
jevois::BoundedBuffer::pop
T pop()
jevois::BoundedBuffer::push
void push(T const &val)
jevois::GUIhelper::toggleButton
bool toggleButton(char const *name, bool *val)
SaveVideo::itsSaving
std::atomic< bool > itsSaving
Definition: SaveVideo.C:450
PATHPREFIX
#define PATHPREFIX
Definition: SaveVideo.C:40
SaveVideo::itsFilename
std::string itsFilename
Definition: SaveVideo.C:453
jevois::warnAndIgnoreException
std::string warnAndIgnoreException(std::string const &prefix="")
jevois::BoundedBuffer::filled_size
size_t filled_size() const
LFATAL
#define LFATAL(msg)
SaveVideo::process
void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, version that also shows a debug video output.
Definition: SaveVideo.C:229
jevois::Module
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
to_string
std::string to_string(T const &val)
jevois::InputFrame
test-model.frame
frame
Definition: test-model.py:22
Utils.H
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
h
int h
jevois::Module::sendSerial
virtual void sendSerial(std::string const &str)
BoundedBuffer.H
SaveVideo::itsRunning
std::atomic< bool > itsRunning
Definition: SaveVideo.C:452
ARtoolkit::JEVOIS_DECLARE_PARAMETER
JEVOIS_DECLARE_PARAMETER(camparams, std::string, "File stem of camera parameters, or empty. Camera resolution " "will be appended, as well as a .dat extension. For example, specifying 'camera_para' " "here and running the camera sensor at 320x240 will attempt to load " "camera_para320x240.dat from within the module's directory (if relative stem) or " "from the specified absolute location (if absolute stem).", JEVOIS_SHARE_PATH "/camera/camera_para", ParamCateg)
Parameter.
SaveVideo::postUninit
void postUninit() override
Get stopped.
Definition: SaveVideo.C:204
LINFO
#define LINFO(msg)
SaveVideo::itsBuf
jevois::BoundedBuffer< cv::Mat, jevois::BlockingBehavior::Block, jevois::BlockingBehavior::Block > itsBuf
Definition: SaveVideo.C:449
SaveVideo::postInit
void postInit() override
Get started.
Definition: SaveVideo.C:193
SaveVideo
Save captured camera frames into a video file.
Definition: SaveVideo.C:179
jevois::Timer
jevois::Component::Module
friend friend class Module