JeVoisBase  1.22
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
SaveVideo.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17
18#include <jevois/Core/Module.H>
19#include <jevois/Debug/Log.H>
20#include <jevois/Debug/Timer.H>
21#include <jevois/Util/Utils.H>
24
25#include <opencv2/core/version.hpp>
26
27#include <opencv2/videoio.hpp> // for cv::VideoCapture
28#include <opencv2/imgproc/imgproc.hpp>
29
30#include <future>
31#include <linux/videodev2.h> // for v4l2 pixel types
32#include <cstdlib> // for std::system()
33#include <cstdio> // for snprintf()
34#include <fstream>
35
36// icon by Madebyoliver in multimedia at flaticon
37
38static jevois::ParameterCategory const ParamCateg("Video Saving Options");
39
40#define PATHPREFIX JEVOIS_ROOT_PATH "/data/savevideo/"
41
42//! Parameter \relates SaveVideo
43JEVOIS_DECLARE_PARAMETER(filename, std::string, "Name of the video file to write. If path is not absolute, "
44 PATHPREFIX " will be prepended to it. Name should contain a printf-like directive for "
45 "one int argument, which will start at 0 and be incremented on each streamoff command.",
46 "video%06d.avi", ParamCateg);
47
48//! Parameter \relates SaveVideo
49JEVOIS_DECLARE_PARAMETER(fourcc, std::string, "FourCC of the codec to use. The OpenCV VideoWriter doc is unclear "
50 "as to which codecs are supported. Presumably, the ffmpeg library is used inside OpenCV. "
51 "Hence any video encoder supported by ffmpeg should work. Tested codecs include: MJPG, "
52 "MP4V, AVC1. Make sure you also pick the right filename extension (e.g., .avi for MJPG, "
53 ".mp4 for MP4V, etc)",
54#ifdef JEVOIS_PRO
55 "X264",
56#else
57 "MJPG",
58#endif
59 boost::regex("^\\w{4}$"), ParamCateg);
60
61//! Parameter \relates SaveVideo
62JEVOIS_DECLARE_PARAMETER(fps, double, "Video frames/sec as stored in the file and to be used both for recording and "
63 "playback. Beware that the video writer will drop frames if you are capturing faster than "
64 "the frame rate specified here. For example, if capturing at 120fps, be sure to set this "
65 "parameter to 120, otherwise by default the saved video will be at 30fps even though capture "
66 "was running at 120fps.",
67 30.0, ParamCateg);
68
69//! Save captured camera frames into a video file
70/*! This module records video and saves it to the MicroSD card inside the JeVois smart camera. It is useful, for
71 example, to record footage that will be used to train some machine vision algorithm.
72
73 Issue the command \c start over the command-line interface to start saving video frames, and \c stop to stop
74 saving. Successive start/stop commands will increment the file number (%d argument in the \p filename
75 parameter. Before a file is written, the module checks whether it already exists, and, if so, skips over it by
76 incrementing the file number. See \ref UserCli for details on how to use the command-line interface.
77
78 This module works with any video resolution and pixel format supported by the camera sensor. Thus, additional video
79 mappings are possible beyond the ones listed here.
80
81 See \ref PixelFormats for information about pixel formats; with this module you can use the formats supported by the
82 camera sensor.
83
84 This module accepts any resolution supported by the JeVois camera sensor.
85
86 This module can operate both with USB/GUI video output, or no USB video output.
87
88 When using with no USB/GUI output (NONE output format), you should first issue a \c streamon command to start video
89 streaming by the camera sensor chip, then issue a \c start when you are ready to start recording. The \c streamon is
90 not necessary when using a video mapping with USB video output, as the host computer over USB triggers video
91 streaming when it starts grabbing frames from the JeVois camera.
92
93 This module internally uses the OpenCV VideoWriter class to compress and write the video file. See the OpenCV
94 documentation for which video formats are supported.
95
96 You should be aware of two things when attempting video recording at high frame rates:
97
98 - If using a video mapping with USB output, the frame rate may be limited by the maximum rate at which the host
99 computer can grab and display. On many host computers, for example, the display refresh rate might be 60Hz. Using
100 a video capture software on these computers is likely going to limit the maximum display rate to 60 frames/s and
101 that will in turn limit the capture and saving rate. This is not an issue when using a video mapping with no USB
102 output.
103
104 - The \p fps parameter should be set to the rate at which you want to save video. If you capture at 60 frames/s
105 according to your video mapping but \p fps is set to 30, saving will be at 30 fps. This limitation of rate is done
106 internally by the OpenCV VideoWriter. So just make sure that you set the \p fps parameter to the rate at which you
107 want to save.
108
109 Note that this module may suffer from DMA coherency artifacts if the \c camturbo parameter of the jevois::Engine is
110 turned on, which it is by default. The \c camturbo parameter relaxes some of the cache coherency constraints on the
111 video buffers captured by the camera sensor, which allows the JeVois processor to access video pixel data from
112 memory faster. But with modules that do not do much processing, sometimes this yields video artifacts, we presume
113 because some of the video data from previous frames still is in the CPU cache and hence is not again fetched from
114 main memory by the CPU. If you see short stripes of what appears to be wrong pixel colors in the video, try to
115 disable \c camturbo: edit JEVOIS:/config/params.cfg on your MicroSD card and in there turn \c camturbo to false.
116
117
118 Example use
119 -----------
120
121 Connect to the JeVois command-line interface (see \ref UserCli), and issue
122
123 \verbatim
124 setmapping2 YUYV 640 480 30.0 JeVois SaveVideo
125 streamon
126 start
127 \endverbatim
128
129 and you will be saving compressed video with 640x480 at 30fps to your microSD card inside JeVois. When you are done,
130 just issue:
131
132 \verbatim
133 stop
134 \endverbatim
135
136 and the video file will be finalized and closed. If you want to access it directly by exporting the microSD inside
137 JeVois as a virtual USB flash drive, issue (with \jvversion{1.3} and later):
138
139 \verbatim
140 streamoff
141 \endverbatim
142
143 to stop camera sensor streaming, and
144
145 \verbatim
146 usbsd
147 \endverbatim
148
149 to export the microSD as a virtual USB flash drive. The video file(s) will be in the <b>JEVOIS:/data/savevideo/</b>
150 folder. You can then use the recorded video to test and debug a new algorithm using your host computer, by running
151 \c jevois-daemon on the host computer with a video file input as opposed to a live camera input, as follows:
152
153 \verbatim
154 jevois-daemon --cameradev=myvideo.avi --videomapping=num
155 \endverbatim
156
157 where you replace \a num above by the video mapping you want to use. See \ref JeVoisDaemon for more info about the
158 \p cameradev parameter.
159
160 @author Laurent Itti
161
162 @videomapping YUYV 320 240 60.0 YUYV 320 240 60.0 JeVois SaveVideo
163 @videomapping YUYV 320 240 30.0 YUYV 320 240 30.0 JeVois SaveVideo
164 @videomapping NONE 0 0 0 YUYV 320 240 60.0 JeVois SaveVideo
165 @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois SaveVideo
166 @videomapping NONE 0 0 0 YUYV 176 144 120.0 JeVois SaveVideo
167 @modulecommand start - start saving video
168 @modulecommand stop - stop saving video and increment video file number
169 @email itti\@usc.edu
170 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
171 @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
172 @mainurl http://jevois.org
173 @supporturl http://jevois.org/doc
174 @otherurl http://iLab.usc.edu
175 @license GPL v3
176 @distribution Unrestricted
177 @restrictions None
178 \ingroup modules */
180 public jevois::Parameter<filename, fourcc, fps>
181{
182 public:
183 // ####################################################################################################
184 //! Constructor
185 // ####################################################################################################
186 SaveVideo(std::string const & instance) : jevois::Module(instance), itsBuf(1000), itsSaving(false),
187 itsFileNum(0), itsRunning(false)
188 { }
189
190 // ####################################################################################################
191 //! Get started
192 // ####################################################################################################
193 void postInit() override
194 {
195 itsRunning.store(true);
196
197 // Get our run() thread going, it is in charge of compresing and saving frames:
198 itsRunFut = jevois::async(std::bind(&SaveVideo::run, this));
199 }
200
201 // ####################################################################################################
202 //! Get stopped
203 // ####################################################################################################
204 void postUninit() override
205 {
206 // Signal end of run:
207 itsRunning.store(false);
208
209 // Push an empty frame into our buffer to signal the end of video to our thread:
210 itsBuf.push(cv::Mat());
211
212 // Wait for the thread to complete:
213 LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
214 if (itsRunFut.valid()) try { itsRunFut.get(); } catch (...) { jevois::warnAndIgnoreException(); }
215 LINFO("Writer thread completed. Syncing disk...");
216 if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
217 LINFO("Video " << itsFilename << " saved.");
218 }
219
220 // ####################################################################################################
221 //! Virtual destructor for safe inheritance
222 // ####################################################################################################
223 virtual ~SaveVideo()
224 { }
225
226 // ####################################################################################################
227 //! Processing function, version that also shows a debug video output
228 // ####################################################################################################
229 void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
230 {
231 // Wait for next available camera image:
232 jevois::RawImage inimg = inframe.get(true); unsigned int const w = inimg.width, h = inimg.height;
233 inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
234
235 if (itsSaving.load())
236 {
237 // Convert image to BGR and push to our writer thread:
238 if (itsBuf.filled_size() > 1000) LERROR("Image queue too large, video writer cannot keep up - DROPPING FRAME");
240 }
241
242 // Copy the input image to output:
243 jevois::RawImage outimg = outframe.get();
244 outimg.require("output", w, h, V4L2_PIX_FMT_YUYV);
245
246 jevois::rawimage::paste(inimg, outimg, 0, 0);
247
248 // Let camera know we are done processing the raw YUV input image:
249 inframe.done();
250
251 // Show some text messages:
252 std::string txt = "SaveVideo: "; if (itsSaving.load()) txt += "RECORDING"; else txt += "not recording";
253 jevois::rawimage::writeText(outimg, txt.c_str(), 3, 3, jevois::yuyv::White);
255
256 // Send output frame over USB:
257 outframe.send();
258 }
259
260 // ####################################################################################################
261 //! Processing function, version with no video output
262 // ####################################################################################################
263 void process(jevois::InputFrame && inframe) override
264 {
265 // Wait for next available camera image:
266 jevois::RawImage inimg = inframe.get(true);
267
268 if (itsSaving.load())
269 {
270 // Convert image to BGR and push to our writer thread:
271 if (itsBuf.filled_size() > 1000) LERROR("Image queue too large, video writer cannot keep up - DROPPING FRAME");
273 }
274
275 // Let camera know we are done processing the raw YUV input image:
276 inframe.done();
277 }
278
279#ifdef JEVOIS_PRO
280 // ####################################################################################################
281 //! Processing function with zero-copy and GUI on JeVois-Pro
282 // ####################################################################################################
283 virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
284 {
285 static jevois::Timer timer("processing", 100, LOG_DEBUG);
286
287 // Start the GUI frame:
288 unsigned short winw, winh;
289 helper.startFrame(winw, winh);
290
291 // Draw the camera frame:
292 int x = 0, y = 0; unsigned short iw = 0, ih = 0;
293 helper.drawInputFrame("camera", inframe, x, y, iw, ih);
294
295 // Wait for next available camera image:
296 jevois::RawImage const inimg = inframe.getp();
297 helper.itext(std::string("JeVois-Pro Save Video: ") + (itsSaving.load() ? "RECORDING" : "not recording"));
298 if (itsFilename.empty() == false) helper.itext("Saving to " + itsFilename);
299 size_t const n = itsBuf.filled_size();
300 if (n) helper.itext(std::to_string(n) + " queued frames waiting to save...");
301
302 // Draw a start/stop button:
303 if (ImGui::Begin("SaveVideo Control"))
304 {
305 ImGui::Text("Click Record to start saving,");
306 ImGui::Text("click it again to stop.");
307 ImGui::Separator();
308 bool save = itsSaving.load(); bool wassaving = save; helper.toggleButton("Record", &save);
309 if (wassaving && !save) itsBuf.push(cv::Mat()); // Let writer thread know to close this file
310 itsSaving.store(save);
311 ImGui::Separator();
312 if (itsSaving.load()) ImGui::Text("Recording...");
313 else if (n) ImGui::Text("Saving queued frames...");
314 else ImGui::Text("Ready to record");
315 }
316 ImGui::End();
317
318 timer.start();
319
320 if (itsSaving.load())
321 {
322 // Convert image to BGR and push to our writer thread:
323 if (itsBuf.filled_size() > 1000)
324 helper.reportError("Image queue too large, video writer cannot keep up - DROPPING FRAME");
325 else
327 }
328
329 // Let camera know we are done processing the input image:
330 inframe.done();
331
332 // Show processing fps:
333 std::string const & fpscpu = timer.stop();
334 helper.iinfo(inframe, fpscpu, winw, winh);
335
336 // Render the image and GUI:
337 helper.endFrame();
338 }
339#endif
340
341 // ####################################################################################################
342 //! Receive a string from a serial port which contains a user command
343 // ####################################################################################################
344 void parseSerial(std::string const & str, std::shared_ptr<jevois::UserInterface> /*s*/) override
345 {
346 if (str == "start")
347 {
348 itsSaving.store(true);
349 sendSerial("SAVESTART");
350 }
351 else if (str == "stop")
352 {
353 itsSaving.store(false);
354 sendSerial("SAVESTOP");
355
356 // Push an empty frame into our buffer to signal the end of video to our thread:
357 itsBuf.push(cv::Mat());
358
359#ifndef JEVOIS_PRO
360 // Wait for the thread to empty our image buffer:
361 while (itsBuf.filled_size())
362 {
363 LINFO("Waiting for writer thread to complete, " << itsBuf.filled_size() << " frames to go...");
364 std::this_thread::sleep_for(std::chrono::milliseconds(200));
365 }
366 LINFO("Writer thread completed. Syncing disk...");
367 if (std::system("/bin/sync")) LERROR("Error syncing disk -- IGNORED");
368 LINFO("Video saved.");
369#endif
370 }
371 else throw std::runtime_error("Unsupported module command");
372 }
373
374 // ####################################################################################################
375 //! Human-readable description of this Module's supported custom commands
376 // ####################################################################################################
377 void supportedCommands(std::ostream & os) override
378 {
379 os << "start - start saving video" << std::endl;
380 os << "stop - stop saving video and increment video file number" << std::endl;
381 }
382
383 protected:
384 void run() // Runs in a thread
385 {
386 while (itsRunning.load())
387 {
388 // Create a VideoWriter here, since it has no close() function, this will ensure it gets destroyed and closes
389 // the movie once we stop the recording:
390 cv::VideoWriter writer;
391 int frame = 0;
392
393 while (true)
394 {
395 // Get next frame from the buffer:
396 cv::Mat im = itsBuf.pop();
397
398 // An empty image will be pushed when we are ready to close the video file:
399 if (im.empty()) break;
400
401 // Start the encoder if it is not yet running:
402 if (writer.isOpened() == false)
403 {
404 // Parse the fourcc, regex in our param definition enforces 4 alphanumeric chars:
405 std::string const fcc = fourcc::get();
406 int const cvfcc = cv::VideoWriter::fourcc(fcc[0], fcc[1], fcc[2], fcc[3]);
407
408 // Add path prefix if given filename is relative:
409 std::string fn = filename::get();
410 if (fn.empty()) LFATAL("Cannot save to an empty filename");
411 if (fn[0] != '/') fn = PATHPREFIX + fn;
412
413 // Create directory just in case it does not exist:
414 std::string const cmd = "/bin/mkdir -p " + fn.substr(0, fn.rfind('/'));
415 if (std::system(cmd.c_str())) LERROR("Error running [" << cmd << "] -- IGNORED");
416
417 // Fill in the file number; be nice and do not overwrite existing files:
418 while (true)
419 {
420 std::string tmp = jevois::sformat(fn.c_str(), itsFileNum);
421 std::ifstream ifs(tmp);
422 if (ifs.is_open() == false) { itsFilename = tmp; break; }
423 ++itsFileNum;
424 }
425
426 // Open the writer:
427 if (writer.open(itsFilename, cvfcc, fps::get(), im.size(), true) == false)
428 LFATAL("Failed to open video encoder for file [" << itsFilename << ']');
429
430 sendSerial("SAVETO " + itsFilename);
431 }
432
433 // Write the frame:
434 writer << im;
435
436 // Report what is going on once in a while:
437 if ((++frame % 100) == 0) sendSerial("SAVEDNUM " + std::to_string(frame));
438 }
439
440 // Our writer runs out of scope and closes the file here.
441 LINFO("Closing " + itsFilename);
442 ++itsFileNum;
443 itsFilename.clear();
444 }
445 }
446
447 std::future<void> itsRunFut;
449 std::atomic<bool> itsSaving;
451 std::atomic<bool> itsRunning;
452 std::string itsFilename;
453};
454
455// Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE(ArUcoBlob)
int h
#define PATHPREFIX
Definition SaveVideo.C:40
Save captured camera frames into a video file.
Definition SaveVideo.C:181
virtual void process(jevois::InputFrame &&inframe, jevois::GUIhelper &helper) override
Processing function with zero-copy and GUI on JeVois-Pro.
Definition SaveVideo.C:283
void run()
Definition SaveVideo.C:384
JEVOIS_DECLARE_PARAMETER(filename, std::string, "Name of the video file to write. If path is not absolute, " PATHPREFIX " will be prepended to it. Name should contain a printf-like directive for " "one int argument, which will start at 0 and be incremented on each streamoff command.", "video%06d.avi", ParamCateg)
Parameter.
void process(jevois::InputFrame &&inframe) override
Processing function, version with no video output.
Definition SaveVideo.C:263
std::string itsFilename
Definition SaveVideo.C:452
std::atomic< bool > itsSaving
Definition SaveVideo.C:449
SaveVideo(std::string const &instance)
Constructor.
Definition SaveVideo.C:186
void parseSerial(std::string const &str, std::shared_ptr< jevois::UserInterface >) override
Receive a string from a serial port which contains a user command.
Definition SaveVideo.C:344
void supportedCommands(std::ostream &os) override
Human-readable description of this Module's supported custom commands.
Definition SaveVideo.C:377
void postInit() override
Get started.
Definition SaveVideo.C:193
jevois::BoundedBuffer< cv::Mat, jevois::BlockingBehavior::Block, jevois::BlockingBehavior::Block > itsBuf
Definition SaveVideo.C:448
void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, version that also shows a debug video output.
Definition SaveVideo.C:229
JEVOIS_DECLARE_PARAMETER(fourcc, std::string, "FourCC of the codec to use. The OpenCV VideoWriter doc is unclear " "as to which codecs are supported. Presumably, the ffmpeg library is used inside OpenCV. " "Hence any video encoder supported by ffmpeg should work. Tested codecs include: MJPG, " "MP4V, AVC1. Make sure you also pick the right filename extension (e.g., .avi for MJPG, " ".mp4 for MP4V, etc)", "X264", boost::regex("^\\w{4}$"), ParamCateg)
Parameter.
std::future< void > itsRunFut
Definition SaveVideo.C:447
virtual ~SaveVideo()
Virtual destructor for safe inheritance.
Definition SaveVideo.C:223
int itsFileNum
Definition SaveVideo.C:450
std::atomic< bool > itsRunning
Definition SaveVideo.C:451
JEVOIS_DECLARE_PARAMETER(fps, double, "Video frames/sec as stored in the file and to be used both for recording and " "playback. Beware that the video writer will drop frames if you are capturing faster than " "the frame rate specified here. For example, if capturing at 120fps, be sure to set this " "parameter to 120, otherwise by default the saved video will be at 30fps even though capture " "was running at 120fps.", 30.0, ParamCateg)
Parameter.
void postUninit() override
Get stopped.
Definition SaveVideo.C:204
size_t filled_size() const
void push(T const &val)
friend friend class Module
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
bool startFrame(unsigned short &w, unsigned short &h)
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
void reportError(std::string const &err)
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
bool toggleButton(char const *name, bool *val)
virtual void sendSerial(std::string const &str)
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
std::string const & stop(double *seconds)
#define LFATAL(msg)
std::string warnAndIgnoreException(std::string const &prefix="")
#define LERROR(msg)
#define LINFO(msg)
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
cv::Mat convertToCvBGR(RawImage const &src)
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
std::string sformat(char const *fmt,...) __attribute__((format(__printf__
unsigned short constexpr White