JeVois  1.2
JeVois Smart Embedded Machine Vision Toolkit
Module.H
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #pragma once
19 
20 #include <memory>
21 #include <jevois/Image/RawImage.H>
22 #include <jevois/Core/VideoBuf.H>
23 #include <jevois/Core/UserInterface.H> // not strictly needed but derived classes will want to use it
25 #include <opencv2/core/core.hpp>
26 #include <ostream>
27 
28 namespace jevois
29 {
30  class VideoInput;
31  class VideoOutput;
32  class Engine;
33 
34  //! Exception-safe wrapper around a raw camera input frame
35  /*! This wrapper operates much like std:future in standard C++11. Users can get the next image captured by the camera
36  by calling get(), which may block if the capture is not complete yet, or may throw if the capture fails for some
37  reason (e.g., the camera is not streaming). The image size and pixel type are as defined by the current
38  VideoMapping, camera section. In addition, a done() function is provided which users may use as soon as they are
39  finished with the pixel data in the image obtained via get(), to allow the camera driver to setup the underlying
40  memory buffer again for capture. If done() has not been called by the time the InputFrame is destroyed, it will be
41  called automatically, if get() had been called. It may in some cases improve your frame rate to call done()
42  manually as early as possible instead of letting the InputFrame destructor do it.
43 
44  InputFrame implements a zero-copy, zero-wait access to input video frames, that is:
45 
46  1. the pixel data of the image you obtain via get() is directly the memory-mapped pixel buffer that the silicon
47  hardware on the JeVois chip uses via direct-memory-access (DMA) to stream the pixel data from the camera chip
48  to processor memory;
49  2. as soon as an image is captured by the camera hardware, get() unblocks and returns it (as opposed to having a
50  fixed, regular interval at which images may be available). Camera has several image buffers, allowing one to be
51  captured while another is being handed over for processing via get(). These buffers are recycled, i.e., once
52  done() is called, the underlying buffer is sent back to the camera hardware for future capture.
53 
54  \ingroup core */
55  class InputFrame
56  {
57  public:
58  //! Move constructor
59  InputFrame(InputFrame && other) = default;
60 
61  //! Get the next captured camera image
62  /*! Throws if we the camera is not streaming or blocks until an image is available (has been captured). */
63  RawImage const & get(bool casync = false) const;
64 
65  //! Indicate that user processing is done with the image previously obtained via get()
66  /*! You should call this as soon after get() as possible, once you are finished with the RawImage data so that it
67  can be recycled and sent back to the camera driver for video capture. */
68  void done() const;
69 
70  //! Shorthand to get the input image as a GRAY cv::Mat and release the raw buffer
71  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
72  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
73  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
74  be released by calling done(), so users should not call done() after using this function. This function is
75  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
76  cv::Mat getCvGRAY(bool casync = false) const;
77 
78  //! Shorthand to get the input image as a BGR cv::Mat and release the raw buffer
79  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
80  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
81  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
82  be released by calling done(), so users should not call done() after using this function. This function is
83  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
84  cv::Mat getCvBGR(bool casync = false) const;
85 
86  //! Shorthand to get the input image as a RGB cv::Mat and release the raw buffer
87  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
88  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
89  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
90  be released by calling done(), so users should not call done() after using this function. This function is
91  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
92  cv::Mat getCvRGB(bool casync = false) const;
93 
94  //! Shorthand to get the input image as a RGBA cv::Mat and release the raw buffer
95  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
96  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
97  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
98  be released by calling done(), so users should not call done() after using this function. This function is
99  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
100  cv::Mat getCvRGBA(bool casync = false) const;
101 
102  //! Destructor, returns the buffers to the driver as needed
103  ~InputFrame();
104 
105  private:
106  InputFrame() = delete;
107  InputFrame(InputFrame const & other) = delete;
108  InputFrame & operator=(InputFrame const & other) = delete;
109 
110  friend class Engine;
111  InputFrame(std::shared_ptr<VideoInput> const & cam, bool turbo); // Only our friends can construct us
112 
113  std::shared_ptr<VideoInput> itsCamera;
114  mutable bool itsDidGet;
115  mutable bool itsDidDone;
116  mutable RawImage itsImage;
117  bool const itsTurbo;
118  };
119 
120  //! Exception-safe wrapper around a raw image to be sent over USB
121  /*! This wrapper operates much like std:future in standard C++11. Users can get the next memory-allocated but blank
122  image to be sent over USB by calling get(), which may block if all buffers are still being sent over USB by Gadget
123  and no blank one is available, or may throw if getting that buffer fails for some reason (e.g., usb disconnect,
124  user just changed video mode in their webcam software or closed it). The allocated image size and pixel type is as
125  defined by the current VideoMapping, USB section, i.e., it is the USB video mode currently selected by the
126  user. To save time, image buffers are not zeroed out, so you should not assume that the image is filled with black
127  pixels, it could contain random pixels, or previous output frames. In addition, a send() function is provided
128  which users may use as soon as they are finished with writing the pixel data into the image obtained via get(), to
129  allow the USB driver to send that image to the connected host computer. If send() has not been called by the time
130  the OutputFrame is destroyed, it will be called automatically, if get() had been called.
131 
132  OutputFrame implements a zero-copy, zero-wait access to output video frames, that is:
133 
134  1. the pixel data of the image you obtain via get() is directly the memory-mapped pixel buffer that the silicon
135  hardware on the JeVois chip will use via direct-memory-access (DMA) to stream the data out over the USB link;
136  2. as soon as you call send() that buffer will be queued for sending over USB (as opposed to having a fixed,
137  regular interval at which images may be streamed out). Gadget has several image buffers, allowing one to be
138  streamed out over USB while another is being handed over for filling by your application via get(). These
139  buffers are recycled, i.e., once send() is called, the underlying buffer is streamed over USB and then sent
140  back to the Gadget for future access by your code.
141 
142  \ingroup core */
144  {
145  public:
146  //! Move constructor
147  OutputFrame(OutputFrame && other) = default;
148 
149  //! Get a pre-allocated image so that we can fill the pixel data and later send out over USB using send()
150  /*! May throw if not buffer is available, i.e., all have been queued to send to the host but have not yet been
151  sent. Application code must balance exactly one send() for each get(). */
152  RawImage const & get() const;
153 
154  //! Send an image out over USB to the host computer
155  /*! May throw if the format is incorrect or std::overflow_error if we have not yet consumed the previous image. */
156  void send() const;
157 
158  //! Shorthand to send a GRAY cv::Mat after converting it to the current output format
159  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
160  processing. The cv::Mat must have same dims as the output frame. C++ module writers should stick to the
161  get()/send() pair as this provides better fine-grained control. Note that the cv::Mat image will always be
162  copied or converted to the destination RawImage and will then be sent out immediately by calling send(), so
163  users should not call send() after using this function. This function is basically equivalent to calling
164  get(), converting the given cv::Mat to the proper output format, and calling send(). quality is used only if
165  the output format is MJPEG and should be between 1 and 100. */
166  void sendCvGRAY(cv::Mat const & img, int quality = 75) const;
167 
168  //! Shorthand to send a BGR cv::Mat after converting it to the current output format
169  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
170  processing. The cv::Mat must have same dims as the output frame. C++ module writers should stick to the
171  get()/send() pair as this provides better fine-grained control. Note that the cv::Mat image will always be
172  copied or converted to the destination RawImage and will then be sent out immediately by calling send(), so
173  users should not call send() after using this function. This function is basically equivalent to calling
174  get(), converting the given cv::Mat to the proper output format, and calling send(). quality is used only if
175  the output format is MJPEG and should be between 1 and 100.*/
176  void sendCvBGR(cv::Mat const & img, int quality = 75) const;
177 
178  //! Shorthand to send a RGB cv::Mat after converting it to the current output format
179  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
180  processing. The cv::Mat must have same dims as the output frame. C++ module writers should stick to the
181  get()/send() pair as this provides better fine-grained control. Note that the cv::Mat image will always be
182  copied or converted to the destination RawImage and will then be sent out immediately by calling send(), so
183  users should not call send() after using this function. This function is basically equivalent to calling
184  get(), converting the given cv::Mat to the proper output format, and calling send(). quality is used only if
185  the output format is MJPEG and should be between 1 and 100.*/
186  void sendCvRGB(cv::Mat const & img, int quality = 75) const;
187 
188  //! Shorthand to send a RGBA cv::Mat after converting it to the current output format
189  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
190  processing. The cv::Mat must have same dims as the output frame. C++ module writers should stick to the
191  get()/send() pair as this provides better fine-grained control. Note that the cv::Mat image will always be
192  copied or converted to the destination RawImage and will then be sent out immediately by calling send(), so
193  users should not call send() after using this function. This function is basically equivalent to calling
194  get(), converting the given cv::Mat to the proper output format, and calling send(). quality is used only if
195  the output format is MJPEG and should be between 1 and 100.*/
196  void sendCvRGBA(cv::Mat const & img, int quality = 75) const;
197 
198  //! Destructor, returns the buffers to the driver as needed
199  ~OutputFrame();
200 
201  private:
202  OutputFrame() = delete;
203  OutputFrame(OutputFrame const & other) = delete;
204  OutputFrame & operator=(OutputFrame const & other) = delete;
205 
206  // Only our friends can construct us:
207  friend class Engine;
208  OutputFrame(std::shared_ptr<VideoOutput> const & gad, RawImage * excimg = nullptr);
209 
210  std::shared_ptr<VideoOutput> itsGadget;
211  mutable bool itsDidGet;
212  mutable bool itsDidSend;
213  mutable RawImage itsImage;
214  jevois::RawImage * itsImagePtrForException;
215  };
216 
217  //! Virtual base class for a vision processing module
218  /*! Module is the base class to implement camera-to-USB frame-by-frame video processing. The Engine instantiates one
219  class derived from Module, according to the current VideoMapping selected by the end user (e.g., current image
220  resolution, format, and frame rate setected by a webcam viewing program on a host computer). The Module is loaded
221  as shared object (.so) file according to the VideoMapping definitions in videomappings.cfg and the current
222  VideoMapping selected by the user.
223 
224  Module derives from Component and as such can contain:
225 
226  - any number of Parameter settings
227 
228  - any arbitrarily complex sub-hierarchy of Component objects to implement various functionality. Parameter
229  settings from all the sub-components are available to the Module and to users connected over Serial ports of the
230  Engine.
231 
232  This allows one to implement complex vision processing pipelines efficiently and with substantial code re-use. For
233  example, one may first want to implement an EdgeDetector or Saliency component, with Parameter settings for
234  various thresholds, features, etc. One can then create any number of top-level objects that derive from Module and
235  that may contain one or more EdgeDetector, Saliency, etc components in their hierarchy of sub-components, with the
236  implementation in the module simply routing images from one Component to another to create a processing pipeline.
237 
238  Classes that derive from Module should implement up to four functions:
239 
240  - process(InputFrame && inframe, OutputFrame && outframe) is called once per iteration of the Engine main loop
241  when the current VideoMapping specifies both a particular Camera resolution and format, and a USB resolution and
242  format. This function should process the received input frame and fill the pixel buffer of the output frame with
243  results. Memory has already been allocated for both the input and output images before process() is
244  called. Because the output image is actually allocated by the USB Gadget driver (and, ultimately, by the Linux
245  kernel), its pixel memory location cannot be moved (hence, do not attempt to copy the output image or replace it
246  by another image, etc; just write pixel data into the output image's pixel array). There is no restriction on
247  video modes or frame rates, except as suported by the Camera hardware, and as limited by USB bandwidth. For most
248  implementations, matching the input and output frame rate is easiest, and means that each invocation of
249  process() would access and use both of the provided InputFrame and OutputFrame (one-input-to-one-output
250  processing pipeline). But this is not mandatory. For example, a motion flow computation Module for use in a
251  flying drone may have 320x240 YUYV 53.0fps inputs and 100x142 GREY 10.6fps output (since output frame rate is 5x
252  lower than input in this example, the process() function would here get, fill, and send the OutputFrame only
253  once every 5 times it is called; implementation of the process() function should keep track of that, e.g.,
254  through a member variable that gets incremented each time process() is called). In addition to filling the pixel
255  data of the OutputFrame, process() may also send results over the serial ports (e.g., for use by an Arduino
256  connected to the JeVois platform hardware) using sendSerial().
257 
258  - process(InputFrame && inframe) is called once per Camera frame when the current VideoMapping specifies a
259  particular Camera resolution and format, and NONE as USB output format. This function should process the
260  received input frame and would typicaly then send results to serial ports (e.g., for use by an Arduino connected
261  to the JeVois platform hardware) using sendSerial(). There is no restriction on video modes or frame rates,
262  except as suported by the Camera hardware.
263 
264  - parseSerial(std::string const & str, std::shared_ptr<UserInterface> s) allows the Module to support custom user
265  commands. Engine will forward to this function any command received over Serial or other UserInterface that it
266  does not understand. You should use this for things that go beyond Parameter settings (which is already natively
267  supported by Engine) or built-in commands of Engine (see \ref UserCli). For example, one could implement here a
268  command called "start" to allow users to start some specific thing.
269 
270  - supportedCommands(std::ostream & os) should stream out a human-readable description of any custom commands
271  supported by parseSerial(). These will be shown to users when they type "help" over a Serial port.
272 
273  \note Every module implementation file should contain a call to #JEVOIS_REGISTER_MODULE(MODULENAME) for the
274  module's class. This creates some plain-C entry points that will be used when the module is loaded from a dynamic
275  library (.so) file to instantiate the module. See \ref ModuleTutorial for examples.
276 
277  \ingroup core */
278  class Module : public Component
279  {
280  public:
281  //! Constructor
282  /*! the instance is a user-defined string that may be used to differentiate between several instances of the
283  same module. */
284  Module(std::string const & instance);
285 
286  //! Virtual destructor for safe inheritance
287  virtual ~Module();
288 
289  //! Processing function, version that receives a frame from camera and sends a frame out over USB
290  /*! This function is called once for each grabbed video frame from the camera, and it should complete within the
291  camera's frame period in order to avoid dropping frames. The InputFrame and OutputFrame objects are simple
292  wrappers to ensure that the low-level video buffers will always be returned to the low-level camera and USB
293  drivers even if the process function throws at any point during the processing. If any error occurs, it is
294  hence ok to throw from within process() at any time, just make sure your locally allocated resources will be
295  freed, which is usually best achieved by using shared_ptr and similar wrappers around them. The Engine (which
296  calls process() on your module for every frame) will catch any exception an proceed to the next frame.
297 
298  Default implementation in the base class just throws. Derived classes should override it. */
299  virtual void process(InputFrame && inframe, OutputFrame && outframe);
300 
301  //! Processing function, version that receives a frame from camera and does not use USB
302  /*! This function is called once for each grabbed video frame from the camera, and it should complete within the
303  camera's frame period in order to avoid dropping frames. The InputFrame object is a simple wrapper to ensure
304  that the low-level video buffers will always be returned to the low-level camera driver even if the process
305  function throws at any point during the processing. If any error occurs, it is hence ok to throw from within
306  process() at any time, just make sure your locally allocated resources will be freed, which is usually best
307  achieved by using shared_ptr and similar wrappers around them. The Engine (which calls process() on your
308  module for every frame) will catch any exception an proceed to the next frame.
309 
310  Default implementation in the base class just throws. Derived classes should override it. */
311  virtual void process(InputFrame && inframe);
312 
313  //! Send a string over the 'serout' serial port
314  /*! The default implementation just sends the string to the serial port specified by the 'serout' Parameter in
315  Engine (which could be the hardware serial port, the serial-over-USB port, both, or none; see \ref UserCli for
316  information about \c serout). No need to override in most cases. Typically, you would use this function from
317  within process() to send out some results of your processing.
318 
319  Note that the default 'serout' Parameter setting in Engine is None. This is to allow users to configure
320  parameters, get parameter values, possibly read the help message, etc before the flow of serial outputs from
321  vision processing starts. Once ready to receive serial outputs, one would typically issue a command 'setpar
322  serout Hard' over the JeVois command line to enable serial outputs to the hardware serial port. An Arduino
323  would issue that setpar commands when it is ready to work. See ArduinoTutorial for an example. */
324  virtual void sendSerial(std::string const & str);
325 
326  //! Receive a string from a serial port which contains a user command
327  /*! This function may be called in between calls to process() with any received string from any of the serial
328  ports. Some commands are parsed upstream already (like "help", set param value, set camera control, etc; see
329  the Engine class) and will not be received here. Only the ones not recognized by the Engine will be received
330  (i.e., custom commands specific to your module).
331 
332  The default implementation just throws std::runtime_error("Unsupported command"), but some modules may want to
333  override this function to handle custom commands. If you successfully process the command, just return;
334  otherwise, throw, and if your exception derives from std::exception, the Engine will append its what() to the
335  error message issued to the user. When you support commands here, you should update the implementation of
336  supportedCommands() to provide some description of those commands to the users.
337 
338  The \c s parameter is the serial port that received the command. You can send any results back to that port
339  using writeString() on it. Note that the Engine will automatically add the 'OK' message upon success, so you
340  do not have to send that here. */
341  virtual void parseSerial(std::string const & str, std::shared_ptr<UserInterface> s);
342 
343  //! Human-readable description of this Module's supported custom commands
344  /*! The format here is free. Just use std::endl to demarcate lines, these will be converted to the appropriate
345  line endings by the serial ports. Default implementation writes "None" to os. */
346  virtual void supportedCommands(std::ostream & os);
347  };
348 }
349 
350 //! Register a module, allowing it to be dynamically loaded from a .so file
351 /* \def JEVOIS_REGISTER_MODULE(MODULENAME)
352  \hideinitializer
353 
354  Every module implementation file should contain a call to JEVOIS_REGISTER_MODULE for the module's class. This creates
355  some plain-C entry points that will be used when the module is loaded from a dynamic library (.so) file to
356  instantiate the module. \relates Module */
357 #define JEVOIS_REGISTER_MODULE(MODULENAME) \
358  extern "C" std::shared_ptr<jevois::Module> MODULENAME##_create(std::string const & instanceid) \
359  { return std::shared_ptr<jevois::Module>(new MODULENAME(instanceid)); } \
360  extern "C" int MODULENAME##_version_major() { return JEVOIS_VERSION_MAJOR; } \
361  extern "C" int MODULENAME##_version_minor() { return JEVOIS_VERSION_MINOR; } \
362 
363 
void done() const
Indicate that user processing is done with the image previously obtained via get() ...
Definition: Module.C:53
void sendCvGRAY(cv::Mat const &img, int quality=75) const
Shorthand to send a GRAY cv::Mat after converting it to the current output format.
Definition: Module.C:142
Exception-safe wrapper around a raw camera input frame.
Definition: Module.H:55
cv::Mat getCvGRAY(bool casync=false) const
Shorthand to get the input image as a GRAY cv::Mat and release the raw buffer.
Definition: Module.C:60
Module(std::string const &instance)
Constructor.
Definition: Module.C:174
virtual void supportedCommands(std::ostream &os)
Human-readable description of this Module's supported custom commands.
Definition: Module.C:205
A component of a model hierarchy.
Definition: Component.H:176
virtual void process(InputFrame &&inframe, OutputFrame &&outframe)
Processing function, version that receives a frame from camera and sends a frame out over USB...
A raw image as coming from a V4L2 Camera and/or being sent out to a USB Gadget.
Definition: RawImage.H:110
void sendCvRGBA(cv::Mat const &img, int quality=75) const
Shorthand to send a RGBA cv::Mat after converting it to the current output format.
Definition: Module.C:165
void sendCvRGB(cv::Mat const &img, int quality=75) const
Shorthand to send a RGB cv::Mat after converting it to the current output format. ...
Definition: Module.C:157
cv::Mat getCvRGBA(bool casync=false) const
Shorthand to get the input image as a RGBA cv::Mat and release the raw buffer.
Definition: Module.C:87
void send() const
Send an image out over USB to the host computer.
Definition: Module.C:134
virtual void sendSerial(std::string const &str)
Send a string over the 'serout' serial port.
Definition: Module.C:191
void sendCvBGR(cv::Mat const &img, int quality=75) const
Shorthand to send a BGR cv::Mat after converting it to the current output format. ...
Definition: Module.C:150
cv::Mat getCvRGB(bool casync=false) const
Shorthand to get the input image as a RGB cv::Mat and release the raw buffer.
Definition: Module.C:78
JeVois processing engine - gets images from camera sensor, processes them, and sends results over USB...
Definition: Engine.H:229
virtual void parseSerial(std::string const &str, std::shared_ptr< UserInterface > s)
Receive a string from a serial port which contains a user command.
Definition: Module.C:200
Virtual base class for a vision processing module.
Definition: Module.H:278
Exception-safe wrapper around a raw image to be sent over USB.
Definition: Module.H:143
~InputFrame()
Destructor, returns the buffers to the driver as needed.
Definition: Module.C:31
cv::Mat getCvBGR(bool casync=false) const
Shorthand to get the input image as a BGR cv::Mat and release the raw buffer.
Definition: Module.C:69
~OutputFrame()
Destructor, returns the buffers to the driver as needed.
Definition: Module.C:102
virtual ~Module()
Virtual destructor for safe inheritance.
Definition: Module.C:179