JeVois  1.0
JeVois Smart Embedded Machine Vision Toolkit
Module.H
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #pragma once
19 
20 #include <memory>
21 #include <jevois/Image/RawImage.H>
22 #include <jevois/Core/VideoBuf.H>
24 #include <ostream>
25 
26 namespace jevois
27 {
28  class VideoInput;
29  class VideoOutput;
30  class Engine;
31  class UserInterface;
32 
33  //! Exception-safe wrapper around a raw camera input frame
34  /*! This wrapper operates much like std:future in standard C++11. Users can get the next image captured by the camera
35  by calling get(), which may block if the capture is not complete yet, or may throw if the capture fails for some
36  reason (e.g., the camera is not streaming). The image size and pixel type are as defined by the current
37  VideoMapping, camera section. In addition, a done() function is provided which users may use as soon as they are
38  finished with the pixel data in the image obtained via get(), to allow the camera driver to setup the underlying
39  memory buffer again for capture. If done() has not been called by the time the InputFrame is destroyed, it will be
40  called automatically, if get() had been called. It may in some cases improve your frame rate to call done()
41  manually as early as possible instead of letting the InputFrame destructor do it.
42 
43  InputFrame implements a zero-copy, zero-wait access to input video frames, that is:
44 
45  1. the pixel data of the image you obtain via get() is directly the memory-mapped pixel buffer that the silicon
46  hardware on the JeVois chip uses via direct-memory-access (DMA) to stream the pixel data from the camera chip
47  to processor memory;
48  2. as soon as an image is captured by the camera hardware, get() unblocks and returns it (as opposed to having a
49  fixed, regular interval at which images may be available). Camera has several image buffers, allowing one to be
50  captured while another is being handed over for processing via get(). These buffers are recycled, i.e., once
51  done() is called, the underlying buffer is sent back to the camera hardware for future capture.
52 
53  \ingroup core */
54  class InputFrame
55  {
56  public:
57  //! Move constructor
58  InputFrame(InputFrame && other) = default;
59 
60  //! Get the next captured camera image
61  /*! Throws if we the camera is not streaming or blocks until an image is available (has been captured). */
62  RawImage const & get(bool casync = false) const;
63 
64  //! Indicate that user processing is done with the image previously obtained via get()
65  /*! You should call this as soon after get() as possible, once you are finished with the RawImage data so that it
66  can be recycled and sent back to the camera driver for video capture. */
67  void done() const;
68 
69  //! Destructor, returns the buffers to the driver as needed
70  ~InputFrame();
71 
72  private:
73  InputFrame() = delete;
74  InputFrame(InputFrame const & other) = delete;
75  InputFrame & operator=(InputFrame const & other) = delete;
76 
77  friend class Engine;
78  InputFrame(std::shared_ptr<VideoInput> const & cam, bool turbo); // Only our friends can construct us
79 
80  std::shared_ptr<VideoInput> itsCamera;
81  mutable bool itsDidGet;
82  mutable bool itsDidDone;
83  mutable RawImage itsImage;
84  bool const itsTurbo;
85  };
86 
87  //! Exception-safe wrapper around a raw image to be sent over USB
88  /*! This wrapper operates much like std:future in standard C++11. Users can get the next memory-allocated but blank
89  image to be sent over USB by calling get(), which may block if all buffers are still being sent over USB by Gadget
90  and no blank one is available, or may throw if getting that buffer fails for some reason (e.g., usb disconnect,
91  user just changed video mode in their webcam software or closed it). The allocated image size and pixel type is as
92  defined by the current VideoMapping, USB section, i.e., it is the USB video mode currently selected by the
93  user. To save time, image buffers are not zeroed out, so you should not assume that the image is filled with black
94  pixels, it could contain random pixels, or previous output frames. In addition, a send() function is provided
95  which users may use as soon as they are finished with writing the pixel data into the image obtained via get(), to
96  allow the USB driver to send that image to the connected host computer. If send() has not been called by the time
97  the OutputFrame is destroyed, it will be called automatically, if get() had been called.
98 
99  OutputFrame implements a zero-copy, zero-wait access to output video frames, that is:
100 
101  1. the pixel data of the image you obtain via get() is directly the memory-mapped pixel buffer that the silicon
102  hardware on the JeVois chip will use via direct-memory-access (DMA) to stream the data out over the USB link;
103  2. as soon as you call send() that buffer will be queued for sending over USB (as opposed to having a fixed,
104  regular interval at which images may be streamed out). Gadget has several image buffers, allowing one to be
105  streamed out over USB while another is being handed over for filling by your application via get(). These
106  buffers are recycled, i.e., once send() is called, the underlying buffer is streamed over USB and then sent
107  back to the Gadget for future access by your code.
108 
109  \ingroup core */
111  {
112  public:
113  //! Move constructor
114  OutputFrame(OutputFrame && other) = default;
115 
116  //! Get a pre-allocated image so that we can fill the pixel data and later send out over USB using send()
117  /*! May throw if not buffer is available, i.e., all have been queued to send to the host but have not yet been
118  sent. Application code must balance exactly one send() for each get(). */
119  RawImage const & get() const;
120 
121  //! Send an image out over USB to the host computer
122  /*! May throw if the format is incorrect or std::overflow_error if we have not yet consumed the previous image. */
123  void send() const;
124 
125  //! Destructor, returns the buffers to the driver as needed
126  ~OutputFrame();
127 
128  private:
129  OutputFrame() = delete;
130  OutputFrame(OutputFrame const & other) = delete;
131  OutputFrame & operator=(OutputFrame const & other) = delete;
132 
133  friend class Engine;
134  OutputFrame(std::shared_ptr<VideoOutput> const & gad); // Only our friends can construct us
135 
136  std::shared_ptr<VideoOutput> itsGadget;
137  mutable bool itsDidGet;
138  mutable bool itsDidSend;
139  mutable RawImage itsImage;
140  };
141 
142  //! Virtual base class for a vision processing module
143  /*! Module is the base class to implement camera-to-USB frame-by-frame video processing. The Engine instantiates one
144  class derived from Module, according to the current VideoMapping selected by the end user (e.g., current image
145  resolution, format, and frame rate setected by a webcam viewing program on a host computer). The Module is loaded
146  as shared object (.so) file according to the VideoMapping definitions in videomappings.cfg and the current
147  VideoMapping selected by the user.
148 
149  Module derives from Component and as such can contain:
150 
151  - any number of Parameter settings
152 
153  - any arbitrarily complex sub-hierarchy of Component objects to implement various functionality. Parameter
154  settings from all the sub-components are available to the Module and to users connected over Serial ports of the
155  Engine.
156 
157  This allows one to implement complex vision processing pipelines efficiently and with substantial code re-use. For
158  example, one may first want to implement an EdgeDetector or Saliency component, with Parameter settings for
159  various thresholds, features, etc. One can then create any number of top-level objects that derive from Module and
160  that may contain one or more EdgeDetector, Saliency, etc components in their hierarchy of sub-components, with the
161  implementation in the module simply routing images from one Component to another to create a processing pipeline.
162 
163  Classes that derive from Module should implement up to four functions:
164 
165  - process(InputFrame && inframe, OutputFrame && outframe) is called once per iteration of the Engine main loop
166  when the current VideoMapping specifies both a particular Camera resolution and format, and a USB resolution and
167  format. This function should process the received input frame and fill the pixel buffer of the output frame with
168  results. Memory has already been allocated for both the input and output images before process() is
169  called. Because the output image is actually allocated by the USB Gadget driver (and, ultimately, by the Linux
170  kernel), its pixel memory location cannot be moved (hence, do not attempt to copy the output image or replace it
171  by another image, etc; just write pixel data into the output image's pixel array). There is no restriction on
172  video modes or frame rates, except as suported by the Camera hardware, and as limited by USB bandwidth. For most
173  implementations, matching the input and output frame rate is easiest, and means that each invocation of
174  process() would access and use both of the provided InputFrame and OutputFrame (one-input-to-one-output
175  processing pipeline). But this is not mandatory. For example, a motion flow computation Module for use in a
176  flying drone may have 320x240 YUYV 53.0fps inputs and 100x142 GREY 10.6fps output (since output frame rate is 5x
177  lower than input in this example, the process() function would here get, fill, and send the OutputFrame only
178  once every 5 times it is called; implementation of the process() function should keep track of that, e.g.,
179  through a member variable that gets incremented each time process() is called). In addition to filling the pixel
180  data of the OutputFrame, process() may also send results over the serial ports (e.g., for use by an Arduino
181  connected to the JeVois platform hardware) using sendSerial().
182 
183  - process(InputFrame && inframe) is called once per Camera frame when the current VideoMapping specifies a
184  particular Camera resolution and format, and NONE as USB output format. This function should process the
185  received input frame and would typicaly then send results to serial ports (e.g., for use by an Arduino connected
186  to the JeVois platform hardware) using sendSerial(). There is no restriction on video modes or frame rates,
187  except as suported by the Camera hardware.
188 
189  - parseSerial(std::string const & str, std::shared_ptr<UserInterface> s) allows the Module to support custom user
190  commands. Engine will forward to this function any command received over Serial or other UserInterface that it
191  does not understand. You should use this for things that go beyond Parameter settings (which is already natively
192  supported by Engine) or built-in commands of Engine (see \ref UserCli). For example, one could implement here a
193  command called "start" to allow users to start some specific thing.
194 
195  - supportedCommands(std::ostream & os) should stream out a human-readable description of any custom commands
196  supported by parseSerial(). These will be shown to users when they type "help" over a Serial port.
197 
198  \note Every module implementation file should contain a call to #JEVOIS_REGISTER_MODULE(MODULENAME) for the
199  module's class. This creates some plain-C entry points that will be used when the module is loaded from a dynamic
200  library (.so) file to instantiate the module. See \ref ModuleTutorial for examples.
201 
202  \ingroup core */
203  class Module : public Component
204  {
205  public:
206  //! Constructor
207  /*! the instance is a user-defined string that may be used to differentiate between several instances of the
208  same module. */
209  Module(std::string const & instance);
210 
211  //! Virtual destructor for safe inheritance
212  virtual ~Module();
213 
214  //! Processing function, version that receives a frame from camera and sends a frame out over USB
215  /*! This function is called once for each grabbed video frame from the camera, and it should complete within the
216  camera's frame period in order to avoid dropping frames. The InputFrame and OutputFrame objects are simple
217  wrappers to ensure that the low-level video buffers will always be returned to the low-level camera and USB
218  drivers even if the process function throws at any point during the processing. If any error occurs, it is
219  hence ok to throw from within process() at any time, just make sure your locally allocated resources will be
220  freed, which is usually best achieved by using shared_ptr and similar wrappers around them. The Engine (which
221  calls process() on your module for every frame) will catch any exception an proceed to the next frame.
222 
223  Default implementation in the base class just throws. Derived classes should override it. */
224  virtual void process(InputFrame && inframe, OutputFrame && outframe);
225 
226  //! Processing function, version that receives a frame from camera and does not use USB
227  /*! This function is called once for each grabbed video frame from the camera, and it should complete within the
228  camera's frame period in order to avoid dropping frames. The InputFrame object is a simple wrapper to ensure
229  that the low-level video buffers will always be returned to the low-level camera driver even if the process
230  function throws at any point during the processing. If any error occurs, it is hence ok to throw from within
231  process() at any time, just make sure your locally allocated resources will be freed, which is usually best
232  achieved by using shared_ptr and similar wrappers around them. The Engine (which calls process() on your
233  module for every frame) will catch any exception an proceed to the next frame.
234 
235  Default implementation in the base class just throws. Derived classes should override it. */
236  virtual void process(InputFrame && inframe);
237 
238  //! Send a string over the 'serout' serial port
239  /*! The default implementation just sends the string to the serial port specified by the 'serout' Parameter in
240  Engine (which could be the hardware serial port, the serial-over-USB port, both, or none; see \ref UserCli for
241  information about \c serout). No need to override in most cases. Typically, you would use this function from
242  within process() to send out some results of your processing.
243 
244  Note that the default 'serout' Parameter setting in Engine is None. This is to allow users to configure
245  parameters, get parameter values, possibly read the help message, etc before the flow of serial outputs from
246  vision processing starts. Once ready to receive serial outputs, one would typically issue a command 'setpar
247  serout Hard' over the JeVois command line to enable serial outputs to the hardware serial port. An Arduino
248  would issue that setpar commands when it is ready to work. See ArduinoTutorial for an example. */
249  virtual void sendSerial(std::string const & str);
250 
251  //! Receive a string from a serial port which contains a user command
252  /*! This function may be called in between calls to process() with any received string from any of the serial
253  ports. Some commands are parsed upstream already (like "help", set param value, set camera control, etc; see
254  the Engine class) and will not be received here. Only the ones not recognized by the Engine will be received
255  (i.e., custom commands specific to your module).
256 
257  The default implementation just throws std::runtime_error("Unsupported command"), but some modules may want to
258  override this function to handle custom commands. If you successfully process the command, just return;
259  otherwise, throw, and if your exception derives from std::exception, the Engine will append its what() to the
260  error message issued to the user. When you support commands here, you should update the implementation of
261  supportedCommands() to provide some description of those commands to the users.
262 
263  The \c s parameter is the serial port that received the command. You can send any results back to that port
264  using writeString() on it. Note that the Engine will automatically add the 'OK' message upon success, so you
265  do not have to send that here. */
266  virtual void parseSerial(std::string const & str, std::shared_ptr<UserInterface> s);
267 
268  //! Human-readable description of this Module's supported custom commands
269  /*! The format here is free. Just use std::endl to demarcate lines, these will be converted to the appropriate
270  line endings by the serial ports. Default implementation writes "None" to os. */
271  virtual void supportedCommands(std::ostream & os);
272  };
273 }
274 
275 //! Register a module, allowing it to be dynamically loaded from a .so file
276 /* \def JEVOIS_REGISTER_MODULE(MODULENAME)
277  \hideinitializer
278 
279  Every module implementation file should contain a call to JEVOIS_REGISTER_MODULE for the module's class. This creates
280  some plain-C entry points that will be used when the module is loaded from a dynamic library (.so) file to
281  instantiate the module. \relates Module */
282 #define JEVOIS_REGISTER_MODULE(MODULENAME) \
283  extern "C" std::shared_ptr<jevois::Module> MODULENAME##_create(std::string const & instanceid) \
284  { return std::shared_ptr<jevois::Module>(new MODULENAME(instanceid)); } \
285  extern "C" int MODULENAME##_version_major() { return JEVOIS_VERSION_MAJOR; } \
286  extern "C" int MODULENAME##_version_minor() { return JEVOIS_VERSION_MINOR; } \
287 
288 
void done() const
Indicate that user processing is done with the image previously obtained via get() ...
Definition: Module.C:52
Exception-safe wrapper around a raw camera input frame.
Definition: Module.H:54
Module(std::string const &instance)
Constructor.
Definition: Module.C:94
virtual void supportedCommands(std::ostream &os)
Human-readable description of this Module's supported custom commands.
Definition: Module.C:125
A component of a model hierarchy.
Definition: Component.H:176
virtual void process(InputFrame &&inframe, OutputFrame &&outframe)
Processing function, version that receives a frame from camera and sends a frame out over USB...
A raw image as coming from a V4L2 Camera and/or being sent out to a USB Gadget.
Definition: RawImage.H:110
void send() const
Send an image out over USB to the host computer.
Definition: Module.C:86
virtual void sendSerial(std::string const &str)
Send a string over the 'serout' serial port.
Definition: Module.C:111
JeVois processing engine - gets images from camera sensor, processes them, and sends results over USB...
Definition: Engine.H:223
virtual void parseSerial(std::string const &str, std::shared_ptr< UserInterface > s)
Receive a string from a serial port which contains a user command.
Definition: Module.C:120
Virtual base class for a vision processing module.
Definition: Module.H:203
Exception-safe wrapper around a raw image to be sent over USB.
Definition: Module.H:110
~InputFrame()
Destructor, returns the buffers to the driver as needed.
Definition: Module.C:30
~OutputFrame()
Destructor, returns the buffers to the driver as needed.
Definition: Module.C:65
virtual ~Module()
Virtual destructor for safe inheritance.
Definition: Module.C:99