JeVois  1.16
JeVois Smart Embedded Machine Vision Toolkit
Share this page:
InputFrame.H
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #pragma once
19 
20 #include <jevois/Image/RawImage.H>
21 #include <opencv2/core/core.hpp>
22 #include <memory>
23 
24 namespace jevois
25 {
26  class VideoInput;
27  class Engine;
28 
29  //! Exception-safe wrapper around a raw camera input frame
30  /*! This wrapper operates much like std:future in standard C++11. Users can get the next image captured by the camera
31  by calling get(), which may block if the capture is not complete yet, or may throw if the capture fails for some
32  reason (e.g., the camera is not streaming). The image size and pixel type are as defined by the current
33  VideoMapping, camera section. In addition, a done() function is provided which users may use as soon as they are
34  finished with the pixel data in the image obtained via get(), to allow the camera driver to setup the underlying
35  memory buffer again for capture. If done() has not been called by the time the InputFrame is destroyed, it will be
36  called automatically, if get() had been called. It may in some cases improve your frame rate to call done()
37  manually as early as possible instead of letting the InputFrame destructor do it.
38 
39  InputFrame implements a zero-copy, zero-wait access to input video frames, that is:
40 
41  1. the pixel data of the image you obtain via get() is directly the memory-mapped pixel buffer that the silicon
42  hardware on the JeVois chip uses via direct-memory-access (DMA) to stream the pixel data from the camera chip
43  to processor memory;
44  2. as soon as an image is captured by the camera hardware, get() unblocks and returns it (as opposed to having a
45  fixed, regular interval at which images may be available). Camera has several image buffers, allowing one to be
46  captured while another is being handed over for processing via get(). These buffers are recycled, i.e., once
47  done() is called, the underlying buffer is sent back to the camera hardware for future capture.
48 
49  \ingroup core */
50  class InputFrame
51  {
52  public:
53  //! Move constructor
54  InputFrame(InputFrame && other) = default;
55 
56  //! Get the next captured camera image
57  /*! Throws if we the camera is not streaming or blocks until an image is available (has been captured). It is ok
58  to call get() several times, but the same image will always be returned. To obtain successive frames from the
59  camera, you must be fed successive InputFrame wrappers by the JeVois Engine. */
60  RawImage const & get(bool casync = false) const;
61 
62  //! Check whether a second input image scaled by the JeVoisPro Platform ISP is available
63  /*! Returns false unless we are on JeVois-Pro Platform and the camera format modifier jevois::CropType::CropScale
64  is currently in use. */
65  bool hasScaledImage() const;
66 
67  //! Get the next captured camera image, ISP-scaled second frame
68  /*! On JeVois-Pro Platform only, the camera ISP can output 2 frames: 1) raw from sensor, 2) scaled by ISP. This
69  function is to access the ISP scaled frame. Throws if not JeVois-Pro Platform or the camera stream type is not
70  jevois::StreamType::RawAndScaled. Throws if we the camera is not streaming or blocks until an image is
71  available (has been captured). It is ok to call get2() several times, but the same image will always be
72  returned. To obtain successive frames from the camera, you must be fed successive InputFrame wrappers by the
73  JeVois Engine. */
74  RawImage const & get2(bool casync = false) const;
75 
76  //! Get the next captured camera image that is intended for processing
77  /*! Same as get() if hasScaledImage() is false, or as get2() if hasScaledImage() is true. */
78  RawImage const & getp(bool casync = false) const;
79 
80  //! Get the DMA-BUF file descriptor of the camera frame
81  /*! This file descriptor can be used to share the pixel buffer across interfaces that support DMA-BUF. The JeVois
82  core uses this to create a zero-copy pipeline from the camera sensor to the hardware-accelerated OpenGL
83  display on JeVois-Pro. Returns -1 if DMA-BUF is not supported (on host, and on JeVois-A33 platform). If get()
84  has not previously been called, it will be called, which blocks until the next camera frame is available. It
85  is ok to call getDmaFd() several times but always the same fd is returned (see get()). */
86  int getDmaFd(bool casync = false) const;
87 
88  //! Get the DMA-BUF file descriptor of the ISP-scaled second camera frame
89  /*! On JeVois-Pro Platform only, the camera ISP can output 2 frames: 1) raw from sensor, 2) scaled by ISP. This
90  function is to access the ISP scaled frame. Throws if not JeVois-Pro Platform or the camera stream type is not
91  jevois::StreamType::RawAndScaled. This file descriptor can be used to share the pixel buffer across interfaces
92  that support DMA-BUF. The JeVois core uses this to create a zero-copy pipeline from the camera sensor to the
93  hardware-accelerated OpenGL display on JeVois-Pro. Returns -1 if DMA-BUF is not supported (on host, and on
94  JeVois-A33 platform). If get() has not previously been called, it will be called, which blocks until the next
95  camera frame is available. It is ok to call getDmaFd() several times but always the same fd is returned (see
96  get2()). */
97  int getDmaFd2(bool casync = false) const;
98 
99  //! Indicate that user processing is done with the image previously obtained via get()
100  /*! You should call this as soon after get() as possible, once you are finished with the RawImage data so that it
101  can be recycled and sent back to the camera driver for video capture. */
102  void done() const;
103 
104  //! Indicate that user processing is done with the ISP-scaled image previously obtained via get2()
105  /*! You should call this as soon after get() as possible, once you are finished with the RawImage data so that it
106  can be recycled and sent back to the camera driver for video capture. */
107  void done2() const;
108 
109  //! Shorthand to get the input image as a GRAY cv::Mat and release the raw buffer
110  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
111  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
112  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
113  be released by calling done(), so users should not call done() after using this function. This function is
114  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
115  cv::Mat getCvGRAY(bool casync = false) const;
116 
117  //! Shorthand to get the input image as a BGR cv::Mat and release the raw buffer
118  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
119  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
120  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
121  be released by calling done(), so users should not call done() after using this function. This function is
122  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
123  cv::Mat getCvBGR(bool casync = false) const;
124 
125  //! Shorthand to get the input image as a RGB cv::Mat and release the raw buffer
126  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
127  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
128  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
129  be released by calling done(), so users should not call done() after using this function. This function is
130  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
131  cv::Mat getCvRGB(bool casync = false) const;
132 
133  //! Shorthand to get the input image as a RGBA cv::Mat and release the raw buffer
134  /*! This is mostly intended for Python module writers, as they will likely use OpenCV for all their image
135  processing. C++ module writers should stick to the get()/done() pair as this provides better fine-grained
136  control. Note that the raw image from the camera will always be copied or converted to cv::Mat and will then
137  be released by calling done(), so users should not call done() after using this function. This function is
138  basically equivalent to calling get(), converting to cv::Mat, and calling done(). */
139  cv::Mat getCvRGBA(bool casync = false) const;
140 
141  //! Shorthand to get the input image for processing as a GRAY cv::Mat and release the raw buffer
142  /*! Returns the frame intended for processing, i.e., either the single camera frame when using single-stream
143  capture, or the second frame when using dual stream capture. This is mostly intended for Python module
144  writers, as they will likely use OpenCV for all their image processing. C++ module writers should stick to the
145  get()/done() pair as this provides better fine-grained control. Note that the raw image from the camera will
146  always be copied or converted to cv::Mat and will then be released by calling done(), so users should not call
147  done() after using this function. This function is basically equivalent to calling get(), converting to
148  cv::Mat, and calling done(). */
149  cv::Mat getCvGRAYp(bool casync = false) const;
150 
151  //! Shorthand to get the input image for processing as a BGR cv::Mat and release the raw buffer
152  /*! Returns the frame intended for processing, i.e., either the single camera frame when using single-stream
153  capture, or the second frame when using dual stream capture. This is mostly intended for Python module
154  writers, as they will likely use OpenCV for all their image processing. C++ module writers should stick to the
155  get()/done() pair as this provides better fine-grained control. Note that the raw image from the camera will
156  always be copied or converted to cv::Mat and will then be released by calling done(), so users should not call
157  done() after using this function. This function is basically equivalent to calling get(), converting to
158  cv::Mat, and calling done(). */
159  cv::Mat getCvBGRp(bool casync = false) const;
160 
161  //! Shorthand to get the input image for processing as a RGB cv::Mat and release the raw buffer
162  /*! Returns the frame intended for processing, i.e., either the single camera frame when using single-stream
163  capture, or the second frame when using dual stream capture. This is mostly intended for Python module
164  writers, as they will likely use OpenCV for all their image processing. C++ module writers should stick to the
165  get()/done() pair as this provides better fine-grained control. Note that the raw image from the camera will
166  always be copied or converted to cv::Mat and will then be released by calling done(), so users should not call
167  done() after using this function. This function is basically equivalent to calling get(), converting to
168  cv::Mat, and calling done(). */
169  cv::Mat getCvRGBp(bool casync = false) const;
170 
171  //! Shorthand to get the input image for processing as a RGBA cv::Mat and release the raw buffer
172  /*! Returns the frame intended for processing, i.e., either the single camera frame when using single-stream
173  capture, or the second frame when using dual stream capture. This is mostly intended for Python module
174  writers, as they will likely use OpenCV for all their image processing. C++ module writers should stick to the
175  get()/done() pair as this provides better fine-grained control. Note that the raw image from the camera will
176  always be copied or converted to cv::Mat and will then be released by calling done(), so users should not call
177  done() after using this function. This function is basically equivalent to calling get(), converting to
178  cv::Mat, and calling done(). */
179  cv::Mat getCvRGBAp(bool casync = false) const;
180 
181  //! Destructor, returns the buffers to the driver as needed
182  ~InputFrame();
183 
184  private:
185  InputFrame() = delete;
186  InputFrame(InputFrame const & other) = delete;
187  InputFrame & operator=(InputFrame const & other) = delete;
188 
189  friend class Engine;
190  InputFrame(std::shared_ptr<VideoInput> const & cam, bool turbo); // Only our friends can construct us
191 
192  std::shared_ptr<VideoInput> itsCamera;
193  mutable bool itsDidGet = false, itsDidGet2 = false;
194  mutable bool itsDidDone = false, itsDidDone2 = false;
195  mutable RawImage itsImage, itsImage2;
196  mutable int itsDmaFd = -1, itsDmaFd2 = -1;
197  bool const itsTurbo;
198  };
199 
200 } // namespace jevois
jevois::InputFrame::getCvRGBA
cv::Mat getCvRGBA(bool casync=false) const
Shorthand to get the input image as a RGBA cv::Mat and release the raw buffer.
Definition: InputFrame.C:143
jevois::InputFrame::getp
const RawImage & getp(bool casync=false) const
Get the next captured camera image that is intended for processing.
Definition: InputFrame.C:79
jevois::InputFrame::getCvRGBAp
cv::Mat getCvRGBAp(bool casync=false) const
Shorthand to get the input image for processing as a RGBA cv::Mat and release the raw buffer.
Definition: InputFrame.C:179
jevois::InputFrame::getDmaFd
int getDmaFd(bool casync=false) const
Get the DMA-BUF file descriptor of the camera frame.
Definition: InputFrame.C:86
jevois::InputFrame::getCvGRAYp
cv::Mat getCvGRAYp(bool casync=false) const
Shorthand to get the input image for processing as a GRAY cv::Mat and release the raw buffer.
Definition: InputFrame.C:152
RawImage.H
jevois::InputFrame::getCvRGB
cv::Mat getCvRGB(bool casync=false) const
Shorthand to get the input image as a RGB cv::Mat and release the raw buffer.
Definition: InputFrame.C:134
jevois::RawImage
A raw image as coming from a V4L2 Camera and/or being sent out to a USB Gadget.
Definition: RawImage.H:110
jevois::InputFrame::getCvRGBp
cv::Mat getCvRGBp(bool casync=false) const
Shorthand to get the input image for processing as a RGB cv::Mat and release the raw buffer.
Definition: InputFrame.C:170
jevois::InputFrame::getCvBGRp
cv::Mat getCvBGRp(bool casync=false) const
Shorthand to get the input image for processing as a BGR cv::Mat and release the raw buffer.
Definition: InputFrame.C:161
jevois::InputFrame::getCvGRAY
cv::Mat getCvGRAY(bool casync=false) const
Shorthand to get the input image as a GRAY cv::Mat and release the raw buffer.
Definition: InputFrame.C:116
jevois::InputFrame::get
const RawImage & get(bool casync=false) const
Get the next captured camera image.
Definition: InputFrame.C:50
jevois
Definition: Concepts.dox:1
jevois::InputFrame::~InputFrame
~InputFrame()
Destructor, returns the buffers to the driver as needed.
Definition: InputFrame.C:31
jevois::InputFrame::getCvBGR
cv::Mat getCvBGR(bool casync=false) const
Shorthand to get the input image as a BGR cv::Mat and release the raw buffer.
Definition: InputFrame.C:125
jevois::InputFrame::hasScaledImage
bool hasScaledImage() const
Check whether a second input image scaled by the JeVoisPro Platform ISP is available.
Definition: InputFrame.C:61
jevois::InputFrame::done
void done() const
Indicate that user processing is done with the image previously obtained via get()
Definition: InputFrame.C:102
jevois::InputFrame::get2
const RawImage & get2(bool casync=false) const
Get the next captured camera image, ISP-scaled second frame.
Definition: InputFrame.C:67
jevois::Engine
JeVois processing engine - gets images from camera sensor, processes them, and sends results over USB...
Definition: Engine.H:381
jevois::InputFrame::getDmaFd2
int getDmaFd2(bool casync=false) const
Get the DMA-BUF file descriptor of the ISP-scaled second camera frame.
Definition: InputFrame.C:94
jevois::InputFrame
Exception-safe wrapper around a raw camera input frame.
Definition: InputFrame.H:50
jevois::InputFrame::done2
void done2() const
Indicate that user processing is done with the ISP-scaled image previously obtained via get2()
Definition: InputFrame.C:109