JeVoisBase  1.17
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
PassThrough.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
21 #include <linux/videodev2.h>
22 
23 // icon by Catalin Fertu in cinema at flaticon
24 
25 //! Simple module that just passes the captured camera frames through to USB host
26 /*! This module makes your JeVois smart camera operate like a regular "dumb" camera. It is intended mainly for use in
27  programming tutorials, and to allow you to debug new machine vision modules that you test on your host computer,
28  using the JeVois camera in pass-through mode as input, to simulate what will happen when your code runs on the
29  JeVois embedded processor.
30 
31  Any video mapping is possible here, as long as camera and USB pixel types match, and camera and USB image
32  resolutions also match.
33 
34  See \ref PixelFormats for information about pixel formats; with this module you can use the formats supported by the
35  camera sensor: YUYV, BAYER, RGB565, and resolutions:
36 
37  - SXGA (1280 x 1024): up to 15 fps
38  - VGA (640 x 480): up to 30 fps
39  - CIF (352 x 288): up to 60 fps
40  - QVGA (320 x 240): up to 60 fps
41  - QCIF (176 x 144): up to 120 fps
42  - QQVGA (160 x 120): up to 60 fps
43  - QQCIF (88 x 72): up to 120 fps
44 
45  Things to try
46  -------------
47 
48  Edit <b>JEVOIS:/config/videomappings.cfg</b> on your MicroSD card (see \ref VideoMapping) and try to add some new
49  pass-through mappings. Not all of the possible pass-through mappings have been included in the card to avoid having
50  too many of these simple "dumb camera" mappings in the base software distribution. For example, you can try
51 
52  \verbatim
53  YUYV 176 144 115.0 YUYV 176 144 115.0 JeVois PassThrough
54  \endverbatim
55 
56  will grab YUYV frames on the sensor, with resolution 176x144 at 115 frames/s, and will directly send them to the
57  host computer over the USB link. To test this mapping, select the corresponding resolution and framerate in your
58  video viewing software (here, YUYV 176x144 \@ 115fps). Although the sensor can capture at up to 120fps at this
59  resolution, here we used 115fps to avoid a conflict with a mapping using YUYV 176x144 \@ 120fps USB output and the
60  \jvmod{SaveVideo} module that is already in the default <b>videomappings.cfg</b> file.
61 
62  Note that this module may suffer from DMA coherency artifacts if the \p camturbo parameter of the jevois::Engine is
63  turned on, which it is by default. The \p camturbo parameter relaxes some of the cache coherency constraints on the
64  video buffers captured by the camera sensor, which allows the JeVois processor to access video pixel data from
65  memory faster. But with modules that do not do much processing, sometimes this yields video artifacts, we presume
66  because some of the video data from previous frames still is in the CPU cache and hence is not again fetched from
67  main memory by the CPU. If you see short stripes of what appears to be wrong pixel colors in the video, try to
68  disable \p camturbo, by editing <b>JEVOIS:/config/params.cfg</b> on your MicroSD card and in there turning \p
69  camturbo to false.
70 
71 
72  @author Laurent Itti
73 
74  @videomapping YUYV 1280 1024 7.5 YUYV 1280 1024 7.5 JeVois PassThrough
75  @videomapping YUYV 640 480 30.0 YUYV 640 480 30.0 JeVois PassThrough
76  @videomapping YUYV 640 480 19.6 YUYV 640 480 19.6 JeVois PassThrough
77  @videomapping YUYV 640 480 12.0 YUYV 640 480 12.0 JeVois PassThrough
78  @videomapping YUYV 640 480 8.3 YUYV 640 480 8.3 JeVois PassThrough
79  @videomapping YUYV 640 480 7.5 YUYV 640 480 7.5 JeVois PassThrough
80  @videomapping YUYV 640 480 5.5 YUYV 640 480 5.5 JeVois PassThrough
81  @email itti\@usc.edu
82  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
83  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
84  @mainurl http://jevois.org
85  @supporturl http://jevois.org/doc
86  @otherurl http://iLab.usc.edu
87  @license GPL v3
88  @distribution Unrestricted
89  @restrictions None
90  \ingroup modules */
92 {
93  public:
94  //! Default base class constructor ok
96 
97  //! Virtual destructor for safe inheritance
98  virtual ~PassThrough() { }
99 
100  //! Processing function
101  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
102  {
103  // Wait for next available camera image:
104  jevois::RawImage const inimg = inframe.get(true);
105 
106  // Wait for an image from our gadget driver into which we will put our results:
107  jevois::RawImage outimg = outframe.get();
108 
109  // Enforce that the input and output formats and image sizes match:
110  outimg.require("output", inimg.width, inimg.height, inimg.fmt);
111 
112  // Just copy the pixel data over:
113  memcpy(outimg.pixelsw<void>(), inimg.pixels<void>(), std::min(inimg.buf->length(), outimg.buf->length()));
114 
115  // Camera outputs RGB565 in big-endian, but most video grabbers expect little-endian:
116  if (outimg.fmt == V4L2_PIX_FMT_RGB565) jevois::rawimage::byteSwap(outimg);
117 
118  // Let camera know we are done processing the input image:
119  inframe.done(); // NOTE: optional here, inframe destructor would call it anyway
120 
121  // Send the output image with our processing results to the host over USB:
122  outframe.send(); // NOTE: optional here, outframe destructor would call it anyway
123  }
124 
125 #ifdef JEVOIS_PRO
126  //! Processing function with zero-copy and GUI on JeVois-Pro
127  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
128  {
129  // Start the frame: Internally, this initializes or resizes the display as needed, polls and handles events
130  // (inputs, window resize, etc), and clears the frame. Variables winw and winh are set by startFrame() to the
131  // current window size, and true is returned if no keyboard/mouse action in a while (can be used to hide any GUI
132  // elements when user is not interacting with JeVois):
133  unsigned short winw, winh;
134  bool idle = helper.startFrame(winw, winh);
135 
136  // Test mode: replace frame by checkerboard:
137  static bool testmode = false;
138  static bool halt = true;
139  static bool valt = false;
140  static bool noalias = true;
141  static bool showscaled = true;
142  static bool usedma = false;
143  static int scaledx = 0;
144  static int scaledy = 0;
145 
146  if (testmode)
147  {
148  jevois::RawImage const & img = inframe.get();
149  unsigned char * pix = (unsigned char *)img.pixels<unsigned char>();
150  int w = img.width, h = img.height;
151  bool oddv = false, oddh = false;
152 
153  switch (img.fmt)
154  {
155  case V4L2_PIX_FMT_RGB24:
156  {
157  for (int j = 0; j < h; ++j)
158  {
159  bool hh = valt ? oddv : halt;
160  oddh = hh ? false : oddv;
161  for (int i = 0; i < w; ++i)
162  {
163  if (oddh) { *pix++ = 0; *pix++ = 0; *pix++ = 0; }
164  else { *pix++ = 255; *pix++ = 255; *pix++ = 255; }
165  if (hh) oddh = !oddh;
166  }
167  oddv = !oddv;
168  }
169  }
170  break;
171  case V4L2_PIX_FMT_YUYV:
172  {
173  for (int j = 0; j < h; ++j)
174  {
175  bool hh = valt ? oddv : halt;
176  oddh = hh ? false : oddv;
177  for (int i = 0; i < w; ++i)
178  {
179  if (oddh) { *pix++ = 0; *pix++ = 0x80; }
180  else { *pix++ = 0xff; *pix++ = 0x80; }
181  if (hh) oddh = !oddh;
182  }
183  oddv = !oddv;
184  }
185  }
186  break;
187  case V4L2_PIX_FMT_RGB32:
188  {
189  for (int j = 0; j < h; ++j)
190  {
191  bool hh = valt ? oddv : halt;
192  oddh = hh ? false : oddv;
193  for (int i = 0; i < w; ++i)
194  {
195  if (oddh) { *pix++ = 0; *pix++ = 0; *pix++ = 0; *pix++ = 255; }
196  else { *pix++ = 255; *pix++ = 255; *pix++ = 255; *pix++ = 255; }
197  if (hh) oddh = !oddh;
198  }
199  oddv = !oddv;
200  }
201  }
202  break;
203  }
204  }
205 
206  jevois::RawImage const & imgdebug = inframe.get(); // to get dims only
207 
208  // In the PassThrough module, just draw the camera input frame, as large as we can, which is achieved by passing
209  // zero dims. The helper with compute the image position and size to make it as large as possible without changing
210  // aspect ratio, and size and position variables will be updated so we know what they are:
211  int x = 0, y = 0; unsigned short w = 0, h = 0;
212  if (usedma) helper.drawInputFrame("dc", inframe, x, y, w, h, noalias);
213  else
214  {
215  jevois::RawImage const & img = inframe.get();
216  helper.drawImage("c", img, x, y, w, h, noalias);
217  }
218  inframe.done();
219 
220  // If we have a second image from the ISP, display it:
221  if (inframe.hasScaledImage() && showscaled)
222  {
223  jevois::RawImage const & img2 = inframe.get2();
224  unsigned short w2 = img2.width, h2 = img2.height;
225  if (usedma) helper.drawInputFrame2("ds", inframe, scaledx , scaledy, w2, h2, noalias);
226  else helper.drawImage("s", img2, scaledx, scaledy, w2, h2, noalias);
227 
228  inframe.done2();
229  }
230 
231  if (idle == false)
232  {
233  // To draw things on top of input video but behind ImGui windows, use ImGui global background draw list:
234  auto dlb = ImGui::GetBackgroundDrawList(); // or use GetForegroundDrawList to draw in front of ImGui
235  ImVec2 const p = ImGui::GetMousePos();
236  dlb->AddRect(ImVec2(p.x-30, p.y-30), ImVec2(p.x+30, p.y+30), ImColor(255, 0, 0, 255) /* red */);
237 
238  // Just draw a simple ImGui window that shows fps if we are not idle:
239  ImGuiIO & io = ImGui::GetIO();
240  if (ImGui::Begin("JeVois-Pro PassThrough Module"))
241  {
242  ImGui::Text("Framerate: %3.2f fps", io.Framerate);
243  ImGui::Text("Video: raw %dx%d aspect=%f, render %dx%d @ %d,%d", imgdebug.width, imgdebug.height,
244  float(imgdebug.width) / float(imgdebug.height), w, h, x, y);
245  if (inframe.hasScaledImage())
246  {
247  ImGui::Checkbox("Show ISP scaled image", &showscaled);
248  ImGui::SliderInt("Scaled image x", &scaledx, 0, winw);
249  ImGui::SliderInt("Scaled image y", &scaledy, 0, winh);
250  }
251 
252  ImGui::Checkbox("Use DMABUF", &usedma);
253  ImGui::Checkbox("Test mode", &testmode);
254  if (testmode)
255  {
256  ImGui::Checkbox("Pattern 1", &halt);
257  ImGui::Checkbox("Pattern 2", &valt);
258  ImGui::Checkbox("No aliasing", &noalias);
259  }
260  }
261  ImGui::End();
262 
263  // To draw things on top of input video and on top of ImGui windows, use ImGui global foregound draw list:
264  auto dlf = ImGui::GetForegroundDrawList();
265  dlf->AddCircle(ImVec2(p.x, p.y), 20, ImColor(0, 255, 0, 128) /* semi transparent green */);
266  }
267 
268  // Render the image and GUI:
269  helper.endFrame();
270  }
271 #endif // JEVOIS_PRO
272 };
273 
274 // Allow the module to be loaded as a shared object (.so) file:
jevois::GUIhelper::drawImage
void drawImage(char const *name, RawImage const &img, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool isoverlay=false)
jevois::GUIhelper::drawInputFrame2
void drawInputFrame2(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::pixels
const T * pixels() const
jevois::OutputFrame
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
Module.H
jevois::GUIhelper
jevois::GUIhelper::endFrame
void endFrame()
jevois::RawImage
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(PassThrough)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::RawImage::width
unsigned int width
jevois::RawImage::pixelsw
T * pixelsw()
jevois::Module
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
PassThrough::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: PassThrough.C:101
PassThrough
Simple module that just passes the captured camera frames through to USB host.
Definition: PassThrough.C:91
jevois::InputFrame
jevois::RawImage::fmt
unsigned int fmt
jevois::RawImage::buf
std::shared_ptr< VideoBuf > buf
h
int h
PassThrough::~PassThrough
virtual ~PassThrough()
Virtual destructor for safe inheritance.
Definition: PassThrough.C:98
jevois::rawimage::byteSwap
void byteSwap(RawImage &img)
jevois::Component::Module
friend friend class Module