JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoCPUGPU.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Types/Enum.H>
20 
24 
26 #include <linux/videodev2.h>
27 
28 #include <opencv2/imgproc/imgproc.hpp>
29 #include <future>
30 
31 // icon by Freepik in computer at flaticon
32 
33 //! Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-400MP2 GPU
34 /*! This module computes saliency and gist over our 4 CPU cores while also computing 4 different image filters over the
35  GPU, finally combining all results into a single grayscale image:
36 
37  - saliency: multicore CPU-based detection of the most conspicuous (most attention-grabbing) object in the field of
38  view.
39  - GPU filter 1: Sobel edge detector
40  - GPU filter 2: Median filter
41  - GPU filter 3: Morphological erosion filter
42  - GPU filter 4: Morphological dilation filter
43 
44  For an introduction to visual saliency, see http://ilab.usc.edu/bu/
45 
46  Also see \jvmod{DemoSaliency}, \jvmod{JeVoisIntro}, \jvmod{DarknetSaliency} for more about saliency.
47 
48  Video output
49  ------------
50 
51  The video output is arranged vertically, with, from top to bottom:
52  - Sobel filter results (same size as input image)
53  - Median filter results (same size as input image)
54  - Morphological erosion filter results (same size as input image)
55  - Morphological dilation filter results (same size as input image)
56  - Saliency results (those are very small): from left to right: saliency map, color map, intensity map, orientation
57  map, flicker map, motion map. Map size is input size divided by 8 horizontally and vertically. Gist vector is
58  appended to the right but note that at 160x120 it is truncated (see source code for details).
59 
60  Serial Messages
61  ---------------
62 
63  This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
64  sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
65  filtered attended (most salient) location:
66 
67  - Serial message type: \b 2D
68  - `id`: always \b sm (shorthand for saliency map)
69  - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
70  - `w`, `h`: always 0, 0
71  - `extra`: none (empty string)
72 
73 
74  @author Laurent Itti
75 
76  @displayname Demo CPU GPU
77  @videomapping GREY 160 495 60.0 YUYV 160 120 60.0 JeVois DemoCPUGPU
78  @email itti\@usc.edu
79  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
80  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
81  @mainurl http://jevois.org
82  @supporturl http://jevois.org/doc
83  @otherurl http://iLab.usc.edu
84  @license GPL v3
85  @distribution Unrestricted
86  @restrictions None
87  \ingroup modules */
89 {
90  public:
91  //! Constructor
92  DemoCPUGPU(std::string const & instance) : jevois::StdModule(instance)
93  {
94  itsFilter = addSubComponent<FilterGPU>("gpu");
95  itsSaliency = addSubComponent<Saliency>("saliency");
96  itsKF = addSubComponent<Kalman2D>("kalman");
97 
98  // Use some fairly large saliency and feature maps so we can see them:
99  itsSaliency->centermin::set(1);
100  itsSaliency->smscale::set(3);
101  }
102 
103  //! Virtual destructor for safe inheritance
104  virtual ~DemoCPUGPU() { }
105 
106  //! Set our GPU program after we are fully constructed and our Component path has been set
107  void postInit() override
108  {
109  itsFilter->setProgram("shaders/simplevertshader.glsl", "shaders/combofragshader.glsl");
110  itsFilter->setProgramParam2f("offset", -1.0F, -1.0F);
111  itsFilter->setProgramParam2f("scale", 2.0F, 2.0F);
112  }
113 
114  //! Processing function
115  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
116  {
117  // Wait for next available camera image:
118  jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
119  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
120 
121  // In this demo, the GPU completes earlier than the CPU. Hence, we are going to wait for the output frame in the
122  // main thread (which runs the GPU code). We use a mutex to signal to the saliency thread when the output image is
123  // available. Using a mutex and unique_lock here ensures that we will be exception-safe (which may be trickier
124  // with a condition variable or such):
125  jevois::RawImage outimg;
126  std::unique_lock<std::mutex> lck(itsOutMtx); // mutex will be released by main thread when outimg is available
127 
128  // Launch the saliency computation in a thread:
129  auto sal_fut = std::async(std::launch::async, [&](){
130  // Compute saliency and gist:
131  itsSaliency->process(inimg, true);
132 
133  // Find most salient point:
134  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
135 
136  // Compute attended location in original frame coordinates:
137  int const smlev = itsSaliency->smscale::get();
138  int const smadj = smlev > 0 ? (1 << (smlev-1)) : 0; // half a saliency map pixel adjustment
139  unsigned int const dmx = (mx << smlev) + smadj;
140  unsigned int const dmy = (my << smlev) + smadj;
141  unsigned int const mapw = itsSaliency->salmap.dims.w, maph = itsSaliency->salmap.dims.h;
142  unsigned int const gistsize = itsSaliency->gist_size;
143  unsigned int const gistw = w - 6 * mapw; // width for gist block
144  unsigned int const gisth = (gistsize + gistw - 1) / gistw; // divide gist_size by w and round up
145 
146  // Filter over time the salient location coordinates:
147  itsKF->set(dmx, dmy, w, h);
148  float kfxraw, kfyraw; itsKF->get(kfxraw, kfyraw, 1.0F); // round to int for serial
149 
150  // Send kalman-filtered most-salient-point info to serial port (for arduino, etc):
151  sendSerialStd2D(kfxraw, kfyraw, 0.0F, 0.0F, "sm");
152 
153  // Wait for output image to be available:
154  std::lock_guard<std::mutex> _(itsOutMtx);
155 
156  // Paste saliency and feature maps:
157  unsigned int offset = 0;
158  pasteGrayMap(outimg, itsSaliency->salmap, offset, h*4, 20);
159  pasteGrayMap(outimg, itsSaliency->color, offset, h*4, 18);
160  pasteGrayMap(outimg, itsSaliency->intens, offset, h*4, 18);
161  pasteGrayMap(outimg, itsSaliency->ori, offset, h*4, 18);
162  pasteGrayMap(outimg, itsSaliency->flicker, offset, h*4, 18);
163  pasteGrayMap(outimg, itsSaliency->motion, offset, h*4, 18);
164 
165  // Paste gist. Note: at 160x120 we will only end up pasting the first 600 gist entries. This code may fail at
166  // different resolutions, beware:
167  unsigned char * d = outimg.pixelsw<unsigned char>() + 4*w*h + 6*mapw;
168  for (int i = 0; i < maph; ++i) memcpy(d + i*w, itsSaliency->gist + i*gistw, gistw);
169  });
170 
171  // Convert input image to grayscale:
172  cv::Mat grayimg = jevois::rawimage::convertToCvGray(inimg);
173 
174  // Once saliency is done using the input image, let camera know we are done with it:
175  itsSaliency->waitUntilDoneWithInput();
176  inframe.done();
177 
178  // Upload the greyscale image to GPU and apply 4 grayscale GPU filters, storing the 4 results into an RGBA image:
179  cv::Mat gpuout(h, w, CV_8UC4);
180  itsFilter->process(grayimg, gpuout);
181 
182  // Wait for an image from our gadget driver into which we will put our results:
183  outimg = outframe.get();
184  lck.unlock(); // let saliency thread proceed with using the output image
185  outimg.require("output", w, h * 4 + (h >> itsSaliency->smscale::get()), V4L2_PIX_FMT_GREY);
186 
187  // Unpack the GPU output into 4 gray images, starting from top-left of the output image:
189 
190  // Wait until saliency computation is complete:
191  sal_fut.get();
192 
193  // Send the output image with our processing results to the host over USB:
194  outframe.send();
195  }
196 
197  // ####################################################################################################
198  //! Paste a map and add its width to the dx offset
199  /*! Beware this is for a gray outimg only. */
200  void pasteGrayMap(jevois::RawImage & outimg, env_image const & fmap, unsigned int & dx, unsigned int dy,
201  unsigned int bitshift)
202  {
203  env_size_t const fw = fmap.dims.w, fh = fmap.dims.h;
204  unsigned int const ow = outimg.width, oh = outimg.height;
205 
206  if (dy + fh > oh) LFATAL("Map would extend past output image bottom");
207  if (fw + dx > ow) LFATAL("Map would extend past output image width");
208 
209  unsigned int const stride = ow - fw;
210 
211  intg32 * s = fmap.pixels; unsigned char * d = outimg.pixelsw<unsigned char>() + dx + dy * ow;
212 
213  for (unsigned int j = 0; j < fh; ++j)
214  {
215  for (unsigned int i = 0; i < fw; ++i)
216  {
217  intg32 v = (*s++) >> bitshift; if (v > 255) v = 255;
218  *d++ = (unsigned char)(v);
219  }
220  d += stride;
221  }
222 
223  // Add the map width to the dx offset:
224  dx += fw;
225  }
226 
227  private:
228  std::shared_ptr<FilterGPU> itsFilter;
229  std::shared_ptr<Saliency> itsSaliency;
230  std::shared_ptr<Kalman2D> itsKF;
231  std::mutex itsOutMtx;
232 };
233 
234 // Allow the module to be loaded as a shared object (.so) file:
Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-...
Definition: DemoCPUGPU.C:88
env_size_t w
The width.
Definition: env_types.h:82
unsigned int height
JEVOIS_REGISTER_MODULE(DemoCPUGPU)
Basic image class.
Definition: env_image.h:43
StdModule(std::string const &instance)
struct env_dims dims
Definition: env_image.h:45
env_size_t h
The height.
Definition: env_types.h:83
void sendSerialStd2D(float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
unsigned long env_size_t
Definition: env_types.h:71
DemoCPUGPU(std::string const &instance)
Constructor.
Definition: DemoCPUGPU.C:92
cv::Mat convertToCvGray(RawImage const &src)
intg32 * pixels
Definition: env_image.h:46
#define LFATAL(msg)
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoCPUGPU.C:115
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
virtual ~DemoCPUGPU()
Virtual destructor for safe inheritance.
Definition: DemoCPUGPU.C:104
void unpackCvRGBAtoGrayRawImage(cv::Mat const &src, RawImage &dst)
void postInit() override
Set our GPU program after we are fully constructed and our Component path has been set...
Definition: DemoCPUGPU.C:107
void pasteGrayMap(jevois::RawImage &outimg, env_image const &fmap, unsigned int &dx, unsigned int dy, unsigned int bitshift)
Paste a map and add its width to the dx offset.
Definition: DemoCPUGPU.C:200
unsigned int width
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const