JeVoisBase  1.20
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoCPUGPU.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Types/Enum.H>
20 
24 
26 #include <linux/videodev2.h>
27 
28 #include <opencv2/imgproc/imgproc.hpp>
29 #include <future>
30 
31 // icon by Freepik in computer at flaticon
32 
33 //! Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-400MP2 GPU
34 /*! This module computes saliency and gist over our 4 CPU cores while also computing 4 different image filters over the
35  GPU, finally combining all results into a single grayscale image:
36 
37  - saliency: multicore CPU-based detection of the most conspicuous (most attention-grabbing) object in the field of
38  view.
39  - GPU filter 1: Sobel edge detector
40  - GPU filter 2: Median filter
41  - GPU filter 3: Morphological erosion filter
42  - GPU filter 4: Morphological dilation filter
43 
44  For an introduction to visual saliency, see http://ilab.usc.edu/bu/
45 
46  Also see \jvmod{DemoSaliency}, \jvmod{JeVoisIntro}, \jvmod{DarknetSaliency} for more about saliency.
47 
48  Video output
49  ------------
50 
51  The video output is arranged vertically, with, from top to bottom:
52  - Sobel filter results (same size as input image)
53  - Median filter results (same size as input image)
54  - Morphological erosion filter results (same size as input image)
55  - Morphological dilation filter results (same size as input image)
56  - Saliency results (those are very small): from left to right: saliency map, color map, intensity map, orientation
57  map, flicker map, motion map. Map size is input size divided by 8 horizontally and vertically. Gist vector is
58  appended to the right but note that at 160x120 it is truncated (see source code for details).
59 
60  Serial Messages
61  ---------------
62 
63  This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
64  sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
65  filtered attended (most salient) location:
66 
67  - Serial message type: \b 2D
68  - `id`: always \b sm (shorthand for saliency map)
69  - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
70  - `w`, `h`: always 0, 0
71  - `extra`: none (empty string)
72 
73  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
74  standardized coordinates.
75 
76  @author Laurent Itti
77 
78  @displayname Demo CPU GPU
79  @videomapping GREY 160 495 60.0 YUYV 160 120 60.0 JeVois DemoCPUGPU
80  @email itti\@usc.edu
81  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
82  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
83  @mainurl http://jevois.org
84  @supporturl http://jevois.org/doc
85  @otherurl http://iLab.usc.edu
86  @license GPL v3
87  @distribution Unrestricted
88  @restrictions None
89  \ingroup modules */
91 {
92  public:
93  //! Constructor
94  DemoCPUGPU(std::string const & instance) : jevois::StdModule(instance)
95  {
96  itsFilter = addSubComponent<FilterGPU>("gpu");
97  itsSaliency = addSubComponent<Saliency>("saliency");
98  itsKF = addSubComponent<Kalman2D>("kalman");
99 
100  // Use some fairly large saliency and feature maps so we can see them:
101  itsSaliency->centermin::set(1);
102  itsSaliency->smscale::set(3);
103  }
104 
105  //! Virtual destructor for safe inheritance
106  virtual ~DemoCPUGPU() { }
107 
108  //! Set our GPU program after we are fully constructed and our Component path has been set
109  void postInit() override
110  {
111  itsFilter->setProgram("shaders/simplevertshader.glsl", "shaders/combofragshader.glsl");
112  itsFilter->setProgramParam2f("offset", -1.0F, -1.0F);
113  itsFilter->setProgramParam2f("scale", 2.0F, 2.0F);
114  }
115 
116  //! Processing function
117  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
118  {
119  // Wait for next available camera image:
120  jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
121  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
122 
123  // In this demo, the GPU completes earlier than the CPU. Hence, we are going to wait for the output frame in the
124  // main thread (which runs the GPU code). We use a mutex to signal to the saliency thread when the output image is
125  // available. Using a mutex and unique_lock here ensures that we will be exception-safe (which may be trickier
126  // with a condition variable or such):
127  jevois::RawImage outimg;
128  std::unique_lock<std::mutex> lck(itsOutMtx); // mutex will be released by main thread when outimg is available
129 
130  // Launch the saliency computation in a thread:
131  auto sal_fut = jevois::async([&](){
132  // Compute saliency and gist:
133  itsSaliency->process(inimg, true);
134 
135  // Find most salient point:
136  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
137 
138  // Compute attended location in original frame coordinates:
139  int const smlev = itsSaliency->smscale::get();
140  int const smadj = smlev > 0 ? (1 << (smlev-1)) : 0; // half a saliency map pixel adjustment
141  unsigned int const dmx = (mx << smlev) + smadj;
142  unsigned int const dmy = (my << smlev) + smadj;
143  unsigned int const mapw = itsSaliency->salmap.dims.w, maph = itsSaliency->salmap.dims.h;
144  //unsigned int const gistsize = itsSaliency->gist_size;
145  unsigned int const gistw = w - 6 * mapw; // width for gist block
146  //unsigned int const gisth = (gistsize + gistw - 1) / gistw; // divide gist_size by w and round up
147 
148  // Filter over time the salient location coordinates:
149  itsKF->set(dmx, dmy, w, h);
150  float kfxraw, kfyraw; itsKF->get(kfxraw, kfyraw, 1.0F); // round to int for serial
151 
152  // Send kalman-filtered most-salient-point info to serial port (for arduino, etc):
153  sendSerialStd2D(kfxraw, kfyraw, 0.0F, 0.0F, "sm");
154 
155  // Wait for output image to be available:
156  std::lock_guard<std::mutex> _(itsOutMtx);
157 
158  // Paste saliency and feature maps:
159  unsigned int offset = 0;
160  pasteGrayMap(outimg, itsSaliency->salmap, offset, h*4, 20);
161  pasteGrayMap(outimg, itsSaliency->color, offset, h*4, 18);
162  pasteGrayMap(outimg, itsSaliency->intens, offset, h*4, 18);
163  pasteGrayMap(outimg, itsSaliency->ori, offset, h*4, 18);
164  pasteGrayMap(outimg, itsSaliency->flicker, offset, h*4, 18);
165  pasteGrayMap(outimg, itsSaliency->motion, offset, h*4, 18);
166 
167  // Paste gist. Note: at 160x120 we will only end up pasting the first 600 gist entries. This code may fail at
168  // different resolutions, beware:
169  unsigned char * d = outimg.pixelsw<unsigned char>() + 4*w*h + 6*mapw;
170  for (unsigned int i = 0; i < maph; ++i) memcpy(d + i*w, itsSaliency->gist + i*gistw, gistw);
171  });
172 
173  // Convert input image to grayscale:
174  cv::Mat grayimg = jevois::rawimage::convertToCvGray(inimg);
175 
176  // Once saliency is done using the input image, let camera know we are done with it:
177  itsSaliency->waitUntilDoneWithInput();
178  inframe.done();
179 
180  // Upload the greyscale image to GPU and apply 4 grayscale GPU filters, storing the 4 results into an RGBA image:
181  cv::Mat gpuout(h, w, CV_8UC4);
182  itsFilter->process(grayimg, gpuout);
183 
184  // Wait for an image from our gadget driver into which we will put our results:
185  outimg = outframe.get();
186  lck.unlock(); // let saliency thread proceed with using the output image
187  outimg.require("output", w, h * 4 + (h >> itsSaliency->smscale::get()), V4L2_PIX_FMT_GREY);
188 
189  // Unpack the GPU output into 4 gray images, starting from top-left of the output image:
191 
192  // Wait until saliency computation is complete:
193  sal_fut.get();
194 
195  // Send the output image with our processing results to the host over USB:
196  outframe.send();
197  }
198 
199  // ####################################################################################################
200  //! Paste a map and add its width to the dx offset
201  /*! Beware this is for a gray outimg only. */
202  void pasteGrayMap(jevois::RawImage & outimg, env_image const & fmap, unsigned int & dx, unsigned int dy,
203  unsigned int bitshift)
204  {
205  env_size_t const fw = fmap.dims.w, fh = fmap.dims.h;
206  unsigned int const ow = outimg.width, oh = outimg.height;
207 
208  if (dy + fh > oh) LFATAL("Map would extend past output image bottom");
209  if (fw + dx > ow) LFATAL("Map would extend past output image width");
210 
211  unsigned int const stride = ow - fw;
212 
213  intg32 * s = fmap.pixels; unsigned char * d = outimg.pixelsw<unsigned char>() + dx + dy * ow;
214 
215  for (unsigned int j = 0; j < fh; ++j)
216  {
217  for (unsigned int i = 0; i < fw; ++i)
218  {
219  intg32 v = (*s++) >> bitshift; if (v > 255) v = 255;
220  *d++ = (unsigned char)(v);
221  }
222  d += stride;
223  }
224 
225  // Add the map width to the dx offset:
226  dx += fw;
227  }
228 
229  private:
230  std::shared_ptr<FilterGPU> itsFilter;
231  std::shared_ptr<Saliency> itsSaliency;
232  std::shared_ptr<Kalman2D> itsKF;
233  std::mutex itsOutMtx;
234 };
235 
236 // Allow the module to be loaded as a shared object (.so) file:
env_dims::w
env_size_t w
The width.
Definition: env_types.h:82
jevois::OutputFrame
DemoCPUGPU::~DemoCPUGPU
virtual ~DemoCPUGPU()
Virtual destructor for safe inheritance.
Definition: DemoCPUGPU.C:106
DemoCPUGPU::DemoCPUGPU
DemoCPUGPU(std::string const &instance)
Constructor.
Definition: DemoCPUGPU.C:94
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
Module.H
DemoCPUGPU
Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-...
Definition: DemoCPUGPU.C:90
env_size_t
unsigned long env_size_t
Definition: env_types.h:71
DemoCPUGPU::postInit
void postInit() override
Set our GPU program after we are fully constructed and our Component path has been set.
Definition: DemoCPUGPU.C:109
jevois::RawImage
jevois::rawimage::convertToCvGray
cv::Mat convertToCvGray(RawImage const &src)
jevois::StdModule::sendSerialStd2D
void sendSerialStd2D(float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
FilterGPU.H
jevois::RawImage::width
unsigned int width
jevois::RawImage::pixelsw
T * pixelsw()
jevois
F
float F
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DemoCPUGPU)
env_dims::h
env_size_t h
The height.
Definition: env_types.h:83
Enum.H
env_image::dims
struct env_dims dims
Definition: env_image.h:45
Kalman2D.H
DemoCPUGPU::pasteGrayMap
void pasteGrayMap(jevois::RawImage &outimg, env_image const &fmap, unsigned int &dx, unsigned int dy, unsigned int bitshift)
Paste a map and add its width to the dx offset.
Definition: DemoCPUGPU.C:202
jevois::StdModule::StdModule
StdModule(std::string const &instance)
LFATAL
#define LFATAL(msg)
jevois::rawimage::unpackCvRGBAtoGrayRawImage
void unpackCvRGBAtoGrayRawImage(cv::Mat const &src, RawImage &dst)
RawImageOps.H
jevois::RawImage::height
unsigned int height
env_image::pixels
intg32 * pixels
Definition: env_image.h:46
jevois::InputFrame
h
int h
DemoCPUGPU::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: DemoCPUGPU.C:117
env_image
Basic image class.
Definition: env_image.h:43
jevois::StdModule
Saliency.H
demo.w
w
Definition: demo.py:85
intg32
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52