JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
FilterGPU.H
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #pragma once
19 
22 
25 
26 #include <map>
27 
28 //! Simple image filtering using OpenGL-ES on the GPU
29 /*! This class first copies the given input image into an OpenGL texture, then applies OpenGL-ES vertex and fragment
30  shaders to achieve some image processing, and finally gets the resulting pixels back into an image.
31 
32  We support two types of input:
33 
34  - CV_8UC1 greyscale image. This is uploaded to GPU as a luminance texture. In the shaders, R=G=B and alpha=1.
35  - CV-8UC4 color images. This is uploaded to GPU as an RGBA image. In the shaders, R,G,B,A are as in the input.
36 
37  We support two types of output:
38 
39  - CV_8UC2 RGB565 images. This is useful to avoid any conversion of color results. GPU results are directly rendered
40  into an RGB565 color buffer, which may directly be the buffer of an RGB565 output RawImage, i.e., the pixel memory
41  was allocated and "belongs to" the USB gadget driver. Thus, the rendered pixels can be directly read back from the
42  GPU into the RawImage and send over USB with no additional copy or conversion. See DemoGPU for an example of this.
43  - CV-8UC4 RGBA images. This may be used either for higher-accuracy color output, but may require conversion since we
44  do not support sending RGBA images over the USB link. Using jevois::rawimage::unpackCvRGBAtoGrayRawImage() one may
45  unpack such a packed RGBA image into 4 single-channel planes. This then makes it possible to run one RGBA shader
46  that produces 4 different filtering effects in each of the 4 color/alpha channels, and to then unpack the results
47  into a 4-up greyscale RawImage buffer for sending over USB. DemoCPUGPU uses this, uploading a greyscale input
48  image and then running 4 different filters to create the R, G, B, and A outputs, finally unpacking those into 4
49  greyscale images sent over USB.
50 
51  Shader conventions:
52 
53  - input texture is in "tex"
54  - current coordinates within texture are in "tcoord"
55  - size of one texture pixel is set in "texelsize" (uniform 2 float)
56  - see DemoGPU and DemoCPUGPU for example shaders
57 
58  This code is inspired from the tutorial and code examples found on this page:
59  http://robotblogging.blogspot.com/2013/10/gpu-accelerated-camera-processing-on.html
60 
61  on host, mesa may be in the way of other drivers, eg nvidia. You need to get them out.
62  \code
63  sudo find / -name libEGL*
64  sudo find / -name libGLESv2*
65 
66  cd /usr/lib/x86_64-linux-gnu
67  ls -l | grep mesa # we will fix those with what we found above
68  sudo ln -fs /usr/lib/nvidia-352-updates/libEGL.so libEGL.so
69  sudo ln -fs /usr/lib/nvidia-352-updates/libGLESv2.so libGLESv2.so
70  sudo ln -fs /usr/lib/nvidia-352-updates/libGL.so libGL.so
71  sudo mv mesa-egl/ld.so.conf mesa-egl/ld.so.conf.disabled
72  sudo ldconfig
73 
74  # back to jevoisbase:
75  ldd hbuild/libjevoisbase.so |grep GL # there should not be any mesa in there
76  \endcode
77 
78  FIXME: yet, this code still does not run on the host. Maybe OpenCV which is using highgui on the host which uses Qt
79  which uses OpenGL (not the ES version) is the issue. \ingroup components */
81 {
82  public:
83  //! Constructor
84  FilterGPU(std::string const & instance);
85 
86  //! Destructor
87  ~FilterGPU();
88 
89  //! Set an image processing program
90  void setProgram(std::string const & vertex_shader, std::string const & frag_shader);
91 
92  //! Set some program parameters, 2 float version
93  /*! OpenGL is not very thread-friendly. Hence the program chosen using setProgram() will actually be instantiated
94  inside the process() function so everything happens in one thread. Here, users can set program parameters, those
95  are just memorized and will be set in process() as well. Each time setProgram() is called, all cached parameters
96  are cleared and they should be set again. */
97  void setProgramParam2f(std::string name, float val1, float val2);
98 
99  //! Set some program parameters, 1 float version
100  void setProgramParam1f(std::string name, float val);
101 
102  //! Set some program parameters, 2 int version
103  void setProgramParam2i(std::string name, int val1, int val2);
104 
105  //! Set some program parameters, 1 int version
106  void setProgramParam1i(std::string name, int val);
107 
108  //! Process an image. The dst image should be allocated with correct image size and pixel type
109  void process(cv::Mat const & src, cv::Mat & dst);
110 
111  private:
112  void initDisplay(); // OpenGL is not thread-safe, so we init in the process() thread rather than at construction
113  std::string itsVshader, itsFshader;
114  volatile bool itsProgramChanged;
115 
116  std::shared_ptr<GPUtexture> itsSrcTex;
117  std::shared_ptr<GPUprogram> itsProgram;
118  GLuint itsQuadVertexBuffer;
119  EGLDisplay itsDisplay;
120  EGLConfig itsConfig;
121  EGLContext itsContext;
122  EGLSurface itsSurface;
123  GLuint itsFramebufferId; // output frame buffer
124  GLuint itsRenderbufferId; // output render buffer
125  GLuint itsRenderWidth, itsRenderHeight;
126  int itsRenderType;
127  enum paramtype { F2, I2, F1, I1 };
128  struct param { paramtype type; float val[2]; };
129 
130  std::map<std::string, param> itsProgramParams;
131 
132  mutable std::mutex itsMutex;
133 };
134 
std::string name
void process(cv::Mat const &src, cv::Mat &dst)
Process an image. The dst image should be allocated with correct image size and pixel type...
Definition: FilterGPU.C:125
void setProgramParam2f(std::string name, float val1, float val2)
Set some program parameters, 2 float version.
Definition: FilterGPU.C:93
void setProgram(std::string const &vertex_shader, std::string const &frag_shader)
Set an image processing program.
Definition: FilterGPU.C:81
FilterGPU(std::string const &instance)
Constructor.
Definition: FilterGPU.C:22
void setProgramParam1f(std::string name, float val)
Set some program parameters, 1 float version.
Definition: FilterGPU.C:101
boost::shared_mutex itsMutex
void setProgramParam2i(std::string name, int val1, int val2)
Set some program parameters, 2 int version.
Definition: FilterGPU.C:109
Simple image filtering using OpenGL-ES on the GPU.
Definition: FilterGPU.H:80
~FilterGPU()
Destructor.
Definition: FilterGPU.C:63
void setProgramParam1i(std::string name, int val)
Set some program parameters, 1 int version.
Definition: FilterGPU.C:117