JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
BurnTest.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 
20 #include <jevois/Debug/Log.H>
21 #include <jevois/Debug/Timer.H>
26 
27 #include <opencv2/core/core.hpp>
28 #include <opencv2/imgproc/imgproc.hpp>
29 
30 #include <future>
31 #include <linux/videodev2.h> // for v4l2 pixel types
32 #include <stdlib.h>
33 
34 // Neon-related:
35 #include <Ne10/inc/NE10_imgproc.h>
36 
37 // GPU-related:
39 
40 // icon by Vectors Market in nature at www.flaticon.com
41 
42 //! This is a burn test: run the quad-core saliency demo while also loading up CPU, GPU and NEON in the background
43 /*! This burn test exercises all aspects of your JeVois smart camera to the maximum, namely:
44 
45  - launch two instances of whetstone (floating point benchmark test) running in the background
46  - launch two instances of dhrystone (integer benchmark test) running in the background
47  - grab frames from the camera sensor
48  - run the quad-core visual attention algorithm
49  - in parallel, run the NEON demo that blurs the video frames using NEON accelerated processor instructions
50  - in parallel, run the GPU demo that processes the video through 4 image filters (shaders)
51  - stream attention video results over USB
52  - issue messages over the serial port
53 
54  This burn test is useful to test JeVois hardware for any malfunction. It should run forever without crashing on
55  JeVois hardware. Demo display layout and markings are the same as for the DemoSaliency module.
56 
57  This burn test is one of the tests that every JeVois camera produced is tested with at the factory, before the unit
58  is shipped out.
59 
60  Things to try
61  -------------
62 
63  Select the burntest video mode (note that it is 640x300 \@ 10fps, while the default MicroSD card also includes a
64  mode with 640x300 \@ 60fps that runs the DemoSaliency module instead). You need to activate it (remove the leading
65  \b # sign) in <b>JEVOIS:/config/videomappings.cfg</b> as it is disabled by default. Observe the CPU temperature at
66  the bottom of the live video window. If it ever reaches 75C (which it should not under normal conditions given the
67  high power fan on the JeVois smart camera), the CPU frequency shown next to the temperature will drop down below
68  1344 MHz, and will then come back up as the CPU temperature drops below 75C.
69 
70  Connect your JeVois camera to your host computer through a USB Tester device that measures voltage, current, and
71  power. You should reach about 3.7 Watts under the burn test, which is the maximum we have ever been able to achieve
72  with a JeVois unit.
73 
74 
75  @author Laurent Itti
76 
77  @videomapping YUYV 640 300 10.0 YUYV 320 240 10.0 JeVois BurnTest
78  @email itti\@usc.edu
79  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
80  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
81  @mainurl http://jevois.org
82  @supporturl http://jevois.org/doc
83  @otherurl http://iLab.usc.edu
84  @license GPL v3
85  @distribution Unrestricted
86  @restrictions None
87  \ingroup modules */
88 class BurnTest : public jevois::Module
89 {
90  public:
91  //! Constructor
92  BurnTest(std::string const & instance) :
93  jevois::Module(instance), itsTimer("BurnTest", 30, LOG_DEBUG)
94  {
95  itsSaliency = addSubComponent<Saliency>("saliency");
96  itsKF = addSubComponent<Kalman2D>("kalman");
97  itsFilter = addSubComponent<FilterGPU>("gpu");
98  }
99 
100  //! Set our GPU program after we are fully constructed and our Component path has been set
101  void postInit() override
102  {
103  itsFilter->setProgram("shaders/simplevertshader.glsl", "shaders/combofragshader.glsl");
104  itsFilter->setProgramParam2f("offset", -1.0F, -1.0F);
105  itsFilter->setProgramParam2f("scale", 2.0F, 2.0F);
106  itsRunning.store(true);
107  }
108 
109  //! Kill our external processes on uninit
110  void postUninit() override
111  {
112  itsRunning.store(false);
113  system("/bin/rm /tmp/jevois-burntest");
114  system("killall -9 dhrystone");
115  system("killall -9 whetstone");
116  try { itsGPUfut.get(); } catch (...) { }
117  try { itsNEONfut.get(); } catch (...) { }
118  }
119 
120  //! Virtual destructor for safe inheritance
121  virtual ~BurnTest() { }
122 
123  //! Processing function
124  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
125  {
126  // Wait for next available camera image:
127  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
128  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
129 
130  itsTimer.start();
131 
132  // Check whether the input image size is small, in which case we will scale the maps up one notch for the purposes
133  // of this demo:
134  if (w < 170) { itsSaliency->centermin::set(1); itsSaliency->smscale::set(3); }
135  else { itsSaliency->centermin::set(2); itsSaliency->smscale::set(4); }
136 
137  // Launch the saliency computation in a thread:
138  auto sal_fut = std::async(std::launch::async, [&](){ itsSaliency->process(inimg, true); });
139 
140  // While computing, wait for an image from our gadget driver into which we will put our results:
141  jevois::RawImage outimg = outframe.get();
142 
143  // Paste the original image to the top-left corner of the display:
144  jevois::rawimage::paste(inimg, outimg, 0, 0);
145  jevois::rawimage::writeText(outimg, "JeVois CPU+GPU+NEON BurnTest", 3, 3, jevois::yuyv::White);
146 
147  // Once saliency is done using the input image, let camera know we are done with it:
148  itsSaliency->waitUntilDoneWithInput();
149  inframe.done();
150 
151  // If our grayimg is empty, compute it and launch the GPU and NEON threads:
152  if (itsGrayImg.empty())
153  {
154  // Create a temp file:
155  system("touch /tmp/jevois-burntest");
156 
157  // start a couple of whetstones:
158  system("( while [ -f /tmp/jevois-burntest ]; do whetstone 2000000000; done ) &");
159  system("( while [ -f /tmp/jevois-burntest ]; do whetstone 2000000000; done ) &");
160 
161  // dhrystne too
162  system("( while [ -f /tmp/jevois-burntest ]; do dhrystone 2000000000; done ) &");
163  system("( while [ -f /tmp/jevois-burntest ]; do dhrystone 2000000000; done ) &");
164 
165  // then load the GPU
167  itsGPUfut = std::async(std::launch::async, [&](unsigned int ww, unsigned int hh) {
168  cv::Mat gpuout(hh, ww, CV_8UC4);
169  while (itsRunning.load()) itsFilter->process(itsGrayImg, gpuout);
170  }, w, h);
171 
172  // and load NEON too
174 
175  itsNEONfut = std::async(std::launch::async, [&](unsigned int ww, unsigned int hh) {
176  cv::Mat neonresult(hh, ww, CV_8UC4);
177  ne10_size_t src_size { ww, hh }, kernel_size { 5, 5 };
178  while (itsRunning.load())
179  {
180 #ifdef __ARM_NEON__
181  // Neon version:
182  ne10_img_boxfilter_rgba8888_neon(itsRGBAimg.data, neonresult.data, src_size, ww * 4, ww * 4, kernel_size);
183 #else
184  // On non-ARM/NEON host, revert to CPU version again:
185  ne10_img_boxfilter_rgba8888_c(itsRGBAimg.data, neonresult.data, src_size, ww * 4, ww * 4, kernel_size);
186 #endif
187  }
188  }, w, h);
189  }
190 
191  // Wait until saliency computation is complete:
192  sal_fut.get();
193 
194  // Get some info from the saliency computation:
195  int const smlev = itsSaliency->smscale::get();
196  int const smfac = (1 << smlev);
197  int const roihw = (smfac * 3) / 2; // roi half width and height
198  int const mapdrawfac = smfac / 4; // factor by which we enlarge the feature maps for drawing
199  int const mapdw = (w >> smlev) * mapdrawfac; // width of the drawn feature maps
200  int const mapdh = (h >> smlev) * mapdrawfac; // height of the drawn feature maps
201 
202  // Enforce the correct output image size and format:
203  outimg.require("output", w + (w & ~(smfac-1)), h + mapdh, V4L2_PIX_FMT_YUYV);
204 
205  // Find most salient point:
206  int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
207 
208  // Compute attended ROI (note: coords must be even to avoid flipping U/V when we later paste):
209  unsigned int const dmx = (mx << smlev) + (smfac >> 2);
210  unsigned int const dmy = (my << smlev) + (smfac >> 2);
211  int rx = std::min(int(w) - roihw, std::max(roihw, int(dmx + 1 + smfac/4)));
212  int ry = std::min(int(h) - roihw, std::max(roihw, int(dmy + 1 + smfac/4)));
213 
214  // Asynchronously launch a bunch of saliency drawings and filter the attended locations
215  auto draw_fut =
216  std::async(std::launch::async, [&]() {
217  // Filter the attended locations:
218  itsKF->set(dmx, dmy, w, h);
219  float kfxraw, kfyraw, kfximg, kfyimg;
220  itsKF->get(kfxraw, kfyraw, kfximg, kfyimg, inimg.width, inimg.height, 1.0F, 1.0F);
221 
222  // Draw a circle around the kalman-filtered attended location:
223  jevois::rawimage::drawCircle(outimg, int(kfximg), int(kfyimg), 20, 1, jevois::yuyv::LightGreen);
224 
225  // Send saliency info to serial port (for arduino, etc):
226  //sendSerial(jevois::sformat("T2D %d %d", int(kfxraw), int(kfyraw)));
227 
228  // Paste the saliency map:
229  drawMap(outimg, &itsSaliency->salmap, w, 0, smfac, 20);
230  jevois::rawimage::writeText(outimg, "Saliency Map", w*2 - 12*6-4, 3, jevois::yuyv::White);
231  });
232 
233  // Paste the feature maps:
234  unsigned int dx = 0; // drawing x offset for each feature map
235  drawMap(outimg, &itsSaliency->color, dx, h, mapdrawfac, 18);
236  jevois::rawimage::writeText(outimg, "Color", dx+3, h+3, jevois::yuyv::White);
237  dx += mapdw;
238 
239  drawMap(outimg, &itsSaliency->intens, dx, h, mapdrawfac, 18);
240  jevois::rawimage::writeText(outimg, "Intensity", dx+3, h+3, jevois::yuyv::White);
241  dx += mapdw;
242 
243  drawMap(outimg, &itsSaliency->ori, dx, h, mapdrawfac, 18);
244  jevois::rawimage::writeText(outimg, "Orientation", dx+3, h+3, jevois::yuyv::White);
245  dx += mapdw;
246 
247  drawMap(outimg, &itsSaliency->flicker, dx, h, mapdrawfac, 18);
248  jevois::rawimage::writeText(outimg, "Flicker", dx+3, h+3, jevois::yuyv::White);
249  dx += mapdw;
250 
251  drawMap(outimg, &itsSaliency->motion, dx, h, mapdrawfac, 18);
252  jevois::rawimage::writeText(outimg, "Motion", dx+3, h+3, jevois::yuyv::White);
253  dx += mapdw;
254 
255  // Blank out free space in bottom-right corner, we will then draw the gist (which may only partially occupy that
256  // available space):
257  unsigned int const gw = outimg.width - dx, gh = outimg.height - h;
258  jevois::rawimage::drawFilledRect(outimg, dx, h, gw, gh, 0x8000);
259 
260  // Draw the gist vector, picking a zoom factor to maximize the area filled:
261  unsigned int const gscale = int(sqrt((gw * gh) / itsSaliency->gist_size));
262 
263  drawGist(outimg, itsSaliency->gist, itsSaliency->gist_size, dx, h, gw / gscale, gscale);
264  jevois::rawimage::drawRect(outimg, dx, h, gw, gh, 0x80a0);
265  jevois::rawimage::writeText(outimg, "Gist", dx+3, h+3, jevois::yuyv::White);
266 
267  // Wait for all drawings to complete:
268  draw_fut.get();
269 
270  // Draw a small square at most salient location in image and in saliency map:
271  jevois::rawimage::drawFilledRect(outimg, dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
272  jevois::rawimage::drawFilledRect(outimg, w + dmx + 1, dmy + 1, smfac/2, smfac/2, 0xffff);
273 
274  // Draw an ROI box around the most salient point:
275  jevois::rawimage::drawRect(outimg, rx - roihw, ry - roihw, roihw*2, roihw*2, 0xf0f0);
276  jevois::rawimage::drawRect(outimg, rx - roihw + 1, ry - roihw + 1, roihw*2-2, roihw*2-2, 0xf0f0);
277 
278  // Show processing fps:
279  std::string const & fpscpu = itsTimer.stop();
280  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
281 
282  // Send the output image with our processing results to the host over USB:
283  outframe.send();
284  }
285 
286  protected:
287  std::shared_ptr<Saliency> itsSaliency;
288  std::shared_ptr<Kalman2D> itsKF;
290  std::future<void> itsGPUfut;
291  std::shared_ptr<FilterGPU> itsFilter;
292  cv::Mat itsGrayImg;
293  std::future<void> itsNEONfut;
294  cv::Mat itsRGBAimg;
295  std::atomic<bool> itsRunning;
296 };
297 
298 // Allow the module to be loaded as a shared object (.so) file:
cv::Mat convertToCvRGBA(RawImage const &src)
friend friend class Module
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
void postUninit() override
Kill our external processes on uninit.
Definition: BurnTest.C:110
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition: BurnTest.C:124
std::future< void > itsGPUfut
Definition: BurnTest.C:290
void drawGist(jevois::RawImage &img, unsigned char const *gist, size_t gistsize, unsigned int xoff, unsigned int yoff, unsigned int width, unsigned int scale)
Definition: Saliency.C:771
cv::Mat itsGrayImg
Definition: BurnTest.C:292
jevois::Timer itsTimer
Definition: BurnTest.C:289
cv::Mat itsRGBAimg
Definition: BurnTest.C:294
void postInit() override
Set our GPU program after we are fully constructed and our Component path has been set...
Definition: BurnTest.C:101
void drawMap(jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale)
Definition: Saliency.C:709
virtual ~BurnTest()
Virtual destructor for safe inheritance.
Definition: BurnTest.C:121
std::shared_ptr< FilterGPU > itsFilter
Definition: BurnTest.C:291
cv::Mat convertToCvGray(RawImage const &src)
std::string const & stop()
std::future< void > itsNEONfut
Definition: BurnTest.C:293
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition: env_types.h:52
std::atomic< bool > itsRunning
Definition: BurnTest.C:295
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
JEVOIS_REGISTER_MODULE(BurnTest)
std::shared_ptr< Saliency > itsSaliency
Definition: BurnTest.C:287
std::string system(std::string const &cmd)
void drawRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int thick, unsigned int col)
BurnTest(std::string const &instance)
Constructor.
Definition: BurnTest.C:92
This is a burn test: run the quad-core saliency demo while also loading up CPU, GPU and NEON in the b...
Definition: BurnTest.C:88
unsigned int width
std::shared_ptr< Kalman2D > itsKF
Definition: BurnTest.C:288
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const