JeVoisBase  1.18
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DNN.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
21 #include <jevois/DNN/Pipeline.H>
22 
23 // icon from opencv
24 
25 //! Detect and recognize multiple objects in scenes using OpenCV, NPU, TPU, or VPU Deep Neural Nets
26 /*! This module runs a deep neural network using the OpenCV #DNN library. Classification networks try to identify the
27  whole object or scene in the field of view, and return the top scoring object classes. Detection networks analyze a
28  scene and produce a number of bounding boxes around detected objects, together with identity labels and confidence
29  scores for each detected box. Semantic segmentation networks create a pixel-by-pixel mask which assigns a class
30  label to every location in the camera view.
31 
32  To select a network, see parameter \p pipe of component Pipeline.
33 
34  The following keys are used in the JeVois-Pro GUI (\p pipe parameter of Pipeline component):
35 
36  - **OpenCV:** network loaded by OpenCV #DNN framework and running on CPU.
37  - **NPU:** network running native on the JeVois-Pro integrated 5-TOPS NPU (neural processing unit).
38  - **SPU:** network running on the optional 26-TOPS Hailo8 SPU accelerator (stream processing unit).
39  - **TPU:** network running on the optional 4-TOPS Google Coral TPU accelerator (tensor processing unit).
40  - **VPU:** network running on the optional 1-TOPS MyriadX VPU accelerator (vector processing unit).
41  - **NPUX:** network loaded by OpenCV and running on NPU via the TIM-VX OpenCV extension. To run efficiently, network
42  should have been quantized to int8, otherwise some slow CPU-based emulation will occur.
43  - **VPUX:** network optimized for VPU but running on CPU if VPU is not available. Note that VPUX entries are
44  automatically created by scanning all VPU entries and changing their target from Myriad to CPU, if a VPU
45  accelerator is not detected. If a VPU is detected, then VPU models are listed and VPUX ones are not.
46  VPUX emulation runs on the JeVois-Pro CPU using the Arm Compute Library to provide efficient implementation
47  of various network layers and operations.
48 
49  For expected network speed, see \subpage JeVoisProBenchmarks
50 
51  Serial messages
52  ---------------
53 
54  For classification networks, when object classes are found with confidence scores above \p thresh, a message
55  containing up to \p top category:score pairs will be sent per video frame. Exact message format depends on the
56  current \p serstyle setting and is described in \ref UserSerialStyle. For example, when \p serstyle is \b Detail,
57  this module sends:
58 
59  \verbatim
60  DO category:score category:score ... category:score
61  \endverbatim
62 
63  where \a category is a category name (from \p namefile) and \a score is the confidence score from 0.0 to 100.0 that
64  this category was recognized. The pairs are in order of decreasing score.
65 
66  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
67  standardized coordinates.
68 
69  For object detection networks, when detections are found which are above threshold, one message will be sent for
70  each detected object (i.e., for each box that gets drawn when USB outputs are used), using a standardized 2D
71  message:
72  + Serial message type: \b 2D
73  + `id`: the category of the recognized object, followed by ':' and the confidence score in percent
74  + `x`, `y`, or vertices: standardized 2D coordinates of object center or corners
75  + `w`, `h`: standardized object size
76  + `extra`: any number of additional category:score pairs which had an above-threshold score for that box
77 
78  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
79  standardized coordinates.
80 
81  @author Laurent Itti
82 
83  @displayname DNN
84  @videomapping NONE 0 0 0.0 YUYV 640 480 15.0 JeVois DNN
85  @videomapping YUYV 640 498 15.0 YUYV 640 480 15.0 JeVois DNN
86  @email itti\@usc.edu
87  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
88  @copyright Copyright (C) 2018 by Laurent Itti, iLab and the University of Southern California
89  @mainurl http://jevois.org
90  @supporturl http://jevois.org/doc
91  @otherurl http://iLab.usc.edu
92  @license GPL v3
93  @distribution Unrestricted
94  @restrictions None
95  \ingroup modules */
96 class DNN : public jevois::StdModule
97 {
98  public:
99  // ####################################################################################################
100  //! Constructor
101  // ####################################################################################################
102  DNN(std::string const & instance) : jevois::StdModule(instance)
103  {
104  itsPipeline = addSubComponent<jevois::dnn::Pipeline>("pipeline");
105  }
106 
107  // ####################################################################################################
108  //! Virtual destructor for safe inheritance
109  // ####################################################################################################
110  virtual ~DNN()
111  { }
112 
113  // ####################################################################################################
114  //! Processing function implementation
115  // ####################################################################################################
116  void doprocess(jevois::InputFrame const & inframe, jevois::RawImage * outimg,
117  jevois::OptGUIhelper * helper, bool idle)
118  {
119  // If we have a second (scaled) image, assume this is the one we want to process:
120  jevois::RawImage const inimg = inframe.getp();
121 
122  // Ok, process it:
123  itsPipeline->process(inimg, this, outimg, helper, idle);
124  }
125 
126  // ####################################################################################################
127  //! Processing function, no video output
128  // ####################################################################################################
129  virtual void process(jevois::InputFrame && inframe) override
130  {
131  doprocess(inframe, nullptr, nullptr, false);
132  }
133 
134  // ####################################################################################################
135  //! Processing function with video output to USB
136  // ####################################################################################################
137  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
138  {
139  // Get the input frame:
140  jevois::RawImage const & inimg = inframe.get();
141  unsigned int const w = inimg.width, h = inimg.height;
142 
143  // Get the output image:
144  jevois::RawImage outimg = outframe.get();
145 
146  // Input and output sizes and format must match:
147  outimg.require("output", w, h, inimg.fmt);
148 
149  // Copy input to output:
150  jevois::rawimage::paste(inimg, outimg, 0, 0);
151 
152  // Process:
153  doprocess(inframe, &outimg, nullptr, false);
154 
155  // Send the output image with our processing results to the host over USB:
156  outframe.send();
157  }
158 
159 #ifdef JEVOIS_PRO
160  // ####################################################################################################
161  //! Processing function with zero-copy and GUI on JeVois-Pro
162  // ####################################################################################################
163  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
164  {
165  // Compute overall frame rate, CPU usage, etc:
166  static jevois::Timer timer("main", 20, LOG_DEBUG);
167  std::string const & fpscpu = timer.stop();
168  timer.start();
169 
170  // Start the display frame:
171  unsigned short winw, winh;
172  bool idle = helper.startFrame(winw, winh);
173 
174  // Display the camera input frame:
175  jevois::RawImage const & img = inframe.get();
176  int x = 0, y = 0; unsigned short w = 0, h = 0;
177  helper.drawInputFrame("c", inframe, x, y, w, h, true);
178 
179  // Process:
180  doprocess(inframe, nullptr, &helper, idle);
181 
182  // Show overall frame rate and camera and display info:
183  helper.iinfo(inframe, fpscpu, winw, winh);
184 
185  // Render the image and GUI:
186  helper.endFrame();
187  }
188 #endif
189 
190  // ####################################################################################################
191  protected:
192  std::shared_ptr<jevois::dnn::Pipeline> itsPipeline;
193 };
194 
195 // Allow the module to be loaded as a shared object (.so) file:
jevois::GUIhelper
Pipeline.H
jevois::OutputFrame
jevois::InputFrame::getp
const RawImage & getp(bool casync=false) const
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
Timer.H
Module.H
jevois::GUIhelper::endFrame
void endFrame()
jevois::RawImage
DNN::~DNN
virtual ~DNN()
Virtual destructor for safe inheritance.
Definition: DNN.C:110
DNN::itsPipeline
std::shared_ptr< jevois::dnn::Pipeline > itsPipeline
Definition: DNN.C:192
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::GUIhelper
jevois::RawImage::width
unsigned int width
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DNN)
jevois
jevois::StdModule::StdModule
StdModule(std::string const &instance)
DNN::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition: DNN.C:129
DNN::DNN
DNN(std::string const &instance)
Constructor.
Definition: DNN.C:102
DNN::doprocess
void doprocess(jevois::InputFrame const &inframe, jevois::RawImage *outimg, jevois::OptGUIhelper *helper, bool idle)
Processing function implementation.
Definition: DNN.C:116
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
jevois::InputFrame
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
h
int h
DNN::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: DNN.C:137
jevois::StdModule
DNN
Detect and recognize multiple objects in scenes using OpenCV, NPU, TPU, or VPU Deep Neural Nets.
Definition: DNN.C:96
demo.w
w
Definition: demo.py:85
jevois::Timer