JeVoisBase  1.16
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DNN.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Debug/Timer.H>
21 #include <jevois/DNN/Pipeline.H>
22 
23 // icon from opencv
24 
25 //! Detect and recognize multiple objects in scenes using OpenCV, NPU, TPU, or VPU Deep Neural Nets
26 /*! This module runs a deep neural network using the OpenCV DNN library. Classification networks try to identify the
27  whole object or scene in the field of view, and return the top scoring object classes. Detection networks analyze a
28  scene and produce a number of bounding boxes around detected objects, together with identity labels and confidence
29  scores for each detected box. Semantic segmentation networks create a pixel-by-pixel mask which assigns a class
30  label to every location in the camera view.
31 
32  To select a network, see parameter \p pipe of component Pipeline.
33 
34  Serial messages
35  ---------------
36 
37  For classification networks, when object classes are found with confidence scores above \p thresh, a message
38  containing up to \p top category:score pairs will be sent per video frame. Exact message format depends on the
39  current \p serstyle setting and is described in \ref UserSerialStyle. For example, when \p serstyle is \b Detail,
40  this module sends:
41 
42  \verbatim
43  DO category:score category:score ... category:score
44  \endverbatim
45 
46  where \a category is a category name (from \p namefile) and \a score is the confidence score from 0.0 to 100.0 that
47  this category was recognized. The pairs are in order of decreasing score.
48 
49  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
50  standardized coordinates.
51 
52  For object detection networks, when detections are found which are above threshold, one message will be sent for
53  each detected object (i.e., for each box that gets drawn when USB outputs are used), using a standardized 2D
54  message:
55  + Serial message type: \b 2D
56  + `id`: the category of the recognized object, followed by ':' and the confidence score in percent
57  + `x`, `y`, or vertices: standardized 2D coordinates of object center or corners
58  + `w`, `h`: standardized object size
59  + `extra`: any number of additional category:score pairs which had an above-threshold score for that box
60 
61  See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
62  standardized coordinates.
63 
64  @author Laurent Itti
65 
66  @displayname DNN
67  @videomapping NONE 0 0 0.0 YUYV 640 480 15.0 JeVois DNN
68  @videomapping YUYV 640 498 15.0 YUYV 640 480 15.0 JeVois DNN
69  @email itti\@usc.edu
70  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
71  @copyright Copyright (C) 2018 by Laurent Itti, iLab and the University of Southern California
72  @mainurl http://jevois.org
73  @supporturl http://jevois.org/doc
74  @otherurl http://iLab.usc.edu
75  @license GPL v3
76  @distribution Unrestricted
77  @restrictions None
78  \ingroup modules */
79 class DNN : public jevois::StdModule
80 {
81  public:
82  // ####################################################################################################
83  //! Constructor
84  // ####################################################################################################
85  DNN(std::string const & instance) : jevois::StdModule(instance)
86  {
87  itsPipeline = addSubComponent<jevois::dnn::Pipeline>("pipeline");
88  }
89 
90  // ####################################################################################################
91  //! Virtual destructor for safe inheritance
92  // ####################################################################################################
93  virtual ~DNN()
94  { }
95 
96  // ####################################################################################################
97  //! Processing function implementation
98  // ####################################################################################################
99  void doprocess(jevois::InputFrame const & inframe, jevois::RawImage * outimg,
100  jevois::OptGUIhelper * helper, bool idle)
101  {
102  // If we have a second (scaled) image, assume this is the one we want to process:
103  jevois::RawImage const inimg = inframe.getp();
104 
105  // Ok, process it:
106  itsPipeline->process(inimg, this, outimg, helper, idle);
107  }
108 
109  // ####################################################################################################
110  //! Processing function, no video output
111  // ####################################################################################################
112  virtual void process(jevois::InputFrame && inframe) override
113  {
114  doprocess(inframe, nullptr, nullptr, false);
115  }
116 
117  // ####################################################################################################
118  //! Processing function with video output to USB
119  // ####################################################################################################
120  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
121  {
122  // Get the input frame:
123  jevois::RawImage const & inimg = inframe.get();
124  unsigned int const w = inimg.width, h = inimg.height;
125 
126  // Get the output image:
127  jevois::RawImage outimg = outframe.get();
128 
129  // Input and output sizes and format must match:
130  outimg.require("output", w, h, inimg.fmt);
131 
132  // Copy input to output:
133  jevois::rawimage::paste(inimg, outimg, 0, 0);
134 
135  // Process:
136  doprocess(inframe, &outimg, nullptr, false);
137 
138  // Send the output image with our processing results to the host over USB:
139  outframe.send();
140  }
141 
142 #ifdef JEVOIS_PRO
143  // ####################################################################################################
144  //! Processing function with zero-copy and GUI on JeVois-Pro
145  // ####################################################################################################
146  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
147  {
148  // Compute overall frame rate, CPU usage, etc:
149  static jevois::Timer timer("main", 20, LOG_DEBUG);
150  std::string const & fpscpu = timer.stop();
151  timer.start();
152 
153  // Start the display frame:
154  unsigned short winw, winh;
155  bool idle = helper.startFrame(winw, winh);
156 
157  // Display the camera input frame:
158  jevois::RawImage const & img = inframe.get();
159  int x = 0, y = 0; unsigned short w = 0, h = 0;
160  helper.drawInputFrame("c", inframe, x, y, w, h, true);
161 
162  // Process:
163  doprocess(inframe, nullptr, &helper, idle);
164 
165  // Show overall frame rate and camera and display info:
166  helper.iinfo(inframe, fpscpu, winw, winh);
167 
168  // Render the image and GUI:
169  helper.endFrame();
170  }
171 #endif
172 
173  // ####################################################################################################
174  protected:
175  std::shared_ptr<jevois::dnn::Pipeline> itsPipeline;
176 };
177 
178 // Allow the module to be loaded as a shared object (.so) file:
Pipeline.H
jevois::OutputFrame
jevois::InputFrame::getp
const RawImage & getp(bool casync=false) const
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
Timer.H
Module.H
jevois::GUIhelper
jevois::GUIhelper::endFrame
void endFrame()
jevois::RawImage
DNN::~DNN
virtual ~DNN()
Virtual destructor for safe inheritance.
Definition: DNN.C:93
DNN::itsPipeline
std::shared_ptr< jevois::dnn::Pipeline > itsPipeline
Definition: DNN.C:175
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::GUIhelper
jevois::RawImage::width
unsigned int width
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(DNN)
jevois
jevois::StdModule::StdModule
StdModule(std::string const &instance)
DNN::process
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition: DNN.C:112
DNN::DNN
DNN(std::string const &instance)
Constructor.
Definition: DNN.C:85
DNN::doprocess
void doprocess(jevois::InputFrame const &inframe, jevois::RawImage *outimg, jevois::OptGUIhelper *helper, bool idle)
Processing function implementation.
Definition: DNN.C:99
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
jevois::InputFrame
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
h
int h
DNN::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: DNN.C:120
jevois::StdModule
DNN
Detect and recognize multiple objects in scenes using OpenCV, NPU, TPU, or VPU Deep Neural Nets.
Definition: DNN.C:79
jevois::Timer