JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ObjectRecognition.H
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #pragma once
19 
21 #include <stdarg.h> // needed by tiny_dnn
22 
23 // Defines used to optimize tiny-dnn:
24 #define CNN_USE_TBB
25 #undef CNN_USE_DOUBLE
26 //#define CNN_USE_NNPACK // This still yieds compilation errors inside tiny-dnn and is not ready yet upstream
27 
28 #include <tiny-dnn/tiny_dnn/config.h> // for float_t, etc. this does not include much code
29 #include <tiny-dnn/tiny_dnn/util/aligned_allocator.h> // for aligned_allocator
30 #include <tiny-dnn/tiny_dnn/util/util.h> // for index3d
31 #include <opencv2/core/core.hpp>
32 
33 namespace tiny_dnn { template <typename NetType> class network; }
34 
35 //! Abstract base class for an object recognition component
36 /*! This base class provides a framework to recognize objects using deep neural networks. The network is implemented
37  using the tiny-dnn library, see https://github.com/tiny-dnn/tiny-dnn
38 
39  Derived classes implement a particular network by overriding the define() method of the base class. They also must
40  implement the other abstract methods provided here.
41 
42  To create a new object recognition component, one would usually derive from the ObjectRecognition class template as
43  opposed to deriving directly from ObjectRecognitionBase.
44 
45  Training
46  --------
47 
48  Training is automatically launched if a pre-trained weight file is not found. Thus, typical workflow is, assuming
49  \jvversion{1.4} or later:
50 
51  - create a new derived class and implement define(), train(), prcess(), etc in there. See ObjectRecognitionMNIST for
52  example.
53 
54  - create a directory <b>/jevois/share/tiny-dnn/NAME/</b> where \a NAME is the instance name you give to your derived
55  object.
56 
57  - copy the training data to that directory (as required by your implementation of train() in your derived class)
58 
59  - run `jevois-daemon` on host, make sure it has write permission to the directory you created. It will fail to load
60  the pre-trained weights and will initiate training.
61 
62  - when training is complete, trained weights will be saved as <b>/jevois/share/tiny-dnn/NAME/weights.tnn</b> (for
63  \jvversion{1.4} or earlier) or as <b>/jevois/share/tiny-dnn/NAME/weights.tnn.host</b> (for \jvversion{1.5} or
64  later).
65 
66  Starting with \jvversion{1.5}, tiny-dnn has been updated to a recent version which uses \b cereal as a back-end to
67  save networks and weights. This yields efficient loading of pre-trained networks but the binary archive format is
68  not portable between Intel x64 hosts and ARM platform. Hence, you should proceed as follows:
69 
70  - run the above steps. Two files will be saved: \b weights.tnn.host (binary) and \b weights.tnn.json (portable text
71  file).
72 
73  - copy \b weights.tnn.json to microSD in <b>JEVOIS:/share//tiny-dnn/NAME/</b> where \a NAME is the instance name you
74  give to your derived object.
75 
76  - insert microSD into JeVois camera and launch the machine vision mode that uses your network.
77 
78  - ObjectRecognition::load() will fail to load the missing \b weights.tnn.platform (binary) and will thus revert to
79  loading \b weights.tnn.json instead. It will then save \b weights.tnn.platform to microSD. Because this was done
80  by the JeVois camera, \b weights.tnn.platform will now be in ARM binary format.
81 
82  - You can now copy \b weights.tnn.platform out of your microSD to <b>~/jevoisbase/share/tiny-dnn/NAME/</b> so that
83  it will be flashed to microSD next time you make one using `jevois-flash-card` and you will avoid having to
84  convert again next time JeVois starts.
85 
86  \ingroup components */
88 {
89  public:
90  //! Type used by tiny-dnn for the results:
91  typedef std::vector<tiny_dnn::float_t, tiny_dnn::aligned_allocator<tiny_dnn::float_t, 64> > vec_t;
92 
93  //! Constructor
94  ObjectRecognitionBase(std::string const & instance);
95 
96  //! Virtual destructor for safe inheritance
97  virtual ~ObjectRecognitionBase();
98 
99  //! Define the network structure
100  /*! Derived classes must implement this function and load a network structure. */
101  virtual void define() = 0;
102 
103  //! Get the input size for the current network, useful to prepare inputs to process()
104  virtual tiny_dnn::index3d<size_t> insize() const = 0;
105 
106  //! Train the network
107  /*! Derived classes must implement this function. */
108  virtual void train(std::string const & path) = 0;
109 
110  //! Process an image, results are confidence for each category
111  virtual vec_t process(cv::Mat const & img, bool normalize = true) = 0;
112 
113  //! Return the name of a given category (0-based index in the vector of results)
114  virtual std::string const & category(size_t idx) const = 0;
115 };
116 
117 //! Wrapper around a neural network implemented by with the tiny-dnn framework by Taiga Nomi
118 /*! Because tiny-dnn is an all-include package, we use the pimpl idiom here to avoid including all the tiny-dnn
119  sources in the header file, and instead only include and compile them once in our ObjectRecognition.C file.
120 
121  \ingroup components */
122 template <typename NetType>
124 {
125  public:
126  //! Constructor allocates the (empty) network
127  ObjectRecognition(std::string const & instance);
128 
129  //! Destructor
130  virtual ~ObjectRecognition();
131 
132  //! Get the input size for the current network, useful to prepare inputs to process()
133  virtual tiny_dnn::index3d<size_t> insize() const override;
134 
135  //! Process an image, results are confidence for each category
136  vec_t process(cv::Mat const & img, bool normalize = true) override;
137 
138  protected:
139  //! Initialize the network, required before one starts using it
140  /*! First, we will call define(). Then, we will look in path for weights.tnn, and if not found, we will call
141  train() to train the network using data in that path, and then we will save weights.tnn. Derived classes may
142  implement a constructor that takes path and then calls init(path) after the base class has been constructed
143  (e.g., in the body of the derived class constructor). */
144  virtual void postInit() override;
145 
146  tiny_dnn::network<NetType> * net; // pointer here to avoid #include'ing tiny_dnn.h
147 };
std::vector< tiny_dnn::float_t, tiny_dnn::aligned_allocator< tiny_dnn::float_t, 64 > > vec_t
Type used by tiny-dnn for the results:
Abstract base class for an object recognition component.
Wrapper around a neural network implemented by with the tiny-dnn framework by Taiga Nomi...
std::string category
tiny_dnn::network< NetType > * net