JeVois  1.18
JeVois Smart Embedded Machine Vision Toolkit
Share this page:
Tensor/Neural Processing networks

Classes and utilities to provide abstraction to deep neural networks. Provides interfacing to OpenCV backends (CPU, OpenCL), tensor processing units (TPU) such as Coral Edge TPU and neural processing units (NPU) such as Amlogic A311D NPU.

Collaboration diagram for Tensor/Neural Processing networks:

Classes

class  jevois::dnn::Network
 Abstract class to represent a neural network. More...
 
class  jevois::dnn::NetworkHailo
 Wrapper around an DNN neural network running on the Hailo8 neural accelerator. More...
 
class  jevois::dnn::NetworkNPU
 Wrapper around a DNN neural network running on Amlogic A311D NPU accelerator (Verisilicon) More...
 
class  jevois::dnn::NetworkOpenCV
 Wrapper around an OpenCV DNN neural network. More...
 
class  jevois::dnn::NetworkPython
 Wrapper around an DNN neural network invoked through python. More...
 
class  jevois::dnn::NetworkTPU
 Wrapper around a Coral TPU neural network. More...
 
class  jevois::dnn::Pipeline
 Neural processing pipeline. More...
 
class  jevois::dnn::PostProcessor
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorClassify
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorDetect
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorPython
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorSegment
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorStub
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorYuNet
 Post-Processor for YuNet face landmarks detector. More...
 
class  jevois::dnn::PreProcessor
 Pre-Processor for neural network pipeline. More...
 
class  jevois::dnn::PreProcessorBlob
 Pre-Processor for neural network pipeline. More...
 
class  jevois::dnn::PreProcessorPython
 Pre-Processor for neural network pipeline written in python. More...
 

Modules

 DNN-related processors written in python
 In addition to writing DNN pre/net/post processors in C++, JeVois supports writing them in Python.
 

Functions

std::map< int, std::string > jevois::dnn::readLabelsFile (std::string const &fname)
 Read a label file. More...
 
std::string jevois::dnn::getLabel (std::map< int, std::string > const &labels, int id)
 Get a label from an id. More...
 
int jevois::dnn::stringToRGBA (std::string const &label, unsigned char alpha=128)
 Compute a color from a label name. More...
 
void jevois::dnn::topK (float const *pfProb, float *pfMaxProb, uint32_t *pMaxClass, uint32_t outputCount, uint32_t topNum)
 Get top-k entries and their indices. More...
 
std::string jevois::dnn::shapestr (cv::Mat const &m)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional cv::Mat with data type TYPE. More...
 
std::string jevois::dnn::shapestr (TfLiteTensor const *t)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE. More...
 
std::string jevois::dnn::shapestr (vsi_nn_tensor_attr_t const &attr)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE. More...
 
std::vector< size_t > jevois::dnn::strshape (std::string const &str)
 Get a vector of size_t from a string containing AxBxC... More...
 
int jevois::dnn::tf2cv (TfLiteType t)
 Convert from TensorFlow data type to OpenCV. More...
 
vsi_nn_type_e jevois::dnn::tf2vsi (TfLiteType t)
 Convert from TensorFlow data type to vsi_nn. More...
 
int jevois::dnn::vsi2cv (vsi_nn_type_e t)
 Convert from NPU data type to OpenCV. More...
 
void jevois::dnn::clamp (cv::Rect &r, int width, int height)
 Clamp a rectangle to within given image width and height. More...
 
void jevois::dnn::clamp (cv::Rect2f &r, float width, float height)
 Clamp a rectangle to within given image width and height. More...
 
std::vector< vsi_nn_tensor_attr_t > jevois::dnn::parseTensorSpecs (std::string const &specs)
 Parse tensor specification. More...
 
cv::Mat jevois::dnn::attrmat (vsi_nn_tensor_attr_t const &attr, void *dataptr=nullptr)
 Construct a cv::Mat from attr and possibly data pointer. More...
 
std::vector< int > jevois::dnn::attrdims (vsi_nn_tensor_attr_t const &attr)
 Get a tensor dims as a vector of int, useful to construct a matching cv::Mat. More...
 
cv::Size jevois::dnn::attrsize (vsi_nn_tensor_attr_t const &attr)
 Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions. More...
 
std::string jevois::dnn::attrstr (vsi_nn_tensor_attr_t const &attr)
 Get a string describing the specs of a tensor, including quantification specs (not provided by shapestr()) More...
 
bool jevois::dnn::attrmatch (vsi_nn_tensor_attr_t const &attr, cv::Mat const &blob)
 Check that a cv::Mat blob matches exactly the spec of an attr. More...
 
vsi_nn_tensor_attr_t jevois::dnn::tensorattr (TfLiteTensor const *t)
 Get tensor shape and type attributes for a TensorFlow Lite tensor. More...
 
size_t jevois::dnn::softmax (float const *input, size_t const n, size_t const stride, float const fac, float *output, bool maxonly)
 Apply softmax to a float vector. More...
 
cv::Mat jevois::dnn::quantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
 Quantize from float32 to fixed-point according to the quantization spec in attr. More...
 
cv::Mat jevois::dnn::dequantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
 Dequantize an output to float32 according to the quantization spec in attr. More...
 
size_t jevois::dnn::effectiveDims (cv::Mat const &m)
 Returns the number of non-unit dims in a cv::Mat. More...
 
std::string jevois::dnn::shapestr (hailo_vstream_info_t const &vi)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE. More...
 
vsi_nn_tensor_attr_t jevois::dnn::tensorattr (hailo_vstream_info_t const &vi)
 Get tensor shape and type attributes for a Hailo tensor. More...
 
vsi_nn_type_e jevois::dnn::hailo2vsi (hailo_format_type_t t)
 Convert from Hailo data type to vsi_nn. More...
 

Function Documentation

◆ attrdims()

std::vector< int > jevois::dnn::attrdims ( vsi_nn_tensor_attr_t const &  attr)

Get a tensor dims as a vector of int, useful to construct a matching cv::Mat.

Definition at line 399 of file Utils.C.

Referenced by jevois::dnn::attrmat(), jevois::dnn::NetworkHailo::load(), jevois::dnn::Network::process(), and jevois::dnn::quantize().

◆ attrmat()

cv::Mat jevois::dnn::attrmat ( vsi_nn_tensor_attr_t const &  attr,
void *  dataptr = nullptr 
)

Construct a cv::Mat from attr and possibly data pointer.

If dataptr is nullptr, new memory will be allocated for the cv::Mat. Caller must ensure data outlives the cv::Mat, and is responsible for eventually de-allocating the data. Usually, with non-null dataptr, this is only to be used as a temporary re-casting, e.g., to recast a received tensor into a Mat before dequantizing it, then forgetting about that Mat.

Definition at line 392 of file Utils.C.

References jevois::dnn::attrdims(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::NetworkNPU::doprocess(), and jevois::dnn::NetworkHailo::load().

◆ attrmatch()

bool jevois::dnn::attrmatch ( vsi_nn_tensor_attr_t const &  attr,
cv::Mat const &  blob 
)

Check that a cv::Mat blob matches exactly the spec of an attr.

Definition at line 625 of file Utils.C.

References jevois::dnn::vsi2cv().

Referenced by jevois::dnn::dequantize(), jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ attrsize()

cv::Size jevois::dnn::attrsize ( vsi_nn_tensor_attr_t const &  attr)

Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions.

Definition at line 408 of file Utils.C.

References jevois::dnn::attrstr().

Referenced by jevois::dnn::PreProcessor::blobsize(), and jevois::dnn::PreProcessorBlob::process().

◆ attrstr()

std::string jevois::dnn::attrstr ( vsi_nn_tensor_attr_t const &  attr)

◆ clamp() [1/2]

void jevois::dnn::clamp ( cv::Rect &  r,
int  width,
int  height 
)

Clamp a rectangle to within given image width and height.

Definition at line 288 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::process(), and jevois::dnn::PostProcessorYuNet::process().

◆ clamp() [2/2]

void jevois::dnn::clamp ( cv::Rect2f &  r,
float  width,
float  height 
)

Clamp a rectangle to within given image width and height.

Definition at line 298 of file Utils.C.

◆ dequantize()

cv::Mat jevois::dnn::dequantize ( cv::Mat const &  m,
vsi_nn_tensor_attr_t const &  attr 
)

Dequantize an output to float32 according to the quantization spec in attr.

attr should have the type and quantization details of m, returned tensor is float32

Definition at line 714 of file Utils.C.

References jevois::dnn::attrmatch(), LFATAL, and jevois::dnn::shapestr().

Referenced by jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ effectiveDims()

size_t jevois::dnn::effectiveDims ( cv::Mat const &  m)

Returns the number of non-unit dims in a cv::Mat.

For example, returns 2 for a 4D Mat with size 1x1x224x224, since it effectively is a 224x224 2D array

Definition at line 753 of file Utils.C.

◆ getLabel()

std::string jevois::dnn::getLabel ( std::map< int, std::string > const &  labels,
int  id 
)

Get a label from an id.

If no entry is found in the map, return the id as a string.

Definition at line 67 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::PostProcessorClassify::process(), and jevois::dnn::PostProcessorDetect::process().

◆ hailo2vsi()

vsi_nn_type_e jevois::dnn::hailo2vsi ( hailo_format_type_t  t)

Convert from Hailo data type to vsi_nn.

Definition at line 274 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::tensorattr().

◆ parseTensorSpecs()

std::vector< vsi_nn_tensor_attr_t > jevois::dnn::parseTensorSpecs ( std::string const &  specs)

◆ quantize()

cv::Mat jevois::dnn::quantize ( cv::Mat const &  m,
vsi_nn_tensor_attr_t const &  attr 
)

Quantize from float32 to fixed-point according to the quantization spec in attr.

m should be float32, typically normalized to [0..1[ or [-1..1[ already. attr is the desired quantized type and method (DFP, AA, etc)

Definition at line 639 of file Utils.C.

References jevois::dnn::attrdims(), LFATAL, jevois::dnn::shapestr(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::PreProcessorBlob::process().

◆ readLabelsFile()

std::map< int, std::string > jevois::dnn::readLabelsFile ( std::string const &  fname)

Read a label file.

Two formats are allowed: one class name per line, or one class number followed by one class name per file.

Definition at line 24 of file Utils.C.

References LERROR, LFATAL, LINFO, and jevois::replaceStringAll().

Referenced by jevois::dnn::PostProcessorClassify::onParamChange().

◆ shapestr() [1/4]

◆ shapestr() [2/4]

std::string jevois::dnn::shapestr ( hailo_vstream_info_t const &  vi)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE.

Definition at line 173 of file Utils.C.

References jevois::dnn::shapestr(), and jevois::dnn::tensorattr().

◆ shapestr() [3/4]

std::string jevois::dnn::shapestr ( TfLiteTensor const *  t)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE.

Definition at line 114 of file Utils.C.

References jevois::to_string().

◆ shapestr() [4/4]

std::string jevois::dnn::shapestr ( vsi_nn_tensor_attr_t const &  attr)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE.

Definition at line 143 of file Utils.C.

References jevois::to_string().

◆ softmax()

size_t jevois::dnn::softmax ( float const *  input,
size_t const  n,
size_t const  stride,
float const  fac,
float *  output,
bool  maxonly 
)

Apply softmax to a float vector.

n is the number of elements to process, stride is the increment in the arrays from one element to the next. So the arrays should have size n * stride. Returns the index in [0..n*stride[ of the highest scoring element. If maxonly is true, only output[returned index] is valid.

Definition at line 589 of file Utils.C.

References LFATAL.

Referenced by jevois::dnn::PostProcessorClassify::process().

◆ stringToRGBA()

int jevois::dnn::stringToRGBA ( std::string const &  label,
unsigned char  alpha = 128 
)

Compute a color from a label name.

Definition at line 75 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::report().

◆ strshape()

std::vector< size_t > jevois::dnn::strshape ( std::string const &  str)

Get a vector of size_t from a string containing AxBxC...

Definition at line 198 of file Utils.C.

References jevois::split().

Referenced by jevois::dnn::parseTensorSpecs().

◆ tensorattr() [1/2]

vsi_nn_tensor_attr_t jevois::dnn::tensorattr ( hailo_vstream_info_t const &  vi)

Get tensor shape and type attributes for a Hailo tensor.

Definition at line 523 of file Utils.C.

References jevois::dnn::hailo2vsi(), and jevois::to_string().

◆ tensorattr() [2/2]

vsi_nn_tensor_attr_t jevois::dnn::tensorattr ( TfLiteTensor const *  t)

Get tensor shape and type attributes for a TensorFlow Lite tensor.

Definition at line 484 of file Utils.C.

References LFATAL, and jevois::dnn::tf2vsi().

Referenced by jevois::dnn::NetworkTPU::inputShapes(), jevois::dnn::NetworkHailo::load(), jevois::dnn::NetworkTPU::outputShapes(), and jevois::dnn::shapestr().

◆ tf2cv()

int jevois::dnn::tf2cv ( TfLiteType  t)

Convert from TensorFlow data type to OpenCV.

Definition at line 207 of file Utils.C.

Referenced by jevois::dnn::NetworkTPU::doprocess().

◆ tf2vsi()

vsi_nn_type_e jevois::dnn::tf2vsi ( TfLiteType  t)

Convert from TensorFlow data type to vsi_nn.

Definition at line 251 of file Utils.C.

Referenced by jevois::dnn::tensorattr().

◆ topK()

void jevois::dnn::topK ( float const *  pfProb,
float *  pfMaxProb,
uint32_t *  pMaxClass,
uint32_t  outputCount,
uint32_t  topNum 
)

Get top-k entries and their indices.

Definition at line 84 of file Utils.C.

Referenced by jevois::dnn::PostProcessorClassify::process().

◆ vsi2cv()

int jevois::dnn::vsi2cv ( vsi_nn_type_e  t)

Convert from NPU data type to OpenCV.

Definition at line 229 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::attrmat(), jevois::dnn::attrmatch(), jevois::dnn::PreProcessorBlob::process(), and jevois::dnn::quantize().