Classes and utilities to provide abstraction to deep neural networks. Provides interfacing to OpenCV backends (CPU, OpenCL), tensor processing units (TPU) such as Coral Edge TPU and neural processing units (NPU) such as Amlogic A311D NPU.

Collaboration diagram for Tensor/Neural Processing networks:

Classes
class	jevois::dnn::CLIP
	Interface to a CLIP model used to compute text and image embeddings. More...

class	jevois::dnn::Network
	Abstract class to represent a neural network. More...

class	jevois::dnn::NetworkHailo
	Wrapper around an DNN neural network running on the Hailo8 neural accelerator. More...

class	jevois::dnn::NetworkNPU
	Wrapper around a DNN neural network running on Amlogic A311D NPU accelerator (Verisilicon) More...

class	jevois::dnn::NetworkONNX
	Wrapper around an ONNX-Runtime neural network. More...

class	jevois::dnn::NetworkOpenCV
	Wrapper around an OpenCV DNN neural network. More...

class	jevois::dnn::NetworkPython
	Wrapper around an DNN neural network invoked through python. More...

class	jevois::dnn::NetworkTPU
	Wrapper around a Coral TPU neural network. More...

class	jevois::dnn::Pipeline
	Neural processing pipeline. More...

class	jevois::dnn::PostProcessor
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorClassify
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorDetect
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorDetectOBB
	Post-Processor for neural network pipeline for oriented bounding box (OBB) object detection. More...

class	jevois::dnn::PostProcessorPose
	Post-Processor for neural network pipeline, for human/animal/other pose detection (skeleton) More...

class	jevois::dnn::PostProcessorPython
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorSegment
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorStub
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorYuNet
	Post-Processor for YuNet face landmarks detector. More...

class	jevois::dnn::PreProcessor
	Pre-Processor for neural network pipeline. More...

class	jevois::dnn::PreProcessorBlob
	Pre-Processor for neural network pipeline. More...

class	jevois::dnn::PreProcessorPython
	Pre-Processor for neural network pipeline written in python. More...

class	jevois::dnn::YOLOjevois
	Helper class for runtime-configurable, quantized open-vocabulary object detection. More...

Modules
	DNN-related processors written in python

Functions
std::map< int, std::string >	jevois::dnn::getClassLabels (std::string const &arg)
	Get class labels from either a list or a file.

std::map< int, std::string >	jevois::dnn::readLabelsFile (std::string const &fname)
	Read a label file.

std::string	jevois::dnn::getLabel (std::map< int, std::string > const &labels, int id, bool namedonly=false)
	Get a label from an id.

int	jevois::dnn::stringToRGBA (std::string const &label, unsigned char alpha=128)
	Compute a color from a label name.

void	jevois::dnn::topK (float const pfProb, float pfMaxProb, uint32_t *pMaxClass, uint32_t outputCount, uint32_t topNum)
	Get top-k entries and their indices.

std::string	jevois::dnn::shapestr (cv::Mat const &m)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional cv::Mat with data type TYPE.

std::string	jevois::dnn::shapestr (std::vector< size_t > dims, int typ)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional size vector and OpenCV data type TYPE.

std::string	jevois::dnn::shapestr (std::vector< int > dims, int typ)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional size vector and OpenCV data type TYPE.

std::string	jevois::dnn::shapestr (TfLiteTensor const *t)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE.

std::string	jevois::dnn::shapestr (vsi_nn_tensor_attr_t const &attr)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE.

std::vector< size_t >	jevois::dnn::strshape (std::string const &str)
	Get a vector of size_t from a string containing AxBxC...

int	jevois::dnn::tf2cv (TfLiteType t)
	Convert from TensorFlow data type to OpenCV.

vsi_nn_type_e	jevois::dnn::tf2vsi (TfLiteType t)
	Convert from TensorFlow data type to vsi_nn.

int	jevois::dnn::vsi2cv (vsi_nn_type_e t)
	Convert from NPU data type to OpenCV.

void	jevois::dnn::clamp (cv::Rect &r, int width, int height)
	Clamp a rectangle to within given image width and height.

void	jevois::dnn::clamp (cv::Rect2f &r, float width, float height)
	Clamp a rectangle to within given image width and height.

std::vector< vsi_nn_tensor_attr_t >	jevois::dnn::parseTensorSpecs (std::string const &specs)
	Parse tensor specification.

cv::Mat	jevois::dnn::attrmat (vsi_nn_tensor_attr_t const &attr, void *dataptr=nullptr)
	Construct a cv::Mat from attr and possibly data pointer.

std::vector< int >	jevois::dnn::attrdims (vsi_nn_tensor_attr_t const &attr)
	Get a tensor dims as a vector of int, useful to construct a matching cv::Mat.

cv::Size	jevois::dnn::attrsize (vsi_nn_tensor_attr_t const &attr)
	Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions.

std::string	jevois::dnn::attrstr (vsi_nn_tensor_attr_t const &attr)
	Get a string describing the specs of a tensor, including quantification specs (not provided by shapestr())

bool	jevois::dnn::attrmatch (vsi_nn_tensor_attr_t const &attr, cv::Mat const &blob)
	Check that a cv::Mat blob matches exactly the spec of an attr.

vsi_nn_tensor_attr_t	jevois::dnn::tensorattr (TfLiteTensor const *t)
	Get tensor shape and type attributes for a TensorFlow Lite tensor.

float	jevois::dnn::fastexp (float x)
	Compute fast exponential using approximation formula.

float	jevois::dnn::sigmoid (float x)
	Compute sigmoid using fastexp.

void	jevois::dnn::sigmoid (cv::Mat &m)
	Compute sigmoid using fastexp on every pixel of a Mat of type CV_32F, in-place.

size_t	jevois::dnn::softmax (float const input, size_t const n, size_t const stride, float const fac, float output, bool maxonly)
	Apply softmax to a float vector.

float	jevois::dnn::softmax_dfl (float const src, float dst, size_t const n, size_t const stride=1)
	Compute softmax and return DFL distance.

cv::Mat	jevois::dnn::quantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
	Quantize from float32 to fixed-point according to the quantization spec in attr.

cv::Mat	jevois::dnn::dequantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
	Dequantize an output to float32 according to the quantization spec in attr.

size_t	jevois::dnn::effectiveDims (cv::Mat const &m)
	Returns the number of non-unit dims in a cv::Mat.

cv::Mat	jevois::dnn::concatenate (std::vector< cv::Mat > const &tensors, int axis)
	Concatenate several tensors into one.

std::vector< cv::Mat >	jevois::dnn::split (cv::Mat const &tensor, int axis, std::vector< int > const &sizes)
	Split a tensor into several, along a given axis.

std::string	jevois::dnn::shapestr (hailo_vstream_info_t const &vi)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE.

vsi_nn_tensor_attr_t	jevois::dnn::tensorattr (hailo_vstream_info_t const &vi)
	Get tensor shape and type attributes for a Hailo tensor.

vsi_nn_type_e	jevois::dnn::hailo2vsi (hailo_format_type_t t)
	Convert from Hailo data type to vsi_nn.

vsi_nn_type_e	jevois::dnn::onnx2vsi (ONNXTensorElementDataType t)
	Convert from ONNX-Runtime data type to vsi_nn.

std::string	jevois::dnn::shapestr (Ort::ConstTensorTypeAndShapeInfo const &ti)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional ONNX tensor with data type TYPE.

vsi_nn_tensor_attr_t	jevois::dnn::tensorattr (Ort::ConstTensorTypeAndShapeInfo const &ti)
	Get tensor shape and type attributes for an ONNX-runtime tensor.

Function Documentation

◆ attrdims()

std::vector< int > jevois::dnn::attrdims ( vsi_nn_tensor_attr_t const & attr )

Get a tensor dims as a vector of int, useful to construct a matching cv::Mat.

Definition at line 519 of file Utils.C.

Referenced by jevois::dnn::attrmat(), jevois::dnn::NetworkHailo::load(), and jevois::dnn::quantize().

◆ attrmat()

cv::Mat jevois::dnn::attrmat	(	vsi_nn_tensor_attr_t const &	attr,
		void *	dataptr = `nullptr`
	)

Construct a cv::Mat from attr and possibly data pointer.

If dataptr is nullptr, new memory will be allocated for the cv::Mat. Caller must ensure data outlives the cv::Mat, and is responsible for eventually de-allocating the data. Usually, with non-null dataptr, this is only to be used as a temporary re-casting, e.g., to recast a received tensor into a Mat before dequantizing it, then forgetting about that Mat.

Definition at line 512 of file Utils.C.

References jevois::dnn::attrdims(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkHailo::load(), and jevois::dnn::Network::process().

◆ attrmatch()

bool jevois::dnn::attrmatch	(	vsi_nn_tensor_attr_t const &	attr,
		cv::Mat const &	blob
	)

Check that a cv::Mat blob matches exactly the spec of an attr.

Definition at line 802 of file Utils.C.

References jevois::dnn::vsi2cv().

Referenced by jevois::dnn::dequantize(), jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ attrsize()

cv::Size jevois::dnn::attrsize ( vsi_nn_tensor_attr_t const & attr )

Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions.

Definition at line 528 of file Utils.C.

References jevois::dnn::attrstr().

Referenced by jevois::dnn::PreProcessor::blobsize(), and jevois::dnn::PreProcessorBlob::process().

◆ attrstr()

std::string jevois::dnn::attrstr ( vsi_nn_tensor_attr_t const & attr )

Get a string describing the specs of a tensor, including quantification specs (not provided by shapestr())

Definition at line 553 of file Utils.C.

Referenced by jevois::dnn::attrsize(), jevois::dnn::NetworkHailo::doprocess(), jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkTPU::inputShapes(), jevois::dnn::NetworkHailo::load(), jevois::dnn::NetworkNPU::load(), jevois::dnn::NetworkTPU::outputShapes(), jevois::dnn::PreProcessorPythonImpl::process(), and jevois::dnn::PreProcessorBlob::process().

◆ clamp() [1/2]

void jevois::dnn::clamp	(	cv::Rect &	r,
		int	width,
		int	height
	)

Clamp a rectangle to within given image width and height.

Definition at line 408 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::process(), jevois::dnn::PostProcessorPose::process(), and jevois::dnn::PostProcessorYuNet::process().

◆ clamp() [2/2]

void jevois::dnn::clamp	(	cv::Rect2f &	r,
		float	width,
		float	height
	)

Clamp a rectangle to within given image width and height.

Definition at line 418 of file Utils.C.

◆ concatenate()

cv::Mat jevois::dnn::concatenate	(	std::vector< cv::Mat > const &	tensors,
		int	axis
	)

Concatenate several tensors into one.

Axis may be positive starting at 0 for the first dimension (when reading dims from left to right), or negative starting at -1 for the last dimension. For example, for a 10x20x30 tensor, axis 0 has size 10 and is also axis -3, axis 1 has size 20 and is also axis -2, and axis 2 has size 30 and is also axis -1. The input tensors must all have the same number of dimensions, same pixel type, and sizes must match for all dimensions except the one that is being concatenated.

Definition at line 937 of file Utils.C.

References jevois::cvBytesPerPix(), jevois::cvtypestr(), and LFATAL.

Referenced by jevois::dnn::Network::process().

◆ dequantize()

cv::Mat jevois::dnn::dequantize	(	cv::Mat const &	m,
		vsi_nn_tensor_attr_t const &	attr
	)

Dequantize an output to float32 according to the quantization spec in attr.

attr should have the type and quantization details of m, returned tensor is float32

Definition at line 888 of file Utils.C.

References jevois::dnn::attrmatch(), LFATAL, and jevois::dnn::shapestr().

Referenced by jevois::dnn::NetworkHailo::doprocess().

◆ effectiveDims()

size_t jevois::dnn::effectiveDims ( cv::Mat const & m )

Returns the number of non-unit dims in a cv::Mat.

For example, returns 2 for a 4D Mat with size 1x1x224x224, since it effectively is a 224x224 2D array

Definition at line 927 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::process().

◆ fastexp()

float jevois::dnn::fastexp ( float x )

Compute fast exponential using approximation formula.

From https://github.com/Qengineering/YoloV8-ncnn-Raspberry-Pi-4/blob/main/yoloV8.cpp

Referenced by jevois::dnn::softmax(), and jevois::dnn::softmax_dfl().

◆ getClassLabels()

std::map< int, std::string > jevois::dnn::getClassLabels ( std::string const & arg )

Get class labels from either a list or a file.

If arg corresponds to a file that exists (either absolute path or relative to JEVOIS_SHARE_PATH), then load the classes from that file. Otherwise, set them from arg, which should then be a comma-separated list of values.

Definition at line 25 of file Utils.C.

References jevois::absolutePath(), JEVOIS_SHARE_PATH, jevois::dnn::readLabelsFile(), and jevois::split().

Referenced by jevois::dnn::PostProcessorClassify::onParamChange(), jevois::dnn::PostProcessorDetect::onParamChange(), jevois::dnn::PostProcessorDetectOBB::onParamChange(), and jevois::dnn::PostProcessorPose::onParamChange().

◆ getLabel()

std::string jevois::dnn::getLabel	(	std::map< int, std::string > const &	labels,
		int	id,
		bool	namedonly = `false`
	)

Get a label from an id.

If no entry is found in the map, return the id as a string (if namedonly is false) or an empty string (if namedonly is true).

Definition at line 85 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::drawWorldGUI(), jevois::dnn::YOLOjevois::load(), jevois::dnn::PostProcessorClassify::process(), jevois::dnn::PostProcessorDetect::process(), jevois::dnn::PostProcessorDetectOBB::process(), and jevois::dnn::PostProcessorPose::process().

◆ hailo2vsi()

vsi_nn_type_e jevois::dnn::hailo2vsi ( hailo_format_type_t t )

Convert from Hailo data type to vsi_nn.

Definition at line 394 of file Utils.C.

Referenced by jevois::dnn::tensorattr().

◆ onnx2vsi()

vsi_nn_type_e jevois::dnn::onnx2vsi ( ONNXTensorElementDataType t )

Convert from ONNX-Runtime data type to vsi_nn.

Definition at line 249 of file Utils.C.

Referenced by jevois::dnn::tensorattr().

◆ parseTensorSpecs()

std::vector< vsi_nn_tensor_attr_t > jevois::dnn::parseTensorSpecs ( std::string const & specs )

Parse tensor specification.

If the specification is empty, an empty vector is returned. Throws std::range_error on any parsing error.

Definition at line 428 of file Utils.C.

References jevois::split(), and jevois::dnn::strshape().

Referenced by jevois::dnn::NetworkNPU::inputShapes(), jevois::dnn::NetworkOpenCV::inputShapes(), jevois::dnn::NetworkPython::inputShapes(), jevois::dnn::NetworkTPU::inputShapes(), jevois::dnn::NetworkNPU::load(), jevois::dnn::NetworkNPU::outputShapes(), jevois::dnn::NetworkOpenCV::outputShapes(), jevois::dnn::NetworkPython::outputShapes(), and jevois::dnn::NetworkTPU::outputShapes().

◆ quantize()

cv::Mat jevois::dnn::quantize	(	cv::Mat const &	m,
		vsi_nn_tensor_attr_t const &	attr
	)

Quantize from float32 to fixed-point according to the quantization spec in attr.

m should be float32, typically normalized to [0..1[ or [-1..1[ already. attr is the desired quantized type and method (DFP, AA, etc)

Definition at line 816 of file Utils.C.

References jevois::dnn::attrdims(), LFATAL, jevois::dnn::shapestr(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::PreProcessorBlob::process().

◆ readLabelsFile()

std::map< int, std::string > jevois::dnn::readLabelsFile ( std::string const & fname )

Read a label file.

Two formats are allowed: one class name per line, or one class number followed by one class name per file.

Definition at line 42 of file Utils.C.

References LERROR, LFATAL, LINFO, and jevois::replaceStringAll().

Referenced by jevois::dnn::getClassLabels().

◆ shapestr() [1/7]

std::string jevois::dnn::shapestr ( cv::Mat const & m )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional cv::Mat with data type TYPE.

Definition at line 126 of file Utils.C.

References jevois::cvtypestr().

Referenced by jevois::dnn::dequantize(), jevois::dnn::NetworkHailo::doprocess(), jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkTPU::doprocess(), jevois::GUIhelper::drawPoly(), jevois::dnn::NetworkONNX::load(), jevois::dnn::NetworkTPU::load(), jevois::dnn::PreProcessorBlob::process(), jevois::dnn::Pipeline::process(), jevois::dnn::Network::process(), jevois::dnn::PostProcessorDetect::process(), jevois::dnn::PostProcessorDetectOBB::process(), jevois::dnn::PostProcessorPose::process(), jevois::dnn::PostProcessorSegment::process(), jevois::dnn::quantize(), jevois::dnn::PreProcessor::sendreport(), jevois::dnn::shapestr(), jevois::dnn::Pipeline::showDataPeekWindow(), jevois::dnn::split(), and jevois::dnn::PostProcessorDetectYOLO::yolo().

◆ shapestr() [2/7]

std::string jevois::dnn::shapestr ( hailo_vstream_info_t const & vi )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE.

Definition at line 293 of file Utils.C.

References jevois::dnn::shapestr(), and jevois::dnn::tensorattr().

◆ shapestr() [3/7]

std::string jevois::dnn::shapestr ( Ort::ConstTensorTypeAndShapeInfo const & ti )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional ONNX tensor with data type TYPE.

Definition at line 215 of file Utils.C.

◆ shapestr() [4/7]

std::string jevois::dnn::shapestr	(	std::vector< int >	dims,
		int	typ
	)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional size vector and OpenCV data type TYPE.

Definition at line 146 of file Utils.C.

References jevois::cvtypestr().

◆ shapestr() [5/7]

std::string jevois::dnn::shapestr	(	std::vector< size_t >	dims,
		int	typ
	)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional size vector and OpenCV data type TYPE.

Definition at line 136 of file Utils.C.

References jevois::cvtypestr().

◆ shapestr() [6/7]

std::string jevois::dnn::shapestr ( TfLiteTensor const * t )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE.

Definition at line 156 of file Utils.C.

◆ shapestr() [7/7]

std::string jevois::dnn::shapestr ( vsi_nn_tensor_attr_t const & attr )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE.

Definition at line 185 of file Utils.C.

◆ sigmoid() [1/2]

void jevois::dnn::sigmoid ( cv::Mat & m )

Compute sigmoid using fastexp on every pixel of a Mat of type CV_32F, in-place.

Definition at line 725 of file Utils.C.

References LFATAL.

◆ sigmoid() [2/2]

float jevois::dnn::sigmoid ( float x )

Compute sigmoid using fastexp.

Referenced by jevois::dnn::PostProcessorDetect::process(), jevois::dnn::PostProcessorDetectOBB::process(), and jevois::dnn::PostProcessorPose::process().

◆ softmax()

size_t jevois::dnn::softmax	(	float const *	input,
		size_t const	n,
		size_t const	stride,
		float const	fac,
		float *	output,
		bool	maxonly
	)

Apply softmax to a float vector.

n is the number of elements to process, stride is the increment in the arrays from one element to the next. So the arrays should have size n * stride. Returns the index in [0..n*stride[ of the highest scoring element. If maxonly is true, only output[returned index] is valid.

Definition at line 733 of file Utils.C.

References jevois::dnn::fastexp(), and LFATAL.

Referenced by jevois::dnn::PostProcessorClassify::process().