#include <jevois/DNN/PreProcessorBlob.H>

Pre-Processor for neural network pipeline.

This is the first step in a deep neural network processing Pipeline.

This pre-processor works as follows. As an example, assume a 1024x576 camera input frame and a 224x224 neural network input:

If camera frame is not RGB or BGR, convert to that (e.g., YUYV to RGB or BGR)
If letterbox is specified, fit the largest possible rectangle, with the aspect ratio of the network input, within the camera frame. For example, 224x224 is square so that would compute a 576x576 square box around the center of the camera frame. Otherwise, use the whole camera frame.
Crop that rectangle and resize it to network input size (possibly with stretching if letterbox was off)
Swap BGR/RGB if needed (combination of rgb parameter and color order in the received camera frame)
Most accurate but also slowest path (may be replaced by an optimized path below):
- convert pixel data to float32
- subtract mean if not zero
- divide by stdev if not 1
- multiply by scale if not 1. At this point, values will typically be in [0..1] or [-1..1]
- quantize if needed. For example, if the network expects uint8 with asymmetric affine quantization NHWC:8U:1x224x224x3:AA:0.0078125:128, divide by quantizer scale (here, 0.0078125, so that multiplies the pixel values by 128) then add zero point (here, 128). The goal here is to use as much of the 8-bit dynamic range as possible. What the network wants (specified by its intensors parameter) is determined during the network quantization process.
- convert to desired data type for the network (e.g., uint8)
- possibly convert from packed RGB from the camera (NHWC) to planar (NCHW)
- convert shape to 4D, with batch size (N) always 1
Because for uint8 (and also signed int8 and dynamic-fixed-point) this leads to nearly a no-op (first transform from native camera range [0..255] to, say, [0..1], then, during quantization, stretch back to [0..255]), fast paths are implemented for these special cases (e.g., uint8 camera input to quantized asymmetric affine uint8 network input). For dynamic fixed point, the fast path uses fast bit-shifting operations; for uint8 asymmetric affine, it is sometimes a no-op.

You can see these steps in the JeVois-Pro GUI (in the window that shows network processing details) by enabling pre-processor parameter details

Definition at line 67 of file PreProcessorBlob.H.

Inheritance diagram for jevois::dnn::PreProcessorBlob:

[legend]

Collaboration diagram for jevois::dnn::PreProcessorBlob:

[legend]

Public Member Functions
virtual	~PreProcessorBlob ()
	Destructor.

void	freeze (bool doit) override
	Freeze/unfreeze parameters that users should not change while running.

	PreProcessor (std::string const &instance)
	Inherited constructor ok.

Public Member Functions inherited from jevois::dnn::PreProcessor
	PreProcessor (std::string const &instance)
	Constructor.

virtual	~PreProcessor ()
	Destructor.

std::vector< cv::Mat >	process (jevois::RawImage const &img, std::vector< vsi_nn_tensor_attr_t > const &attrs)
	Extract blobs from input image.

virtual void	sendreport (jevois::StdModule mod, jevois::RawImage outimg=nullptr, jevois::OptGUIhelper *helper=nullptr, bool overlay=true, bool idle=false)
	Report what happened in last process() to console/output video/GUI.

cv::Size const &	imagesize () const
	Access the last processed image size.

std::vector< cv::Mat > const &	blobs () const
	Access the last computed blobs (or empty if process() has not yet been called)

cv::Size	blobsize (size_t num) const
	Access the width and height of a given blob, accounting for NCHW or NHWC.

void	b2i (float &x, float &y, size_t blobnum=0)
	Convert coordinates from blob back to original image.

void	b2i (float &x, float &y, cv::Size const &bsiz, bool letterboxed)
	Convert coordinates from blob back to original image, given a known blob size.

void	b2is (float &sx, float &sy, size_t blobnum=0)
	Convert box size from blob back to original image.

void	b2is (float &sx, float &sy, cv::Size const &bsiz, bool letterboxed)
	Convert box size from blob back to original image, given a known blob size.

cv::Rect	getUnscaledCropRect (size_t blobnum=0)
	Get unscaled crop rectangle in image coordinates.

void	getUnscaledCropRect (size_t blobnum, int &tlx, int &tly, int &brx, int &bry)
	Get unscaled crop rectangle in image coordinates.

void	i2b (float &x, float &y, size_t blobnum=0)
	Convert coordinates from image to blob.

void	i2b (float &x, float &y, cv::Size const &bsiz, bool letterboxed)
	Convert coordinates from image to blob.

std::shared_ptr< PreProcessorForPython >	getPreProcForPy () const
	Get a pointer to our python-friendly interface.

Public Member Functions inherited from jevois::Component
	Component (std::string const &instance)
	Constructor.

virtual	~Component ()
	Virtual destructor for safe inheritance.

template<class Comp , typename... Args>
std::shared_ptr< Comp >	addSubComponent (std::string const &instance, Args &&...args)
	Pseudo-constructor: construct and add another component as a subcomponent of this one.

template<class Comp >
void	removeSubComponent (std::shared_ptr< Comp > &component)
	Remove a sub-Component from this Component, by shared_ptr.

void	removeSubComponent (std::string const &instance, bool warnIfNotFound=true)
	Remove a sub-Component from this Component, by instance name.

template<class Comp = jevois::Component>
std::shared_ptr< Comp >	getSubComponent (std::string const &instance) const
	Get a sub-component by instance name.

bool	isTopLevel () const
	Returns true if this component is top-level, i.e., its parent is jevois::Manager.

bool	initialized () const
	Has this component been initialized yet?

std::string const &	className () const
	The class name of this component.

std::string const &	instanceName () const
	The instance name of this component.

template<typename T >
std::vector< std::string >	setParamVal (std::string const &paramdescriptor, T const &val)
	Set a parameter value.

template<typename T >
void	setParamValUnique (std::string const &paramdescriptor, T const &val)
	Set a parameter value, simple version assuming only one parameter match.

template<typename T >
std::vector< std::pair< std::string, T > >	getParamVal (std::string const &paramdescriptor) const
	Get parameter(s) value(s) by descriptor.

template<typename T >
T	getParamValUnique (std::string const &paramdescriptor) const
	Get a parameter value, simple version assuming only one parameter match.

std::vector< std::string >	setParamString (std::string const &paramdescriptor, std::string const &val)
	Set a parameter value, by string.

void	setParamStringUnique (std::string const &paramdescriptor, std::string const &val)
	Set a parameter value by string, simple version assuming only one parameter match.

std::vector< std::pair< std::string, std::string > >	getParamString (std::string const &paramdescriptor) const
	Get a parameter value, by string.

std::string	getParamStringUnique (std::string const &paramdescriptor) const
	Get a parameter value by string, simple version assuming only one parameter match.

void	freezeParam (std::string const &paramdescriptor, bool doit)
	Freeze/unfreeze a parameter, by name, see ParameterBase::freeze()

void	freezeAllParams (bool doit)
	Freeze all parameters.

std::string	descriptor () const
	Get our full descriptor (including all parents) as [Instancename]:[...]:[...].

void	setParamsFromFile (std::string const &filename)
	Set some parameters from a file.

std::istream &	setParamsFromStream (std::istream &is, std::string const &absfile)
	Set some parameters from an open stream.

virtual void	paramInfo (std::shared_ptr< UserInterface > s, std::map< std::string, std::string > &categs, bool skipFrozen, std::string const &cname="", std::string const &pfx="")
	Get machine-oriented descriptions of all parameters.

void	foreachParam (std::function< void(std::string const &compname, ParameterBase *p)> func, std::string const &cname="")
	Run a function on every param we hold.

template<typename T >
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ParameterCategory const &category)
	Add a new parameter after the Component has already been constructed.

template<typename T , template< typename > class ValidValuesSpec>
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ValidValuesSpec< T > const &validValuesSpec, ParameterCategory const &category)
	Add a new parameter after the Component has already been constructed.

template<typename T >
void	setDynamicParameterCallback (std::string const &name, std::function< void(T const &)> cb, bool callnow=true)
	Register a callback with a previously created dynamic parameter.

void	removeDynamicParameter (std::string const &name, bool throw_if_not_found=true)
	Remove a previously added dynamic parameter.

void	setPath (std::string const &path)
	Assign a filesystem path to this component.

std::filesystem::path	absolutePath (std::filesystem::path const &path="")
	If given path is relative (not starting with /), prepend the Component path to it.

Public Member Functions inherited from jevois::ParameterRegistry
virtual	~ParameterRegistry ()
	Virtual destructor for safe inheritance.

Protected Member Functions
std::vector< cv::Mat >	process (cv::Mat const &img, bool swaprb, std::vector< vsi_nn_tensor_attr_t > const &attrs, std::vector< cv::Rect > &crops) override
	Extract blobs from input image.

void	report (jevois::StdModule mod, jevois::RawImage outimg=nullptr, jevois::OptGUIhelper *helper=nullptr, bool overlay=true, bool idle=false) override
	Report what happened in last process() to console/output video/GUI.

Protected Member Functions inherited from jevois::Component
virtual void	preInit ()
	Called before all sub-Components are init()ed.

virtual void	postInit ()
	Called after all sub-Components are init()ed.

virtual void	preUninit ()
	Called before all sub-Components are uninit()ed.

virtual void	postUninit ()
	Called after all sub-Components are uninit()ed.

Protected Member Functions inherited from jevois::ParameterRegistry
void	addParameter (ParameterBase *const param)
	The Parameter class uses this method to register itself on construction with its owning Component.

void	removeParameter (ParameterBase *const param)
	The Parameter class uses this method to un-register itself on destruction with its owning Component.

void	callbackInitCall ()
	For all parameters that have a callback which has never been called, call it with the default param value.

Protected Attributes
std::vector< std::string >	itsInfo

Related Symbols
(Note that these are not member symbols.)
	JEVOIS_DECLARE_PARAMETER (numin, size_t, "Number of input blobs to generate from the received video image. " "Any additional inputs required by the network would have to be specified using " "Network parameter extraintensors", 1, ParamCateg)
	Parameter.

Related Symbols inherited from jevois::dnn::PreProcessor
	JEVOIS_DECLARE_PARAMETER (rgb, bool, "When true, model works with RGB input images instead BGR ones", true, ParamCateg)
	Parameter.

	JEVOIS_DECLARE_PARAMETER (scale, float, "Value scaling factor applied to input pixels after mean subtraction, " "or 0.0 to extract an unscaled UINT8 blob, typically for use with quantized networks", 2.0F/255.0F, ParamCateg)
	Parameter.

	JEVOIS_DECLARE_PARAMETER (mean, cv::Scalar, "Mean values subtracted from input image, in the same RGB/BGR " "order as the network's input", cv::Scalar(127.5F, 127.5F, 127.5F), ParamCateg)
	Parameter.

	JEVOIS_DECLARE_PARAMETER (stdev, cv::Scalar, "Input image is divided by stdev after mean subtraction and scale " "factor are applied. This is rarely used. Same RGB/BGR order as the network's input", cv::Scalar(1.0F, 1.0F, 1.0F), ParamCateg)
	Parameter.

	JEVOIS_DECLARE_PARAMETER (letterbox, bool, "When true, extract the largest possible box from the input image " "with same aspect ratio as the network's input tensor, and then rescale it to that " "tensor's width and height (hence with cropping but no distortion). Otherwise, use " "the whole image and rescale it to the network's input width and height with some " "possible stretching.", false, ParamCateg)
	Parameter.

	JEVOIS_DECLARE_PARAMETER (showin, bool, "Show outline of cropped image fed to network", true, ParamCateg)
	Parameter.

	JEVOIS_DECLARE_PARAMETER (details, bool, "Show more details about the pre-processing steps", false, ParamCateg)
	Parameter.

	JEVOIS_DEFINE_ENUM_CLASS (InterpMode,(Nearest)(Linear)(Cubic)(Area)(Lanczos4))
	Enum for image resizing modes.

	JEVOIS_DECLARE_PARAMETER (interp, InterpMode, "Image interpolation to use when resizing the input image " "from camera to network input dims", InterpMode::Nearest, InterpMode_Values, ParamCateg)
	Parameter.

Constructor & Destructor Documentation

◆ ~PreProcessorBlob()

jevois::dnn::PreProcessorBlob::~PreProcessorBlob ( )

virtual

Destructor.

Definition at line 32 of file PreProcessorBlob.C.

Member Function Documentation

◆ freeze()

void jevois::dnn::PreProcessorBlob::freeze ( bool doit )

overridevirtual

Freeze/unfreeze parameters that users should not change while running.

Implements jevois::dnn::PreProcessor.

Definition at line 36 of file PreProcessorBlob.C.

◆ PreProcessor()

jevois::dnn::PreProcessor::PreProcessor ( std::string const & instance )

Inherited constructor ok.

Definition at line 114 of file PreProcessor.C.

◆ process()

std::vector< cv::Mat > jevois::dnn::PreProcessorBlob::process	(	cv::Mat const &	img,
		bool	swaprb,
		std::vector< vsi_nn_tensor_attr_t > const &	attrs,
		std::vector< cv::Rect > &	crops
	)

overrideprotectedvirtual

Extract blobs from input image.

Implements jevois::dnn::PreProcessor.

Definition at line 42 of file PreProcessorBlob.C.

References jevois::applyLetterBox(), jevois::dnn::attrsize(), jevois::dnn::attrstr(), jevois::cvtypestr(), DETAILS, DETAILS2, LFATAL, jevois::dnn::quantize(), jevois::dnn::shapestr(), and jevois::dnn::vsi2cv().

◆ report()

void jevois::dnn::PreProcessorBlob::report	(	jevois::StdModule *	mod,
		jevois::RawImage *	outimg = `nullptr`,
		jevois::OptGUIhelper *	helper = `nullptr`,
		bool	overlay = `true`,
		bool	idle = `false`
	)

overrideprotectedvirtual

Report what happened in last process() to console/output video/GUI.

Implements jevois::dnn::PreProcessor.

Definition at line 379 of file PreProcessorBlob.C.

Friends And Related Symbol Documentation

◆ JEVOIS_DECLARE_PARAMETER()

JEVOIS_DECLARE_PARAMETER	(	numin	,
		size_t	,
		"Number of input blobs to generate from the received video image. " "Any additional inputs required by the network would have to be specified using " "Network parameter extraintensors"	,
		1	,
		ParamCateg
	)

Member Data Documentation

◆ itsInfo

std::vector<std::string> jevois::dnn::PreProcessorBlob::itsInfo

protected

Definition at line 90 of file PreProcessorBlob.H.

The documentation for this class was generated from the following files:

include/jevois/DNN/PreProcessorBlob.H
include/jevois/DNN/PreProcessor.H
src/jevois/DNN/PreProcessorBlob.C

Public Member Functions

Protected Member Functions

Protected Attributes

Related Symbols

Constructor & Destructor Documentation

◆ ~PreProcessorBlob()

Member Function Documentation

◆ freeze()

◆ PreProcessor()

◆ process()

◆ report()

Friends And Related Symbol Documentation

◆ JEVOIS_DECLARE_PARAMETER()

Member Data Documentation

◆ itsInfo