#include <jevois/DNN/YOLOjevois.H>

Helper class for runtime-configurable, quantized open-vocabulary object detection.

YOLO-JeVois splits the YOLO-World model into 3 components to allow runtime changes of class definitions on a running quantized model: 1) a CLIP model to convert text or image class definitions into 512D embeddings; 2) A YOLOjevois helper that runs an ONNX model on CPU when classes are updated, to convert the 512D CLIP embeddings into 5 tensors that will be used by the quantized object detection model; 3) a trimmed YOLO-World model (usually quantized for NPU) that takes an image plus those 5 tensors as an input, to generate detection boxes. In addition, a second variant is available, which is a bit slower, where: 1) class names or images are converted to CLIP embeddings; 2) these are input along with an image to a full YOLO-World model. This approach is slower and only works well on NPU when using 16-bit quantization.

Definition at line 59 of file YOLOjevois.H.

Inheritance diagram for jevois::dnn::YOLOjevois:

[legend]

Collaboration diagram for jevois::dnn::YOLOjevois:

[legend]

Public Member Functions
	YOLOjevois (std::string const &instance, std::map< int, std::string > &labels)
	Inherited constructor ok; must call setup() before using.

void	setup (size_t nclass, GUIhelper *helper, std::shared_ptr< jevois::dnn::Network > net)

virtual	~YOLOjevois ()
	Virtual destructor.

void	freeze (bool doit)
	Freeze/unfreeze parameters that users should not change while running.

bool	ready ()
	Are we ready to work, or still loading our networks?

int	textEmbeddingSize ()
	Get CLIP text embedding size, or 0 if we do not have a CLIP text encoder.

int	imageEmbeddingSize ()
	Get CLIP image embedding size, or 0 if we do not have a CLIP image encoder.

void	update (size_t const classnum, std::string const &label)
	Update one class using text.

void	update (size_t const classnum, cv::Mat const &img)
	Update one class using an RGB image.

cv::Mat const &	image (size_t const classid) const
	Access our class definition images.

Public Member Functions inherited from jevois::Component
	Component (std::string const &instance)
	Constructor.

virtual	~Component ()
	Virtual destructor for safe inheritance.

template<class Comp , typename... Args>
std::shared_ptr< Comp >	addSubComponent (std::string const &instance, Args &&...args)
	Pseudo-constructor: construct and add another component as a subcomponent of this one.

template<class Comp >
void	removeSubComponent (std::shared_ptr< Comp > &component)
	Remove a sub-Component from this Component, by shared_ptr.

void	removeSubComponent (std::string const &instance, bool warnIfNotFound=true)
	Remove a sub-Component from this Component, by instance name.

template<class Comp = jevois::Component>
std::shared_ptr< Comp >	getSubComponent (std::string const &instance) const
	Get a sub-component by instance name.

bool	isTopLevel () const
	Returns true if this component is top-level, i.e., its parent is jevois::Manager.

bool	initialized () const
	Has this component been initialized yet?

std::string const &	className () const
	The class name of this component.

std::string const &	instanceName () const
	The instance name of this component.

template<typename T >
std::vector< std::string >	setParamVal (std::string const &paramdescriptor, T const &val)
	Set a parameter value.

template<typename T >
void	setParamValUnique (std::string const &paramdescriptor, T const &val)
	Set a parameter value, simple version assuming only one parameter match.

template<typename T >
std::vector< std::pair< std::string, T > >	getParamVal (std::string const &paramdescriptor) const
	Get parameter(s) value(s) by descriptor.

template<typename T >
T	getParamValUnique (std::string const &paramdescriptor) const
	Get a parameter value, simple version assuming only one parameter match.

std::vector< std::string >	setParamString (std::string const &paramdescriptor, std::string const &val)
	Set a parameter value, by string.

void	setParamStringUnique (std::string const &paramdescriptor, std::string const &val)
	Set a parameter value by string, simple version assuming only one parameter match.

std::vector< std::pair< std::string, std::string > >	getParamString (std::string const &paramdescriptor) const
	Get a parameter value, by string.

std::string	getParamStringUnique (std::string const &paramdescriptor) const
	Get a parameter value by string, simple version assuming only one parameter match.

void	freezeParam (std::string const &paramdescriptor, bool doit)
	Freeze/unfreeze a parameter, by name, see ParameterBase::freeze()

void	freezeAllParams (bool doit)
	Freeze all parameters.

void	hideParam (std::string const &paramdescriptor, bool doit)
	Hide/unhide a parameter, by name, see ParameterBase::hide()

void	hideAllParams (bool doit)
	Hide all parameters.

std::string	descriptor () const
	Get our full descriptor (including all parents) as [Instancename]:[...]:[...].

void	setParamsFromFile (std::string const &filename)
	Set some parameters from a file.

std::istream &	setParamsFromStream (std::istream &is, std::string const &absfile)
	Set some parameters from an open stream.

virtual void	paramInfo (std::shared_ptr< UserInterface > s, std::map< std::string, std::string > &categs, bool skipFrozen, std::string const &cname="", std::string const &pfx="")
	Get machine-oriented descriptions of all parameters.

void	foreachParam (std::function< void(std::string const &compname, ParameterBase *p)> func, std::string const &cname="")
	Run a function on every param we hold.

template<typename T >
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ParameterCategory const &category)
	Add a new parameter after the Component has already been constructed.

template<typename T , template< typename > class ValidValuesSpec>
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ValidValuesSpec< T > const &validValuesSpec, ParameterCategory const &category)
	Add a new parameter after the Component has already been constructed.

template<typename T >
void	setDynamicParameterCallback (std::string const &name, std::function< void(T const &)> cb, bool callnow=true)
	Register a callback with a previously created dynamic parameter.

void	removeDynamicParameter (std::string const &name, bool throw_if_not_found=true)
	Remove a previously added dynamic parameter.

void	setPath (std::string const &path)
	Assign a filesystem path to this component.

std::filesystem::path	absolutePath (std::filesystem::path const &path="")
	If given path is relative (not starting with /), prepend the Component path to it.

Public Member Functions inherited from jevois::ParameterRegistry
virtual	~ParameterRegistry ()
	Virtual destructor for safe inheritance.

Protected Member Functions
void	load ()
	Load CLIP and ONNX networks, in a non-blocking thread.

Protected Member Functions inherited from jevois::Component
virtual void	preInit ()
	Called before all sub-Components are init()ed.

virtual void	postInit ()
	Called after all sub-Components are init()ed.

virtual void	preUninit ()
	Called before all sub-Components are uninit()ed.

virtual void	postUninit ()
	Called after all sub-Components are uninit()ed.

Protected Member Functions inherited from jevois::ParameterRegistry
void	addParameter (ParameterBase *const param)
	The Parameter class uses this method to register itself on construction with its owning Component.

void	removeParameter (ParameterBase *const param)
	The Parameter class uses this method to un-register itself on destruction with its owning Component.

void	callbackInitCall ()
	For all parameters that have a callback which has never been called, call it with the default param value.

Protected Attributes
std::map< int, std::string > &	itsLabels

size_t	itsNumClasses = 0

cv::Mat	itsEmbeddings

std::vector< cv::Mat >	itsCLIPimages

std::shared_ptr< jevois::dnn::Network >	itsNetwork
	Pointer to the main YOLO to update its extra inputs.

std::shared_ptr< CLIP >	itsCLIP
	CLIP network to get embeddings from text or image queries.

std::shared_ptr< Network >	itsAuxNet
	Optional aux network to process CLIP embeddings.

Constructor & Destructor Documentation

◆ YOLOjevois()

jevois::dnn::YOLOjevois::YOLOjevois	(	std::string const &	instance,
		std::map< int, std::string > &	labels
	)

Inherited constructor ok; must call setup() before using.

Definition at line 27 of file YOLOjevois.C.

◆ ~YOLOjevois()

jevois::dnn::YOLOjevois::~YOLOjevois ( )

virtual

Virtual destructor.

Definition at line 40 of file YOLOjevois.C.

Member Function Documentation

◆ freeze()

void jevois::dnn::YOLOjevois::freeze ( bool doit )

Freeze/unfreeze parameters that users should not change while running.

Definition at line 44 of file YOLOjevois.C.

◆ image()

cv::Mat const & jevois::dnn::YOLOjevois::image ( size_t const classid ) const

Access our class definition images.

Returned vector always has one cv::Mat per class, but that Mat may be empty if class was not updated by image. Caution not thread-safe.

Definition at line 250 of file YOLOjevois.C.

References LFATAL.

◆ imageEmbeddingSize()

int jevois::dnn::YOLOjevois::imageEmbeddingSize ( )

Get CLIP image embedding size, or 0 if we do not have a CLIP image encoder.

Definition at line 59 of file YOLOjevois.C.

References LFATAL.

◆ load()

void jevois::dnn::YOLOjevois::load ( )

protected

Load CLIP and ONNX networks, in a non-blocking thread.

Definition at line 129 of file YOLOjevois.C.

References jevois::absolutePath(), jevois::dnn::getLabel(), JEVOIS_CUSTOM_DNN_PATH, JEVOIS_SHARE_PATH, LFATAL, LINFO, and jevois::stringStartsWith().

Referenced by ready().

◆ ready()

bool jevois::dnn::YOLOjevois::ready ( )

Are we ready to work, or still loading our networks?

Definition at line 104 of file YOLOjevois.C.

References jevois::async(), LINFO, load(), and jevois::warnAndRethrowException().

◆ setup()

void jevois::dnn::YOLOjevois::setup	(	size_t	nclass,
		jevois::GUIhelper *	helper,
		std::shared_ptr< jevois::dnn::Network >	net
	)

Initialize for nclass object classes. All labels in the label map will be processed upon load(), which is triggered by ready(), giving rise to embeddings; or, if the label map is missing some class labels, default ones will be created. Note that YOLOjevois may modify some of the labels during load() and update().

Definition at line 32 of file YOLOjevois.C.

◆ textEmbeddingSize()

int jevois::dnn::YOLOjevois::textEmbeddingSize ( )

Get CLIP text embedding size, or 0 if we do not have a CLIP text encoder.

Definition at line 51 of file YOLOjevois.C.

References LFATAL.

◆ update() [1/2]

void jevois::dnn::YOLOjevois::update	(	size_t const	classnum,
		cv::Mat const &	img
	)

Update one class using an RGB image.

Definition at line 90 of file YOLOjevois.C.

References LFATAL.

◆ update() [2/2]

void jevois::dnn::YOLOjevois::update	(	size_t const	classnum,
		std::string const &	label
	)

Update one class using text.

Definition at line 76 of file YOLOjevois.C.

References LFATAL.

Member Data Documentation

◆ itsAuxNet

std::shared_ptr<Network> jevois::dnn::YOLOjevois::itsAuxNet

protected

Optional aux network to process CLIP embeddings.

Definition at line 104 of file YOLOjevois.H.

◆ itsCLIP

std::shared_ptr<CLIP> jevois::dnn::YOLOjevois::itsCLIP

protected

CLIP network to get embeddings from text or image queries.

Definition at line 103 of file YOLOjevois.H.

◆ itsCLIPimages

std::vector<cv::Mat> jevois::dnn::YOLOjevois::itsCLIPimages

protected

Definition at line 101 of file YOLOjevois.H.

◆ itsEmbeddings

cv::Mat jevois::dnn::YOLOjevois::itsEmbeddings

protected

Definition at line 100 of file YOLOjevois.H.

◆ itsLabels

std::map<int, std::string>& jevois::dnn::YOLOjevois::itsLabels

protected

Definition at line 98 of file YOLOjevois.H.

◆ itsNetwork

std::shared_ptr<jevois::dnn::Network> jevois::dnn::YOLOjevois::itsNetwork

protected

Pointer to the main YOLO to update its extra inputs.

Definition at line 102 of file YOLOjevois.H.

◆ itsNumClasses

size_t jevois::dnn::YOLOjevois::itsNumClasses = 0

protected

Definition at line 99 of file YOLOjevois.H.

The documentation for this class was generated from the following files:

include/jevois/DNN/YOLOjevois.H
src/jevois/DNN/YOLOjevois.C

Public Member Functions

Protected Member Functions

Protected Attributes

Constructor & Destructor Documentation

◆ YOLOjevois()

◆ ~YOLOjevois()

Member Function Documentation

◆ freeze()

◆ image()

◆ imageEmbeddingSize()

◆ load()

◆ ready()

◆ setup()

◆ textEmbeddingSize()

◆ update() [1/2]

◆ update() [2/2]

Member Data Documentation

◆ itsAuxNet

◆ itsCLIP

◆ itsCLIPimages

◆ itsEmbeddings

◆ itsLabels

◆ itsNetwork

◆ itsNumClasses