Saliency Class Reference

Simple wrapper class around Rob Peter's C-optimized, fixed-point-math visual saliency code. More...

#include <jevoisbase/Components/Saliency/Saliency.H>

Inheritance diagram for Saliency:

Collaboration diagram for Saliency:

Classes
struct	visitor_data

Public Member Functions
	Saliency (std::string const &instance)
	Constructor. More...
virtual	~Saliency ()
	Destructor. More...
void	process (jevois::RawImage const &input, bool do_gist)
	Process a raw YUYV image. Results are stored in the Saliency class. More...
void	process (cv::Mat const &input, bool do_gist)
	Process an RGB image. Results are stored in the Saliency class. More...
void	waitUntilDoneWithInput () const
	Wait until process() is done using the input image. More...
void	getSaliencyMax (int &x, int &y, intg32 &value)
	Get location and value of max point in the saliency map. More...
void	inhibitionOfReturn (int const x, int const y, float const sigma)
	Inhibit the saliency map around a point, sigma is in pixels at the sacle of the map. More...
Public Member Functions inherited from jevois::Component
	Component (std::string const &instance)
virtual	~Component ()
std::shared_ptr< Comp >	addSubComponent (std::string const &instance, Args &&...args)
void	removeSubComponent (std::shared_ptr< Comp > &component)
void	removeSubComponent (std::string const &instance, bool warnIfNotFound=true)
std::shared_ptr< Comp >	getSubComponent (std::string const &instance) const
bool	isTopLevel () const
Engine *	engine () const
bool	initialized () const
const std::string &	className () const
const std::string &	instanceName () const
std::vector< std::string >	setParamVal (std::string const &paramdescriptor, T const &val)
void	setParamValUnique (std::string const &paramdescriptor, T const &val)
std::vector< std::pair< std::string, T > >	getParamVal (std::string const &paramdescriptor) const
T	getParamValUnique (std::string const &paramdescriptor) const
std::vector< std::string >	setParamString (std::string const &paramdescriptor, std::string const &val)
void	setParamStringUnique (std::string const &paramdescriptor, std::string const &val)
std::vector< std::pair< std::string, std::string > >	getParamString (std::string const &paramdescriptor) const
std::string	getParamStringUnique (std::string const &paramdescriptor) const
void	freezeParam (std::string const &paramdescriptor)
void	unFreezeParam (std::string const &paramdescriptor)
void	freezeAllParams ()
void	unFreezeAllParams ()
std::string	descriptor () const
void	setParamsFromFile (std::string const &filename)
std::istream &	setParamsFromStream (std::istream &is, std::string const &absfile)
virtual void	paramInfo (std::shared_ptr< UserInterface > s, std::map< std::string, std::string > &categs, bool skipFrozen, std::string const &cname="", std::string const &pfx="")
void	foreachParam (std::function< void(std::string const &compname, ParameterBase *p)> func, std::string const &cname="")
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ParameterCategory const &category)
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ValidValuesSpec< T > const &validValuesSpec, ParameterCategory const &category)
void	setDynamicParameterCallback (std::string const &name, std::function< void(T const &)> cb, bool callnow=true)
void	removeDynamicParameter (std::string const &name)
void	setPath (std::string const &path)
std::filesystem::path	absolutePath (std::filesystem::path const &path="")
std::shared_ptr< Comp >	addSubComponent (std::string const &instance, Args &&...args)
void	removeSubComponent (std::shared_ptr< Comp > &component)
void	removeSubComponent (std::string const &instance, bool warnIfNotFound=true)
std::shared_ptr< Comp >	getSubComponent (std::string const &instance) const
bool	isTopLevel () const
Engine *	engine () const
bool	initialized () const
const std::string &	className () const
const std::string &	instanceName () const
std::vector< std::string >	setParamVal (std::string const &paramdescriptor, T const &val)
void	setParamValUnique (std::string const &paramdescriptor, T const &val)
std::vector< std::pair< std::string, T > >	getParamVal (std::string const &paramdescriptor) const
T	getParamValUnique (std::string const &paramdescriptor) const
std::vector< std::string >	setParamString (std::string const &paramdescriptor, std::string const &val)
void	setParamStringUnique (std::string const &paramdescriptor, std::string const &val)
std::vector< std::pair< std::string, std::string > >	getParamString (std::string const &paramdescriptor) const
std::string	getParamStringUnique (std::string const &paramdescriptor) const
void	freezeParam (std::string const &paramdescriptor)
void	unFreezeParam (std::string const &paramdescriptor)
void	freezeAllParams ()
void	unFreezeAllParams ()
std::string	descriptor () const
void	setParamsFromFile (std::string const &filename)
std::istream &	setParamsFromStream (std::istream &is, std::string const &absfile)
virtual void	paramInfo (std::shared_ptr< UserInterface > s, std::map< std::string, std::string > &categs, bool skipFrozen, std::string const &cname="", std::string const &pfx="")
void	foreachParam (std::function< void(std::string const &compname, ParameterBase *p)> func, std::string const &cname="")
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ParameterCategory const &category)
std::shared_ptr< DynamicParameter< T > >	addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ValidValuesSpec< T > const &validValuesSpec, ParameterCategory const &category)
void	setDynamicParameterCallback (std::string const &name, std::function< void(T const &)> cb, bool callnow=true)
void	removeDynamicParameter (std::string const &name)
void	setPath (std::string const &path)
std::filesystem::path	absolutePath (std::filesystem::path const &path="")
Public Member Functions inherited from jevois::ParameterRegistry
virtual	~ParameterRegistry ()

Public Attributes
struct env_image	salmap
	The saliency map. More...
struct env_image	intens
struct env_image	color
struct env_image	ori
struct env_image	flicker
struct env_image	motion
unsigned char *	gist
	Gist vector has 1152 entries, 72 feature maps * 16 values/map. More...
const size_t	gist_size

Related Functions
(Note that these are not member functions.)
	JEVOIS_DECLARE_PARAMETER (cweight, byte, "Color channel weight", 255, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (iweight, byte, "Intensity channel weight", 255, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (oweight, byte, "Orientation channel weight", 255, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (fweight, byte, "Flicker channel weight", 255, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (mweight, byte, "Motion channel weight", 255, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (centermin, size_t, "Lowest (finest) of the 3 center scales", 2, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (deltamin, size_t, "Lowest (finest) of the 2 center-surround delta scales", 3, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (smscale, size_t, "Scale of the saliency map", 4, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (mthresh, byte, "Motion threshold", 0, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (fthresh, byte, "Flicker threshold", 0, ParamCateg)
	Parameter. More...
	JEVOIS_DECLARE_PARAMETER (msflick, bool, "Use multiscale flicker computation", false, ParamCateg)
	Parameter. More...
void	drawMap (jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale)
	Draw a saliency map or feature map in a YUYV image. More...
void	drawMap (jevois::RawImage &img, env_image const *fmap, unsigned int xoff, unsigned int yoff, unsigned int scale, unsigned int bitshift)
	Draw a saliency map or feature map in a YUYV image, applying some right bitshift to values. More...
void	drawGist (jevois::RawImage &img, unsigned char const *gist, size_t gistsize, unsigned int xoff, unsigned int yoff, unsigned int width, unsigned int scale)
	Draw a gist vector in a YUYV image as a rectangle of width width*scale and correct height. More...

Additional Inherited Members
Protected Member Functions inherited from jevois::Component
virtual void	preInit ()
virtual void	postInit ()
virtual void	preUninit ()
virtual void	postUninit ()
virtual void	preInit ()
virtual void	postInit ()
virtual void	preUninit ()
virtual void	postUninit ()
Protected Member Functions inherited from jevois::ParameterRegistry
void	addParameter (ParameterBase *const param)
void	removeParameter (ParameterBase *const param)
void	callbackInitCall ()

Detailed Description

Simple wrapper class around Rob Peter's C-optimized, fixed-point-math visual saliency code.

This component implements a visual saliency detector using the algorithm from Itti et al., IEEE Trans PAMI,

1. The saliency map highlights locations in an video stream that stand out and would attract the attention of a human. In addition, this component computes the gist of the scene, a low-dimentional statistical summary of the whole scene that can be used for scene classification.

Note that the Saliency component here imposes some restrictions on the saliency computations:

• we enforce that always 6 feature maps are computed for each channel, which is obtained by using 3 center scales and 2 center-surround scale deltas. In this component, only the finest center scale and smallest delta can be specified, as well as the saliency map scale. This is so that the gist vector can have a fixed size.
• number of orientations in the orientation channel is fixed at 4, number of directions in the motion channel is fixed at 4. This is again to obtain a fixed gist vector size.
• we always consider all of C, I O, F and M channels as opposed to having a more dynamic collection of channels as done in other implementations of this model (see, e.g., http://iLab.usc.edu/toolkit/). This is again so that we have fixed gist size and available output maps. Note that some channels will not be computed if their weight is set to zero, and instead the maps will be empty and the gist entries will be zeroed out.

See the research paper at http://ilab.usc.edu/publications/doc/Itti_etal98pami.pdf

Definition at line 95 of file Saliency.H.

Constructor & Destructor Documentation

Saliency()

Saliency::Saliency

(

std::string const &

instance

)

Constructor.

Definition at line 81 of file Saliency.C.

References env_init_integer_math(), and env_params_set_defaults().

~Saliency()

Saliency::~Saliency ( )

virtual

Destructor.

Definition at line 106 of file Saliency.C.

References env_img_make_empty(), env_motion_channel_destroy(), env_pyr_make_empty(), and gist.

Member Function Documentation

getSaliencyMax()

void Saliency::getSaliencyMax	(	int &	x,
		int &	y,
		intg32 &	value
	)

Get location and value of max point in the saliency map.

Definition at line 672 of file Saliency.C.

inhibitionOfReturn()

void Saliency::inhibitionOfReturn	(	int const	x,
		int const	y,
		float const	sigma
	)

Inhibit the saliency map around a point, sigma is in pixels at the sacle of the map.

Definition at line 686 of file Saliency.C.

process() [1/2]

void Saliency::process	(	cv::Mat const &	input,
		bool	do_gist
	)

Process an RGB image. Results are stored in the Saliency class.

Definition at line 225 of file Saliency.C.

References jevois::async(), env_params::chan_c_weight, color, env_image::dims, ENV_ASSERT, env_c_luminance_from_byte(), env_chan_color(), env_img_init(), env_total_weight(), env_dims::h, env_math::nbits, salmap, env_dims::w, and WEIGHT_SCALEBITS.

process() [2/2]

void Saliency::process	(	jevois::RawImage const &	input,
		bool	do_gist
	)

Process a raw YUYV image. Results are stored in the Saliency class.

Definition at line 339 of file Saliency.C.

References jevois::Profiler::checkpoint(), env_image::dims, env_img_init(), env_dims::h, jevois::RawImage::height, jevois::RawImage::pixels(), jevois::Profiler::start(), and jevois::RawImage::width.

waitUntilDoneWithInput()

void Saliency::waitUntilDoneWithInput

(

)

const

Wait until process() is done using the input image.

This assumes that you are running process() in a different thread and here just want to wait until the initial processing that uses the input image is complete, so you can return that input image to the camera driver.

Definition at line 217 of file Saliency.C.

Friends And Related Function Documentation

drawGist()

void drawGist ( jevois::RawImage &  img, unsigned char const *  gist, size_t  gistsize, unsigned int  xoff, unsigned int  yoff, unsigned int  width, unsigned int  scale  )

related

Draw a gist vector in a YUYV image as a rectangle of width width*scale and correct height.

Definition at line 771 of file Saliency.C.

drawMap() [1/2]

void drawMap ( jevois::RawImage &  img, env_image const *  fmap, unsigned int  xoff, unsigned int  yoff, unsigned int  scale  )

related

Draw a saliency map or feature map in a YUYV image.

Definition at line 709 of file Saliency.C.

drawMap() [2/2]

void drawMap ( jevois::RawImage &  img, env_image const *  fmap, unsigned int  xoff, unsigned int  yoff, unsigned int  scale, unsigned int  bitshift  )

related

Draw a saliency map or feature map in a YUYV image, applying some right bitshift to values.

Definition at line 740 of file Saliency.C.

JEVOIS_DECLARE_PARAMETER() [1/11]

JEVOIS_DECLARE_PARAMETER ( centermin  , size_t  , "Lowest (finest) of the 3 center scales"  , 2  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [2/11]

JEVOIS_DECLARE_PARAMETER ( cweight  , byte  , "Color channel weight"  , 255  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [3/11]

JEVOIS_DECLARE_PARAMETER ( deltamin  , size_t  , "Lowest (finest) of the 2 center-surround delta scales"  , 3  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [4/11]

JEVOIS_DECLARE_PARAMETER ( fthresh  , byte  , "Flicker threshold"  , 0  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [5/11]

JEVOIS_DECLARE_PARAMETER ( fweight  , byte  , "Flicker channel weight"  , 255  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [6/11]

JEVOIS_DECLARE_PARAMETER ( iweight  , byte  , "Intensity channel weight"  , 255  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [7/11]

JEVOIS_DECLARE_PARAMETER ( msflick  , bool  , "Use multiscale flicker computation"  , false  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [8/11]

JEVOIS_DECLARE_PARAMETER ( mthresh  , byte  , "Motion threshold"  , 0  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [9/11]

JEVOIS_DECLARE_PARAMETER ( mweight  , byte  , "Motion channel weight"  , 255  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [10/11]

JEVOIS_DECLARE_PARAMETER ( oweight  , byte  , "Orientation channel weight"  , 255  , ParamCateg    )

related

Parameter.

JEVOIS_DECLARE_PARAMETER() [11/11]

JEVOIS_DECLARE_PARAMETER ( smscale  , size_t  , "Scale of the saliency map"  , 4  , ParamCateg    )

related

Parameter.

Member Data Documentation

color

struct env_image Saliency::color

Definition at line 127 of file Saliency.H.

Referenced by process().

flicker

struct env_image Saliency::flicker

Definition at line 129 of file Saliency.H.

gist

unsigned char* Saliency::gist

Gist vector has 1152 entries, 72 feature maps * 16 values/map.

The 16 values are in raster order. If a feature weight is zero, the corresponding gist values are all zero. The feature maps are in order:

red/green (6*16), blue/yellow (6*16), intens (6*16), ori (6*4*16), flicker (6*16), motion (6*4*16)

so the offsets are: red/green: offset 0 len 6*16 blue/yellow: offset 1*6*16 len 6*16 intens: offset 2*6*16 len 6*16 ori0: offset 3*6*16 len 6*16 ori1: offset 4*6*16 len 6*16 ori2: offset 5*6*16 len 6*16 ori3: offset 6*6*16 len 6*16 flicker: offset 7*6*16 len 6*16 motion0: offset 8*6*16 len 6*16 motion1: offset 9*6*16 len 6*16 motion2: offset 10*6*16 len 6*16 motion3: offset 11*6*16 len 6*16

Definition at line 151 of file Saliency.H.

Referenced by ~Saliency().

gist_size

const size_t Saliency::gist_size

Definition at line 152 of file Saliency.H.

intens

struct env_image Saliency::intens

Definition at line 126 of file Saliency.H.

motion

struct env_image Saliency::motion

Definition at line 130 of file Saliency.H.

ori

struct env_image Saliency::ori

Definition at line 128 of file Saliency.H.

salmap

struct env_image Saliency::salmap

The saliency map.

Definition at line 118 of file Saliency.H.

Referenced by process().

---

The documentation for this class was generated from the following files:

• include/jevoisbase/Components/Saliency/Saliency.H
• src/Components/Saliency/Saliency.C