Can the image be processed in raw image format?

Question

We would like to process the image in YCbCr format, as far as I understand this is the digital version of YUYV which should come directly from the sensor. Is it possible to process the image directly in that color format or is the conversion -->BGR -->YCbCr required?

Answer 1

Yes, we do that quite a lot, though mostly for drawing as opposed to deeper processing.

See for example

https://github.com/jevois/jevoisbase/blob/master/src/Modules/TensorFlowEasy/TensorFlowEasy.C

Getting the central ROI to be sent to the neural net, and doing all the drawings, is done directly in YUYV.

However, in that example, we do convert the contents of the ROI to RGB because the deep network was trained using RGB.

In principle, you could re-train the deep net to work direclty with YUYV (or even raw Bayer) inputs, but we have not tried it so we don't know how well it will work. Also note that our tensorflow module currently expects either rgb or gray, so it would have to be extended to also accept YUYV inputs.

The following functions operate on raw YUYV; look in the various jevoisbase modules for examples of how to use them:

http://jevois.org/doc/RawImageOps_8H.html

Comments

Thanks for the answer. I tried it this way but it seems that the cv::Mat obtained from jevois::rawimage::cvImage only has two channels, so the data type is 8UC2.

Wouldn't it need at least 3 channels?
How can I continue processing this format?

Edit: Understood now the encoding and checked the source code and found the appropriate functions:)

If someone else needs this: the 1rst channel is multiplexed so it contains odd columns U and even columns V, while Y is the 0th channel