Firstly, thanks for making a great product. It's awesome.
I have a bee detection project that I have running in TF on a RaspPi ( I'm battling getting it working on the pi using a movidus neural compute stick due to bugs in their tensorflow support but that's another story ) For now I want to spend some time porting this to the JeVois.
My network is not a standard classification network so i won't be able to use TensorFlowEasy directly (though I'm sure it's going to be a great reference point for code). Instead it's a fully convolution network going from (H,W) camera input to a binary (H/2,W/2) image (representing bee/no bee). See that linked blog post above for more info.
As I understand it, to run at 120fps I can't use RGB (since the conversion from native camera sensor format to RGB will be too slow) Instead I'm going to have to work in BAYER RGGB (or YUYV) . Is this correct?
Since I'm training the network for scratch, and conv nets can take whatever format, I'm happy to work in BAYER (or whatever) I just a way of capturing data in this encoding that I can ship to my desktop (to train a model there using that format)
What I've done far is just captured some sample video (using the PassThrough module & ffmpeg)
YUYV 176 144 120.0 YUYV 176 144 120.0 JeVois PassThrough # mapping
ffmpeg -f v4l2 -pixel_format yuyv422 -video_size 176x144 -framerate 120 -i /dev/video0 output.mp4
Is there a simple way to convert an output.mp4 like this back to the Bayer format? I know for RGB encoding my tensors are (H, W, 3) and I guess what I'm really trying to understand for Bayer is what's the equivalent tensor structure... I'm a total pixel format newb, so any insights much appreciated :)
: after thinking about this a bit more it might be that the single format conversation might be the least of my problems; my network has multiple convolutions, up sampling through near neighbour interpolation (though it could be deconvolution if required) as well as big chunks of channel concatenation... I should get everything going with the simple RGB conversion (even if it's on the cpu) and actually benchmark everything...