Have a look at DemoSalGistFaceObj.C in jevoisbase, starting at line 191:
for both greyscale and color, we convert from YUYV to either RGB or GRAY cv::Mat in there. Then that cv::Mat is sent to process() of the ObjectRecognition component.
Now if you look at process() in ObjectRecognition.C you will see how in there we convert from cv::Mat (either gray or RGB) to tiny_dnn vec_t
This has worked well for us for MNIST (gray) and CIFAR (color).