Yes, we do that quite a lot, though mostly for drawing as opposed to deeper processing.
See for example
Getting the central ROI to be sent to the neural net, and doing all the drawings, is done directly in YUYV.
However, in that example, we do convert the contents of the ROI to RGB because the deep network was trained using RGB.
In principle, you could re-train the deep net to work direclty with YUYV (or even raw Bayer) inputs, but we have not tried it so we don't know how well it will work. Also note that our tensorflow module currently expects either rgb or gray, so it would have to be extended to also accept YUYV inputs.
The following functions operate on raw YUYV; look in the various jevoisbase modules for examples of how to use them: