This is a known issue with Windows and the greyscale modes.
And because this morning I had a sudden urgent need to capture a video of me flipping the bird using the edge detect module (my friend with an openmv board made one and I simply *had* to respond in kind) and I couldn't be bothered switching over to my linux machine, I went and solved it.
To get the greyscale capture modes working under windows:
- Download http://www.gdcl.co.uk/YUVxfm.zip
- Extract it to c:\windows\system32
- Open a command prompt window with administrator privileges
- Enter "regsvr32 c:\windows\system32\YUVxfm.dll" in the command window.
That'll install a directshow filter which supports the Y800 modes.