Yes, we do use similar flags (but for A7). The part of the INSTALL file you are referring to is to compile opencv for host. So no arm flags here as the assumption is intel host.
Opencv for platform is compiled with arm flags as part as buildroot, in jevois-sdk
In jevois, we support compiling and running natively on an intel host (assumed fast) and cross compiling for the platform hardware (one specific arm A7 chip). Native compilation on intel is to allow fast development cycles: try your code on your fast desktop before you cross compile it for jevois, copy to microSD, etc. native compilation on platform is too slow and annoying and is not supported. It takes over an hour to compile jevois and jevoisbase on a raspberry pi, but only 2 minutes on a dual xeon...
To create the entire cross-compiled OS with all utilities and libraries, we use buildroot. This is in jevois-sdk. The assumption here again is that a fast intel host will be used to cross compile buildroot. It does indeed take on the order of 10 hours to build the whole thing on a 48-core xeon with 256GB ram.
These have more details about this
http://jevois.org/doc/ProgrammerSource.html and following pages