Yes, 4GB is fine, I believe 512MB would even work right now given the OS size and number of vision modules.
But it is quite desirable to have a class 10 card speed, and those were just not available in less than 8GB in bulk quantities (or the price of the smaller class 10 cards was more than the 8GB price). Even 8GB class 10 is getting hard to find, 16GB is more common.
Have you installed the jevois SDK? If so, you can tweak the numbers in jevois-sdk/jevois-build/jevois-flash-card.sh, as follows:
look for p1siz (size of BOOT partition), keep this one at 16MB.
Then p2siz (size of LINUX partition) can be smaller, 300MB is likely to be ok.
The script will then size the JEVOIS partition to fill up the card.
Follow the steps in INSTALL of jevois to compile all the code and flash it.