I'm not sure about the file extension. Wouldn't this depend on what the FPGA would want to read from the SPI flash?
The Fx3's function here is only to transfer the file from the PC to the SPI flash. So as far as the Fx3/USB_interface is concerned, the file type does not really matter.
The 4kB limit for control transfers is because of a lack of buffer availability in Fx3 for EP0.
If you want to transfer the whole file in one shot, you could first send a vendor request to setup the SPI etc. in Fx3 firmware, and then send out the whole file through a bulk out endpoint. In firmware, once you get a buffer of data, you can start the Spi writes and continue until you have no more data coming in from that endpoint.