Question: FX2LP has clearly committed a buffer of size of 512 bytes. When host requests a packet of size smaller than 512 bytes, the transfer fails. Why?
Answer: The USBD_STATUS_BUFFER_OVERRUN (0xC000000C) error code is set by the USB stack drivers when the device transfers more data than is requested by the host. This is what is happening when the device transmits a 512 byte packet in response to an 'IN' request of packet length less than 512.
For IN transfers, USBDI requires a data buffer of "expected size" at each IN request call (URB or USB Request Block). When the incoming bytes from device exceed this expected size, USBD_STATUS_BUFFER_OVERRUN occurs.
The only solution for this is to either request 512 bytes each time, or to take care in the firmware that only the short packet is committed to the host using the INPKTEND pin or by writing into the INPKTEND register.