FX3 camera board has high sensitivity to USB3 errors

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

We have an FX3 based camera board that is using Bulk transfers to send uncompressed data from a 12 megapixel imager. The design has always been unstable, as the slightest USB error will cause it to crash the camera and/or software. A recent test with a fixed pattern generator in the FPGA gives very similar results, but our USB analyser shows no CRC errors, only the occasional 'Running Disparity Error' which it says are just warnings. However, these warnings still seem to be crashing the link. I would have expected the Bulk mode error correction to deal with this sort of problem, but apparently not. Does anyone have a suggestion as to why the system is so sensitive to errors? Thanks!

0 Likes
1 Solution

Hello Terry,

After some debugging I found that more than 4MB data( ~17MB) is requested by the host application.

Please refer to GetXferSize API in CyApi Programmer's reference or  Xfersize() in CYUSB.NET programmer's reference which mentions maximum transfer size limit set into cyusb3.sys driver.

The reference guide for CyAPI.lib is available at C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\doc\SuiteUSB\CyAPI.pdf and the reference guide for CyUSB.dll is available at C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\doc\SuiteUSB\CyUSB.NET.pdf.

For Bulk transfers cyusb3.sys sets the limit  to 4MB.

To transfer the whole frame you can transfer/requests chunks of 4 MB video data from the device.

Please use the driver shared in this thread Solved: Win10 specific issue in data transfer from an FX3 ... - Cypress Developer Community . The driver is not signed hence can be used when PC in test mode.

Edited: For USB 2.0 (high speed), the maximum BULK transfer supported is 4 MB USB Bandwidth Allocation - Windows drivers | Microsoft Docs.  so the host application should requests for chunks of 4 MB

Regards,
Rashi

View solution in original post

0 Likes
57 Replies

Hello Rashi,

I tried the new version, but had a problem with the new error handler.

// Setup the Callback to Handle the GPIF INTR event
// CyU3PGpifRegisterCallback(CyFxIntrGpifCB);
// backFlowDetected = 0;
// CyU3PPibRegisterCallback(SxVc3AppPibCallback, CYU3P_PIB_INTR_ERROR);

It errored out with an 'undefined' message, so I commented it out for a trial of the firmware. Rather as expected, it ran for a few seconds, but then failed with an overflow. Here is a clip of the UART output:

No Error :0
No Error :0
No Error :0
No Error :0
No Error :0
DMA Reset Event: Commit buffer failure: failure code = 71
CYU3P_PIB_ERR_THR1_WR_OVERRUNDMARESET: APP re initialized = 0
CYU3P_PIB_ERR_THR1_WR_OVERRUNNo Error :0
No Error :0
No Error :0

I couldn't get the software to resume after the error. It just jammed and a restart gave a blank frame.

Regards,

Terry

0 Likes
lock attach
Attachments are accessible only for community members.
Rashi_Vatsa
Moderator
Moderator
Moderator
5 likes given 500 solutions authored 1000 replies posted

Hello Terry,

I had commented out the debug print " No Error" but this is still seen in the UART debug prints.

I have also added a debug print to print only the producer and consumer events. I wanted to check if the data is received properly after the error occurs.

Please import and build the attached firmware and share the complete UART debug prints. 

Also share the USB traces using USB software analyzer (like Wireshark)

 

Regards,
Rashi
0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins
Hi Rashi,
 
OK, I imported the project and loaded it into my test board. Here is the UART output it gave.
I noted that ‘Manual’ DMA operation is currently disabled – is that correct?
 
Thanks!
Regards,
Terry
0 Likes
lock attach
Attachments are accessible only for community members.

Hello Terry,

- Please find the attached firmware with  few more debug prints added.

- The UART debug prints that you shared is with "STREAM_MANUAL" macro enabled.  Please confirm if there are no UART debug prints seen after the ones you shared in the document.

- Also probe and share the USB traces using Wireshark.

- Please confirm if the Build variable of the firmware is 1_3_4

 

Regards,
Rashi
0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

Thanks for the latest version. I can confirm that the Build Variable is 1_3_4.

I tried to use WireShark, but couldn't find out how to make it see the FX3 board - it seems to be heavily biased towards Networks. As an alternative, I ran 'USB Analyser' alongside Termite and got the attached logs. This version fell over very quickly, so there may not be any valid frames in the output.

0 Likes
lock attach
Attachments are accessible only for community members.

Hello Terry,

From the UART debug prints it seems that for the very first producer event (first buffer received) the CyU3PDmaMultiChannelCommitBuffer API failed. To confirm this I have added a debug print in the DMA RESET event handler. Please share the UART debug prints

To capture the USB trace using Wireshark, USBCap needs to be installed while installing Wireshark (Wireshark installer will have an option to install USBCap.

You can refer to this link for more details. Please plug the device to the host after starting the usbcap (in wireshark). Please share the .pcap file after the capture.

Also, let me know the original GPIF state machine (.cyfx) which is used in the firmware "cyfxgpif2config-Fifo.h".  which file in the \SxFX3\GPIF II Designer is used to generate "cyfxgpif2config-Fifo.h". This will help me to modify the state machine as per AN75779. The state machine used in your firmware is  different from the AN75779 state machine.

 

 

 

Regards,

Rashi

Regards,
Rashi
0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

For quickness, I tried the new version without USB monitoring, and it ran much better. I don't know if you know, but the Pclock is set at 24MHz in that version (and most previous ones), so I tried it at both 24 and 98MHz. At 98, it ran, but eventually failed. Here are the UART traces for 24 and 98MHz. I'll get Wireshark running later today and try again.

0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

The original source for the FIFO state machine is in the attached zip file. I hope that it is useful. I also tried Wireshark and got the attached file.

Regards,

Terry

0 Likes
lock attach
Attachments are accessible only for community members.

Hello Terry,

For 24 MHz, from the traces I couldn't see any failure. Please confirm if the streaming is proper (without any error) for 24 MHz PCLK

For 98 MHz, from the UART traces I found that few frames of (18324003 Bytes) are sent and then host completely stops consuming the data i.e. the producer count is  8 (i.e. total number of DMA buffers) and consumer count is 0 which means that none of the DMA buffers are consumed.

In the USB traces, URB_FUNCTION_ABORT_PIPE is sent by the host when transfer fails with USBD_STATUS_BABBLE_DETECTED and after that there is not video streaming.

To solve this we can do the following:

Test 1: To avoid the error I have modified the DMA buffer size (36KB) and the endpoint size (16KB). Please try building the attached firmware and program to FX3 and probe the USB lines (using Wireshark (USBPCap)) for both MANUAL (#define STREAM_MANUAL enabled) and AUTO (#define STREAM_MANUAL disabled or comment out) DMA channel. By default the MANUAL channel is enabled. Also, share the UART Debug print for both cases

Test 2: This error can be recovered  by implementing the steps mentioned in this document on the host application  https://docs.microsoft.com/en-us/windows-hardware/drivers/usbcon/how-to-recover-from-usb-pipe-errors...

Please carry out Test 1 first and let me know the results an later follow the test 2

 

Regards,
Rashi
0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

I tested the firmware at 24MHz Pclock and it ran for a few frames, but the images were distorted (attached frame). I see that the byte count is different (18322512), so this probably accounts for the distortion and may account for the fairly rapid crash. Running Wireshark makes it fall over on the first frame, so it is clearly sensitive to USB events. The attached files show what I got with Wireshark running, so the sequence fails immediately.

0 Likes
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

I'm convinced that using the FPGA FIFO is the best way to fix this problem. We have about 4000 pixels capacity in the FIFO, but it doesn't have any control feedback from the FX3, so it just overflows the DMA buffers and doesn't help to control the data input. There are a few test outputs from the FX3 that could be used to control the FIFO, so it would be great to use one or two of these to halt the FIFO, as necessary.  The FX3 pads B11, C10, E7, E8 and F6 are currently available as test pads.

Regards,

Terry

0 Likes
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hi Rashi, Did you receive my private reply yesterday?

Regards,

Terry

0 Likes

Hello Terry,

The UART debug prints are different every time. Sometimes the error is seen early and sometimes its seem after some time.  Please probe the PCLK, HSYNC(LV) and VSYNC (FV) signals and check if the FPS from the sensor is as per the sensor settings. Please confirm if the FPS of the video is as expected.

Also, the .pcap (USB traces) file is not received for the AUTO channel and MANUAL channel for the latest firmware. Please share the same. This will help me to check if the USBD_STATUS_BABBLE_DETECTED is causing the failure for both the cases. Also let me know the behavior for  for 98 MHz clock.

From your earlier post, the video frame size  is expected to be 4104*3000*12 bits/pixel =  18468000 bytes. But the received frame size is less than expected frame size. The frame size is expected to remain same irrespective of the DMA buffer size. 

Please confirm if you use a hub between the host and device. If yes please try connecting the device directly to the usb host (PC). You can also try connecting Fx3 to another host.

 

Regards,
Rashi
0 Likes
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

I'm using a board that has a test pattern programmed into the FPGA - there is no sensor connected to the board. A Lattice support engineer created the test pattern firmware and so I'm confident that it is operating properly. It uses the Pclock to generate the pattern, so it is synchronous with the FX3. The pattern runs at around 5 frames per second, so it isn't super fast. Just for info. I've used the FX3 firmware (various versions that you have sent) in an operational camera and get very similar failures. 

The pattern frames are 4072 x 3000 pixels, but the software downloads a 4104 x 3000 frame and disposes of the last 32 pixels in each line, so we expect downloads of 18468000 bytes. This is all to do with removing blanking and sync zones  in the images. 

There is a direct connection to the PC (no hub) and we have tried at least 4 different PCs and cables during our testing. The performance varies from 'Instant failure' to successful streaming for several minutes at a time. Using a very short USB3 cable is definitely best, but not a complete cure.

Can you please clarify what you would like me to try with the USB traces? Do you want both Auto and Manual traces? thanks!

Regards,

Terry

0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

I've run Wireshark with the board in both Auto and Manual at 98MHz Pclock. Here are the Wireshark files.

Regards,

Terry

0 Likes
lock attach
Attachments are accessible only for community members.

Hello Terry,

The commit buffers are seen when a USB transaction throws USBD_STATUS_BABBLE_DETECTED error. The USB PIPE is aborted by the USB host dur to which all the other USB IN Transfers are cancelled. This causes the video stream to stop. So the reason for commit buffer failure is  USBD_STATUS_BABBLE_DETECTED  and not the slow host. So adding the flow control flags will not be helpful to recover from this error.

I had seen some differences in the GPIF state machine and I have modified them. Please find the folder with different firmware version. Please build them and test them all an let me know if there is some improvement. Please share the UART debug prints for all of them.

I have configured the PCLK from FX3 to FPGA to 80MHz. Please confirm this by probing the PCLK pin.

Please let me know to which USB  driver  FX3 binds  and which host controller is used for the application.

Regards,
Rashi
0 Likes
lock attach
Attachments are accessible only for community members.
TePl_1066611
Level 3
Level 3
25 replies posted 10 replies posted 10 sign-ins

Hello Rashi,

I've compiled and run most versions - one wouldn't compile because of errors (attached screen shot). I can confirm that the clock is running at 80MHz and the UART outputs are attached for the successful compiles. The version which has the name 'IN_DATA_REMOVED_SETWRP_DISABLED' seems to be a bit more stable and will stream for about 50 seconds before failing. The others fail more quickly - usually at around 10 seconds.

The driver is based on the Cypress driver from 8/11/2017 - version 1.2.3.20 .

The host computer is currently a i7-4790 HP machine running Windows10 Home release 19041.746 in 'Test' mode.

0 Likes

Hello Terry,

After some debugging I found that more than 4MB data( ~17MB) is requested by the host application.

Please refer to GetXferSize API in CyApi Programmer's reference or  Xfersize() in CYUSB.NET programmer's reference which mentions maximum transfer size limit set into cyusb3.sys driver.

The reference guide for CyAPI.lib is available at C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\doc\SuiteUSB\CyAPI.pdf and the reference guide for CyUSB.dll is available at C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\doc\SuiteUSB\CyUSB.NET.pdf.

For Bulk transfers cyusb3.sys sets the limit  to 4MB.

To transfer the whole frame you can transfer/requests chunks of 4 MB video data from the device.

Please use the driver shared in this thread Solved: Win10 specific issue in data transfer from an FX3 ... - Cypress Developer Community . The driver is not signed hence can be used when PC in test mode.

Edited: For USB 2.0 (high speed), the maximum BULK transfer supported is 4 MB USB Bandwidth Allocation - Windows drivers | Microsoft Docs.  so the host application should requests for chunks of 4 MB

Regards,
Rashi
0 Likes