How to debug device stall

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
WGT_4383351
Level 5
Level 5
First like received

When use Master/Slave example in AN87216 to constantly transfer data between two CYUSB3KIT-003 (from master to slave) with two custom C# programs, sometimes (sometimes after seconds, sometimes after a very long period) the transfer stalls - at this time, even with Control Center, the slave bulk IN and bulk OUT endpoints doesn't respond (timeout after 2secs) , control endpoint OUT return error code 997 immediately.

The problem is not in the C# program since it can work for very long time and Control Center can't send/receive data too after the pair is dead.

The master also doesn't respond to OUT but the reason may be buffer full after slave "dead".

The master doesn't respond to IN but the reason is at least that there is no data in the master IN buffer since only the slave can send data to master but in the experiment only master send data to slave, and after the slave is dead, there is noway to send data to the master.

Abort Pipe and Reset Pipe button in Control Center doesn't resolve the problem, only device reset works.

There is no serial port message since there is no code that send messages - where to insert some print code?

How to debug this situation?

How to know what is the status of the slave device?

How to bring the slave back as fast as possible?

Could there be any bugs in the firmware?

0 Likes
1 Solution

Hello Tom,

Apologies, there was a typing mistake by me.

Please go through Section 2.3 (IV) of the FX3_SDK_TroubleShooting_Guide.pdf provided in the SDK that explains about the CYU3P_USBEP_SS_RESET_EVT to get a better understanding on how to handle the request.

In cyu3usb.h, the events are explained as:

     CYU3P_USBEP_SS_RETRY_EVT          /**< An ACK TP with the RETRY bit was received on the IN endpoint. */

     CYU3P_USBEP_SS_RESET_EVT          /**< USB 3.0 endpoint reset event. A stall recovery is required. */

These are the descriptions of the USB events.

The handling on the CYU3P_USBEP_SS_RETRY_EVT has been taken care in the cyfxbulksrcsink example provided in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxbulksrcsink

The handling on the CYU3P_USBEP_SS_RESET_EVT has been taken care in the cyfxbulksrcsink example provided in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxgpiftousb

Regards,

Yashwant

View solution in original post

0 Likes
24 Replies
YashwantK_46
Moderator
Moderator
Moderator
100 solutions authored 50 solutions authored 50 likes received

Hello,

Can you please share your test procedure as to when you see the stalling issue?

Also, how much data are you sending and how many times are you transferring the data?

The slavefifo example has default UART debug prints mentioning the PROD_COUNT in both U-port to P-port and P-port to U-port transfers.

Can you please add two more global variables in the firmware and increment the variables in CY_U3P_DMA_CB_CONS_EVENT, one in CyFxSlFifoUtoPDmaCallback and the other one in CyFxSlFifoPtoUDmaCallback?

After incrementing the variables, in SlFifoAppThread_Entry,  add the variables in the debugprint call as below:

if (glIsApplnActive)

        {

            /* Print the number of buffers received so far from the USB host. */

            CyU3PDebugPrint (6, "Data tracker: buffers received: %d, buffers sent: %d. cons count(tx): %d cons count(rx): %d\n",

                    glDMARxCount, glDMATxCount, cons_flag_in_PtoU, cons_flag_in_UtoP);

        }

This can help in understanding if there is any difference in the prod count and the cons count on the master and slave side as well.

Please add the above snippet of code in both the master and slave and share the debug logs with me.

Also, we can analyze this issue better if you can take USB traces using a hardware analyzer or a software analyzer software such as Wireshark and see if the device is sending NAK's?

Regards,
Yashwant

0 Likes
WGT_4383351
Level 5
Level 5
First like received

"Also, how much data are you sending and how many times are you transferring the data?"

Constanly sending data from master to slave at >2Gbps for minutes.

"This can help in understanding if there is any difference in the prod count and the cons count on the master and slave side as well"

"take USB traces using a hardware analyzer or a software analyzer software such as Wireshark and see if the device is sending NAK's"

We have no USB analyzer. Shouldn't the firmware recover if any data is loss? The firmware shouldn't be dead even data are loss. How to know what is the exact status of the firmware when it is dead looking?

0 Likes
WGT_4383351
Level 5
Level 5
First like received

Another clue is, only for once the transfer process keep working for minutes at ~2.5Gbps, for other cases it stall soon after the transfer begin.

0 Likes

Hello,


Can you please register for USB event so that we can see if there's any SS_RESET events in the firmware?

You can go through gpiftousb example found in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxgpiftousb

Please register for the USB events like: CyU3PUsbRegisterEpEvtCallback (CyFxApplnEpCallback, 0x1B0, 0x04, 0x06);

And add the corresponding callback function that's mentioned in the firmware which handles the SS_RESET_EVENT in the callback

Please try a similar implementation in your firmware and share the UART debug logs with me.


Also, we would need Wireshark traces for the USB transactions. Please take the traces and share it with me.

Without the traces, its not possible to understand what is causing the issue.

Link for Wireshark: Wireshark · Download

Regards,

Yashwant

0 Likes
WGT_4383351
Level 5
Level 5
First like received

Should I use WireShark or https://desowin.org/usbpcap/ ?

0 Likes
WGT_4383351
Level 5
Level 5
First like received

I guess all the problem about Master/Slave example is these examples are only tested with send/receive 1 packet in Control Center as described in AN87216. The example are not ready for continuous data transfer.

Can you test the example with any continuous data transfer and publish a working version?

0 Likes
WGT_4383351
Level 5
Level 5
First like received

With WireShark, the related log is:

No. Time Source Destination Protocol Length Info

309 38.086140 host 3.5.1 USB 27 URB_BULK in

310 38.186795 host 3.5.1 USB 27 URB_FUNCTION_ABORT_PIPE

why URB_FUNCTION_ABORT_PIPE?

0 Likes
WGT_4383351
Level 5
Level 5
First like received

Here is the log around the dead:

No. Time Source Destination Protocol Length Info

132971 110.006909 3.23.1 host USB 27 URB_BULK out

132972 110.006967 3.22.1 host USB 1048603 URB_BULK in

132973 110.007125 host 3.23.1 USB 1048603 URB_BULK out

132974 110.007222 host 3.22.1 USB 27 URB_BULK in

132975 110.009897 3.23.1 host USB 27 URB_BULK out

132976 110.009930 3.22.1 host USB 1048603 URB_BULK in

132977 110.010055 host 3.23.1 USB 1048603 URB_BULK out

132978 110.010129 host 3.22.1 USB 27 URB_BULK in

132979 110.014618 3.22.1 host USB 342043 URB_BULK in

132980 110.014836 host 3.22.1 USB 27 URB_BULK in

132981 110.109949 host 3.23.1 USB 27 URB_FUNCTION_ABORT_PIPE

132982 110.110115 3.23.1 host USB 27 URB_BULK out

132983 110.110177 3.23.1 host USB 27 URB_FUNCTION_ABORT_PIPE

132984 110.112476 host 3.23.1 USB 1048603 URB_BULK out

132985 110.212353 host 3.23.1 USB 27 URB_FUNCTION_ABORT_PIPE

132986 110.212489 3.23.1 host USB 27 URB_BULK out

132987 110.212555 3.23.1 host USB 27 URB_FUNCTION_ABORT_PIPE

132988 110.214518 host 3.23.1 USB 1048603 URB_BULK out

there is a IN packet with smaller size, what does this mean the problem could be?

0 Likes

Hello,


Thank you for sharing the Wireshark traces. I will review it on my end.


In my response no. 4, as i have suggested, have you implemented the CyU3PUsbRegisterEpEvtCallback() function in your firmware?


Along with the Wireshark traces, could you please share UART debug logs with the CyU3PUsbRegisterEpEvtCallback() added in your firmware?

This is important to understand what might be causing the issues on the device side.

Can you please register for USB event so that we can see if there's any SS_RESET events in the firmware?

You can go through gpiftousb example found in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxgpiftousb

Regards,

Yashwant

0 Likes
WGT_4383351
Level 5
Level 5
First like received

More experiments shows that when dead there is always a short packet, example are 334K/302K/63K.

Some problem should have happened during transfering of an URB, can the driver print the related information?

0 Likes

Hello,

Can you please provide the UART debug logs after implementing the CyU3PUsbRegisterEpEvtCallback() as i have requested in the responses 4 and response 9?


I would need to go through it to understand the issue and assist you better further.

The short packet can also be related to the issue of URB_FUNCTION_ABORT_PIPE in the USB transactions.

Please provide the needful.

Regards,

Yashwant

0 Likes
WGT_4383351
Level 5
Level 5
First like received

URB_FUNCTION_ABORT_PIPE seems to be normal.

0 Likes
WGT_4383351
Level 5
Level 5
First like received

BTW there is no "CyU3PMemSet ((uint8_t *)&dmaCfg, 0, sizeof (dmaCfg));" in Slave firmware, although every field is initialized later, future extention of that structure may introduce bugs.

0 Likes
WGT_4383351
Level 5
Level 5
First like received

Here is the serial log:

▒▒EP Event: ep=81 event=16

▒▒EP Event: ep=81 event=16

▒▒EP Event: ep=81 event=16

▒▒

P Event: ep=81 event=256

▒▒Halting USB Consumer EP: 0

1.There are CYU3P_USBEP_SS_RETRY_EVT every a few seconds during normal condition

2.CYU3P_USBEP_SS_RESET_EVT causes the dead, how to recover? Why this event is sent?

3.Why there are serial glitches when printing the dead declaration?

0 Likes

Hello,

Regarding respone 14:

Please go through Section 2.3 (IV) of the FX3_SDK_TroubleShooting_Guide.pdf provided in the SDK that explains about the CYU3P_USBEP_SS_RETRY_EVT to get a better understanding.

The path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\doc\firmware\FX3_SDK_TroubleShooting_Guide.pdf

Also, CYU3P_USBEP_SS_RESET_EVT and CYU3P_USBEP_SS_RETRY_EVT are set by the library due to various link errors and protocol errors as explained in the above document.

Please read through it to get a better understanding about the issue.


Regards,

Yashwant

0 Likes

Yashwant,

Regarding your response 17, you noted that:

Section 2.3 (IV) of the FX3_SDK_TroubleShooting_Guide.pdf provided in the SDK explains about the CYU3P_USBEP_SS_RETRY_EVT

I have version 1.3.4 of that Guide. I looked through that section, and then searched through the Guide, and did not find any description for handling the CYU3P_USBEP_SS_RETRY_EVT.

Please provide details on handling the CYU3P_USBEP_SS_RETRY_EVT.

Thanks,

Tom

0 Likes

Hello Tom,

Apologies, there was a typing mistake by me.

Please go through Section 2.3 (IV) of the FX3_SDK_TroubleShooting_Guide.pdf provided in the SDK that explains about the CYU3P_USBEP_SS_RESET_EVT to get a better understanding on how to handle the request.

In cyu3usb.h, the events are explained as:

     CYU3P_USBEP_SS_RETRY_EVT          /**< An ACK TP with the RETRY bit was received on the IN endpoint. */

     CYU3P_USBEP_SS_RESET_EVT          /**< USB 3.0 endpoint reset event. A stall recovery is required. */

These are the descriptions of the USB events.

The handling on the CYU3P_USBEP_SS_RETRY_EVT has been taken care in the cyfxbulksrcsink example provided in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxbulksrcsink

The handling on the CYU3P_USBEP_SS_RESET_EVT has been taken care in the cyfxbulksrcsink example provided in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxgpiftousb

Regards,

Yashwant

0 Likes
WGT_4383351
Level 5
Level 5
First like received

When dead, resetting that endpoint doesn't solve the problem, the device is still dead.

0 Likes
WGT_4383351
Level 5
Level 5
First like received

1. Why CYU3P_USBEP_SS_RESET_EVT is sent? This seems to be not normal since reset the endpoint takes seconds which make the device not responding to the user. How to debug the reason for CYU3P_USBEP_SS_RESET_EVT ?

2. How to know a deviced is dead in C# program?

CyUSBDevice.UsbdStatusString(inEndpoint.UsbdStatus);

returns "[state=HALTED status=UNKNOWN]" whenever inEndpoint.XferData fails, even the device is not dead, like when the master is not running and the slave inEndpoint.XferData timeout.

0 Likes

Hello,

Regarding the C# application.

The UsbdStatus  is a member of CyUSBEndPoint, indicating the status of the respective endpoint and the  UsbdStatusString( ) function will return  "SUCCESS",  "PENDING",  "STALLED", or  "ERROR". Please refer to the CyUSB.NET.pdf document. I hope you are receiving "STALLED".

This is because the endpoint is stalled (indicating an endpoint error), you can reset the endpoint when this occurs using the Reset( ) method or clear the stall in the firmware by sending a vendor command to do so.

Thanks,

Yatheesh

0 Likes
WGT_4383351
Level 5
Level 5
First like received

The problem seems to be CyFxBulkLpApplnUSBSetupCB in Slave example:

if ((bTarget == CY_U3P_USB_TARGET_ENDPT) && (bRequest == CY_U3P_USB_SC_CLEAR_FEATURE)

                && (wValue == CY_U3P_USBX_FS_EP_HALT))

        {

            if ((wIndex == CY_FX_EP_PRODUCER) || (wIndex == CY_FX_EP_CONSUMER))

            {

                if (glIsApplnActive)

                {

                CyU3PDebugPrint (4, "CyFxBulkLpApplnUSBSetupCB: CY_U3P_USB_TARGET_ENDPT-CY_U3P_USB_SC_CLEAR_FEATURE-CY_U3P_USBX_FS_EP_HALT\r\n");

                   CyU3PDmaChannelReset (&glChHandleBulkLpUtoP);

                    CyU3PUsbFlushEp(CY_FX_EP_PRODUCER);

                    CyU3PUsbFlushEp(CY_FX_EP_CONSUMER);

                    CyU3PUsbResetEp (CY_FX_EP_PRODUCER);

                    CyU3PUsbResetEp (CY_FX_EP_CONSUMER);

                    CyU3PDmaChannelSetXfer (&glChHandleBulkLpUtoP, CY_FX_DMA_TX_SIZE);

                    CyU3PUsbStall (wIndex, CyFalse, CyTrue);

                    isHandled = CyTrue;

                }

            }

        }

The existing code reset only UtoP DMA channel on both endpoint's clear feature request. It should reset the DMA of the requested endpoint. Please fix the bug in this problematic official example.

0 Likes
WGT_4383351
Level 5
Level 5
First like received

Now the problem become that after dead the master can't be reset. The C# reset code is like:

private void button2_Click(object sender, EventArgs e)

        {

            var status = CyUSBDevice.UsbdStatusString(outEndpoint.UsbdStatus);

            if (status.Contains("state=HALTED"))

            {

                bool b = outEndpoint.Reset();

                Debug.Assert(b);

                stReset++;

            }

        }

after click the button the master is still HALTED and each time the button is pressed, serial port will print message that the CLEAR_FEATURE event is received by the master on the PRODUCER endpoint which meas the CyU3PDmaChannelReset/CyU3PUsbFlushEp/CyU3PUsbResetEp/CyU3PDmaChannelSetXfer/CyU3PUsbStall will be executed on the glChHandleBulkLpUtoP/CY_FX_EP_PRODUCER by the master.

0 Likes
WGT_4383351
Level 5
Level 5
First like received

Yes, Yashwant why are you talking about CYU3P_USBEP_SS_RETRY_EVT?

0 Likes

Hello,

Apologies, i meant the CYU3P_USBEP_SS_RESET_EVT  and not CYU3P_USBEP_SS_RETRY_EVT.

Please go through Section 2.3 (IV) of the FX3_SDK_TroubleShooting_Guide.pdf provided in the SDK that explains about the CYU3P_USBEP_SS_RESET_EVT to get a better understanding on how to handle the request.

Regards,

Yashwant

0 Likes