1 2 Previous Next 24 Replies Latest reply on Mar 26, 2020 1:20 AM by YashwantK_46

    How to debug device stall

    WGT_4383351

      When use Master/Slave example in AN87216 to constantly transfer data between two CYUSB3KIT-003 (from master to slave) with two custom C# programs, sometimes (sometimes after seconds, sometimes after a very long period) the transfer stalls - at this time, even with Control Center, the slave bulk IN and bulk OUT endpoints doesn't respond (timeout after 2secs) , control endpoint OUT return error code 997 immediately.

       

      The problem is not in the C# program since it can work for very long time and Control Center can't send/receive data too after the pair is dead.

       

      The master also doesn't respond to OUT but the reason may be buffer full after slave "dead".

      The master doesn't respond to IN but the reason is at least that there is no data in the master IN buffer since only the slave can send data to master but in the experiment only master send data to slave, and after the slave is dead, there is noway to send data to the master.

       

      Abort Pipe and Reset Pipe button in Control Center doesn't resolve the problem, only device reset works.

       

      There is no serial port message since there is no code that send messages - where to insert some print code?

      How to debug this situation?

      How to know what is the status of the slave device?

      How to bring the slave back as fast as possible?

      Could there be any bugs in the firmware?

        • 1. Re: How to debug device stall
          YashwantK_46

          Hello,

           

          Can you please share your test procedure as to when you see the stalling issue?

          Also, how much data are you sending and how many times are you transferring the data?

           

           

          The slavefifo example has default UART debug prints mentioning the PROD_COUNT in both U-port to P-port and P-port to U-port transfers.

          Can you please add two more global variables in the firmware and increment the variables in CY_U3P_DMA_CB_CONS_EVENT, one in CyFxSlFifoUtoPDmaCallback and the other one in CyFxSlFifoPtoUDmaCallback?

           

          After incrementing the variables, in SlFifoAppThread_Entry,  add the variables in the debugprint call as below:

          if (glIsApplnActive)

                  {

                      /* Print the number of buffers received so far from the USB host. */

                      CyU3PDebugPrint (6, "Data tracker: buffers received: %d, buffers sent: %d. cons count(tx): %d cons count(rx): %d\n",

                              glDMARxCount, glDMATxCount, cons_flag_in_PtoU, cons_flag_in_UtoP);

                  }

           

          This can help in understanding if there is any difference in the prod count and the cons count on the master and slave side as well.

          Please add the above snippet of code in both the master and slave and share the debug logs with me.

           

          Also, we can analyze this issue better if you can take USB traces using a hardware analyzer or a software analyzer software such as Wireshark and see if the device is sending NAK's?

           

          Regards,
          Yashwant

          • 2. Re: How to debug device stall
            WGT_4383351

            "Also, how much data are you sending and how many times are you transferring the data?"

             

            Constanly sending data from master to slave at >2Gbps for minutes.

             

            "This can help in understanding if there is any difference in the prod count and the cons count on the master and slave side as well"

            "take USB traces using a hardware analyzer or a software analyzer software such as Wireshark and see if the device is sending NAK's"

             

            We have no USB analyzer. Shouldn't the firmware recover if any data is loss? The firmware shouldn't be dead even data are loss. How to know what is the exact status of the firmware when it is dead looking?

            • 3. Re: How to debug device stall
              WGT_4383351

              Another clue is, only for once the transfer process keep working for minutes at ~2.5Gbps, for other cases it stall soon after the transfer begin.

              • 4. Re: How to debug device stall
                YashwantK_46

                Hello,


                Can you please register for USB event so that we can see if there's any SS_RESET events in the firmware?

                 

                You can go through gpiftousb example found in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxgpiftousb

                 

                Please register for the USB events like: CyU3PUsbRegisterEpEvtCallback (CyFxApplnEpCallback, 0x1B0, 0x04, 0x06);

                 

                And add the corresponding callback function that's mentioned in the firmware which handles the SS_RESET_EVENT in the callback

                Please try a similar implementation in your firmware and share the UART debug logs with me.


                Also, we would need Wireshark traces for the USB transactions. Please take the traces and share it with me.

                Without the traces, its not possible to understand what is causing the issue.

                 

                Link for Wireshark: Wireshark · Download

                 

                Regards,

                Yashwant

                • 5. Re: How to debug device stall
                  WGT_4383351

                  Should I use WireShark or https://desowin.org/usbpcap/ ?

                  • 6. Re: How to debug device stall
                    WGT_4383351

                    I guess all the problem about Master/Slave example is these examples are only tested with send/receive 1 packet in Control Center as described in AN87216. The example are not ready for continuous data transfer.

                     

                    Can you test the example with any continuous data transfer and publish a working version?

                    • 7. Re: How to debug device stall
                      WGT_4383351

                      With WireShark, the related log is:

                       

                      No. Time Source Destination Protocol Length Info

                      309 38.086140 host 3.5.1 USB 27 URB_BULK in

                      310 38.186795 host 3.5.1 USB 27 URB_FUNCTION_ABORT_PIPE

                       

                      why URB_FUNCTION_ABORT_PIPE?

                      • 8. Re: How to debug device stall
                        WGT_4383351

                        Here is the log around the dead:

                         

                        No. Time Source Destination Protocol Length Info

                        132971 110.006909 3.23.1 host USB 27 URB_BULK out

                        132972 110.006967 3.22.1 host USB 1048603 URB_BULK in

                        132973 110.007125 host 3.23.1 USB 1048603 URB_BULK out

                        132974 110.007222 host 3.22.1 USB 27 URB_BULK in

                        132975 110.009897 3.23.1 host USB 27 URB_BULK out

                        132976 110.009930 3.22.1 host USB 1048603 URB_BULK in

                        132977 110.010055 host 3.23.1 USB 1048603 URB_BULK out

                        132978 110.010129 host 3.22.1 USB 27 URB_BULK in

                        132979 110.014618 3.22.1 host USB 342043 URB_BULK in

                        132980 110.014836 host 3.22.1 USB 27 URB_BULK in

                        132981 110.109949 host 3.23.1 USB 27 URB_FUNCTION_ABORT_PIPE

                        132982 110.110115 3.23.1 host USB 27 URB_BULK out

                        132983 110.110177 3.23.1 host USB 27 URB_FUNCTION_ABORT_PIPE

                        132984 110.112476 host 3.23.1 USB 1048603 URB_BULK out

                        132985 110.212353 host 3.23.1 USB 27 URB_FUNCTION_ABORT_PIPE

                        132986 110.212489 3.23.1 host USB 27 URB_BULK out

                        132987 110.212555 3.23.1 host USB 27 URB_FUNCTION_ABORT_PIPE

                        132988 110.214518 host 3.23.1 USB 1048603 URB_BULK out

                         

                        there is a IN packet with smaller size, what does this mean the problem could be?

                        • 9. Re: How to debug device stall
                          YashwantK_46

                          Hello,


                          Thank you for sharing the Wireshark traces. I will review it on my end.


                          In my response no. 4, as i have suggested, have you implemented the CyU3PUsbRegisterEpEvtCallback() function in your firmware?


                          Along with the Wireshark traces, could you please share UART debug logs with the CyU3PUsbRegisterEpEvtCallback() added in your firmware?

                          This is important to understand what might be causing the issues on the device side.

                           

                          Can you please register for USB event so that we can see if there's any SS_RESET events in the firmware?

                           

                          You can go through gpiftousb example found in the SDK path: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\basic_examples\cyfxgpiftousb

                           

                          Regards,

                          Yashwant

                           

                          • 10. Re: How to debug device stall
                            WGT_4383351

                            More experiments shows that when dead there is always a short packet, example are 334K/302K/63K.

                            Some problem should have happened during transfering of an URB, can the driver print the related information?

                            • 11. Re: How to debug device stall
                              YashwantK_46

                              Hello,

                               

                              Can you please provide the UART debug logs after implementing the CyU3PUsbRegisterEpEvtCallback() as i have requested in the responses 4 and response 9?


                              I would need to go through it to understand the issue and assist you better further.

                              The short packet can also be related to the issue of URB_FUNCTION_ABORT_PIPE in the USB transactions.

                              Please provide the needful.

                               

                              Regards,

                              Yashwant

                              • 12. Re: How to debug device stall
                                WGT_4383351

                                URB_FUNCTION_ABORT_PIPE seems to be normal.

                                • 13. Re: How to debug device stall
                                  WGT_4383351

                                  BTW there is no "CyU3PMemSet ((uint8_t *)&dmaCfg, 0, sizeof (dmaCfg));" in Slave firmware, although every field is initialized later, future extention of that structure may introduce bugs.

                                  • 14. Re: How to debug device stall
                                    WGT_4383351

                                    Here is the serial log:

                                     

                                    ▒▒EP Event: ep=81 event=16

                                    ▒▒EP Event: ep=81 event=16

                                    ▒▒EP Event: ep=81 event=16

                                    ▒▒

                                    P Event: ep=81 event=256

                                    ▒▒Halting USB Consumer EP: 0

                                     

                                    1.There are CYU3P_USBEP_SS_RETRY_EVT every a few seconds during normal condition

                                    2.CYU3P_USBEP_SS_RESET_EVT causes the dead, how to recover? Why this event is sent?

                                    3.Why there are serial glitches when printing the dead declaration?

                                    1 2 Previous Next