4 Replies Latest reply on Feb 14, 2020 1:04 AM by BragadeeshV_41

    Cannot recover from a failed I2C send start

    BrBr_1320441

      Hi,

       

      I am working with the PSoC 6 BLE module (CYBLE-416045-02) and am having a lot of trouble with the behavior of the I2C bus. Everything works great except for one error I am getting, but it is fatal when it happens. Luckily (and unluckily) I am able to reproduce it with my hardware easily.

       

      I have a custom designed PCB, with a number of I2C peripherals attached. One of them is a familiar audio codec which is used by the cypress TFT shield (the ak4954a), but this happened with a different audio amp previously (for likely the same reason) and this particular device is causing some problems with the I2C bus which I cannot recover from.

       

      The problem is that I keep the audio codec powered down in hardware, and then turn it on (with a gpio connected to an i2c controlled gpio expander) when it is needed.

       

      About 20% of the time, when I power on this amp, my next I2C transaction gets a failure on I2C_MasterSendStart.

       

      The return code on the first failure is always 11149316 (0x00aa2004). After this first failure, every single I2C transaction will receive a failure on I2C_MasterSendStart as well, but with error code 11149314  (0x00aa2002), which basically means the device can no longer function because the i2c bus is in an unusable state.

       

      I am not 100% sure what those two errors mean, but from what I can tell, 0xaa refers to this being an error (CY_PDL_STATUS_ERROR), the 0x02 just indicates the SCB I2C ID (1<<13), and the 0x04 part of the first error is the important bit which indicates that it is a CY_SCB_I2C_MASTER_MANUAL_ADDR_NAK, which means:

       

      "The slave NACKed the address."

       

      After that first error, all of the other errors appear to be from: CY_SCB_I2C_MASTER_NOT_READY which means:

       

      "The master is not ready to start a new transaction.

          * Either the master is still processing a previous transaction or in the

          * master-slave mode, the slave operation is in progress."

       

      Simply doing a cpu reset, not even a power cycle, is sufficient to recover the device operation. Unfortunately that isn't a practical solution in production, so I am hoping to find a way to restart the i2c bus without restarting the CPU bus.

       

      I believe the issue is ultimately caused because the amp is powering up during some other transaction, and causing some contention on the bus when it wakes up mid-transaction. Of course I can try to implement a locking system so that all other threads are locked out from i2c access until the amp is powered up, but I want to also solve the root issue in case this happens randomly.

       

      I am using FreeRTOS on this particular build, and there are at least 4 threads with access to the i2c bus. I implemented a thread-safe i2c accessor method which is definitely sharing the bus safely, so I do not think that is related to the issue.

       

      Thanks for any help!

      Brian

        • 1. Re: Cannot recover from a failed I2C send start
          BrBr_1320441

          As a follow up, I discovered that I was able to recover from the initial failure simply enough by sending a Stop on the bus and then retrying the failed transaction. This is a good solution for now, but I am still interested to know if anyone has any insight as to why I am getting the initial error, and if you believe this is a correct way to go about cleaning up the bus.

           

          Brian

          • 2. Re: Cannot recover from a failed I2C send start
            RodolfoG_11

            Hi Brian,

            I have observed the exact same problem with the AK4954. I could easily reproduce this problem on power up of the CY8CKIT-062-WIFI-4343W + TFT Shield. The only way I found to recover was to perform a software reset when the attempt to init AK4954 fails.

            Thanks for sharing your findings. We will try your method.

            • 3. Re: Cannot recover from a failed I2C send start
              BrBr_1320441

              Thanks for the follow up! I am glad I am not alone in this issue. I am confident sending the Stop will solve your problem if it is the same, since I have now tested it quite rigorously on various hardware.

               

              My i2c access thread simply looks like this for now

               

              for(;;)

                  {

                      //Wait for a transaction request.

                      if(xQueueReceive(transaction_queue, &transaction, portMAX_DELAY) == pdTRUE)

                      {

                          cy_en_scb_i2c_status_t ret = handle_i2c_transaction(&transaction);

                          if (ret != CY_SCB_I2C_SUCCESS){

                              //Send a stop and retry this transaction.

                              I2C_MasterSendStop(transaction.timeout);

                              handle_i2c_transaction(&transaction);

                          }

                          if(transaction.i2cm_complete_sem)

                          {

                              xSemaphoreGive(transaction.i2cm_complete_sem);

                          }

                      }

                  }

               

              Obviously "transaction" is a custom struct with parameters describing the transaction, but I think you get the point.

               

              This very simple solution is working perfectly, I am now getting the first failure as always, but I have recovered from the failure 100% of the time.

               

              However, I would still like to share further information that this is certainly not isolated to the AK4954A. I actually supported a different codec on this hardware previously, and had exactly the same issue when I brought it out of power down. I was using the TLV320AIC3107IRSBR.

               

              I moved away from that amp partly because of this issue. So as it turns out I may have been a bit hasty in my finding that this was an issue with that particular chip (it was also very complicated for my needs though so I don't feel too bad for moving on from it, but it turns out the chip was fine in this regard since this technique solves that problem too.)

               

              I guess my questions now are:

               

              Is this the expected usage for the low level APIs?

              Should we always send the stop on a failed transaction?

              Is there documentation somewhere about how to handle various errors?

              Is it clear why this may be happening? Is it because the slave is taking to long to wake up and coming alive after a transaction started?

              Is there anything else I can do to prevent the initial error from happening at all?

               

              Thanks for further clarification,

              Brian

              • 4. Re: Cannot recover from a failed I2C send start
                BragadeeshV_41

                Hi BrBr_1320441,

                 

                The I2C master will send a start condition only when the bus is free. The bus busy flag is set by hardware whenever a start condition is received on the bus. The busy flag is reset upon a stop condition. But when there is a false start condition in the bus due to noise/ ESD event, this will be registered as a start condition from the I2C master hardware.  The hardware can not differentiate between a false start condition and an actual condition. It is the upper firmware layer that needs to detect this error and reset the I2C component.

                 

                If you are the only master in the bus, and you receive a bus busy error, it might be due to a false start condition. If that is the case you can simply restart the I2C hardware by calling the Cy_SCB_I2C_Disable and Cy_SCB_I2C_Enable functions one after the other.

                 

                If you have multiple masters in the bus, then you have to set a timeout like after x failed transactions reset the I2C component.

                 

                If the slave is not ready to receive the data or is in sleep mode/ turned OFF, the master will see a NACK and not a bus busy flag due to the pull up network.

                 

                To see if there is any false start condition, probe the I2C lines and see if you see any glitch in the bus when the slave device is powering up.

                 

                Regards,

                Bragadeesh

                1 of 1 people found this helpful