BLE Stack/Controller Hanging in DeepSleep, CY_BLE_ERROR_INSUFFICIENT_RESOURCES

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
AdHi_2630466
Level 3
Level 3
First like received

Hi All,

I have two projects using the PSOC 63 (CYBLE-416045-02) modules, as Central and Peripheral.

Both devices use Dual Core BLE, Controller on CM0+ and Host on CM4.

Peripheral is a battery operated device and must use DeepSleep when not active.

I have ran extensive testing (~120hrs continuous) on the communications keeping the Peripheral "awake", and unable to reproduce the issue.

Peripheral handles incoming data (write without response from central) from BLE AppCallBack, and sends Notification right the way with data already available.

What is happening:
- Communication runs fine (7.5ms intervals, 3x16byte packets every 8ms + 16byte every 1sec) in DeepSleep for seemingly random periods

- Anywhere between 5mins - 5hrs after, the Peripheral BLE Stack or Controller hangs somehow

- Central Stack stays CY_BLE_STACK_STATE_BUSY, and will not become FREE, unless I issue Cy_BLE_GAP_Disconnect(), to get back to "Scanning State"

- Peripheral still thinks it is connected, and CM4 goes on doing it's 500ms supervisory tasks in DeepSleep.

- Peripheral functions fine for the rest of the code, I can wake it up, go back to sleep again, and manually trigger BLE operations that fail.

- Attaching to CM0+ Controller core shows nothing suspicious, it only has Cy_BLE_ProcessEvents() and Cy_SysPm_CpuEnterDeepSleep(CY_SYSPM_WAIT_FOR_INTERRUPT), breakpoints here are constantly hit.

After Peripheral hangs, I tried:

- Send Notification - After about 2-3 notifications peripheral reports "CY_BLE_STACK_STATE_BUSY" and stays that way

- Cy_BLE_GAP_Disconnect() reports CY_BLE_ERROR_INSUFFICIENT_RESOURCES every time

Some of the things I tried to narrow things down and help the issue, with no luck:

- Increased Stack and Heap size on peripheral to 1600/1600 and 800/800 on Central

- Disabled debugging printf() on peripheral

- All kinds of ways and places to process received data, process data to be transmitted, call Cy_BLE_ProcessEvents(), Interrupt priorities

Anyone has encountered similar problems in DeepSleep?
How can I effectively debug what is happening under the hood of the BLE component?

Any way to recover from such a state?

Any pointers would be greatly appreciated.

Regards,

Adam

0 Likes
1 Solution

After some extended testing I can confidently say the odd hanging problem was caused by the Timer ISR, executing BLE related tasks exactly between Cy_BLE_ProcessEvents() and Cy_SysPm_CpuEnterDeepSleep(). This was hard to reproduce due to the very small time window of this happening, and the very slow (1Hz) Timer ISR.

I am still getting Stack Busy states when the signal levels are low, but I think I will be able to manage it and throttle communications when it falls behind.

Ganesh and Nazar, thank you very much for your guidance, your time and efforts in the process are greatly appreciated!

View solution in original post

0 Likes
24 Replies
VenkataD_41
Moderator
Moderator
Moderator
750 replies posted 500 replies posted 250 solutions authored

Hi Adam,

1. Before sending a Notification from the Server or before Writing a data from the client are you checking whether the stack is busy or not as follows?

        if(charNotificationEnabled == true)

        {

            if(Cy_BLE_GATT_GetBusyStatus(appConnHandle.attId) ==\

               CY_BLE_STACK_STATE_FREE)

            {

                /* Send notification data to the GATT Client*/

                SendNotification();

            }

        }

2. Please check the firmmware flow of the BLE_Find me project to use the DeepSleep APIs in your project. You can see the above code in BLE Throughput code example. You can rever SERVER and CLIENT projects for the usage of

https://www.cypress.com/documentation/code-examples/ce222046-psoc-6-mcu-bluetooth-low-energy-ble-con...

3. You can try removing the DeepSleep call in one project and check if you are able to resolve. This will give an idea on which project is causing the issue.

If checking the above three points doesn't resolve the issue, please attach your project once to check at our side.

Thanks

Ganesh

0 Likes

Thank you Ganesh,

1. Yes, I do check for Stack BUSY status before sending:

        if (Cy_BLE_GATT_GetBusyStatus(connHandle.attId) == CY_BLE_STACK_STATE_FREE) {

            apiResult = Cy_BLE_GATTS_Notification(&uartTxDataNtf);

            //Cy_BLE_ProcessEvents();

           

            if(apiResult == CY_BLE_SUCCESS){           

                // Clear the TX buffer

                BLE_UARTs.tx_index = 0;

                for(uint8 i=0; i<sizeof(BLE_UARTs.tx_buffer); i++){

                    BLE_UARTs.tx_buffer=0;

                }

               

            }          

            else if(apiResult != CY_BLE_SUCCESS){

                DBG_PRINTF("Cy_BLE_GATTS_Notification API Error: 0x%x ", apiResult);

                if (apiResult == CY_BLE_ERROR_INVALID_PARAMETER){

                    DBG_PRINTF("INVALID_PARAMETER");

                }

                else if (apiResult == CY_BLE_ERROR_MEMORY_ALLOCATION_FAILED){

                    DBG_PRINTF("MEMORY ALLOCATION FAILED");  

                    BLE_app.mem_alloc_fail++;

                }

                DBG_PRINTF("\r\n");

            } 

        }

        else {

            DBG_PRINTF("BLE_UART_Send_TX_Data Error - CY_BLE_STACK_STATE_BUSY\r\n");

        }

2. I have used those examples as references, and I will study them again

3. There is no DeepSleep in the Client/Central device. It's BLE will hang in the CY_BLE_STACK_STATE_BUSY state, I have to run a timer to detect it hanging. When detected I manually issue Cy_BLE_GAP_Disconnect(), which succeeds and the Client will start scanning again. This makes me confident that the Client is not the problem.

Some additional details:
- After the client disconnects, I manually try to send a notification from the peripheral. I can issue the Cy_BLE_GATTS_Notification() successfully a 5 times, before the stack becomes busy. How many times I can send the notification varies between ~2-5 every time.

See debug printout:

WDT CTR0 Inactivity Interrupt

Battery Level Update: 18881= 3721mV, 52%

Charge Status ADC: 136, State: 0, NO CHARGE

Activity Flags: 0x0

Wake Up Interrupt

Exit Inactive State

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

CY_BLE_EVT_STACK_BUSY_STATUS: 1

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

BLE_UART_Send_TX_Data Error - CY_BLE_STACK_STATE_BUSY

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

BLE_UART_Send_TX_Data Error - CY_BLE_STACK_STATE_BUSY

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

BLE_UART_Send_TX_Data Error - CY_BLE_STACK_STATE_BUSY

0 Likes

Read through the app notes and example projects (FindMe and BLE Throughput) and made some changes:


About the app notes:
- BLE Throughput: This example only has one-way traffic, from Server to Client. This gave me an idea to try, see below.

- Find Me: This example uses the pre-defined Immediate Alert Service with it's own library and callbacks.
My projects use a Custom Service. This service doesn't have it's own Init() and CallBack() functions, but handled from the generic BLE Event Handler defined by Cy_BLE_Start(AppCallBack).
This Custom Service is ported from PSoC 4 example: https://www.cypress.com/blog/problem-solver/project-020-uart-bluetooth-low-energy-bridge

What I did:

I stopped the Client/Host sending out WriteWithoutResponse() updates, and let the Server/Peripheral send Notifications at it's discretion. Server sends Notifications (3x16byte packets) in it's 500ms supervisory WDT WakeUp interrupt. Client only sends WriteWithoutResponse() if it needs to wake up the Server (manually triggered for testing).

So far this has been running all day without any problems, but I'm also not sending data nearly as often as before (every 500ms vs 8ms), so I'm going to wait until tomorrow and see if I can isolate the problem.

If I don't get a BLE hang, I think we can safely isolate the issue to the Client's WriteWithoutResponse() transmission to the Server when in DeepSleep.

After running for about a day in DeepSleep (with short interruptions due to other developments) as described above, I wasn't able to reproduce the issue.

I am now pretty confident the Server's BLE stack/controller gets lost in the weeds due to the WriteWithoutResponse() from the Client. The random nature of the symptoms point to a timing related issue, perhaps receiving data just in the wrong time (Server in the middle of going to DeepSleep)?

I will try to see if the frequency of these WriteWithoutResponse() calls has any effect on the problem, expecting that lowering the frequency will need more time to fail.

I'm looking to figure out how I can prevent this from happening, and also if it ever happens, how I can detect and recover from it on the Server side.

0 Likes

Hi,

That is a great observation. We will look into the possible issues of calling WriteWithoutResponse() when Server is in DeepSleep().

Can you use normal write instead of WriteWithoutResponse()? (if this doesn't affect your application)

Print the error code in the CY_BLE_EVT_GATTS_WRITE_CMD_REQ event everytime when the server received it. The eventParam that comes with event is of the type cy_stc_ble_gatts_write_cmd_req_param_t . Based on this we can debug the rootcause.

Thanks

Ganesh

0 Likes

Thank you Ganesh,


I am seeing rather interesting results trying to print. I added the condition to only print when in DeepSleep not to get overwhelmed in normal operation.

        case CY_BLE_EVT_GATTS_WRITE_CMD_REQ:

            {

                cy_stc_ble_gatts_write_cmd_req_param_t* uartRxDataWrReg;

                if (Device.Activity_State == INACTIVE){

                    DBG_PRINTF("CY_BLE_EVT_GATTS_WRITE_CMD_REQ attrHandle: %x, bdHandle: %x:  attId:%x \r\n",

                                ((cy_stc_ble_gatts_write_cmd_req_param_t *)eventParam)->handleValPair.attrHandle,

                                ((cy_stc_ble_gatts_write_cmd_req_param_t *)eventParam)->connHandle.bdHandle,

                                ((cy_stc_ble_gatts_write_cmd_req_param_t *)eventParam)->connHandle.attId);

                }

                uartRxDataWrReg = (cy_stc_ble_gatts_write_cmd_req_param_t *) eventParam;

             

                if(uartRxDataWrReg->handleValPair.attrHandle == CY_BLE_UART_SERVICE_SERVER_UART_RX_CHAR_HANDLE){

                    BLE_UART_Handle_Peripheral_RX_Data( uartRxDataWrReg->handleValPair.value.val, uartRxDataWrReg->handleValPair.value.len);

                }

            }

            break;

This prints as expected:

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19849= 3911mV, 76%

Charge Status ADC: 150, State: 0, NO CHARGE

Activity Flags: 0x0

Send Device Info Notification

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19860= 3912mV, 76%

Charge Status ADC: 135, State: 0, NO CHARGE

Activity Flags: 0x0

Send Device Info Notification

CY_BLE_EVT_GATTS_WRITE_CMD_REQ attrHandle: 55, bdHandle: f:  attId:0

What is rather interesting is that this ONLY WORKS IF THE DEBUGGER IS ATTACHED AND RUNNING from PSoC Creator (4.3).

- To reproduce this, I can run the application while Debugging, everything looks ok.
- I then stop the debugger and disconnect it, Server hangs.

- "Attach to running target" I can gather the following:

Server hangs itself in EnterDeepSleepRam(), inside this loop:

    /* Acquire the IPC to prevent changing of the shared resources at the same time */

    while(0U == _FLD2VAL(IPC_STRUCT_ACQUIRE_SUCCESS, REG_IPC_STRUCT_ACQUIRE(CY_IPC_STRUCT_PTR(CY_IPC_CHAN_DDFT))))

    {

        /* Wait until the IPC structure is released by another CPU */

    }

The Call Stack looks like this:

0 Cy_SysLib_ProcessingFault() Generated_Source\PSoC6\pdl\drivers\peripheral\syslib\cy_syslib.c 463 0x10036B7C (All)

1 Cy_SysLib_FaultHandler(const uint32_t * faultStackAddr = <optimized out>) Generated_Source\PSoC6\pdl\drivers\peripheral\syslib\cy_syslib.c 444 0x10036C0C (All)

2 UsageFault_Handler() gcc/startup_psoc6_01_cm4.S 455 0x1003036A (All)

3 <signal handler called>() ?????? ?????? 0xFFFFFFF9 (All)

4 EnterDeepSleepRam(cy_en_syspm_waitfor_t waitFor = CY_SYSPM_WAIT_FOR_INTERRUPT, cy_en_syspm_waitfor_t waitFor@entry = CY_SYSPM_WAIT_FOR_INTERRUPT) Generated_Source\PSoC6\pdl\drivers\peripheral\syspm\cy_syspm.c 2883 0x08010D96 (All)

5 Cy_SysPm_CpuEnterDeepSleep(cy_en_syspm_waitfor_t waitFor = <optimized out>) Generated_Source\PSoC6\pdl\drivers\peripheral\syspm\cy_syspm.c 705 0x1003729C (All)

6 DeepSleep_Sequence() device.c 177 0x10035474 (All)

7 HostMain() host_main.c 494 0x100352A4 (All)

8 main() main_cm4.c 62 0x10030374 (All)

After attaching, observing, and restarting the Server, it keeps going just fine WITH DEBUGGER STILL ATTACHED, then hangs again when I stop debugging.

0 Likes

Hi Adam,

>>"What is rather interesting is that this ONLY WORKS IF THE DEBUGGER IS ATTACHED AND RUNNING from PSoC Creator (4.3)."

--> This is expected because while debugging the device will not go to DeepSleep. So you cannot debug the applications with DeepSleep feature.

Thank you for attaching the call stack and pointing the stuck location. We will check at our side and update you.

Meanwhile can you please check using normal GATT write and see if it is working fine?

Thanks

Ganesh

0 Likes

Thank you Ganesh very much,

I will be running my testing with normal write, making this change:

/* apiResult = Cy_BLE_GATTC_WriteWithoutResponse(&uartTxDataWriteCmd); */

apiResult = Cy_BLE_GATTC_WriteCharacteristicValue(&uartTxDataWriteCmd);

I'll report back the results.

0 Likes

Changing the WriteWithoutResponse() to WriteCharacteristicValue() definitely helped, but I still caught it hanging one time only in the ~24hrs of testing. I will keep running it and try to catch it while sitting next to it.

A few things I feel that are worth noting:

- WriteCharacteristicValue() often returns with CY_BLE_ERROR_INVALID_OPERATION. I assume this happens because my loop calls it while the stack is still waiting for the CY_BLE_EVT_GATTC_WRITE_RSP event? This causes inconsistency in the amount of packets I send out in 1 second, where 376 packets is the steady norm, I get fluctuations between 370-385 (3 packets/connection interval).


- When the Client gets stuck with CY_BLE_STACK_STATUS_BUSY-> timeout runs out-> try to disconnect Server, Cy_BLE_GAP_Disconnect() will return SUCCESS the first time I call, but the Server is still connected until the "Connection supervision timeout" (in BLE GAP settings). My code keeps calling Cy_BLE_GAP_Disconnect() thinking it is still connected, (while the stack is still busy, and disconnection events have not occured yet) and returns with CY_BLE_ERROR_NO_DEVICE_ENTITY. Right after "connection supervision timeout" the stack becomes FREE, Disconnect events come in and scanning is re-initiated (back to the beginning).

0 Likes

Could there be hardware related problems causing the IPC to hang in this loop?

    /* Acquire the IPC to prevent changing of the shared resources at the same time */

    while(0U == _FLD2VAL(IPC_STRUCT_ACQUIRE_SUCCESS, REG_IPC_STRUCT_ACQUIRE(CY_IPC_STRUCT_PTR(CY_IPC_CHAN_DDFT))))

    {

        /* Wait until the IPC structure is released by another CPU */

    }

I have been testing on two boards, and one of them started hanging in this loop, the other one has been running fine.

EDIT: This seems to be related to the I2C hardware, if I disable my display it looks to be running fine.

0 Likes

Here is a debug capture of the BLE hanging with WriteCharacteristicValue(), as comprehensive as I can get it so far:
The capture was taken from two different hangs. I ommitted repeating events and SPI report for clarity.
This is more rare than WriteWithoutResponse(), but still managed to catch it:

// Everything is reporting fine, BLE and SPI Reports printing every 1second

--- BLE Report --- Updates: 125Hz, Stack Busy: 1, TX Buff Discarded: 198

        Interface Status: 6, Activity State: 0

        Battery Level: 91% (4029mV)

        TX Frames: 346/s, T:9514822     RX Frames: 346/s, T:9514440 (Invalid: 0)

        TX Signal: -38dBm,              RX Signal: -30dBm

// Response Event CY_BLE_EVT_GATTC_WRITE_RSP not received, WriteCharacteristicValue() responds with CY_BLE_ERROR_INVALID_OPERATION, software FIFO buffer fills up since BLE is stopped flushing

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

...Repeating

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

// Next 1second Reports

--- SPI Report ---

        RX DMA Length: 32 x 16bits,             TX DMA Length: 32 x 16bits

        RX DMA Updates: 998/s, 63872 Bytes/s    TX DMA Updates: 999/s, 63936 Bytes/s

SXP TX Error: FIFO FULL

--- BLE Report --- Updates: 125Hz, Stack Busy: 1, TX Buff Discarded: 198

        Interface Status: 6, Activity State: 0

        Battery Level: 91% (4029mV)

        TX Frames: 0/s, T:9514822       RX Frames: 0/s, T:9514440 (Invalid: 0)

        TX Signal: -38dBm,              RX Signal: -37dBm

// Notice: TX/RX Frames Count has not changed, no Write sent out, no Notification received.

// Notice: RSSI (RX Signal) updated, indicating BLE connection is still ok. (TX Signal is Server's RSSI, not received in Notification Data)

// Notice: Stack doesn't become BUSY with normal WriteCharacteristicValue()

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

...Repeating

// Here I let it run to observe behavior. I notice Timeouts happening, resetting the BLE TX Buffer, flushing out normal 128byte transfers

// Somehow, somewhere one or more Notification finds it's way back.

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

--- BLE Report --- Updates: 125Hz, Stack Busy: 1, TX Buff Discarded: 198

        Interface Status: 6, Activity State: 1

        Battery Level: 91% (4029mV)

        TX Frames: 0/s, T:9514926       RX Frames: 0/s, T:9514447 (Invalid: 0)

        TX Signal: -35dBm,              RX Signal: -33dBm

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

...Repeating

SXP TX Error: FIFO FULL

BLE_EVT_TIMEOUT, Reason: CY_BLE_GATT_RSP_TO:

SXP TX Error: FIFO FULL

...Repeating

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

// BLE Report directly after the last one

// Notice: 8 TX Frames were flushed out (16byte frames = 128bytes total = BLE transfer size)

// Notice: No RX Frames received

// Notice: Stack doesn't become BUSY with normal WriteCharacteristicValue()

--- BLE Report --- Updates: 125Hz, Stack Busy: 1, TX Buff Discarded: 198

        Interface Status: 6, Activity State: 1

        Battery Level: 91% (4029mV)

        TX Frames: 8/s, T:9514934       RX Frames: 0/s, T:9514447 (Invalid: 0)

        TX Signal: -35dBm,              RX Signal: -29dBm

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

// This keeps repeating, stopped the test after a while in order to keep developing the SPI side of the communications.

// TX Buffers are not discarded since STACK_STATE_FREE, but software FIFO gets full and software won't keep pushing to it

************************************************************************************

// This happened before the debug capture above, causing the only STACK_STATE_BUSY event

// Debug capture when I wasn't watching, but TeraTerm logged the events:

...Repeating

SXP TX Error: FIFO FULL

SXP TX Error: FIFO FULL

CY_BLE_EVT_TIMEOUT, Reason: CY_BLE_GATT_RSP_TO:

CY_BLE_EVT_STACK_BUSY_STATUS: 1

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

...Repeating

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

--- BLE Report --- Updates: 125Hz, Stack Busy: 1, TX Buff Discarded: 65

        Interface Status: 6, Activity State: 0

        Battery Level: 92% (4043mV)

        TX Frames: 8/s, T:4652515       RX Frames: 0/s, T:4652139 (Invalid: 0)

        TX Signal: -33dBm,              RX Signal: -37dBm

...Repeating

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

// Software Timeout of STACK_STATE_BUSY stuck, calling Cy_BLE_GAP_Disconnect() with SUCCESS the first time, CY_BLE_ERROR_NO_DEVICE_ENTITY every consecutive time until "Connection supervision timeout" (in BLE GAP settings).

Stack Busy Timeout - Handle Stack Failure

Disconnect Initiated, apiResult: 0x0 CY_BLE_SUCCESS

Stack Busy Timeout - Handle Stack Failure

Disconnect Initiated, apiResult: 0x160007 CY_BLE_ERROR_NO_DEVICE_ENTITY

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded

...Repeating

Stack Busy Timeout - Handle Stack Failure

Disconnect Initiated, apiResult: 0x160007 CY_BLE_ERROR_NO_DEVICE_ENTITY

BLE_UART_Handle_Host_TX_Data Error: CY_BLE_STACK_STATE_BUSY. TX Buffer discarded


// "Connection supervision timeout" releases the Stack, Server Disconnects, Scanning re-started

CY_BLE_EVT_STACK_BUSY_STATUS: 0

CY_BLE_EVT_GATT_DISCONNECT_IND: 0, f

CY_BLE_EVT_GAP_DEVICE_DISCONNECTED: bdHandle=f, reason=22, status=0

CY_BLE_EVT_GAPC_SCAN_START_STOP, scanstate = CY_BLE_SCAN_STATE_SCANNING

0 Likes

Ok, so the hang inside the IPC loop came back on the same board when trying to go back to sleep in the EnterDeepSleepRam().

    /* Acquire the IPC to prevent changing of the shared resources at the same time */

    while(0U == _FLD2VAL(IPC_STRUCT_ACQUIRE_SUCCESS, REG_IPC_STRUCT_ACQUIRE(CY_IPC_STRUCT_PTR(CY_IPC_CHAN_DDFT))))

    {

        /* Wait until the IPC structure is released by another CPU */

    }


I disabled the I2C display again, and a few other things, but nothing helped this time. I have more boards on the way to see if this is an isolated case of component damage, or something else.

EDIT: Testing KBA229335 that seems to relate to exactly this kind of problem.
(Updating PDL syspm Driver for Hard Fault Problem – KBA229335 )

Hi Adam,

Thank you for the information.

>>" WriteCharacteristicValue() often returns with CY_BLE_ERROR_INVALID_OPERATION. I assume this happens because my loop calls it while the stack is still waiting for the CY_BLE_EVT_GATTC_WRITE_RSP event

--> Yes. The Write is considered as completed only when the Server responds and the Client recives CY_BLE_EVT_GATTC_WRITE_RSP event. Please include this logic in your application.

>>"I disabled the I2C display again, and a few other things, but nothing helped this time. I have more boards on the way to see if this is an isolated case of component damage, or something else."

--> Have you tested the other boards? Have you tried the KBA229335 ? Are there any positive results?

Thanks

Ganesh

0 Likes

Hi Ganesh,

Thank you for the clarification.

KBA229335 seemed to help with the IPC hanging, so far no issues with normal writes with responses.

I'm planning on going back to the WriteWithoutResponse() to see if anything changed, as it is my preferred way to handle the communications. I need to tie up some loose ends with other components of the code before I make fundamental changes like this but I'll be sure to report back on my results.

Unfortunately I'm still observing the behavior I posted in Reply #11 above.

I don't know if this is related to the IPC hanging, as the server still goes on, thinking it is still connected.

An other DeepSleep related issue I discovered:

Series of events and reproduction:
- Main controller starts DFU process of the Client

- Client calls Cy_BLE_GAP_Disconnect() while server is in DeepSleep.

- Client goes into DFU operations with main controller

- Server gets into CY_BLE_EVT_GAP_DEVICE_DISCONNECTED event

- Server calls Cy_BLE_GAPP_StartAdvertisement() from Disconnect event

-**** Server gets into CY_BLE_EVT_GAPP_ADVERTISEMENT_START_STOP event and start advertising ***

- Client finishes DFU operations and starts Scanning

- Client finds Server and re-connects

Sometimes the Server doesn't get into the event CY_BLE_EVT_GAPP_ADVERTISEMENT_START_STOP after calling Cy_BLE_GAPP_StartAdvertisement(), and advertisement does not start. The Client will not find and connect to it after DFU. This is not consistent and it takes a number of tries.

After this happens, the Server gets into a similar situation with the same kind of errors as after the random hangs, observed by issuing SendNotification() and Disconnect() manually:

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19584= 3860mV, 70%

Charge Status ADC: 135, State: 0, NO CHARGE

Activity Flags: 0x0

Wake Up Interrupt

Exit Inactive State

USER SINGLE RELEASE TRIGGER - Hold Timer (mS): 0

Send Device Info Notification

Cy_BLE_GATTS_Notification API Error: 0x160001 INVALID_PARAMETER

USER DOUBLE RELEASE TRIGGER - Hold Timer (mS): 0

Disconnect Initiated, apiResult: 0x160007 CY_BLE_ERROR_NO_DEVICE_ENTITY

Manually trying to start advertisement gives CY_BLE_ERROR_INVALID_STATE:
Cy_BLE_GAPP_StartAdvertisement API Error: 0x1600ff CY_BLE_ERROR_INVALID_STATE

Which returns  "On calling this function not in Stopped state." So advertising is NOT in Stopped state, but it has not started.
Further debugging shows state is CY_BLE_ADV_STATE_ADV_INITIATED -> BLE Stack is hanging somewhere?

EDIT: I caught it just in the right time with CY_BLE_ERROR_INSUFFICIENT_RESOURCES:

CY_BLE_EVT_GATT_DISCONNECT_IND: 0, 10

CY_BLE_EVT_GAP_DEVICE_DISCONNECTED: bdHandle=10, reason=13, status=0

Cy_BLE_GAPP_StartAdvertisement API Error: 0x160004 CY_BLE_ERROR_INSUFFICIENT_RESOURCES

Debug log from Server:

Server normal operation:

....

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19645= 3873mV, 71%

Charge Status ADC: 135, State: 0, NO CHARGE

Activity Flags: 0x0

CY_BLE_EVT_GATT_DISCONNECT_IND: 0, 10

CY_BLE_EVT_GAP_DEVICE_DISCONNECTED: bdHandle=10, reason=13, status=0

CY_BLE_EVT_GAPP_ADVERTISEMENT_START_STOP, state: 2

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19645= 3873mV, 71%

Charge Status ADC: 136, State: 0, NO CHARGE

Activity Flags: 0x0

....

Server missing ADVERTISEMENT_START_STOP event:

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19594= 3862mV, 70%

Charge Status ADC: 135, State: 0, NO CHARGE

Activity Flags: 0x0

CY_BLE_EVT_GATT_DISCONNECT_IND: 0, 10

CY_BLE_EVT_GAP_DEVICE_DISCONNECTED: bdHandle=10, reason=13, status=0

WDT CTR0 Inactivity Interrupt

Battery Level Update: 19590= 3862mV, 70%

Charge Status ADC: 135, State: 0, NO CHARGE

Activity Flags: 0x0

0 Likes

Hi Adam,

I have some questions.

What version of PDL do you use?

Does BLESS Interrupt have the highest priority in system (on CM0p core)? Did you do any modification in system_psoc6.h?

What do you do on CM0 core (e.g. i2c or other peripheral operation)?
Please confirm that Clk_LF is sourced from WCO (See in DWR->Clocks->Miscellaneous Clocks)
Could you check that Cy_BLE_ProcessEvent function is calling at least once per CI?

Regards,

Nazar



0 Likes

What version of PDL do you use?

     - PDL 3.1.2

Does BLESS Interrupt have the highest priority in system (on CM0p core)? Did you do any modification in system_psoc6.h?

    pastedImage_0.png

     - No changes to system_psoc6.h

What do you do on CM0 core (e.g. i2c or other peripheral operation)?

     - The only other thing running on CM0 is the Crypto Server. See main loop:

   for(;;)

    {

        /* Process crypto server requests */

        Cy_Crypto_Server_Process();       

    #if(CY_BLE_STACK_MODE_IPC)

        /* Process BLE events continuously for controller in dual core mode */

        Cy_BLE_ProcessEvents();

        /* To achieve low power in the device */

        Cy_SysPm_CpuEnterDeepSleep(CY_SYSPM_WAIT_FOR_INTERRUPT);

    #endif /* CY_BLE_STACK_MODE_IPC */

    }

Please confirm that Clk_LF is sourced from WCO (See in DWR->Clocks->Miscellaneous Clocks)

pastedImage_15.png

Could you check that Cy_BLE_ProcessEvent function is calling at least once per CI?

     - In CM4 core Cy_BLE_ProcessEvents() is called after every major BLE operation, such as HID and BAS characteristic writes. It is also      called at the end of main(), before callind DeepSleep:

     Cy_BLE_ProcessEvents();

    if ((Device.Activity_State == INACTIVE) && (UART_DEB_IS_TX_COMPLETE() == 1u)){

        cy_en_syspm_status_t stat = Cy_SysPm_CpuEnterDeepSleep(CY_SYSPM_WAIT_FOR_INTERRUPT);

    }


How can I make 100% sure this is called every CI even in DeepSleep? Does it need to be called on both CM0+ and CM4 independently at every CI?

0 Likes

Hi Nazar,

As I gather more experience with this and observe the stack hanging and misbehaving, it would make sense that Cy_BLE_ProcessEvent() falls behind somehow.

Just to summarize:
- CM0+ runs the BLE Controller. It is in DeepSleep at all times, and woken up by the BLE interrupts (every connection interval I presume) and calls Cy_BLE_ProcessEvent() to handle communications.

- CM4 runs the higher level Host and Profile of the BLE. In normal operation Cy_BLE_ProcessEvent() is called from main() constantly

- On CM4 the BLE callback function AppCallBack() handles all received events, and sends out Notifications directly from the CB.

- When CM4 is in DeepSleep, main() loop halts, and Cy_BLE_ProcessEvent() is not called, only if CM4 comes out of DeepSleep.

I don't completely understand how Cy_BLE_ProcessEvent() and AppCallBack() are handled between CM0+ and CM4 cores.

Will CM4 also wake up when CM0+ handles BLE traffic?
Will CM0+ execute AppCallBack() or CM4?

Is it a good idea to call Cy_BLE_ProcessEvent() directly from the AppCallBack()?

0 Likes
NazarP_56
Employee
Employee
25 solutions authored 10 sign-ins 50 replies posted

Hi Adam,

Will CM4 also wake up when CM0+ handles BLE traffic?
yes. Controller run on CM0+: BLESS interrupt wakes up CM0+ core (if it in deep sleep), than Cy_BLE_ProcessEvent on (CM0+) will communicate to Host core CM4 ( via IPC), and IPC interrupt wakes up CM4 core.

Will CM0+ execute AppCallBack() or CM4?

see my answer below. CM4 will execute only if need some action from host side. If we start Adv, the host side will receive only START/STOP ADV event, rest handling of ADV generation will process on CM0+ side, CM4 will in deep sleep... 

Is it a good idea to call Cy_BLE_ProcessEvent() directly from the AppCallBack()?
No. AppCallBack is called from Cy_BLE_ProcessEvent. So you will have stuck.

Do you use some kind of RTOS? please check that you do not call Cy_BLE_ProcessEvent (or another BLE API) from different threads (rtos thread, or ISR context like timers).

Regards,
Nazar

Will CM0+ execute AppCallBack() or CM4?

0 Likes

Thanks Nazar,

I do have a Timer ISR executing at 1Hz in DeepSleep, to do some basic supervisory tasks.

One of those tasks is to measure the battery level, and update the HID device for Win10. This is important because if I don't update it, the battery level will freeze at the last updated value, and the user won't know if their device is about to shut off.

The function does call Cy_BLE_ProcessEvents():

void BAS_Battery_Update(cy_stc_ble_conn_handle_t connHandle){

    cy_en_ble_api_result_t apiResult;

    uint16_t cccd;

    (void) Cy_BLE_BASS_GetCharacteristicDescriptor(connHandle, CY_BLE_BAS_BATTERY_LEVEL, CY_BLE_BAS_BATTERY_LEVEL, CY_BLE_BAS_BATTERY_LEVEL_CCCD, CY_BLE_CCCD_LEN, (uint8_t*)&cccd);

    if(cccd == CY_BLE_CCCD_NOTIFICATION)

    {

        do{

            Cy_BLE_ProcessEvents();

        }

        while(Cy_BLE_GATT_GetBusyStatus(connHandle.attId) == CY_BLE_STACK_STATE_BUSY);

        if(Cy_BLE_GetConnectionState(connHandle) >= CY_BLE_CONN_STATE_CONNECTED)

        {

            /* Update Battery Level characteristic value and send Notification */

            apiResult = Cy_BLE_BASS_SendNotification(connHandle, CY_BLE_BAS_BATTERY_LEVEL, CY_BLE_BAS_BATTERY_LEVEL,

                                                     sizeof(BAS.batteryLevel), &BAS.batteryLevel);

            if(apiResult != CY_BLE_SUCCESS){

                DBG_PRINTF("Cy_BLE_BASS_SendNotification API Error: 0x%x \r\n", apiResult);

            }

        }

    }   

       

    /* Update Battery Level characteristic value */

    apiResult = Cy_BLE_BASS_SetCharacteristicValue(CY_BLE_BAS_BATTERY_LEVEL, CY_BLE_BAS_BATTERY_LEVEL,

                                                   sizeof(BAS.batteryLevel), &BAS.batteryLevel);

    if(apiResult != CY_BLE_SUCCESS){

        DBG_PRINTF("Cy_BLE_BASS_SetCharacteristicValue API Error: 0x%x \r\n", apiResult);

    }

    else {

        DBG_PRINTF("Battery Level Update: %d= %dmV, %d%%\r\n Charge Status ADC: %d, State: %d",BAS.adcResult, BAS.mvolts_smooth, (int)BAS.batteryLevel, BAS.charging_adc, BAS.charging);

        if(BAS.charging == 0){

            DBG_PRINTF(", NO CHARGE\r\n");

        }

        else if (BAS.charging == 1){

            DBG_PRINTF(", CHARGING\r\n");       

        }

        else DBG_PRINTF(", FULL\r\n");

    }

    Cy_BLE_ProcessEvents();

}

0 Likes

HI Adam,

in RTOS flow you should have one BLE task, which call processEvent all incoming signals (send notification, etc) from other tasks.

see example:

PSoC-6-MCU-RTOS-Based-Design/CE218138_BLE_Thermometer_RTOS at master · cypresssemiconductorco/PSoC-6... Please look at Figure#14 RTOS Firmware Flow (p.9)  in datasheet for this example.

In Bare metal flow,  you should implement BLE functionality in main loop (CM4). So your main loop shoud have:

1. BLE_ProcessEvent

2. implement some logic to send notification (BAS, HID, etc). Timer ISR can set some flag "we have the data for sending", or store data to some queue (finally, this data should be notifyed from main loop).

3. Enter in to deep-sleep.

NOTE: you can register addition SysPm deep-sleep callback to control entry to deep-sleep by some additional condition from the application.


Please update your app, and share results.

Regards,
Nazar

0 Likes

That makes sense, your help is much appreciated!

If I understand correctly Cy_SysPm_CpuEnterDeepSleep(CY_SYSPM_WAIT_FOR_INTERRUPT) will "hang" the main loop, until a wakeup interrupt happens. How do I make sure that the main loop will "wake up" from this and execute the tasks from a flag?

Does the Timer ISR running at 1Hz take care of it? So if I initiate a BAS characteristic update from the ISR, after exiting that ISR the code will return to the main loop. In main() the Cy_SysPm_CpuEnterDeepSleep() will return, therefore I get one round of main() to examine the flag, handle BLE transfers, call Cy_BLE_ProcessEvents(), then go back to DeepSleep?

0 Likes
NazarP_56
Employee
Employee
25 solutions authored 10 sign-ins 50 replies posted

HI Adam,
Timer ISR is deep-sleep wakeup capable and it will wake up CM4.

Also, need to handle the case, when the timer is triggered between  #2 and #3 (see my post above), in this case you do not send notification and go to deep-sleep, so you will miss this notification and probably send in the next timer event (after 2 sec).


To manage this you can:
1. check flag before into deep-sleep (simple way)
2. or you can register addition SysPm deep-sleep callback to control entry to deep-sleep by some additional condition (check flag  or..) in the registered callback.

Regards,
Nazar

0 Likes

After some extended testing I can confidently say the odd hanging problem was caused by the Timer ISR, executing BLE related tasks exactly between Cy_BLE_ProcessEvents() and Cy_SysPm_CpuEnterDeepSleep(). This was hard to reproduce due to the very small time window of this happening, and the very slow (1Hz) Timer ISR.

I am still getting Stack Busy states when the signal levels are low, but I think I will be able to manage it and throttle communications when it falls behind.

Ganesh and Nazar, thank you very much for your guidance, your time and efforts in the process are greatly appreciated!

0 Likes