6 Replies Latest reply on Mar 8, 2019 3:50 AM by m.lagrassa_3768146

    OTA2 Failsafe app not starting

    m.lagrassa_3768146

      Hi,

      I'm quite new to the Wiced Studio environment, so I hope I'm not forgetting something.

      I have a CYW943907AEVAL1F Kit and I'm testing the OTA2 features. Here is what a tried:

      1 - Build and download the manufacturing image with the "ota2_manuf_download" makefile target. I understand that this image should contain everything needed for the OTA2 update.

      2 - Modified a constant in the app so that I could see the difference between the "factory" firmware and the "updated" one (the constant is printed to the console).

      3 - I used the "ota2_download" makefile target to build the image and download it to the staging area

      4 - Reboot the board and check that the "old" firmware is still running.

       

      At this point I press button1 of the board. In the firmware, the button is programmed to do the following:

              wiced_log_msg(WLF_DEF, WICED_LOG_INFO, "Setting OTA2 header to EXTRACT_ON_NEXT_BOOT");

              wiced_ota2_image_fakery(WICED_OTA2_IMAGE_EXTRACT_ON_NEXT_BOOT);

       

      So basically when I press the button, I use the fakery function to "fake" a successful download of the new firmare image, which I previously saved in the flash.

      Now, if I simply reboot the board, everything works as expected: the OTA2 extraction app runs, extracts the new firmware and runs it after rebooting.

       

      The problem is that if I reset the board *during* the extraction, the Failsafe app never runs. I enabled the printf macros in ota2_bootloader.c and slightly edited it to print more info. In particular, I added 2 printf in this part (line ~209 of ota2_bootloader.c):

       

          /* Check boot_type FAILSAFE - if so, use ota2_failsafe to recover */

          BOOTLOADER_PRINTF(("Bootloader: boot_type = %d!\r\n", dct_ota2_config.boot_type));

          if ((dct_ota2_config.boot_type == OTA2_BOOT_FAILSAFE_FACTORY_RESET) ||

              (dct_ota2_config.boot_type == OTA2_BOOT_FAILSAFE_UPDATE))

          {

              uint32_t entry_point;

              if (ota2_failsafe_app_load( OTA2_IMAGE_FAILSAFE_APP_AREA_BASE, &entry_point ) == WICED_SUCCESS)

              {

                  BOOTLOADER_PRINTF(("Bootloader: Starting failsafe app ... address %ld\r\n", entry_point));

                  wiced_waf_start_app( entry_point );

              } else {

                  BOOTLOADER_PRINTF(("Bootloader: failed to load failsafe app!\r\n"));

              }

          }

       

      Now on the console, here is what I see after rebooting the board during the extraction from the staging area:

      OTA2 Bootloader -- start_sequence : 2 wiced_waf_start_app() 0x4a0140

      Hi, I'm the OTA2 extraction app (ota2_extract).

      Extract from Staged Area!

      Copy Current DCT to Saved DCT.

      extracting: APPS.bin.

      extracting: DCT.bin.

      extracting: snip.ota2_extract-CYW943907AEVA..................

      extracting: filesystem.bin <----- HERE IS WHERE I RESET THE BOARD

      ..OTA2 Bootloader!

      Bootloader: boot_type = 9!

      Bootloader: Starting failsafe app ... address 4849696

       

      At this point I'd expect the failsafe app to run and extract the image from the staging area... but nothing happens and it remains stuck there forever. Rebooting the board again will result in the last two lines being repeated...

      Do you have any suggestion? Am I doing something wrong?

       

      Thank you in advance!

       

      Marco

        • 1. Re: OTA2 Failsafe app not starting
          ZhengbaoZ_96

          Hello:

           

            We have a similar thread which talked about OTA2 issues,  Would you please check this thread to see if you can get help ?

           

          Re: Problems with OTA2: flashing device and firmware_reset (con't)

          • 2. Re: OTA2 Failsafe app not starting
            m.lagrassa_3768146

            Hello,

            I had already checked that thread before opening this one. Unfortunately they talk about different issues. My issue is not with the firmware upgrade itself, but with the failsafe app in the case of board reset during the extraction procedure.

            I don't even use the SoftAp http server to upload the new firmware, I just use the make target "ota2_download" as per documentation.

             

            As you can see from the console log, it is like the ota2_bootloader is failing to launch the failsafe app for some reason (the failsafe app should print on console too, which clearly isn't happening in my case). As you can see I also tried to print the "entry_point" variable value, but I have no idea how to check if that address is correct or not.

             

            Marco

            • 3. Re: OTA2 Failsafe app not starting
              ZhengbaoZ_96

              hello:

               

              Yes, I can duplicate the issue .  Seems it has some logic error if we do the reset .

              but really need more time to track the issue, need your help to dig more also.

              • 4. Re: OTA2 Failsafe app not starting
                m.lagrassa_3768146

                Hello,

                I investigated more today and found out something interesting. What I've tried is to force the bootloader to keep launching the failsafe app. To do that, I modified the ota2_extract.c file and added the following code between /** DEBUG */ and /** END OF DEBUG */ (line ~327):

                 

                switch ( extract_result )

                    {

                        case EXTRACT_RESULT_OK:

                            /* set APP0 as initial app to run - we set it to OTA_APP to get to this application */

                            wiced_waf_app_set_boot( DCT_APP0_INDEX, PLATFORM_DEFAULT_LOAD );

                 

                            /* clear force_factory_reset Flag */

                            dct_ota2_config.force_factory_reset = 0;

                            wiced_dct_write( &dct_ota2_config, DCT_OTA2_CONFIG_SECTION, 0, sizeof(platform_dct_ota2_config_t) );

                 

                            /** DEBUG **/

                            dct_ota2_config.boot_type = OTA2_BOOT_FAILSAFE_FACTORY_RESET;

                            wiced_dct_write( &dct_ota2_config, DCT_OTA2_CONFIG_SECTION, 0, sizeof(platform_dct_ota2_config_t) );

                            OTA2_EXTRACT_PRINTF(("\r\nDEBUG: Extract result OK - DCT boot type set to FAILSAFE_FACTORY_RESET...\r\n"));

                            /** END OF DEBUG **/

                 

                            break;

                 

                The idea is to set the boot_type to FAILSAFE_FACTORY_RESET after a valid ota2 extraction, so that at the next boot the bootloader will think that something went wrong during the ota2 extraction and try to launch the Failsafe app to restore the flash. The very first ota2 extraction is triggered in the same way as before, i.e. by setting a handler on the board button that writes the ota2 config dtc using the following code:

                 

                wiced_dct_read_with_copy( &ota2_config, DCT_OTA2_CONFIG_SECTION, 0, sizeof(platform_dct_ota2_config_t) );

                ota2_config.boot_type = OTA2_BOOT_FAILSAFE_FACTORY_RESET;

                wiced_dct_write( &ota2_config, DCT_OTA2_CONFIG_SECTION, 0, sizeof(platform_dct_ota2_config_t) );

                 

                The behaviour expected from this edit is the following:

                1. After launching the ota2_manuf_download, the firmware starts and the main application is run
                2. Pressing the button will set the boot_type to FAILSAFE_FACTORY_RESET
                3. Resetting the board will cause the bootloader to launch the Failsafe app, which launches the ota2_extract app that finally restores the flash from the factory reset area.
                4. After the successful extraction, the ota2_extract app reboots the board
                5. Points 3 and 4 repeats over and over in a loop.

                 

                However, what happened is that after pressing the board button (point 2) and resetting the board (point 3), the Failsafe app runs correctly and the same happens for the Ota2 extract app. However, after the next reboot, the failsafe app fails to run in the very same way I reported yesterday.

                 

                The problems seems to arise when ota2_extract.c calls wiced_dct_ota2_save_copy(). In particular, the following block of code:

                 

                switch (dct_ota2_config.boot_type)

                        {

                           [...]

                            case OTA2_BOOT_EXTRACT_FACTORY_RESET:

                                OTA2_EXTRACT_PRINTF(("Extract the Factory Reset!\r\n"));

                                result =WICED_SUCCESS;

                 

                                result = wiced_dct_ota2_save_copy(OTA2_BOOT_FACTORY_RESET);

                                if (result != WICED_SUCCESS)

                                {

                                    extract_result = EXTRACT_RESULT_ERROR_DCT_COPY_FAIL;

                                    break;

                                }

                                result = wiced_ota2_image_extract ( WICED_OTA2_IMAGE_TYPE_FACTORY_RESET_APP );

                               [...]

                                break;

                 

                            case OTA2_BOOT_EXTRACT_UPDATE:

                                OTA2_EXTRACT_PRINTF(("Extract from Staged Area!\r\n"));

                                result = wiced_dct_ota2_save_copy(OTA2_BOOT_UPDATE);

                                if (result != WICED_SUCCESS)

                                {

                                    extract_result = EXTRACT_RESULT_ERROR_DCT_COPY_FAIL;

                                    break;

                                }

                                result = wiced_ota2_image_extract ( WICED_OTA2_IMAGE_TYPE_STAGED );

                 

                When commenting these two calls to wiced_dct_ota2_save_copy() the Failsafe app seems to restart correctly and the firmware gets stuck in the boot loop as expected. Obviously this breaks the DCT saving mechanisms so it is not a solution.

                Going deeper, I think the problem is the call to wiced_dct_ota2_erase_save_area_and_copy_dct(), inside wiced_dct_ota2_save_copy().

                As I said I'm quite new to this IDE and I was unable to investigate these functions much more than this.

                 

                Finally, I tried to print a sort of dump of the first 1000 bytes starting from the position pointed by the entry_point variable in ota2_bootloader.c, with the following code:

                 

                /* Check boot_type FAILSAFE - if so, use ota2_failsafe to recover */

                    BOOTLOADER_PRINTF(("Bootloader: boot_type = %d!\r\n", dct_ota2_config.boot_type));

                    if ((dct_ota2_config.boot_type == OTA2_BOOT_FAILSAFE_FACTORY_RESET) ||

                        (dct_ota2_config.boot_type == OTA2_BOOT_FAILSAFE_UPDATE))

                    {

                        uint32_t entry_point;

                        if (ota2_failsafe_app_load( OTA2_IMAGE_FAILSAFE_APP_AREA_BASE, &entry_point ) == WICED_SUCCESS)

                        {

                            ptr = (char *)entry_point;

                            BOOTLOADER_PRINTF(("Bootloader: Starting failsafe app ... address %ld\r\n", entry_point));

                            BOOTLOADER_PRINTF(("\r\n---- BEGIN DUMP ---\r\n"));

                            for (i = 0; i < 1000; i++) {

                                ptr = ptr + i;

                                BOOTLOADER_PRINTF(("%2x ", *ptr));

                            }

                            BOOTLOADER_PRINTF(("\r\n---- EOF ---\r\n"));

                            wiced_waf_start_app( entry_point );

                        } else {

                            BOOTLOADER_PRINTF(("Bootloader: failed to load failsafe app!\r\n"));

                        }

                    }

                 

                I compared the dump before and after the Ota2 extraction (so between points 3 and 4 of the previous list) and I found out that they are different after a certain number of bytes (see the attached image). Is that supposed to happen? I still don't have a clear picture of what the memory dump should look like, but I don't understand why the two dumps are different. Is it possible that the memory is corrupted and wiced_waf_start_app() results in a crash?

                 

                Thank for the assistance!

                 

                Marco

                • 5. Re: OTA2 Failsafe app not starting
                  ZhengbaoZ_96

                  Hi:

                   

                  Finally, I found the reason,  every time when the issue happened I only updated failsafe app , the extract will go to normal.

                  so check the mk file:

                  # OTA2 Failsafe Application Storage

                  OTA2_IMAGE_FAILSAFE_APP_AREA_BASE := 0x001F4000  #  72k  0x00014000

                   

                  The size of Failsafe Application is 72k,  Actually,  our app size of compiled is 85k.

                  after adding 20k size more, all the function goes  well.

                  please have a try.

                  1 of 1 people found this helpful
                  • 6. Re: OTA2 Failsafe app not starting
                    m.lagrassa_3768146

                    Hello,

                    from a preliminary test, setting OTA2_IMAGE_FAILSAFE_APP_AREA_BASE := 0x001EF000 in ota2_image_defines.mk fixes the problem. Can't believe it was something so simple!

                    Anyway the original size set was

                    OTA2_IMAGE_FAILSAFE_APP_AREA_BASE := 0x001F4000  #  72k  0x00014000

                    I suppose this was taken from an older configuration, since the actual dimension is 80K and not 72K, so the comment is wrong...

                     

                    I'll keep testing next week and let you know if I find any other related issue.

                     

                    PS: if anyone is experiencing the same issue and changing OTA2_IMAGE_FAILSAFE_APP_AREA_BASE doesn't fix it, make sure to perform a clean, otherwise the constant may not be updated correctly.

                     

                    Thank you for the help