7 Replies Latest reply on Apr 1, 2020 1:37 AM by RashiV_61

    How to dynamically change resolution of image sensor - AN75779

    BoJo_4384951

      Hello guys,

       

      We were able to successfully benefit from the application note AN75779 and use FX3 to transfer video streams from our image sensors (IMX250, IMX253, IMX420) to the PC. VLC player has been used to play the streams. Thanks for your support in that sense (thread 1, thread 2)! It is really appreciated.

       

      We are now in the process of replacing VLC player by our custom software using UVC class drivers. In that sense, we would like to be able to dynamically change video resolution during system runtime. Here are some use-cases where this dynamic resolution change would be necessary:

      1) During system start-up, we will read the type of the image sensor we have connected with FX3 and set video resolution accordingly.

      2) There will be a need to change video mode or simply video resolution during runtime of the system by sending appropriate command message from PC.

       

      In this thread you explained us how to include different resolutions into USB descriptor chain and we did it successfully. We also noticed that the last Class specific Uncompressed VS frame descriptor in the list represents the default video resolution.

       

      Our FX3 firmware is based on AN75779 example. What are the steps/commands to take in order to change video resolution of the stream dynamically, during runtime? We can change a resolution of the image sensor successfully, we only need to inform PC about that.

       

      As always, thank you very much for your time and efforts!

       

      Sincerely,

      Bojan.

        • 1. Re: How to dynamically change resolution of image sensor - AN75779
          RashiV_61

          Hello Bojan,

           

          I will explain the whole procedure of changing video resolution dynamically with VLC media player

          When AN75779 firmware is modified to add multiple resolution, on opening the host application (VLC) the options of the video resolutions supported by FX3 (as per modifications in the firmware) will be visible. Among the multiple resolutions one resolution will be set by default

           

          1) To change the video resolution dynamically, one among the multiple resolutions shown in host application (VLC) need to be chosen.

          2) On changing the video resolution, UVC class request will be sent by the host to FX3. Please refer to figure 6 of AN75779 and section 2.3.2.2 for the flow of the requests and response between host and device.

          3) The UVC requests (sent by host) need to be handled in the firmware. Changing video resolution is for Streaming interface. Streaming interface can handle two requests CY_FX_UVC_PROBE_CTRL and CY_FX_UVC_COMMIT_CTRL requests.

           

          - Please confirm that you are sending the probe control structures for all the video resolutions when this CY_FX_USB_UVC_GET_DEF_REQ request is sent by the host

          - Under  CY_FX_UVC_COMMIT_CTRL, when CY_FX_USB_UVC_SET_CUR_REQ  occurs then appropriate sensor settings need to be change (according to the video resolution set) by the host

           

           

          // UVC class request

           

                  if (wValue == FX3_UVC_VS_COMMIT_CONTROL)

                  {

                  

          switch(bRequest)

          case FX3_USB_UVC_SET_CUR_REQ:

                      switch (CY_U3P_GET_LSB(wIndex))

                      {

                      case FX3_UVC_STREAM_INTERFACE_1:

                      {

                          CyU3PThreadSleep(900);

           

                          CyFx3UvcAppSetVideoResolution(glCommitCtrl[2], glCommitCtrl[3], frameInterval, 0);

                     }

                      break;

                      }

                  }

           

          void

          CyFx3UvcAppSetVideoResolution(

          uint8_t format_index,

          uint8_t resolution_index,

          uint32_t frame_interval,

           

          )

          {

           

          glFrameSizeExpected = 0;

          if (CyU3PUsbGetSpeed() == CY_U3P_SUPER_SPEED)

          {

          if(format_index == UYVY_FORMAT_INDEX)

          {

          switch (resolution_index)

          {

          case SS_YUY2_1920x1080_INDEX:

          sensor_settings_1920*1080();                             //setting the sensor to stream 1920*1080

          break;

          case SS_YUY2_1280x720_INDEX:

          sensor_settings_1280*720();                                 //setting the sensor to stream 1280*1720

          break;

          case SS_YUY2_640x640_INDEX:

          sensor_settings_640x640();

          break;

           

          The fields like format_index, resolution_index etc needs to be parsed from glCommitCtrl  when bRequest is CY_FX_USB_UVC_SET_CUR_REQ and wValue is CY_FX_UVC_PROBE_CTRL

           

          Please share the modified firmware if you have more queries

           

          Regards,

          Rashi

          1 of 1 people found this helpful
          • 2. Re: How to dynamically change resolution of image sensor - AN75779
            BoJo_4384951

            Hello, RashiV_61!

             

            Thank you very much for your reply!

             

            The firmware we are developing is based on AN75779. We did not change the structure of the AN75779 firmware, we just added a few things on the top of it for the moment (mainly relative to I2C communication with our image sensor). We also added a few more Class specific Uncompressed VS format descriptors on appropriate place, as per your advice (link).

            When we connect our system to VirtualDub player, we can see that VirtualDub recognizes all the resolutions we included within USB descriptor which proves that capability request phase (initial communication between the Host and FX3 firmware) is successfully taking place even though I am not aware where exactly in the AN75779 firmware we are responding to CY_FX_USB_UVC_GET_DEF_REQ requests. Can you point me that place, please?

            Untitled.png

            Thank you for describing the process of changing resolution that is initiated by the Host. The same question again... where is the place in AN75779 firmware where we detect CY_FX_USB_UVC_SET_CUR_REQ commands from the Host? Is this somewhere within UVCAppEP0Thread_Entry() ?

            Call me a dumb but I could not find where the function CyFx3UvcAppSetVideoResolution() is defined and used in AN75779 firmware. Can you point that place to me as well? (or this is something I need to add myself).

             

            Regards,

            Bojan.

            • 3. Re: How to dynamically change resolution of image sensor - AN75779
              RashiV_61

              Hello Bojan,

               

              Thank you for confirming that video resolutions are visible in host application

              I am not aware where exactly in the AN75779 firmware we are responding to CY_FX_USB_UVC_GET_DEF_REQ requests. Can you point me that place, please?

              >> This request is handled in AN75779 where all the resolutions supported by the device needs to be sent. As the video resolutions are seen in the host application probe control of all the video resolutions need to be sent as mentioned in the last response of this thread How to support different video resolutions - AN75779

              For multiple resolution

              you can parse the frame index number and send the probe control data of the appropriate  resolution (as per the request from the host).

              case CY_FX_USB_UVC_GET_DEF_REQ: /* There is only one setting per USB speed. */

                                  if (usbSpeed == CY_U3P_SUPER_SPEED)

                                  {

                                    if(frame_index == 2)

                                      CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *glProbeCtrl_1280x720_60fps));

                                  }... so on

               

              where is the place in AN75779 firmware where we detect CY_FX_USB_UVC_SET_CUR_REQ commands from the Host? Is this somewhere within UVCAppEP0Thread_Entry() ?

               

              >>CY_FX_USB_UVC_SET_CUR_REQ request is handled in UVCHandleVideoStreamingRqts. You can search for this in the firmware.

              When this request comes the sensor needs to be configured for that resolution.

              set_Cur_req.PNG

              For multiple resolutions, CyFx3UvcAppSetVideoResolution API , is just for reference (is not in AN75779 firmware ) for you to understand what needs to be done when the request CY_FX_USB_UVC_SET_CUR_REQ  occurs.

              You can parse the frame index number and  set the sensor  for appropriate resolution.

              The requested frame index number  (as you are using single format so no need to parse format index) can be parsed from g1CommitCtrl global variable. For more details i.e. which array element need to parsed for frame index you can refer to UVC spec or refer the sample code in previous response                                  

              Regards,

              Rashi

              1 of 1 people found this helpful
              • 4. Re: How to dynamically change resolution of image sensor - AN75779
                BoJo_4384951

                Hello, RashiV_61!

                 

                Thanks for your clarifications! Let me summarize my understandings so far and ask few more related questions...

                 

                When the change resolution request comes from the Host, UVCHandleVideoStreamingRqts() function will be called.

                wValue = CY_FX_UVC_COMMIT_CTRL whereas

                bRequest = CY_FX_USB_UVC_SET_CUR_REQ.

                 

                apiRetStatus = CyU3PUsbGetEP0Data (CY_FX_UVC_MAX_PROBE_SETTING_ALIGNED,  glCommitCtrl, &readCount);

                 

                function will be called and, as a consequence, glCommitCtrl[] array will be filled with the details of the Host request.

                glCommitCtrl[2] = format_index => we currently have only one format: YUY2

                glCommitCtrl[3] = resolution_index

                 

                When we receive such a request from the Host, FX3 firmware should perform the following:

                1) Set image sensor to start streaming desired resolution

                2) Reply to the host with the following:

                 

                CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_XXX);

                 

                where glProbeCtrl_XXX probe we point to depends on the resolution_index. Each resolution supported should have its own  glProbeCtrl_XXX settings where Video format (YUY2 in our case) as well as Video frame index (resolution_index) will be specified together with desired frame interval and max video frame size. Video frame index (resolution_index) is unique for each resolution supported.

                 

                In our case we currently have 7 different resolution specified with the following indexes:

                 

                typedef enum CyFx3SSYUY2VideoRes_t

                {

                    YUY2_3214x2206_INDEX=1,

                    YUY2_4110x3006_INDEX,

                    YUY2_2054x1502_INDEX,

                    YUY2_2462x2054_INDEX,

                    YUY2_1934x1094_INDEX,

                    YUY2_1230x1026_INDEX,

                    YUY2_1606x1102_INDEX,

                } CyFx3SSYUY2VideoRes;

                 

                Please let me know if above understanding is correct.

                 

                Thanks once again for your great support!

                Cheers,

                Bojan.

                • 5. Re: How to dynamically change resolution of image sensor - AN75779
                  RashiV_61

                  Hello Bojan,

                   

                  Please refer to the attachment which mentions the negotiation between host and device while changing settings dynamically

                  streaming.PNG

                   

                  - VS_PROBE_CONTROL (SET_CUR)

                  This request will get the data  (video resolution to be streamed) from the host using this API and store it in g1CommitCtrl

                  apiRetStatus = CyU3PUsbGetEP0Data (CY_FX_UVC_MAX_PROBE_SETTING_ALIGNED,  glCommitCtrl, &readCount);

                   

                  The g1CommitCtrl data should be parsed and stored in global variables like

                  Please refer to Video Probe and Commit Controls in UVC spec

                   

                          glCurrentVideoFormatIndex = glCommitCtrl[2];

                          glCurrentVideoFrameIndex = glCommitCtrl[3];

                         glCurrentVideoFrameInterval = glCommitCtrl[3]; // 4bytes

                   

                   

                  - VS_PROBE_CONTROL(GET_CUR)

                  On this request the device will send the probe control structure of the video resolution asked by the host in the previous

                  This is as per the previous thread

                  For multiple resolution

                  you can parse the frame index number and send the probe control data of the appropriate  resolution (as per the request from the host).

                   

                   

                                  case FX3_USB_UVC_GET_CUR_REQ:

                                  case FX3_USB_UVC_GET_MIN_REQ:

                                  case FX3_USB_UVC_GET_MAX_REQ:

                                  case CY_FX_USB_UVC_GET_DEF_REQ: /* There is only one setting per USB speed. */

                                      if (usbSpeed == CY_U3P_SUPER_SPEED)

                                      {

                                        if(glCurrentVideoFrameIndex == 2)

                                          CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *glProbeCtrl_1280x720_60fps));

                                      }... so on

                                     break;

                   

                  - VS_COMMIT_CONTROL (SET_CUR)

                  When this request is sent from host sensor needs to be configured for appropriate video resolution

                  case FX3_USB_UVC_SET_CUR_REQ:

                              switch (CY_U3P_GET_LSB(wIndex))

                              {

                              case FX3_UVC_STREAM_INTERFACE_1:

                              {

                                  CyU3PThreadSleep(900);

                   

                                  CyFx3UvcAppSetVideoResolution(glCommitCtrl[2], glCommitCtrl[3], frameInterval, 0);

                             }

                              break;

                              }

                          }

                   

                  void

                  CyFx3UvcAppSetVideoResolution(

                  uint8_t format_index,

                  uint8_t resolution_index,

                  uint32_t frame_interval,

                   

                  )

                  {

                   

                  glFrameSizeExpected = 0;

                  if (CyU3PUsbGetSpeed() == CY_U3P_SUPER_SPEED)

                  {

                  if(format_index == UYVY_FORMAT_INDEX)

                  {

                  switch (resolution_index)

                  {

                  case SS_YUY2_1920x1080_INDEX:

                  sensor_settings_1920*1080();                             //setting the sensor to stream 1920*1080

                  break;

                  case SS_YUY2_1280x720_INDEX:

                  sensor_settings_1280*720();                                 //setting the sensor to stream 1280*1720

                  break;

                  case SS_YUY2_640x640_INDEX:

                  sensor_settings_640x640();

                  break;

                   

                  Can we implement this advanced communication scheme within Video control interface or we need to implement some third interface between the Host and FX3 to achieve this? In Extension Unit perhaps?

                  >> . You can use extension unit to add vendor specific blocks to the UVC spec.  This unit can have one or more inputs but only ony output. To access this  extension unityou need to develop your custom host application

                   

                  What I am missing here? How FX3 can tell the Host during start-up process which resolution will be played?

                  >> You can have more than one streaming interface. Yo can refer to UVC spec for basic details.

                   

                   

                  Regards,

                  Rashi

                   

                   

                  1 of 1 people found this helpful
                  • 6. Re: How to dynamically change resolution of image sensor - AN75779
                    BoJo_4384951

                    I don't see any attachment, RashiV_61. It seems you missed to attach the document.

                    • 7. Re: How to dynamically change resolution of image sensor - AN75779
                      RashiV_61

                      Hello Bojan,

                       

                      Apologies for the mistake.

                      I was referring to the image which i shared in the previous response. This was to make explanation more clear.

                       

                      streaming.PNG

                       

                      Regards,

                      Rashi