14 Replies Latest reply on Jun 13, 2020 8:39 PM by JayakrishnaT_76

    How to dynamically change video encoding format during runtime - AN75779

    AbMa_4712156

      We are successfully streaming a video from an image sensor to our host PC through an FX3 module. An FPGA acts as a bridge between the FX3 module and the sensor.

      Our FX3 firmware is based on AN75779. We have read the AN75779 chapter 2 to get an understanding of the UVC control and streaming interface. We have also referred to the following threads to get an idea of the modifications we need to make:

      Use FX3 as UVC camera based on AN75779

      Re: How to dynamically change resolution of image sensor - AN75779

      Exchange resolution control messages between the Host and FX3 configured as UVC-class device - AN75779

       

      We wish to be able to change the streaming video encoding format from the default 'YUY2' to 'Y16 ' dynamically during runtime by sending an input from the host PC. For that we have made the following modifications:

       

      1.) In the cyfxuvcdsrc.c file, within CyFxUSBSSConfigDscr[] under /* Class-specific Video Streaming Input Header Descriptor */ we have changed  the number of format desciptor follows from 0x01 to 0x02.

       

      2.) We have added an additional /* Class specific Uncompressed VS format descriptor */  for the Y16 format along with its own Class specific Uncpompressed VS frame descriptor. We have set the Format desciptor index as 0x02 for Y16.

       

      3.) We have added a probe structure in the uvc.h file for streaming Y16 format video upon receiving the SET_CUR from the probe request, and modified the following line in the probe structure:

      0x02,                       /* Use 1st Video format index */

       

      4.) We have added the following lines in the uvc.c file under UVCHandleVideoStreamingRqts{}:

       

      case CY_FX_USB_UVC_GET_DEF_REQ: /* There is only one setting per USB speed. */

                          if (usbSpeed == CY_U3P_SUPER_SPEED)

                          {

                              if(glProbeCtrl[2] == 1)

                                         CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl);

                              else

                                         CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_Y16);  //Probe structure for streaming Y16 encoded video

                          }

                          else

                          {

                              CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl20);

                          }

                          break;

       

      5.) We understand we have to edit the descriptor size in each of the appropriate descriptors in the cyfxuvcdsrc.c file, but we are unsure how to calculate the appropriate size of all the class-specific descriptors as well as the overall size. For trial purposes we have made an upper limit estimation of the descriptor sizes.

       

      Currently we have an OpenCV Python program to stream the video. The stream is coming out fine when using the unmodified firmware(default YUY2 format basic firmware that came directly with FX3 SDK from the cypress website) to run the UVC device. However the program is unable to detect the video stream from the modified firmware.

      Is our understanding correct ? Have all the above mentioned modifications been done correctly and with the proper understanding ? If not what are the required modifications that need to be done ?

       

      We sincerely thank you for your time and efforts !

       

      Sincerely,

      Abdul Mannan

        • 1. Re: How to dynamically change video encoding format during runtime - AN75779
          JayakrishnaT_76

          Hello,

           

          Please let me know if the modified firmware is enumerating properly on the host. From this we can understand if there are any problems on the descriptors or not.

          Please refer to the following thread to understand how the descriptors are to be changed for using multiple formats:

          i want to enumerate more than one GUID in SS mode,what should i do? dose this just like enumerating multi frames?

           

          Also, please share the complete project so that we can check how the streaming related requests are handled in the firmware.

           

          Best Regards,

          Jayakrishna

          • 2. Re: How to dynamically change video encoding format during runtime - AN75779
            AbMa_4712156

            Hello,

             

            It is unsure if the modified firmware is enumerating properly on the host. We don't know a proper method of ensuring whether enumeration is taking place, so we kindly request help with that.

             

            Please find the complete project in a zip file in the gdive link below:

            https://drive.google.com/file/d/1PJ981oijKjQWwfQHgEcbCjFOWLqzeJ4h/view?usp=sharing

            • 3. Re: How to dynamically change video encoding format during runtime - AN75779
              JayakrishnaT_76

              Hello,

               

              If possible, please share the project here as the drive link denies access.

              Also, please go through the following link to understand how the descriptors are to be changed for supporting multiple formats.

              i want to enumerate more than one GUID in SS mode,what should i do? dose this just like enumerating multi frames?

               

              Please let me know the following details also:

              1. Which OS are you working with?

              2. Did you try testing the firmware with a standard UVC host application (like AMCap or MPC-HC in Windows) ? This will not stream Y16 format as it is not UVC compliant. But it should stream YUY2 format properly.

               

              Best Regards,

              Jayakrishna

              • 4. Re: How to dynamically change video encoding format during runtime - AN75779
                AbMa_4712156

                Hello,

                 

                Apologies for the previous link, here is the updated link for the project:

                USBVideoClass_UVC_CDC.zip - Google Drive

                 

                We are currently working on Windows 10. We did not use any standard UVC host application to test the firmware, we are using a custom python program on our PC from which we are streaming directly through an IDE (for trial purposes, until we have developed our own custom application). We have been successful in streaming in YUY2 as well as Y16 formats in two separate sessions. We did so by only changing the GUID line in the /* Class specific Uncompressed VS format descriptor */.

                 

                Now we wish to be able to change the format dynamically during runtime. For this purpose, we have made the descriptor changes according to the link you have shared. But our PC fails to detect the FX3 device after uploading the img file. As you said this is probably because of enumeration problems due to improper modifications in the cyfxuvcdscr.c descriptor file.

                 

                Regards,

                Abdul Mannan

                • 5. Re: How to dynamically change video encoding format during runtime - AN75779
                  JayakrishnaT_76

                  Hello,

                   

                  Please find the modified .dscr file. The modifications that I have made are:

                  1. The descriptor size field of Class-specific Video Streaming Input Header Descriptor is changed to 0x0F. An extra 0x00 is added at the end for to make the size equal to 0x0F. This change was mentioned in the thread link that I shared before.

                  2. Low Byte of Class Specific VS descr is changed to 0x81 from 0x80.

                  3. The total size of Superspeed configuration descriptors is 0x167. So, the corresponding field in the descriptor is made as 0x67, 0x01.

                   

                  Please find the snapshot of the device enumerated in the device manager below:

                   

                  Please try this and let me know if you still face the issue of enumeration. If the device is enumerated properly, please let me know if the format switching is happening or not.

                   

                  Best Regards,

                  Jayakrishna

                  1 of 1 people found this helpful
                  • 6. Re: How to dynamically change video encoding format during runtime - AN75779
                    AbMa_4712156

                    Hello,

                     

                    Yes, the enumeration problem has been solved. The device is being detected properly. Thank you so much for helping with that.

                     

                    However, the format switching is not taking place. We understand the Class specific Uncompressed VS format descriptor for Y16 is below that of YUY2 and thus the Y16 format is being selected by default, but the output format is not changing regardless of which input we give (earlier we switched to YUY2 by going directly to the .dscr file and changing the GUID). Is this because of an error in the probe structure in the uvc.c file ?

                     

                    I have attached the python code we are using below in for reference.

                     

                    import cv2

                     

                    device_index = 0

                    cap = cv2.VideoCapture(device_index + cv2.CAP_DSHOW)

                    cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('Y','U','Y','2'))              #selecting YUY2 by default

                    cap.set(cv2.CAP_PROP_CONVERT_RGB, True)

                    print(cap.isOpened())

                    while(True):

                                    code, frame = cap.read()

                                    img_scaled = cv2.normalize(frame, dst=None, alpha=0, beta=63255, norm_type=cv2.NORM_MINMAX)

                                    cv2.imshow('frame', img_scaled)  

                                    k = cv2.waitKey(1)


                                    #if 'q' is pressed, the exit

                                    if k == ord('q'):

                                           break


                                    #if 'c' is pressed, switch to YUY2 format

                                    if k == ord('c'):

                                            cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('Y','U','Y','2'))

                                            cap.set(cv2.CAP_PROP_CONVERT_RGB, True)


                                    #if 'g' is pressed, switch to Y16 format

                                    if k == ord('g'):

                                            cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('Y','1','6',' '))

                                            cap.set(cv2.CAP_PROP_CONVERT_RGB, False)

                    cap.release()

                    cv2.destroyAllWindows()

                     

                     

                    The underlined lines above are where we are trying to switch the format. However this isn't changing anything. Could you help with this ?

                    Thank you for your time and efforts.

                     

                    Best Regards,

                    Abdul Mannan

                    • 7. Re: How to dynamically change video encoding format during runtime - AN75779
                      JayakrishnaT_76

                      Hello,

                       

                      Please confirm if you are using the firmware that you shared with me or not. Or are you using a different firmware?

                      Also, please confirm which format is streaming by default. Is it Y16 or YUY2?

                      To isolate if it is a host application fault, can you please try testing the firmware with AMCap or MPC-HC? As i mentioned in my previous response, these wont stream Y16 format. Still, it can be used to understand if the switching happens or not.

                       

                      Best Regards,

                      Jayakrishna

                      • 8. Re: How to dynamically change video encoding format during runtime - AN75779
                        AbMa_4712156

                        Hello

                         

                        Yes I can confirm that we are using the same firmware that I shared with you. I confirm now that YUY2 format is the default format. As per your suggestion, we have tried streaming with both AMCap and MPC-HC. Both MPC-HC and AMCap are streaming YUY2 properly but they encounter an error or don't display anything while trying to switch to Y16. Your help is appreciated.

                         

                        Best Regards,

                        Abdul Mannan

                        • 9. Re: How to dynamically change video encoding format during runtime - AN75779
                          JayakrishnaT_76

                          Hello,

                           

                          Please find the following snapshot which shows the sequence of requests that host use to switch the streaming format or resolution.

                          I see that you have handled PROBE: SET_CUR request properly in the firmware. But after this, the host will issue PROBE: GET_CUR request to read the updated values from the UVC device. This is not handled in your firmware. Also, if the updated probe/commit control structure sent by the device matches with that sent by the host, then the host will send COMMIT:SET_CUR request. When this is received, you should update the sensor registers to stream with the new format.

                          Please make the following modifications to your code:

                          1. Use two global variables glCurrentFormatIndex and glCurrentFrameIndex.

                           

                          2. When PROBE: SET_CUR request is obtained, replace the existing code by the following lines of code:

                            

                                glCurrentFrameIndex = glCommitCtrl[3];

                               glCurrentFormatIndex = glCommitCtrl[2];

                           

                          3. Now, when PROBE:GET_CUR request is received, under the check for usb speed, check what is the value of  glCurrentFormatIndex. If the value is 1, then send the probe control structure for YUY2. And if it is 2, then send the probe control structure for Y16. (You have made this change in COMMIT: GET_CUR which is not needed).

                           

                          4. After this, the host will issue COMMIT: SET_CUR. At that time, you need to configure the sensor to output the corresponding format (by checking glCurrentFormatIndex) by writing the sensor registers. At the moment, always it is calling the function SensorScaling_HD720p_30fps. Which I assume is writing the sensor registers for streaming YUY2 data and not Y16.

                           

                          Please let me know if you have any queries on this.

                           

                          Best Regards,

                          Jayakrishna

                          1 of 1 people found this helpful
                          • 10. Re: How to dynamically change video encoding format during runtime - AN75779
                            AbMa_4712156

                            Hello,

                             

                            Thank you for the information. I have declared the two global variables and updated the PROBE:SET_CUR request as you suggested. I have now modified the PROBE:GET_CUR request with the if/else condition according to your suggestion and removed the condition from the COMMIT:GET_CUR request that I had put earlier.

                             

                            I have one query: what is the purpose of COMMIT:GET_CUR ?

                             

                            And in modification 4.) you have asked me to "configure the sensor to output the corresponding format (by checking glCurrentFormatIndex) by writing the sensor registers". I am unaware how to configure the sensors according to the need. Kindly suggest how to go about configuring the sensors.

                             

                            Note: Having done modification 1,2,3 which you suggested, the format still doesn't switch. But I assume this is expected because I still haven't configured the sensors ?

                             

                            Your help is extremely appreciated.

                             

                            Best Regards,

                            Abdul Mannan

                            • 11. Re: How to dynamically change video encoding format during runtime - AN75779
                              JayakrishnaT_76

                              Hello,

                               

                              Apologies for making a mistake in my previous response. COMMIT: GET_CUR request is needed in the firmware. You can use the same code as used in PROBE: GET_CUR here also. Please refer to Table 4-49 of UVC specification to understand the need for COMMIT: GET_CUR. Again apologies for making the mistake.

                              http://www.cajunbot.com/wiki/images/8/85/USB_Video_Class_1.1.pdf

                               

                              Regarding sensor configuration settings, as you might be knowing, a sensor should be configured to stream at a desired format and resolution. This is done by writing the corresponding configuration settings into the sensor registers. Please contact the sensor vendor to get the settings that are to be used for making the sensor output the desired format, resolution and frame rate. Once this is done, you need to write your own function to write the register by referring to the existing function. Please let us know if you face any trouble while writing the sensor registers. But the sensor configurations should be obtained from the sensor vendor itself.

                               

                              Best Regards,

                              Jayakrishna

                              • 12. Re: How to dynamically change video encoding format during runtime - AN75779
                                AbMa_4712156

                                Hello,

                                 

                                The sensor module we are using is custom made. It is connected to the FX3 device through an FPGA. The signal connected to the FX3 from the FPGA are Frame_valid, Line_valid and a 16-bit data bus in a UART method. Bottom line is : there is no sensor dependency as far as the sensor module is concerned on the FX3. Is there any way to bypass any sensor dependency in the firmware code to do the format switch ?

                                 

                                Regards

                                Abdul Mannan

                                • 13. Re: How to dynamically change video encoding format during runtime - AN75779
                                  AbMa_4712156

                                  Hello,

                                   

                                  Format switching is working now. The problem was that in the Class-specific Video Streaming Input Header Descriptor the element at index 7 responsible for enabling format switching was set to zero. It has now been set to one. Thank you so much for your time and help. We highly appreciate it. You are amazing !!

                                   

                                  Best Regards,

                                  Abdul Mannan

                                  • 14. Re: How to dynamically change video encoding format during runtime - AN75779
                                    JayakrishnaT_76

                                    Hello,

                                     

                                    Thanks for the update. We are glad to hear that the issue is resolved!

                                     

                                    Best Regards,

                                    Jayakrishna