Exchange resolution control messages between the Host and FX3 configured as UVC-class device - AN75779

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
bojankoce
Level 3
Level 3
25 replies posted 10 replies posted 5 replies posted

Hello, guys.

In our system we have FX3 controller used to control various IMX image sensors through FPGA. In FX3 we have implemented the firmware from AN75779 (as a starting point) which makes our system recognized as UVC-class device from the Host PC.

There is a need to dynamically change the resolution of the video streams during runtime of the system. In this thread, you explained how we can change the resolution of the stream when the request is initiated by the Host PC.

1) Now, there is one more use case when the request for the resolution change will come from the FX3 device. This is the request to change the resolution from the default one to the desired one.

Namely, our system is designed to support different image sensors from IMX family. Consequently, we would like to be able to start streaming different resolutions depending on the IMX sensor connected (e.g. IMX250, IMX253, IMX420...). We are able to get the IMX sensor ID within CyFxUVCApplnInit() function. However, I am not sure how to inform the Host which resolution will be played. Host expects DEFAULT resolution to be played which is always THE LAST Class specific Uncompressed VS frame descriptor described in CyFxUSBSSConfigDscr[] descriptor!

Untitled.png

I tried with sending

CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_1934x1094_30fps);

to the Host as soon as I get IMX sensor ID but to no avail! It seems something more is needed.

What I am missing here? How FX3 can tell the Host during start-up process which resolution will be played?

2) We need to have a closer communication between the Host and FX3. In addition to requesting basic parameters like change resolution, get gain etc. we would like to be able to send some more requests from the Host to FX3 like:

- Start playing patterns

- Read the value of image sensor register and send it to me (e.g. register value that stores temperature).

To the best of my understanding, extension unit can be used to implement custom (vendor-specific) features (I am aware we need to customize Host application for that).

On page 11 (Section 2.3.2.1) of AN75779 document, you give some explanations on how to design UVC extension unit in FX3 firmware (this thread). I would like to try my luck and implement that. At the same place you mention uvc_extension_app_x64.exe file as well as

"Guidelines to run the Host application is provided in the readme.txt file in the attached project"

Where I can find that uvc_extension_app_x64.exe file and attached project you are talking about?

Thanks for your time and efforts.

Sincerely,

Bojan.

0 Likes
1 Solution

Hello Bojan,

From the explanation this is what i understood

- The sensor cannot be changed dynamically.

- Only one sensor will be connected at a time so one ID will be fetched.

- Based on this ID the default video stream need to be decided.

The default video settings for a sensor is set when the host requests Probe_Ctrl (get_cur)

  switch (wValue)

    {

        case CY_FX_UVC_PROBE_CTRL:

            switch (bRequest)

            {

                case CY_FX_USB_UVC_GET_INFO_REQ:

                    glEp0Buffer[0] = 3;                /* GET/SET requests are supported. */

                    CyU3PUsbSendEP0Data (1, (uint8_t *)glEp0Buffer);

                    break;

                case CY_FX_USB_UVC_GET_LEN_REQ:

                    glEp0Buffer[0] = CY_FX_UVC_MAX_PROBE_SETTING;

                    CyU3PUsbSendEP0Data (1, (uint8_t *)glEp0Buffer);

                    break;

                /* There is only one setting per USB speed. */

                    if (usbSpeed == CY_U3P_SUPER_SPEED)

                      if (sensor id == 1)

                    {

                        CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_{resolution_of_appropriate_frame_index});

                    }

                    else if (sensor_id == 2)

                    {

                        CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_{resolution_of_appropriate_frame_index});

                    }

                    break;

This request comes after device is enumerated. So if you want the default video settings based on sensor id, you have to get the sensor id from FPGA/sensor before enumeration starts.

Regards,

Rashi

Regards,
Rashi

View solution in original post

0 Likes
13 Replies
Rashi_Vatsa
Moderator
Moderator
Moderator
5 likes given 500 solutions authored 1000 replies posted

Hello Bojan,

Now, there is one more use case when the request for the resolution change will come from the FX3 device. This is the request to change the resolution from the default one to the desired one.

>> Please refer to section 9 of AN75779 and let me know the block diagram of the application you are working on. Are you using any FPGA or ISP before for interfacing multiple cameras to FX3.

>> As per UVC specification, you can have multiple streaming interface (i.e. one streaming interface per sensor). If the request from the host is for streaming interface 1 then that particular sensor can stream and so on. Only the host can initiate the resolution change request (as per spec). But in your custom host application this request can be triggered by FX3 through extension unit.

We need to have a closer communication between the Host and FX3. In addition to requesting basic parameters like change resolution, get gain etc. we would like to be able to send some more requests from the Host to FX3 like

>> Yes your understanding is right about extension unit. Please refer to this thread for the source code UVC Extension unit

Regards,

Rashi

Regards,
Rashi
0 Likes

Hello, RashiV_61​.

Here is a block diagram of the system:

ZaandamBlockDiagram.png

What is important to say is that we have only ONE image sensor connected in the system. Either IMX250 or IMX253 or IMX420 or any other image sensor from IMX family. There are 3 boards in the system connected in mezzanine way so that we can easily replace image sensor used.

FX3 communicates with FPGA through I2C link whereas FGA can configure IMX through SPI link. During initialization phase we are able to read ID of the IMX sensor and, consequently, to conclude which video resolution we would like to stream to the Host PC. For every resolution we plan to support we added Class specific Uncompressed VS frame descriptor in descriptor chain and defined appropriate probe control structure like:

uint8_t glProbeCtrl_1606x1102[FX3_UVC_MAX_PROBE_SETTING];

uint8_t glProbeCtrl_2054x1502[FX3_UVC_MAX_PROBE_SETTING];

uint8_t glProbeCtrl_1230x1026[FX3_UVC_MAX_PROBE_SETTING];

uint8_t glProbeCtrl_1934x1094[FX3_UVC_MAX_PROBE_SETTING];

uint8_t glProbeCtrl_4110x3006[FX3_UVC_MAX_PROBE_SETTING];

uint8_t glProbeCtrl_3214x2206[FX3_UVC_MAX_PROBE_SETTING];

uint8_t glProbeCtrl_2462x2054[FX3_UVC_MAX_PROBE_SETTING];

What we would like to implement in FX3 firmware would be the following initialization steps:

1) Read ID of the IMX image sensor

2) Based on IMX ID, set one of the 7 possible resolutions supported as default resolution and stream it to the Host.

There should not be any intervention of the Host during that initialization phase.

Currently, THE LAST Class specific Uncompressed VS frame descriptor described in CyFxUSBSSConfigDscr[] descriptor is considered as DEFAULT. We would like to change that default from FX3 firmware during initialization phase without any Host intervention. I hope I was clear in my explanations.

Do you have any idea how we can do it?

Thanks for the UVC Extension App for the Host. I will try to benefit from it!

Sincerely,

Bojan.

0 Likes

Hello Bojan,

From the explanation this is what i understood

- The sensor cannot be changed dynamically.

- Only one sensor will be connected at a time so one ID will be fetched.

- Based on this ID the default video stream need to be decided.

The default video settings for a sensor is set when the host requests Probe_Ctrl (get_cur)

  switch (wValue)

    {

        case CY_FX_UVC_PROBE_CTRL:

            switch (bRequest)

            {

                case CY_FX_USB_UVC_GET_INFO_REQ:

                    glEp0Buffer[0] = 3;                /* GET/SET requests are supported. */

                    CyU3PUsbSendEP0Data (1, (uint8_t *)glEp0Buffer);

                    break;

                case CY_FX_USB_UVC_GET_LEN_REQ:

                    glEp0Buffer[0] = CY_FX_UVC_MAX_PROBE_SETTING;

                    CyU3PUsbSendEP0Data (1, (uint8_t *)glEp0Buffer);

                    break;

                /* There is only one setting per USB speed. */

                    if (usbSpeed == CY_U3P_SUPER_SPEED)

                      if (sensor id == 1)

                    {

                        CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_{resolution_of_appropriate_frame_index});

                    }

                    else if (sensor_id == 2)

                    {

                        CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_{resolution_of_appropriate_frame_index});

                    }

                    break;

This request comes after device is enumerated. So if you want the default video settings based on sensor id, you have to get the sensor id from FPGA/sensor before enumeration starts.

Regards,

Rashi

Regards,
Rashi
0 Likes

Hello RashiV_61​.

Your understanding is correct! Only one sensor will be connected and, based on its ID, default video resolution needs to be decided.

I tried what you suggest:

         case CY_FX_UVC_PROBE_CTRL:

            switch (bRequest)

            {

          ...

                case CY_FX_USB_UVC_GET_CUR_REQ:

                case CY_FX_USB_UVC_GET_MIN_REQ:

                case CY_FX_USB_UVC_GET_MAX_REQ:

                case CY_FX_USB_UVC_GET_DEF_REQ: /* There is only one setting per USB speed. */

                    if (usbSpeed == CY_U3P_SUPER_SPEED)

                    {

                          switch(_imx_id.model){

                               case IMX_250:

                                    CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_2462x2054_20fps);

                                    break;

                               case IMX_253:

                                    CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_4110x3006_10fps);

                                    break;

                               case IMX_420:

                                    CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_3214x2206_10fps);

                                    break;

                               default:

                                    CyU3PUsbStall (0, CyTrue, CyFalse);

                                    break;

                          }

                    }

else

...

But with NO SUCCESS! Only when I manually put desired resolution at THE LAST place within CyFxUSBSSConfigDscr[] Descriptor chain only in that case there is a success.

I tried to get sensor ID within CyFxUVCApplnInit (); function (on the same place your Aptina sensor is initialized) but also before entering kernel with CyU3PKernelEntry ();. In both cases there were no success.

It seems something more is needed?!

Cheers,

Bojan.

0 Likes

Hello Bojan,

Please try getting the sensor id  before CyFxUVCApplnInit (); is called.

After getting it, please print the value of the sensor id so that we can come to know which probe control structure is sent.

Please add necessary debug prints in the code where Probe control structure is sent.

If possible probe the usb lines using wireshark and share the traces. Start the wireshark capture before programming the FX3

Regards,

Rashi

Regards,
Rashi
0 Likes

Hi, RashiV_61

I tried to get sensor ID right after I2C driver is initialized with CyFxUVCApplnI2CInit (); function. I believe this is the earliest place in the code I can try to get sensor ID.

However, there were no success in changing default resolution!

I tried to record the traffic on USB link for the case I have a good stream (captureFile_SUCCESS) as well as for the case when I try to change default resolution without success (captureFile_FAILURE). Find attached both files. I hope I captured what you need.

Sincerely,

Bojan.

0 Likes

Hello Bojan,

The wire shark traces do not capture the get/set request sent by the host.

Please let me know what is the sensor id  and which probe control structure is being sent for the failure case.

If possible share the firmware for us to check.

Regards,

Rashi

Regards,
Rashi
0 Likes

Hello, RashiV_61​.

I am currently using IMX250 sensor whose ID is 0xFA. This value is obtained by reading two internal registers of IMX250 sensor and it is not communicated over USB. It is needed for FX3 firmware locally - only in order to know which IMX sensor is connected so that default resolution and operation mode can be set properly.

I am sharing with you complete EZ-USB project. Unzip password is sent in private message.

Probe structure that is sent in the case of failure is glProbeCtrl_2462x2054_20fps() (see line 2068 in uvc.c file). The only way to play the stream successfully is to set       /* Class specific Uncompressed VS frame descriptor - 2462 x 2054 @ 20FPS */ on THE LAST place in CyFxUSBSSConfigDscr[] descriptor.

Please take a deeper look and let me know if there is something I am missing. Thanks in advance!

Sincerely,

Bojan.

P.S. Does the wire shark traces capture the get/set request sent by the host for the case I have a good stream (captureFile_SUCCESS file)?

0 Likes

Hello Bojan,

The descriptors sent to the host should be in sequential manner i.e in sequence of their frame index. Pleas make necessary changes for that.

Generally, the video resolution with frame index 1 can be set as default but we can try sending probe control with different frame index (in the GET_CUR request)

Probe structure that is sent in the case of failure is glProbeCtrl_2462x2054_20fps() (see line 2068 in uvc.c file).

>> This is what is expected right? In your firmware glProbeCtrl_2462x2054_20fps() is sent when you are using IMX250.

Please check the  glProbeCtrl[3] value when there is CY_FX_USB_UVC_SET_CUR_REQ: (PROBE_CTRL). You can print this value using debug print and let me know the results

Also, i went through the traces again but didn't find probe set and get requests. Please take another trace by starting the capture before programming FX3 and then open the host application. After some data transfers then stop the capture

Meanwhile i am going through the firmware you shared in more detail

Regards,

Rashi

Regards,
Rashi
0 Likes

Hi there!

The descriptors sent to the host should be in sequential manner i.e in sequence of their frame index. Pleas make necessary changes for that.

I tried to arrange descriptors in sequential manner (in sequence of their frame indexes). There were no success!

Probe structure that is sent in the case of failure is glProbeCtrl_2462x2054_20fps() (see line 2068 in uvc.c file).

>> This is what is expected right? In your firmware glProbeCtrl_2462x2054_20fps() is sent when you are using IMX250.

Yes, this is what is expected.

Please check the  glProbeCtrl[3] value when there is CY_FX_USB_UVC_SET_CUR_REQ: (PROBE_CTRL). You can print this value using debug print and let me know the results

Unfortunately, I don't have a possibility to debug with print() function because we are on custom PCB. I only have JTAG debugging possibilities. Anyway, as you will see in a code I shared with you, glProbeCtrl[] is not used. I send one of new glProbeCtrl_xxxx probes that I defined for every resolution supported.

Also, i went through the traces again but didn't find probe set and get requests. Please take another trace by starting the capture before programming FX3 and then open the host application. After some data transfers then stop the capture

Even my previous traces are captured the way you suggested (I started capturing before programming FX3). I did it again now by taking a closer look what is going on while I am doing the activities like programming FPGA and downloading the firmware into FX3. Here is what I noted:

1) In the case of captureFile_SUCCESS_1 file:

a) I give power supply to the system: Frames 1 - 12 are recorded

b) I download the code to FPGA (FX3 is still not programmed): Frames 12 - 6054 are recorded

c) I download the code into FX3: Frames 6054 - 65xxx are recorded (not sure about exact number)

d) I start streaming with VLC player: the rest of the frame are recorded

2) In the case of captureFile_FAILURE_1 file:

a) I give power supply to the system: Frames 1 - 12 are recorded

b) I download the code to FPGA (FX3 is still not programmed): Frames 12 - 6054 are recorded

c) I download the code into FX3: Frames 6054 - 6360 are recorded (not sure about exact number)

d) I start streaming with VLC player: NOTHING HAPPENS!

This might indicate that FX3 stalls even before starting exchanging data with VLC player!!!

Sincerely,

Bojan.

0 Likes

Hello Bojan,

Yes, this is what is expected.

>> Please let me know where is the problem. What is the default value set in the host application (VLC) when the FX3 enumerates as Camera

Unfortunately, I don't have a possibility to debug with print() function because we are on custom PCB. I only have JTAG debugging possibilities. Anyway, as you will see in a code I shared with you, glProbeCtrl[] is not used. I send one of new glProbeCtrl_xxxx probes that I defined for every resolution supported

>> I have taken this snippet from the firmware you shared. Please let me know the value of g1ProbeCtrl[3]. You can use JTAG for that

snippet.PNG

Please refer to wireshark traces i took with default AN75779 firmware and the requests which are sent by host. Refer to 1599 trace which is Get CUR response. I want to check the same with your firmware. Please share similar trace. Can you use AmCap instead to VLC just for convenience

Regards,

Rashi

Regards,
Rashi
0 Likes

Hello, RashiV_61​.

I have taken this snippet from the firmware you shared. Please let me know the value of g1ProbeCtrl[3]. You can use JTAG for that

In both cases, when I am successfully playing the stream and when I am unable to play it, the value of glProbeCtrl[3] is equal to 0x01.

By debugging over JTAG I also noted that this part of the code is properly running:

Untitled.png

CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_2462x2054_20fps);

is properly called which means that Host PC is informed about our intentions. That gave me a hint to try playing the stream with different video players. I used VLC, AmCap and Camera app that comes with Windows 10. VLC and AmCap were unable to play the stream (in the case when I wanted to play the stream that is different from default one). However, Camera app were able to adapt and play the desired stream!

It can be simply that the problem is on the Host side. Could be that VLC and AmCap are designed to simply start playing default stream not taking into account what the FX3 is initially telling them. What do you think about that?

Sincerely,

Bojan.

0 Likes

I updated the firmware so that I now keep track of active resolution (active glProbeCtrl_xxxx[] probe) and covered every resolution supported.

VLC and AmCap players always start playing default resolution (the one with the resolution_index = 1) whereas Camera app from Windows 10 is able to start with resolution I want (other than default one). I was able to dynamically change resolution by sending requests from all the three players!

This proves me that the firmware on FX3 is properly doing its job. It is now up to our custom Host software to behave the same way like Camera app! Anyway, I think it is time to close this thread given the fact we have something.

I will open a new thread for the issue 2) in my initial question.

Thank you very much for being with me! You are awesome!

Sincerely,

Bojan.