Streaming raw10 1080P 60 fps

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
Rohanth
Level 4
Level 4
First like received 50 sign-ins 25 replies posted

Hello Team,

I'm trying to stream 1080P MIPI Raw 10 image sensor . I if configure the MIPI transmitter section and receiver section for 1080p 30 Fps There is no problem in receiver side and It streams correctly. When I done this for 60 FPS I cant change the the output pixel clock in receiver configuration and it shows minimum is 129 and maximum is 100 even if my output video format size is 24-bit.

my camera can support 8/10/20 bit raw output of Bayer format and both continues and gated MIPI clock mode.

Kindly suggest me some settings for 1080P 60 fps.

Thankyou

0 Likes
1 Solution

Hello,

I still suspect that the sensor configuration settings are not proper. As mentioned before, CX3 is just a bridge between image sensor and host. It just forwards the data obtained from the image sensor to the host after adding UVC header. No additional frame data is added by CX3. The prints that you are seeing indicates the total amount of data sent by the image sensor. As mentioned before, the expected frame size is calculated as follows:

Frame size = (1920 x 1080 x 10) / 8 = 2592000 bytes

But, from the debug prints, the frame size received by CX3 is 3246480 bytes which is again greater than 2592000 bytes. Initially, you were receiving 3903348 bytes from the image sensor instead of 2592000 bytes. This was brought down to 3246480 bytes by changing the sensor configuration settings. I strongly suspect that the sensor configuration settings used currently is also not correct for your desired resolution. So, kindly check the configuration settings again with the image sensor vendor.

Best Regards,
Jayakrishna

View solution in original post

0 Likes
24 Replies
JayakrishnaT_76
Moderator
Moderator
Moderator
First question asked 1000 replies posted 750 replies posted

Hello,

Please share the snapshot of CX3 Receiver configuration tab of MIPI configuration utility so that we can check if the settings are proper or not.

Best Regards,
Jayakrishna
0 Likes

Hello,

I tried with 16 bit output video format and 24 bit output video format, hear I'm sharing the  screenshots for my configurations.

Image sensor configurations for 16 bit

Sensor_config_16b.png

 Receiver configurations

receiver_config_16b.png

Sensor configuration for 24 bit

Sensor_config_24b.png

 Receiver configuration for 24 bit

receiver_config_24b.png

Kindly check this and say what I should change.

*image sensor frame configurations are based on the image sensor that I have been using and it working for 30 fps Configurations.*

Thankyou

0 Likes

Hello,

The configuration settings used for 24 bit output is correct. The error shown in the CX3 configuration Utility is due to a bug which will be fixed in the next release of FX3 SDK. You can ignore this error and go forward to build the project with the configuration settings used for 24 bit output. Please let me know if you face any difficulties while doing this.

Best Regards,
Jayakrishna
0 Likes

Hello,

I tried The same 24 bit configuration from the above images and build the project the in my camera app it says that media capture failed.

Should I change any format/descriptors internally or should I try using some other camera application. 

*The camera application that I'm using is amcap, e-cam-view and default camera*

Kindly say what will be the problem

Thankyou 

0 Likes

Hello,

Please refer to the following KBA for packing the incoming data into 24 bits:

https://community.cypress.com/t5/Knowledge-Base-Articles/Streaming-RAW10-Format-Input-Data-to-16-24-...

The modifications mentioned in the KBA mentioned above should be added in your project also. If you are facing issues even after adding the modifications mentioned in the KBA, then kindly share the project after removing the sensor configuration settings so that I can check it.

Best Regards,
Jayakrishna
0 Likes
lock attach
Attachments are accessible only for community members.

Hello, 

I tried the KBA which u send previously and my cx3is got enumerated as UVC and There are some mismatch in the frame size  

"these ware the frame size that I got in UART logs

Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3325 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3326 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3327 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3328 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3329 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3330 Frm_Sz = 3903348 B


TimeDiff = 895 ms FPS = 33


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3331 Frm_Sz = 3903348 B

Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3332 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3333 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3334 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3335 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3336 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3337 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3338 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3339 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3340 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3341 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3342 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3343 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3344 Frm_Sz = 3903348 B


Prod = 106 Cons = 105 Prtl_Sz = 852 Frm_Cnt = 3345 Frm_Sz = 3903348 B

"

If I dint change anything in descriptor I got " media capture failed" but still its has been enumerated as UVC is there any option to view image in 24 bit.

If possible kindly suggest that.

hear I'm attached my project file kindly check for that.

Thankyou.

 

0 Likes

Hello,

The modifications mentioned in the descriptors needs to be done for streaming the video data properly. If the modifications are not done, then the descriptors will not be set correctly and can cause troubles to render the video stream.

From the UART debug logs, I understand that the frame size is constant but greater than the required size. The required frame size can be calculated as follows:

Frame size = (1920 x 1080 x 10) / 8 = 2592000 bytes

The data size shown in the UART debug log is 3903348 bytes

This means that the sensor is configured to send more data. CX3 is just a bridge that accepts the MIPI data from the sensor and forward the same to the host. If the sensor is sending more data in one frame, then the same will be forwarded to the host. In addition to this, you can also find that the fps is also shown as 33. This means that the fps set for the sensor is also not correct.

Kindly approach the image sensor vendor and check if the image sensor configuration settings are proper or not. 

Best Regards,
Jayakrishna
0 Likes

Hello,

I tried connecting this UVC device in Linux I got "CB failure" in UART logs, Kindly clarify what is "CB failure".

In mean while I requested for new sensor configuration probably it may take 2 more days to verify that.

Thankyou

0 Likes

Hello,

CB failure occurs when the API CyU3PDmaMultiChannelCommitBuffer() fails. We have seen this issue when the host is slow to issue IN tokens to the device. As you can see that the issue is seen only on Linux host, the same reason holds valid. Please let me know if the device is able to recover by itself when the CB failure is seen or not. That is, are you able to see the data being transferred in the UART logs after the CB failure is seen?

In mean while I requested for new sensor configuration probably it may take 2 more days to verify that

>> Sure, please let me know if you face any issues after trying with the new configurations.

Best Regards,
Jayakrishna
0 Likes

Hello,

*are you able to see the data being transferred in the UART logs after the CB failure is seen?

No,  I dint get any data transfer logs in UART.

Thankyou

0 Likes

Hello,

I would recommend you to create a new thread for discussing the CB failure issue. This could help other customers also who face the same issue.  We can continue our discussion on the RAW10 60fps streaming issue in this thread.

Best Regards,
Jayakrishna
0 Likes

Hello,

I have been created a new thread for CB failure

https://community.cypress.com/t5/USB-Superspeed-Peripherals/CB-failure-while-

streaming-1080P-60fps/m-p/283731#M27629

hear is the link for new thread.

Thankyou

0 Likes

Hello,

We got the new sensor configuration for 60 fps and still its is not streaming the data.

1. I tried generating code for 24 bit raw 10 60fps and replaced the dummy registers with our image sensor registers and tried to stream it says media capture fail in my camera app.

2. I tried replacing "Number of bits per pixel" in" Class specific Uncompressed VS format descriptor" from 10 t0 16. then I got  below lines in UART logs.


"Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 869 Frm_Sz = 3246480 B
Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 870 Frm_Sz = 3246480 B
TimeDiff = 503 ms FPS = 59
Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 871 Frm_Sz = 3246480 B
Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 872 Frm_Sz = 3246480 B
Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 873 Frm_Sz = 3246480 B
Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 874 Frm_Sz = 3246480 B
Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 875 Frm_Sz = 3246480 B"

3. I have tried streaming 1080p 60fps register in 1080p 30fps code, Which is streaming and got frame rate as 33fps.

according that KBA that you shared previously we are changing the descriptor form 1920X1080 to 1200x1080, so its not 1920x1080.

My question is can we stream 1080p 60fps directly or not ?

0 Likes

Hello,

As you might be knowing, UVC applications will not support any formats that are not UVC compliant. RAW10 format is not UVC compliant. So, it is not supported by UVC applications.

Now, the video that you are trying to stream is 1080P RAW10 at 60FPS. Therefore, the bandwidth requirement can be found out as follows:

Data rate = 1920 * 1080 * 10 * 60 = 1.15Gbps

This is supported by CX3. But, CX3 needs to either pad this incoming data into 16 bits or it needs to pack the incoming data into 16/24 bits. If CX3 is configured to pack the incoming data, then the data rate calculated above will not change. But, if CX3 is configured to pad the incoming data into 16 bits, then the data rate will change. The new data rate is given below:

Data rate = 1920 * 1080 * 16 * 60 = 1.85Gbps

But, when a 16 bit parallel bus is used, the maximum bandwidth supported by CX3 is only 1.6Gbps. So, it is not possible to pad the incoming data into 16 bits for this case.

So, the only left over options are either to pack the incoming data into 16 or into 24 bits. For this, you need to change the descriptors as mentioned in the KBA. After these modifications are made, you can use a default UVC player to check if you are receiving any data or not. But, to obtain the original video stream, you need to develop a custom host application itself.

Please do not do any other modifications other than the ones given in the KBA. Please share the UART debug logs if you are facing issues after making the modifications mentioned in the KBA.

Best Regards,
Jayakrishna
0 Likes
lock attach
Attachments are accessible only for community members.

Hello,

>>regarding KBA that you shared

I tried to modify the descriptor , But still I get media capture failed in the default camera application.

In UART logs I get 

"Sensor is in sleep
EnterSuspendMode Status = 0x0, Wakeup reason = 0x8
Sensor is in sleep
EnterSuspendMode Status = 0x0, Wakeup reason = 0x8"

I dint see any data in UART logs.

I do verified that MIPI is streaming. 

Hear I'm uploading my project without sensor register configuration file Kindly check the UVC descriptors  changes.

 

>> regarding custom host application.

pastedImage_20.png

 I found this format in the KBA that you previously shared,

How to arrange these pixel data in rows and column of  1920x1080 frame.

Kindly explain how to split and re-arrange pixel data in my custom host application.

Thankyou.

0 Likes

Hello,

Can you share the project again without removing the cyu3imagesensor.c file? You can remove the sensor configurations inside the structures declared as it is confidential. Basically, I would like to have a firmware that can be compiled without any errors for testing at my end. You can just remove the sensor configuration settings only from the project.

Please share the project with the modification mentioned above so that I can try to test it at my end.

Regarding the custom host application, please find my comments below:

1. Based on the KBA, when 24 bit output is used, the GPIF II block samples the data as follows:

  • First clock cycle - P1[9:2], P2[9:2], P3[9:2],
  • Second clock cycle - P4[9:2], P1[1:0], P2[1:0], P3[1:0], P4[1:0], P5[9:2]
  • Third clock cycle –  and so on.

2. For First clock cycle - P1[9:2], P2[9:2], P3[9:2], the data is presented to the GPIF II pins as follows:

Pixel     GPIFII pins

P1[2] - DQ0 LSB

P1[3] - DQ1

....

....

P3[9] - DQ23 MSB

This data will then be put in little endian format into the DMA buffers. CX3 will add a 12 byte UVC header at the start of each DMA buffer. This gives information on the End of frame. The data that follows after this UVC header will be the frame data that was sampled and stored into the DMA buffers by the GPIF II block.

Best Regards,
Jayakrishna
0 Likes
lock attach
Attachments are accessible only for community members.

Hello,

Hear I'm attaching my project file as you mentioned.

Thank you

0 Likes

Hello,

Thank you for sharing the project as requested. I tried to build the project that was shared in your previous response. After this, I downloaded the firmware into Denebola kit. I tested with the Camera Application of Windows 10 and also using AMCap. I did not find the media capture failed error that you saw while testing. Please refer to the snapshot of AMCap below:

JayakrishnaT_76_0-1627468705633.png

Please try testing on a different host as I feel that this issue is specific to the host that you are using for testing.

Best Regards,
Jayakrishna
0 Likes

Apology for that I tried the same thing but I dint get  the below image only after switching the workspace and importing the same project I got some some UART logs but I don't see any image.

Snapshot :

Screenshot (36).png

UART log "Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 147 Frm_Sz = 3246480 B

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 148 Frm_Sz = 3246480 B

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 149 Frm_Sz = 3246480 B

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 150 Frm_Sz = 3246480 B

TimeDiff = 504 ms FPS = 59

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 151 Frm_Sz = 3246480 B

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 152 Frm_Sz = 3246480 B

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 153 Frm_Sz = 3246480 B

Prod = 88 Cons = 87 Prtl_Sz = 6672 Frm_Cnt = 154 Frm_Sz = 3246480 B"

Is thare any option to pass the data to uvc ?

Thankyou

 

0 Likes

Hello,

I still suspect that the sensor configuration settings are not proper. As mentioned before, CX3 is just a bridge between image sensor and host. It just forwards the data obtained from the image sensor to the host after adding UVC header. No additional frame data is added by CX3. The prints that you are seeing indicates the total amount of data sent by the image sensor. As mentioned before, the expected frame size is calculated as follows:

Frame size = (1920 x 1080 x 10) / 8 = 2592000 bytes

But, from the debug prints, the frame size received by CX3 is 3246480 bytes which is again greater than 2592000 bytes. Initially, you were receiving 3903348 bytes from the image sensor instead of 2592000 bytes. This was brought down to 3246480 bytes by changing the sensor configuration settings. I strongly suspect that the sensor configuration settings used currently is also not correct for your desired resolution. So, kindly check the configuration settings again with the image sensor vendor.

Best Regards,
Jayakrishna
0 Likes

Hello,

I tried configuring the sensor register, finally I brought up to this in my UART,

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16302 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16303 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16304 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16305 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 18888 Frm_Cnt = 16306 Frm_Sz = 2596008 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16307 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16308 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16512 Frm_Cnt = 16309 Frm_Sz = 2593632 B

Prod = 70 Cons = 69 Prtl_Sz = 16512 Frm_Cnt = 16310 Frm_Sz = 2593632 B

Prod = 70 Cons = 69 Prtl_Sz = 18924 Frm_Cnt = 16311 Frm_Sz = 2596044 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16312 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16313 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16314 Frm_Sz = 2593644 B

Prod = 70 Cons = 69 Prtl_Sz = 16512 Frm_Cnt = 16315 Frm_Sz = 2593632 B

Prod = 70 Cons = 69 Prtl_Sz = 16524 Frm_Cnt = 16316 Frm_Sz = 2593644 B

But this is not continues and also constant I don't know why, Each time I rewrite my image sensor configuration I got  a single frame. 

Sensor configuration flow

>>pause the streaming 

>>writing the registers

>>Start streaming

This done by and external microcontroller and each time I  pause and resume the image senor I t prints the frame size. but its not going continues

the same thing I did to get 1280X720 60fps It started streaming  when it reaches almost the nearest value it started streaming, but the same thing is not working for 1920X1080 60fps.

 

For 1920X1080 30fps the frame size is(1920x1080x16)/8 =  4147200 Due to 6 more bit padded to this.

What should be the exact frame size if it is packed as 24 bit ?

Should I try increasing the frame size ?

Thankyou

0 Likes

Hello,

The frame size when you pack the incoming data into 24 bits will be the same as the actual frame size. When you pack the incoming data, there are no additional padded bits. Please refer to the following KBA again as it explains in detail about how CX3 packs and pads the incoming data:

https://community.cypress.com/t5/Knowledge-Base-Articles/Streaming-RAW10-Format-Input-Data-to-16-24-...

So, the frame size in this case will be:

frame size = (1920x1080x10)/8 =  2592000

For the UVC player to stream the data, the frame size should be equal to this value. I find that the frame size is still greater and is varying. Please check the sensor configuration settings again with the sensor vendor.

Best Regards,
Jayakrishna
0 Likes

Hello, 

Thanks for the support, As per your suggestion I tried the modifying the register configuration. This time instead of changing the frame size, I changed the MIPI CSI clock from 720Mhz to 1000Mhz which solves the problem, but still we dint developed any custom Host to view the proper image. 

Due to packing 10 bits into 24 bits the image is look like this

WIN_20210730_15_32_55_Pro.jpg

Thus the data from cx3 is correct and only thing is, cx3 combines multiple frames in one image.

to get this image I made modifications according to this KBA .

https://community.cypress.com/t5/Knowledge-Base-Articles/Streaming-RAW10-Format-Input-Data-to-16-24-...

Thankyou.

0 Likes

Hello,

Thank you for the confirmation. We are glad to hear that the issue is resolved.

Best Regards,
Jayakrishna
0 Likes