Streaming RAW8 or RAW10 using CX3

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
user_2884081
Level 4
Level 4
50 replies posted 25 replies posted 10 replies posted

I have a Denebola kit from e-Con Systems. It was initially configured to send images in a YUY2 format.

I managed to modify the firmware to send image in RGB565 format. For this purpose, I changed the following parameters:

  • Camera format registers
  • Media Type identifier (GUID)
  • MIPI CSI-2 format (in 'CyU3PMipicsiCfg_t')

Since the bits-per-pixel of both formats is the same (16), I can see the video. Even when the camera format is set to YUY2, I can see the image in the PC, but color distorted.

The problem is, I want to stream in RAW8 or RAW10, which have a different bpp. I have changed the same parameters as before, but I can't get it running.

I have read this and this thread, but I'm still missing something. I also posted a case: case#00435749.

Should I change the previous parameters? What values should I use?

Other parameters I think that should be changed, based on what I have read here, are the 'hResolution' in 'CyU3PMipicsiCfg_t', used to calculate BYTE_COUNT; or the MIPI CSI-2 to GPIF bus used in 'CyU3PMipicsiGpifLoad' (now it is set to 16).

Should I change this parameters? What values should I use?

What format, media type identifier and bits-per-pixel should I use in the USB descriptor?

0 Likes
1 Solution
KandlaguntaR_36
Moderator
Moderator
Moderator
25 solutions authored 10 solutions authored 5 solutions authored

Edited:

1. What is the host application that you are using here?

Is it able to receive the RAW 8 or RAW 10 and able to stream the video?

2. The UVC descriptors part will depends on the configuration of MIPI Interface.

Case A: Let us say, you are using the RAW 10 input and RAW10 output format in the MIPI and you have set the GPIF Bus to 16 bit data bus using MipicsiGpifLoad API.

In this case, the GPIF will sample the 16 lines and store in the buffer as 16 bits per clock, but the MIPI block puts only 10 bits per clock, the rest 6 bit are sampled as zero or floating values. Hence, out 16 bits, only 10 bits belong to a single pixel, rest are in-valid data w.r.t image.

If you set the UVC descriptors as 16-bit pixel (instead of 10 bits per pixel) and set the frame size by caluclatuing 16 bits per pixel, and put YUY2 format; the standard UVC host applications will receive the data, and stream the video. But the video will be distorted because it is actually RAW 10 image is sending in 16 bit format.

In order to stream the video, you need to modify the host application in such away, it removes the additional bits per pixel and display image (video). No need to change the descriptors here.

Case B: if you set the RAW 10 as input and CY_U3P_CSI_DF_YUV422_8_1 as output format (pass this in Mipicsiintfparams becasue the tool does not this option) as said in this Re: CX3 configuration for OV5680 RAW10 format , the mipi will pack input RAW 10 into 16 bit format.

i.e This 16-bit format will have 10 bits of first pixel and rest 6 bits of second pixel.

This is explained below:

The MIPI bridge will receive RAW 10 in the following format as per the MIPI CSI-2 Spec.

RAW10_Format.PNG

The CX3 MIPI will convert the FIRST 16 serial bits into parallel 16 bits. It does not bother about the data sequence.

So, the GPIF will receive these first 16-bit in the RAW 10 format.

i.e First clock cycle - P1[9:2] P2[9:2]

Second clock cycle - P3[9:2], P4[9:2];

Third clock cycle - P1[1:0], P2[1:0], P3[1:0], P4[1:0]; P5[9:2]

Fourth cycle - continue

In this case, you have to modify the descriptors with the consideration of 10 bits per pixel and also calculate the frame format descriptors with 10 Bpp.

Note that you also need to modify the Probe Control Structures with the consideration of 10 bits per pixel.

Here, the host application should have capability to  receive the data (RAW 10) and able to display the same and since this is not as the same as RGB565 and YUY2 formats (16 bits per pixel).

View solution in original post

0 Likes
17 Replies
KandlaguntaR_36
Moderator
Moderator
Moderator
25 solutions authored 10 solutions authored 5 solutions authored

Edited:

1. What is the host application that you are using here?

Is it able to receive the RAW 8 or RAW 10 and able to stream the video?

2. The UVC descriptors part will depends on the configuration of MIPI Interface.

Case A: Let us say, you are using the RAW 10 input and RAW10 output format in the MIPI and you have set the GPIF Bus to 16 bit data bus using MipicsiGpifLoad API.

In this case, the GPIF will sample the 16 lines and store in the buffer as 16 bits per clock, but the MIPI block puts only 10 bits per clock, the rest 6 bit are sampled as zero or floating values. Hence, out 16 bits, only 10 bits belong to a single pixel, rest are in-valid data w.r.t image.

If you set the UVC descriptors as 16-bit pixel (instead of 10 bits per pixel) and set the frame size by caluclatuing 16 bits per pixel, and put YUY2 format; the standard UVC host applications will receive the data, and stream the video. But the video will be distorted because it is actually RAW 10 image is sending in 16 bit format.

In order to stream the video, you need to modify the host application in such away, it removes the additional bits per pixel and display image (video). No need to change the descriptors here.

Case B: if you set the RAW 10 as input and CY_U3P_CSI_DF_YUV422_8_1 as output format (pass this in Mipicsiintfparams becasue the tool does not this option) as said in this Re: CX3 configuration for OV5680 RAW10 format , the mipi will pack input RAW 10 into 16 bit format.

i.e This 16-bit format will have 10 bits of first pixel and rest 6 bits of second pixel.

This is explained below:

The MIPI bridge will receive RAW 10 in the following format as per the MIPI CSI-2 Spec.

RAW10_Format.PNG

The CX3 MIPI will convert the FIRST 16 serial bits into parallel 16 bits. It does not bother about the data sequence.

So, the GPIF will receive these first 16-bit in the RAW 10 format.

i.e First clock cycle - P1[9:2] P2[9:2]

Second clock cycle - P3[9:2], P4[9:2];

Third clock cycle - P1[1:0], P2[1:0], P3[1:0], P4[1:0]; P5[9:2]

Fourth cycle - continue

In this case, you have to modify the descriptors with the consideration of 10 bits per pixel and also calculate the frame format descriptors with 10 Bpp.

Note that you also need to modify the Probe Control Structures with the consideration of 10 bits per pixel.

Here, the host application should have capability to  receive the data (RAW 10) and able to display the same and since this is not as the same as RGB565 and YUY2 formats (16 bits per pixel).

0 Likes

The host application I'm using right now is e-CAMView from e-Con Systems, which I think does not support RAW format. But if I can get it to send data like it is YUV or RGB, I can then remake the picture (with MATLAB for example).

I will try your solution and come back to you.

Thanks

0 Likes

First question:

In Case A, why would the host application not be able to stream video? Wouldn't the 6 extra zero bits treated as part of the YUY2 format?

I know the image wouldn't appear as such, but I would expect a color distorted image at least.

0 Likes

I have tried the following:

  • Camera set to RAW8
  • MIPI CSI set to RAW8
  • GPIF bus set to 8 bits
  • USB descriptor: RGB565, 16bpp, height:1080, width:1920

MIPI CSI configuration:

/* Configuration parameters for 1080p @30FPS for the OV5640 sensor */

CyU3PMipicsiCfg_t cfgUvc1080p30NoMclk =  {

CY_U3P_CSI_DF_RAW8,      /* dataFormat   */

2,                          /* numDataLanes */

1,                        /* pllPrd       */

62,                          /* pllFbd       */

CY_U3P_CSI_PLL_FRS_500_1000M, /* pllFrs      */

CY_U3P_CSI_PLL_CLK_DIV_8,    /* csiRxClkDiv  */

CY_U3P_CSI_PLL_CLK_DIV_8,    /* parClkDiv    */

0x00,                        /* mclkCtl      */

CY_U3P_CSI_PLL_CLK_DIV_8,    /* mClkRefDiv   */

1920,                        /* hResolution  */

0x01                        /* fifoDelay    */

};

USB descriptor:

/* GUID, globally unique identifier used to identify streaming-encoding format: RGB565  */

0x7B, 0xEB, 0x36, 0xE4,             /*MEDIASUBTYPE_RGB565 e436eb7b-524f-11ce-9f53-0020af0ba770  */

0x4F, 0x52, 0xCE, 0x11,

0x9F, 0x53, 0x00, 0x20,

0xAF, 0x0B, 0xA7, 0x70,

    0x10,                               /* Number of bits per pixel: 16*/

    0x01,                               /* Optimum Frame Index for this stream: 1 (1080p) */

    0x00,                               /* X dimension of the picture aspect ratio; Non-interlaced */

    0x00,                               /* Y dimension of the pictuer aspect ratio: Non-interlaced */

    0x00,                               /* Interlace Flags: Progressive scanning, no interlace */

    0x00,                               /* duplication of the video stream restriction: 0 - no restriction */

    /* Class specific Uncompressed VS Frame Descriptor 1 - 1080p@30fps */

    0x1E,                               /* Descriptor size */

    ES_UVC_CS_INTRFC_DESCR,                /* Descriptor type*/

    0x05,                               /* Subtype: Uncompressed frame interface*/

    0x01,                               /* Frame Descriptor Index: 1 */

    0x00,                               /* No Still image capture method supported */

    0xC0, 0x03,                         /* Width in pixel:  1920 */

    0x38, 0x04,                         /* Height in pixel: 1080 */

    0x00, 0xC0, 0xA9, 0x1D,             /* Min bit rate (bits/s): 1080 x 1920 x 2 x 30 x 8 = 995328000 */

    0x00, 0xC0, 0xA9, 0x1D,             /* Max bit rate (bits/s): Fixed rate so same as Min */

    0x00, 0xEC, 0x5E, 0x00,             /* Maximum video or still frame size in bytes(Deprecated): 1920 x 1080 x 3 */

    0x15, 0x16, 0x05, 0x00,             /* Default frame interval (in 100ns units): (1/30)x10^7 */

    0x01,                               /* Frame interval type : No of discrete intervals */

    0x15, 0x16, 0x05, 0x00,             /* Frame interval 3: Same as Default frame interval */

The host application shows a black screen at 0 FPS. What am I doing wrong?

0 Likes

We have to confirm whether there is data flow from MIPI to USB side first.

Enable the Debug Prints and see whether the buffers are getting the data.

Also probe the H_SYNC_ and V_SYNC_Test pins and share the values of HSYNC and VSYNC.

Regards,

Sridhar

0 Likes

I assume they are working properly since RGB565 and YUV422 modes are working just fine

0 Likes

Also I have probed the HSYNC and VSYNC lanes and they seem to be sending 1080 rows at 30 FPS.

0 Likes

Then MIPI configuratio is fine.

Check whether you are receiving the the producer and consumer events. Put a counter in Producer and conusmer events and also frame size. Then print them in the main thread.

This confirms the data flow ftom MIPi to USB side.

0 Likes

I probed it wrong: with the RAW8 configuration, the sensor gives a 2592x1944 resolution, and I thought it was 1920x1080. I can see around 2590 rows when probing it (can't know the exact number).

Given this information, I have tried setting the MIPI for a RAW8, hResolution=2592 using the 5Mp configuration:

/* Configuration parameters for 5Mp @15FPS for the OV5640 sensor */

CyU3PMipicsiCfg_t cfgUvc5Mp15NoMclk =  {

CY_U3P_CSI_DF_RAW8,      /* dataFormat   */

    2,                          /* numDataLanes */

    1,                        /* pllPrd       */

    64,                          /* pllFbd       */

    CY_U3P_CSI_PLL_FRS_500_1000M, /* pllFrs       */

    CY_U3P_CSI_PLL_CLK_DIV_8,    /* csiRxClkDiv  */

    CY_U3P_CSI_PLL_CLK_DIV_8,    /* parClkDiv    */

    0x00,                        /* mclkCtl      */

    CY_U3P_CSI_PLL_CLK_DIV_2,    /* mClkRefDiv   */

    2592,                        /* hResolution  */

    0x01                        /* fifoDelay    */

};

In the USB descriptor, the format is still RGB565, and 16bpp. I'm using 1296x1944 (width x height) to compensate the format being 8bpp.

Still, the host application shows a black image at 0 FPS. Any idea what could be wrong?

0 Likes

I have added a little code to send a pulse whenever a producer or consumer event took place, and I can confirm both of them are happening. However, they are not as regular as when I use the RGB565 configuration (the one that works correctly).

0 Likes

Have you changed the probe control settings as per 1296x1944 resolution?

Can you check what is the frame size that you are receiving? this can be checked by adding the number of full buffers of bytes received  with the partial buffer size received in the each producer event.

0 Likes

Corrected the Case A after the comment.

0 Likes
user_2884081
Level 4
Level 4
50 replies posted 25 replies posted 10 replies posted

I had not changed the probe control settings, now I have and still not working.

With the working RGB565 configuration I have probed the number of full buffers and it is similar to the total number of packs to send. 1080x1920x2 = 4147200, DMA size is 24576, so 169 packs must be sent.

With the RAW8 configuration, I can only see half the full buffers I expect. The total size of a frame in bytes is 2592x1944, so 206 packs should be sent, while only around 100 are full.

Edit: These are the modifications so far:

  • Cam set to RAW8, and as seen in MIPI lanes, sending 2592 rows.
  • MIPI configured for RAW8, hResolution=2592
  • GPIF bus set to 16bits
  • Probe control settings: 2592x1944x1 Bytes
  • USB descriptor: 16bpp, RGB565, frame size = 1296x1944; bit rate = 2592x1944x15x8

Should the GPIF bus be 8 bits since I am using RAW 8?

0 Likes

Check whether you configured the Fixed GPIF with 8-bit data bus width through this API: CyU3PMipicsiGpifLoad(8,24560).

For 2592x1944, RAW 8, the frame size is 5038848 bytes. Since the DMA size is 24576 bytes, if we remove the header and footer, the effective DMA Buffer size is 24560 bytes (24576 -16). So you need 205 full buffer of 24560 bytes and one parial buffer of 4048 bytes per frame.

If you are not receiving the same, probe the HSYNC, VSYNC and PCLK test pins check whether these values are matching the 2592x1944 frame size.

0 Likes

Now I have managed to get it working. First of all, thank you for your help, it has been really important.

For the record, and for people with the same problem, I will update the configuration. It is the same as before, but I had the sensor configured for 30FPS and the CX3 was expecting 15FPS.

  • Camera settings: 2592x1944, RAW8, 15FPS
  • MIPI Configuration: RAW8, hResolution = 2592
  • GPIF Bus: 8 bits
  • Probe control settings: max bytes per frame = 2592x1944
  • USB descriptor: RGB565, 16bpp, frame size = 2592x1944, bit rate = 2592x1944x15x8
0 Likes

Now I would like to get it to work at higher FPS. My first guess is trying binning the image, do you have any recommendations?

Also, do you know if I can read RAW8 images with MATLAB?

Thanks

0 Likes

I do not have any recommendations for the both of your requirements at the moment.

Please create a new thread for the newer queries instead of adding different queries in single forum thread as a good practice.

0 Likes