Note: See AN90369 - How to Interface a MIPI CSI-2 Image Sensor With EZ-USB® CX3 for information additional to the contents of this article.
1. How does CX3 pack the input RAW10 format into the 16/24-bit output format?
CX3 has a MIPI CSI-2 receiver to convert the serial MIPI data into parallel format. The parallel data is sampled by the fixed GPIF II block and sent to USB through the DMA Channel.
You can use the CX3 configuration utility to select the input data format and output data format as shown in Figure 1.
Assume that the image sensor is streaming RAW 10 data in 1080p at 30 fps.
- Select RAW 10 as the input data format, 16-bit as output data format, and provide other necessary details for streaming 1080p at 30 fps in Image Sensor Configuration Tab.
- Navigate to the CX3 Receiver Configuration tab to adjust the MIPI CSI-2 receiver block parameters as shown in Figure 2. The data format under MIPI CSI-2 inputs is RAW10 and data format under CX3 MIPI interface configuration is RAW10. This means that MIPI CSI-2 is converting the input serial RAW10 data into parallel 10-bit data.
- However, fixed GPIF II supports configuring 8/16/24-bit data bus. In this example, GPIF II is configured in 16-bit data bus mode; therefore, it samples 16 bits per clock. Out of 16 bits, only 10 bits are actual data and rest 6 bits are additional bits padded for every clock. This increases the pixel size to 1.6 times of actual pixel size. Therefore, the frame size increases to 1.6 times of actual frame size. You should remove additional 6 bits per pixel and display the actual frame size in the host application.
Figure 1. Image Sensor Configuration
Figure 2. CX3 MIPI Receiver Configuration
The MIPI CSI-2 configuration for the above settings is as follows:
/* XYZ_Sensor_RAW10_Resolution0 : */
CyU3PMipicsiCfg_t XYZ_Sensor_RAW10_Resolution0 =
{
CY_U3P_CSI_DF_RAW10, /* CyU3PMipicsiDataFormat_t dataFormat */
2, /* uint8_t numDataLanes */
2, /* uint8_t pllPrd */
89, /* uint16_t pllFbd */
CY_U3P_CSI_PLL_FRS_250_500M, /* CyU3PMipicsiPllClkFrs_t pllFrs */
CY_U3P_CSI_PLL_CLK_DIV_4, /* CyU3PMipicsiPllClkDiv_t csiRxClkDiv */
CY_U3P_CSI_PLL_CLK_DIV_4, /* CyU3PMipicsiPllClkDiv_t parClkDiv */
0, /* uint16_t mClkCtl */
CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t mClkRefDiv */
1920, /* uint16_t hResolution */
50 /* uint16_t fifoDelay */
};
These additional 6 bits per pixel can be removed by replacing CY_U3P_CSI_DF_RAW10 with CY_U3P_CSI_DF_YUV422_8_2 in the above MIPI configuration or selecting 24-bit as output data format in the image sensor configuration.
Note that the output data format kept in the MIPI CSI configuration decides the padding of zeros and will not affect the streaming video data format.
These two cases are discussed as follows:
A. Replacing CY_U3P_CSI_DF_RAW10 with CY_U3P_CSI_DF_YUV422_8_2
Setting CY_U3P_CSI_DF_YUV422_8_2 as the output data format enables the packing of incoming RAW10 data into 16-bit parallel data and sending over the MIPI CSI2 parallel interface. There are no additional padded bits in this case.
Note that in this case, only the output data format in the cyu3mipicsi.c is being modified; clock settings in CX3 Receiver Configuration tab are not changed. Therefore, this configuration may not work in cases which need modification in clock settings. Please contact Infineon Technical Support if you come across this issue.
The packing of RAW 10 input data format into 16-bit will be done as follows:
a. MIPI CSI2 receives the data in standard RAW 10 format as shown in the Figure 3. (Source: MIPI Alliance Specification for Camera Serial Interface 2 (CSI-2))
b. The input serial RAW 10 data will be converted to parallel 16-bit data as follows; this will be sampled by GPIF II.
i.e. For every clock cycle, GPIF II receives 16-bit data:
- First clock cycle: P1[9:2] P2[9:2]
- Second clock cycle: P3[9:2], P4[9:2]
- Third clock cycle: P1[1:0], P2[1:0], P3[1:0], P4[1:0]; P5[9:2]
- Fourth cycle – and so on
Figure 3. RAW 10 Transmission (Source: MIPI Alliance Specification for Camera Serial Interface 2)
Note that GPIF II should be configured in 16-bit data bus format when selecting 16-bit as output data format. The packing of RAW 10 into 16-bit format has the constraint that the line size of the frame should be a multiple of 16 bits.
B. Selecting 24-bit as output data format
Setting 24-bit as the output data format in Image Sensor Configuration Tab enables the packing of incoming RAW10 data into 24-bit parallel data and sending it over the MIPI CSI2 parallel interface. There are no additional padded bits in this case.
The input serial RAW 10 data will be converted to parallel 24-bit data as follows and this will be sampled by GPIF II.
i.e. For every clock cycle, GPIF II receives 24-bit data
- First clock cycle - P1[9:2], P2[9:2], P3[9:2],
- Second clock cycle - P4[9:2], P1[1:0], P2[1:0], P3[1:0], P4[1:0], P5[9:2]
- Third clock cycle – and so on.
Note that GPIF II should be configured in 24-bit data bus format when selecting 24-bit as output data format. The packing of RAW 10 into 24-bit format has the constraint that the line size of the frame should be a multiple of 24 bits.
2. How to set USB Descriptors and Probe Control structures to stream in RAW 10 format?
The standard UVC driver from Microsoft streams video from UVC devices with the GUID values provided by Microsoft. The standard driver cannot stream the RAW video format; there is no GUID provided by Microsoft for RAW video format. Therefore, the standard UVC driver does not stream the RAW format video.
However, you can use the following alternative method to stream the RAW format video with significant limitations in playback—you will need to customize the host application for a fully functional video playback.
1. Use the GUID defined by Microsoft (for example, GUID of YUY2 format) in Uncompressed Video Streaming (VS) format descriptor.
2. Set USB descriptors (Uncompressed VS format descriptor and Uncompressed VS Frame Descriptor) and probe control structures based on the input and output data format of MIPI bridge of CX3.[BJ1]
With these settings, the UVC driver streams the UVC data; the host application decodes the data according to the GUID value that you assigned in Step 1. However, you cannot see the exact video in the host application; instead you will see some colors[BJ2] because the incoming video is in RAW format and the host application decodes it as standard format (YUY2). Even though the playback functionality is significantly reduced, you can use this approach for debugging a UVC device.
The following examples of UVC descriptors and probe control structures for streaming a RAW 10 video format with three variants of MIPI Output data format are provided below. Make the required changes in the UVC Descriptor and probe control structures of your firmware as demonstrated in this example. You will need to customize the host application to obtain a fully functional video playback.
Example 1: RAW 10 output data format (CY_U3P_CSI_DF_RAW10)
Assume that you are streaming 1920x1080 resolution at 30 fps in RAW 10 format with the following values:
a. Width in pixel : 1920
b. Height in pixel : 1080
c. Bits per pixel : 10
d. Frames per second : 30
e. Frame size : 1920 x 1080 x 10 bits
f. Bit rate : 1920 x 1080 x 10 x 30 bits per second
In this setting, the GPIF II interface should be set to 16-bit parallel data interface for RAW 10 MIPI output format, and so has padding of 6 bits for each pixel, causing the frame size to increase. The above calculations will be changed as follows:
a. Width in pixel : 1920
b. Height in pixel : 1080
c. Bits per pixel :16 (due to padded 6 bits per pixel)
d. Frames per second : 30
e. Frame size : 1920 x 1080 x 16 bits
f. Bit rate : 1920 x 1080 x 16 x 30 bits per second
USB descriptors and probe control structures are as follows:
/* Class specific Uncompressed VS format descriptor */
0x1B, /* Descriptor size */
CX3_CS_INTRFC_DESCR, /* Class-specific VS interface Type */
0x04, /* Subtype : VS_FORMAT_UNCOMPRESSED */
0x01, /* Format desciptor index */
0x04, /* Number of Frame Descriptors that follow this descriptor: 3 */
/* GUID, globally unique identifier used to identify streaming-encoding format: YUY2 */
0x59, 0x55, 0x59, 0x32, /*MEDIASUBTYPE_YUY2 GUID: 32595559-0000-0010-8000-00AA00389B71 */
0x00, 0x00, 0x10, 0x00,
0x80, 0x00, 0x00, 0xAA,
0x00, 0x38, 0x9B, 0x71,
0x10, /* Number of bits per pixel: 16*/
0x01, /* Optimum Frame Index for this stream: 2 (1080p) */
0x00, /* X dimension of the picture aspect ratio; Non-interlaced */
0x00, /* Y dimension of the pictuer aspect ratio: Non-interlaced */
0x00, /* Interlace Flags: Progressive scanning, no interlace */
0x00, /* duplication of the video stream restriction: 0 - no restriction */
/* Class specific Uncompressed VS Frame Descriptor - 1080p@30fps */
0x1E, /* Descriptor size */
CX3_CS_INTRFC_DESCR, /* Descriptor type*/
0x05, /* Subtype: Uncompressed frame interface*/
0x01, /* Frame Descriptor Index: 1 */
0x00, /* No Still image capture supported */
0x80, 0x07, /* Width in pixel: 1920 */
0x38, 0x04, /* Height in pixel: 1080 */
0x00, 0x80, 0x53, 0x3B, /* Min bit rate (bits/s): 1080 x 1920 x 16 x 30 = 995328000 */
0x00, 0x80, 0x53, 0x3B, /* Max bit rate (bits/s): Fixed rate so same as Min */
0x00, 0x48, 0x3F, 0x00, /* Maximum video or still frame size in bytes(Deprecated: 1920x1080 x2*/
0x15, 0x16, 0x05, 0x00, /* Default frame interval (in 100ns units): (1/30)x10^7 */
0x01, /* Frame interval type : No of discrete intervals */
0x15, 0x16, 0x05, 0x00, /* Frame interval 3: Same as Default frame interval */
/* UVC Probe Control Setting - 1080p@30FPS */
uint8_t const gl1080pProbeCtrl[CX3_APP_MAX_PROBE_SETTING] = {
0x00, 0x00, /* bmHint : No fixed parameters */
0x01, /* Use 1st Video format index */
0x01, /* Use 1st Video frame index */
0x15, 0x16, 0x05, 0x00, /* Desired frame interval in 100ns = (1/30)x10^7 */
0x00, 0x00, /* Key frame rate in key frame/video frame units */
0x00, 0x00, /* PFrame rate in PFrame / key frame units */
0x00, 0x00, /* Compression quality control */
0x00, 0x00, /* Window size for average bit rate */
0x00, 0x00, /* Internal video streaming i/f latency in ms */
0x00, 0x48, 0x3F, 0x00, /* Max video frame size in bytes = 1920 x 1080 x 2 */
#ifdef CX3_UVC_1_0_SUPPORT
0x00, 0x90, 0x00, 0x00 /* No. of bytes device can rx in single payload: 36KB */
#else
/* UVC 1.1 Probe Control has additional fields from UVC 1.0 */
0x00, 0x90, 0x00, 0x00, /* No. of bytes device can rx in single payload: 36KB */
0x00, 0x60, 0xE3, 0x16, /* Device Clock */
0x00, /* Framing Information - Ignored for uncompressed format*/
0x00, /* Preferred payload format version */
0x00, /* Minimum payload format version */
0x00 /* Maximum payload format version */
#endif
};
Note that the host application needs to decode the received RAW 10 data by removing the padded 6 bits for each pixel to get the correct frame size and display the RAW10 video.
Example 2: 16-bit output data format (CY_U3P_CSI_DF_YUV422_8_2)
Set CY_U3P_CSI_DF_YUV422_8_2 as the output format in cyu3mipicsi.c by replacing CY_U3P_CSI_DF_RAW10, to remove padded zeros.
Assume that you are streaming 1920 x 1080 resolution at 30 fps in RAW 10 format. The calculations of width in pixel, height in pixel, bits per pixel, frames per second, frame size and bit rate are as follows:
a. Width in pixel : 1920
b. Height in pixel : 1080
c. Bits per pixel : 10
d. Frames per second : 30
e. Frame size : 1920 x 1080 x 10 bits
f. Bit rate : 1920 x 1080 x 10 x 30 bits per second
Re-writing the above calculations:
a. Width in pixel : 2400
b. Height in pixel : 540
c. Bits per pixel : 16
d. Frames per second : 30
e. Frame size : 1200 x 1080 x 16 bits
f. Bit rate : 1200 x 1080 x 16 x 30 bits per second
Make the following changes in the UVC Descriptor and probe control structures of your firmware.
/* Class specific Uncompressed VS format descriptor */
0x1B, /* Descriptor size */
CX3_CS_INTRFC_DESCR, /* Class-specific VS interface Type */
0x04, /* Subtype : VS_FORMAT_UNCOMPRESSED */
0x01, /* Format desciptor index */
0x04, /* Number of Frame Descriptors that follow this descriptor: 3 */
/* GUID, globally unique identifier used to identify streaming-encoding format: YUY2 */
0x59, 0x55, 0x59, 0x32, /*MEDIASUBTYPE_YUY2 GUID: 32595559-0000-0010-8000-00AA00389B71 */
0x00, 0x00, 0x10, 0x00,
0x80, 0x00, 0x00, 0xAA,
0x00, 0x38, 0x9B, 0x71,
0x10, /* Number of bits per pixel: 16*/
0x01, /* Optimum Frame Index for this stream: 1 (1080p) */
0x00, /* X dimension of the picture aspect ratio; Non-interlaced */
0x00, /* Y dimension of the pictuer aspect ratio: Non-interlaced */
0x00, /* Interlace Flags: Progressive scanning, no interlace */
0x00, /* duplication of the video stream restriction: 0 - no restriction */
/* Class specific Uncompressed VS Frame Descriptor - 1080p@30fps */
0x1E, /*Descriptor size */
CX3_CS_INTRFC_DESCR, /* Descriptor type*/
0x05, /* Subtype: Uncompressed frame interface*/
0x01, /* Frame Descriptor Index: 1 */
0x00, /* No Still image capture supported */
0xB0, 0x04, /* Width in pixel: 1200 */
0x38, 0x04, /* Height in pixel: 1080 */
0x00, 0x30, 0x14, 0x25, /* Min bit rate (bits/s): 1080 x 1200 x 16 x 30 = 622080000*/
0x00, 0x80, 0x53, 0x3B, /* Max bit rate (bits/s): Fixed rate so same as Min */
0x00, 0x8D, 0x27, 0x00, /* Maximum video or still frame size in bytes(Deprecated: 1200x1080 x2*/
0x15, 0x16, 0x05, 0x00, /* Default frame interval (in 100ns units): (1/30)x10^7 */
0x01, /* Frame interval type : No of discrete intervals */
0x15, 0x16, 0x05, 0x00, /* Frame interval 3: Same as Default frame interval */
/* UVC Probe Control Setting - 1080p@30FPS */
uint8_t const gl1080pProbeCtrl[CX3_APP_MAX_PROBE_SETTING] = {
0x00, 0x00, /* bmHint : No fixed parameters */
0x01, /* Use 1st Video format index */
0x01, /* Use 1st Video frame index */
0x15, 0x16, 0x05, 0x00, /* Desired frame interval in 100ns = (1/30)x10^7 */
0x00, 0x00, /* Key frame rate in key frame/video frame units */
0x00, 0x00, /* PFrame rate in PFrame / key frame units */
0x00, 0x00, /* Compression quality control */
0x00, 0x00, /* Window size for average bit rate */
0x00, 0x00, /* Internal video streaming i/f latency in ms */
0x00, 0x8D, 0x27, 0x00, /* Max video frame size in bytes = 1200 x 1080 x 2 */
#ifdef CX3_UVC_1_0_SUPPORT
0x00, 0x90, 0x00, 0x00 /* No. of bytes device can rx in single payload: 36KB */
#else
/* UVC 1.1 Probe Control has additional fields from UVC 1.0 */
0x00, 0x90, 0x00, 0x00, /* No. of bytes device can rx in single payload: 36KB */
0x00, 0x60, 0xE3, 0x16, /* Device Clock */
0x00, /* Framing Information - Ignored for uncompressed format*/
0x00, /* Preferred payload format version */
0x00, /* Minimum payload format version */
0x00 /* Maximum payload format version */
#endif
};
Example 3: 24-bit output data format (CY_U3P_CSI_DF_RGB888)
Set 24-bit as the output data format in Image Sensor Configuration Tab to enable packing of incoming RAW10 data into 24-bit parallel data and sending it over the MIPI CSI2 parallel interface. There are no additional padded bits in this case.
There is no difference in the descriptors and probe control structures in packing of RAW 10 data into 16 bit or 24-bit parallel output. Therefore, you can use the descriptors and probe control structures in Example 2 in this case as well.
Author: KandlaguntaR_36 Version: **
Translation - Japanese: RAW10形式の入力データをCX24 MIPI CSI-2の16/24ビット出力形式にストリーミングする - KBA224387 - Community Translated (JA)