8 Replies Latest reply on Jan 24, 2019 10:42 PM by KandlaguntaR_36

    CX3 OV5647 RAW10 data display

    ScGr_289066

      Hi All,

       

      I'm having a couple of issues this is the second thread to discuss them.

       

      In this thread I will discuss displaying RAW10 data from an OV5647 sensor in 10-bit mode.

       

      We have successfully configured the CX3 to transport RAW10 data from an OV5647 sensor so it is viewable by e-camView and OpenCV applications.  As several posts on this forum suggested, we did this by tricking the CX3 into believing it is receiving YUY2 data from the 5647 sensor, which can only produce RAW10 data, and sending it to the PC applications.  As noted in the posts, the data isn't displayed correctly because the PC also believes it is in YUY2 format, so the images are generally without much detail and green.

       

      Today I created an OpenCV application that reads image data and does a simple memory dump so I could make sense of its format to figure out how to properly convert it to a known, preferably RGB, format.  The results so far have been confusing.  First let me say that I have configured the sensor to generate its bar test pattern (8 vertical bars across the width of the screen or varying colors.

       

      When I examine the data collected by OpenCV, I don't see a stream of 16-bit values but instead I see repeating groups of 3-bytes (24-bit) values.  Each 1920x1080 (1080p) frame contains 3-bytes per pixel (6,220,800 bytes/frame).  Since the test pattern is saturated bars and the middle byte of each group is 255, I figured the data was coming across in UYV or VYU format, so I tried the standard conversion B=1.164*(y-16)+2.018(u-128); G=1.164*(y-16)-0.813*(v-128)-0.391*(u-128) and R=1.164*(y-16)+1.596*(v-128).  However this conversion doesn't give me the colors that match the sensor's test pattern (or even close).

       

      As I understand the 5647's RAW10 output, I am expecting 16-bits of Bayer patterned pixels:

       

      B G B G B G...

      G R G R G R...

      B G B G B G...

      ...

      G R G R G R ...

       

      Why is the input data in 24-bit format rather than 16-bit as I have specified in the CX3 configuration?  Is the PC driver further jumbling the data it receives from the CX3 or is something else happening?

       

       

      Thanks,

      Scott

        • 1. Re: CX3 OV5647 RAW10 data display
          KandlaguntaR_36

          Please let me know what is the MIPI Output data format you have set. Please check whether you have received correct frame size. Please go through the following and it may help you.

           

          1. Let us assume, you are using the RAW 10 input and RAW10 output format in the MIPI and you have set the GPIF Bus to 16 bit data bus using MipicsiGpifLoad API.

           

          In this case, the GPIF will sample the 16 lines and store in the buffer as 16 bits per clock, but the MIPI block puts only 10 bits per clock, the rest 6 bit are sampled as zero or floating values. Hence, out 16 bits, only 10 bits belong to a single pixel, rest are in-valid data w.r.t image.

           

          If you set the UVC descriptors as 16-bit pixel (instead of 10 bits per pixel) and set the frame size by caluclatuing 16 bits per pixel, and put YUY2 format; the standard UVC host applications will receive the data, and stream the video. But the video will be distorted because it is actually RAW 10 image is sending in 16 bit format.

           

          In order to stream the video, you need to modify the host application in such away, it removes the additional bits per pixel and display image (video). No need to change the descriptors here.

           

          2. If you set the RAW 10 as input and CY_U3P_CSI_DF_YUV422_8_1 as output format (pass this in Mipicsiintfparams becasue the tool does not have this option) as said in this Re: CX3 configuration for OV5680 RAW10 format , the mipi will pack input RAW 10 into 16 bit format.

           

          This is explained below:

           

          The MIPI bridge will receive RAW 10 in the following format as per the MIPI CSI-2 Spec.

           

          The CX3 MIPI will convert the FIRST 16 serial bits into parallel 16 bits. It does not bother about the data sequence.

           

          So, the GPIF will receive this first 16-bit in the RAW 10 format.

           

          i.e First clock cycle - P1[9:2] P2[9:2]

          Second clock cycle - P3[9:2], P4[9:2];

          Third clock cycle - P1[1:0], P2[1:0], P3[1:0], P4[1:0]; P5[9:2]

          Fourth cycle - continue

           

          3. If you set the RAW 10 as input and CY_U3P_CSI_DF_RGB888 as output format, the MIPI will pack input RAW 10 into 24-bit parallel format.

           

          This is explained below:

           

          The MIPI bridge will receive RAW 10 in the following format as per the MIPI CSI-2 Spec.

          The CX3 MIPI will convert the FIRST 24 serial bits into parallel 24 bits. It does not bother about the data sequence.

           

          So, the GPIF will receive this first 24-bit in the RAW 10 format then convert into 24-bit parallel bits as follows:

           

          i.e First clock cycle - P1[9:2] P2[9:2] P3[9:2]

          Second clock cycle -  P4[9:2] P1[1:0], P2[1:0], P3[1:0], P4[1:0]; P5[9:2]

          Third clock cycle - continue

           

           

           

           

           

          In 2 &3, you have to modify the descriptors with the consideration of 10 bits per pixel and also calculate the frame format descriptors with 10 Bpp.

          Note that you also need to modify the Probe Control Structures with the consideration of 10 bits per pixel.

          Here, the host application should have the capability to receive the data (RAW 10) and able to display the same and since this is not as the same as RGB565 and YUY2 formats (16 bits per pixel).

          • 2. Re: CX3 OV5647 RAW10 data display
            ScGr_289066

            Hi srdr,

             

            Your assumptions in section #1 of your reply are all correct: I am configured for RAW 10 input and RAW10 output for MIPI and have set the GPIF Bus to 16 bit (data below).  All the UVC descriptors are calculated using 16-bits per pixel.  OpenCV receives what appears to be data in multiples of 3-bytes (24-bits) with the OV5647 running a vertical bar test pattern (8 different color vertical bars each 240 pixels wide across the image).

             

            One confusing part is OpenCV reports there are 6,220,800 bytes for a 1920x1080 frame (that's 3-bytes/pixel), not the 1920x1080x2=4,147,200 bytes that I expect.  OpenCV does report the correct image dimensions.

             

            Per section 1, if I assume the data format is P1[9:2],P2[9:2],P3[9:2],P4[9:2],P1[1:0],P2[1:0],P3[1:0],P4[1:0] for the first 5 bytes in the image, corresponding to the first 4 pixels in line 1 which according to sensor data sheets are: Blue,Green,Blue,Green, and I try rendering the image this way, the result is trash.

             

            I'm only running in 1080p resolution in USB3 (SS mode), so I've omitted the non-revilement tables:

             

            /* OV5647_RAW10_1080p :  */

            CyU3PMipicsiCfg_t OV5647_RAW10_1080p = 

            {

                CY_U3P_CSI_DF_RAW10,  /* CyU3PMipicsiDataFormat_t dataFormat */

                2,                          /* uint8_t numDataLanes */

                1,                /* uint8_t pllPrd */

                62,            /* uint16_t pllFbd */

                CY_U3P_CSI_PLL_FRS_500_1000M, /* CyU3PMipicsiPllClkFrs_t pllFrs */ 

                CY_U3P_CSI_PLL_CLK_DIV_8,    /* CyU3PMipicsiPllClkDiv_t csiRxClkDiv */

                CY_U3P_CSI_PLL_CLK_DIV_8,    /* CyU3PMipicsiPllClkDiv_t parClkDiv */

                0,                        /* uint16_t mClkCtl */

                CY_U3P_CSI_PLL_CLK_DIV_2,    /* CyU3PMipicsiPllClkDiv_t mClkRefDiv */

                1920,                /* uint16_t hResolution */

                200                            /* uint16_t fifoDelay */

            };

             

            Here are select portions of CyCx3USBSSConfigDscr[]:

             

            /* Class specific Uncompressed VS format descriptor */
            0x1B,                           /* Descriptor size */
            CX3_CS_INTRFC_DESCR,            /* Class-specific VS interface Type */
            0x04,                           /* Subtype : VS_FORMAT */
            0x01,                           /* Format desciptor index */
            0x04,                           /* Number of Frame Descriptors that follow this descriptor: 4 */

               

            /* GUID, globally unique identifier used to identify streaming-encoding format*/
            //    YUY2: 32595559-0000-0010-8000-00AA00389B71

             

            0x59, 0x55, 0x59, 0x32,
            0,   0, 0x10, 0x00,
            0x80, 0x00, 0x00, 0xaa,
            0x00, 0x38, 0x9b, 0x71,

             

            0x10,                           /* Number of bits per pixel: 16*/
            0x01,                           /* Optimum Frame Index for this stream: 1 (5M) */
            0x00,                           /* X dimension of the picture aspect ratio; Non-interlaced */
            0x00,                           /* Y dimension of the pictuer aspect ratio: Non-interlaced */
            0x00,                           /* Interlace Flags: Progressive scanning, no interlace */
            0x00,                           /* duplication of the video stream restriction: 0 - no restriction */

             

            ...

             

            /* Class specific Uncompressed VS Frame Descriptor 2 - 1080p corrected 1/8/19 */
            0x1E,                           /* Descriptor size */
            CX3_CS_INTRFC_DESCR,            /* Descriptor type*/
            0x05,                           /* Subtype:  frame interface*/
            0x02,                           /* Frame Descriptor Index: 2 */
            0x00,                           /* No Still image capture method supported */
            0x80,0x07,                     /* Width in pixel:  1920 */
            0x38,0x04,                     /* Height in pixel: 1080 */
            0x00,0xc0,0xa9,0x1d,         /* Min bit rate (bits/s): 1920 x 1080 x No Of Bits per Pixel x FrameRate = 497,664,000 */
            0x00,0xc0,0xa9,0x1d,         /* Max bit rate (bits/s): Fixed rate so same as Min */
            0x00,0x48,0x3f,0x00,         /* Maximum video or still frame size in bytes(Deprecated): 1920 x 1080 x 2 = 4,147,200 */
            0x2b,0x2c,0x0a,0x00,         /* Default frame interval (in 100ns units): (1/15)x10^7 = 666,667 ns*/
            0x01,                           /* Frame interval type : No of discrete intervals */
            0x2b,0x2c,0x0a,0x00,         /* Frame interval 3: Same as Default frame interval */

             

            Sorry for the pasting being formatted in tables, this seems to be a new feature of the web site.

             

             

            Also, in section 3 the format for 24-bit is identical to that for 16-bit:

             

            So, the GPIF will receive this first 24-bit in the RAW 10 format then convert into 24-bit parallel bits as follows:

             

            i.e First clock cycle - P1[9:2] P2[9:2] P3[9:2]

            Second clock cycle -  P4[9:2] P1[1:0], P2[1:0], P3[1:0], P4[1:0]; P5[9:2]

            Third clock cycle - continue

             

             

            Thanks,

            Scott

            • 3. Re: CX3 OV5647 RAW10 data display
              KandlaguntaR_36

              Well.

              Is it possible to measure the frame size in the firmware (in DMA call back Producer event)? provide the UART logs for the same?

               

              This will confirm whether the CX3 is receiving additional bytes from the sensor itself or something is going wrong in CX3.

               

              You may also probe PCLK, VSYNC and HSYNC test pins and measure the PCLK frequency, Hactive and Vactive time duration then provide same here.

              • 4. Re: CX3 OV5647 RAW10 data display
                ScGr_289066

                Hi,

                 

                The DMA callback shows there are 1920*1080*2=4,147,200 bytes per frame.  The three test points are correct, at least they match the values shown by the image sensor configuration tool.

                 

                However when these frames are received in an OpenCV Mat there are 3-bytes/pixel: 1920*1080*3=6,220,800.  I've tried every conversion I can think of and nothing produces color-correct images of the sensor test pattern or live images when the sensor test pattern is disabled.  So, where is that extra byte/pixel coming from?  Is it added by the PC's UVC or OpenCV's driver?

                 

                As an additional data point, I have tried a variety of GUID's in the CX3's CyCx3USBSSConfigDscr.  These GUIDs came from Microsoft's UVC video sub-type page (Media Type Identifiers - Windows applications | Microsoft Docs).  I've listed my trials and the Microsoft GUIDs here:

                 

                //    YUY2: 32595559-0000-0010-8000-00AA00389B71 (display)
                0x59, 0x55, 0x59, 0x32, 0, 0, 0x10, 0x00, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71,
                //    MEDIASUBTYPE_RGB32:    E436EB7E-524F-11CE-9F53-0020AF0BA770 (no display)
                //    0x7e, 0xeb, 0x36, 0xe4, 0x4f, 0x52, 0xce, 0x11, 0x9f, 0x53, 0x00, 0x20, 0xaf, 0x0b, 0xa7, 0x70,
                //    MEDIASUBTYPE_UYVY     59565955-0000-0010-8000-00AA00389B71 (display)
                //    0x55, 0x59, 0x56, 0x59, 0, 0, 0x10, 0, 0x80, 0, 0, 0xaa, 0, 0x38, 0x9b, 0x71,
                //    MEDIASUBTYPE_RGB1     e436eb78-524f-11ce-9f53-0020af0ba770 (no display)
                //    0x78, 0xeb, 0x36, 0xe4, 0x4f, 0x52, 0xce, 0x11, 0x9f, 0x53, 0, 0x20, 0xaf, 0x0b, 0xa7, 0x70,
                //    MEDIASUBTYPE_RGB24     e436eb7d-524f-11ce-9f53-0020af0ba770 (no display)
                //    0x7d, 0xeb, 0x36, 0xe4, 0x4f, 0x52, 0xce, 0x11, 0x9f, 0x53, 0, 0x20, 0xaf, 0xb, 0xa7, 0x70,
                //    MEDIASUBTYPE_YV12     31313259-0000-0010-8000-00AA00389B71 (no display)
                //    0x59, 0x32, 0x31, 0x31, 0, 0, 0x10, 0, 0x80, 0, 0, 0xaa, 0, 0x38, 0x9b, 0x71,
                //    MEDIASUBTYPE_YVYU     55595659-0000-0010-8000-00AA00389B71 (same as YUY2)
                //    0x59, 0x56, 0x59, 0x55, 0, 0, 0x10, 0, 0x80, 0, 0, 0xaa, 0, 0x38, 0x9b, 0x71,
                //    WMMEDIATYPE_Video     73646976-0000-0010-8000-00AA00389B71 (no display)
                //    0x76, 0x69, 0x64, 0x73, 0, 0, 0x10, 0, 0x80, 0, 0, 0xaa, 0, 0x38, 0x9b, 0x71,
                //    MEDIASUBTYPE_VIDEOIMAGE     1d4a45f2-e5f6-4b44-8388-f0ae5c0e0c37 (no display)
                //    0xf2, 0x45, 0x4a, 0x1d, 0xf6, 0xe5, 0x44, 0x4b, 0x83, 0x88, 0xf0, 0xae, 0x5c, 0xe, 0xc, 0x37,
                //    MEDIASUBTYPE_RGB4     e436eb79-524f-11ce-9f53-0020af0ba770 (no display)
                //    0x79, 0xeb, 0x36, 0xe4, 0x4f, 0x52, 0xce, 0x11, 0x9f, 0x53, 0, 0x20, 0xaf, 0xb, 0xa7, 0x70,
                //    MEDIASUBTYPE_RGB8     e436eb7a-524f-11ce-9f53-0020af0ba770 (display)
                //    0x7a, 0xeb, 0x36, 0xe4, 0x4f, 0x52, 0xce, 0x11, 0x9f, 0x53, 0, 0x20, 0xaf, 0xb, 0xa7, 0x70,

                 

                Only four (marked above as (display)) allow OpenCV (and e-camView) to receive data.  Of the four, the displays look virtually identical except the display colors change.

                 

                When I dumping the data in hex, the three byte groups appear to be UYV or YUV formatted because the middle byte appears to be luminance (proportional to brightness), but standard UYV to RGB conversions don't produce correct images.  Any thoughts on the received data format (in the PC) are appreciated.

                 

                Thanks,

                Scott

                • 5. Re: CX3 OV5647 RAW10 data display
                  KandlaguntaR_36

                  >>The DMA callback shows there are 1920*1080*2=4,147,200 bytes per frame.  The three test points are correct, at least they match the values shown by the image sensor configuration tool.<<

                   

                  Then, the CX3 and Image sensor are working fine. The addition of bytes per frame is on the host side. Please check it on host/driver side.

                   

                   

                  • 6. Re: CX3 OV5647 RAW10 data display
                    KandlaguntaR_36

                    You can even collet the USB 3.0 Trace using Protocol Analyzer in between Device and host then see a frame data and confirm size of a frame.

                    • 7. Re: CX3 OV5647 RAW10 data display
                      ScGr_289066

                      I have collected frame data using OpenCV.  So far, no OpenCV conversion, or any conversion I've tried manually has rendered color correct images.

                       

                      Scott

                      • 8. Re: CX3 OV5647 RAW10 data display
                        KandlaguntaR_36

                        Scott,

                         

                        As per the Debug Prints collected in DMA Producer event, the frame is correct. It means CX3 is streaming the correct frame.

                         

                        Please try with different resolution (instead 1080 P) and check.