11 Replies Latest reply on Sep 21, 2018 4:21 PM by tzu-pingchen

    how to stream video from direct MIPI input without calling sensor initializing via I2C?

    tzu-pingchen

      Hi,

       

      I want to make CX3 receive MIPI sent from my ISP chip directly and stream video through USB3.0 using e-CAM tool.

      Which means, I don't need the CX3 send I2C to initialize the sensor.

      I've configured the sensor and bypassed the "cyu3imagesensor.c" and "cyu3imagesensor.h" by commenting them.

      But the video stream didn't come out, and also the UART didn't print any frame count there.

      I've done some hardware debugging (oscilloscope) and confirm that the MIPI signal is successfully received by CX3.

      And also do UART debug print to check

      the result I get from debug print :

       

      (power on the CX3)

       

      bRType = 0x81, bRequest = 0x0, wValue = 0x0, wIndex = 0x0, wLength= 0x2

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

      bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

      bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

      bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

      StpCB:In SET_FTR 0::1

      AppInit:GpifSMStart passed

       

      (Open e-CAM after powering up CX3)

       

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

      bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

      StpCB:In SET_FTR 0::1

      EnterSuspendMode Status =  0x0, Wakeup reason = 0x8

      bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

      StpCB:In SET_FTR 0::1

      bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

      bRType = 0x21, bRequest = 0x1, wValue = 0x100, wIndex = 0x1, wLength= 0x22

      bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

      bRType = 0xA1, bRequest = 0x83, wValue = 0x100, wIndex = 0x1, wLength= 0x22

      bRType = 0xA1, bRequest = 0x82, wValue = 0x100, wIndex = 0x1, wLength= 0x22

      bRType = 0x21, bRequest = 0x1, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Set cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x81, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x82, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x83, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0x21, bRequest = 0x1, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Set cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x81, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x82, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x83, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0x21, bRequest = 0x1, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Set cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x81, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x82, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0xA1, bRequest = 0x83, wValue = 0x300, wIndex = 0x1, wLength= 0xB

      Get cur Still probe index = 1

      bRType = 0x21, bRequest = 0x1, wValue = 0x200, wIndex = 0x1, wLength= 0x22

      AplnStrt:SMState = 0x2

       

      And it stucks here without any video stream.

      Usually if it start streaming it'll print out :

      "Prod = %d Cons = %d  Prtl_Sz = %d Frm_Cnt = %d Frm_Sz = %d " etc.

       

      I couldn't spot any suspicious activity from these print outs,

      Any suggestions ?

      Is it because I done something wrong on the bypassing part?

      Something like the stream request is waiting an ACK which will never been sent because I commented some essential part?

       

      Attachment is the project I'm currently working on, please have a look.

       

      Thank you,

      Paddy

        • 1. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
          srdr

          It means you do not require any communication from CX3 to ISP. If no, how do you set the resolution that you want to the sensor/ ISP to stream?

          Please explain the control and data flow in this case.

           

          In general,

           

          Control Flow:

          When there is Commit Control request from the host, we set the GPIF (if needed), MIPI CS-2 receiver, Sensor to requested video format and resolution. Then we call UVCStartApp to configure endpoint, DMA, Wake up MIPI and Sensor.

           

          i.e. Commit Control Request -> GPIF II -> MIPI CSI-2 -> Sensor

           

          Here MIPI is configured first then Sensor.

           

          Data Flow:

          Sensor -> MIPI CSI-2 Bridge -> GPIF II -> USB

           

          Refer OV5640 example firmware.

          • 2. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
            tzu-pingchen

            Hi srdr,

            Thanks for your quick reply

            Here is my work flow :

             

            CX3_Work_Flow.png

            The resolution is fixed to 1080p right now.

            The ISP and the firmware inside it will determine the resolution I need and also communicate with sensor. So CX3 won't need to worry about this part.

            And I connect my ISP and CX3 board using a MIPI bridge board, make CX3 receive MIPI package ISP sent.

            Then after CX3 board received MIPI data from ISP, I expect CX3 to unpack MIPI data using my given receiver configuration, then output video data through USB.

            In other word, I want CX3 work as a simple "MIPI in, USB out" board. And stream video on my computer.

             

            Expected Data Flow:

            Sensor -> ISP -> MIPI CSI-2 Bridge -> GPIF -> DMA Channel -> USB

             

             

            I assumed the Eclipse configuration project will configure the CX3 receiving after I key in all the parameters in "CX3 Receiver Configuration".

            And set the receiver configuration in this function:

             

            CyCx3UvcAppImageSensorSetVideoResolution(

                    uint8_t resolution_index

                    ){

            ...

            #ifndef FX3_STREAMING

                 status = CyU3PMipicsiSetIntfParams (&IMX390_1080p_sample2_UYVY_1080P_Ver2, CyFalse);

                 if (status != CY_U3P_SUCCESS)

                 {

                      CyU3PDebugPrint (4, "\n\rUSBStpCB:SetIntfParams SS1 Err = 0x%x", status);

                 }

            #endif

            ...

            }

             

             

            I use UART print out to ensure "CyU3PMipicsiSetIntfParams" had been accessed after I open e-CAMView application.

             

            And I've also checked and found that the function "CyCx3UvcAppDmaCallback" was never been accessed.

            Any suggestions?

             

            Thank you,

            Paddy

            • 3. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
              srdr

              Paddy,

               

              In order to know whether the MIPI CSI-2 receiver is configured properly, we should probe VSYNC and HSYNC test pins and measure the timings of VSYNC and HSYNC then check whether they meet the requirement.

               

              If the MIPI CSI-2 receiver configuration is wrong and GPIF II cannot get the data. Hence, there is no DMA Call back.

              Please share the screenshot of the MIPI receiver configuration tab of MIPI configuration tool for review.

              1 of 1 people found this helpful
              • 4. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                tzu-pingchen

                Hi srdr,

                 

                I couldn't be too sure if I'm probing the right H5 (VSYNC), G6(HSYNC) signal.

                I made my guess based on the "e-con_CX3RDK_Hardware_UserManual" and the ball map. And the output waveform seems to be wrong.

                Is there any guidance on which is the right ball to probe?

                Or is there any other output pins to check with VSYNC and HSYNC?

                Currently I guess it's these two:

                probe.png

                And the output is more like a VCC instead of VSYNC signal.

                below is my MIPI receiver configuration.

                config.png

                 

                Thank you,

                Paddy

                • 5. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                  tzu-pingchen

                  Hi srdr,

                   

                  I checked the MIPI data signal again, and found that termination doesn't seem to be started.

                   

                  I'm sending two type of MIPI signal lines,

                  one type is Start of frame, End of frame data, with Tzero around 150ns

                  and one is the HS MIPI data with 843ns Tzero.

                  In my FPGA, these are two different type of packets.

                  Would this be the problem that CX3 didn't able to receive anything?

                   

                  Thanks,

                  Paddy

                  • 6. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                    tzu-pingchen

                    Hi srdr,

                     

                    I've found some problem in my previous configuration,

                    this is the number I'm using now.

                    config.png

                     

                    e-CAM is still black, do i need to tune any clock value under "CX3 MIPI Interface Configuration"? I'm currently using almost the same setting as OV5640 example 1080p 30fps because I'm pretty new to this area and doesn't know what would be the best settings.

                     

                    Also, do you have any recommend PHY Time delay (CyU3PMipicsiSetPhyTimeDelay()) value I can set based on this configuration?

                    Currently I'm setting it 9 in my firmware.

                     

                    Thanks,

                    Paddy

                    • 7. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                      srdr

                      Paddy,

                       

                      Note that you should not directly use the MIPI configuration of OV5640, if your ISP MIPI transceiver configuration is not same as OV5640's configuration.

                       

                      You have to configure the MIPI Receiver based on MIPI Transceiver configuration.

                       

                      CSI Clock

                      Data lane

                      THS_Prepare

                      THS_Zero

                      H_Active

                      H_Blanking

                      V_Active

                      V_Blanking

                      Frames per second

                      Data Format

                       

                      Please probe the following pins to know VSYNC and HSYNC values:

                       

                      G6         HSYNC_test

                      H5         VSYNC_test

                      H8         PCLK_test

                       

                      Why do you have two MIPI configurations as said above? It should be one for a single configuration. i.e one MIPI Receiver configuration should be mapped to one MIPI Transceiver configuration.

                       

                      You have set the THS Settle value (PHY Time Delay Value) of MIPI Receiver to the value generated by CX3 MIPI configuration tool using CyU3PMipicsiSetPhyTimeDelay() API.

                      Refer right bottom section - CX3 MIPI Interface Configuration.

                      • 8. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                        tzu-pingchen

                        Hi srdr,

                         

                        I've probe the PCLK, VSYNC and HSYNC output signal and only PCLK is working.

                        According to trouble shooting guide, this means there's problems in my configuration.

                         

                        Also I'm sure CSI CLK I'm using is 148.5 instead of 297, I was testing something.

                        I've changed MIPI Interface Configuration back to the auto generated one.

                        config.png

                        The auto generated configuration has an error in Output Pixel Clock, and I've tried to increase Multiplier of Unit Clk to make pixel clock larger than 74.25.

                        But there's still no output stream.

                         

                        Any suggestions on how to configure it?

                         

                        Thank you,

                        Paddy

                        • 9. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                          tzu-pingchen

                          Hi srdr,

                          Til now, CX3 still refuse to process the MIPI data I gave it.

                          So I'm currently trying to make my ISP send gated clock as default instead of continuous clock, to make it "less complicate".

                          And also continue to tweak configuration parameters.

                          If nothing works, next I'll try to made my ISP send the same MIPI as OV5640 sensor. But it's like lots of works.

                           

                          There are still some questions.

                           

                          The only reason that cause no GPIF callback interrupt is only because the MIPI receiver configuration?

                          Or is there any other possible reasons I can look into?

                          Like missing some essential settings.

                          Or maybe the sensor I2C communication shouldn't be disabled no matter I want to communicate through it or not.

                           

                          Also, I assume the definition of MIPI CSI2 CLK is the CLK not data bit rate. Is that right?

                           

                          Thanks,

                          Paddy

                          • 10. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                            srdr

                            If there is no VSYNC and HSYNC, it can be due to MIPI configuration error.

                            Yes, the definition of MIPI CSI2 CLK is the CLK not bit rate.

                             

                             

                            Please try the following MIPI CSI-2 receiver configuration.

                            CyU3PMipicsiCfg_t null_UYVY_Resolution0 = 

                            {

                                CY_U3P_CSI_DF_YUV422_8_1,  /* CyU3PMipicsiDataFormat_t dataFormat */

                                4,                          /* uint8_t numDataLanes */

                                2, /* uint8_t pllPrd */

                                99, /* uint16_t pllFbd */

                                CY_U3P_CSI_PLL_FRS_125_250M, /* CyU3PMipicsiPllClkFrs_t pllFrs */ 

                                CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t csiRxClkDiv */

                                CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t parClkDiv */

                                0,                 /* uint16_t mClkCtl */

                                CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t mClkRefDiv */

                                1920,         /* uint16_t hResolution */

                                80                         /* uint16_t fifoDelay */

                            };

                             

                            Configure the Phy Delay values using CyU3PMipicsiSetPhyTimeDelay API. Pass 12 in the second argument and keep 1 in the first argument. Refer API Guide for more info.

                             

                            Please check for MIPI errors once you configure the MIPI using CyU3PMipicsiGetErrors API. Refer OV5640 firmware for this.

                            • 11. Re: how to stream video from direct MIPI input without calling sensor initializing via I2C?
                              tzu-pingchen

                              Hi srdr,

                              Thanks for the reply!

                              I tried with this MIPI CSI2 receiver configuration.

                              Set the CyU3PMipicsiSetPhyTimeDelay to 12 after setting CyU3PMipicsiSetIntfParams() with CyFalse in second argument.

                              phytimedelay.png

                              Unfortunately, the video is still not streaming.

                              (commenting CyCx3_ImageSensor_Set_1080p_vol2 is because I don't need Cx3 to write anything to my ISP through I2C)

                               

                               

                              I use CyU3PMipicsiGetErrors API to get error counts after setting the configuration.

                              And sometimes Control Error (Incorrect Line State Sequence) Count is 1 when I open e-CAMView (send commit control).

                              phytimedelay.png

                              What does it mean? Sometimes it happens and sometimes don't.

                               

                              I also tried create MIPI error thread refering to OV5640 example, the thread had initialized successfully, but the thread seems to stuck at "CyU3PEventGet", According to api description it's because none of the flag are signaled. But I have no idea why.

                               

                              And the GPIF still stays at SMState= 0x2 (Waiting) after AppStart and never receive a callback interrupt.

                               

                              Thanks,

                              Paddy