how to stream video from direct MIPI input without calling sensor initializing via I2C?

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
TzCh_3504586
Level 3
Level 3
5 likes given First like given

Hi,

I want to make CX3 receive MIPI sent from my ISP chip directly and stream video through USB3.0 using e-CAM tool.

Which means, I don't need the CX3 send I2C to initialize the sensor.

I've configured the sensor and bypassed the "cyu3imagesensor.c" and "cyu3imagesensor.h" by commenting them.

But the video stream didn't come out, and also the UART didn't print any frame count there.

I've done some hardware debugging (oscilloscope) and confirm that the MIPI signal is successfully received by CX3.

And also do UART debug print to check

the result I get from debug print :

(power on the CX3)

bRType = 0x81, bRequest = 0x0, wValue = 0x0, wIndex = 0x0, wLength= 0x2

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

StpCB:In SET_FTR 0::1

AppInit:GpifSMStart passed

(Open e-CAM after powering up CX3)

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x100, wLength= 0x0

StpCB:In SET_FTR 0::1

EnterSuspendMode Status =  0x0, Wakeup reason = 0x8

bRType = 0x1, bRequest = 0x3, wValue = 0x0, wIndex = 0x0, wLength= 0x0

StpCB:In SET_FTR 0::1

bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0x21, bRequest = 0x1, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x83, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x82, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0x21, bRequest = 0x1, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Set cur Still probe index = 1

bRType = 0xA1, bRequest = 0x81, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0xA1, bRequest = 0x82, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0xA1, bRequest = 0x83, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0x21, bRequest = 0x1, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Set cur Still probe index = 1

bRType = 0xA1, bRequest = 0x81, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0xA1, bRequest = 0x82, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0xA1, bRequest = 0x83, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0x21, bRequest = 0x1, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Set cur Still probe index = 1

bRType = 0xA1, bRequest = 0x81, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0xA1, bRequest = 0x82, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0xA1, bRequest = 0x83, wValue = 0x300, wIndex = 0x1, wLength= 0xB

Get cur Still probe index = 1

bRType = 0x21, bRequest = 0x1, wValue = 0x200, wIndex = 0x1, wLength= 0x22

AplnStrt:SMState = 0x2

And it stucks here without any video stream.

Usually if it start streaming it'll print out :

"Prod = %d Cons = %d  Prtl_Sz = %d Frm_Cnt = %d Frm_Sz = %d " etc.

I couldn't spot any suspicious activity from these print outs,

Any suggestions ?

Is it because I done something wrong on the bypassing part?

Something like the stream request is waiting an ACK which will never been sent because I commented some essential part?

Attachment is the project I'm currently working on, please have a look.

Thank you,

Paddy

0 Likes
1 Solution

It is good to know that the issue on the data lanes is solved.

Please probe VSYNC and HSYNC now and share the HSYNC High and Low; VSYNC high and low timings.

Answering other queries:

1. Note that the GPIF interrupt comes when there is Frame End.

     In the frame end state of the GPIF state machine, we have set Interrupt CPU action. This triggers the GPIFCB function. Hence, DMA      Callback event can come irrespective of GPIFCB.

2. Please remove the debug prints in Callback functions. This may block normal functionality.

     Put a flag and print in the infinite for loop running in Main Thread.

3. Please share the snippet that printed 0x45 (CY_U3P_ERROR_TIMEOUT = Timeout on relevant operation). This error comes when there is      a timeout on relevant operation

I hope that you might have referred AN75779 App. Note to understand the UVC application firmware. If not, please refer.

View solution in original post

20 Replies
KandlaguntaR_36
Moderator
Moderator
Moderator
25 solutions authored 10 solutions authored 5 solutions authored

It means you do not require any communication from CX3 to ISP. If no, how do you set the resolution that you want to the sensor/ ISP to stream?

Please explain the control and data flow in this case.

In general,

Control Flow:

When there is Commit Control request from the host, we set the GPIF (if needed), MIPI CS-2 receiver, Sensor to requested video format and resolution. Then we call UVCStartApp to configure endpoint, DMA, Wake up MIPI and Sensor.

i.e. Commit Control Request -> GPIF II -> MIPI CSI-2 -> Sensor

Here MIPI is configured first then Sensor.

Data Flow:

Sensor -> MIPI CSI-2 Bridge -> GPIF II -> USB

Refer OV5640 example firmware.

0 Likes

Hi srdr,

Thanks for your quick reply

Here is my work flow :

CX3_Work_Flow.png

The resolution is fixed to 1080p right now.

The ISP and the firmware inside it will determine the resolution I need and also communicate with sensor. So CX3 won't need to worry about this part.

And I connect my ISP and CX3 board using a MIPI bridge board, make CX3 receive MIPI package ISP sent.

Then after CX3 board received MIPI data from ISP, I expect CX3 to unpack MIPI data using my given receiver configuration, then output video data through USB.

In other word, I want CX3 work as a simple "MIPI in, USB out" board. And stream video on my computer.

Expected Data Flow:

Sensor -> ISP -> MIPI CSI-2 Bridge -> GPIF -> DMA Channel -> USB

I assumed the Eclipse configuration project will configure the CX3 receiving after I key in all the parameters in "CX3 Receiver Configuration".

And set the receiver configuration in this function:

CyCx3UvcAppImageSensorSetVideoResolution(

        uint8_t resolution_index

        ){

...

#ifndef FX3_STREAMING

     status = CyU3PMipicsiSetIntfParams (&IMX390_1080p_sample2_UYVY_1080P_Ver2, CyFalse);

     if (status != CY_U3P_SUCCESS)

     {

          CyU3PDebugPrint (4, "\n\rUSBStpCB:SetIntfParams SS1 Err = 0x%x", status);

     }

#endif

...

}

I use UART print out to ensure "CyU3PMipicsiSetIntfParams" had been accessed after I open e-CAMView application.

And I've also checked and found that the function "CyCx3UvcAppDmaCallback" was never been accessed.

Any suggestions?

Thank you,

Paddy

0 Likes

Paddy,

In order to know whether the MIPI CSI-2 receiver is configured properly, we should probe VSYNC and HSYNC test pins and measure the timings of VSYNC and HSYNC then check whether they meet the requirement.

If the MIPI CSI-2 receiver configuration is wrong and GPIF II cannot get the data. Hence, there is no DMA Call back.

Please share the screenshot of the MIPI receiver configuration tab of MIPI configuration tool for review.

Hi srdr,

I couldn't be too sure if I'm probing the right H5 (VSYNC), G6(HSYNC) signal.

I made my guess based on the "e-con_CX3RDK_Hardware_UserManual" and the ball map. And the output waveform seems to be wrong.

Is there any guidance on which is the right ball to probe?

Or is there any other output pins to check with VSYNC and HSYNC?

Currently I guess it's these two:

probe.png

And the output is more like a VCC instead of VSYNC signal.

below is my MIPI receiver configuration.

config.png

Thank you,

Paddy

0 Likes

Hi srdr,

I checked the MIPI data signal again, and found that termination doesn't seem to be started.

I'm sending two type of MIPI signal lines,

one type is Start of frame, End of frame data, with Tzero around 150ns

and one is the HS MIPI data with 843ns Tzero.

In my FPGA, these are two different type of packets.

Would this be the problem that CX3 didn't able to receive anything?

Thanks,

Paddy

0 Likes

Hi srdr,

I've found some problem in my previous configuration,

this is the number I'm using now.

config.png

e-CAM is still black, do i need to tune any clock value under "CX3 MIPI Interface Configuration"? I'm currently using almost the same setting as OV5640 example 1080p 30fps because I'm pretty new to this area and doesn't know what would be the best settings.

Also, do you have any recommend PHY Time delay (CyU3PMipicsiSetPhyTimeDelay()) value I can set based on this configuration?

Currently I'm setting it 9 in my firmware.

Thanks,

Paddy

0 Likes

Paddy,

Note that you should not directly use the MIPI configuration of OV5640, if your ISP MIPI transceiver configuration is not same as OV5640's configuration.

You have to configure the MIPI Receiver based on MIPI Transceiver configuration.

CSI Clock

Data lane

THS_Prepare

THS_Zero

H_Active

H_Blanking

V_Active

V_Blanking

Frames per second

Data Format

Please probe the following pins to know VSYNC and HSYNC values:

G6         HSYNC_test

H5         VSYNC_test

H8         PCLK_test

Why do you have two MIPI configurations as said above? It should be one for a single configuration. i.e one MIPI Receiver configuration should be mapped to one MIPI Transceiver configuration.

You have set the THS Settle value (PHY Time Delay Value) of MIPI Receiver to the value generated by CX3 MIPI configuration tool using CyU3PMipicsiSetPhyTimeDelay() API.

Refer right bottom section - CX3 MIPI Interface Configuration.

Hi srdr,

I've probe the PCLK, VSYNC and HSYNC output signal and only PCLK is working.

According to trouble shooting guide, this means there's problems in my configuration.

Also I'm sure CSI CLK I'm using is 148.5 instead of 297, I was testing something.

I've changed MIPI Interface Configuration back to the auto generated one.

config.png

The auto generated configuration has an error in Output Pixel Clock, and I've tried to increase Multiplier of Unit Clk to make pixel clock larger than 74.25.

But there's still no output stream.

Any suggestions on how to configure it?

Thank you,

Paddy

0 Likes

Hi srdr,

Til now, CX3 still refuse to process the MIPI data I gave it.

So I'm currently trying to make my ISP send gated clock as default instead of continuous clock, to make it "less complicate".

And also continue to tweak configuration parameters.

If nothing works, next I'll try to made my ISP send the same MIPI as OV5640 sensor. But it's like lots of works.

There are still some questions.

The only reason that cause no GPIF callback interrupt is only because the MIPI receiver configuration?

Or is there any other possible reasons I can look into?

Like missing some essential settings.

Or maybe the sensor I2C communication shouldn't be disabled no matter I want to communicate through it or not.

Also, I assume the definition of MIPI CSI2 CLK is the CLK not data bit rate. Is that right?

Thanks,

Paddy

0 Likes

If there is no VSYNC and HSYNC, it can be due to MIPI configuration error.

Yes, the definition of MIPI CSI2 CLK is the CLK not bit rate.

Please try the following MIPI CSI-2 receiver configuration.

CyU3PMipicsiCfg_t null_UYVY_Resolution0 = 

{

    CY_U3P_CSI_DF_YUV422_8_1,  /* CyU3PMipicsiDataFormat_t dataFormat */

    4,                          /* uint8_t numDataLanes */

    2, /* uint8_t pllPrd */

    99, /* uint16_t pllFbd */

    CY_U3P_CSI_PLL_FRS_125_250M, /* CyU3PMipicsiPllClkFrs_t pllFrs */ 

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t csiRxClkDiv */

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t parClkDiv */

    0,                 /* uint16_t mClkCtl */

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t mClkRefDiv */

    1920,         /* uint16_t hResolution */

    80                         /* uint16_t fifoDelay */

};

Configure the Phy Delay values using CyU3PMipicsiSetPhyTimeDelay API. Pass 12 in the second argument and keep 1 in the first argument. Refer API Guide for more info.

Please check for MIPI errors once you configure the MIPI using CyU3PMipicsiGetErrors API. Refer OV5640 firmware for this.

0 Likes

Hi srdr,

Thanks for the reply!

I tried with this MIPI CSI2 receiver configuration.

Set the CyU3PMipicsiSetPhyTimeDelay to 12 after setting CyU3PMipicsiSetIntfParams() with CyFalse in second argument.

phytimedelay.png

Unfortunately, the video is still not streaming.

(commenting CyCx3_ImageSensor_Set_1080p_vol2 is because I don't need Cx3 to write anything to my ISP through I2C)

I use CyU3PMipicsiGetErrors API to get error counts after setting the configuration.

And sometimes Control Error (Incorrect Line State Sequence) Count is 1 when I open e-CAMView (send commit control).

phytimedelay.png

What does it mean? Sometimes it happens and sometimes don't.

I also tried create MIPI error thread refering to OV5640 example, the thread had initialized successfully, but the thread seems to stuck at "CyU3PEventGet", According to api description it's because none of the flag are signaled. But I have no idea why.

And the GPIF still stays at SMState= 0x2 (Waiting) after AppStart and never receive a callback interrupt.

Thanks,

Paddy

0 Likes

Hi srdr,

I found that I couldn't set gated clock on my ISP side. I'll just continue to use continuous clock instead.

Just make sure I'm not messing up this part.

To set continuous clock mode.

Once CX3 receive the commit control, do a MIPI hard reset, initialize the MIPI bridge, set the MIPI receiver configuration then reset of sensor.

setcur.png

For the MIPI receiver configuration setting I just leave "CyCx3UvcAppImageSensorSetVideoResolution (glCommitCtrl[3])" untouched.

And the Sensor reset I just power cycle my ISP.

Is this correct?

Thanks!
Paddy

0 Likes

The above snippet is fine.

Along with the above, please check whether you are doing the following in CyCx3UvcAppImageSensorSetVideoResolution

i.e

status = CyU3PMipicsiSetIntfParams (&null_YUY2_1080p, CyFalse);

  CyU3PMipicsiSetPhyTimeDelay(1, Phydelay value); // Enter the PHY Delay Value here

   

  break;

after CyCx3AppImageSensorSetVideoResolution you should wakeup the mipi using CyU3PMipicsiWakeup in CyCx3AppStart function.

Refer OV5640 example firmware located here in the SDK: C:\Program Files (x86)\Cypress\EZ-USB FX3 SDK\1.3\firmware\cx3_examples\cycx3_uvc_ov5640

Hi srdr,

Yes, my PHY Time Delay is set right after CyU3PMipicsiSetIntfParams with second argument set CyFalse.

There are some improvement based on using the configuration you gave us.

Although there's still no VSYNC/HSYNC, but the termination of MIPI signal toggled. I checked it with probing the input MIPI signal.

Does it means at least CX3 recognize the MIPI signal?

Checked the debug output, GPIF II still stays at state 2 and DMA callback not been triggered.

There are some questions I want to ask :

1. Is LP to HS transition necessary to CLK?

2. Based on the GPIF II state machine, state 2 to state 3 transition need to wait for frame valid. Is there any API I can use to check if GPIF II is getting any MIPI "line" instead of the whole MIPI "frame", the reason is I want to know why my GPIF II always stuck at state 2.

3. I've done some minor change to my input configuration again (based on the read of my probing)

MIPI CSI clock :             148.5 --> 150  (Mhz)

THS-ZERO :                  158    --> 472  (ns)

Horizontal blanking :      280    --> 302  (pixel)

I've changed the THS Settle based on the new PHY Time Delay, how about others? Will it affect any other receiver settings you gave me?

Thanks,

Paddy

0 Likes

1. LP to HS transition is not necessary to CLK.

2. If there is no VSYNC High from MIPI block of CX3, the GPIF stuck at state 2 itself. Please probe

G6 HSYNC_test and H5 VSYNC_test  pins for HSYNC and VSYNC signals. There is no APIs as you requested.

Note that the VSYNC goes high when there is Frame Start on the MIPI lane and goes Low when there is Frame End on the MIPI Lane.

3. If you change any parameters in the Transmitter side, enter the same in MIPI configuration tool then use the MIPI configuration generated by it.

Have you checked whether there is any MIPI errors using CyU3PMipicsiGetErrors?

Please refer CX3_ERROR_THREAD_ENABLE MACRO in OV5640 example firmware to monitor the MIPI errors.

Hi srdr,

Thanks for the information,

I probe the VSYNC_test, HSYNC_test and PCLK again , VSYNC_test and HSYNC_test are not toggled. The PCLK is 80Mhz, same as the MIPI receiver confguraion I gave to CX3.

And I enable the CX3_ERROR_THREAD_ENABLE to print error count every 5 seconds, all of the error counts are zeros.

config.png

When I enable CX3_ERROR_THREAD_ENABLE I also change the CyU3PEventGet() wait Option in CyCx3UvcAppThread_Entry() function.

To made it do CyU3PEventGet() every 5 seconds instead of doing busy waiting the whole time.

Also I made an experiment to test if the GPIF II indeed is not receiving anything.

I call the GPIF II call back function manually, 3 seconds after receiving SET_CUR request.

The GPIF --> DMA --> UVC workflow works perfectly and sends nothing out as I expected.

config.png

Any Suggestions on what to do next?

Feels like it's still a configuration problem but I'm not too sure.

I'm still using the MIPI receiver configuration you gave me, since the configs generated by Eclipse didn't even trigger the MIPI HS termination.

Thanks,

Paddy

0 Likes

I have entered your settings in the MIPI tool as follows and obtain MIPI configuration.

pastedImage_0.png

The MIPI Config. parameters are as follows:

CyU3PMipicsiCfg_t null_UYVY_Resolution0 =

{

    CY_U3P_CSI_DF_YUV422_8_2,  /* CyU3PMipicsiDataFormat_t dataFormat */

    4,                          /* uint8_t numDataLanes */

    2, /* uint8_t pllPrd */

    99, /* uint16_t pllFbd */

    CY_U3P_CSI_PLL_FRS_125_250M, /* CyU3PMipicsiPllClkFrs_t pllFrs */ 

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t csiRxClkDiv */

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t parClkDiv */

    0,                 /* uint16_t mClkCtl */

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t mClkRefDiv */

    1920,         /* uint16_t hResolution */

    80                         /* uint16_t fifoDelay */

};

We can see there is no change in above parameters.

Since you changed the THS Zero to 472 ns, the THS Settle value i.e PHY Delay Value should change to 24 from 8.

You can see this number in the right bottom corner under CX3 MIPI Interface Configuration.

As per the above responses from you, the MIPI configuration is fine and there are no MIPI errors.

Can you please check whether the ISP is streaming data correctly (MIPI Transmitter settings)? Hence, the CX3 MIPI can decode accordingly.

Please let me know the packet format of MIPI Transmitter.

Please check whether the connection between MIPI Receiver and MIPI Transmitter (ISP) are correct for all four data lanes and clock lane.

You told that there are terminations. However, check once again for all the lanes.

Hi srdr,

Thanks for the notice,

I checked four lanes again, found that one of our lane had some problem.

Fixed it.

And now I'm able to get in to DMA Call Back event, but the producer callback failed with Error code = 0x45.

config.png

Sorry for having this messy debug message, I've added tons of things myself.

Beside the error 0x45,

Why does the DMA callback have been triggered, while GPIF II seems not to be triggered at all?

I have this question is because I got a debug print message in GPIF II call back function like this:

CyU3PDebugPrint (4,"\n\r*********************GPIF Interrupt*************************");

And I assume that the print out message should be happening before "********DMA Call Back*******" message.

While it never shows up.

Thanks,

Paddy

0 Likes

It is good to know that the issue on the data lanes is solved.

Please probe VSYNC and HSYNC now and share the HSYNC High and Low; VSYNC high and low timings.

Answering other queries:

1. Note that the GPIF interrupt comes when there is Frame End.

     In the frame end state of the GPIF state machine, we have set Interrupt CPU action. This triggers the GPIFCB function. Hence, DMA      Callback event can come irrespective of GPIFCB.

2. Please remove the debug prints in Callback functions. This may block normal functionality.

     Put a flag and print in the infinite for loop running in Main Thread.

3. Please share the snippet that printed 0x45 (CY_U3P_ERROR_TIMEOUT = Timeout on relevant operation). This error comes when there is      a timeout on relevant operation

I hope that you might have referred AN75779 App. Note to understand the UVC application firmware. If not, please refer.

Hi srdr,

When I disable the print debug message.

The video finally came out.

Thanks for all the support !!

You're the best

Best,

Paddy

0 Likes