https get can not work correctly

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
Us16_3774766
Level 1
Level 1
First like received First like given

I am testing snip.httpbin_org on CWY943907AEVAL1F. I use WICED-Studio-6.2.

The original sample work correctly. I modified the  original source code to send data to my local server. I removed the CA check.

I can send small file (size = 300kbyte). but  I could not send a file that is over 400kbyte size.

Is it needed to change mbedtls/config.h ?

0 Likes
1 Solution

Please increase the packet pool size to avoid the assert issues.

GLOBAL_DEFINES += TX_PACKET_POOL_SIZE=xx \

  RX_PACKET_POOL_SIZE=xx

The value of packet pools is dependent on the amount of data exchanged. Hence the value "xx" will be higher for applications where large amount of data is transferred.

+= TX_PACKET_POOL_SIZE=14 \

                  RX_PACKET_POOL_SIZE=12

View solution in original post

0 Likes
18 Replies
PriyaM_16
Moderator
Moderator
Moderator
250 replies posted 100 replies posted 50 replies posted

Hello,

Root CA is to ensure the authenticity of the server. If you do not want a secure conneection, you can omit the certificates. In general, when a client wants to connect to a server, it sends a connection request. The server then sends a certificate in the server hello message. The client can verify the authenticity of the server as a trusted root ca is present with the client.

As you were already able to establish a connection with your local server, I believe you are familiar with the flow. Can you give more details about the failure of sending the 400kB file? At what point it fails? Is your connection still intact with the server? Do you specifically have problems with "POST" request or with "GET" request only?

Thank you for your replay.

I use "400k.bin" that is filled with "0x35".  The size is just 400KB. I use "GET" request.

I am checking the payload_data_length.  The server finished sending the data. But Device received only 16384byte data.

When I checked by debug mode. "wiced_assert"  was called in tls_host_create_buffer function.

I have checked "PUT" request . It's OK. But I have not checked "POST" request.

0 Likes

Hello,

Can you please post the initial response from the server where the size of the file that needs to be transferred is passed?

I have set up a local server with some text to replicate the issue. I was able to receive the files up to 510KB. Please check the response I received form the server:

HTTP Header Information for response2 :

==============================================

HTTP/1.1 200 OK

server: ecstatic-3.3.0

cache-control: max-age=3600

last-modified: Wed, 21 Nov 2018 10:22:43 GMT

etag: W/"23925373020530490-512656-2018-11-21T10:22:43.372Z"

content-length: 512656

content-type: text/html; charset=UTF-8

Date: Wed, 21 Nov 2018 13:17:34 GMT

Connection: keep-alive

==============================================

Note: I am testing with httpbin_org snip with no security to check the variable size issue. I have not modified the variable size to uint32_t.

I can share the snip and local server details if required.

lock attach
Attachments are accessible only for community members.

Hello riya

> Can you please post the initial response from the server where the size of the file that needs to be transferred is passed?

I got a log with three patterns.

I changed httpbin_org.c slightly.

The value of response-> payload_data_length is added to another variable and it is printed at the end.

(This is the line "recv_total_size = ....".)

* httpbin_org.c is attached for reference.

File size is below.

1) 5MB

  -> 1024 * 1024 * 5    = 5242880 = 0x00500000

3) 5MB + 1Byte

  -> 1024 * 1024 * 5 + 1 = 5242881 = 0x00500001

2) 5MB - 1Byte

  -> 1024 * 1024 * 5 - 1 = 5242879 = 0x004FFFFF

The logs is below.

1) 5MB

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

Server: 04WebServer/1.92

Date: Thu, 22 Nov 2018 04:16:20 GMT

Content-Type: application/octet-stream

Connection: Close

Last-modified: Fri, 12 Oct 2018 05:10:16 GMT

ETag: "500000-5bc02cb8:5237"

Content-Length: 5242880

==============================================

Received total payload data for response1

recv_total_size = 0

2) 5MB + 1Byte

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

Server: 04WebServer/1.92

Date: Thu, 22 Nov 2018 04:16:52 GMT

Content-Type: application/octet-stream

Connection: Close

Last-modified: Thu, 22 Nov 2018 04:06:40 GMT

ETag: "500001-5bf62b50:5237"

Content-Length: 5242881

==============================================

Received total payload data for response1

recv_total_size = 5242881

3) 5MB - 1Byte

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

Server: 04WebServer/1.92

Date: Thu, 22 Nov 2018 04:17:26 GMT

Content-Type: application/octet-stream

Connection: Close

Last-modified: Thu, 22 Nov 2018 04:07:11 GMT

ETag: "4fffff-5bf62b6f:5237"

Content-Length: 5242879

==============================================

Received total payload data for response1

recv_total_size = 5242879

> I was able to receive the files up to 510KB. Please check the response I received form the server:

The response Content-Length is correct.
It is because it directly printed the character string in the header.

The actual length of the data received by the application is the sum of "response-> payload_data_length".

Also, I think that this problem will not be reproduced by the size of the test data of your log.
Since 512656(Byte) is 0x7D290(Byte), the values ​​of the lower 16bits are not taken into account.

Please try with data whose lower 16bits of the file size will be the following numerical values.
0x0000: for example 5 MB (0x00500000)
0x4000: for example 400 KB (0x00064000)

Regards

0 Likes
lock attach
Attachments are accessible only for community members.

Hello,

Local Server details:

Started a local server through cgywin using package http-server

1. Install the package:

    npm install http-server -g

2.  Create a dire named public and place the sometext_409600 file attached here. The sometext_409600.html file contains some junk data.

    mkdir public

3. start http server

    http-server ./public

I tested the httbin_org which you have attached and I got the following response:

Connecting to 192.168.43.121

Connected

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

server: ecstatic-3.3.0

cache-control: max-age=3600

last-modified: Tue, 27 Nov 2018 13:02:55 GMT

etag: W/"3659174697948289-409600-2018-11-27T13:02:55.485Z"

content-length: 409600

content-type: text/html; charset=UTF-8

Date: Wed, 28 Nov 2018 05:34:17 GMT

Connection: keep-alive

==============================================

Received total payload data for response1

recv_total_size = 409600

Note: I have changed the SERVER_PORT to 8080.

Can you please attach the 5MB file that you are using for testing? I will test it on my side and try to replicate your issue.

lock attach
Attachments are accessible only for community members.

Hello

> Can you please attach the 5MB file that you are using for testing? I will test it on my side and try to replicate your issue.

I attached 5MB file.

Please change the file name so that it is easy to understand.

Thanks for the information on the details of the local server.

I have not prepared the server yet, but I tried it with IIS.

(Tested with Non-Secure)

--------------------------------------------------------------------------------------------

Connecting to 192.168.2.101

Connected

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

Content-Type: text/html

Last-Modified: Fri, 12 Oct 2018 05:10:16 GMT

Accept-Ranges: bytes

ETag: "efa66dde961d41:0"

Server: Microsoft-IIS/7.5

Date: Thu, 29 Nov 2018 06:17:05 GMT

Content-Length: 5242880

==============================================

Received total payload data for response1

recv_total_size : 0

Disconnected from 192.168.2.101

--------------------------------------------------------------------------------------------

You tested it without using the patch from https_client.c I attached before, right?

0 Likes

Hello

I tried some additional tests.

I've sorted out my test results so far.

Secure

or

Non-Secure

FileSize

Result

  OK : Issue Not Occured

  NG : Issue Occred,

Non-Secure5MBNG
Non-Secure5MB - 1ByteOK
Non-Secure5MB + 1ByteOK
Non-Secure400KBOK
Secure5MBNG
Secure400KBNG

On my side, with 5MB file, issue is occured both Secure and Non-Secure.

With 400KB file, no problem occurred when Non-Secure, but a problem occurred when it was Secure.

・Non-Secure-400KB Response Log

--------------------------------------------------------------

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

Server: 04WebServer/1.92

Date: Thu, 29 Nov 2018 06:38:34 GMT

Content-Type: application/octet-stream

Connection: Close

Last-modified: Mon, 12 Nov 2018 04:28:48 GMT

ETag: "64000-5be90180:5237"

Content-Length: 409600

==============================================

Received total payload data for response1

recv_total_size : 409600

Disconnected from 192.168.2.101

--------------------------------------------------------------

・Secure-400KB Response Log

--------------------------------------------------------------

HTTP Header Information for response1 :

==============================================

HTTP/1.1 200 OK

Server: 04WebServer/1.92

Date: Thu, 29 Nov 2018 06:40:09 GMT

Content-Type: application/octet-stream

Connection: Close

Last-modified: Mon, 12 Nov 2018 04:28:48 GMT

ETag: "64000-5be90180:5237"

Content-Length: 409600

==============================================

Received total payload data for response1

recv_total_size : 16384

--------------------------------------------------------------

Regards

I think we figured out the problem in receiving large file i.e., 5MB(NS).

The http_library has a function for capturing the content length from the http response. The value is type casted to uint16_t which should be changed to uint32_t.

Kindly change the data type at the following places:

1. In file http_client.c, change static uint16_t get_content_length( char* data , uint16_t data_len) to static uint32_t get_content_length( char* data , uint16_t data_len)

2. change return ( (uint16_t) ( strtol( data + 1, NULL, 10 ) ) ); to return ( (uint32_t) ( strtol( data + 1, NULL, 10 ) ) );

3. Change the content length variable in static wiced_result_t client_receive_handler( void* arg 😞

  uint32_t content_length = 0;

4. In file http_client.h change the payload_data_length in structure http_response_t

  uint32_t payload_data_length; /* This length indicates only payload length */ //uint16_t

Please check with these changes and revert back the result.

riya wrote:

I think we figured out the problem in receiving large file i.e., 5MB(NS).

The http_library has a function for capturing the content length from the http response. The value is type casted to uint16_t which should be changed to uint32_t.

Isn't this the same finding nunokawa.akira_2072836 already told you on Oct 15, 2018 in this thread?

0 Likes

Yes Sir but there was still problem in sending 5MB data(5242880 bytes) which goes through if the mentioned changes are done.

0 Likes

This fix is ​​for http_client.c that has not changed anything, right?

If so, for example to change (1) and (2), it seems necessary to change other variables to 32 bit.
For example, response_info-> total_remaining_length or http_response.remaining_length.

Please provide a patch file as I did, not a procedure on how to fix it.

0 Likes

Besides, where does the connection fail in secure mode. Please capture the TLS handshake information as mentioned in Debugging TLS in WICED​ and share the log file.

0 Likes

Do you have anything to do with the modification of http_client.c and TLS?

I think TLS is related to the problem where wiced_assert occurs in wiced_tls.c.

If so, look at that log as it already shares it to you in another way.

0 Likes

Please increase the packet pool size to avoid the assert issues.

GLOBAL_DEFINES += TX_PACKET_POOL_SIZE=xx \

  RX_PACKET_POOL_SIZE=xx

The value of packet pools is dependent on the amount of data exchanged. Hence the value "xx" will be higher for applications where large amount of data is transferred.

+= TX_PACKET_POOL_SIZE=14 \

                  RX_PACKET_POOL_SIZE=12

0 Likes

PriyaM_16 wrote:

Please increase the packet pool size to avoid the assert issues.

GLOBAL_DEFINES += TX_PACKET_POOL_SIZE=xx \

  RX_PACKET_POOL_SIZE=xx

The value of packet pools is dependent on the amount of data exchanged. Hence the value "xx" will be higher for applications where large amount of data is transferred.

+= TX_PACKET_POOL_SIZE=14 \

                  RX_PACKET_POOL_SIZE=12

Generally you won't know exactly the amount of data exchanged by every user. You cannot avoid the assert issues this way.

0 Likes

I made simple files.  I have checked on AWS S3.

HTTPS Get

File Name "5M.bin"  the file size is just 5MByte ,  filled with 0xFF.

->  NG 

File Name "5M1.bin"  the file size is 5MByte +1 Byte , filled with 0xFF

-> OK

I don't call  "wiced_tls_init_root_ca_certificates" function in "application_start" function.

0 Likes
lock attach
Attachments are accessible only for community members.

It seems that the remaining data size is not managed properly.

Since the variable is uint16, it seems that it is doing abnormal operation by judging with only the lower 16 bits of the actual file size.

(For example,

5MB is 0x00500000, it seems that it is recognized as 0 byte on the program.

400KB is 0x00064000, so it seems that it is recognized as 16384 bytes on the program. )

Attach the modified file for reference.

43xxx_Wi-Fi\libraries\protocols\HTTP_client\http_client.c

This seems like a cause, but what about it?

I have tried the patch(http_client.c.zip).  and It worked correctly.

But I checked between local server.  It does not work correctly.  

I run "testing snip.httpbin_org-debug download"  by WICED debug.

the system stopped by "wiced_assert"  at tls_host_create_buffer fucntion of wiced_tls.c.

And MBEDTLS_SSL_DEBUG_MSG( 1, ( "unknown record type" ) )  was called in ssl_tls.c.

Is this another issue?