CYW43438: Pi 3B - Wifi stops transmitting every minute during iperf test

[MiKi_4159711]

iperf3 tests are run with UDP between 2 Pi 3B boards over the on-board WiFi ( channel 6 = 2.437 GHz)

Server Pi is set up as Access Point (hostapd/dnsmasq) and client Pi is attached.

Below is an example where iperf3 has zero data transfer for a full 1 second period (32-33 seconds).

Just prior there is high jitter with packet loss and after the event as well.

Test setup is running in a RF screen room with no interfering WiFi devices.

RF spectrum analyzer (2.437 GHz) shows complete loss of WiFi RF signal during this event - so Pi is not transmitting and nothing else is interfering.

Bluetooth is disabled on both Pi's.  Part is this chip:  https://www.cypress.com/file/298076/download

Does the CYW43438 periodically run calibration and shut down WiFi for 2 -4 seconds every minute?

Are there any kernel tasks in Raspbian that would shut down WiFi?  Can a log file be inspected to determine why WiFi stopped?

iperf3 -c 192.168.0.1 -t 60 -f m -u -b 50M
  <<< ******* SKIP TO MIDPOINT OF IPERF3 TEST ************* >>>>
[ 5] 26.00-27.00 sec 5.20 MBytes 43.6 Mbits/sec 1.020 ms 0/666 (0%) 
[ 5] 27.00-28.00 sec 5.02 MBytes 42.1 Mbits/sec 1.105 ms 0/643 (0%) 
[ 5] 28.00-29.00 sec 2.88 MBytes 24.1 Mbits/sec 0.955 ms 0/368 (0%) <<< WHY FEWER DATAGRAMS HERE?
[ 5] 29.00-30.00 sec 5.13 MBytes 43.1 Mbits/sec 1.166 ms 0/657 (0%) 
[ 5] 30.00-31.00 sec 5.23 MBytes 43.8 Mbits/sec 0.944 ms 0/669 (0%) 
[ 5] 31.00-32.00 sec 144 KBytes 1.18 Mbits/sec 20.133 ms 26/44 (59%) <<< HIGH JITTER, PACKET LOSS
[ 5] 32.00-33.00 sec 0.00 Bytes 0.00 Mbits/sec 20.133 ms 0/0 (0%) <<< COMPLETE LOSS - NO DATA TRANSFER
[ 5] 33.00-34.00 sec 136 KBytes 1.12 Mbits/sec 156.233 ms 11/28 (39%) <<< RECOVERY, HIGH JITTER, LOW PACKET TRANSFER
[ 5] 34.00-35.00 sec 2.15 MBytes 18.0 Mbits/sec 1.175 ms 0/275 (0%) << BACK TO NORMAL
[ 5] 35.00-36.00 sec 5.16 MBytes 43.3 Mbits/sec 0.981 ms 0/660 (0%) 
[ 5] 36.00-37.00 sec 2.78 MBytes 23.3 Mbits/sec 6.918 ms 1/357 (0.28%) 
[ 5] 37.00-38.00 sec 5.00 MBytes 41.9 Mbits/sec 1.020 ms 0/640 (0%) 
[ 5] 38.00-39.00 sec 5.23 MBytes 43.9 Mbits/sec 1.090 ms 0/670 (0%) 
[ 5] 39.00-40.00 sec 5.24 MBytes 44.0 Mbits/sec 1.171 ms 0/671 (0%) 
<<<< ************ CUT HERE TO END OF TEST ************* >>>>
[ 5] 59.00-60.00 sec 2.67 MBytes 22.4 Mbits/sec 6.552 ms 0/342 (0%) <<<<<< END OF TEST
[ 5] 60.00-60.07 sec 384 KBytes 43.7 Mbits/sec 1.505 ms 0/48 (0%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-60.07 sec 0.00 Bytes 0.00 Mbits/sec 1.505 ms 39/34519 (0.11%) << THE 32-33s INTERVAL ABOVE IS NOT INCLUDED HERE.

Can some parameter be changed in iperf3 (buffer sizing?) or in the WiFi configuration to avoid loss?

Or is there a known issue in the WiFi chipset or driver that could cause this loss?

It appears that iperf3 just hangs during this event - the stats do not reflect the gap, only the lost datagrams before and after.

Similar event is seen with WiFi TCP test - but only with Pi AP as client and attached Pi as server.

Reversing roles only has a few lower datagram rate events (loss), but never complete dropout.

Running the same test with a wired 100Mbps Ethernet connection yields 0% datagram loss (rate is 93.2 Mbps).

Any insights appreciated.

[MiKi_4159711]

Part 2:    Switched to nuttcp tool (similar to iperf) - seeing WiFi loss every 60 seconds over 4 minute test (repeatable) with TCP packets.

Not seeing any TCP retries - so link is good.

DETAILS:

I am getting a periodic (1 minute) dropout of WiFi signal (no RF on spectrum analyzer) using the nuttcp tool.

I can't understand what would shutdown WiFi, the onboard Bluetooth is disabled. Log file is below.

One Pi 3B is set up as Access Point (hostapd/dnsmasq) and other Pi3B is attached.

Attached Pi is the nuttcp server and AP Pi is the client.  Have tried 3 different Pi's as server and as client. Same result.

If I use my laptop as client, same result. - periodic 60 second WiFi dropout.

Does WiFi have to stop and do periodic calibration? Could WiFi be going in to sleep mode?

If i reverse roles, AP as server and attached Pi as client, I do not see these dropouts.

There are cases where data rate drops down to 10-12 Mbps, but never zero with no RF signal.

Any insights appreciated.

CLIENT COMMAND:   sudo nice -n -20 nuttcp -tvv -i 1 -frunningtotal -T240 -R20m 192.168.0.74

SERVER COMMAND:   sudo nice -n -20 nuttcp -Sv

2.3750 MB / 1.00 sec = 19.9230 Mbps 0 retrans Tot: 76.2500 MB / 32.00 sec = 19.9885 Mbps 0 retrans
  0.6250 MB / 1.00 sec = 5.2428 Mbps 0 retrans Tot: 76.8750 MB / 33.00 sec = 19.5416 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 76.8750 MB / 34.00 sec = 18.9669 Mbps 0 retrans 
  *************** First loss – 34 seconds into test ( No RF)
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 76.8750 MB / 35.00 sec = 18.4250 Mbps 0 retrans
  1.0000 MB / 1.00 sec = 8.3887 Mbps 0 retrans Tot: 77.8750 MB / 36.00 sec = 18.1462 Mbps 0 retrans
  4.9375 MB / 1.00 sec = 41.4178 Mbps 0 retrans Tot: 82.8125 MB / 37.00 sec = 18.7751 Mbps 0 retrans 
  ****************** Higher data rates here to maintain 20 Mbps average.
  4.8125 MB / 1.00 sec = 40.3709 Mbps 0 retrans Tot: 87.6250 MB / 38.00 sec = 19.3434 Mbps 0 retrans
  4.7500 MB / 1.00 sec = 39.8459 Mbps 0 retrans Tot: 92.3750 MB / 39.00 sec = 19.8691 Mbps 0 retrans
=========== skip forward
  2.3750 MB / 1.00 sec = 19.9202 Mbps 0 retrans Tot: 219.2500 MB / 92.00 sec = 19.9913 Mbps 0 retrans
  0.6250 MB / 1.00 sec = 5.2436 Mbps 0 retrans Tot: 219.8750 MB / 93.00 sec = 19.8327 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 219.8750 MB / 94.00 sec = 19.6217 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 219.8750 MB / 95.00 sec = 19.4152 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 219.8750 MB / 96.00 sec = 19.2130 Mbps 0 retrans 
  ********************* 60 seconds after first loss – another loss ( No RF)
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 219.8750 MB / 97.00 sec = 19.0149 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 219.8750 MB / 98.00 sec = 18.8209 Mbps 0 retrans
  3.6250 MB / 1.00 sec = 30.4100 Mbps 0 retrans Tot: 223.5000 MB / 99.00 sec = 18.9379 Mbps 0 retrans
  4.8750 MB / 1.00 sec = 40.8921 Mbps 0 retrans Tot: 228.3750 MB / 100.00 sec = 19.1575 Mbps 0 retrans
============= skip forward
  2.3750 MB / 1.00 sec = 19.9234 Mbps 0 retrans Tot: 359.9375 MB / 151.00 sec = 19.9959 Mbps 0 retrans
  2.3750 MB / 1.00 sec = 19.9228 Mbps 0 retrans Tot: 362.3125 MB / 152.00 sec = 19.9954 Mbps 0 retrans
  0.6250 MB / 1.00 sec = 5.2429 Mbps 0 retrans Tot: 362.9375 MB / 153.00 sec = 19.8990 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 362.9375 MB / 154.00 sec = 19.7697 Mbps 0 retrans 
  ******************** 120 seconds later – another loss ( No RF)
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 362.9375 MB / 155.00 sec = 19.6422 Mbps 0 retrans
  1.0625 MB / 1.00 sec = 8.9130 Mbps 0 retrans Tot: 364.0000 MB / 156.00 sec = 19.5734 Mbps 0 retrans
  4.8125 MB / 1.00 sec = 40.3708 Mbps 0 retrans Tot: 368.8125 MB / 157.00 sec = 19.7059 Mbps 0 retrans
============= skip forward
  2.3750 MB / 1.00 sec = 19.9230 Mbps 0 retrans Tot: 505.3750 MB / 212.00 sec = 19.9971 Mbps 0 retrans
  0.6250 MB / 1.00 sec = 5.2429 Mbps 0 retrans Tot: 506.0000 MB / 213.00 sec = 19.9279 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 506.0000 MB / 214.00 sec = 19.8347 Mbps 0 retrans
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 506.0000 MB / 215.00 sec = 19.7425 Mbps 0 retrans
  *********************** 180 seconds later - another loss ( No RF)
  0.0000 MB / 1.00 sec = 0.0000 Mbps 0 retrans Tot: 506.0000 MB / 216.00 sec = 19.6511 Mbps 0 retrans
  4.7500 MB / 1.00 sec = 39.8438 Mbps 0 retrans Tot: 510.7500 MB / 217.00 sec = 19.7441 Mbps 0 retrans
  4.8750 MB / 1.00 sec = 40.8977 Mbps 0 retrans Tot: 515.6250 MB / 218.00 sec = 19.8412 Mbps 0 retrans
 
  SUMMARY
572.2500 MB / 240.04 sec = 19.9986 Mbps 83 %TX 4 %RX 0 retrans 1.97 msRTT

[MiKi_4159711]

From the datasheet, this is Calibration section

CalibrationThe CYW43438 radio transceiver features an automated calibration scheme that is self contained in the radio. No user interaction is required during normal operation or during manufacturing to optimize performance. Calibration optimizes the performance of all the major blocks within the radio to within 2% of optimal conditions, including filter gain and phase characteristics, matching between key components, and key gain blocks. This takes into account process variation and temperature variation. Calibration occurs transpar-ently during normal operation during the settling time of the hops and calibrates for temperature variations as the device cools and heats during normal operation in its environment

How often does this run?   Would it block pending WiFi traffic for 2-4 seconds?

[Zhengbao_Zhang · Accepted solution]

Hello:

  sorry , I didn't find any info about the frequency for the calibration period.

but from previous throughput test experience, it will not influence the value to 0 for a 2-4 second shot.

So my suggestions are:

1.  use one PI board to do Throughput test with another PC which confirmed T/P is ok.

    test TX/RX direction.  to see if still have problems.

2.  try to find clean place for the test.

3.  if possible , try a conductive test in different channel like 1, 6, 11.

4.  try to find evb to use the same firmware.

5. to check if some T/P related defines are consistent with the release.