emFile writing is inefficient, any ways to speed it up?

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
PhMa_4608351
Level 2
Level 2
First like received First like given

Hello community,

I'm trying to write to the microSD card as fast as I can. Unfortunately, there seems to be a considerable amount of time between each byte.

Oscilloscope showing the clock emFile.sclk0 clocking out bytes (ignore the upper line).

emFile clock BYTES.jpg

(Apologies for the low quality oscilloscope)

As shown above, the period time between bytes is nearly four times as much as the time it takes to clock out the byte itself. I assumed that emFile is inflexible and a hidden layer of the code, but the amount of time between bytes makes the write time quite long, even though my master clock (MASTER_CLK) is running at 48MHz. Is there any way of increasing the efficiency of writing short of getting a new file system module?

Thanks!

- Phoenix

0 Likes
1 Solution

As I was writing this question to the SEGGER forum, I suddenly realised the cause - which is less technical than it appeared to be.

This graph explains the gaps between the bytes.

IMG_2218.jpg

The top line shows those "clumps" mentioned earlier, and the bottom is a pin I raise high during the sample-timer interrupt.

Since emFile writing is not interrupt-driven, my interrupt that triggers 10000 times a second pauses the writing. I hadn't realised how long is spent in the sample interrupt. Frustratingly, most of that time is spent writing zeros to the ADC so that it can be read (since SPIM is synchronous). Thankfully, sending those zero bytes via DMA makes the interrupt take much less time, and the oscilloscope now looks like this:

IMG_2219.jpg

Much better!

Takeaways:

  • emFile will stop writing for an interrupt of any priority
  • using DMA with the SPI is much faster

Thanks for the replies Len, turns out it was my code that caused the issue - as usual.

- Phoenix

View solution in original post

0 Likes
9 Replies
Len_CONSULTRON
Level 9
Level 9
Beta tester 500 solutions authored 1000 replies posted

Phoenix,

There are a number of factors limiting the aggregate write speed.

Make sure your SD card can support a 48MHz SPI clock.

You can see from the chart below (from the emFile datasheet) that if you write more bytes at a time, the aggregate write speed increases.

If you're only writing one byte at a time, you're not making use of the batch write feature of the SD card.  This is because you are executing a certain amount of PSoC code for each write operation.  If you cache your bytes to RAM first, you minimize the PSoC code executed.  Then when you dump the cached data to emFile, you execute the minimum emFile code to perform the operation.

pastedImage_0.png

Len

Len
"Engineering is an Art. The Art of Compromise."
0 Likes

Hi Len,

Apologies for not being very specific. My CPU speed is 48MHz, while my SPI's is 24MHz, although I have tested it at 12MHz - which made it write even slower. I am also batch writing large buffers, with 512*18 samples (SD cards write in blocks of 512) of 3 bytes per sample. Thus, the buffer size 27648 bytes per buffer, and only half of each buffer is writing while the other half is being written to. Since I am using two buffers, one for each channel of my ADC, 27648 bytes are written to the SD card at once.

Since it usually takes 0.25 seconds to complete this batch write, the write speed is 27648 / 0.25 =  110.592 KBps despite running the SPI at 24MHz and batch writing ~28 KB at a time. To clarify, the faster clock rate and larger buffers that I'm using do in fact make the write time faster according to my tests. Since the write speed is supposed to be 262.56 KBps when the CPU speed is 48MHz, SPI speed is 12MHz, and batchs of 8KB are being written, I should be writing to the SD even faster than that. My optimisation level is also set to "speed".

Edit: fixed miscalculation

Thanks again,

- Phoenix

0 Likes

Phoenix,

Based on the scope plot above,  I gain the following information:

  • Each burst is one byte transferred by SPI.
  • Each burst takes about 350ns.  This means your SPI clock is about 22.857MHz.  This is close to your stated SPI clock.
  • The period of the bursts is 1625ns.  This is a approximate data rate of 615KBps.
  • There is some SPI addressing overhead (yet to be computed).

The published data rate for 8KB based on your Master_Clk and half your SPI clock (12MHz) from the chart is 553KBs.

How are you clocking the 0.25 seconds for a write operation?

Len

Len
"Engineering is an Art. The Art of Compromise."
0 Likes

Len,

Sorry again for not mentioning important information. I wrote a pin high before executing the FS_Write function and wrote it low afterwards. The time occasionally varies but below is the usual time spent in FS_Write, 250ms:

IMG_2214.jpg

This is much slower than what you calculated the data rate to be. I gave you a zoomed in image of two bytes, but unfortunately the bytes aren't always 1625ns, and the bytes come in "clumps".

IMG_2215.jpg

Strangely, zooming out shows that the size of these "clumps" are consistently 13 bytes - I do not know why this may be.

This final image shows the distance between the groups of bytes:

IMG_2216.jpg

(I have since shifted the horizontal position to line us with the time measurements, showing that the clumps are indeed ~75μs apart)

This explains why it takes much longer to write than if each byte was only 1625ns apart. I do not have a clue why these clumps exist, so I'll briefly outline how my code for writing to the SD card works in case that's the reason:

  • Interrupt triggers 10000 times a second, reading a 24 bit number from each of the channels of an ADC, and writing each number to a buffer dedicated to that channel (e.g. channel 2's number is written to sample_buffer2).
  • Each interrupt, it checks if the first or second half of the buffers are full. If one is, set a variable that indicates which half to write in the main loop.
  • In the main loop, call FS_Write for the full half of the buffers, each channel/buffer writes to a different TXT file. Any samples received during this time are stored in the vacant half of the buffers.
  • New TXT files are created after a period of time (1 hour or 36,000,000 samples)
  • Repeat

Thanks for the replies Len,

- Phoenix

0 Likes

Phoenix,

You are thorough.  This might be a question of Seeger.  They are the company that creates the emFile library.  Since you can't peek into the library code (except for assembly), they might be able to give you some answers (or code optimizations).

I've visited their website with a technical question about emFile and received a helpful answer.  What have you got to lose?

Len

Len
"Engineering is an Art. The Art of Compromise."

As I was writing this question to the SEGGER forum, I suddenly realised the cause - which is less technical than it appeared to be.

This graph explains the gaps between the bytes.

IMG_2218.jpg

The top line shows those "clumps" mentioned earlier, and the bottom is a pin I raise high during the sample-timer interrupt.

Since emFile writing is not interrupt-driven, my interrupt that triggers 10000 times a second pauses the writing. I hadn't realised how long is spent in the sample interrupt. Frustratingly, most of that time is spent writing zeros to the ADC so that it can be read (since SPIM is synchronous). Thankfully, sending those zero bytes via DMA makes the interrupt take much less time, and the oscilloscope now looks like this:

IMG_2219.jpg

Much better!

Takeaways:

  • emFile will stop writing for an interrupt of any priority
  • using DMA with the SPI is much faster

Thanks for the replies Len, turns out it was my code that caused the issue - as usual.

- Phoenix

0 Likes

Phoenix,

Excellent!   It appears your troubleshooting skills have been ramped-up a notch!

I find to become a better designer, you have to become a even better troubleshooter.  Who creates flawless designs the first time?  No one.

You did a great job of breaking down the problem into its primary elements and determined the root cause.  This is not always easy when you don't have control over all parts of a system.  I believe your inferences of the emFile component is "spot-on".

Len

Len
"Engineering is an Art. The Art of Compromise."

Len,

I'm glad you think that! It's good to know I'm going about this in the right way, as I am quite new to firmware.

You've also been a big help, and have responded to every thread and comment I've made! You're the reason why forums like these have a nice communal feel to them. I look forward to further learn and enhance my skills with PSoC. Thanks for the replies Len!

Phoenix

0 Likes

Phoenix,

I'm glad I was a help in this matter.  (Although my help was more encouragement since you solved it yourself).

If you're into learning more about advanced firmware, you might want to check out FreeRTOS.  This Real-Time Operating System elevates firmware to the next level with threading (tasks), events, queues, etc.   I believe FreeRTOS lets you compartmentalize your SW better eventually leading to better and more robust SW.

Happy Holidays!

Len

Len
"Engineering is an Art. The Art of Compromise."