NVME performance test

[bls]

Results from my early/quick NVME performance test

Hardware: Pi5, Geekworm X1001, Kingston NV2 500GB SSD
Test:

Code: Select all

# all run as root
rm -f bar.img
echo 1 | tee /proc/sys/vm/drop_caches
time cp foo.img bar.img

Test results:

Code: Select all

Nothing in config.txt: 33.266  0.040   7.153
                       33.282  0.091   7.133
                       33.266  0.043   7.051

pciex1 (only):         33.286  0.048   7.107
                       33.367  0.047   7.080
                       33.321  0.047   7.158

pciex1, pciex1_gen=2   33.425  0.040   7.047
                       33.260  0.043   6.983

pciex1_gen3            18.070  0.046   6.900
                       18.754  0.068   6.819
                       18.478  0.041   6.841

pciex1, pciex1_gen=3   18.077  0.050   6.779
                       18.386  0.059   6.818
                       18.032  0.059   6.847

Conclusions:

• Apparently "pciex1_gen=n" implies "pciex1"
• pciex1_gen=3 is FAST

[bls]

And, for comparison...

NVME system disk, Kingston A400 connected to USB3 Hub:

Code: Select all

44.998  0.028  7.320
44.265  0.053  7.008

NVME system disk, Kingston A400 directly into Pi5 USB3 port:

Code: Select all

47.089  0.053  6.959
45.543  0.053  6.959
44.446  0.066  6.963

Unsurprisingly, tests done when running on a Kingston A400 system disk were similar to the above.

The variability in the last test (Kingston A400 directly into Pi5 USB3 port) is curious

[fanoush]

You could add "sync" call to the time, cp does not wait until everything is written so if there is enough ram you are measuring only reading, not writing. And BTW, how big your foo.img file actually is?

[bensimmo]

It amazing how we're all saying it blaming fast when it is actually amazingly slow (in in PCIE4) compared to any recent era 5/6 years PC or laptop, when NVME M.2. started to be use
Even my laptop is at least twice as fast but then it uses an extra lane to the Pi and these are excruciatingly slow compared to modern NVMe setup with all lanes and PCIE4 or5.

That's in benchmarks, in the real world use you don't really notice that pcie2 to 3 jump in a single lane nor that PCIE3 to 4.
Unless your transferring a clone of one drive to another.

Anyway, I came to say there is a Iong topic on this comparing different boards and settings and the errors people are seeing.

[bls]

You could add "sync" call to the time, cp does not wait until everything is written so if there is enough ram you are measuring only reading, not writing. And BTW, how big your foo.img file actually is?

Yea, this was just a "smoke test", I was trying to get *indicative* data, not scientifically perfect reproducible data. foo.img is unfortunately long gone, but I ISTR it was about 5-6GB.

Sorry for the lack of precision.

[bls]

It amazing how we're all saying it blaming fast when it is actually amazingly slow (in in PCIE4) compared to any recent era 5/6 years PC or laptop, when NVME M.2. started to be use
Even my laptop is at least twice as fast but then it uses an extra lane to the Pi and these are excruciatingly slow compared to modern NVMe setup with all lanes and PCIE4 or5.

That's in benchmarks, in the real world use you don't really notice that pcie2 to 3 jump in a single lane nor that PCIE3 to 4.
Unless your transferring a clone of one drive to another.

Anyway, I came to say there is a Iong topic on this comparing different boards and settings and the errors people are seeing.

Absolutely agree, but the disk I/O performance improvement on the Pi5 is pretty impressive, and it's also quite impressive for a sub-$100 highly flexible computer.

And, it's worth noting that there are other ways to improve performance in this highly contrived benchmark such as using a copy-on-write file system (btrfs), where the copy is instantaneous.

Lastly, disk I/O is a small (but obviously important) piece of the total performance picture.

[bensimmo]

Could you do a similar 6GB transfer but with loads of random small files, that would be more interesting, only as I don't do a lot of 6GB file transfers.

Where to get that number from, cookies after a day or two of tracking us, browser cache files?

By the way, I'm assuming the first column is time, as it get quicker, more than I expected on the switch from USB3 to standard PCIe, assuming drives are OS similar spec. (edit, ah a while generation of age newer being PCIE4, yet both are £35 on Amazon for the ~500GB versions.) bargain for the nvme once you have a adaptor board.

But what are the other two. I know not important, just curious.

[bls]

Could you do a similar 6GB transfer but with loads of random small files, that would be more interesting, only as I don't do a lot of 6GB file transfers.

Where to get that number from, cookies after a day or two of tracking us, browser cache files?

By the way, I'm assuming the first column is time, as it get quicker, more than I expected on the switch from USB3 to standard PCIe, assuming drives are OS similar spec. (edit, ah a while generation of age newer being PCIE4, yet both are £35 on Amazon for the ~500GB versions.) bargain for the nvme once you have a adaptor board.

But what are the other two. I know not important, just curious.

I'll look into doing a bit more testing, as you've suggested, but have a few other TODOs ahead of it

As far as the 3 columns, they are the 3 figures output by the time command, rotated around from vertical to horizontal formatting: real, user, sys, which are (AIUT):
real: wall-clock time
user: user-mode time consumed
sys: system-mode time consumed

From `man bash`: If the time reserved word precedes a pipeline, the elapsed as well as user and system time consumed by its execution are reported when the pipeline terminates.