Help me to benchmark Raspberry Pi 3

[kirankankipati]

Hey guys,

I have developed this new CPU benchmarking system titled: TrueBench

With which I benchmarked my Raspberry Pi2.
I do not have currently Raspberry Pi 3. If you own a Raspberry Pi3, can you please benchmark (as per the instructions mentioned in TrueBench website) and submit me your results please. I need this for research standpoint.

http://truebench.the-toffee-project.org/

Thank you, Kiran

[DNPNWO]

A bit useless as a benchmark as it only utilizes 1 core...

[kirankankipati]

No. It is on purpose benchmark to measure single core. We need single core benchmark for few core apps which cannot be scaled easily.

If you have a Raspberry Pi 3, can you test the same. And report back its performance please.

[kirankankipati]

and when you run this app make sure you do not pollute the results by running any other apps. So that it is as precise as possible.

[DNPNWO]

Fine. But if it only utilizes 25% and a single core, then it is not truly bench marking the performance of the Pi3.

[kirankankipati]

exactly. That depends on the CPU architecture. And how far the hardware is tuned to perform under that specific task. Which is the reason we measure to access.

In most cases we do not need turbo mode to kick in. We do not need special conditions (including RAM overclock).

A pure system benchmarking mechanism so that we can compare directly ARM with Xeons. Per-core vs Per-core basis.
---------
This is very important. Since most apps are not load-shared. Not threaded to be precise. Hence this benchmark system (as we get more and more samples, or examples), we may find very useful in future.

Thank you, Kiran

[kirankankipati]

another example. Suppose if we have a base measurements under standard conditions. Assume by adding a heatsink to RPi or some active cooling via fan/liquid cooling, we can measure fresh benchmarks with and without any modifications.

In many cases measuring multi-thread benchmarks (and multi-core cpu) benchmarks pollutes this analysis.

[davidcoton]

This is very important.

But not important enough for you to buy a Pi3 and test it under exactly the conditions you want?

[kirankankipati]

David. As I mentioned I tested personally with RPi2.
For now I am fine with RPi2. I may skip RPi3 may buy RPi4 sometime next year.
I am very busy with other research meanwhile.

-----------
It is not a question about money/price. I do not want eWaste. I do not like buying something and then tossing it since it is outdated.
Hope you understand.

[DNPNWO]

Code: Select all

pi@raspberrypi:~ $ ./truebench_rpi2
Elapsed Duration:
in Seconds: 1156
in Milliseconds: 1156626
in Microseconds: 1156626169


-------------------------
dummy_result=63

[kirankankipati]

wow. SWEET
Now we can compare it with RPi2. I want to add this in online graph.

""THANKS A LOT""

[DougieLawson]

For now I am fine with RPi2. I may skip RPi3 may buy RPi4 sometime next year.

You have more chance of getting a pink pony and a unicorn. There's no such thing as a RPi4 (or anything beyond RPi3).

Apart from stealing all my data and rendering my RPi as a spam spewing zombie what does your benchmark do? Is it just a cpu soaker loop?

[DougieLawson]

pi@challenger:/tmp $ ./truebench_rpi2
Elapsed Duration:
in Seconds: 1316
in Milliseconds: 1315933
in Microseconds: 1315933666


-------------------------
dummy_result=63
pi@challenger:/tmp $

[DNPNWO]

another example. Suppose if we have a base measurements under standard conditions. Assume by adding a heatsink to RPi or some active cooling via fan/liquid cooling, we can measure fresh benchmarks with and without any modifications.

In many cases measuring multi-thread benchmarks (and multi-core cpu) benchmarks pollutes this analysis.

I have active cooling (heatsink and fan), and I run 1.3 GHz OC. Bench ran at 41C, 25% CPU usage. Ambient 26C.

[kirankankipati]

see Dougle. I do research. I want to benchmark ARM on par with Intel and as well MIPS.
I do not see anything currently which fits into this criteria. I am a kernel/networking systems architect. And I want this result.
---------
I am not doing this to promote CPU sales or some Windows benchmarking app nonsense. What I did is a very lightweight app.

I honor passmark benchmarks (and a fan of it). It is useful for large scale app performance analysis. Such as Database apps, Apache, etc. But I want a precise way to restrict and expand the test for testing single core performance. Lock on to that core and test. It is done on purpose. Trust me.

Anyways thanks a lot for the results

[kirankankipati]

DNPNWO,
Funny. All I am doing is a precise combination of memcpy()s and memset() just trillion operations per test.
May be CPU is adjusting performance based on temperature (due to load on a core), etc.

[DNPNWO]

DNPNWO,
Funny. All I am doing is a precise combination of memcpy()s and memset() just trillion operations per test.
May be CPU is adjusting performance based on temperature (due to load on a core), etc.

My DDR2-800 SDRAM (memory) is also OC'd to 575 MHz (1.15 GHz). My SoC temps never exceeded 41C.

[DougieLawson]

I also ran it on my RPi2B just for comparison

pi@beaufort:/tmp $ ./truebench_rpi2
Elapsed Duration:
in Seconds: 1858
in Milliseconds: 1857701
in Microseconds: 1857701284


-------------------------
dummy_result=63
pi@beaufort:/tmp $

It can't hurt to have more samples in your results set.

[Laurens-wuyts]

pi@raspberrypi:~/truebench $ ./truebench_rpi2
Elapsed Duration:
in Seconds: 1311
in Milliseconds: 1310619
in Microseconds: 1310619137


-------------------------
dummy_result=63
pi@raspberrypi:~/truebench $

[kirankankipati]

Thanks a Ton guys.

I am taking: this for RPi3
by Laurens-wuyts » Fri Sep 02, 2016 1:50 am
pi@raspberrypi:~/truebench $ ./truebench_rpi2
Elapsed Duration:
in Seconds: 1311
in Milliseconds: 1310619
in Microseconds: 1310619137

since it is middle of:
in Microseconds: 1156626169
vs
in Microseconds: 1315933666

and going to add RPi2B
by DougieLawson » Fri Sep 02, 2016 1:50 am
I also ran it on my RPi2B just for comparison

pi@beaufort:/tmp $ ./truebench_rpi2
Elapsed Duration:
in Seconds: 1858
in Milliseconds: 1857701
in Microseconds: 1857701284

Thank you. I am going to populate in the website ASAP.

Have a great day

[kirankankipati]

Done guys

The page is updated: http://truebench.the-toffee-project.org/

Thank you.

[kirankankipati]

so looks like as they quoted RPi3 is almost double the performance of RPi2. Its really impressive.
RPi3 is 1.86 times faster than RPi2 according to TrueBench calculations.

It is interesting to note that Intel is unable to double their core performance year after year. Instead they offer better scalability options by adding more and more cores. That is needed to run many apps at once. But for a given app, the throughput suffers since they are not drastically improving core performance generation after generation.

But it is a great leap in the case of RPi (ARM). I cant wait to test the latest upcoming AMD ZEN processor. And compare it against Intel.

[ejolson]

DNPNWO,
Funny. All I am doing is a precise combination of memcpy()s and memset() just trillion operations per test.
May be CPU is adjusting performance based on temperature (due to load on a core), etc.

I may have missed the link for the source code. Is it there?

If you are using memset and memcpy from the standard library in Raspbian, it is worth mentioning that this library was compiled to be compatible with the ARM6 CPU of the original Pi. Since it sounds like you are testing memory speed rather than processor speed this may or may not make a difference. The default machine architecture used by the compiler in Raspbian is also ARM6 and often 2 to 3 times slower than optimal on a Pi 3.

A sensible benchmark needs to compute and check a verifiable result, otherwise data flow analysis in an optimizing compiler might eliminate some or all of the code you are trying to time. Moreover, a verifiable result is important to ensure that the timing is valid and not the result of a hardware malfunction. Is that what dummy_result=63 means at the end of the run?

I like your website. You have been working on some interesting projects.

[kirankankipati]

Hi ejolson,

The end dummy result is to avoid compiler to do any optimization and skipping the core operation. So the end result 63 ensures that after doing trillions of iterations, memory bytes set at the end is still intact. It denotes the end result of trillions of iterations done successful. But the catch is I need to do dummy print on the screen. This way compiler will not escape and will not do code optimization.

Besides, it is part CPU intensive. Assume you have a network stack, (VPN, IPS, server,) or a database. In most cases it is about movement of memory bytes to and fro along with some amount of CPU computation (I mean the basic cpu number crunching logic). And this is the reason in these applications cpu computational latency becomes a bottleneck. And for the same reasons it is wise to buy a hardware-offload networking device vs. a traditional cpu software bound networking device.

But the Raspberry Pi results show a complete different story. I am confident that in next 2-3 years, a typical raspberry pi sized hardware which is ARM based can compete Intel x86 chips. But it is even more makes sense to cross check with new upcoming AMD ZEN architecture.

I am currently not releasing my source-code. Since it is too simple. The problem in general public (and the programmer community), is that they have a collective conclusion about things. They do and believe things which are very common and mostly practised. They fail to adapt sometimes. And they now start comparing this technique with other methods. Which is something I do not want.

I do not want to test scalability of CPU. I do not want to test its multidimensional aspect. Instead I want to calibrate precisely single dimension performance. And just judge with that. Lets say we have 4GB of RAM vs 8GB vs 16GB of RAM. More CPU cores means better support for more threads and or sometimes more processes (and threads within the same). Which is easy to compute. And this is vital aspect for Server Hardware design. But in desktop computing (especially gaming), they need combination of all. They need best scalability as well best core-wise performance (lets ignore GPU aspect in this case). GPU is useful for better frame-rates(or game rendering), but CPU is needed for better Game AI, overall app performance.

But single dimension tests tells the real story about CPU innovation progress is what I believe

I did recently a video on Threads vs Processes in my channel The Linux Channel. This somewhat explains this logic.
https://www.youtube.com/watch?v=OuXWiqcMvhA

Thank you

[ejolson]

I am currently not releasing my source-code. Since it is too simple.

Fair enough. Last year I compared the speed of the Raspberry Pi to other computers when running an inefficient but simple algorithm to find prime numbers. This test indicates the single core performance of the Pi 3B to be twice the speed of a Pi 2B and about the same as a 3.4Ghz Pentium 4. Note also that using -O3 optimization with the default compiler resulted in a program that was 4.89 times slower then when optimizing for the correct architecture with -O3 -mcpu=cortex-a53. This illustrates a trade-off between performance and compatibility with all models of Pi.