64 Bit Raspberry Pi 3 Benchmarks via SUSE

[RoyLongbottom]

I am converting my Raspberry Pi benchmarks to run via 64 bit SUSE (or openSUSE) for Raspberry Pi 3. I had lots of issues in producing bootable micro SD cards, the final procedure for openSUSE was to select a Leap 42.2 raw.xz version, extract the .raw file on a PC via Linux Ubuntu and burn the card on a Windows based PC using Win32 Disk Imager. Then, I installed GCC-6 to generate appropriate 64 bit code. Example command:

gcc-6 linpack.c cpuidc.c -Wa,-march=armv8-a -lm -lrt -O3 -march=armv8-a -o linpackPi64

The first impression was that measured speeds were exceptionally inconsistent. I tried SUSE Linux Enterprise Server (same installation procedure) and that produced the same variable performance. Examples below are for the Linpack benchmark. The source code is essentially the same as Linpack-pc.c, that I submitted to Netlib 20 years ago. This was specifically designed to produce consistent speeds, unaffected by timer resolution. The program executes the same code ten times, using the same pass count, the second half using slightly different memory addressing. The benchmark has been run without this inconsistency, at 32 bits and 64 bits, where appropriate, on PCs from DOS to Windows 10, various Linux distros, including OpenSuse 11.3, Android devices, and RPis at 32 bits.

Below are the first set of five Raspberry Pi 3 measurements for 32 bit operation via Raspbian Operating System and at 64 bits using SUSE. The first results demonstrate constant MFLOPS performance, as expected, but SUSE speeds appear to be far too variable. I have run my Android version on a Tablet, using a slightly faster Cortex-A53 CPU, with both 32 bit and 64 bit compilations. Resultant average MFLOPS were 178 and 348 respectively.

Code: Select all

  RPi 3 Raspbian 

    dgefa    dgesl    total   Mflops     unit    ratio
  0.00374  0.00012  0.00386   177.81   0.0112    0.069
  0.00374  0.00012  0.00386   177.83   0.0112    0.069
  0.00374  0.00012  0.00386   177.81   0.0112    0.069
  0.00374  0.00012  0.00386   177.81   0.0112    0.069
  0.00374  0.00012  0.00386   177.81   0.0112    0.069
 Average                      177.81

 RPI3 SuSe 

    dgefa    dgesl    total   Mflops     unit    ratio
  0.00340  0.00006  0.00346   198.33   0.0101    0.062
  0.00363  0.00013  0.00376   182.80   0.0109    0.067
  0.00191  0.00006  0.00197   348.54   0.0057    0.035
  0.00186  0.00006  0.00192   357.37   0.0056    0.034
  0.00407  0.00013  0.00420   163.36   0.0122    0.075
 Average                      250.08

In order to see what was happening, I decided to run the benchmark on a selected CPU [taskset 0x00000001 ./linpackPi64], measure CPU utilisation, clock MHz and CPU temperature. Results are shown below. I installed sysstat for CPU utilisation using mpstat and sensors for temperature.

CPU Utilisation - 25 x 1 second samples were produced, with results essentially the same as shown, whilst the benchmark was running, with a constant 100% indicated for the chosen CPU core.

Temperature - This was measured using watch at 1 second intervals, to show changes in a fixed window position. This rose by no more than 4°C from the temperature shown below.

Power Supply - I have a meter that measures power supply voltage and current, indicating 5.14 to 5.15 volts and less than 0.5 amps.

CPU frequency - This varied, switching between 1200 MHz and 600 MHz both when the program was running, no doubt the cause of varying performance. This behaviour was repeated when the CPU was not running benchmarks.

Raspbian - On running the 32 bit benchmark via Raspbian, CPU frequency was constantly 1200 MHz and 600 MHz with nothing running. When the CPU is overheating, CPU MHz is reduced, the frequency being used is identified by the command vcgencmd measure_clock arm. This command is not available via SUSE.

Code: Select all

mpstat -P ALL 1 25 

 CPU    %usr   %nice    %sys %iowait   %idle
 all   26.38    0.00    0.25    0.00   73.37
   0  100.00    0.00    0.00    0.00    0.00
   1    0.00    0.00    1.00    0.00   99.00
   2    0.00    0.00    0.00    0.00  100.00
   3    4.95    0.00    0.00    0.00   95.05
 --------------------------------------------------------

 sensors - watch -n 1 sensors

 bcm2835_thermal-virtual-0
 Adapter: Virtual device
 temp1:        +47.2°C  

 --------------------------------------------------------

 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq

 1200000   Each 600000 sometimes
 1200000
 1200000
 1200000

Stumbleupon Solution

On experimenting with watch sampling interval, I changed the command to, for example, watch -n 0.1 sensors, when repeated runs showed consistent performance, with occasional minor variation. Below shows results of the ten sets of calculations.

I will upload the 64 bit benchmarks in due course, obtainable (free with no ads) via my web site in the Raspberry Pi pages:

http://www.roylongbottom.org.uk/

Is there anything that I have missed?

Code: Select all

 RPI3 SuSe with watch -n 0.1 sensors

 Times for array with leading dimension of 201

    dgefa    dgesl    total   Mflops     unit    ratio
  0.00193  0.00006  0.00199   345.48   0.0058   0.0355
  0.00192  0.00006  0.00198   346.67   0.0058   0.0354
  0.00192  0.00006  0.00198   346.40   0.0058   0.0354
  0.00192  0.00006  0.00198   346.57   0.0058   0.0354
  0.00192  0.00006  0.00198   346.51   0.0058   0.0354
 Average                      346.30

 Times for array with leading dimension of 200

    dgefa    dgesl    total   Mflops     unit    ratio
  0.00180  0.00006  0.00186   369.72   0.0054   0.0332
  0.00179  0.00006  0.00186   370.15   0.0054   0.0331
  0.00179  0.00006  0.00186   370.09   0.0054   0.0331
  0.00179  0.00006  0.00185   370.78   0.0054   0.0331
  0.00179  0.00006  0.00186   370.07   0.0054   0.0331
 Average                      370.16

[dom]

CPU frequency - This varied, switching between 1200 MHz and 600 MHz both when the program was running, no doubt the cause of varying performance. This behaviour was repeated when the CPU was not running benchmarks.

The 64-bit kernel isn't widely tested. There could be an issue with the cpufreq governor which determines when to switch into turbo mode.
To rule that out it may be worth adding "force_turbo=1" and testing both 32 and 64-bit kernels.

(a correctly functioning cpufreq driver should be running at 1200MHz when any core is more than 50% busy which should be the case for running benchmarks).

[pi-anazazi]

Hi!

Maybe a suggestion:

- Try this image

http://download.opensuse.org/repositori ... i3/images/

- Simply copy to SD card with

Code: Select all

xzcat [image].raw.xz | dd bs=4M of=/dev/sdX iflag=fullblock oflag=direct; sync

as described here

https://en.opensuse.org/HCL:Raspberry_P ... pstream.29

- After the initial repartitioning (wait for SOME time. Really be patient) you should have a GUI. User root, password linux.

- Do a reboot, do

Code: Select all

sudo zypper -dup --no-allow-vendor-change

and reboot, repeat as long as the latest kernel is installed (4.4.39-5.1 64 bit here)

Maybe have a better experience. This image is doing fine here for some months now...

[RoyLongbottom]

CPU frequency - This varied, switching between 1200 MHz and 600 MHz both when the program was running, no doubt the cause of varying performance. This behaviour was repeated when the CPU was not running benchmarks.

The 64-bit kernel isn't widely tested. There could be an issue with the cpufreq governor which determines when to switch into turbo mode.
To rule that out it may be worth adding "force_turbo=1" and testing both 32 and 64-bit kernels.

(a correctly functioning cpufreq driver should be running at 1200MHz when any core is more than 50% busy which should be the case for running benchmarks).

Thanks Dom, config.txt had force_turbo=0. Now the benchmark runs continuously at full speed with force_turbo=1 but measured MHz appears to still run at `1200 when the CPU is idle. Below is config.txt. Are other changes required?

I wonder what will happen when I run MP stress tests.

Code: Select all

# Get more options/information on http://elinux.org/RPiconfig
# or on https://www.raspberrypi.org/documentation/configuration/config-txt.md
# Our kernels are located on a Linux partition. Chainload U-Boot to load them.
kernel=u-boot.bin
# Use 64 MB for GPU for RPi with 256 MB (Min 16 - Max 192 MB)
gpu_mem_256=64
# Use 64 MB for GPU for RPi with 512 MB (Min 16 - Max 448 MB)
gpu_mem_512=64
# Use 128 MB for GPU for RPi with 1024 MB (Min 16 - Max 944 MB)
gpu_mem_1024=128
# Turbo mode: 0 = enable dynamic freq/voltage - 1 = always max
force_turbo=1
# Start in turbo mode for 30 seconds or until cpufreq sets a frequency
initial_turbo=30
# DO NOT overvoltage manually to not void warranty!
over_voltage=0
# Boot in AArch64 (64-bit) mode
arm_control=0x200
# Fix mini UART input frequency, and setup/enable up the UART.
enable_uart=1
# Disable warning overlays as they don't work well together with linux's graphical output
avoid_warnings=1

[RoyLongbottom]

I tried to copy Tumbleweed with

xzcat [image].raw.xz | dd bs=4M of=/dev/sdX iflag=fullblock oflag=direct; sync

but the SD card would not boot. I think that it might be due to using Ubuntu on my PC with UEFI. Using Gparted, the boot partition was FAT32 (or Ext4 when initially formatted as that), instead of FAT16. Then my method, of extracting via Ubuntu and copying via Win32 Disk Imager, did not produce an image that would boot.

Anyway, Leap 42 will meet my needs

[dom]

Thanks Dom, config.txt had force_turbo=0. Now the benchmark runs continuously at full speed with force_turbo=1 but measured MHz appears to still run at `1200 when the CPU is idle.

This is expected. force_turbo=1 means force turbo mode all the time. It's a workaround for the behaviour you have described rather than a final solution.

[jahboater]

Thanks Dom, config.txt had force_turbo=0. Now the benchmark runs continuously at full speed with force_turbo=1 but measured MHz appears to still run at `1200 when the CPU is idle.

This is expected. force_turbo=1 means force turbo mode all the time. It's a workaround for the behaviour you have described rather than a final solution.

It should be fine for benchmarks though.
You don't want the slight lag when the clock speed ramps up.

I don't know how it detects the "50% busy" but it must involve a delay.
(its probably 100% busy for 50% of the time in the last interval).

[dom]

I don't know how it detects the "50% busy" but it must involve a delay.
(its probably 100% busy for 50% of the time in the last interval).

Yes, there will be some lag in detecting being busy (in the tens of milliseconds range).
It won't have much effect on a benchmark that takes many seconds to complete but would be bad for a sub-second duration benchmark (although you could run it 100 times to avoid that).

[RoyLongbottom]

I have compiled my RPi Floating point and integer CPU stress tests, at 64 bits for OpenSUSE, and run them on my RPi 3. The purpose was to see what happens when the system is running with the CPU frequency governor settings of performance and on-demand, bearing in mind that the first converted single core benchmarks demonstrated unexpected slow performance with the latter setting. Details of the tests are in the following and benchmarks, with source code, in the tar.gz file.

http://www.roylongbottom.org.uk/SUSE%20 ... 0Tests.htm
http://www.roylongbottom.org.uk/SuseRpi3Stress.tar.gz

The first tests were with the RPi 3 fitted in a FLIRC case, where 15 minute 32 bit tests, under Raspbian, showed no degradation in performance, with limited increase in CPU temperature. The next ones had a copper heat sink on the CPU and no case, where the original tests demonstrated CPU MHz throttling and slower performance, with increases in temperature.

A summary of OpenSUSE results are below, showing average speeds per core for multi-core tests, with floating point test results in MFLOPS and integer tests in MB/second.

Single vs Multi Core - With performance setting, up to 10% per core degradation could be expected. With on demand, even an overheated MP core can be faster than a cold single core.

Performance vs On Demand - Multi core performance is essentially the same but single core OD speeds can be too slow.

FLIRC case - Note constant performance over 15 minutes MP tests.

Copper Heatsink - CPU speed throttling kicks in at over 80°C CPU temperature, recorded as on-going small reductions in MHz, and slower measured performance.

Code: Select all

                         Per Core Average Speeds

                       On Demand      Performance     OD/Perf
                       MFLOPS MB/sec  MFLOPS MB/sec   MFL MB/s

 Single core 1 pass      1996   2278    3832   2768   52%  82%

 FLIRC Case
 4 cores first 2 passes  3623   2465    3644   2528   99%  98%
 4 cores last  2 passes  3591   2476    3645   2618   99%  95%

 Copper Heatsink
 4 cores first 2 passes  3603   2485    3463   2477  104% 100%
 4 cores last  2 passes  3152   2017    3104   1975  102% 102%

The above htm report includes 8 graphs, with 2 represented below, indicating variable recordings with on demand setting and fairly constant speeds using the performance option, with limited increases in CPU temperature.

HotSUSEPi.gif (7.02 KiB) Viewed 31391 times

[RoyLongbottom]

The initial 32 bit benchmarks have been compiled to run at 64 bits and results are reported in my Raspberry Pi Benchmarks thread:

viewtopic.php?p=1110102#p1110102

The initial benchmarks were Whetstone, Dhrystone, Linpack and Livermore Loops, with performance gains of up to 40%.

[dukla2000]

I tried to copy Tumbleweed with

xzcat [image].raw.xz | dd bs=4M of=/dev/sdX iflag=fullblock oflag=direct; sync

but the SD card would not boot. I think that it might be due to using Ubuntu on my PC with UEFI. Using Gparted, the boot partition was FAT32 (or Ext4 when initially formatted as that), instead of FAT16. Then my method, of extracting via Ubuntu and copying via Win32 Disk Imager, did not produce an image that would boot.

Anyway, Leap 42 will meet my needs

I have had a couple of problems getting the SuSE images to boot. In short I really suspect the xzcat | dd mechanism is flawed as I have tried on multiple sdcards and multiple different images from download.opensuse! Which sounds ludicrous but ... I have a couple of times been able to get an sdcard to work by running fsck on all partitions (and correcting the errors!). And once I got an image to work the second time I burned it. I do all my downloads and SDcard burning on my Pi3, and have always first checked the sha256sum is OK!

I installed sysstat for CPU utilisation using mpstat and sensors for temperature.

Curious which SuSE you are running? I am currently running openSUSE-Tumbleweed-ARM-XFCE-raspberrypi3.aarch64-2017.01.31-Build1.1.raw.xz and sensors-detect found nothing. Similarly for MHz I have no cpufreq structure under /sys/devices/system/cpu/cpu*

[EDITS] Changed fdisk to fsck. Also re-reading thread notice you are LEAP 42.2 - not sure why that has CPU governors and I don't but ... [/EDITS]

[pi-anazazi]

re problems burning images: Most likely cause is the USB-card reader used, had problems with one of those recently, others work just fine.

[RoyLongbottom]

I
Anyway, Leap 42 will meet my needs

I have had a couple of problems getting the SuSE images to boot. In short I really suspect the xzcat | dd mechanism is flawed as I have tried on multiple sdcards and multiple different images from download.opensuse! Which sounds ludicrous but ... I have a couple of times been able to get an sdcard to work by running fsck on all partitions (and correcting the errors!). And once I got an image to work the second time I burned it. I do all my downloads and SDcard burning on my Pi3, and have always first checked the sha256sum is OK!

I installed sysstat for CPU utilisation using mpstat and sensors for temperature.

Curious which SuSE you are running? I am currently running openSUSE-Tumbleweed-ARM-XFCE-raspberrypi3.aarch64-2017.01.31-Build1.1.raw.xz and sensors-detect found nothing. Similarly for MHz I have no cpufreq structure under /sys/devices/system/cpu/cpu*

[EDITS] Changed fdisk to fsck. Also re-reading thread notice you are LEAP 42.2 - not sure why that has CPU governors and I don't but ... [/EDITS]

Besides OpenSUSE, I have SUSE Linux Enterprise Server, both LEAP 42.2. SUSE, Raspbian and Gentoo, that I am also playing with, all have different methods of determining CPU temperature and actual CPU MHz that controls throttling. It seems that the OS can interpret hardware and save results in one of those areas or provide a function for direct interpretation.

With SLES, you have to register it in order to install new apps or updates. In running my stress tests, a system crash made the SD unbootable. I reinstalled the image, requiring registration again, but the old number was not recognised. I can still use it to compile and run my benchmarks (so far but Java and OpenGL might need more SW???).

[ktb]

I very well could be wrong, but isn't the cpufreq driver missing upstream? I thought it was missing based on my playing around with kraxel.org aarch64 Fedora images.

[RoyLongbottom]

I very well could be wrong, but isn't the cpufreq driver missing upstream? I thought it was missing based on my playing around with kraxel.org aarch64 Fedora images.

I don’t know whether the cpufreq driver is missing, but SUSE frequency measurement is different to Raspbian and Linux Gentoo. For the latter, there is a specific ARM frequency, obtained via command vcgencmd measure_clock arm. With SUSE, MHz is measured using:

cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq

As indicated in the above graphs, this shows the varying frequency used for controlling throttling, but only shows 600 or 1200 MHz for the other two.

With all three, default frequency control on is “on demand” where, idle MHz is 600, switching to 1200 MHz when processing (with no throttling). The SUSE EFI boot partition config.txt file has a parameter to set “ondemand” or “performance”, where the latter provides a continuous 1200 MHz. I have not found the equivalent for the other two systems, but did not need it.

The difference with SUSE “on demand” was the it lead to inconsistent benchmark performance, overcome by switching to “performance” mode or running another program in a different CPU core. This different behaviour was quite clear until I compiled and ran my Fast Fourier Transform benchmarks - see:

http://www.roylongbottom.org.uk/Raspber ... m#anchor27

This measures execution time of FFTs sized 1K to 1024K, where the shortest take less than 0.5 milliseconds. With these short runs, all systems could produce inconsistent speeds. To investigate this, I produced another test that executes 30 1K sized FFTs 500 times. Results in the report shows that consistent speeds were produced on all systems with “performance” setting, or running a background program.

A summary of measured “on demand” execution times is below. Raspbian and Gentoo performance could be slow over the first 3 or 4 measurements (90 to 120 FFTs), then consistent speeds afterwards. SUSE indicated similar early performance but, at random intervals, could run at half speed for a relatively long time.

Code: Select all

       RPi 3 500 x 30 1K Single Precision FFT milliseconds
 
                   32 Bit Raspbian On Demand

  12.9  12.2   7.4   6.0   6.0   6.4   6.0   6.0   6.0   6.0
   6.1   6.0   6.0   6.0   6.0   6.0   6.1   6.1   6.0   6.2
   6.2   6.0   6.0   6.1   6.0   6.0   6.0   6.0   6.1   6.0
   6.2   6.0   6.0   7.0   6.1   6.0   6.0   6.0   6.1   6.0
   6.2   6.1   6.0   6.0   6.2   6.0   6.0   6.0   6.0   7.2
 To
   6.5   6.3   6.1   6.2   6.1   6.1   6.1   6.1   6.1   6.1
   6.5   6.3   6.1   6.1   6.1   6.1   6.1   6.1   6.1   6.1
   6.4   6.2   6.1   6.1   6.2   6.1   6.1   6.1   6.1   6.1

                   64 Bit Gentoo On Demand

  17.5  15.4  11.8   8.6   5.4   5.4   5.4   5.4   5.4   5.4
   5.5   5.8   6.0   5.4   5.5   5.4   5.5   5.4   5.4   5.4
   5.5   5.6   6.1   5.4   5.5   5.4   5.5   5.5   5.4   5.4
To
   5.7   6.9   5.7   5.4   5.4   5.4   5.5   5.4   5.4   5.4
   5.8   6.8   5.8   5.6   5.4   5.4   5.4   5.5   5.4   5.4
   5.7   6.4   5.7   5.5   5.4   5.4   5.5   5.4   5.4   5.4

                 64 Bit OpenSUSE On Demand

  12.1  12.5   8.9   5.3   5.3   5.3   5.3   5.3   5.3   5.3
   5.3   5.7   5.3   5.3   5.3   5.3   5.3   5.3   5.3   5.3
   5.3   5.6   5.3   5.3   5.3   5.3   5.3   5.3   5.3   5.3
 To
   7.9  11.7  10.7  10.6  10.6  10.6  10.6  10.6  10.6  10.7
  11.6  11.2  10.6  10.7  10.6  10.6  10.6  10.6  10.6  10.6
  11.7  11.5  10.6  10.6  10.6  10.6  10.6  10.6  10.6  10.6
  11.8  11.1  10.6  10.6  10.7  10.6  10.6  10.7  10.6  10.6
 To
   5.5   6.0   5.8   5.3   5.3   5.3   5.3   5.3   5.3   5.3
   5.5   5.9   5.7   5.3   5.3   5.3   5.3   5.3   5.3   5.3
   5.5   6.0   5.8   5.3   5.3   5.3   5.3   5.3   5.3   5.4