jdonald
Posts: 449
Joined: Fri Nov 03, 2017 4:36 pm

Re: Why moving to 64bit?

Mon Aug 19, 2019 3:03 am

Here are some decompression benchmarks. While programs like gzip or lzma don't seem to have a standard built-in test like openssl, I get pretty consistent results with time unlzma -c filename.lzma > /dev/null.

Raspbian (ARMv6):

Code: Select all

pi@raspberrypi:~ $ time unlzma -c ~/70_mb_image.lzma -c > /dev/null

real	0m11.360s
user	0m11.276s
sys	0m0.084s

Debian armhf chroot (ARMv7 maximum compatibility):

Code: Select all

(pi32)pi@raspberrypi:~ $ time unlzma -c ~/70_mb_image.lzma -c > /dev/null

real	0m11.115s
user	0m11.037s
sys	0m0.076s

Debian armhf chroot Cortex-A72 tuned (ARMv7 -march=armv8-a+crc+simd -mtune=cortex-a72 -mfpu=neon-fp-armv8):

Code: Select all

(pi32)pi@raspberrypi:~/xz-utils-5.2.4/debian/normal-build $ time LD_LIBRARY_PATH=src/liblzma/.libs unlzma -c ~/70_mb_image.lzma -c > /dev/null

real	0m10.451s
user	0m10.401s
sys	0m0.048s

Debian arm64 chroot:

Code: Select all

(pi64)pi@raspberrypi:~ time unlzma -c ~/70_mb_image.lzma -c > /dev/null

real	0m9.451s
user	0m9.330s
sys	0m0.120s

Debian arm64 chroot Cortex-A72 tuned (ARMv8 -march=armv8-a+crc -mtune=cortex-a72):

Code: Select all

(pi64)pi64@raspberrypi:~/xz-utils-5.2.4/debian/normal-build $ time LD_LIBRARY_PATH=src/liblzma/.libs unlzma -c ~pi/70_mb_image.lzma -c > /dev/null

real	0m9.348s
user	0m9.232s
sys	0m0.116s
So for LZMA decompression, going to ARMv7 alone improves performance by only 2%, Cortex-A72 tuning (still 32-bit) bumps that up to 9%, 64-bit out-of-the-box gets 19%, 64-bit tuned for the Pi 4 reaches 21%. (All speedup percentages relative to the ARMv6 baseline.) It doesn't seem to matter whether the test archive is 70 MB or a gigabyte.

This is in line with jerrm's earlier result where going from the Raspbian userland to 64-bit Debian took a zbackup restore from 103 minutes down to 91 minutes (13% speedup).

ejolson
Posts: 8897
Joined: Tue Mar 18, 2014 11:47 am

Re: Why moving to 64bit?

Thu Aug 22, 2019 1:25 am

Here are the openssl results for a Pi 4B running this 64-bit Gentoo image.

Code: Select all

$ openssl speed -evp aes-256-gcm
Doing aes-256-gcm for 3s on 16 size blocks: 6665953 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 64 size blocks: 1789838 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 456301 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 114937 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 14443 aes-256-gcm's in 3.00s
OpenSSL 1.0.2s  28 May 2019
built on: reproducible build, date unspecified
options:bn(64,64) rc4(ptr,char) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(ptr) 
compiler: aarch64-unknown-linux-gnu-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -Wall -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -march=armv8-a+crc -mtune=cortex-a72 -O2 -pipe -fno-strict-aliasing -Wa,--noexecstack
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-gcm      35551.75k    38183.21k    38937.69k    39231.83k    39439.02k
I've added this result to the previous table and obtained

Code: Select all

ARM1176JZF-S 32-bit 700MHz noAES (Pi B+ Raspbian):
aes-256-gcm       5283.77k     6570.91k     6961.58k     7102.24k     7099.73k
Cortex-A53 32-bit 1.4GHz noAES (Pi 3B+ Raspbian):
aes-256-gcm      19046.13k    22753.49k    24055.64k    24405.67k    24510.46k
Cortex-A72 32-bit 1.5GHz noAES (Pi 4B Raspbian):
aes-256-gcm      27712.91k    30615.40k    31924.65k    32312.51k    32328.36k
Cortex-A72 64-bit 1.5GHz noAES (Pi 4B Gentoo):
aes-256-gcm      35551.75k    38183.21k    38937.69k    39231.83k    39439.02k
Cortex-A53 64-bit 1.4GHz AES (NanoPi T3 Ubuntu):
aes-256-gcm      85789.11k   216272.58k   363160.92k   449810.09k   480291.50k
Cortex-A57 64-bit 2GHz AES (Jetson TX2 Ubuntu):
aes-256-gcm     182532.45k   451431.66k   628317.10k   733904.90k   767257.26k
Denver-2 64-bit 1.8Ghz AES (Jetson TX2 Ubuntu):
aes-256-gcm     225900.97k   518809.43k   820897.28k   982568.28k  1050839.72k

jdonald
Posts: 449
Joined: Fri Nov 03, 2017 4:36 pm

Re: Why moving to 64bit?

Thu Aug 22, 2019 4:50 am

This result running 5% faster than Debian arm64 could be due to any of the following differences:
* -march=armv8-a+crc -mtune=cortex-a72
* gcc 9.1 (vs gcc 8.3)
* openssl 1.0.2s (vs 1.1.1c)

On Gentoo I no longer see any performance difference with OPENSSL_armcap=0. It appears the bug with ARMV7_NEON mysteriously causing a slowdown is not present in this version.

Of course it's still rather crippled by lacking a -DAES_ASM implementation, much slower than the Debian armhf configuration (gcc 8.3 and no -mtune).

Cob
Posts: 28
Joined: Tue Mar 05, 2013 2:03 am

Re: Why moving to 64bit?

Thu Aug 22, 2019 8:50 am

Manjaro 19.08 is also now available for the Rpi4 in 64bit, both in desktop and minimal flavours.

https://forum.manjaro.org/t/manjaro-arm ... ased/99031


Being based on Arch linux it's quite a lot easier to work with than Gentoo.

pica200
Posts: 281
Joined: Tue Aug 06, 2019 10:27 am

Re: Why moving to 64bit?

Thu Aug 22, 2019 10:19 am

This is what i have been using for the past week.And believe it or not Firefox was already working better without GPU driver than ESR on Raspian. And on top of that now that the GPU driver landed in the kernel (need to rename fbturbo config) Firefox is even working with acceleration force enabled while doing the same on Raspian with ESR crashed all tabs.

Now the disadvantage right now is the GPU driver doesn't give nearly as smooth graphics on 64 bits than on 32 bits because RPiF/T doesn't give a damn about the former. There is noticeable lag.

jdonald
Posts: 449
Joined: Fri Nov 03, 2017 4:36 pm

Re: Why moving to 64bit?

Thu Aug 22, 2019 2:44 pm

Cob wrote:
Thu Aug 22, 2019 8:50 am
Being based on Arch linux it's quite a lot easier to work with than Gentoo.
Both Gentoo and Arch arm64 leave me in unknown territory on how to run 32-bit baseline tests. On Ubuntu/Debian/Raspbian it's straighforward with debootstrap or Docker for a 32-bit container, apt-get to retrieve source with all the appropriate patches, and dpkg-buildpackage to build something like OpenSSL with its OS-specific flags. What are the "easy" recipes for this on Arch and Gentoo?
pica200 wrote:
Thu Aug 22, 2019 10:19 am
believe it or not Firefox was already working better without GPU driver than ESR on Raspian.
This is expected even if comparing Raspbian to other 32-bit distros, due to the ARMv6 problem again. Firefox's 2D rendering is built on Skia which is preferably compiled with NEON. I recall the WebRTC code also has NEON pieces which likewise require ARMv7.

Well, assuming Raspbian's firefox-esr maintained compatibility with Pi Zero.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 30436
Joined: Sat Jul 30, 2011 7:41 pm

Re: Why moving to 64bit?

Thu Aug 22, 2019 7:00 pm

pica200 wrote:
Thu Aug 22, 2019 10:19 am
This is what i have been using for the past week.And believe it or not Firefox was already working better without GPU driver than ESR on Raspian. And on top of that now that the GPU driver landed in the kernel (need to rename fbturbo config) Firefox is even working with acceleration force enabled while doing the same on Raspian with ESR crashed all tabs.

Now the disadvantage right now is the GPU driver doesn't give nearly as smooth graphics on 64 bits than on 32 bits because RPiF/T doesn't give a damn about the former. There is noticeable lag.
We give many damns. Stop making assumptions on what do or don't care about.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

User avatar
Gavinmc42
Posts: 6438
Joined: Wed Aug 28, 2013 3:31 am

Re: Why moving to 64bit?

Thu Aug 22, 2019 11:41 pm

We give many damns. Stop making assumptions on what do or don't care about.
Take s big deep breath, exhale slowly, think calming thoughts.

RPF, a victim of their own success?
A huge company that is taking over the world and can do anything?
Perhaps RPF can take over the World, but it will happen slowly.
Just not fast enough for some people. :lol:

I wonder how many of these whingers can actually code?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

jdonald
Posts: 449
Joined: Fri Nov 03, 2017 4:36 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 3:04 am

Running Manjaro now. Unlike Gentoo, and similar to Debian arm64, it defaults to poor SSL performance then shows that same 60% performance jump just by setting OPENSSL_armcap=0. This pointed to a key factor being OpenSSL 1.0.2s vs 1.1.1c.

Then looking at a source file that is unique to 1.1.1 (vpaes-armv8.pl), this partly aligns with ejolson's theory:
jdonald wrote:
Sat Aug 17, 2019 8:59 pm
ejolson wrote:
Sat Aug 17, 2019 7:43 pm
timing-related side-channel attacks. Therefore, it is possible that the current 64-bit ARM assembler in openssl works but was deemed unsafe and that is why it is currently disabled.
Is compiled code necessarily more resistant against timing attacks?

Code: Select all

#                   CBC enc     ECB enc/dec(*)   [bit-sliced enc/dec]
# Cortex-A53        21.5        18.1/20.6        [17.5/19.8         ]
# Cortex-A57        36.0(**)    20.4/24.9(**)    [14.4/16.6         ]
# ...
# (**)  these results are worse than scalar compiler-generated
#       code, but it's constant-time and therefore preferred;
(Note: This actually somewhat the opposite of the earlier hypothesis, rather compiled code here is considered less safe while the assembler path for VPAES_ASM is slower.)

As the comment says they are indeed sacrificing 64-bit performance further for the intent of making it more resistant to side-channel attacks. At the same time, that's a whole additional 60% performance cost in a CPU-constrained system.

Gentoo avoids the performance loss at the expense of remaining theoretically more vulnerable. I doubt this was a conscious decision, just an artifact of sticking with OpenSSL 1.0.2 for other reasons.

And non-Gentoo 64-bit users are put into the odd situation that they can choose to reduce overhead by running Chromium or Firefox with OPENSSL_armcap=0

Funny how such intricate consequences ultimately stemmed from the decision not to put ARMv8 cryptographic extensions in the Pi 4.

Heater
Posts: 18854
Joined: Tue Jul 17, 2012 3:02 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 4:06 am

I think assembler might required for timing attack resistance so that one can be sure the output code is actually what you want rather than being rearranged and/or optimized away by a compiler.
Memory in C++ is a leaky abstraction .

pica200
Posts: 281
Joined: Tue Aug 06, 2019 10:27 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 5:39 am

Gavinmc42 wrote:
Thu Aug 22, 2019 11:41 pm
I wonder how many of these whingers can actually code?
Are you implying people who spent money on a product have no voice when they are not capable of coding the software for it themselves? This is not how it works...

__________________________________________________________________________________________________

Yeah, so we would not have this problem if they included it. From the first Q&A video it seems they don't even know why they didn't which doesn't make this any better. For local network or file/disk encryption you can probably sacrifice security for speed. For connections outside your local network better safe than sorry.

ejolson
Posts: 8897
Joined: Tue Mar 18, 2014 11:47 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 5:44 am

jdonald wrote:
Fri Aug 23, 2019 3:04 am
Funny how such intricate consequences ultimately stemmed from the decision not to put ARMv8 cryptographic extensions in the Pi 4.
By now some might suspect AES is not a convenient cypher for streaming data across the world wide web. Fortunately most uses of https are not cryptographic confidentiality, but to prevent add injection by internet service providers.

While I'm not an expert, it is notable that wireguard--the fast, modern, secure VPN tunnel--avoids AES entirely. Wireguard instead employs chacha20 from the danceable-name family of cyphers.
Last edited by ejolson on Fri Aug 23, 2019 5:51 am, edited 2 times in total.

User avatar
Gavinmc42
Posts: 6438
Joined: Wed Aug 28, 2013 3:31 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 5:50 am

Are you implying people who spent money on a product have no voice when they are not capable of coding the software for it themselves? This is not how it works...
Nope, not at all, I am implying non-coders might not have an idea of how long it takes to do complex software.
You buy the Pi hardware, the software is free and you are free to use any software you like on it, if it works.
If it does not work you are free to fix it yourself too.
That is how it works.

I moved to Gentoo64, OpenSCAD builds and works, that is one reason to move.
Warzone2100 works, another reason.
How many more "reasons" will work on Gentoo64?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

pica200
Posts: 281
Joined: Tue Aug 06, 2019 10:27 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 7:16 am

ejolson wrote:
Fri Aug 23, 2019 5:44 am
While I'm not an expert, it is notable that wireguard--the fast, modern, secure VPN tunnel--avoids AES entirely. Wireguard instead employs chacha20 from the danceable-name family of cyphers.
Not an expert either but i'm sceptical if this is as secure as they say. Curve 25519 is proven secure and therefore fine but not sure about the other bits of the implementation. The fact it's a kernel module worries me too because in case there is a vulnerability the system is immediately completely compromised (even if the attack surface is small that doesn't mean vulns are not possible).

User avatar
Gavinmc42
Posts: 6438
Joined: Wed Aug 28, 2013 3:31 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 7:43 am

Not an expert either but i'm sceptical if this is as secure as they say
Nothing is secure, which is why for the last 2-3 years I have been learning baremetal so I can drop Linux ;)

Perhaps Harmony OS will takeover, I bet that will get a good look at for back doors.
And then there is the hardware issues of the chips, which is why I am looking at RISC-V or even making my own cpu's with FPGA's.

All this can be done on Pi's, designing the next generation of hardware and software.
Some kid in his/her bedroom could be working on it this now.
Progress is made when someone does not like the status quo and invents something new.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Heater
Posts: 18854
Joined: Tue Jul 17, 2012 3:02 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 12:16 pm

Gavinmc42,

There is a growing tribe of people working on that very idea.

RISC V for the safer processor. Bootloader, OS and up written in Rust for the safer software.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 8897
Joined: Tue Mar 18, 2014 11:47 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 2:02 pm

pica200 wrote:
Fri Aug 23, 2019 7:16 am
ejolson wrote:
Fri Aug 23, 2019 5:44 am
While I'm not an expert, it is notable that wireguard--the fast, modern, secure VPN tunnel--avoids AES entirely. Wireguard instead employs chacha20 from the danceable-name family of cyphers.
Not an expert either but i'm sceptical if this is as secure as they say. Curve 25519 is proven secure and therefore fine but not sure about the other bits of the implementation. The fact it's a kernel module worries me too because in case there is a vulnerability the system is immediately completely compromised (even if the attack surface is small that doesn't mean vulns are not possible).
In my opinion Wireguard is very easy to set up compared to IPsec and doesn't have the complication or commercial leanings of OpenVPN. In many ways it is similar to CIPE updated with newer cyphers and a cleaner codebase.

Creating an encrypted virtual network device at the kernel level using as few lines of code as possible seems like a reasonable approach to me. Wireguard avoids having too many cyphers and much of the committee-designed baggage present in SSL and IPsec. The point, however, is that AES was adopted by people who did not understand how susceptible the algorithm itself was to timing attacks. Even if the algorithm is built into the hardware, rather than using AES for a VPN, it's probably better to set it aside for the garbage collector.

Back on topic, does 64-bit do the cha-cha-chá faster than 32-bit?

fik
Posts: 46
Joined: Thu Jan 17, 2013 1:34 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 2:47 pm

ejolson wrote: Back on topic, does 64-bit do the cha-cha-chá faster than 32-bit?
Raspbian vs. Debian aarch64:

Code: Select all

openssl speed -evp chacha20-poly1305

Pi4B
32: 138.2 MB/s
64: 266.4 MB/s

Pi3B+
32:   96.0 MB/s
64: 221.3 MB/s
Note: OPENSSL_armcap=0 halves the speed of chacha in 64bit Debian.

Code: Select all

time dd if=/dev/zero bs=1M count=4096 status=progress | ssh localhost  'cat > /dev/zero'

Pi4B
32: 52.9 MB/s
64: 51.9 MB/s

Pi3B+
32: 27.0 MB/s
64: 27.4 MB/s
Note: ssh uses by default chacha20-poly1305, as can be seen by ssh -v localhost

jdonald
Posts: 449
Joined: Fri Nov 03, 2017 4:36 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 2:57 pm

Hi fik, that's clearly yet another ARMv6 vs aarch64 comparison, which provides an upper bound on measuring "64-bit faster than 32-bit?" . It's some data, but the lower bound remains zero.

Could you provide the Debian armhf (ARMv7) datapoints?

ejolson
Posts: 8897
Joined: Tue Mar 18, 2014 11:47 am

Re: Why moving to 64bit?

Fri Aug 23, 2019 3:08 pm

fik wrote:
Fri Aug 23, 2019 2:47 pm

Code: Select all

time dd if=/dev/zero bs=1M count=4096 status=progress | ssh localhost  'cat > /dev/zero'

Pi4B
32: 52.9 MB/s
64: 51.9 MB/s

Pi3B+
32: 27.0 MB/s
64: 27.4 MB/s
Note: ssh uses by default chacha20-poly1305, as can be seen by ssh -v localhost
Thanks for the chacha20 timings.

Does SSH use compression before encrypting the input stream?

If so, then you might be comparing the speed at which a stream of nulls is compressed. While still interesting, that may not be the speed of the cypher.

fik
Posts: 46
Joined: Thu Jan 17, 2013 1:34 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 3:50 pm

ejolson wrote: Does SSH use compression before encrypting the input stream?

If so, then you might be comparing the speed at which a stream of nulls is compressed. While still interesting, that may not be the speed of the cypher.
compression is off, that's the default, from ssh -v localhost:

Code: Select all

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

fik
Posts: 46
Joined: Thu Jan 17, 2013 1:34 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 6:02 pm

jdonald wrote:
Fri Aug 23, 2019 2:57 pm
Hi fik, that's clearly yet another ARMv6 vs aarch64 comparison, which provides an upper bound on measuring "64-bit faster than 32-bit?" . It's some data, but the lower bound remains zero.

Could you provide the Debian armhf (ARMv7) datapoints?
OK, here is Debian armhf vs. Debian aarch64

Code: Select all

openssl speed -evp chacha20-poly1305

Pi4B
32: 242.5 MB/s
64: 266.4 MB/s

Pi3B+
32: 186.6 MB/s
64: 221.3 MB/s

Code: Select all

time dd if=/dev/zero bs=1M count=4096 status=progress | ssh localhost  'cat > /dev/zero'

Pi4B
32: 51.9 MB/s
64: 51.9 MB/s

Pi3B+
32: 26.1 MB/s
64: 27.4 MB/s

jdonald
Posts: 449
Joined: Fri Nov 03, 2017 4:36 pm

Re: Why moving to 64bit?

Fri Aug 23, 2019 8:37 pm

Thanks for running that, fik. So 64-bit does help chacha20-poly1305, but closer to +10% instead of 2X over a decent 32-bit implementation.

I hope by now everybody gets the point that it's questionable to compare Raspbian (ARMv6) binary performance directly against aarch64. For some benchmarks ARMv6 and ARMv7 performance are close, but cryptography applications are so frequently implemented with NEON. Comparing a non-NEON 32-bit implementation against NEON 64-bit doesn't say a whole lot.

If anyone has trouble setting up a 32-bit chroot, here's an example:

Code: Select all

sudo apt-get install -y debootstrap schroot
cat << EOF | sudo tee /etc/schroot/chroot.d/pi32
[pi32]
description=Debian armhf
type=directory
directory=/srv/chroot/pi32
users=pi
root-groups=root
profile=desktop
personality=linux
preserve-environment=true
EOF
sudo debootstrap --arch=armhf buster /srv/chroot/pi32
sudo schroot -c pi32 -- apt-get install -y sudo
schroot -c pi32

Code: Select all

(pi32)pi@raspberrypi:~ $ openssl speed -evp aes-256-gcm
...
The chroot is more useful than merely running benchmarks. You can run your backup tasks or everyday web browser in there and make use of the performance boost as we've seen anywhere from +13% to +100%. This doesn't even require a custom kernel: Debian armhf or other ARMv7 chroots will work under Raspbian's kernel on a Pi 2B v1.1 or newer.

Still too complicated? Use Docker.

Code: Select all

docker run -it debian:buster
Can't easily do graphical applications, but it has everything needed for benchmarks.

Pimaxin
Posts: 78
Joined: Sat Jan 19, 2019 7:34 pm

Re: Why moving to 64bit?

Sat Aug 24, 2019 5:24 am

so should I download a 32 bit or 64 bit OS for the 4B?

User avatar
Gavinmc42
Posts: 6438
Joined: Wed Aug 28, 2013 3:31 am

Re: Why moving to 64bit?

Sat Aug 24, 2019 5:38 am

so should I download a 32 bit or 64 bit OS for the 4B?
Try them all and use what works for you.
What you use is up to your needs, if your needs change you can change your OS.
No one is forcing you to use any OS.

OS's are a bit like computer languages, learn as many as you can.
Languages and OS's are just tools in your software toolbox.
One thing is pointed out very clearly in this digital disruption world we are now living in.

There will be change.
The more you can be prepared for change the better you will survive.
CPU/Software History is filled will dead ends and non survivors.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Return to “General discussion”