Liberation through Computer Literacy

[ejolson]

Lor' lummy, are youse all still at this? Perl is Fast Enough for me, so I've not given it much more thought, except for:

• it may be quicker to find anagrams of words all the same length, since by definition two words can only be anagrams of each other if they're the same length. Reading and classifying strings by length is a bit slow under the Unix paradigm as we don't have fixed-length records baked into the filesystem. Either way, you only have to hash far fewer words against one another even if you still need a master array for the final output order.
• each of those comparisons of words of the same length might work quite well as separate threads.

Here's how that insane word list stacks up by word length:

Code: Select all

Length	Count	% of total
1	26	0%
2	240	0%
3	1910	0%
4	7349	2%
5	17257	4%
6	32498	8%
7	46596	11%
8	57535	13%
9	59453	14%
10	54659	13%
11	45499	11%
12	35410	8%
13	25569	6%
14	17592	4%
15	11264	3%
16	6855	2%
17	3939	1%
	… (longer words aren't significant as there are so few)

Speed isn't the primary objective, but to avoid the digital apocalypse by increasing computer literacy. In particular, with the insane British anagram challenge, any entry that obtains the correct answer wins.

I previously found 32-bit hash algorithms that are sufficient if the number of letters in the word were included as a secondary key. As it was easier to use 64-bit hashes, I never posted the code. It's possible using 32-bit hashes would make the existing C code faster when run on a 32-bit architecture; however, I was hesitant to give the zombies any more access to the encabulator.

[ejolson]

"A Second Age of Personal Computing"
"A Final Fibonacci Challenge"
"Project Digital Apocalypse Not Now"
"Liberation through Computer Literacy"

Can I suggest the next change of thread title is to "Best Troll. Ever"

There are still a number of scheduled titles coming up. In the sense of Tolkien, I do like the idea of a good troll. I wonder if your proposed title could be modified so it fit better within the forum topic "general programming chat and advice for beginners."

What do you think about "Programming Advice for the Best Troll Ever" as a coming title?

As always, thoughtful commentary and insight concerning how to liberate all the beings in Middle Earth from the oppression of digital feudalism is welcome.

[John_Spikowski]

"A Second Age of Personal Computing"
"A Final Fibonacci Challenge"
"Project Digital Apocalypse Not Now"
"Liberation through Computer Literacy"

Can I suggest the next change of thread title is to "Best Troll. Ever"

How about 'Worthless Challenges Defined'?

[Paeryn]

The word list is already sorted. If you use a hash method properly the only sorting going on is the dictionary word to build the hash key.

I think the ScriptBasic approach is the most efficient but using associative arrays rather than the hash extension module is its downfall.

I already know how to do the mapping (Kira's code is very fast), the Haskell code was a quick (in terms of writing it) implementation that performs each stage separately.

I admit I was surprised when I saw how much memory it took, I thought it would be able to throw away the old lists once a stage had finished with them (as each list is only needed to link stages) but Haskell is lazy in that it doesn't evaluate anything until it is used and for some reason it is holding on to all those temporary lists created during the sort even though after there isn't (shouldn't) be anything holding references to them, something must be though.

[ejolson]

The canine coder said with a growl, de-mouser is just a clever way of saying cat and I'm not logging in as a cat. Consequently, there have been some delays with the 64-bit testing.

I ran some of the insane British anagram challenge winners in 64-bit mode on the Raspberry Pi 4B using this Gentoo image. The results are



Numbers greater than 1 mean the program ran faster in 64-bit mode; less than 1 mean the program ran slower.

Unfortunately, the version of Node.js was too old to run iba-hash.js and the hashbrown crate was missing in Rust for iba-main.rs. Moreover, I didn't want to clean potatoes from the flux ports of the turbo encabulator again. Note also that I've not installed 8th or any of the Basic compilers and interpreters.

It is interesting that Python3 is slower on the Gentoo image compared to Raspbian. It would be 64-bit liberating if someone has the computer literacy to fix that.

[Gavinmc42]

Apart from Python, 64bit is about 10% faster.
Are they all A72 optimized compilers?
What speed difference between A53/A72/?

I would have expected a bit better than 10% with the extra registers available in 64bit mode.
Rust is looking good
Not bad for someone still learning Rust

Would a D version be better than C?

[ejolson]

Apart from Python, 64bit is about 10% faster.
Are they all A72 optimized compilers?
What speed difference between A53/A72/?

I would have expected a bit better than 10% with the extra registers available in 64bit mode.
Rust is looking good
Not bad for someone still learning Rust

Would a D version be better than C?

I believe the C and C++ codes were compiled on the respective platforms using reasonable tuning and optimization settings. I'm not so sure about Rust as I didn't explicitly specify A72 or A53 architectures in any of the compilations. The Perl and Python interpreters are whatever versions came included with the respective operating systems. That makes the low performance of Python3 on Gentoo quite mysterious. Since Gentoo is a source-based distribution, one would have expected the Python interpreter to be well tuned for the Pi 4B and faster.

[scruss]

At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.

Gentoo isn't always the newest, fastest or best-maintained. It's quite often the crashiest distro, especially if the user doesn't know what they are doing. Building everything with -Ol33t_h4kz0r can be counterproductive, unless spending your time building and rebuilding packages is what you actually want to do with your computer.

[ejolson]

At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.

So far the prebuilt Gentoo image appears complete, functional and crash free. That is a good point about Python versions. Does anyone know how to fix that on the Gentoo image?

The bar chart isn't meant to compare optimal performance of 64-bit to 32-bit mode, but rather what a beginning programmer might encounter using the bootable images easily available. Given the recent improvements, it would be nice to compare the same versions of the Python interpreters. I understand there is a now a 64-bit version of Manjaro Linux that is likely more up to date. However, as this thread is currently about the freedom which comes from being able to read and write computer programs, I would encourage speed comparisons of the different operating system images be performed elsewhere.

More pressing from a computer literacy point of view is whether the recently provided insane Haskell anagram finder will run on the 4GB Pi 4B and whether Haskell, by it's lazy functional nature, simply uses to much memory to be digitally liberating for users of previous Raspberry Pi models.

[John_Spikowski]

Another candidate for the mega memory class programming environment.

Memory is cheap if you have access to it.

[sakaki]

At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.

So far the prebuilt Gentoo image appears complete, functional and crash free. That is a good point about Python versions. Does anyone know how to fix that on the Gentoo image?

On the 64-bit 1.5.0 Gentoo image, python 3 is v3.6.9. On Debian Buster it appears to be v3.7.3, which could well explain the difference in timings.

v3.7.4 is in the standard Gentoo ebuild tree, so I'm building it on the binhost now. I'll post again once that's done, then it should be a simple matter to install it (from the resulting binary package) and make it the default v3.

Most of the stuff on the image is pretty current as regards version, so not sure why I had python 3.7 masked back - probably an issue in early testing, most likely resolved upstream now.

best, sakaki

[Paeryn]

I've improved the memory usage of my Haskell code, the culprit was the amount of ByteStrings it generated. The overheads in using them for small strings is quite big, they require 9 words + length of string each, that's 36 bytes extra per word (72 on 64-bit) and due to how they are allocated the GC can't move them to compact the heap. I've switched to normal Strings which are literally a linked list of Char, they require (1+3*length) words + 2*length bytes (Chars are UTF-16) but the GC can move them around. It's a bit slower (6.8s vs 4s) on my PC but at least the RPi can comfortably run it, it only uses up 166 MB. My RPi3 does it in 24.6s, though half that time is spent in the GC.

Code: Select all

import qualified Data.List
import qualified Data.Map.Strict as Map
import Data.Char (isAsciiLower)

printRest :: [String] -> IO ()
printRest []     = return ()
printRest [x]    = putStrLn x
printRest (x:xs) = do
  putStr x
  putStr ", "
  printRest xs

printAnagram :: [String] -> IO ()
printAnagram []     = return ()
printAnagram (x:xs) = do
  putStr x
  putStr ": "
  printRest xs

main :: IO ()
main = do
  f <- readFile "/usr/share/dict/british-english-insane"
  let wordList = filter (all isAsciiLower) (lines f)
  let hashWords = [(hw, [w]) | w <- wordList, let hw = Data.List.sort w]
  let anagrams = Map.elems $ Map.fromListWith (\ x y -> y ++ x) hashWords
  let nonUniqueWords = Data.List.filter ((> 1) . length) anagrams
  mapM_ printAnagram $ Data.List.sortOn head nonUniqueWords

Code: Select all

pi@rpi4:~/Programming/banagrams $ ./haskell_anagrams >insane-h +RTS -s
   1,370,853,644 bytes allocated in the heap
     550,927,724 bytes copied during GC
      76,971,508 bytes maximum residency (10 sample(s))
         111,116 bytes maximum slop
             164 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1303 colls,     0 par    8.470s   8.476s     0.0065s    0.0096s
  Gen  1        10 colls,     0 par    3.635s   3.639s     0.3639s    1.5546s

  INIT    time    0.001s  (  0.001s elapsed)
  MUT     time   12.474s  ( 12.488s elapsed)
  GC      time   12.106s  ( 12.116s elapsed)
  EXIT    time    0.000s  (  0.000s elapsed)
  Total   time   24.581s  ( 24.605s elapsed)

  %GC     time      49.2%  (49.2% elapsed)

  Alloc rate    109,893,198 bytes per MUT second

  Productivity  50.7% of total user, 50.8% of total elapsed

[ejolson]

It's a bit slower (6.8s vs 4s) on my PC but at least the RPi can comfortably run it, it only uses up 166 MB. My RPi3 does it in 24.6s, though half that time is spent in the GC.

It appears liberating after all for a Raspberry Pi owner to be literate in reading and writing Haskell programs. Running on the Raspberry Pi 4B obtains

Code: Select all

$ ghc -O3 haskgram.hs
$ time ./haskgram >haskgram.insane

real    0m10.600s
user    0m10.297s
sys 0m0.302s
$ 
$ time ./haskgram >haskgram.insane

real    0m10.589s
user    0m10.264s
sys 0m0.322s
$ time ./haskgram >haskgram.insane

real    0m10.603s
user    0m10.251s
sys 0m0.352s
$ time perl selfgrams4.pl >selfgrams4.insane

real    0m5.454s
user    0m5.232s
sys 0m0.221s
$ md5sum *.insane
addeda36a4631afc983f3bff13742e36  haskgram.insane
addeda36a4631afc983f3bff13742e36  selfgrams4.insane

The insane British anagram bar chart has been updated and now looks like



Unfortunately, not all seems well with my turbo encabulator today: I tried to connect it to a gigabit switch, but it won't connect. The network lights don't even turn on.

Could one of the hydrocoptic marzlevanes be loose? Would that cause the waneshaft to fumble?

I tried three different cables known to work at gigabit speeds with other machines. After repeatedly unplugging and plugging the cables it connected once or twice, but then lost connection again when I rebooted. I finally went back to fast Ethernet.

[davidcoton]

Urgent programming challenge.

Submit code for "teboot" and the companion program "coffeboot".
Marks for adherence to @DougieLawson's spec and imagination beyond that.
Fido and Kira may develop additional variations.

Deadline for submissions:

• teboot -- 4pm
• coffeboot -- 11am

[scruss]

but tea is taken at 3 and coffee at 11 anywhere civilized. Also, Dougie's process is incorrect (by definition, the way everyone else makes tea is incorrect) and excludes critical points of doctrine such as prelactarianism vs - gasp! - postlactarianism, sugar shunning, spoon heft, and the proper observance of biscuitry.

[DougieLawson]

You must know that pre-lactarianism is the first stage of a debilitating and ultimately fatal disease (you will be dead by age 110 if you continue on that course). Post-lactarianism is the one true way (and the only way that's tolerated in Yorkshire).

Dunking of biscuits is clearly documented by Philip Schofield. https://www.youtube.com/watch?v=L0Hf_zwJpm0

[Paeryn]

Kira says that tea and coffee is overrated and that pre-lactarinism is the one true way so that he can drink the milk before it is tainted by anything afterwards (and it had better be lactose free).

Personally I'm post, if there's too much milk I can't drink my tea.

[ejolson]

Kira says that tea and coffee is overrated and that pre-lactarinism is the one true way so that he can drink the milk before it is tainted by anything afterwards (and it had better be lactose free).

I've never heard of lactose-free pre-lactarianism. The lead developer of FidoBasic prefers the biscuits.

The Pi 4B still refuses to connect to my gigabit switch. More details are here, if anyone is interested.

On a similar note, I spent yesterday helping move two servers that were noticeably heavier than Raspberry Pi only to discover the networking didn't work at the new location because the required IP numbers hadn't been requisitioned yet. How can one have liberation through computer literacy if faulty networking prevents accessing the computer?

[DougieLawson]

That brings another thought to mind. Never trust a computer you can lift - regardless of its tea making abilities.

[John_Spikowski]

Great to see ScriptBasic at the top of the charts.

[rpdom]

That brings another thought to mind. Never trust a computer you can lift - regardless of its tea making abilities.

I couldn't lift my old Unix box. It weighed around 130Kg (280lbs). I had to dismantle it every time I need to move it to a different floor. It was fine on the same floor as it was on rollers. Fortunately it was fairly modular and easy to strip down and rebuild with just two screwdrivers. Three 5.25" hard disks, one of them full height (about 3GB total storage). Two tape drives (150MB QIC and 2.3GB Exabyte). Two battery modules with 4 6Ah 12V batteries in each. Two redundant PSUs. One fan tray. One CPU card (68040 @33MHz). Two memory cards (240MB total). One serial card (with 16 micro-DB9 connectors). One network card (10Mb RJ45 and 2Mb? AUI). Some assorted metal work and covers. Then it was light enough to lift a little with some help.

List price for that spec when new would have been £75K, but it was not available as a standard option. I paid £0 for it as it was going to be scrapped. It was noisy and used a lot of power, but It was cool to be able to practice Unix stuff at home and start compiling in C. I compiled a webserver for it. Due to the ancient OS it was quite a task and ended up only supporting a single session. No threading on that system. Even some of the standard libc routines were missing and I had to hack some together to make it work. It never went on the public internet. Just as well, as the security was probably terrible, but I could access it from my 486 laptop.

[ejolson]

At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.

So far the prebuilt Gentoo image appears complete, functional and crash free. That is a good point about Python versions. Does anyone know how to fix that on the Gentoo image?

v3.7.4 is in the standard Gentoo ebuild tree, so I'm building it on the binhost now. I'll post again once that's done, then it should be a simple matter to install it (from the resulting binary package) and make it the default v3.

Did Python version 3.7.4 build? If so, how to I update my Gentoo image to include it?

[Gavinmc42]

Did Python version 3.7.4 build?

I have this bookmarked
https://isshoni.org/pi64pie/
Looks like 3.7.4.-R1/2 is there in dev/lang. release candidates 1 and 2?

I tried a sudo genup and it posted an error

Code: Select all

Local copy of remote index is up-to-date and will be used.
Calculating dependencies... done!
[binary   R    ] dev-lang/python-exec-2.4.6-r1  PYTHON_TARGETS="(python3_7*)" 
[binary  NS    ] dev-lang/python-3.7.4-r2 [2.7.16, 3.5.7, 3.6.9] USE="pgo%*" 
[binary     U  ] dev-python/certifi-2019.6.16 [2018.11.29] PYTHON_TARGETS="python3_7*" 
[binary     U  ] dev-python/setuptools-41.2.0 [41.0.1] PYTHON_TARGETS="python3_7*" 
[binary   R    ] app-portage/gemato-14.1  PYTHON_TARGETS="python3_7*" 
[binary     U  ] sys-apps/portage-2.3.72 [2.3.71] PYTHON_TARGETS="python3_7*" 
[blocks B      ] <app-portage/gentoolkit-0.4.6 ("<app-portage/gentoolkit-0.4.6" is blocking sys-apps/portage-2.3.72)

 * Error: The above package list contains packages which cannot be
 * installed at the same time on the same system.

[sakaki]

Did Python version 3.7.4 build? If so, how to I update my Gentoo image to include it?

Yes, I now have 3.7.3 and 3.7.4 built with pgo etc. enabled. Turned out to be more involved than I thought: 3.7 was masked upstream for arm64, so unmasking it caused the slot to be added to PYTHON_TARGETS which then caused rebuilds for everything exporting python APIs etc...

I tried a sudo genup and it posted an error

There's another step required to upgrade from v1.5.0 to v1.5.1 (which includes not only the updated python version, but also updated kernels & boot firmware, and fixes for various reported issues (glitchy audio, wifi connection problems, monitor autodetection regression etc.)).

Hoping to ship the updated image by the end of this weekend (real-world NMIs permitting), and will also post a link to the announce post, which will include v1.5.0 -> v1.5.1 upgrade instructions, here when that's done.

Apologies for the delay ><

Best, sakaki

[Gavinmc42]

Apologies for the delay ><

No need to apologise Sakaki, we should apologise for finding bugs in your great release
It is only teething troubles and we know this is a spare time thing for you.

Has it been only two months since the Pi4 has been released?
That's about 4 months ahead of my expectations of a 64bit OS
Gentoo64 in another 4 months?
I expect by then it will be able to do nearly ALL my PI developments.