plugwash
Forum Moderator
Forum Moderator
Posts: 3770
Joined: Wed Dec 28, 2011 11:45 pm

The raspbian jessie autobuilders (warning: image heavy)

Tue Mar 18, 2014 11:51 pm

At the raspbian project I (as project cofounder) decided that the existing infastructure we had built for raspbian wheezy needed to be replaced before we started on raspbian jessie for a few reasons. Firstly because having all the build hardware in the founders basement on the end of a comcast cable line was considered far from ideal. Secondly because the software set-up was a horrible hack based on an ancient version of wanna-build*, and that building with only 1GB of ram was making some builds excruciatingly slow. Bytemark had previously offered to sponsor and host build hardware for us and so we opened discussions with them.

We build natively because cross-building support in Debian was (and still is) far too immature to be used to rebuild the whole distro as we do. So the hardware had to be arm based and powerful. My requirements included being reasonably affordable, having at least 2GB of ram, having native SATA (I've learnt the hard way that USB on arm systems is often unstable) and preferable having native Ethernet.

I evaluated the hardware options and decided the 2GB nitrogen6x was the best available. The wandboard models available at the time were rejected due to lack of native sata and only having 1GB of ram.

I also opened up discussions with bytemark at this point, they were cautiously optimistic but it took some time for them to deal with the formailities of making such a donation and I also said that I wanted to test the hardware before asking them to commit to it.

I ordered a 2GB nitrogen6x but it took ages for boundary devices to actually supply it, apparently they were having trouble getting the memory chips. I tested it by starting it working on building raspbian jessie (hosted at my place connected to the repo server over a VPN). It was also somewhat expensive (about $470 including the board itself, the 2GB ram option, sata cable, shipping, VAT and the fee for collecting the VAT)

When I first started testing I had some stability problems and it took a few kernel updates, addition of a heatsink and a lot of testing before I decided it was stable enough to use as an auto builder. It also became clear that one autobuilder was not enough to keep up with jessie and I calculated that two autobuilders was the absoloute minimum to keep up. I thought it was prudent to have two more to give us some breathing room for large updates and to give us some protection against hardware failure taking the total requirement to four.

While I was testing the nitrogen6x it came to my attention that the wandboard quad had been released offering much the same hardware as the 2GB nitrogen6x but on a physically smaller board and at about half the price. I ordered one to evaluate and after some initial bumps with the software setup set it building alongside the nitrogen6x.

Image
The two boards side by side.

A few weeks of testing proved it to be sufficiently reliable and we moved to discussions about geting a cluster of four of them hosted at bytemark. Here are some pics from the build.

Image
One of he power supplies, 5V 5A (2A for the wandboard, 1A for the HDD, 2A safety margin) with an IEC inlet (mouser part number 709-GS40A05-P1J) seperate power supplies were used for each board so we could reset crashed boards using bytemarks existing systems. The option of one big power supply and a custom relay board was considered (and is probablly what i'd have done if I was buliding it myself) but this was decided to be simpler to implement given the conditions.

Image
Unboxing shot, well packaged.

Image
500GB laptop hdd sitting on an adapter bracket, laptop hdds were used so we didn't have to worry about 12V power.

Image
A 2U rackmount case, presumablly the remains of a gutted server.

Image
The case from the front, presumablly those bays were originally hotswap but the backplane is no longer there.

Image
Plywood mounting board to hold the wandboards.

Image
Power supplies in case and connected to terminal blocks on mounting board, IEC extensions on the front two so all power connections can be made from outside the box.

Image
Power for hdds and fans.

Image
Power supplies, hard drives and associated wiring in place,

Image
Closeup of the power wiring.

Image
Wandboard quad with serial adapter, presumablly testing fit.

Image
Pillars mounted on board

Image
Marked out for remaining boards

Image
Fully drilled.

Image
All pillars mounted.

Image
Boards in place.

Image
IO panel, apparently the "result of an old Hellermann Tyton 1U 24 port back-to-back patch panel meeting a hacksaw"

Image
Power and sata connected to boards.

Image
Inside of the IO panel.

Image
All wired up

Software wise we use Debian wheezy armhf as the "outer OS" and raspbian in the build chroots. We use Debian's buildd/sbuild software with BTRFS snapshots. The boards boot off microSD cards with the hard drives used for build chroots, various other temporary files from the buildd process and swap space. We use a version of u-boot built from robert nelsons sources (don't remember the exact version offhand). We currently use a multiplatform kernel built from debian's 3.13-1~exp1 source package. Prior to that we were using robert nelson's kernels.

The autobuilders have been building raspbian jessie for some time (I plan to move wheezy over as well but it's going to take some time to get around to it). The two test autobuilders at my place are currently also still active, we don't strictly need them but it gives us faster turnaround times.

* Mike at the time had failed to get current versions to work and decided hacking up an older and broken but slightly better docuemnted version was the way to go......

Big thanks to bytemark for donating, building and hosting this for us.

User avatar
blachanc
Posts: 466
Joined: Sat Jan 26, 2013 5:03 am
Location: Quebec,canada(french)

Re: The raspbian jessie autobuilders (warning: image heavy)

Wed Mar 19, 2014 3:14 am

thanks for sharing,

you know you will get the "when should we expect an official release of jessie ? ;) "

but more seriously, doe you have a blog/info or link to a tutorial giving details on what is going on in the background
when you build for a new distro for a specific platform.
I have no visibility on what human or cpu time is requiered here.

Thanks,

Ben

PS: and of course a big thank you for you and everybody working on the PI...
Autism/Asperger syndrome: what is your score on this quiz?
http://www.raspberrypi.org/forums/viewtopic.php?f=62&t=70191

plugwash
Forum Moderator
Forum Moderator
Posts: 3770
Joined: Wed Dec 28, 2011 11:45 pm

Re: The raspbian jessie autobuilders (warning: image heavy)

Thu Mar 20, 2014 10:32 pm

blachanc wrote: you know you will get the "when should we expect an official release of jessie ? ;) "
Define "official release"

Raspbian jessie follows debian jessie, which is supposed to freeze in november this year, so it will probablly go stable some time in the first half of next year.

We don't build images ourselves, at what point those building images (including the raspberry pi foundation) switch over to jessie is their descision not ours.
but more seriously, doe you have a blog/info or link to a tutorial giving details on what is going on in the background
when you build for a new distro for a specific platform.
I have no visibility on what human or cpu time is requiered here.
Not really,

In practice most of the work is dealing with cases where some package fails to build for some stupid reason and needs to be got building to unjam progress.

I'd probablly feel suicidal if I actually tried to add up all the time I put into raspbian ;)

User avatar
blachanc
Posts: 466
Joined: Sat Jan 26, 2013 5:03 am
Location: Quebec,canada(french)

Re: The raspbian jessie autobuilders (warning: image heavy)

Fri Mar 21, 2014 3:45 am

Thank you for your reply,

The linux developement has alway looked to me like black magic, probably because I am clueless on that subject.
I really need to find the time to learn about this.

thanks again for you involvment into Raspian !

Ben
Autism/Asperger syndrome: what is your score on this quiz?
http://www.raspberrypi.org/forums/viewtopic.php?f=62&t=70191

AudibleBag
Posts: 6
Joined: Tue Feb 04, 2014 12:48 pm

Re: The raspbian jessie autobuilders (warning: image heavy)

Tue Mar 25, 2014 10:46 am

Thanks for the work on Raspbian! I enjoyed reading your post quite a bit.

Return to “Raspberry Pi OS”