First "REAL" RPi Rack

[hrs]

Hi guys,

we built what i believe is the first "REAL" RPi Rack. It has 3U size and we are able to fit 70 rpis inside @600mm case depth. Inside this custom case we integrate switches already so there are just a few LAN outputs outside + we managed to get remote reboots for each raspi on our pcb boards. Each rpi is easy to manage, no screws or tools required to exchange defective part.
This is our 3rd version. It supports Rpi 1/2/3/ and Banana PI.



Our v4 will be able @42U rack @950mm depth fit 1350 RPIs inside.

Any suggestions for improvements are welcome.

[PiAhoy]

Uhh..what do you use it for?

[ejolson]

Any suggestions for improvements are welcome.

It is not clear from the picture, but are there heatsinks and/or heatpipes on the Pi 3s? Have you stress tested the unit with all Pi computers under load?

As for uses, maybe your company could sponsor a team in one of the supercomputer student cluster competitions. Such competitions have 3kw power limit and ARM is starting to look competitive from a flop/watt point of view.

[hrs]

@PiAhoy - our client is using them for clustering. Currently we got 5 of there which makes 350 RPIs total in cluster.

@yes each rpi got own heatsink, stress test was running for hours without problems.

[dasmanul]

Very nice! Although I'll admit I don't personally see the point of a cluster of Pi's...

Naive question regarding the heatsinks: Wouldn't it have been better to attach them turned 90° to better align with the airflow?

[hrs]

@dasmanul - the potential is large. Imagine 70 standalone computers ( or clustered ) @150Watt in 3U form. Some of our customers use it for dedicated server rental ( even its rpi its still better than virtual server ) and some using as hardware h264 transcoder. About heatsinks u right, but in version1 case we used different fixation, so now they look like turned wrong. But it does not matter they do not generate too much heat, even without heatsinks they working fine.

[ejolson]

@dasmanul - the potential is large. Imagine 70 standalone computers ( or clustered ) @150Watt in 3U form. Some of our customers use it for dedicated server rental ( even its rpi its still better than virtual server ) and some using as hardware h264 transcoder. About heatsinks u right, but in version1 case we used different fixation, so now they look like turned wrong. But it does not matter they do not generate too much heat, even without heatsinks they working fine.

Heat convection is naturally vertical and likely to be more significant near the surface of the heatsink than the air movement caused by the fans. The fans are needed to prevent heat from accumulating inside the box, but it is likely the current configuration for the heatsinks is the right one.

The only way to know for sure is to measure the internal CPU temperature with the heatsinks in one orientation and again in the other orientation. To negate the effects of silicon variations, the exact same Pi computers should be used for both tests.

For dedicated server rental, individual Pi computers may have certain security advantages over virtual servers. For example side-channel attacks based on shared caches could be avoided. At the same time 1GB RAM and 100mbit networking speed could be limiting. Does the h264 transcoder application leverage the GPUs?

[hrs]

@ejolson - that is correct. h264 transcoder uses just GPU and small amount of cpu if audio conversion is needed. Our tests show that single rpi2 can make up to 5 h264 conversions real time @ static bitrate 1.2Mbit.

[asavah]

@hrs
May I ask what software with hw transcode support is being used?

Awesome rack btw.

[hrs]

@asavah - its our own developed software based on omx

[ejolson]

@ejolson - that is correct. h264 transcoder uses just GPU and small amount of cpu if audio conversion is needed. Our tests show that single rpi2 can make up to 5 h264 conversions real time @ static bitrate 1.2Mbit.

That is very impressive. I would have thought the GPU in the Pi could only transcode one stream at a time. What is the resolution and frame rate?

[hrs]

@ejolson SD at 25fps

[kevinpr]

This is a very impressive solution! Can you share the name of your company along with contact information?

[jrm]

@rps do you plan to sell the rack in a near future ?
Regards,

Jeremy

[tutzeroone]

@hrs

Hello, i'm very interested by this project, andI've got a few questions :

- Are there units for sale, do you have a website or some pricing you could communicate ?
- How are the RPis connected to the main boards underneath (usb/ethernet/custom), is that same connection used to old the cards in place ?
- Would the project be compatible with other cards, (Odroid boards for example) ?

Thanks in advance for your answers !

[Chorizo.Omelet]

Has anyone been successful in contacting HRS?

Thanks

[hrs]

pretty late but still

@tutzeroone
- we did that for our client, we did not think about the sale yet
- connected via eth, it holds rpi in place and is connected to switch
- yes its compatible with most credit card sized computers

[HawaiianPi]

Whenever I see one of these all I can think about is all the wasted space and ports. Why are these always based on the consumer models with all the extra parts that aren't needed or used?

I mean, I understand that when it's done on a small scale by a hobbyist or school, but if you are going to all the trouble of making a custom rack for high density clusters, then why not build a rack based on the CM3. You could eliminate a lot of the interconnecting cables and fit a whole lot more in the same space.

Imagine a backplane with rows (and rows) of CM3 slots. Power distribution, networking and other I/O all routed through the backplane. Much neater and cleaner (no birds nest of cables), and you could fit a ton more CM3 boards in the same space.

[jamesh]

Whenever I see one of these all I can think about is all the wasted space and ports. Why are these always based on the consumer models with all the extra parts that aren't needed or used?

I mean, I understand that when it's done on a small scale by a hobbyist or school, but if you are going to all the trouble of making a custom rack for high density clusters, then why not build a rack based on the CM3. You could eliminate a lot of the interconnecting cables and fit a whole lot more in the same space.

Imagine a backplane with rows (and rows) of CM3 slots. Power distribution, networking and other I/O all routed through the backplane. Much neater and cleaner (no birds nest of cables), and you could fit a ton more CM3 boards in the same space.

I'd guess its more expensive to build you own carrier boards with ethernet chips etc, than just use the Pi3.

[HawaiianPi]

I'd guess its more expensive to build you own carrier boards with ethernet chips etc, than just use the Pi3.

True, but it would be so awesome!

[Heater]

HawaiianPi,

..why not build a rack based on the CM3...?

It's more expensive. Isn't it?

CM3 from Digikey are $53. The Pi 3 only $40.

[mic_s]

It's more expensive.

The other way around.

Element14:
Pi3B, 10+ 36.9$, 150+ 34.12$
CM3, 10+ 30.49$, 100+ 29.68$
CM3LT, 10+ 25.25$, 100+ 24,31$

[Heater]

Hmm...Digkey seems to be way out of line there then.

[jamesh]

Hmm...Digkey seems to be way out of line there then.

Yup. Not sure what they are thinking.

[MarkTF]

. . . if you are going to all the trouble of making a custom rack for high density clusters, then why not build a rack based on the CM3. You could eliminate a lot of the interconnecting cables and fit a whole lot more in the same space. . .

It's probably relevant that the original post is from July 2016 and the CM3 wasn't announced until January 2017.