jimboprogram
Posts: 5
Joined: Mon May 14, 2018 11:53 am

Does Overlay FS prevent SD card to be damaged?

Fri Dec 11, 2020 5:06 pm

I wonder if enabling Overlay FS and Read-only boot partition from raspi-config panel prevents SD card to be damaged when RPi is switched off without proper shutdown?

tommy55
Posts: 18
Joined: Sat Nov 14, 2020 4:18 pm

Re: Does Overlay FS prevent SD card to be damaged?

Fri Dec 11, 2020 5:39 pm

I believe it helps. I am using Overlay FS (both Overlay and Boot Partition) exactly for this purpose, to be able to turn off Raspberry improperly.

I have my Raspberry Pi 4 connected to a power-strip with a switch. Using it daily for over a month and improperly turning it off several times a week and so far I had no problems. In case SD card gets damaged, I have a backup.

The only little downside is that if you want to change something in your system (and keep the changes), you have to disable overlay, restart pi, make changes, then enable overlay again and restart. But this is something I can live with :)
Last edited by tommy55 on Fri Dec 11, 2020 6:37 pm, edited 1 time in total.

epoch1970
Posts: 7997
Joined: Thu May 05, 2016 9:33 am
Location: France

Re: Does Overlay FS prevent SD card to be damaged?

Fri Dec 11, 2020 6:33 pm

Yes. That is the purpose of that setup.
Of course if you mount data volumes RW, they are not protected themselves. But the system will reboot correctly for sure.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Fri Dec 11, 2020 7:48 pm

Both should help greatly with preventing card corruption but my understanding is these are both software protections. While the card is still accessible it is potentially possible to instruct the card to do something which results in an adverse outcome.

The chance of an adverse outcome should be very low but not as low as when the card is truly inaccessible.

jimboprogram
Posts: 5
Joined: Mon May 14, 2018 11:53 am

Re: Does Overlay FS prevent SD card to be damaged?

Sat Dec 12, 2020 11:40 am

hippy wrote:
Fri Dec 11, 2020 7:48 pm
Both should help greatly with preventing card corruption but my understanding is these are both software protections. While the card is still accessible it is potentially possible to instruct the card to do something which results in an adverse outcome.

The chance of an adverse outcome should be very low but not as low as when the card is truly inaccessible.
Thanks

And is there a way to make the card truly inaccessible?

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Sat Dec 12, 2020 1:44 pm

jimboprogram wrote:
Sat Dec 12, 2020 11:40 am
And is there a way to make the card truly inaccessible?
Turn its power off, disconnect the signalling wires, eject it from the card socket.

AIUI that's not possible with Raspberry Pi OS running when booted from SD Card.

User avatar
HermannSW
Posts: 5307
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: Does Overlay FS prevent SD card to be damaged?

Sat Dec 12, 2020 2:00 pm

I don't know OverlayFS, but just read about readonly SD card running Raspberry Pi OS on twitter:
"Make your Raspberry Pi file system read-only (Raspbian Buster)"
https://medium.com/swlh/make-your-raspb ... 558694de79
https://hermann-sw.github.io/planar_graph_playground
https://stamm-wilbrandt.de/en#raspcatbt
https://github.com/Hermann-SW/memrun
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

jimboprogram
Posts: 5
Joined: Mon May 14, 2018 11:53 am

Re: Does Overlay FS prevent SD card to be damaged?

Sat Dec 12, 2020 11:30 pm

HermannSW wrote:
Sat Dec 12, 2020 2:00 pm
I don't know OverlayFS, but just read about readonly SD card running Raspberry Pi OS on twitter:
"Make your Raspberry Pi file system read-only (Raspbian Buster)"
https://medium.com/swlh/make-your-raspb ... 558694de79
It is the same as Overlaying FS. Isnt it?

cleverca22
Posts: 6328
Joined: Sat Aug 18, 2012 2:33 pm

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 6:58 am

https://www.usenix.org/system/files/con ... inal80.pdf

after reading this pdf, i fear that mounting ANY partition writable, puts the ENTIRE card at risk


basically, any SSD with wear leveling needs to maintain a mapping of block# to flash-sector#
and if that happens to get corrupted due to an improper shutdown, it can hose the entire card, and it wont care about partition borders

one drive from the pdf above, suffered from IO errors on any attempt to read beyond ~1/3rd into the disk, so 2/3rds was just hard lost, with no way to recover


so the only safe option i can see is to entire every partition is read-only, and then optionally use overlayfs to fake it being writable for apps that cant handle pure RO
any state, would have to go into a seperate device, either another sd on some interface, or a plain usb drive

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 10:27 am

cleverca22 wrote:
Sun Dec 13, 2020 6:58 am
so the only safe option i can see is to entire every partition is read-only
Still won't be a 100% solution because the card doesn't know about partitions or anything, whether read-only or not. It will simply do what it's told to do and if if it's ever told to do something which results in an adverse outcome it will.

All these suggestions are good, most likely do deliver low enough acceptable risk of corruption. I'm just pointing out that while a card is still accessible at the hardware level the risk of corruption can never be entirely eliminated.

User avatar
HermannSW
Posts: 5307
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 10:31 am

hippy wrote:
Sun Dec 13, 2020 10:27 am
All these suggestions are good, most likely do deliver low enough acceptable risk of corruption. I'm just pointing out that while a card is still accessible at the hardware level the risk of corruption can never be entirely eliminated.
I see how a single write can damage SD card.
But if using whole SD card as readonly, how can that damage it?
https://hermann-sw.github.io/planar_graph_playground
https://stamm-wilbrandt.de/en#raspcatbt
https://github.com/Hermann-SW/memrun
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 10:48 am

HermannSW wrote:
Sun Dec 13, 2020 10:31 am
hippy wrote:
Sun Dec 13, 2020 10:27 am
All these suggestions are good, most likely do deliver low enough acceptable risk of corruption. I'm just pointing out that while a card is still accessible at the hardware level the risk of corruption can never be entirely eliminated.
I see how a single write can damage SD card.
But if using whole SD card as readonly, how can that damage it?
When read-only the software should prevent anything being written to the card, but it's no guarantee that it will in all circumstances.

Think of it this way; if there is a mechanism which could write to the card itself, even though it's read-only in software, then that could be invoked accidentally or when the hardware is being powered down when what's happening is far from predictable or certain.

The card is connected to an on-chip peripheral, that peripheral appears as a set of registers in the memory space. What happens if you have access to that memory space and start writing to those registers ?

Will that ever happen ? Possibly not, but it's not an impossibility.

I am merely clarifying that "should never happen", "is unlikely to ever happen" is not "will never happen".

User avatar
HermannSW
Posts: 5307
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 2:45 pm

hippy wrote:
Sun Dec 13, 2020 10:48 am
Will that ever happen ? Possibly not, but it's not an impossibility.

I am merely clarifying that "should never happen", "is unlikely to ever happen" is not "will never happen".
But it should be detectable.

This command determined all files under / that were modified within last day, quite quick:

Code: Select all

pi@raspberrypi400:~ $ time sudo find / -cmin -1440 | wc --lines
find: ‘/run/user/1000/gvfs’: Permission denied
find: ‘/proc/21246/task/21246/fd/5’: No such file or directory
find: ‘/proc/21246/task/21246/fdinfo/5’: No such file or directory
find: ‘/proc/21246/fd/6’: No such file or directory
find: ‘/proc/21246/fdinfo/6’: No such file or directory
163130

real	0m3.876s
user	0m0.676s
sys	0m3.130s
pi@raspberrypi400:~ $ 

Instead of counting, those files that were modified need to be looked at to determine those that were modified. Then all need to be skipped that are not located on SD card. That way you can at least identify (not prevent) SD card file modifications. If there are none, then all is fine -- if there are, ro configuration needs to be improved.
https://hermann-sw.github.io/planar_graph_playground
https://stamm-wilbrandt.de/en#raspcatbt
https://github.com/Hermann-SW/memrun
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 4:22 pm

HermannSW wrote:
Sun Dec 13, 2020 2:45 pm
hippy wrote:
Sun Dec 13, 2020 10:48 am
Will that ever happen ? Possibly not, but it's not an impossibility.

I am merely clarifying that "should never happen", "is unlikely to ever happen" is not "will never happen".
But it should be detectable.
Sure, but that wasn't the question posed.

How detectable corruption would be, how recoverable a card and its data is, depends on the damage done.

At the worst extreme a card can simply be rendered 'invisible'; not even detected when it's plugged in to anything one tries. I recall a few reports of that on the forum and I've corrupted cards that way though not with a Pi.

If it prevents booting one would have to mount the card on something else to access what there is, assuming it's not corruption which also prevents it being mounted.

At the other end of the scale it could be that the card is entirely intact save for becoming physically read-only.

cleverca22
Posts: 6328
Joined: Sat Aug 18, 2012 2:33 pm

Re: Does Overlay FS prevent SD card to be damaged?

Sun Dec 13, 2020 8:28 pm

hippy wrote:
Sun Dec 13, 2020 10:27 am
cleverca22 wrote:
Sun Dec 13, 2020 6:58 am
so the only safe option i can see is to entire every partition is read-only
Still won't be a 100% solution because the card doesn't know about partitions or anything, whether read-only or not. It will simply do what it's told to do and if if it's ever told to do something which results in an adverse outcome it will.

All these suggestions are good, most likely do deliver low enough acceptable risk of corruption. I'm just pointing out that while a card is still accessible at the hardware level the risk of corruption can never be entirely eliminated.
if every partition is mounted read-only, then you cant issue any writes to the SD card
if you never issue a write, then you cant interrupt it in the middle of a write, problem solved!

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Mon Dec 14, 2020 11:42 am

cleverca22 wrote:
Sun Dec 13, 2020 8:28 pm
if every partition is mounted read-only, then you cant issue any writes to the SD card
That's the general case because the software prevents such writes, but that doesn't mean it is impossible to sneak round the back of the software or issue a command direct to the card.

Nothing should, but that can be achieved by deliberate design, accident, or in the chaos of power collapsing.

It's like saying a doorman fronting a club will prevent anyone barred getting in. That's true so long as those barred are queuing up and presenting themselves to the doorman. It does nothing to prevent them getting in through the back door.

cleverca22
Posts: 6328
Joined: Sat Aug 18, 2012 2:33 pm

Re: Does Overlay FS prevent SD card to be damaged?

Mon Dec 14, 2020 7:12 pm

hippy wrote:
Mon Dec 14, 2020 11:42 am
cleverca22 wrote:
Sun Dec 13, 2020 8:28 pm
if every partition is mounted read-only, then you cant issue any writes to the SD card
That's the general case because the software prevents such writes, but that doesn't mean it is impossible to sneak round the back of the software or issue a command direct to the card.

Nothing should, but that can be achieved by deliberate design, accident, or in the chaos of power collapsing.

It's like saying a doorman fronting a club will prevent anyone barred getting in. That's true so long as those barred are queuing up and presenting themselves to the doorman. It does nothing to prevent them getting in through the back door.
yeah, you can still write to /dev/mmcblk0 directly, or just "mount / -o remount,rw"
but you should know if the software is doing such things, and make it stop, if you want things to be secured against power failures

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Mon Dec 14, 2020 9:00 pm

The other issue I forgot to mention is; what happens when the power is failing and a read command to the card gets corrupted as a write or something else, or similar such thing ?

Even "std r0,[r1]" will be unpredictable if r0 or r1 contains data which has become corrupted while power is failing, one can't even guarantee an instruction being fetched is as it should be, hasn't been corrupted.

I recall an issue with a PIC micro where on-chip Data Eeprom which was only ever read was being corrupted when power cycled. All sort of odd and random things can happen as power fades away.

cleverca22
Posts: 6328
Joined: Sat Aug 18, 2012 2:33 pm

Re: Does Overlay FS prevent SD card to be damaged?

Mon Dec 14, 2020 11:45 pm

hippy wrote:
Mon Dec 14, 2020 9:00 pm
The other issue I forgot to mention is; what happens when the power is failing and a read command to the card gets corrupted as a write or something else, or similar such thing ?
the SD protocol works by sending a command over the CMD+CLK pins

Code: Select all

393     /* enter READ mode */
394     if (count == 1) {
395       send_raw(MMC_READ_BLOCK_SINGLE | SH_CMD_READ_CMD_SET | SH_CMD_BUSY_CMD_SET, sector);
396     } else {
397       send_raw(MMC_READ_BLOCK_MULTIPLE | SH_CMD_READ_CMD_SET | SH_CMD_BUSY_CMD_SET, sector);
398     }
#define MMC_READ_BLOCK_SINGLE              17      /* R1 */
#define MMC_READ_BLOCK_MULTIPLE            18      /* R1 */
#define MMC_WRITE_BLOCK_SINGLE             24      /* R1 */
#define MMC_WRITE_BLOCK_MULTIPLE   25      /* R1 */
the read/write commands also take a sector# as well
after that command has been sent, the SD card will stall some until it is ready, then the data is sent over either DAT1 or DAT{1,2,3,4}, one chunk per CLK (though pi4 has DDR modes, 2 chunks per clock)

the payload being sent over the DAT lines also includes a checksum at the end

Code: Select all

> (18).toString(2)
'10010'
> (25).toString(2)
'11001'
even if you managed to get the right bits flipped in the command, the controller wont send a data payload over because its expecting a reply
and if the SD card somehow thinks a payload was sent over, it wouldnt have a valid checksum, so the write command would be rejected

though the arm cpu could maybe somehow malfunction in just the right way to go down an entirely different codepath, and send a valid write
but the PMIC should pull RUN low when it detects a major power fault, causing all cpu cores to instantly halt

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Tue Dec 15, 2020 11:06 am

cleverca22 wrote:
Mon Dec 14, 2020 11:45 pm
though the arm cpu could maybe somehow malfunction in just the right way to go down an entirely different codepath, and send a valid write
Exactly that. Or one of the VPU's going rogue. Or the on-chip hardware interface to SD Card.

Something like that is what I posited was happening when we used to have more reports of corruption, and there were even reports of corruption when using a read-only file system.

Things seem to have got better over the years but that doesn't mean all edge cases have been handled.
cleverca22 wrote:
Mon Dec 14, 2020 11:45 pm
but the PMIC should pull RUN low when it detects a major power fault, causing all cpu cores to instantly halt
But does it ? And what of Pi's which have earlier PMIC or discrete regulators ?

We have been repeatedly told that having the low-voltage indicator come on is a fatal error condition which may cause incorrect operation and cause damage. And that's a stage things will go through when something worse happens.

cleverca22
Posts: 6328
Joined: Sat Aug 18, 2012 2:33 pm

Re: Does Overlay FS prevent SD card to be damaged?

Tue Dec 15, 2020 7:23 pm

recent firmware will also enforce a fairly strong arm freq throttle, any time the low voltage signal goes off
which would reduce the demand for power, and possibly keep it stable under that low-voltage condition

Paul Hutch
Posts: 699
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA

Re: Does Overlay FS prevent SD card to be damaged?

Wed Dec 16, 2020 4:24 pm

HermannSW wrote:
Sun Dec 13, 2020 10:31 am
I see how a single write can damage SD card.
But if using whole SD card as readonly, how can that damage it?
If the micro-controller built into the uSD card is doing wear levelling when power is lost the card may become corrupted. Likely a very rare event but it is a risk that can't be avoided by making the uSD card read-only.

BTW - I have still never had a uSD card in the normal R/W state corrupted due to unexpected power loss. I ran an automated test rig that cut power at a random interval between 10 and 30 minutes on four PI3B+ for a total of over 2100 power cuts with no failures. Since that is more than the number of household power failures I get in many decades I stopped testing. So I never bother with RO mode and simply use Log2RAM to prevent wearing out the card via the system logs. I hope to fire up the tester again some day so I can kill a uSD card from power cuts.

cleverca22
Posts: 6328
Joined: Sat Aug 18, 2012 2:33 pm

Re: Does Overlay FS prevent SD card to be damaged?

Wed Dec 16, 2020 9:51 pm

Paul Hutch wrote:
Wed Dec 16, 2020 4:24 pm
If the micro-controller built into the uSD card is doing wear levelling when power is lost the card may become corrupted. Likely a very rare event but it is a risk that can't be avoided by making the uSD card read-only.
the wear leveling stuff is likely only going to be active if you had written to it "recently" (including just before the last shutdown)
if it hasnt been written to for a while, i would expect the wear leveling to not have any pending work, and to stop writing entirely
Paul Hutch wrote:
Wed Dec 16, 2020 4:24 pm
BTW - I have still never had a uSD card in the normal R/W state corrupted due to unexpected power loss. I ran an automated test rig that cut power at a random interval between 10 and 30 minutes on four PI3B+ for a total of over 2100 power cuts with no failures. Since that is more than the number of household power failures I get in many decades I stopped testing. So I never bother with RO mode and simply use Log2RAM to prevent wearing out the card via the system logs. I hope to fire up the tester again some day so I can kill a uSD card from power cuts.
2 ideas on why your tests never failed
  1. your cards have better firmware, and can just ignore the partial writes, and revert to an older state, hiding the corruption and making it seem like the power failure happened a little earlier then it really did
  2. perhaps cutting the power too quickly causes the PMIC to assert RUN, and halt everything, a much slower decay could leave you at 4.5v for long enough to make it malfunction

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Thu Dec 17, 2020 9:51 am

Paul Hutch wrote:
Wed Dec 16, 2020 4:24 pm
BTW - I have still never had a uSD card in the normal R/W state corrupted due to unexpected power loss.
Me neither but it does seem it's 'luck of the draw'. Others like yourself have run power-cycling tests and have never witnessed a single failure but others have had card corruption on their first power loss, or after being lucky enough to avoid that previously.

It is possibly one of those 'two types of people in the world' issue; those who have suffered corruption, and those yet to experience it.

hippy
Posts: 12485
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Does Overlay FS prevent SD card to be damaged?

Thu Dec 17, 2020 10:02 am

cleverca22 wrote:
Wed Dec 16, 2020 9:51 pm
perhaps cutting the power too quickly causes the PMIC to assert RUN, and halt everything, a much slower decay could leave you at 4.5v for long enough to make it malfunction
That's a good point. When I unplug my Pi PSU from the mains after shutdown the Pi's red power LED will stay on for many seconds then slowly fade away. Unplugging the USB power cable appears instant.

I expect most domestic power outages and 'throwing the master switch' power-offs will be more like the former.

It's hard to predict what happens or compare one with another because it all depends on PSU, how the Pi's configured and what it's doing, what's connected, the randomness of what may be going on as power fails.

Return to “Advanced users”