lighttpd 503, php handlers down, OS corruption?

[martinmarty]

I have a Raspberry Pi OS Bullseye 5.15.84-v7+ installation that seems to have become corrupt. It has been working well since about February 2023 until the other day I could no longer connect to its lighttpd server and received HTTP messages 500 Internal Server Error and 503 Unavailable. However, other parts of the software seemed to be working fine. I rebooted the machine and the problem persisted.

I logged in through SSH and found in the lighttpd log, a lot of:

2023-06-04 22:39:03: gw_backend.c.475) unlink /run/lighttpd/php.socket-0 after connect failed: Connection refused
2023-06-04 22:39:03: gw_backend.c.332) child signalled: 11
2023-06-04 22:39:03: gw_backend.c.970) all handlers for /index.php? on .php are down.

There would be several thousand "connection refused" msgs and then one of the php handler msgs.

Attempts to restart lighttpd were successful but did not get rid of the HTTP 503 error. As I said, reboot also did not fix.

Eventually I figured out that even "php -v" returns Segmentation Fault. So somehow php and who knows what else have become corrupt on this Linux system though it had been working fine for several months. The machine is on a UPS but could occasionally take a power hit due to extended outages. The building it is in has had many power problems recently.

I took the sdhc card and mounted it on another RPi and ran fsck on the two partitions, one small boot (FAT) and one with everything else (EXT4) and they report clean, BUT, I think this system was set up to run fsck automatically (if it needed to) on a reboot. I suspect that it took a power hit and when it came back up it fixed the file system but maybe some of the files were still damaged. Does this sound like a reasonable explanation?

Neither partition is full, they both have like 90% free space.

I intended to copy the bad sdhc card to another and then try doing apt update/upgrade and see if it would fix php. Normally I use rpi-clone to backup my sdhc card but it fails trying to copy this card, even though the fsck runs clean.

I have backups and everything is back up and running but I would like to know what happened just to know it wasn't something caused by me, i.e. that I need to fix.

I am seeking opinions on how to proceed or if it is even worth chasing. If you have read all this, what are your thoughts?
THANK YOU

[rpdom]

Yes, power issues like that are very likely to cause file system corruption.

The proper procedure is to use your backup. But if you don't have one then I would recommend creating a new card from scratch, then install extra software as needed and copy all your config and data files from the bad card to the new one.

I wouldn't rely on a file system that has damaged files in it. Even if you reinstalled everything using apt reinstall it is likely that there would still be damaged files.

Then do a backup (or three).

[martinmarty]

Thank you very much for the reply.

I do have backups, so it was not a problem recovering the system.

What is your opinion on the SDHC card itself? Do you think the card would be damaged from an event like we are supposing? I have seen test utilities available from certain manufacturers but I could not locate one for this card (Gskill). My thinking is that if I am able to reformat it and run any kind of basic I/O test successfully that it would be safe to use it again.

Thanks again.

p.s. and you're right - even if the apt stuff made it run again I would never be able to trust it because who knows what else was hosed up in there.