How to check how long RPi had power outage?

[Bosse_B]

I have a number of RPi units running in different locations and I have set them up to report when they start up by calling a web script which sends me an email with their IP address.
This works fine.
Today I found 3 such messages coming from units on the same site all at 8:34 CET.
So there had been a power outage at the location.

Question:
Is there some way to check (after the system is up again) for how long it was down?
I.e. get the outage duration...
I have VPN connections to all of these sites.
So I can connect SSH with PuTTY.

[Heater]

Is there some way to check (after the system is up again) for how long it was down?

No.

Well, you could look in the system log (/var/syslog). Find where the last boot up is logged. Then see when the last log messages before that we written. If you have nothing that logs frequently on your system then make something.

I have VPN connections to all of these sites.

You could check the log of your VPN server. That will tell you when connection was lost and when it was reestablished. Of course that happens on normal reboots as well as power outages.

Often people implement a "ping" message to some server that happens every few seconds or so. Then the server receiving that can tell when something has gone offline.

[cleverca22]

Code: Select all

[root@amd-nixos:~]# journalctl -b -1 | tail
Jan 30 23:38:59 amd-nixos systemd[1]: Reached target Unmount All Filesystems.
Jan 30 23:38:59 amd-nixos systemd[1]: Reached target Late Shutdown Services.
Jan 30 23:38:59 amd-nixos systemd[1]: systemd-poweroff.service: Deactivated successfully.
Jan 30 23:38:59 amd-nixos systemd[1]: Finished System Power Off.
Jan 30 23:38:59 amd-nixos systemd[1]: Reached target System Power Off.
Jan 30 23:38:59 amd-nixos systemd[1]: Shutting down.
Jan 30 23:38:59 amd-nixos systemd-shutdown[1]: Syncing filesystems and block devices.
Jan 30 23:38:59 amd-nixos systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jan 30 23:38:59 amd-nixos systemd-journald[1068]: Received SIGTERM from PID 1 (systemd-shutdow).
Jan 30 23:38:59 amd-nixos systemd-journald[1068]: Journal stopped

the "-b -1" tells journalctl to show the logs from the previous boot, and |tail limits it to the last 10 lines
in this case, you can see i did a clean shutdown, because there was a scheduled power outage the next day

but i didnt bother shutting down the router:

Code: Select all

[root@router:~]# journalctl -b -1 | tail
Jan 31 13:00:15 router nix-gc-start[13288]: note: currently hard linking saves 14412.22 MiB
Jan 31 13:00:15 router nix-gc-start[13288]: 0 store paths deleted, 0.00 MiB freed
Jan 31 13:00:15 router systemd[1]: nix-gc.service: Deactivated successfully.
Jan 31 13:00:15 router systemd[1]: nix-gc.service: Consumed 5.724s CPU time, no IP traffic.
Jan 31 13:00:15 router kernel: nfs: server nas not responding, timed out
Jan 31 13:00:20 router kernel: nfs: server nas not responding, timed out

you can see some messages right up to 1pm, the time of the scheduled outage, then it just went dead

Code: Select all

[root@router:~]# journalctl -b 0 | head
Jan 31 16:00:29 router kernel: Linux version 5.15.70 (nixbld@localhost) (gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.38) #1-NixOS SMP Fri Sep 23 12:15:52 UTC 2022

and switching to the start of the current boot, you can see it came back at almost exactly 4pm

timestamps are going to be a bit more broken on the rpi however, because of the lack of an RTC

[bjtheone]

Unless you have a method of monitoring power supply and ensuring a clean shutdown, you will not get clean logging on power outage. Once the system comes up and gets a time sync it can easily log/phone home.

Issues with any sort of ping, phone home, or central monitoring may have issues with distinguishing between end device issues, and network issues. It depends on what you really care about.

[Bosse_B]

I tried the journalctl suggestion but on the RPi3 it did not work, just threw out this:

Code: Select all

$ sudo journalctl -b -1 | tail
Specifying boot ID or boot offset has no effect, no persistent journal was found.

Then I switched to an Ubuntu Server 20.04 LTS that also suffered the reboot following power outage.
But here the command just hangs an does not return anything at all, just a nonmoving cursor.
So I had to exit using Ctrl-C

I thought of trying to find the info inside syslog, but it is too big to navigate in an editor like nano...

So I will skip this for now and try to put in place a logging script that just writes a timestamp to a file every few minutes or so.
When it starts up it can put a unique marker in the logfile which is easy to search for.
Then I could start it from cron on @reboot and leave it running "forever".
Or put the log command itself into cron with a log every minute or so and do a startup log using @reboot in crontab.

This will at least give me the info next time there is a crash.

[cleverca22]

Specifying boot ID or boot offset has no effect, no persistent journal was found.

ack!, sounds like raspi-os doesnt have the journal saved to disk

But here the command just hangs an does not return anything at all, just a nonmoving cursor.

it can sometimes take 2 minutes to respond, depending on how fast the hdd is, and how much logs you have

[Bosse_B]

Right now the problem location had an issue again so my VPN was broken, but this time it was the fiber Internet connection that was down for a few minutes only. Back up again.
This is in Austin TX and they are having problems with the weather offloading ice on trees, electricity wiring and roads making for a difficult place to move about just now....

I will figure out a simple power outage scripted logging system and install it while the connection is in place.

EDIT:
Turns out that there are both local power outages and some that only affects the Internet connection, so I have created the following script to log (via crontab) the state on boot and then every 10 minutes including a note on the connectivity:

Code: Select all

#!/bin/bash
#This script logs the runstate to a logfile in logdir $HOME/log for each day
#Start arguments are one of B (for boot) and R (for runstate)

if [ -z "$1" ]; then
  echo "Usage: $0 B | R"
  echo "where B = boot log and R = runstate log"
  exit 0
else
  MODE="$1"
fi

LOGDIR="$HOME/log"
DAY=$(date "+%F")
TIM=$(date "+%T")
LOGFILE="${LOGDIR}/${DAY}_runstate.log"

if [ $MODE == "B" ]; then
  MESSAGE="$TIM ------------ JUST BOOTED ------------"
  eval "echo $MESSAGE >> $LOGFILE"
  exit 0
fi

if [ "$MODE" == "R" ]; then
  MESSAGE="$TIM running"
  eval "ping -4 -c 1 google.com"
  if [ $? -ne 0 ]; then
    MESSAGE="$MESSAGE - No Internet"
  fi
  eval "echo $MESSAGE >> $LOGFILE"
  exit 0
fi

echo "Please use argument B (for boot) or R (for runstate) and try again"
exit 0

[Bosse_B]

How does this relate to the topic of this thread?
It is about figuring out the start and end of a power outage using data on the Linux system.
Seems like one has to beforehand arrange a system where a log file is kept updated say every 10 minutes with a timestamp written to it.
Then on boot also write that with a timestamp.
Then one can inspect the log file after the fact and see when the last normal log was done prior to the boot record.
That is when and for how long the power outage happened.

This has nothing to do with for how long a battery supplied RPi will run...

[stevend]

I have noted that on a clean shutdown the Pi clearly stores the shutdown time. On startup this time (incrementing at the expected rate) is used until clock is synchronised with an externa time source. So you could note the time on startup, and then keep polling until there's a jump. That jump will approximate to the power down time. Possibly the same will happen on an uncontrolled shutdown.

As an alternative, fit an RTC but don't let the OS loose on it. Manage it yourself, so you can access all features of the device (the OS just deals with the time-related registers, and last time I checked, using any other feature of the RTC was difficult to impossible). The Microchip MCP79410/MCP79411/MCP79412 actually log the power fail time into a battery backed register.

[simon-useless]

Specifying boot ID or boot offset has no effect, no persistent journal was found.

ack!, sounds like raspi-os doesnt have the journal saved to disk

Yes mine does the same, I remember setting the logs to be to RAM and only written to disk occasionally (the idea being to save writing to SSD too much)

[JohnsUPS]

There is a Raspberry Pi project that I made a while back, and it is plugged into a (supposedly) UPS and generator backed power source. However, it seems to have lost power a couple of times and so I was faced with this same question. One of my thoughts was to take an 18650 cell and connect it to a clock or counter that would only increment when there was no 5v to the Pi. Sure, the battery is only going to last so long, but a low power clock or counter would run for quite a while.

[cleverca22]

but thats just an RTC with extra steps!

[JohnsUPS]

but thats just an RTC with extra steps!

If I used a RTC, then probably. However, I'd just like to open the enclosure with the circuit and read the number off of the counter (probably make it increment once per second) without any additional hardware. A battery backed RTC would do it, properly wired to start and stop, then the clock could be interrogated via the Pi. Most of the RTC chips I've seen do not have an enable pin to start and stop the clock though......

[trejan]

A I2C FRAM would work for a log of power up/down. FRAM can withstand a huge number of write cycles and is non-volatile.

A circular log of records contain the boot timestamp and the latest timestamp updated regularly will show how long the Pi was on and off for.

[Milliways]

Code: Select all

last -x

is a useful indicator. It shows each login, but occasional crash.

Code: Select all

ian      pts/0        10.1.2.108       Sat Feb 11 17:02   still logged in
ian      pts/0        fe80::1894:6927: Fri Feb 10 10:29 - 23:04  (12:35)
…
reboot   system boot  5.15.84-v7+      Thu Jan  1 10:00   still running
ian      pts/0        fe80::100c:d8ff: Sun Jan 29 12:22 - crash (-19386+01:22)

In conjunction with

Code: Select all

uptime -p
up 1 week, 6 days, 4 hours, 14 minutes

may be helpful.

My Pi has been running for 13 days.