User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Detect a shutdown event in Python [obsolete]

Tue May 31, 2016 5:39 am

I posted this question on another forum without success, perhaps someone here can help me?

I'm running a Raspberry Pi v2 with Jessy.

Is there a way from a Python program to detect that a "shutdown" or "restart" event has been initiated? I have a Python program which runs 24x7 in a Screen session window on one of my Raspberry Pi's. I also have a cron job which reboots the Raspberry Pi (shutdown -r 2) every couple days. When it does that I lose any information in the file buffers.

I have read about signal and have tried to follow the information on the web. No success so far. Here is my code:

Code: Select all

import time
import signal
import sys
#import syslog

Log_File = '/usr/pgms/TestSignal.txt'

#Define function which will process Signal event
def CloseAll(Code, Frame):
    print(time.strftime("%Y-%m-%d %H:%M")+'\r\nshutdown detected')
#    syslog.syslog(syslog.LOG_CRIT, 'Signal Number:%d {%s}' % (Code, Frame))
    f = open(Log_File,'a')
    f.write(time.strftime("%Y-%m-%d %H:%M")+'\r\nSignal Code:' + str(Code) + ' ')
    f.write('Signal Frame:' + str(Frame))
    f.write('\r\n')
    f.close()
    sys.exit(0)

#Register CloseAll function with Signal    
signal.signal(signal.SIGTERM,CloseAll)

f = open(Log_File,'a')
f.write('\r\n')
f.write('Program Started')
f.write('\r\n')
f.close()
print(time.strftime("%Y-%m-%d %H:%M")+'\r\nProgram is running')

try:
  while True:
#write something every 15 seconds 
    time.sleep(15)

    f = open(Log_File,'a')
    f.write(time.strftime("%Y-%m-%d %H:%M")+'\r\nHello ')
    f.write('\r\n')
    f.close()

# trap cntl-c keyboard entry
except KeyboardInterrupt:
     f = open(Log_File,'a')
     f.write(time.strftime("%Y-%m-%d %H:%M")+'\r\nCntl-C detected')
     f.write('\r\n')
     f.close()
The program runs in a Screen session/window and reacts as expected to a Cntl-C from within the Screen session window and to a kill -15 <pid> from the main console window. However, when I exit the Screen session, leaving the program running, and enter sudo shutdown -r 2, the Pi reboots as expected after 2 minutes, but the TestSignal.txt file does not show that the signal.SIGTERM event was processed.

What am I doing wrong? Or better yet, how can I trap the shutdown event, usually initiated by a cron job, and close my Python program running in a screen session gracefully? Thanks....RDK

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Tue May 31, 2016 7:27 am

My guess would be that the screen process gets the TERM signal before your script. It will then probably send HUP signals to its child processes. Might be worth a shot to try trapping these as well and see where that gets you. If that doesn't work, try trapping each signal you can think of and log which signals you have received. Then you should know what to actually trap to look for a shutdown.

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Tue May 31, 2016 3:09 pm

OK, I tried adding these to my test program

Code: Select all

signal.signal(signal.SIGHUP,CloseAll)
signal.signal(signal.SIGCHLD,CloseAll)
signal.signal(signal.SIGALRM,CloseAll)
, with no positive results. The other signal options do not seem appropriate?

Perhaps I should say no consistent results, every now and again (randomly to my eye) the CloseAll function does trap a SIGTERM (code 15)event. However running the test program again after the reboot does not trap the event a second time. Makes me think it is a "timing" thing??...RDK

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Tue May 31, 2016 3:54 pm

Your results not being consistent could be because shutdown will not kill processes in any particular order. I guess you catch a signal 15 when your process is actually killed before the controlling screen process.

I repeat my recommendation of trying to trap *every* signal with a short function that just prints the number of the caught signal to some log file. That should give you a clue what's happening to your script.

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Tue May 31, 2016 4:07 pm

OK, how does one go about trapping "every" signal/event?....RDK

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Tue May 31, 2016 4:14 pm

A web search for "python trap all signals" turned up this page.

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Tue May 31, 2016 5:38 pm

OK, I added the following code:

Code: Select all

signals = {
        signal.SIGABRT: 'signal.SIGABRT',
        signal.SIGALRM: 'signal.SIGALRM',
        signal.SIGBUS: 'signal.SIGBUS',
        signal.SIGCHLD: 'signal.SIGCHLD',
        signal.SIGCONT: 'signal.SIGCONT',
        signal.SIGFPE: 'signal.SIGFPE',
        signal.SIGHUP: 'signal.SIGHUP',
        signal.SIGILL: 'signal.SIGILL',
        signal.SIGINT: 'signal.SIGINT',
        signal.SIGPIPE: 'signal.SIGPIPE',
        signal.SIGPOLL: 'signal.SIGPOLL',
        signal.SIGPROF: 'signal.SIGPROF',
        signal.SIGQUIT: 'signal.SIGQUIT',
        signal.SIGSEGV: 'signal.SIGSEGV',
        signal.SIGSYS: 'signal.SIGSYS',
        signal.SIGTERM: 'signal.SIGTERM',
        signal.SIGTRAP: 'signal.SIGTRAP',
        signal.SIGTSTP: 'signal.SIGTSTP',
        signal.SIGTTIN: 'signal.SIGTTIN',
        signal.SIGTTOU: 'signal.SIGTTOU',
        signal.SIGURG: 'signal.SIGURG',
        signal.SIGUSR1: 'signal.SIGUSR1',
        signal.SIGUSR2: 'signal.SIGUSR2',
        signal.SIGVTALRM: 'signal.SIGVTALRM',
        signal.SIGXCPU: 'signal.SIGXCPU',
        signal.SIGXFSZ: 'signal.SIGXFSZ',
        }

for num in signals:
    signal.signal(num, CloseAll)   
No messages and no happiness

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Tue May 31, 2016 5:52 pm

Strange. Have you tried just killing the screen process? Do you get any messages when you do that?

Out of curiosity: What is the reason you want to reboot your Pi every couple of days?

Do you close your logfile after each write access? I thought that should flush the buffers for that file, but I might be mistaken. You could of course issue a sync call after each write to be on the safe side.

Another option would be to somehow monitor for an existing shutdown process and trigger your CloseAll function by that.

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Tue May 31, 2016 6:24 pm

My work-around is to open and close the files for each access. This is ok, but not elegant which is why I wanted to learn about SIGNAL and event processing.

As I noted in my original posting "kill -15 <pid>" does work and is trapped by my code.

Not often, but occasionally my program hangs, hence the cron job for rebooting.

I've thought about modifying the cron job to include a "kill -15 ..." command but I am not sure how to determine the PID real time for use in the kill command.

Trying to keep the brain active....RDK

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Tue May 31, 2016 6:47 pm

So did you try just killing the screen process and seeing which signals you get?

A few more possibilities: Search for "python unbuffered io". This should allow you to open your output file in a way that anything you write to it is written to disk (well, SD card) immediately.

As for finding the PID, have a look at 'pgrep' or 'pkill' - you might have to install them first.

Also, to my taste, rebooting the Pi because a program *might* be stuck is not especially elegant anyway. Try to figure out a way to find out whether your program *is* actually stuck and in that case, kill and restart it.

Just my 2 cents of course. ;)

User avatar
CarlRJ
Posts: 598
Joined: Thu Feb 20, 2014 4:00 am
Location: San Diego, California

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 6:15 am

A couple of suggestions that aren't directly the answers you're looking for:

It sounds like you've got a long-running program running under screen, primarily so that you can watch the output. Consider converting it to run as a background process, entirely detached from the terminal. Have it get its configuration/options from a config file, and write its output to a file - you can just redirect the script's stderr and stdout to a file and then continue printing to stdout. Set up a signal handler so that the script responds to a particular signal (signal 1, HUP, was traditional) by flushing it's output buffers and rereading it's config file; this gives you the ability to interact with the running script, changing it's behavior on the fly as needed. There have been many _many_ programs that worked in this manner (sendmail is the first that comes to mind). Instead of connecting to a screen session to monitor the script's status, you can simply "tail -f" on the file where it's writing its output. As a side benefit, if the script is entirely divorced from a terminal session, it should be higher up the food chain as far as receiving signals - that is, whereas now, on shutdown, screen hears that the system is going down and goes about trying to kill its children, including your script, instead, your script will hear directly that the system is going down and can clean up as necessary.

And, echoing what others have said, having the system reboot every few days, "just because", is both overkill, and is simply treating the symptoms of a problem rather than finding the problem itself. Better to have something that starts from cron periodically (say, every hour), and does some sort of functional test to see if the app is still running properly, then shuts down and restarts the app if actually necessary. Further on down this line, you could look into something like Nagios, which extends this test-and-report-or-restart functionality to the whole system, or a networked group of systems.
Last edited by CarlRJ on Wed Jun 01, 2016 9:23 am, edited 2 times in total.

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 8:33 am

To come back to your original question (although I agree with all of CarlRJ's comments, but I was curious):

I just did the experiment of letting a python script which does nothing but trap signals and log the signal numbers and times to a file run inside a screen session. I then issued a 'shutdown -r +2' from outside that session. Here's what I got:

09:44:48: shutdown +2 issued
09:46:48: shutdown scheduled
09:46:48: Caught signal 15
09:46:48: Caught signal 1
09:46:48: Caught signal 18
09:46:48: Caught signal 1
09:46:48: Caught signal 18

For some reason, this looks quite different from your results. This was also on a Pi 2 running Jessie but not updated in a while.

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 9:56 am

Well, I'm confused! My Raspberry Pi v2 does have the PiModules http://www.pimodules.com/ UPS PIco HAT module installed. I wonder if it or their software is doing something?

Two things come to mind for next steps: In a couple weeks I will have another Raspberry Pi v2 and I can try some of these ideas on it. Also, if you could post or PM me your Python program I could run it now. Maybe there is something wrong with my code?

Thanks for all of your time and ideas....RDK

ps....I need to read up on how to convert my code to a background process...

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 10:07 am

Here's the script I've been using, it's basically pilfered from an answer on the Stackoverflow page I linked earlier:

Code: Select all

#!/usr/bin/env python
import signal,time

f = open("/path/to/trap.log",'w')
f.close()

def receive_signal(signum, stack):
    f = open("/path/to/trap.log",'a')
    print >>f, '%s: Caught signal %s, ignoring.' %(time.strftime("%H:%M:%S"),str(signum))
    f.close()

uncatchable = ['SIG_DFL','SIGSTOP','SIGKILL']
for i in [x for x in dir(signal) if x.startswith("SIG")]:
    if not i in uncatchable:
        signum = getattr(signal,i)
        signal.signal(signum,receive_signal)
while True:
    time.sleep(1)

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 11:37 am

WELL!!! Looks like there is something wrong with my code. Here is the output I got from your code:
3:30:26: Caught signal 15, ignoring.
13:30:26: Caught signal 18, ignoring.
13:30:26: Caught signal 15, ignoring.
13:30:26: Caught signal 1, ignoring.
13:30:26: Caught signal 1, ignoring.
13:30:26: Caught signal 18, ignoring.
13:30:26: Caught signal 18, ignoring.
Not exactly the same as yours, but shows it is working. Back to the drawing boards.....RDK

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 11:44 am

Good luck, have fun and let us know how it went! :)

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 1:40 pm

I'm throwing in the towel for a while. I tried a few modifications to my code, paralleling some of your code, but I still am not seeing the expected results. I'll get back to this when I get my new Raspberry Pi v2.

By the way, what happens if an event occurs during a "sleep()" period. I would hope that the event signal would be queued?

Thanks again for you help....RDK

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 2:06 pm

RDK wrote:By the way, what happens if an event occurs during a "sleep()" period. I would hope that the event signal would be queued?
The script should jump into the trap immediately. But why don't you try it out? Just increase the sleep period's duration to, say, 1000 seconds and kill your script within that period from launch.

smm
Posts: 23
Joined: Thu May 19, 2016 12:05 pm
Location: OF78wb

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 2:09 pm

Might I suggest: -

Code: Select all

pkill TestSignal.py &&sudo shutdown -r 2


Assuming your script name is "TestSignal.py"

pkill sends SIGTERM by default.

Steve

User avatar
dasmanul
Posts: 502
Joined: Wed Sep 30, 2015 10:20 am
Location: Frankfurt, Germany

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 3:11 pm

I have played with your code a bit and found the following: If I comment out the print statement from your CloseAll function, I see the signals do indeed get trapped when I kill the screen. I guess this is because print tries to write to stdout which used to be connected to the screen process which in turn no longer exists. Thus, your python code fails and the part where you write to the log file is never reached.

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Wed Jun 01, 2016 4:11 pm

Thanks guys, you are great!!! I have to get onto some other things I have let lapse while I was playing with this problem. I will try your ideas out this weekend on this Raspberry Pi v2 and an older B+ (not sure if it is Wheezy or Jessie?).

Anyway, thanks for all of these ideas and efforts......RDK

User avatar
RDK
Posts: 405
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Detect a shutdown event in Python

Sun Jun 05, 2016 8:10 am

Ok, I have completed my tests with my Pi's: v2 with Jessie and B+ with Wheezy. I modified my program from shown above to trap the following events:
SIG name = SIGTSTP; SIG number = 20
SIG name = SIGTTIN; SIG number = 21
SIG name = SIGTTOU; SIG number = 22
SIG name = SIGURG; SIG number = 23
SIG name = SIGUSR1; SIG number = 10
SIG name = SIGUSR2; SIG number = 12
SIG name = SIGVTALRM; SIG number = 26
SIG name = SIGWINCH; SIG number = 28
SIG name = SIGXCPU; SIG number = 24
SIG name = SIGXFSZ; SIG number = 25
SIG name = SIG_IGN; SIG number = 1
and SIGTERM code 15
Each had its own CloseAll function so I could, for sure, identify the event which was trapped. Below are my results, note kill, pkill and shutdown commands were run from the main console session:

Pi B+ Wheezy
  • 1. kill -15 <pid> : only trapped with code 15 (SIGTERM)
    2. pkill <pgm name> : no action (pkill does not work for screen objects)
    3. pkill screen (kill screen and its sessions) : trapped with code 1 (SIG_IGN)
    4. shutdown -r 1 : trapped with code 1
Pi v2 Jessie with a PiModules UPS PIco Hat module installed
  • 1. kill -15 <pid> : only trapped with code 15 (SIGTERM)
    2. pkill <pgm name> : no action (pkill does not work for screen objects)
    3. pkill screen (kill screen and its sessions) : trapped with code 1 (SIG_IGN)
    4. shutdown -r 1 : trapped with code 1
Thus the results are identical for the two Pi's, OS's and hardware configurations.

Observations and answers to questions/comments made previously in this thread:
  • A. pkill does not work for processes running in a "screen" window
    B. these Pi's are running "headless" at remote locations. Thus my access is via ssh and Putty. That is why I'm running the Python programs in the "screen" sessions. I can "disconnect" from the "screen" session and exit Putty without terminating the Python program. And, I can reconnect later to monitor activity.
    C. Why the cron job to reboot? One of the programs seems to randomly hang, no errors shown and no messages when I reconnect to the "screen" session.
    D. to address the loss of logging data when the reboot takes place I have modified my open to include a buffer option, ie open(Log_File,'w', 1).
    E. I need to look into setting this program up as a background process. I need to understand how it is done and what are the operational issues.
    F.
    "having the system reboot every few days, "just because", is both overkill, and is simply treating the symptoms of a problem rather than finding the problem itself. Better to have something that starts from cron periodically (say, every hour), and does some sort of functional test to see if the app is still running properly, then shuts down and restarts the app if actually necessary. "
    This is an interesting idea. I will look into it
Again, thanks to all of you for your help, comments and ideas.....RDK

metinsenturk
Posts: 1
Joined: Thu Dec 17, 2020 5:30 pm

Re: Detect a shutdown event in Python

Thu Dec 17, 2020 5:41 pm

This might be too late but I guess it may help for future reference.

I believe there are two ways to detect this.

1. Signal library
2. Starting python as another process and passing process id inside.

Both have their ups and downs.

The Signal library has a flag for the shutdown but only works in Unix-like systems. Check this link https://docs.python.org/3/library/signa ... nal.SIGHUP.

The second way is to start python as a process and have the ID passed to python as an argument. For example, in PowerShell, this would be like

``` ps1
$main = [System.Diagnostics.Process]::GetCurrentProcess()
Start-Process ` 'python' "main.py --processid $($main.Id)"
```

Inside python, have a thread or a process on the side where you check this process ID if exists or not. `psutil` has a function for this, `pid_exists()` can check if the process exists.

If the main process id does not exist, python will get to know that something is wrong and you can do your cleanup over here.

Note: I have partially tested this so try it out.

danieldean
Posts: 40
Joined: Tue Nov 24, 2020 4:44 pm

Re: Detect a shutdown event in Python

Sat Dec 19, 2020 2:20 pm

It might be better to have a flag on the write loop which the signal handler sets when it receives SIGTERM. This way the loop will finish and you can run cleanup code after it then allow the script exit. Otherwise your script may still be waiting in the loop and might either overrun or be issued a SIGKILL which you cannot handle.

Doing this you also would not need to call sys.exit() which is fairly brute force.

I used a threading.Event() object which you can use as a loop control flag and call set on when you want the script to exit.

My script is started and stopped as a service and has been running perfectly for months doing this even with restarts.

https://github.com/danieldean/Vivarium_ ... um_ctrl.py

Return to “Python”