How to deal with the "logs filling up space" problem?

[allfox]

Greetings ~

I just witnessed logs filling up the whole hard drive, and wondering if there is some better way I could deal with the log size.

Last night, just before I go to sleep, my router Pi suddenly started to work SLOWLY. I mean, really slowly. WiFi speed dropped to 100 kilobyte per second.

Since I wanted to go to bed. I just left it there.

And in the next morning, when I'm up to deal the problem, I found it become unable to answer DHCP request, so unable to access through network.

As it's headless, I couldn't see what is happening. So I pulled out the power cable.

Then it didn't boot up normally, I can hear the hard driver is working constantly, so I guessed it's performing a fsck. I left it there and wait. After about 20 minutes, the hard driver is stopped. But it didn't brought up the network interface. After think a while, I plugged out the power cable again. Maybe I can plug in a keyboard at this point and press Enter to see what would happen.

It boots up, and I got SSH into it. I headed straight toward logs. However, Vim couldn't open the /var/log/syslog, it froze up. The system was responding, as I could use ^a + c to start a new Screen window. I saw Vim is busy with something, using up a whole CPU core. I killed Vim, then did a "ls -lh" on logs. The "syslog", "messages" and "kern.log" sized 109G. And "df -h" told me the root file system has no space left.

I erased these logs.

They were filled up with some error lines. I don't quite care these lines for now.

Code: Select all

3473298 Feb  3 06:25:18 pi1 kernel: [23095.286352] ------------[ cut here ]------------
3473299 Feb  3 06:25:18 pi1 kernel: [23095.286371] WARNING: CPU: 1 PID: 11 at net/sched/sch_hfsc.c:1429 hfsc_dequeue+0x348/0x3        70 [sch_hfsc]()
3473300 Feb  3 06:25:18 pi1 kernel: [23095.286378] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_TCPMSS xt_tcpud        p xt_conntrack iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_fil        ter ip_tables cfg80211
3473301 Feb  3 06:25:18 pi1 kernel: [23095.286433]  x_tables rfkill pppoe pppox ppp_generic slhc cls_u32 sch_hfsc sch_fq_codel         bridge stp llc rtc_ds1307 sg 8192eu(O) asix libphy bcm2835_rng bcm2835_gpiomem i2c_bcm2708 uio_pdrv_genirq uio i2c_de        v snd_bcm2835 snd_pcm snd_timer snd fuse ipv6
3473302 Feb  3 06:25:18 pi1 kernel: [23095.286577] CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G        W  O    4.1.16-v7+ #833
3473303 Feb  3 06:25:18 pi1 kernel: [23095.286596] Hardware name: BCM2709
3473304 Feb  3 06:25:18 pi1 kernel: [23095.286630] [<800180c0>] (unwind_backtrace) from [<80013b88>] (show_stack+0x20/0x24)
3473305 Feb  3 06:25:18 pi1 kernel: [23095.286660] [<80013b88>] (show_stack) from [<80554e10>] (dump_stack+0x80/0x98)
3473306 Feb  3 06:25:18 pi1 kernel: [23095.286691] [<80554e10>] (dump_stack) from [<80026970>] (warn_slowpath_common+0x8c/0xc8        )
3473307 Feb  3 06:25:18 pi1 kernel: [23095.286716] [<80026970>] (warn_slowpath_common) from [<80026a68>] (warn_slowpath_null+0        x2c/0x34)
3473308 Feb  3 06:25:18 pi1 kernel: [23095.286747] [<80026a68>] (warn_slowpath_null) from [<7f220658>] (hfsc_dequeue+0x348/0x3        70 [sch_hfsc])
3473309 Feb  3 06:25:18 pi1 kernel: [23095.286780] [<7f220658>] (hfsc_dequeue [sch_hfsc]) from [<8049d34c>] (__qdisc_run+0x40/        0x1a0)
3473310 Feb  3 06:25:18 pi1 kernel: [23095.286809] [<8049d34c>] (__qdisc_run) from [<80479258>] (net_tx_action+0x1d0/0x274)
3473311 Feb  3 06:25:18 pi1 kernel: [23095.286835] [<80479258>] (net_tx_action) from [<8002a2c4>] (__do_softirq+0x124/0x334)
3473312 Feb  3 06:25:18 pi1 kernel: [23095.286860] [<8002a2c4>] (__do_softirq) from [<8002a514>] (run_ksoftirqd+0x40/0x6c)
3473313 Feb  3 06:25:18 pi1 kernel: [23095.286890] [<8002a514>] (run_ksoftirqd) from [<80045fec>] (smpboot_thread_fn+0x124/0x1        98)
3473314 Feb  3 06:25:18 pi1 kernel: [23095.286918] [<80045fec>] (smpboot_thread_fn) from [<800426d0>] (kthread+0xe8/0x104)
3473315 Feb  3 06:25:18 pi1 kernel: [23095.286944] [<800426d0>] (kthread) from [<8000f858>] (ret_from_fork+0x14/0x3c)
3473316 Feb  3 06:25:18 pi1 kernel: [23095.286963] ---[ end trace d0e5e44167ca3b56 ]---

I didn't know the Raspbian is able to boot with 0 byte space left on the root file system.

I don't quite get the log system's design, I thought it's a FIFO queue that new messages would overwrite the old one. It looks like not, it would just eat up whole hard disk and die.

I thought logs are for rescuing the system, so if something happened, I could check the logs. So the system would keep running all the time. However now I know the logs would stuff the system to die before other problem. I could only check the corpse.

So the joke is it would be better to be stuffed by logs than let the actual problem trigger the nukes.

I'm thinking about mounting the /var/log to some other partition to limit it.

Do you have any better suggestion on this?

[rpdom]

Have you got logrotate installed? That compresses old log file, creates a new empty log, deletes older ones and restarts services as needed.

[mung]

Didn't really read your post properly, but I am guessing here there are a few options.

You could setup a cron job (try 'man cron' to read the manual page) to erase the logs regularly or you could get the same cron job to gzip compress the logs and upload them to some other storage (server, nas, cloud, etc)

You could setup the logs to be written to some other network storage rather than the SD card (nfs, smb, cloud, etc)

You could setup logs to write to a small ram disk.

Or you could try something else I have not thought of.

I always used to have a cron job that did 'mv logfile newlogfile(date) ; gzip newlogfile(date)' not done anything like that in years so my memory maybe failing me.

[RaTTuS]

what RPdom says - log rotate
but them what you really want to know is why they are filling up
what is it that is crashing all the time and filling them up

[jamesh]

Your router problem may, if you are in the UK and using BT, down to yesterdays massive problems with their network. Certainly took out our work BT link, and slowed me down at home.

[DougieLawson]

Do you need the logs? Do you ever look at the logs? If not, then configure rsyslogd to write them to /dev/null or configure the product writing the logs to not bother logging.

[dgordon42]

Allfox,
When I first started using Jessie (from a clean install on a B+), I noticed that there was no log rotation or compression going on, and that the logs were filling up the SD card.
The most common error at the time was

Code: Select all

rsyslogd-2007: action 'action 17' suspended, next retry is ...

but I caught this before the SD card was full and so my Pi did not crash.
A search of the internet suggested that a possible fix is to disable logging to /dev/xconsole.
I followed this suggestion by commenting out the last 4 lines of /etc/rsyslog.conf, like this:

Code: Select all

# ********** Commenting out the next 4 lines *********
#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole

I was then able to force the logs to rotate with the command:

Code: Select all

sudo logrotate -d /etc/logrotate.conf

Note: this command would not run (it just wrote error messages to my logs!) before making the above changes.
After that, cron started to rotate and compress my logs as normal, before that, they were growing by many megabytes per day and would have filled up the SD card and crashed the Pi.

This may or may not be relevant to your problem, I thought I'd mention it as it was a problem I had on a clean install of Raspbian Jessie about six month ago.

Dave.

[allfox]

Thank you, guys.

I don't know the actual problem producing so much log yet, still working on it.

There is the package "logrotate" installed on my Pi. I didn't know that before.

I read its configuration, it looks like "/var/log/syslog" would rotate daily. So if the logs fill up during a single night, it won't rotate. I think it's all right anyway. There shouldn't be so much log.

I've seen those "rsyslogd-2007 action 17 suspended" lines before, I didn't pay attention to them. Now I commented out those four lines either as dgordon42's suggestion.

[dgordon42]

I've seen those "rsyslogd-2007 action 17 suspended" lines before

From this, I suspect you have the same problem I had. The lines in /etc/rsyslog.conf piping logs into /dev/xconsole prevent logrotate from working, and so old logs don't get deleted, and /var/log grows towards infinity.

You can see a working logrotate doing it's thing in /var/log, here are some examples:

Code: Select all

-rw-r----- 1 root adm  2.6K Feb  3 17:17 syslog
-rw-r----- 1 root adm  5.4K Feb  3 06:25 syslog.1
-rw-r----- 1 root adm   12K Feb  2 06:25 syslog.2.gz
-rw-r----- 1 root adm   13K Feb  1 06:25 syslog.3.gz
-rw-r----- 1 root adm   561 Jan 31 06:25 syslog.4.gz
-rw-r----- 1 root adm   23K Jan 30 06:25 syslog.5.gz
-rw-r----- 1 root adm  1.3K Jan 29 06:25 syslog.6.gz
-rw-r----- 1 root adm   23K Jan 28 06:25 syslog.7.gz

Code: Select all

-rw-r----- 1 root adm   25K Feb  1 10:22 kern.log
-rw-r----- 1 root adm  183K Jan 31 10:02 kern.log.1
-rw-r----- 1 root adm   10K Jan 19 22:11 kern.log.2.gz
-rw-r----- 1 root adm   14K Jan 17 18:50 kern.log.3.gz
-rw-r----- 1 root adm   14K Jan  9 13:52 kern.log.4.gz

Dave.

[DougieLawson]

You can fix that with a quick'n'dirty cd /var/log;find . -name "*gz" | xargs sudo rm

[jojopi]

The lines in /etc/rsyslog.conf piping logs into /dev/xconsole prevent logrotate from working, and so old logs don't get deleted, and /var/log grows towards infinity.

If true, would that not affect everyone? logrotate works for me.

Anyway, OP's problem appears to be a kernel bug, possibly the same one that is discussed here: https://bugzilla.kernel.org/show_bug.cgi?id=109581

Once the kernel starts logging thousands of warnings, no amount of logrotate will stop the disk filling up. Even turning off the log files completely is not a fix, because the kernel will slow itself to a crawl just generating all the text.

[dgordon42]

jojopi wrote:

dgordon42 wrote:
The lines in /etc/rsyslog.conf piping logs into /dev/xconsole prevent logrotate from working, and so old logs don't get deleted, and /var/log grows towards infinity.

If true, would that not affect everyone? logrotate works for me.

I agree it's odd that other people did not have this problem on their Pi's, but I did. I found the solution through a quick Google search but, if I remember, it was a Linux or Debian answer, not a RaspberryPi answer, it did not come from these forums.

My first post was just a suggestion for the OP to try, based on my experience in the past. My second post was to show what part of /var/log would look like if logrotate was working properly, and the OP's problem is something like you suggest, that is some other bug that is flooding his logs.

EDIT: Found an old thread

Dave.

[allfox]

Well, if anyone see the "rsyslogd-2007 action 17 suspend" lines in their logs, dgordon42 do offer a solution. There is a Debian bug report either: https://bugs.debian.org/cgi-bin/bugrepo ... bug=742113

I did recently switch to hfsc+fq_codel like those kernel bug reporter. I was using htb+sfq, then I read something about CoDel and HFSC. I'll keep watching that kernel bug.

Thanks.

In fact, I was thinking about the well known "I need another powered USB hub" problem, as the fist symptom was things slow down. Reading that kernel bug, I think I'd better switch back to HTB to see if it is the hub's problem.

[tpylkko]

An old practice in the linux world has been to move /var to its own partition on servers. That way, if something goes wrong and the logs fill up everything, at least your system will be able to run normally as root will not be full.

[mutley]

I personally think your best bet is, as you said to mount /var/log to another partition, the last thing you want is a ton of junk writes going to the CF card if you want long term stability. I found a ram disk to be the best option for this on the PI. Just add something like this to /etc/fstab
tmpfs /var/log tmpfs defaults,noatime,mode=0755,size=20M 0 0

I'd also recommend to add 2 more as well :-
tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=6M 0 0
tmpfs /var/log tmpfs defaults,noatime,mode=0755,size=20M 0 0
tmpfs /var/lock tmpfs defaults,noatime,mode=0755,size=12M 0 0

[Rsiesta]

The rsyslog error "rsyslogd-2007: action 'action 17' suspended" is probably due to the /dev/xconsole pipe being full, when it is full it retries the write. That pipe is used by the xconsole program, ie a console in xwindows. Since a lot of us run our pi headless xconsole is not installed or running. Commenting out the piping of logs to xconsole in the /etc/rsyslog.conf should fix the error. Once you comment it out restart rsyslog and the error should go away.

sudo service rsyslog restart

[doublehp]

To avoid any/many problems, always do this on new machines:
- install munin (at least the node, maybe the master if you don't have any other master; save CPU with CGI)
- smartctl plugin
- df plugin
- munin plugin
- loggrep configured to scan warnings and error in all your log files
- set alarms on almost all plugins
- send one email per alarm and per day
- also monitor your exim
- when your machine is not able to send email, make it blink something, like a LED, easy on a pi
- monitor temperatures, voltages and fan speeds; set your important machines to shutdown on stopped fan
- change your disks as soon as they have ONE badblock
- always keep at least 25% free disk space, especially on SSD
... and so much more ...

I have set a syslog server; it collects not only my machines, but also my NVR and switches logs and cams. Always ask syslog to use timestamp including year. Rotate logs over at least 80 weeks (sometimes problems may grow for months before you see them; and the seed may get lost if you leave default 4 weeks policy)

For pis, make sure you are comitting your ext4 at most twice per minute.

OrangePi are keeping syslog in RAMdisk, and limited to 50M. I have set a cron policy to monitor when there is less than 20% free space in that disk, and then, remove the biggest file. Anyway, logs are never lost since they are sent to my network server, even if machine crashes and can't sync to disk.

[barsznica]

I remember this happening to me years ago, which at that time I started fresh, but now I not willing to just move on, so here is how I solved my issue.

Most of my answers came from Ask Ubuntu: Very large log files, what should I do?
Namely, how to view the end (and determine reoccurring errors) of the logs and how to empty, but not delete the files.

Logrotate looks like it is working...
We have a problem!
In /var/log/ my syslog, syslog.1, kern.log & ker.log.1 were Gigs! I could see the offenders using

Code: Select all

du -sh *

Using

Code: Select all

tail -n 100 /var/log/syslog

to show the last 100 lines, showed a very similar error in all four of the logs, so I didn't use:

if the offending lines are too deeply buried to easily see what's occurring, something like

Code: Select all

for log in /var/log/{dmesg,syslog,kern.log}; do 
  echo "${log} :"
  sed -e 's/\[[^]]\+\]//' -e 's/.*[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}//' ${log} \
  | sort | uniq -c | sort -hr | head -10
done

(note: this may take some time, given such large files) which will attempt to strip off the timestamps and then count the most frequently occurring messages.

, but YMMV!

Once I resolved my issue, I cleared the logs using

Code: Select all

sudo su
> syslog
> syslog.1
> kern.log
> kern.log.1
exit

You will know if your fix has worked by repeating

Code: Select all

du -sh *

to see if the files are rapidly growing again.

I must keep an eye on this, but I do remember when I introduced the root cause Whew, not what was about to start!!

[piFixer]

or........you could just use bleachbit.
It worked for me.

[JohnBeardmore]

Probably just me being stupid, but I've never been able to get

> syslog
> syslog.1
> kern.log
> kern.log.1
etc

to work from a shell script, or from system() commands within a C++ program. Is there a way ?

Cheers, J/.