/var/log in tmpfs - errors

[Joltline]

Hello.

I have used linux extensively in the past so I know my way around the shell, but this is my first time using a Raspberry Pi. Here's the thing. Me and my colleague are trying to setup a monitoring system for some sensors using a Raspberry Pi. I performed a fresh minimal netinstall of the raspbian linux. Now since this device is expected to have a long life I'm trying to minimize the amount of write cycles performed to the SD card.

Among the things I did is the fstab modification where I set some folders (i.e. /tmp, /var/cache/apt and /var/log) to reside in RAM by using tmpfs. Yes yes, I know that the logs won't survive a reboot or a power cut. If the device starts crashing we'll simply disable the tmpfs on /var/log to see what's going on.

Anyway, I'm having some problems running /var/log on tmpfs, because some services (i.e. nginx) refuse to run and complain they can't write to /var/log or create new files in there. I was wondering, if there is a good solution to this. Is it a good idea to mount /var/log folder as world-writable (mode=1777)?

My current fstab looks like this:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>

# System partitions
/dev/mmcblk0p1 /boot vfat utf8 0 0
/dev/mmcblk0p2 / ext4 noatime,errors=remount-ro 0 1

# Temporary mount points
tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev,noexec,mode=1777 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,nodev,noexec,mode=0755 0 0
tmpfs /var/cache/apt tmpfs defaults,noatime,nosuid,nodev,noexec 0 0

# External HDD or USB key
/dev/sda1 /mnt/flash auto defaults,noatime,nosuid,nodev,noexec 0 0

Do you have any other tips on how to reduce writes to the SD card?

[Zielony]

In my opinion, the problem lies in booting order. I mean /var/log may be written before mounting fstab entries, so file handles to logs are opened to those on SD card. And then you mount tmpfs and the system is lost. Maybe providing starting all daemons after mounting fstab entries would work. It's the matter of correct systemd services configuration (if you use systemd).

In Arch we have such script like anything-sync-daemon, but it doesn't solve your problem. It solves the problem of saving logs do SD from time to time (for example every one hour) if you need this.

The other way of taking care of SD card may be setting write back cache to ext4 with 60 s commit. My /boot/cmdline.txt is:

Code: Select all

sdhci-bcm2708.enable_llm=1 sdhci-bcm2708.sync_after_dma=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 loglevel=2 root=/dev/mmcblk0p2 init=/bin/systemd rootfstype=ext4 rootflags=commit=60,data=writeback elevator=deadline noatime nodiratime data=writeback rootwait

As you can see, also noatime and nodiratime reduces write cycles.

To monitor, what daemon writes to disk I recommend:

Code: Select all

# iotop -oa

And then you can fight with them, for example by placing some daemon's directory to tmpfs with anything-sync-daemon.

[Joltline]

In my opinion, the problem lies in booting order. I mean /var/log may be written before mounting fstab entries, so file handles to logs are opened to those on SD card. And then you mount tmpfs and the system is lost. Maybe providing starting all daemons after mounting fstab entries would work. It's the matter of correct systemd services configuration (if you use systemd).

I'm not sure this is happening here. If services were writing logs to /var/log before it is mounted then I assume those logs should still be present inside /var/log on the SD card. When I accessed the SD card from my laptop via a card reader, the /var/log folder was empty.

In Arch we have such script like anything-sync-daemon, but it doesn't solve your problem. It solves the problem of saving logs do SD from time to time (for example every one hour) if you need this.

I do not. At least not for now.

The other way of taking care of SD card may be setting write back cache to ext4 with 60 s commit. My /boot/cmdline.txt is:

Code: Select all

sdhci-bcm2708.enable_llm=1 sdhci-bcm2708.sync_after_dma=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 loglevel=2 root=/dev/mmcblk0p2 init=/bin/systemd rootfstype=ext4 rootflags=commit=60,data=writeback elevator=deadline noatime nodiratime data=writeback rootwait

As you can see, also noatime and nodiratime reduces write cycles.

Thanks for bringing this up. I have not yet touched /boot/cmdline, but now that you mentioned it I'll do some research.

To monitor, what daemon writes to disk I recommend:

Code: Select all

# iotop -oa

And then you can fight with them, for example by placing some daemon's directory to tmpfs with anything-sync-daemon.

I'll check it out.


I did some more testing. It would seem that nginx is not having any trouble crating log files inside /var/log with 0755 permission mask. However it will refuse to start when the full path does not exist - it has problems creating folders when they don't exist. For example, I had to go to /var/log and create a folder called nginx in there as root. Afterwards the service started normally and an access.log and error.log were created inside. I see two options here - either to modify the init script to create this folder at boot or add another mount point for /var/log/nginx. Changing log path in nginx.conf doesn't seem to work.

http://stackoverflow.com/questions/1050 ... g-on-tmpfs

This solves one service, but there were some other services that complained about access i.e. exim. I made the log folder world-writable by mounting it with mode=1777 parameter, and the warnings stopped. I'm still not sure this is the best configuration, but I'll go along with it for now.

[1pi2much]

Related issue to this topic.

I have removed dphys-sysfile and moved /tmp and /var/log to tmpfs.
Using iotop -oa I still see jbd2 writing about 4KB / min.
/ has noatime set in fstab
I do have lighttpd running with logs on, but they go to the tmpfs

Any chance that noatime is not working?

This is with the latest raspbian wheezy on a 512MB model B

--

[AnythingIsFine]

I had a similar issue when mounting '/var/log' as 'tmpfs'.
Services like Apache2, Mysql and Redis would refuse to start as they could not create their respective log directories...

I want to add, that I also mounted '/var/log' with 'mode=1777', yet this did not help as the services were not configured to create their own log directories at start.

EDIT:

Solution 1(preffered)

I found a more elgant solution that the one I first posted, by using systemd.

When mounting '/var/log' as 'tmpfs', you need to edit the below file, to add all the files/directories your current services need to find.

Code: Select all

/usr/lib/tmpfiles.d/var.conf

For example, I added some apache2, mysql, redis log dirs as below.

Code: Select all

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

q /var 0755 - - -

L /var/run - - - - ../run

d /var/log 0755 - - -
f /var/log/wtmp 0664 root utmp -
f /var/log/btmp 0600 root utmp -
### Below dirs need to be created in /var/log(tmpfs)
d /var/log/apache2 750 root adm -
d /var/log/mysql 2750 mysql adm -
d /var/log/redis 750 redis redis -
d /var/log/apt 755 root root -
d /var/log/ntpstats 755 ntp ntp -
d /var/log/samba 750 root adm -
d /var/log/systat 755 root root -
### Below files need to be created in /var/log(tmpfs)
f /var/log/lastlog 0664 root utmp -
f /var/log/faillog 0644 root utmp -
f /var/log/nextcloud.log 640 www-data www-data -

d /var/cache 0755 - - -

d /var/lib 0755 - - -

d /var/spool 0755 - - -

You may also change '/var/log' 's default permissions from '755' as below, although I would advise against it...

Code: Select all

d /var/log 0777 - - -

Solution 2 (as alternative)

I found that you need to edit the systemd service files for the respective services as below, so they may create their log directories.

[*]Solution:

Code: Select all

sudo vim /lib/systemd/system/apache2.service

Add the below lines under the '[Service]' block

Code: Select all

[Service]
Type=forking
Environment=APACHE_STARTED_BY_SYSTEMD=true
### remove below 3 lines unless logging to tmpfs
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir /var/log/apache2
ExecStartPre=-/bin/chmod 750 /var/log/apache2
ExecStartPre=-/bin/chown root:adm /var/log/apache2
### end tmpfs

[*]Explanation:

Code: Select all

ExecStartPre=

Is a directive in the systemd service files, which executes specified commands before the service is started, in my example it creates the log directories and respective permissions.

Code: Select all

PermissionsStartOnly=true

Is a directive which tells systemd that the 'ExecStartPre' directives commands must be executed by user 'root' as oppossed to the normal apache users 'www-data'

[img]Note: [/img]

You need to do this for all services which write to '/var/log', by editing their respective service files.

* If you find this solution is not particularly suitable, you may simply write a bash script that runs before the other services start up and creates the respective directories.use the same command

[NicoD]

Solution 1(preffered)

I found a more elgant solution that the one I first posted, by using systemd.

When mounting '/var/log' as 'tmpfs', you need to edit the below file, to add all the files/directories your current services need to find.

Code: Select all

/usr/lib/tmpfiles.d/var.conf

Brilliant! Thanks for the tip, I didn't know this system existed. However in stead of changing /usr/lib/tmpfiles.d/var.conf create a file in /etc/tmpfiles.d/ in stead e.g.

Code: Select all

# cat /etc/tmpfiles.d/mosquitto.conf 
d /var/log/mosquitto 0755 mosquitto mosquitto - -

The behaviour is exactly the same but files in /usr/lib/... are part of a package and may be overwritten by an upgrade.