Disabling services, reducing footprint and I/O

[Anonymous]

I've started playing with eliminating services and reducing IO. Partly, to save SD card wear and reduce footprint, mostly out of curiosity ("ps -aux" prints out alot of stuff!). If this has been asked and answered or otherwise discussed to death please point me there and maybe a mod can delete this. I didn't find anything, but it seems like it would have been brought up.

I've got a Pi Zero W which, right now, is just running as a Pimoroni Blinkt status server. I have a daemon on it that sets up a fifo, and a little client to send it commands to show patterns & colors; right now I ssh in to write to run the client. I do this at the end of builds and periodically with my old laptop's (now a server) HDD power state. It's fun!

So far, I've cut back on logging (not critical on this machine) and mounted my SD card's filesystems with the noatime option; I don't really want to have atime on anyway.

Any ideas or pointers to (somewhat remedial) references would be appreciated!

[cleverca22]

Partly, to save SD card wear

if the goal is to reduce disk IO, then running services are not your target, services can run without causing a single write to the card

Code: Select all

c2d ~ # echo 1 > /proc/sys/vm/block_dump ; sleep 30 ; echo 0 > /proc/sys/vm/block_dump 
c2d ~ # dmesg | tail
[1729080.207389] nfsd(2616): READ block 4283175376 on sdb1 (256 sectors)
[1729080.208873] nfsd(2616): READ block 4283175632 on sdb1 (256 sectors)
[1729080.209979] nfsd(2616): READ block 4283175888 on sdb1 (256 sectors)
[1729080.210981] nfsd(2616): READ block 4283176144 on sdb1 (256 sectors)
[1729080.212480] nfsd(2616): READ block 4283176400 on sdb1 (256 sectors)
[1729080.213353] nfsd(2616): READ block 4283176656 on sdb1 (256 sectors)
[1729080.214847] nfsd(2616): READ block 4283176912 on sdb1 (256 sectors)
[1729080.215506] nfsd(2616): READ block 4283177168 on sdb1 (256 sectors)
[1729080.217249] nfsd(2616): READ block 4283177424 on sdb1 (256 sectors)
[1729080.217985] nfsd(2616): READ block 4283177680 on sdb1 (256 sectors)
c2d ~ # dmesg | grep inode | tail     
[1729075.458673] rrdtool(32113): dirtied inode 279490245 (squid_http_requests_199.rrd) on sda1
[1729075.458751] rrdtool(32113): dirtied inode 279490244 (squid_http_hits_200.rrd) on sda1
[1729075.458836] rrdtool(32113): dirtied inode 278924506 (localhost_traffic_in_315.rrd) on sda1
[1729075.458924] rrdtool(32113): dirtied inode 278924507 (localhost_traffic_in_344.rrd) on sda1
[1729079.121480] vnstatd(2847): dirtied inode 270026586 (.eth0) on sda1
[1729079.121485] vnstatd(2847): dirtied inode 270026586 (.eth0) on sda1
[1729079.121959] vnstatd(2847): dirtied inode 270949987 (eth0) on sda1
[1729079.121963] vnstatd(2847): dirtied inode 270949987 (eth0) on sda1
[1729079.246309] syslog-ng(32099): dirtied inode 200363276 (messages.log) on sda1
[1729079.246313] syslog-ng(32099): dirtied inode 200363276 (messages.log) on sda1

this will give you a record of every single program doing disk IO for a 30 second window
in my example, both rrdtool and vnstatd are doing disk activity regularly
and syslog-ng is also trying to save the dmesg, and is more noisy then normal

[Anonymous]

I was using fatrace but this looks better! I'll give it a spin.

Reducing memory footprint and CPU usage is also a fun game. Or just having a system depend on less code. No good reason for it, just curiosity (code golf maybe?).

Thanks!

[scruss]

systemd-analyze blame is also useful. It shows you which services take the most time on startup. These may not consume any resources later on, though.

[Anonymous]

systemd-analyze blame is also useful. It shows you which services take the most time on startup. These may not consume any resources later on, though.

Sweet! I'm away from my Pis, but at the top of the list on my laptop is:

6.224s apt-daily-upgrade.service
5.284s apt-daily.service

It already starts up pretty fast, but it can always be faster!

Thanks!

[cleverca22]

`systemd-analyze plot` is also a handy one

[Anonymous]

`systemd-analyze plot` is also a handy one

Thanks!

(I want a button!)

[HPCguy]

Partly, to save SD card wear

if the goal is to reduce disk IO, then running services are not your target, services can run without causing a single write to the card

Code: Select all

c2d ~ # echo 1 > /proc/sys/vm/block_dump ; sleep 30 ; echo 0 > /proc/sys/vm/block_dump 
c2d ~ # dmesg | tail
[1729080.207389] nfsd(2616): READ block 4283175376 on sdb1 (256 sectors)
[1729080.208873] nfsd(2616): READ block 4283175632 on sdb1 (256 sectors)
[1729080.209979] nfsd(2616): READ block 4283175888 on sdb1 (256 sectors)
[1729080.210981] nfsd(2616): READ block 4283176144 on sdb1 (256 sectors)
[1729080.212480] nfsd(2616): READ block 4283176400 on sdb1 (256 sectors)
[1729080.213353] nfsd(2616): READ block 4283176656 on sdb1 (256 sectors)
[1729080.214847] nfsd(2616): READ block 4283176912 on sdb1 (256 sectors)
[1729080.215506] nfsd(2616): READ block 4283177168 on sdb1 (256 sectors)
[1729080.217249] nfsd(2616): READ block 4283177424 on sdb1 (256 sectors)
[1729080.217985] nfsd(2616): READ block 4283177680 on sdb1 (256 sectors)
c2d ~ # dmesg | grep inode | tail     
[1729075.458673] rrdtool(32113): dirtied inode 279490245 (squid_http_requests_199.rrd) on sda1
[1729075.458751] rrdtool(32113): dirtied inode 279490244 (squid_http_hits_200.rrd) on sda1
[1729075.458836] rrdtool(32113): dirtied inode 278924506 (localhost_traffic_in_315.rrd) on sda1
[1729075.458924] rrdtool(32113): dirtied inode 278924507 (localhost_traffic_in_344.rrd) on sda1
[1729079.121480] vnstatd(2847): dirtied inode 270026586 (.eth0) on sda1
[1729079.121485] vnstatd(2847): dirtied inode 270026586 (.eth0) on sda1
[1729079.121959] vnstatd(2847): dirtied inode 270949987 (eth0) on sda1
[1729079.121963] vnstatd(2847): dirtied inode 270949987 (eth0) on sda1
[1729079.246309] syslog-ng(32099): dirtied inode 200363276 (messages.log) on sda1
[1729079.246313] syslog-ng(32099): dirtied inode 200363276 (messages.log) on sda1

this will give you a record of every single program doing disk IO for a 30 second window
in my example, both rrdtool and vnstatd are doing disk activity regularly
and syslog-ng is also trying to save the dmesg, and is more noisy then normal

Hi,

/proc/sys/vm/block_dump is an absolutely amazing capability I used often on Buster, but I just switched to Bullseye where it seems to be disabled.

Does anyone know the mechanism in Bullseye to generate this same diagnostic information without rebuilding the kernel?

Code: Select all

$ echo 1 > /proc/sys/vm/block_dump
bash: /proc/sys/vm/block_dump: No such file or directory

[rpdom]

Does anyone know the mechanism in Bullseye to generate this same diagnostic information without rebuilding the kernel?

Not quite the same, but you could look at installing fatrace

Code: Select all

fatrace - report system wide file access events

[Marty]

Does anyone know the mechanism in Bullseye to generate this same diagnostic information without rebuilding the kernel?

Code: Select all

$ echo 1 > /proc/sys/vm/block_dump
bash: /proc/sys/vm/block_dump: No such file or directory

It is removed from the kernel, but in Bookworm you can do the following instead:

Code: Select all

sudo blktrace -d /dev/mmcblk0 -a pc -o - | blkparse -i -

For it to work you need to have the following kernel option enabled:

CONFIG_BLK_DEV_IO_TRACE

Check it with:

Code: Select all

sysctl -a

Source: (https://askubuntu.com/questions/1406434 ... untu-22-04)
See also: https://www.ibm.com/docs/en/linux-on-sy ... g-blktrace
Output interpretation: https://unix.stackexchange.com/question ... trace-work

Unfortunately when you want to install blktrace it pulls in the whole universe (Wayland, etc. etc.).

Tracing is also possible using the ftrace interface, e.g.:

Code: Select all

echo 1 > /sys/block//mmcblk0/mmcblk0p2/trace/enable
echo blk > /sys/kernel/debug/tracing/current_tracer
cat /sys/kernel/debug/tracing/trace_pipe 

See: https://www.kernelconfig.io/config_blk_dev_io_trace
For more information see: https://opensource.com/article/21/7/linux-kernel-ftrace

[peterlite]

You need bullseye in searches. There are tricky little changes from previous versions.

I remove things like Bluetooth instead of just disabling them. Looking at all the things I do not use, there are many using similar resources when allowed and disabled. You need to actually delete them to stop resource usage.

In Linux Mint, I use the System Monitor to see things that use resources after boot or randomly. I do not know the equivalent in the Pi OS. I have tested top and its variation but the instantaneous display is too random.

I use a delete script as an easy way to keep track of what to delete. Just keep expanding the script. You can also add deletes for things like cached package downloads. They all add up to a noticeable saving in my 16 GB microSD card.

[Anonymous]

You need bullseye in searches. There are tricky little changes from previous versions.

I remove things like Bluetooth instead of just disabling them. Looking at all the things I do not use, there are many using similar resources when allowed and disabled. You need to actually delete them to stop resource usage.

In Linux Mint, I use the System Monitor to see things that use resources after boot or randomly. I do not know the equivalent in the Pi OS. I have tested top and its variation but the instantaneous display is too random.

I use a delete script as an easy way to keep track of what to delete. Just keep expanding the script. You can also add deletes for things like cached package downloads. They all add up to a noticeable saving in my 16 GB microSD card.

All that is great advice! I’m less concerned at the moment about cached downloads sitting there, as I’m not doing anything that uses much disk. It’s cpu, ram, power, and general complexity I’m curious about lowering. But that may change: a —purge option to the script is easy enough. I’m learning to make scripts, or even snippets for bash, rather than just taking notes (although I occasionally add a comment or two), because I’ve got a few Pis up and running.

One thing I’ve noticed is interdependencies between packages are sometimes frustrating: if I’m too trusting “apt remove” something x-windows related will take out stuff I need to use GLES using kms. Getting rid of a font I don’t use will take out a whole subsystem I do use or purging window server stuff will take out that nifty font I do use with Python’s PIL for a hat. So I’m still learning how to best manage those interdependencies.

[rpdom]

So I’m still learning how to best manage those interdependencies.

Setting the "no install recommends" option (on apt command line, or in a file in /etc/apt.conf.d) can reduce the number of extra packages installed. It can occasiaonally bit you when you find a particular feature w=you wanted is a "recommended" additional package, not a "depends" one, but I find that doesn't happen very often.

Also you might like to install and look at deborphan which can identify packages that may no longer be required.

I’m less concerned at the moment about cached downloads sitting there, as I’m not doing anything that uses much disk.

apt is quite good at cleaning up after itself, unlike apt-get which doesn't do that by default. Again it's a setting that can be put in apt.conf.d/... A quick "sudo apt clean" will delete any downloaded files in the cache.

[ar51an]

Many good suggestions in above comments.

`systemd-analyze plot` is also a handy one

+1. I used this most when I was trimming down. Then open the file in browser. Very easy to understand this way:

Code: Select all

systemd-analyze plot > bootup.svg

6.224s apt-daily-upgrade.service
5.284s apt-daily.service

Disable both services and timers.

Code: Select all

sudo systemctl disable apt-daily-upgrade.timer
sudo systemctl disable apt-daily-upgrade.service
sudo systemctl disable apt-daily.timer
sudo systemctl disable apt-daily.service

Here is one more cmd:

Code: Select all

lsmod

Find modules that you do not use and stop loading those modules at boot. Blacklist those modules by putting them in file:

Code: Select all

/etc/modprobe.d/my-blacklist.conf

I use my RP4 headless as a DNS server. Here is a list of my blacklisted modules:

Code: Select all

## Blacklist kernel modules in bullseye
# Camera
blacklist bcm2835_isp
blacklist bcm2835_v4l2
blacklist bcm2835_codec
# Audio
blacklist snd_bcm2835
# Video
blacklist vc_sm_cma
blacklist rpivid_hevc
# Wireless
blacklist cfg80211
blacklist rfkill
# Vlan
install 8021q /bin/true
install garp /bin/true
install stp /bin/true
install llc /bin/true

Read more about blacklisting here:
https://wiki.debian.org/KernelModuleBlacklisting

[ar51an]

Regarding packages, below cmd does a pretty job. It removes package, dependencies & configs:

Code: Select all

sudo apt --purge autoremove <pkg-name>

It can breaks stuff as well. One in particular is alsa, it made raspi-config unusable. For removing alsa, first find all alsa related packages:

Code: Select all

dpkg-query -l 'alsa*'

Code: Select all

Do not remove 'alsa-utils'
sudo apt --purge autoremove alsa-ucm-conf
sudo apt --purge autoremove alsa-topology-conf

Below cmd can be handy to find any residual packages left by apt:

Code: Select all

dpkg -l | grep '^rc'

Remove those residual packages:

Code: Select all

sudo apt remove --purge $(dpkg -l | grep "^rc" | awk '{print $2}')

[bls]

Great tips in this thread, both for things to disable and techniques for how to find them. Thanks!

Here's what I do programmatically on every one of my systems:

• Disable sshd servicee and use ssh.socket instead. One less process to start, works great, less filling
• Disable triggerhappy on "desktop" versions. I don't use it. If you're not using it, you should disable it too.

  Code: Select all

  systemctl disable triggerhappy.service
  systemctl disable triggerhappy.socket
  # Eliminate thd.socket errors from udev
  [ -f /lib/udev/rules.d/60-triggerhappy.rules ] && mv /lib/udev/rules.d/60-triggerhappy.rules /lib/udev/rules.d/.60-triggerhappy.rules
  

• Disable rsyslog. Who needs the text files when it's all obtainable via `sudo journalctl`?

  Code: Select all

  mkdir -p /etc/systemd/journald.conf.d
  cat > /etc/systemd/journald.conf.d/030-journal.conf <<EOF
  [Journal]
  Storage=persistent
  ForwardToSyslog=no
  EOF
  systemctl disable rsyslog
  systemctl mask rsyslog
  

• Use systemd-cron instead of cron daemon. Don't run cron@hourly at all unless there's something there. MUCH less log spew, less disk writes
• Use haveged instead of rngd. rngd spews to the log every hour unless you use "-S 0", but this evokes a nasty bug in rngd after 8 days uptime

[ar51an]

systemctl disable triggerhappy.service
systemctl disable triggerhappy.socket
# Eliminate thd.socket errors from udev
[ -f /lib/udev/rules.d/60-triggerhappy.rules ] && mv /lib/udev/rules.d/60-triggerhappy.rules /lib/

Just purge it, I do not use it on "lite" OS as well. You can re-install if needed for any reason:

Code: Select all

sudo apt --purge autoremove triggerhappy

mkdir -p /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/030-journal.conf <<EOF

Why not make changes in journald.conf directly? (recommended in bullseye):

Code: Select all

/etc/systemd/journald.conf

Use haveged instead of rngd. rngd spews to the log every hour unless you use "-S 0", but this evokes a nasty bug in rngd after 8 days uptime

haveged is considered obsolete starting from kernel 5.6:
https://github.com/jirka-h/haveged/issues/57
https://wiki.archlinux.org/title/Haveged

Use rng-tools5 package instead of default "rng-tools" and "rng-tools-debian". Below cmd will install rng-tools5 and automatically removes "rng-tools" and "rng-tools-debian"

Code: Select all

sudo apt install rng-tools5

[bls]

systemctl disable triggerhappy.service
systemctl disable triggerhappy.socket
# Eliminate thd.socket errors from udev
[ -f /lib/udev/rules.d/60-triggerhappy.rules ] && mv /lib/udev/rules.d/60-triggerhappy.rules /lib/

Just purge it, I do not use it on "lite" OS as well. You can re-install if needed for any reason:

Code: Select all

sudo apt --purge autoremove triggerhappy

mkdir -p /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/030-journal.conf <<EOF

Why not make changes in journald.conf directly? (recommended in bullseye):

Code: Select all

/etc/systemd/journald.conf

Use haveged instead of rngd. rngd spews to the log every hour unless you use "-S 0", but this evokes a nasty bug in rngd after 8 days uptime

haveged is considered obsolete starting from kernel 5.6:
https://github.com/jirka-h/haveged/issues/57
https://wiki.archlinux.org/title/Haveged

Use rng-tools5 package instead of default "rng-tools" and "rng-tools-debian". Below cmd will install rng-tools5 and automatically removes "rng-tools" and "rng-tools-debian"

Code: Select all

sudo apt install rng-tools5

While it's true triggerhappy can be removed, I typically just disable stuff. Either way works.

rng-tools5: I'm going to have a look into that. I wasn't happy about using haveged, but I've had no issues. If rng-tools5 solves the rng spew AND the compute loop after 8 days, I'm all in!

[bls]

mkdir -p /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/030-journal.conf <<EOF

Why not make changes in journald.conf directly? (recommended in bullseye):

Code: Select all

/etc/systemd/journald.conf

I'm going to disagree with you on this one. By adding the subdirectory journal.conf.d it is completely clear what has been changed for the journal, and completely eliminates the confusion going back and looking at it several months later and wondering what was changed.

Of course, to each their own, but my preference has always been to leave as many historical "bread crumbs" as possible.

Use rng-tools5 package instead of default "rng-tools" and "rng-tools-debian". Below cmd will install rng-tools5 and automatically removes "rng-tools" and "rng-tools-debian"

Code: Select all

sudo apt install rng-tools5

I've had this running on one of my systems since you mentioned it yesterday and there's no log spew whatsoever. Thanks for the enlightenment! It's yet another reminder of why I invest time reading these forums.

[ar51an]

I'm going to disagree with you on this one. By adding the subdirectory journal.conf.d it is completely clear what has been changed for the journal, and completely eliminates the confusion going back and looking at it several months later and wondering what was changed.

All options are commented out in "/etc/systemd/journald.conf" and these are the default options of Journald. Whatever change you are going to make just edit the file and add that at the bottom keeping all the default commented options as it is. I am not sure what is not clear in that. Anyways do it the way you prefer.

This is what it says in the /etc/systemd/journald.conf:

Code: Select all

# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.

[Anonymous]

I’m pretty mellow about other people’s preferences, whether coding or sysadmin; I’ve got my quirks. Some people like breadcrumbs, others like chalk arrows on the cave walls….

I’m starting up a script that will check for “issues” first, then suggest or make changes - a bit of effort, but I always have to go an extra mile just to stay organized.

[peterlite]

script that will check for “issues”

Display the microSD card id in your script. Some are junk. You might not find out the problems until later. You could then post good and bad IDs. An example of good.

Code: Select all

SE16G
02/2016
0x5344

[rpdom]

I'm going to disagree with you on this one. By adding the subdirectory journal.conf.d it is completely clear what has been changed for the journal, and completely eliminates the confusion going back and looking at it several months later and wondering what was changed.

Of course, to each their own, but my preference has always been to leave as many historical "bread crumbs" as possible.

The other important issue is that it keeps your configuration away from the default file, so when the file gets upgraded with a new version it won't lose your changes. That is the main reason for it being like that in Debian. It applies to reinstalls too.

I know it will ask if the file should be overwritten, but I prefer not to have that question be asked.

[Anonymous]

script that will check for “issues”

Display the microSD card id in your script. Some are junk. You might not find out the problems until later. You could then post good and bad IDs. An example of good.

Code: Select all

SE16G
02/2016
0x5344

By check for “issues” I meant look for things that might lighten up the OS install (even if just for the principle of it), as in the suggestions in this topic. I’m not sure I would be likely to identify SD card problems. This is more labor intensive than a script that just makes the changes, but will help me learn, remember. Etc.

Either way, how does one get the microSD card ID?

[peterlite]

Code: Select all

#!/bin/bash

# Identify the microSD card.
ALOG="install.log"
echo "install-identify-microsd-card start.
" | tee -a $ALOG

echo /sys/block/mmcblk0/device/name
cat  /sys/block/mmcblk0/device/name  >> $ALOG
cat  /sys/block/mmcblk0/device/date  >> $ALOG
cat  /sys/block/mmcblk0/device/oemid >> $ALOG

echo "install-identify-microsd-card end.
" | tee -a $ALOG