Niklo
Posts: 28
Joined: Wed Feb 28, 2018 12:53 pm

kworker processes consume CPU with kernel 5.4.79

Tue Dec 08, 2020 4:17 pm

Hi,
one year ago I installed Raspian Buster on my external harddrive and boot it on my Raspberry Pi 3b. At first I noticed that the kworker processes consumed about 7 to 10 % CPU in idle mode (Please see here)
The solution was to add the following line to /boot/config.txt.

Code: Select all

dtoverlay=sdtweak,poll_once
Afterwards everything was fine until the last update and upgrade to kernel 5.4.79.
(uname -a shows "Linux wega 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l GNU/Linux")
Now the kworker processes consume 8 to 12% CPU in idle mode again.
I checked to /boot/config.txt and

Code: Select all

dtoverlay=sdtweak,poll_once
is still there but unfortunately it doesn't help.

Here are some examples from top:
top - 16:06:07 up 1 day, 18:25, 1 user, load average: 0.13, 0.19, 0.18
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 2.1 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 924.8 total, 682.4 free, 42.6 used, 199.9 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 813.5 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3468 root 20 0 0 0 0 D 11.2 0.0 1:06.20 kworker/2:0+events_freezable
3526 roland 20 0 10692 2900 2540 R 0.7 0.3 0:00.27 top
20 root 20 0 0 0 0 S 0.3 0.0 2:00.12 ksoftirqd/2

3008 root 20 0 0 0 0 R 11.2 0.0 1:17.20 kworker/2:2+events_freezable
516 root 20 0 27656 80 0 S 0.3 0.0 0:14.13 rngd
3036 root 20 0 0 0 0 I 0.3 0.0 0:20.09 kwork
3008 root 20 0 0 0 0 I 4.0 0.0 1:20.70 kworker/2:2-events
3044 root 20 0 0 0 0 I 4.0 0.0 0:05.08 kworker/2:3-events_freezable
20 root 20 0 0 0 0 S 0.3 0.0 1:39.41 ksoftirqd/2
3042 root 20 0 0 0 0 I 0.3 0.0 0:00.07 kworker/0:3-events

Me I am running the 32 Bit version but one of my friends is running the 64 Bit version on his Raspberry Pi 2 Model B Rev 1.2. Like me he is booting and running the Raspberry Pi OS from an external harddrive over USB. After updating to the 5.4.79 kernel he has the same behaviour.
Afterwards my friend downgraded to the older kernel and with the old kernel 5.4.72-v8+ everything is fine without busy kworker processes and without CPU consumtion by kworker in idele mode.
(The working kernel on his 64 bit Raspberry Pi 2 is 5.4.72-v8+ #1354 SMP PREEMPT Mon Oct 19 13:41:31 BST 2020 aarch64 GNU/Linux).
What can be done to avoid this problem wit the latest kernel version?
For any advice I would be very happy.

Kind regards,
Roland

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4733
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: kworker processes consume CPU with kernel 5.4.79

Tue Dec 08, 2020 4:26 pm

The sdtweak overlay wasn't compatible with all Pis and all situations, so it was replaced with a set of dtparams. Use this instead:

Code: Select all

dtparam=sd_poll_once

Niklo
Posts: 28
Joined: Wed Feb 28, 2018 12:53 pm

Re: kworker processes consume CPU with kernel 5.4.79

Tue Dec 08, 2020 7:42 pm

PhilE wrote:
Tue Dec 08, 2020 4:26 pm
The sdtweak overlay wasn't compatible with all Pis and all situations, so it was replaced with as set of dtparams. Use this instead:

Code: Select all

dtparam=sd_poll_once
Thank you very much. After changing the line in /boot/config.txt to "dtparam=sd_poll_once" everything works fine again :)
Best regards,
Roland

User avatar
Skraaj
Posts: 39
Joined: Wed Nov 20, 2019 11:35 pm

Re: kworker processes consume CPU with kernel 5.4.79

Tue Dec 08, 2020 11:34 pm

Any chance someone could explain what the following actually does when put in config.txt?

Code: Select all

dtparam=sd_poll_once
dtoverlay=sdtweak,poll_once
When I migrated my server to USB pendrive, kworker's variety of processes seemed to put great load on my system and I saw numbers as high as 5.50 on average. The processes were as follows (taken from iotop, sometimes they had 0:2 and + instead of -)

Code: Select all

kworker/0:0-events
kworker/0:1-events_power_efficient
kworker/0:2-events_power_efficient
kworker/0:1-events_freezable_power_
kworker/0:0-mm_percpu_wq
jbd2/sda1
Additionally when I tried using a stock 2020-12-02-raspios-buster-armhf-lite and I also saw quite high loads, 0.50 compared to usual 0.01 on freshly flashed SD. Same processes in iotop

dtparm and dtoverlay lines seemed to have resolved the problem on a clean install, I'm yet to test on the server as I can't take it down currently.

The jbd2/sda1 seems to be some journaling issue, however, I was only able to find solutions to specific problems people had with variety of packages.

Any insight would be welcome as those processes slow everything to a crawl, even website becomes unresponsive. I just have no idea if this is a bug or a feature I have no idea how to use
codedoneright.eu – newbie friendly raspberry tutorials

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4733
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: kworker processes consume CPU with kernel 5.4.79

Wed Dec 09, 2020 8:49 am

Both of those lines do (or did - sdtweak is history) the same thing - they stop Linux polling the SD card interface to check for card removal and insertion, something it will do by default because there is no Card Detect signal from the socket to the SoC. Polling when a card is present it fairly quick, but when there is no card in the slot it relies on timeouts and can result in error messages being recorded in the kernel log, all of which boosts the CPU usage.

It's unfortunate that the lower levels of the SD card stack can't tell a simple poll from a vital data access, which would allow the error reporting and retries to be turned off - there might be some heuristics one could use in the driver, but it isn't standard.

User avatar
Skraaj
Posts: 39
Joined: Wed Nov 20, 2019 11:35 pm

Re: kworker processes consume CPU with kernel 5.4.79

Thu Dec 10, 2020 3:12 am

Thank you for the reply. So basically the problem with high load can be solved by simply placing any SD card in the SD slot or by adding those lines in config.txt? If so, which one, or both?

Basically which approach would be more stability / performance oriented in terms of negating load on CPU?

I tried searching for those lines but search engines did not help much, is there any documentation that I could take a peek at?
codedoneright.eu – newbie friendly raspberry tutorials

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4733
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: kworker processes consume CPU with kernel 5.4.79

Thu Dec 10, 2020 2:19 pm

Raspberry Pi OS ships with a "dtoverlay" and "dtparam" commands that can be used to search the documentation (found in /boot/overlays/README):

Code: Select all

pi@raspberrypi:~$ dtparam -h sd_poll_once
sd_poll_once            Looks for a card once after booting. Useful
                        for network booting scenarios to avoid the
                        overhead of continuous polling. N.B. Using
                        this option restricts the system to using a
                        single card per boot (or none at all).
                        (default off)
pi@raspberrypi:~$ dtoverlay -h sdtweak
Name:   sdtweak

Info:   This overlay is now deprecated. Use the sd_* dtparams in the
        base DTB, e.g. "dtoverlay=sdtweak,poll_once" becomes
        "dtparam=sd_poll_once".

Usage:  <Deprecated>
Using the sd_poll_once parameter and not inserting an SD card is the better solution, since a) it doesn't tie up an SD card, and b) even with an SD card it will check once a second without the parameter.

User avatar
Skraaj
Posts: 39
Joined: Wed Nov 20, 2019 11:35 pm

Re: kworker processes consume CPU with kernel 5.4.79

Thu Dec 10, 2020 9:10 pm

Thank you for the reply - much appreciated, I had no idea about dtoverlay/dtparm and their help :)
codedoneright.eu – newbie friendly raspberry tutorials

User avatar
HawaiianPi
Posts: 7532
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: kworker processes consume CPU with kernel 5.4.79

Thu Dec 10, 2020 9:27 pm

More specifically, using dtparam=sd_poll_once will poll the SD card once, and if no card is present the card slot is disabled. If there is a card, then it's mounted as non-removable storage. So you can still use the SD card slot with dtparam=sd_poll_once but you can't hot-swap cards.

I often leave a non-bootable FAT32 formatted card in my USB booted systems for backups of scripts, configuration files and other important data. On the older 3B/3B+ models it also reduces boot times over an empty slot (probably does on the 4B as well, but that system is so much faster that it's not very noticable).
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups, and where is that annoying music coming from?

Return to “Troubleshooting”