any chance to fix dependency ordering cycle of systemd?

[beta-tester]

hi, i have a problem, that i got a dependency ordering cycle for unknown reason recently after an apt-get upgrade.
because of that systemd tried to resolve the cycle by deleting the rpcbind.service some needed resources services are not available after system boot.

is there a way to fix that problem?

Code: Select all

...
[    3.494252] systemd[1]: Found ordering cycle on basic.target/start
[    3.506955] systemd[1]: Found dependency on sysinit.target/start
[    3.519265] systemd[1]: Found dependency on rpcbind.service/start
[    3.531550] systemd[1]: Found dependency on network-online.target/start
[    3.544316] systemd[1]: Found dependency on network.target/start
[    3.556489] systemd[1]: Found dependency on dhcpcd.service/start
[    3.568562] systemd[1]: Found dependency on basic.target/start
[    3.580419] systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/start
[    3.594361] systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start
...

i am using the official raspbian jessie image and added several packages.
all was working perfectly well until i got a dhcpcd5 package update via "apt-get upgrade"
from [dhcpcd5 6.7.1-1+rpi3] to (dhcpcd5 6.7.1-1+rpi5)

[ShiftPlusOne]

Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.

[weboide]

Installing nfs-common triggered it (or made it worse) for me. I haven't looked after doing a clean install to see if there's already dependency cycles.

[ 3.206659] systemd[1]: Found ordering cycle on basic.target/start
[ 3.215051] systemd[1]: Found dependency on sysinit.target/start
[ 3.223178] systemd[1]: Found dependency on kbd.service/start
[ 3.230924] systemd[1]: Found dependency on remote-fs.target/start
[ 3.239209] systemd[1]: Found dependency on mnt-nas.mount/start
[ 3.247214] systemd[1]: Found dependency on network.target/start
[ 3.255123] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.262958] systemd[1]: Found dependency on basic.target/start
[ 3.270518] systemd[1]: Breaking ordering cycle by deleting job kbd.service/start
[ 3.281399] systemd[1]: Job kbd.service/start deleted to break ordering cycle starting with basic.target/start
[ 3.298003] systemd[1]: Found ordering cycle on basic.target/start
[ 3.306175] systemd[1]: Found dependency on sysinit.target/start
[ 3.314134] systemd[1]: Found dependency on raspi-config.service/start
[ 3.322582] systemd[1]: Found dependency on remote-fs.target/start
[ 3.330677] systemd[1]: Found dependency on mnt-nas.mount/start
[ 3.338467] systemd[1]: Found dependency on network.target/start
[ 3.346300] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.354099] systemd[1]: Found dependency on basic.target/start
[ 3.361624] systemd[1]: Breaking ordering cycle by deleting job raspi-config.service/start
[ 3.373313] systemd[1]: Job raspi-config.service/start deleted to break ordering cycle starting with basic.target/start
[ 3.390077] systemd[1]: Found ordering cycle on basic.target/start
[ 3.398201] systemd[1]: Found dependency on sysinit.target/start
[ 3.406150] systemd[1]: Found dependency on console-setup.service/start
[ 3.414724] systemd[1]: Found dependency on remote-fs.target/start
[ 3.422825] systemd[1]: Found dependency on mnt-nas.mount/start
[ 3.430707] systemd[1]: Found dependency on network.target/start
[ 3.438677] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.446610] systemd[1]: Found dependency on basic.target/start
[ 3.454285] systemd[1]: Breaking ordering cycle by deleting job console-setup.service/start
[ 3.466139] systemd[1]: Job console-setup.service/start deleted to break ordering cycle starting with basic.target/start
[ 3.483119] systemd[1]: Found ordering cycle on basic.target/start
[ 3.491219] systemd[1]: Found dependency on sysinit.target/start
[ 3.499192] systemd[1]: Found dependency on rpcbind.service/start
[ 3.507244] systemd[1]: Found dependency on network-online.target/start
[ 3.515827] systemd[1]: Found dependency on network.target/start
[ 3.523802] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.531736] systemd[1]: Found dependency on basic.target/start
[ 3.539491] systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/start
[ 3.550976] systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start

[larch]

Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.

Hi Shift - yes this is easy a clean install plus:

apt-get install -y nfs-kernel-server nfs-common

Breaks this feature once you add an export.

[chme]

I can also reproduce this issue with a clean jessie-lite image on my rpi 1 model B.
I just have to add a nfs mount in /etc/fstab and enable the rpcbind service with "sudo systemctl enable rpcbind".

The entry in /etc/fstab looks like:

Code: Select all

192.168.178.24:/DataVolume/Public /public nfs defaults,noatime,noauto,x-systemd.automount 0 0

On reboot the rpcbind service does not start automatically and dmesg shows following error:

Code: Select all

[    3.408744] systemd[1]: Found dependency on network.target/start
[    3.416628] systemd[1]: Found dependency on dhcpcd.service/start
[    3.424401] systemd[1]: Found dependency on basic.target/start
[    3.432044] systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/start
[    3.443360] systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start

Starting the service manually succeeds.

There is another report on this forum, that seems to be related:
viewtopic.php?f=66&t=128816

[beta-tester]

Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.

i just noticed, that there is activity on this thread...
this is ma initial post nfs-kernel-server not starting anymore since few days/weeks.

as workaround, i installed a fresh Raspbian Jessie image, and marked the dhcpcd5 package as HOLD with

Code: Select all

sudo apt-mark hold dhcpcd5

then it is save to install everything newly and get rid of the dependency ordering circle,
even after an apt-get upgrade.
but that means i do not get any updates for dhcpcd5.

i not figured out, what steps exactly triggers that issue, because i use a bigger script, to install/configure my RPi as PXE server from scratch,
but as larch sayed

Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.

Hi Shift - yes this is easy a clean install plus:
apt-get install -y nfs-kernel-server nfs-common
Breaks this feature once you add an export.

those kind of steps are also included on my script.

Code: Select all

...
sudo apt-get -y install nfs-kernel-server
sudo apt-get -y install pxelinux syslinux-common
sudo apt-get -y install dnsmasq
sudo apt-get -y install samba
...

plus some fstab and exports.

Code: Select all

## /etc/fstab
/srv/tftp/iso/win-pe-x86.iso                  /srv/tftp/nfs/winpe         auto  loop,ro  0  0
/srv/tftp/iso/ubuntu-15.10-desktop-amd64.iso  /srv/tftp/nfs/ubuntu/amd64  auto  loop,ro  0  0
...

Code: Select all

## /etc/exports
/srv/tftp                   *(ro,no_subtree_check,async,no_root_squash)
/srv/tftp/nfs/winpe         *(ro,no_subtree_check,async,no_root_squash)
/srv/tftp/nfs/ubuntu/amd64  *(ro,no_subtree_check,async,no_root_squash)
...

[beta-tester]

dropped way down in the threads again...
bump...

[ShiftPlusOne]

Thanks for all the extra info. It will get sorted.

[sielenk]

As I found out trying to mount a windows (samba) share:
The cycle problem with systemd gets triggered by adding a cifs mount in fstab, too.

[ShiftPlusOne]

At this stage, it appears to be a bug in the way Debian handles sysvinit scripts in systemd.

https://bugs.debian.org/cgi-bin/bugrepo ... bug=763315
https://bugs.debian.org/cgi-bin/bugrepo ... bug=622394
https://bugs.debian.org/cgi-bin/bugrepo ... bug=761951

[beta-tester]

At this stage, it appears to be a bug in the way Debian handles sysvinit scripts in systemd.

https://bugs.debian.org/cgi-bin/bugrepo ... bug=763315
https://bugs.debian.org/cgi-bin/bugrepo ... bug=622394
https://bugs.debian.org/cgi-bin/bugrepo ... bug=761951

thank you for the links and spending your time to this...
but all these links were initially opened between 2011 and 2014...
i would expect, that these should be fixed in the meantime.
as i said, if i install everything to a virgine Raspbian Jessie (2015-11-21) with marked dhcpcd5 on hold, everything is working without any problems.
i run in trouble only, when the dhcpcd5 package is updating.
i mean, it looks to me like a very new issue, and not an old one.

[ShiftPlusOne]

http://unix.stackexchange.com/questions ... ycle-start

It is a debian bug that we've hit by making a change to dhcpcd. Fundamentally, the problem is as described in those bug reports. The system wants all the sysvinit-style scripts from that runlevel started first, and one of them relies on having networking ready, which the dhcpcd service starts before. This can't happen, since the network won't be up before dhcpcd is started. The fix might be replacing all the nfs-related services with dhcpcd units.

Don't know if this will be addressed in the upcoming image, the one after that or when it's fixed in debian. Still looking at the options.

[dagon666]

Until we'll get a true solution for this problem, I've hacked a simple python script which restarts the rpcbind and the nfs-kernel-server after reboot, if it's not up. Seems to do the job for me and maybe it will help some of you.

https://github.com/dagon666/rpcbindplumber

[beta-tester]

... if i install everything to a virgine Raspbian Jessie (2015-11-21) with marked dhcpcd5 on hold, everything is working without any problems.
i run in trouble only, when the dhcpcd5 package is updating.

damn, with the new Raspbian from 2016-02-XX i run immediately into trouble with no chance to do my workaround of last time with hold dhcpcd5 package to keep things running...
ok, as new WORKAROUND i now (re)start my broken services via rc.local command entry after boot...

Code: Select all

# /etc/rc.local
...
sudo service nfs-kernel-server restart &
exit 0

when is debian fixing that...

[359]

Hi.. After tearing out hair, I found the post above from beta-tester, I'm up and running with nfs server running automatically, thank you to you beta-tester!

The post above from February 11th, wondering if there is any other better solution, I'm on the latest Jessie Lite on Pi3 now.

Thank you if anyone can update the situation

[DirkS]

The post above from February 11th, wondering if there is any other better solution, I'm on the latest Jessie Lite on Pi3 now.

Not AFAIK. Only workarounds, no solutions

[vahagn]

I am absolute novice in systemd and had to spend quite time to figure out this solution. But here is my workaround.
Copy rpcbind.service (and possibly other dependent services e.g. nfs-common.service, nfs-kernel-server.service) from /run/systemd/generator.late/ to /lib/systemd/system. These are the services that sysv generator creates automatically for systemd. Next edit the files to remove dependency from sysinit.target which causes the loop. Instead I added:

[Install]
WantedBy=multi-user.target

Then install these services through "sudo systemctl enable <service name>". Basically, that's it. It worked for me.

[charango]

Hello there, I stjumbled upon this thread while looking for a solution to a problem I have which is very similar to the one posted here. Let me expalain:
I am a fairly new Debian user, I installed Jessie on a dual boot system with Win 10. After installing a few packages I got a systemd ordering cycle error message at boot which I am trying to understand and solve. It is very similar to the one evoked here. Could some good soul have a look at the log and point me in the right direction for a fix? Thanks very much!
Here is the message in the log:

May 9 05:34:48 debian systemd[1]: Found ordering cycle on sysinit.target/stop
May 9 05:34:48 debian systemd[1]: Found dependency on rpcbind.service/stop
May 9 05:34:48 debian systemd[1]: Found dependency on network-online.target/stop
May 9 05:34:48 debian systemd[1]: Found dependency on NetworkManager-wait-online.service/stop
May 9 05:34:48 debian systemd[1]: Found dependency on basic.target/stop
May 9 05:34:48 debian systemd[1]: Found dependency on sysinit.target/stop
May 9 05:34:48 debian systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/stop
May 9 05:34:48 debian systemd[1]: Job rpcbind.service/stop deleted to break ordering cycle starting with sysinit.target/stop

[beta-tester]

at the moment, my workaround to get my network work again is to restart the nfs-kernel-server service after each reboot.
for that i patched the /etc/rc.local file by adding an addition line with the restart of the service...

Code: Select all

# /etc/rc.local
...

# fix for systemd dependency cycle
sudo service nfs-kernel-server restart &

exit0

but i think it depends, what service is broken with the dependency ordering cycle...

[charango]

Thanks for the reply, yes of course I understand it depends on the service that is "broken" so can't you tell by looking at my specific error message where the problem lies? Everything seems to be working when I boot, I mean I can go on the internet, so my network is not down. So I don't know if your fix (nfs-kernel-server) will be suitable for my system, will it?

[DirkS]

Thanks for the reply, yes of course I understand it depends on the service that is "broken" so can't you tell by looking at my specific error message where the problem lies? Everything seems to be working when I boot, I mean I can go on the internet, so my network is not down. So I don't know if your fix (nfs-kernel-server) will be suitable for my system, will it?

I have the same error and to make sure I actually restart both services.
I added

Code: Select all

systemctl restart rpcbind
systemctl restart nfs-kernel-server

before the 'exit 0' in rc.local. (note that 'sudo' is not needed)

[charango]

Thanks again for the help but those two commands don't seem to change anything. After executing the first one there was no message and the second one said there was no such file.

[beta-tester]

Everything seems to be working when I boot, I mean I can go on the internet, so my network is not down. So I don't know if your fix (nfs-kernel-server) will be suitable for my system, will it?

does it mean, you do not notice any downsides/impact in your work...
nothing seems to be broken or you do not run into any problems...
the only thing you can see is the bool log...
so for you it is more or less a cosmetical issue only?

what did you installed, that caused that dependency ordering cycle?

if you don't see any impacts, i wouldn't care about...

[charango]

what did you installed, that caused that dependency ordering cycle?

That's a good question. If I knew how it happened I would be nearer a fix.
It seems only a cosmetic thing as you say but I hate to have something nagging at me like that, I like to know why things happen.
I have only installed Debian since Sunday and there is a certain instability sometimes, for example the usb mouse will act strangely or the window I am working on will freeze up...I don't know if any of these problems are related..I am slowly trying to get accustomed to a linux system again after a few years away from it.

[beta-tester]

me too, i also like to know, why things happen and i hate error messages and warnings.
but sometimes you can't do anything as normal user

PS.: i don't think that the issues you have are related to the dependency order cycle issue. otherwise, you would see more and other additional errors in the logs...

PPS.: once i installed a package, that caused the issue, even with uninstalling that installed package(s) completely again didn't fix that issue.
only a new clean installation of the OS solved the issue. but i needed to install the package again, because i need its service. so i have to live with that issue and the workaround until a fix is available...