1ma
Posts: 2
Joined: Fri Jul 02, 2021 8:40 pm

network not working during cloud-init provisioning

Fri Jul 02, 2021 9:06 pm

I bough a Raspberry Pi 4 to use it as a home server (no peripherals attached, only the Ethernet cable), and I'm trying to provision a fresh install of Ubuntu with cloud-init. However it seems like all network-related operations are failing (downloading SSH pubkeys from GitHub, updating the APT repos, installing packages, etc.).

The weird thing is that if I ssh'd into the Raspberry and manually do what cloud-init is supposed to do the network works (e.g. "sudo apt-get install syncthing" or "ssh-import-id gh:1ma"), even without needing to reboot. So I'm a bit of a loss at how to identify the problem.

Troubleshooting Info
Model: Raspberry Pi 4 8GB model rev1.4
OS: Ubuntu Server 20.04.2 LTS for Raspberry Pi (https://ubuntu.com/download/raspberry-pi)

my custom userdata:

Code: Select all

#cloud-config

users:
  - default
  - name: syncthing
    gecos: Syncthing System User
    sudo: false
    shell: /usr/sbin/nologin
  - name: marcel
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_import_id: gh:1ma
    ssh_authorized_keys:
      - <a hardcoded SSH pubkey for logging in, since ssh_import_id from GitHub does not work ATM>

apt:
  sources:
    syncthing:
      source: "deb https://apt.syncthing.net/ syncthing stable"
      keyid: 37C84554E7E0A261E4F76E1ED26E6ED000654A3E

packages:
  - apt-transport-https
  - syncthing

apt_update: true
apt_upgrade: true
/var/log/cloud-init.log (too big to paste it inline): https://pastebin.com/raw/6WP7XuBD

/var/log/cloud-init-output.log:

Code: Select all

Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'init-local' at Wed, 01 Apr 2020 17:23:49 +0000. Up 14.14 seconds.
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'init' at Wed, 01 Apr 2020 17:23:52 +0000. Up 16.58 seconds.
ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: |  eth0  | False |     .     |     .     |   .   | e4:5f:01:1b:53:6c |
ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |
ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
ci-info: | wlan0  | False |     .     |     .     |   .   | e4:5f:01:1b:53:6d |
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: +-------+-------------+---------+-----------+-------+
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:GvQPCLeJP606fQ2tu+hXhk030UF6Fit7m5jZk/tvwEA root@ubuntu
The key's randomart image is:
+---[RSA 3072]----+
|             o+. |
|            E..o |
|    . o    .o.+  |
|     = =  . +=   |
|    . = S= ..+.  |
|     . +oo=  *o+ |
|     .+ .*. + *. |
|    . .++ .    o.|
|    .++ooo    .o+|
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub
The key fingerprint is:
SHA256:a9l16tIsdDlLS4iT+bHP6AtuucCdTXBuXZO/sdE32OM root@ubuntu
The key's randomart image is:
+---[DSA 1024]----+
|                 |
|               . |
|        . .   +  |
|         + . .oo.|
|        S++.oo.*+|
|     . .=O+.*o. O|
|      o B=oO.+ E |
|       +o.=+=    |
|       .oo+=+    |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:/1yGCV/vUGffX9sVM62rmyo49OoMPPGHTdxbPp63eBc root@ubuntu
The key's randomart image is:
+---[ECDSA 256]---+
|                 |
|                 |
|                 |
|       . .      .|
|    .   S o . .+=|
|   . o.+ . * + EB|
|    +.ooo o * = B|
|     +o.o  + Bo+B|
|     .+o ...X=o++|
+----[SHA256]-----+
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
The key fingerprint is:
SHA256:cpS1ezZYI8x+K0CSxMUvSwcp8B0Dd0qXmKiSAhcyDUI root@ubuntu
The key's randomart image is:
+--[ED25519 256]--+
|*E...oo=*++.     |
|ooo  o+=*@..     |
|.. . .+.*o= o    |
|. o .  +o.o= .   |
| . .  ..S++ =    |
|       o.. + o   |
|          . .    |
|           .     |
|                 |
+----[SHA256]-----+
2020-04-01 17:24:09,458 ERROR HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/1ma/keys (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'modules:config' at Wed, 01 Apr 2020 17:24:07 +0000. Up 32.21 seconds.
2020-04-01 17:24:09,573 - util.py[WARNING]: Failed to run command to import marcel SSH ids
2020-04-01 17:24:09,577 - util.py[WARNING]: ssh-import-id failed for: marcel ['gh:1ma']
2020-04-01 17:24:09,579 - util.py[WARNING]: Running module ssh-import-id (<module 'cloudinit.config.cc_ssh_import_id' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_import_id.py'>) failed
Ign:1 https://apt.syncthing.net syncthing InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Err:3 https://apt.syncthing.net syncthing Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 82.196.13.137 443]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease [101 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
Reading package lists...
E: The repository 'https://apt.syncthing.net syncthing Release' does not have a Release file.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/focal/InRelease is not valid yet (invalid for another 22d 0h 8min 55s). Updates for this repository will not be applied.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/focal-updates/InRelease is not valid yet (invalid for another 457d 1h 39min 40s). Updates for this repository will not be applied.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/focal-backports/InRelease is not valid yet (invalid for another 457d 1h 22min 19s). Updates for this repository will not be applied.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/focal-security/InRelease is not valid yet (invalid for another 457d 0h 55min 48s). Updates for this repository will not be applied.
2020-04-01 17:24:27,760 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  apt-transport-https syncthing
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 5625 kB of archives.
After this operation, 16.7 MB of additional disk space will be used.
Err:1 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 apt-transport-https all 2.0.4
  404  Not Found [IP: 91.189.91.38 80]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 syncthing arm64 1.1.4~ds1-4ubuntu1 [5624 kB]
Fetched 5624 kB in 2s (3405 kB/s)
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/a/apt/apt-transport-https_2.0.4_all.deb  404  Not Found [IP: 91.189.91.38 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'modules:final' at Wed, 01 Apr 2020 17:24:29 +0000. Up 53.46 seconds.
2021-07-02 20:12:33,506 - util.py[WARNING]: Failed to install packages: ['apt-transport-https', 'syncthing']
2021-07-02 20:12:33,577 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
2021-07-02 20:12:33,579 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
ci-info: no authorized SSH keys fingerprints found for user syncthing.
ci-info: no authorized SSH keys fingerprints found for user ubuntu.
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 finished at Fri, 02 Jul 2021 20:12:33 +0000. Datasource DataSourceNoCloud [seed=/dev/mmcblk0p1][dsmode=net].  Up 59.02 seconds

epoch1970
Posts: 8590
Joined: Thu May 05, 2016 9:33 am
Location: France

Re: network not working during cloud-init provisioning

Sat Jul 03, 2021 12:18 pm

At least in the syncthing install the problem appears to come from system date (04/2020)

Code: Select all

 Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 82.196.13.137 443]
I’ve no idea how cloud init works.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

1ma
Posts: 2
Joined: Fri Jul 02, 2021 8:40 pm

Re: network not working during cloud-init provisioning

Tue Oct 12, 2021 10:49 am

I had another go at this and this time I drilled deep enough to indentify the problem. The critical pieces of information I missed last time were:

* The Raspberry Pi does not have a hardware clock, and whenever it boots its time is incorrect. It relies on NTP to get the correct time from the internet.
* APT needs the correct current time in order to work.
* Back in 2016 the Cloud Init maintainers moved the NTP setup stage down after the APT configuration stage, because someone was dealing with an Ubuntu image without an NTP client and they could not install the NTP package before setting up APT. But obviously they had a hardware clock and the APT configuration worked for them. Since Cloud Init made this change its been broken as-is for the Raspberry.

This explains why I wasn't able to reproduce the problem. Once I SSH'ed into the Raspberry the NTP service was already humming and all APT commands I ran manually were already working.



Now, to fix this mess I needed to do two things. First, enabling the NTP setup module in my cloud-config template. The official Ubuntu 20.04 image for the Raspberry comes with the NTP client preinstalled, so no problem here. This directive simply syncs the device's datetime.

Code: Select all

#cloud-config

ntp:
  enable: true

...
Then, after flashing the SD card but before booting it for the first time, I had to edit the /etc/cloud/cloud.cfg file and undo the change they made in 2016, moving the ntp stage up before apt-pipelining and apt-configure. Without this the datetime synchronization happens too late and the APT configuration stage fails despite having activated the ntp stage.

Image

Return to “Ubuntu”