Rasbian Lite DNS Resolution sometimes takes forever

[TechnoSwiss]

Interesting DNS problem that I don't know if I've ever seen before.

I have 4 Rapberry Pi 4 4GB models. I set all 4 up at the same time (created an installation wiki when installing the first one, then followed that for the next 3), so they should all have identical installations on them. I use the Rasbian Lite image (don't recall which one I downloaded, pulled it down for this install in July) and used Etcher to write that image to the SD cards. They're all currently on Rasbian 10.

The problem had mostly been happening when they were attempting to connect to the OpenVPN server, they would hang in the client service trying to resolve the DNS for the server, and just stick there. I have the config setup with a second server address which is just a static IP address, but they would never attempt to connect to that address because they would never fail the connection to the first server, they would just hang trying to resolve DNS. I'd run over to the building, SSH into the unit from the local network, and to "fix" the problem I'd ping google.com, wait a second ctrl-c out of then when it wouldn't resolve google.com, ping google.com again, ctrl-c out, then usually on the 3rd time I'd ping google.com and it would resolve the DNS, and then OpenVPN client would also resolve the address and connect. If I just let that first ping keep running and not ctrl-c out, after about 2 min. it would finally time out with an error that it was unable to resolve the address, or occasionally after a minute or two it would finally resolve. Sometimes I'd have a problem when I went to start the live stream where it would sit on the first connection to YouTube (and API call to get the stream ID) for 1-2 minutes before the connection finally went through. This last Sunday one of the Raspberry Pi's instead of just delaying starting the stream, it would just fail to stream, and I'd have to go through the ping process to "unlock" the DNS. (rebooting the system didn't seem to solve the issue, the ping trick seems to be the only way to solve this)

It's a pretty minimal install, they have OpenVPN installed on them, and a source version of ffmpeg compiled on the Raspberry Pi's for doing YouTube live broadcasts. They're installed at 4 different locations (being used for live streams of Sunday services because of the pandemic). They're all connected to a Meraki managed switch, and that switch is then connected to a Meraki firewall. Three of the units are on Ziply fiber service, one is on Comcast cable service. The problem I'm seeing was first noticed after I set them up on my home network, which they were connected to a Netgear 8-port switch, and that into a Netgear R7000 router running Asuswrt-merlin port Xwrt-Vortex versin 380.69 version. All units were hard wired on my home network, no wifi setup on them, one of the units was setup for wifi when it was setup in the building it's located in, but has since been moved to wired.

I am manually setting the nameserver on all of these units (a change I made after having the problem during testing at home), and they're all set to the following:
nameserver 1.1.1.1
nameserver 8.8.8.8
nameserver 8.8.4.4

I have disabled the avahi-dameon on all these units using systemctl disable avahi-daemon in case that might have been causing some conflicts with the DNS. I thought I had disabled IPV6 (none of our ISPs currently offer IPV6, at least in our area) by adding 'ipv6.disable=1' but I just noticed when I checked to verify IPV6 was disabled that it didn't seem to work, so maybe that's still my issue.

I've seen a lot of issues with DNS in the past, but I just don't recall ever seeing this particular issue, where DNS seems to just get stuck, it doesn't really error out, it just hangs everything up until. I haven't had any luck finding a similar issue online (or the couple that I found suggested setting the nameserver manually, which was the first thing I tried). Anybody ever seen this before, have any ideas what I might have miss-configured?

Thanks

[TechnoSwiss]

Okay, disabled IPv6 by creating the file /etc/sysctl.d/disable-ipv6.conf and adding "net.ipv6.conf.all.disable_ipv6 = 1" to it.

DNS resolution still taking forever occasionally.

[epoch1970]

Most likely reason would be 53/udp to/from external addresses being blocked by the router/firewall. Sometimes by the ISP, even.
Piling up nameservers in resolv.conf doesn’t help a lot in general. If multiple upstream servers are a requirement, consider using a cache forwarder, like dnsmasq.

Up to MAXNS (currently 3, see <resolv.h>) name servers may be listed, one per keyword. If there are multiple servers, the resolver library queries them in the order listed. ... The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made.

[TechnoSwiss]

Several weeks of trying to solve this, and of course after I post I find what might at least be a similar issue, however I don't get the "ping: www.google.com: Temporary failure in name resolution" error, but the other symptoms do seem to match up.

viewtopic.php?t=273602

Adding options single-request-reopen to /etc/resolv.conf doesn't solve the issue.

[TechnoSwiss]

Most likely reason would be 53/udp to/from external addresses being blocked by the router/firewall. Sometimes by the ISP, even.
Piling up nameservers in resolv.conf doesn’t help a lot in general. If multiple upstream servers are a requirement, consider using a cache forwarder, like dnsmasq.

Up to MAXNS (currently 3, see <resolv.h>) name servers may be listed, one per keyword. If there are multiple servers, the resolver library queries them in the order listed. ... The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made.

So possibly the fix is using the ISP's DNS server instead of using Google or Cloudflare? I'll give that a shot.

[TechnoSwiss]

Hmm... if I use the DNS server that's being handed out via DHCP by the firewall, the problem remains the same. It's also the same if I use the DNS server provided by the ISP.

If I use the OpenVPN server as the DNS server (it's connected to the same Comcast provided router that the firewall is connected which one of these Rasberry Pi's is on) then if I do a ping, the name resolution happens in a couple of seconds, and then another 10-15 second before the ping actually starts happening (the first response still says 5-10ms even though I'm timing at least 10 seconds before it starts) If I ping the IP address directly it starts in what appears to be instantaneously, with reported times between 5-10ms.

[TechnoSwiss]

I took a Windows laptop and Android phone over to the building on Comcast and tried using the DNS version that is set by DHCP, and then using 1.1.1.1 as the DNS server, and in both cases they resolve the DNS what appears to be instantaneously, so I wouldn't expect either the firewall or ISP to be blocking port 53. I setup stubby on one of the Raspberry Pi's to see if DNS over TLS was any faster/different, and I'm still seeing the same delay issues with stubby.

[DougieLawson]

Okay, disabled IPv6 by creating the file /etc/sysctl.d/disable-ipv6.conf and adding "net.ipv6.conf.all.disable_ipv6 = 1" to it.

DNS resolution still taking forever occasionally.

So it's clearly nothing to do with the 21 year old protocol that you've disabled.
Try adding some IPv6 resolvers.

Code: Select all

nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

[TechnoSwiss]

Try adding some IPv6 resolvers.

Code: Select all

nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

Happy to try anything at this point as I'm really at a loss as to why it's only these Raspberry Pi's on the network having resolution issues, however since neither ISP involved here supports ipv6 at the endpoint (Ziply finally announced they'll be turning it on next month in the next town over) is turning ipv6 back on and setting ipv6 nameservers going to do anything other than potentially add delay as it tries name servers it can't hit because there's no ipv6 support out to the internet from these locations?

[DougieLawson]

If you don't have a /64 IPv6 prefix then having IPv6 enabled or not makes not a jot of difference.

The most likely delay is the Cloudflare resolver at 1.1.1.1. As soon as it fails to resolve a public address you'll wait for the timeout before forwarding to Google @ 8.8.8.8.

[TechnoSwiss]

Yup, I originally only had the Google 8.8.8.8 nameserver in the list, after having the resolution delay issues and doing some searching I found a couple of people suggesting that Google could be kind of slow, and to use Cloudflair's 1.1.1.1 nameserver because it was much faster. So I've had both setup as the only nameserver, and have the same delays either way.

[DougieLawson]

Google has always been as fast as a roadrunner with Wile E. Coyote chasing him for me.

You can change the fallback timeout with

Code: Select all

options timeout:30

in /etc/resolv.conf

[TechnoSwiss]

I got dnsutils installed and have been trying dig and nslookup

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @1.1.1.1 youtube.com A
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @8.8.8.8 youtube.com A
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @8.8.4.4 youtube.com A
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

nslookup youtube.com also returns ;; connection timed out; no servers could be reached

So I ping youtube.com and 2 min. later it finally resolves and the ping starts, I ctrl-c out of that and then run dig and nslookup again, and everthing still reports timeouts.

Is there a way to find out how ping is eventually resolving youtube.com?

[TechnoSwiss]

You can change the fallback timeout with

Code: Select all

options timeout:30

in /etc/resolv.conf

Minute and 30 seconds for ping to resolve, which I believe makes sense if it has a 30 second timeout on 3 servers.

[DougieLawson]

It's odd that your ISP is breaking your connection to Cloudflare and Google's resolver (unless they're trying to impose some form of national firewall restrictions on you).
Is something (invalidly) blocking port UDP 53 or TCP 53?
Does your ISP have their own local resolver?

I have no problems digging around for Youtube

Code: Select all

dougie@apollo:~$ dig @8.8.8.8 youtube.com A

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @8.8.8.8 youtube.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6822
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;youtube.com.                   IN      A

;; ANSWER SECTION:
youtube.com.            238     IN      A       216.58.210.238

;; Query time: 15 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Nov 25 22:18:24 UTC 2020
;; MSG SIZE  rcvd: 56

dougie@apollo:~$ dig @8.8.4.4 youtube.com A

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @8.8.4.4 youtube.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51987
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;youtube.com.                   IN      A

;; ANSWER SECTION:
youtube.com.            299     IN      A       172.217.169.46

;; Query time: 18 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Wed Nov 25 22:18:34 UTC 2020
;; MSG SIZE  rcvd: 56

dougie@apollo:~$ dig @1.1.1.1 youtube.com A

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @1.1.1.1 youtube.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50446
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;youtube.com.                   IN      A

;; ANSWER SECTION:
youtube.com.            187     IN      A       216.58.204.238

;; Query time: 9 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Nov 25 22:18:41 UTC 2020
;; MSG SIZE  rcvd: 56

dougie@apollo:~$ dig @9.9.9.9 youtube.com A

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> @9.9.9.9 youtube.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3273
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;youtube.com.                   IN      A

;; ANSWER SECTION:
youtube.com.            208     IN      A       216.58.212.238

;; Query time: 9 msec
;; SERVER: 9.9.9.9#53(9.9.9.9)
;; WHEN: Wed Nov 25 22:18:52 UTC 2020
;; MSG SIZE  rcvd: 56

dougie@apollo:~$

[TechnoSwiss]

I've been able to resolve DNS against 1.1.1.1 and 8.8.8.8 from a Windows PC, so I don't think the firewall or ISP are blocking them (although I haven't tested that since a couple months ago when I first was seeing this problem, I should run over to one of the buildings tonight and try it with a Debian systems).

Followup on changing the timeout option, if I set that to:

Code: Select all

options timeout:1

Then a ping to youtube.com still takes a minute 30 seconds to resolve and start.

[TechnoSwiss]

Okay, so I did a tcpdump while trying to ping youtube.com.

It looks like it tries doing a lookup against 8.8.8.8 port 53 8 times, then it switches to doing a TLS lookup against port 853, that resolves the name and the timing lines up with the delay I'm seeing (which makes sense).

So that would seem to be proof that port 53 is getting blocked somewhere, and that's what's causing the name resolution delays, and DoT gets around this.

It does however raise a couple of questions still:
1) If I use the DHCP provided nameserver from the firewall, I still get this delay issue (I should do a tcpdump on that one)
2) I had tried doing DoT earlier with stubby with the same delay results (again should look at a tcpdump there to see what's going on)
3) How is the system switching to DoT, I didn't think the default resolver on Rasbian could do DoT (the reason I tried stubby)

[DougieLawson]

You really need to discover what's blocking DNS and forcing you to DNS over TLS.

What does a sudo resolvectl status command give you?
You can change settings in /etc/systemd/resolved.conf

[TechnoSwiss]

This is after switching the unit I've been running testing on back over to the DHCP provided nameserver (or rather removing the static nameservers and letting DHCP assign it). The listed nameservers are for Zscaler, a company that provides DNS based content filtering, which I'm fairly certain is getting set by the firewall, and so I would assume the firewall is the culprit for the port 53 block, to make sure eveyrthing is going though the Zscaler DNS.

With the firewall provided DNS servers if I ping youtube.com I'm getting about 30ms response times, although it resolves the address, and then there's a pause for about 20 seconds before the first ping response comes back, although the response time doesn't reflect that 20 second pause. That 20 second pause I assume is still something related to name resolution because if I ping 172.27.5.110 there is not pause. (should to a packet capture on that I suppose as well)

Code: Select all

PING youtube.com (172.217.5.110) 56(84) bytes of data.
64 bytes from 172.217.5.110 (172.217.5.110): icmp_seq=1 ttl=111 time=31.10 ms
64 bytes from 172.217.5.110 (172.217.5.110): icmp_seq=2 ttl=111 time=31.4 ms
64 bytes from 172.217.5.110 (172.217.5.110): icmp_seq=3 ttl=111 time=30.9 ms
64 bytes from 172.217.5.110 (172.217.5.110): icmp_seq=4 ttl=111 time=31.8 ms
64 bytes from 172.217.5.110 (172.217.5.110): icmp_seq=5 ttl=111 time=31.3 ms
64 bytes from 172.217.5.110 (172.217.5.110): icmp_seq=6 ttl=111 time=31.3 ms

From the resolvectl status it looks to me like DoT is turned off:

Code: Select all

Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 8.35.35.101
         DNS Servers: 8.34.34.101
                      8.35.35.101
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (tun0)
      Current Scopes: LLMNR/IPv4
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 3 (wlan0)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 2 (eth0)
      Current Scopes: LLMNR/IPv4
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

and the contents of resolved.conf also look like DoT is disabled, so still curious why it's eventually failing over to DoS

Code: Select all

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=yes
#MulticastDNS=yes
#DNSSEC=allow-downgrade
#DNSOverTLS=no
#Cache=yes
#DNSStubListener=yes
#ReadEtcHosts=yes

[DougieLawson]

ZScaler works by using DNS poisoning (among other things) to prevent you reaching unapproved sites (it's a whitelist system). That explains why you can't use 1.1.1.1 or 8.8.8.8 or 8.8.4.4 or 9.9.9.9 on your tightly controlled "corporate" network.

You should have told us that it's behind ZScaler earlier it would have saved wasting my time.

[TechnoSwiss]

Yup, if I had realized we were behind Zscaler earlier I would have, that must be a recent change.

Any idea on why it's eventually resolving against 8.8.8.8 via DoT when DoT is not enabled? I understand it's getting compiled into newer releases of the systemd resolver, a but as far as I can tell it's still disabled on these systems.

Thanks for the help and insight.

[DougieLawson]

Ask ZScaler support.