I recently acquired some new desktop hardware, and went about imaging a new Fedora 43 workstation for my home "office" (an office in an NYC apartment is a lonely corner of the room designated for work). Everything has been working rather swimmingly; even the venerable RTX 2080 Super I have installed worked with minimal coaxing. This might be the year of the linux desktop!
Everything was coming up Milhouse for a few days, until I rebooted to complete the installation of some application and system updates. On reboot, I found that my internet connection was reporting a loss of connectivity. Indeed, I was unable to browse to any website in my web browser.
Putting on the ol' IT thinking cap, I brought up a terminal and started doing some basic network troubleshooting.
❯ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=20.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=18.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=14.7 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 14.725/17.822/20.707/2.446 ms
❯ ping google.com
ping: google.com: Temporary failure in name resolution
It's always DNS™. Digging into my DNS configuration with nmcli:
❯ nmcli
...
DNS configuration:
servers: 192.168.1.1
domains: lan
...
That's definitely the DNS server (my ISP router).
❯ nslookup google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused
;; no servers could be reached
❯ nslookup google.com 192.168.1.1
Server: 192.168.1.1
Address: 192.168.1.1#53
Non-authoritative answer:
Name: google.com
Address: 142.250.80.110
Name: google.com
Address: 2607:f8b0:4006:820::200e
"Why am I directing DNS requests to localhost?" I initially thought. I went to check /etc/resolv.conf.
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search lan
Of course, me and my old fashioned ideas. Fedora switched to using systemd-resolved for DNS in 2020. In the caveman days, we checked /etc/resolv.conf for the address of the DNS server, but via systemd-resolved, a systemd service handles DNS requests. Through this service, we theoretically benefit from its own DNS cache, easier per interface DNS server configuration, better integration with other systemd services, security via DNS over TLS, support for multicast DNS, etc.
Right now, though, it appears to be busted. What's happening? Let's take a look at the journalctl logs
Jan 03 16:45:50 redbird systemd-resolved[1812]: Failed to connect to system bus: Permission denied
Jan 03 16:45:50 redbird systemd-resolved[1812]: Could not create manager: Permission denied
Jan 03 16:45:50 redbird systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE
Jan 03 16:45:50 redbird systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Jan 03 16:45:50 redbird systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution.
Jan 03 16:45:50 redbird systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
Jan 03 16:45:50 redbird systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Jan 03 16:45:50 redbird systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Jan 03 16:45:50 redbird systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution.
Jan 03 16:45:54 redbird systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Jan 03 16:45:54 redbird systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Jan 03 16:45:54 redbird systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution.
Looks like it's just trying to start up too quickly, seemingly before D-Bus (a message system by which Linux services communicate) has initialized. If we restart it now, everything should be fine.
❯ sudo systemctl restart systemd-resolved
❯ nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: google.com
Address: 142.251.41.174
Name: google.com
Address: 2607:f8b0:4006:817::200e
There's one good thing about a manually configured DNS server file; a file tends not to try to start-up too quickly and then crash. Anyhow, I'll add an override to ensure the services starts after dbus.service.
[Unit]
After=dbus.service
We'll put the above into the file /etc/systemd/system/systemd-resolved.service.d/override.conf. Using an override file allows us to add options like the above without messing with the base file. After running a sudo systemctl daemon-reload, we can reboot and see if it works.
I won't keep you in suspense, DNS worked on startup. Let's see what's going on in the logs again.
Jan 03 17:25:33 redbird systemd[1]: [🡕] sysinit.target: Found ordering cycle: systemd-resolved.service/start after dbus-broker.service/start after dbus.socket/start after sysinit.target/start - after systemd-resolved.service
Jan 03 17:25:33 redbird systemd[1]: [🡕] sysinit.target: Job systemd-resolved.service/start deleted to break ordering cycle starting with sysinit.target/start
Jan 03 17:25:34 redbird systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Jan 03 17:25:34 redbird systemd-resolved[1887]: Positive Trust Anchors:
Jan 03 17:25:34 redbird systemd-resolved[1887]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jan 03 17:25:34 redbird systemd-resolved[1887]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16
Jan 03 17:25:34 redbird systemd-resolved[1887]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 2>
Jan 03 17:25:34 redbird systemd-resolved[1887]: Using system hostname 'redbird'.
Jan 03 17:25:34 redbird systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Jan 03 17:25:38 redbird systemd-resolved[1887]: wlo2: Bus client set search domain list to: lan
Jan 03 17:25:38 redbird systemd-resolved[1887]: wlo2: Bus client set default route setting: yes
Jan 03 17:25:38 redbird systemd-resolved[1887]: wlo2: Bus client set DNS server list to: 192.168.1.1
Jan 03 17:25:40 redbird systemd-resolved[1887]: wlo2: Bus client set DNS server list to: 192.168.1.1, 2603:7000:a700:c45::1
As you can see in the first two lines here, I've created a circular dependency. sysinit.target actually lists systemd-resolved as one of its dependencies (i.e. one of the services that needs to start for the target to be reached), and the dbus services conversely start after sysinit.target is reached. Systemd resolves this by ignoring the systemd-resolved requirement of sysinit.target. This doesn't seem to break anything else, and DNS works, so I'm just gonna leave it like this.
Seems like a lot of hullabaloo vs. editing a file, especially for a personal desktop set-up. But I think we've learned a lot. One thing I still don't know: what caused this to start happening? Was it something in one of the updates? Let me know what you think.
Until next time
-Div