Discovery Service Failing over IPv6

[TymanLS]

I have been running an Talos Linux cluster in an IPv6 only environment for some time without trouble. Sometime over the past day or two, though I'm not sure exactly when, I've been running into issues with the discovery service. I started getting warnings in the console:
[talos] hello failed {"component": "controller-runtime", "controller": "cluster.DiscoveryServiceController", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready", "endpoint": "discovery.talos.dev:443"}

When I run talosctl get affiliates and talosctl get members against one node, the results list only has one entry corresponding to that specific node.

I tried using talosctl debug with the docker.io/jonlabelle/network-tools:latest image to test connectivity to the public discovery service endpoint. I was able to resolve "discovery.talos.dev" with nslookup, but I was unable to receive ICMP ping responses, nor was I able to open a TLS connection using openssl s_client -connect discovery.talos.dev:443. I then tried curl -v https://discovery.talos.dev:443, and the request timed out.

I tried these connectivity tests again on some other non-Talos hosts on the same local network to make sure that my network firewall wasn't the issue, even though I never had issues with outbound traffic before. Here, I noticed that resolving with nslookup succeeded, ICMP ping failed, and openssl s_client -connect failed just like it was on the Talos host. However, I discovered that when I attempted curl on IPv6/IPv4 dual-stack hosts, the request succeeded; attempting curl on other IPv6-only hosts failed just like with Talos. I then went back to the dual-stack hosts and tried to force IPv6-only connection by using curl -6 -v https://discovery.talos.dev:443, and those connections then failed. I also tested forcing IPv4 with openssl s_client -4 -connect discovery.talos.dev:443, and that ended up succeeding.

After my testing, it seems as though the public Talos discovery service is not functioning properly over IPv6, but it is working fine over IPv4. On dual-stack hosts, it seems like the behavior may or may not fall back to IPv4 depending on the tool (curl seemed to fall back whereas openssl s_clientdid not). However, in my scenario of IPv6-only Talos hosts, the IPv6 discovery service is causing issues since I can't fall back to IPv4.

If there is any other possible explanation for this behavior, please let me know what I can do to troubleshoot the issue further. Thank you!

[smira]

Yes, this is not an issue in Talos itself, but an issue with discovery.talos.dev. @rsmitty or @TimJones should be able to provide more updates.

[TymanLS]

Thank you for the update, it seems the discovery service is still not working over IPv6. I am running into issues that seem to stem from broken communication between nodes. I assume this is likely due to KubeSpan being non-functional due to the discovery service, but all of my nodes are on the same L2 network segment so I thought they'd be able to fall back to direct communication.

While I am waiting for the discovery service to become functional over IPv6 again, would it make sense to temporarily disable KubeSpan on all my nodes to try and force them to directly communicate without the need for the discovery service, and would it cause any issues if I were to re-enable KubeSpan on all of my nodes again after the discovery service comes back up over IPv6?

[TymanLS]

@rsmitty @TimJones, are you able to provide any updates on the status of the discovery service? I tried testing again today, and it appears the discovery service is still not working properly over IPv6. Thank you!

[rsmitty]

Hi there,

Yes, we have this working properly in our staging environment. We'll get this rolled out to prod (should be tomorrow morning).

[smira]

This should work now over IPv6 as well, thanks for reporting!

[TymanLS]

I have confirmed as well that https://discovery.talos.dev is now responding over IPv6 as expected. Thank you all!