TLDR@1qzxcThere is still version 1.27… . Tried a lot. The only thing that worked was to revert to the version you suggested. Here’s everything I tried for future reference.

I installed kubernetes following the steps described in _readme/vps-kubernetes.md. I tried to start nginx and certbot on k8s. However, I was having persistent issues restarting kube-proxy. I checked the event log and, with the help of GPT and StackOverflow, discovered that the problem was caused by duplicate DNS entries. This appears immediately after droplet initialization.

Summary

• /etc/resolv.conf
• Old files, everyone gangs up on them
• /run/systemd/resolve/resolv.conf
• New file – created at runtime to avoid gangbang issues and avoid losing system settings. Your changes will be overwritten.
• sudo nano /etc/netplan/50-cloud-init.yaml' - this generates the one from /run - this has double entries for and

  eth0

eth1\`. Basically it comes from DO droplets.

• Creates 6 DNS entries.
• DNS Ubuntu supports only 3.
• Anything greater than 3 will be ignored in legacy programs.
• K8S kube-proxy sees this and is confused.
• Forced use of kube-proxy.etc/resolv.conf
• The problem no longer exists, but the kube-proxy pods still keep dying.
• I’ve dug into all the logs I can to increase the details. Nothing
• Finally I tried rolling back to a previous version and it worked.

Detailed log

• Inspect eventskubectl get events --all-namespacesNameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: xx.xxx.xx.2 xx.xxx.xx.3 xx.xxx.xx.2

  \-

• Errors were found in several pods while checking for events.

• Adjust kubelet flagsK8s coredns and flannel nameserver limits exceeded. [Close]

  \- If necessary, you can specify an alternative resolv.conf file by passing the –resolv-conf flag to the kubelet.

• Some Linux distributions (e.g. Ubuntu) use the local DNS resolver (systemd-resolved) by default. Systemd-resolved moves and replaces /etc/resolv.conf with a stub file, which can cause fatal forwarding loops when resolving names on upstream servers. This issue can be resolved manually by using the kubelet’s –resolv-conf flag to point to the correct resolv.conf (/run/systemd/resolve/resolv.conf for systemd-resolved). kubeadm (>= 1.11) automatically detects systemd-resolved and adjusts kubelet flags accordingly. Debugging DNS resolution

  .

• This is not my first time seeing this page. I ran into a similar issue when trying to understand why Minikube wouldn’t start. At that time there was a problem with the defaults and the only solution was a complete reinstall. It doesn’t work this time.
• kubelet configuration file

  Edit the GPT. Kubelet configuration is typically controlled by files located in /var/lib/kubelet/config.yaml. You must edit this file to change your DNS settings.

sudo nano /var/lib/kubelet/config.yaml

• To specify an alternate resolv.conf, add the following line under the kubelet configuration.

Copy code
resolvConf: /run/systemd/resolve/resolv.conf

• Already checked for modifications

  . Then I checked /run/systemd/resolve/resolv.conf. establish:

nameserver xx.xxx.xx.2
nameserver xx.xxx.xx.3
nameserver xx.xxx.xx.2
# Too many DNS servers configured, the following entries may be ignored.
nameserver xx.xxx.xx.3
nameserver xx.xxx.xx.2
nameserver xx.xxx.xx.3
search .

• Remove excessive nameserver entriesK8s coredns and flannel nameserver limits exceeded

  \- GPT: Only a maximum of 3 nameserver rows should be maintained. Select the primary DNS server you want to use. Also make sure there are no duplicate entries. For example, after editing, the file should look like this: Here’s another new answer suggesting the same fix.

• More to say that deduplication works – Rancher node: Nameservers limits were exceeded, some nameservers are being omitted

nameserver xx.xxx.xx.2
nameserver xx.xxx.xx.3
search .

• Restart kubeletsudo systemctl restart kubelet

  \-

• Recheck events – new issueskubectl get events --all-namespaces

  \-

• I tried it and it seemed to work, but I have a new problem.

• Full restart does not help – changes are overwritten/run/systemd/resolve/resolv.conf/run/systemd/resolve/resolv.conf

  \- A full restart was performed. edited

. Removed cheat. I started reinstalling everything. Same problem. I checked again on

. Overwritten.

• Also edit /etc/systemd/resolved.conf.

  \- GPT: It is important to note that if systemd-resolved is actively managing resolv.conf, these changes may be overwritten by systemd-resolved. In these cases, you must configure systemd-resolved to use the correct DNS servers. This typically involves editing /etc/systemd/resolved.conf, setting the DNS= line to the desired nameserver, and then restarting systemd-resolved.

• sudo nano /etc/systemd/resolved.conf

DNS=xx.xxx.xx.2 xx.xxx.xx.3

• sudo systemctl restart systemd-resolved
• Restart
• /run/systemd/resolve/resolv.conf still overwrittensystemd-resolved

  \- This also applies if K8 is not installed. It’s enough to start again

The file will be overwritten.

sudo systemctl restart systemd-resolved
kubectl get events --all-namespaces | grep kube-proxy
kubectl get events --all-namespaces --sort-by='.lastTimestamp'

• Ideas, further research

  \- This seems to be a very well known issue.

• Rollback – it still works, but I found out laterThis problem appears on k8s v1.26 however it was absent on v1.23 ​ kube-proxy pod constantly crashes in CrashLoopBackOff

  Someone suggested:

• Add flag to kubelet configmy current workaround is to have a /etc/resolv-static.conf and add --resolv-conf=/etc/resolv-static.conf to /var/snap/microk8s/current/args/kubelet but this is really not very satisfyingName server limit exceeded due to more than 3 DNS servers in /etc/resolv.conf

  – Another flag here:

  . Not directly related. We don’t use microk8.

  – Increase 3 DNS limit using dnsmasqNameserver limit exceeded with more than 3 DNS servers in /etc/resolv.conf

  – GPT: Increase nameserver records using dnsmasq: If you have reached the limit of three nameserver records, setting up dnsmasq on your node may help. It acts as a local DNS cache and allows you to specify additional name servers. Here’s a brief introduction to how to set it up. Same as here:

  – I don’t want to use this approach right now. It is very complex and prone to failure.

  • Install dnsmasq on the node.
  • Configure dnsmasq to include the cluster DNS and other name servers as needed.
  • Specify the kubelet’s –resolv-conf as the dnsmasq management resolv.conf.
  • Known issues, unclear fixesK8s coredns and flannel nameserver limit exceeded

    –

  • Netplan duplicate DNS entrysudo nano /etc/netplan/50-cloud-init.yaml

    \- Some duplicates are

. GPT: Configuration fragments found in /etc/netplan provide important clues as to what happens as DNS settings are reset on restart. Netplan is a network configuration abstraction renderer used in Ubuntu and other Linux distributions that allows you to easily configure networking through YAML files. When the system boots, netplan applies these configurations, including DNS server settings, to the system’s network interfaces.

• Understanding duplication issues

  – Duplicate DNS servers in /run/systemd/resolve/resolv.conf after rebooting or restarting systemd-resolved are most likely due to the way netplan handles network configuration and interacts with systemd-resolved. If both the eth0 and eth1 interfaces are assigned the same DNS servers and these settings are applied to the system’s DNS resolver, duplication may be observed.

• Not sure if it’s okay to change

  . I don’t like changing the DO droplet defaults. I will try another method.

• Why K8s are of interest – because they may lose their own DNS after 3 entries.

  \- GPT: Kubernetes’ warning about nameserver limit exceeded is due to an inherent limitation of the Linux kernel’s DNS resolver, which by default only considers up to three nameservers. Entry in the /etc/resolv.conf file for DNS resolution. When setting up a pod, Kubernetes (specifically the kubelet) configures DNS settings based on the node’s /etc/resolv.conf or another resolver configuration specified with the –resolv-conf kubelet flag. Kubernetes attempts to merge your cluster’s internal DNS service (such as CoreDNS or kube-dns) with the existing name servers specified in the node’s DNS configuration. If the node’s DNS configuration already specifies three name servers, adding the cluster’s DNS service will exceed this limit and displays a warning.

• Specify Kube as //etc/resolv.conf instead of /run/systemd/resolve/resolv.conf/etc/resolv.conf/run/systemd/resolve/resolv.conf

  \-GPT:

Instead, specifying the kubelet’s –resolv-conf flag is recommended.

It’s a workable approach. This is especially true if /etc/resolv.conf contains: A DNS configuration that better suits the DNS resolution needs of Kubernetes Pods. This method can be especially effective if /etc/resolv.conf is stable, properly configured, and not subject to undesirable automatic updates that could interfere with Kubernetes operations.

• sudo nano /var/lib/kubelet/config.yaml

• Change: resolvConf: /run/systemd/resolve/resolv.confresolvConf: /etc/resolv.conf

  to

  • If you want to keep it “modern” you shouldn’t do this, but as long as you don’t rely too much on automatic DNS changes, you should be fine.
  • sudo systemctl restart kubelet
  • Finally starting again
  • New issue – unclear what it is, no problem shown in log

    \- In the previous issue, the name server limit exceeded error was fixed. Now we have something new.

• kubectl describe pod kube-proxy-qv47n -n kube-system

  • It is of no use here. Just mark it as dead.
• kubectl logs kube-proxy-mg49v --previous -n kube-system
  • There’s nothing here either.
• No entries in one of the following:

journalctl -u kubelet
journalctl -u docker
journalctl -u containerd
journalctl -p err

• Edit detailed information of kube-proxy – still no resultskubectl edit daemonset kube-proxy -n kube-system:wq

  \-

• Save and Exit: After adding the verbose flag, save your changes and exit the editor. Kubernetes automatically updates the DaemonSet and restarts the kube-proxy pod to apply the new log level.

• Create vim and exit.

containers:
- command:
  - kube-proxy
  - --config=/var/lib/kube-proxy/config.conf
  - -v=8

• Despite level 8 verbosity, GPT cannot find anything wrong.

• Installing previous version 1.23.17-00 – this is the answersudo apt-get install -y kubelet=1.23.17 kubeadm=1.23.17 kubectl=1.23.17This problem appears on k8s v1.26 however it was absent on v1.23 ​kube-proxy pod continuously crashes with CrashLoopBackOff

  \- Full reinstall –

• Someone suggested:

• [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.5. Latest validated version: 20.10