Related to:

CoreDNS

Problem:

The ITRS Analytics web console may become inaccessible. When checking the ITRS Analytics debug shell, multiple pods failed to start or in CrashLoopBackOff state. The log files for pods may show error symptoms on resolving DNS hostnames.

User may want to check the "platformd" pod status and its log file.

kubectl get pods -n kotsadm | grep platformd
kubectl logs platformd-<pod-id> -n kotsadm

This is an excerpt of the DNS error that may return in the log.

java.lang.IllegalArgumentException: Environment variable STATSD_SERVER value 'statsd-internal.kotsadm.svc.cluster.local' is not a valid address or hostname

Possible causes:

• Root Cause 1:

The CoreDNS pod in Kubernetes may have failed or does not respond properly.

• Root Cause 2:

The node may have antivirus or security tools running that interfere Kubernetes operations.

• Root Cause 3:

The node may have SELinux enabled in enforcing mode.

• Root Cause 4:

Certain Linux kernel parameters may be changed unexpectedly that affects Kubernetes operations.

Possible solutions:

• Solution Root Cause 1:

User may check the coredns pod status and its log file.

kubectl get pods -n kube-system | grep coredns
kubectl logs coredns-<pod-id> -n kube-system

If it fails to start or the log contains unexpected timeouts or errors, you may consider restarting the pod.

kubectl rollout restart deployment/coredns -n kube-system

• Solution Root Cause 2:

User may check the node for security software or tool installed or running. Please contact our support teams and provide a support bundle (details in link provided under below section) for further investigation.

• Solution Root Cause 3:

User may execute the below commands to verify SELinux status.

sestatus
getenforce
cat /etc/selinux/config

Please liaise with the IT security or related teams in your organization if SELinux settings need to be updated.

• Solution Root Cause 4:

User may verify the net.ipv4.conf.default.rp_filter and net.ipv4.conf.all.rp_filter settings. The values should be set to 2. These kernel parameters affect routing of packets between network interfaces created by Kubernetes.

Please note that Embedded Cluster already sets rp_filter sysctl parameters in /etc/sysctl.d/99-embedded-cluster.conf file. The parameters may have been overridden in a different conf file in /etc/sysctl.d/ or in /etc/sysctl.conf. You may look for clues with the below command.

grep rp_filter -R /etc/sysctl*

Related articles:

• ITRS Analytics documentation
• ITRS Analytics - Support Bundle

If you need further help:

• Please contact our support team via the chat service box on any of our websites or raise a support request.
  
• Make sure you provide us with:
  • Background of the issue or request.
  • Use cases, requirements, business impact, etc.
  • Encountered error messages.
  • Log files or diagnostic files.
  • Screenshots.
  • And other important information relevant to your inquiry.