Related to:

ITRS Analytics, Evicted, Eviction

Problem:

When user looks at the ITRS Analytics web console, some pods are not running under Ingestion => Workloads.

User may find the pods in Evicted status when checking the ITRS Analytics debug shell.

sudo ./itrs-analytics shell
kubectl get pods -A

Possible causes:

The current versions of ITRS Analytics are built on top of Kubernetes. If the node is under resource pressure, some pods may be terminated to reclaim resources and prevent starvation. The resources that are generally involved could be memory, disk space, and inodes.

Possible solutions:

User should review the default eviction thresholds and resources available on the nodes where ITRS Analytics is installed.

Start a proxy server using kubectl proxy in your terminal.

kubectl proxy

Open another terminal window and use curl to fetch the kubelet configuration. Obtain the node name and replace in the second command.

kubectl get nodes
curl -s -X GET http://127.0.0.1:8001/api/v1/nodes/<my_node_name>/proxy/configz | jq .kubeletconfig.evictionHard

The command returns the hard eviction thresholds in JSON format, which should look like this:

In particular, please note the disk space thresholds which could often be overlooked by users. We have provided link to Kubernetes documentation below for your reference.

Related articles:

• ITRS Analytics Resource and Hardware Requirements
• Node-pressure Eviction (linked to Kubernetes documentation)

If you need further help:

• Please contact our support team via the chat service box on any of our websites or raise a support request.
  
• Make sure you provide us with:
  • Background of the issue or request.
  • Use cases, requirements, business impact, etc.
  • Encountered error messages.
  • Log files or diagnostic files.
  • Screenshots.
  • And other important information relevant to your inquiry.