OpenShift Kubelet Reserved

The “kubelet” is an essential component in a Kubernetes cluster responsible for managing the state of individual nodes and ensuring that containers are running in a Pod as expected. It communicates with the Kubernetes control plane to receive instructions about which containers should be running on a node.

“Reserved resources” in Kubernetes generally refer to the resources (CPU and memory) that are reserved on a node to ensure that there is capacity for critical system processes and other essential tasks. These reserved resources might not be available for regular Pod scheduling.

Combining these terms, “kubelet reserved” could potentially refer to the resources reserved by the kubelet itself to ensure the proper functioning of Kubernetes node-level operations. These reserved resources could include CPU, memory, and other system resources necessary for the kubelet’s operation.

Here are some potential consequences when kubelet reserved resources are insufficient:

• Pod Scheduling Failures: Kubernetes may not be able to schedule new Pods on the node due to the lack of available resources. This can lead to delays in deploying applications and workloads.

• System Instability: Insufficient resources can cause system processes and kubelet-related operations to become unstable. Critical system components might not get the resources they need to function properly, leading to crashes or hang-ups.

• Degraded Performance: When kubelet operations and system processes don’t have the necessary resources, they might start competing with application containers for resources like CPU and memory. This can result in slower response times and degraded overall application performance.

• Pod Evictions: Kubernetes may start evicting Pods from the node in an attempt to free up resources for higher-priority Pods. This can disrupt running applications and result in unexpected downtime.

• Increased Latency: Applications that depend on low-latency access to resources might experience increased latency due to resource contention and competition with other processes.

• Container Failures: If containers within Pods don’t have the necessary resources to run, they might crash or fail to start properly. This can lead to application disruptions and data loss.

• Resource Starvation: The kubelet itself might experience resource starvation, leading to its inability to function correctly. This could impact its ability to communicate with the control plane, handle Pod lifecycle operations, and report node status.

Calculating Kubelet Reserved

CPU Reservation Formula;

• 6% of the first core.
• 1% of the next core (up to 2 cores).
• 0.5% of the next 2 cores (up to 4 cores).
• 0.25% of any cores above four cores.

Memory Reservation Formula;

• 25% of the first 4 GB of memory.
• 20% of the following 4 GB of memory (up to 8 GB).
• 10% of the following 8 GB of memory (up to 16 GB).
• 6% of the next 112 GB of memory (up to 128 GB).
• 2% of any memory above 128 GB

Here’s a simplified example of calculating kubelet reserved resources:

Let’s say you have a node with a total of 12 vCPU cores and 32GB of memory. Based on your container runtime and Kubernetes distribution documentation, you allocate:

CPU:

Total for 12vCPU reservation = 0.1025 vCPU / 102.5 milicore ~ 103milicore (I usually add it up to next half of CPU for more buffer, ie 500milicore.)

Memory:

Total for 32GB memory reservation = 5.9GB ~ 6GB

Applying kubelet reserved

We can enforce this setting via the KubeletConfig custom resource definition and apply it based on the MachineConfigPool selector. For example;

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-kubelet-config 
spec:
  machineConfigPoolSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: worker 
  kubeletConfig: 
    podsPerCore: 10
    maxPods: 250
    systemReserved:
      cpu: 500m
      memory: 6Gi