Kubernetes resource concepts deep dive
This page explains the underlying Kubernetes mechanics behind resource requests, limits, CPU throttling, and OOM kills. It is background reading — you do not need to understand all of this to right-size your deployments, but it helps when you are debugging unexpected throttling, OOM kills, or autoscaling behavior.
For practical guidance on what values to set, see the Resource details page.
When Kubernetes schedules a pod, it looks at the request values to find a node with enough unreserved capacity. Once scheduled, the pod is guaranteed to get at least what it requested — but it can use more if the node has spare capacity available.
The limit is a hard cap enforced by the Linux kernel’s cgroups mechanism. The behavior when a limit is hit differs between CPU and memory:
- CPU: the kernel throttles the process — it is slowed down but continues running
- Memory: the kernel sends an OOM (out of memory) kill signal — the container is terminated immediately and Kubernetes restarts it
This asymmetry matters for how you set headroom. CPU limits can be set tighter because throttling degrades performance gracefully. Memory limits need more headroom because the consequence of hitting them is an immediate restart.
For the full Kubernetes reference on resource management, see the official documentation.
CPU is measured in cores or millicores (m). 1000m = 1 core.
cpu: "250m" # quarter of a core
cpu: "1" # one full core
cpu: "2000m" # two cores
How throttling works
The Linux CFS (Completely Fair Scheduler) enforces CPU limits using quota periods — by default, 100ms windows. If a container’s CPU limit is 250m, it is allowed to use 25ms of CPU time per 100ms window. If it consumes its quota before the window ends, it is throttled (paused) for the remainder of that window.
This means throttling is not just about sustained high CPU usage. A container that has a brief burst within a single 100ms window — even if its average usage is well below the limit — can get throttled. This is especially relevant for latency-sensitive applications: a 50ms pause in a 100ms window can add visible latency to a request.
The 100m minimum
CPU limits below 100m mean the container gets less than 10ms per 100ms window. At this granularity, scheduling jitter makes behavior unreliable. We ask for a minimum CPU limit of 100m for this reason.
Requests can be lower than limits
A pod with requests: cpu: 10m and limits: cpu: 1000m is valid. It reserves almost nothing on the node but can burst up to a full core if the node has spare capacity. This can be the correct configuration for mostly-idle workloads.
Memory is measured in bytes. Common units:
memory: "128Mi" # 128 mebibytes (~134 MB)
memory: "1Gi" # 1 gibibyte (~1.07 GB)
How OOM kills work
When a container exceeds its memory limit, the kernel kills it with a SIGKILL. Kubernetes detects the exit and restarts the container according to the pod’s restart policy. In kubectl describe pod output this appears as:
State: Terminated
Reason: OOMKilled
Unlike CPU throttling, there is no warning or graceful degradation. The process simply dies.
Memory requests and node scheduling
Like CPU, the memory request is what Kubernetes uses for scheduling — it reserves that amount on the node. If a node runs low on actual memory (not just reserved memory), the kernel’s OOM killer can evict pods regardless of their limits, prioritizing pods that are using more memory relative to their request.
JVM memory
JVM applications have multiple memory regions beyond the heap: metaspace, code cache, thread stacks, off-heap allocations (direct buffers, NIO). -Xmx is the JVM flag that sets the maximum heap size — it does not account for these non-heap regions. Setting -Xmx to 100% of the memory limit is a common mistake — the JVM will OOM kill itself when non-heap usage pushes total consumption over the limit.
A safe rule: set -Xmx to ~75% of the memory limit. For a 1Gi limit:
-Xmx 768m # 75% of 1024Mi
This leaves ~256Mi for non-heap overhead.
The Horizontal Pod Autoscaler calculates utilization as:
utilization % = sum(actual CPU usage across all pods) / sum(CPU requests across all pods)
It then compares this to targetAverageUtilization and scales replicas up or down accordingly.
The key consequence: the request is the denominator. If requests are inflated, the denominator is large, the percentage is always low, and HPA never scales up — even when pods are genuinely busy. If requests are right-sized to reflect actual typical usage, HPA math becomes accurate and scaling happens when it should.
See the HPA algorithm documentation for the full calculation details.
Kubernetes assigns each pod a QoS class based on its resource configuration. This affects which pods get evicted first when a node runs low on resources:
| QoS Class | Condition | Eviction priority |
|---|---|---|
| Guaranteed | requests == limits for all containers | Last to be evicted under resource pressure |
| Burstable | requests < limits (or only some containers have limits) | Middle priority |
| BestEffort | no requests or limits set | First to be evicted |
Note that QoS class only affects kubelet eviction ordering under resource pressure. Node drains, maintenance events, and autoscaler scale-downs will evict all pods regardless of QoS class.
Most GAP workloads are Burstable (requests < limits), which is the right trade-off: some eviction risk in exchange for the ability to burst beyond the request when the node has spare capacity.
Setting requests equal to limits (Guaranteed) prevents bursting and wastes capacity — it is only appropriate for latency-critical workloads where predictable performance matters more than efficiency.
GAP currently sets default resource requests in the platform charts, so no GAP deployment will land in BestEffort unless resource configuration is explicitly removed. Making resource requests a mandatory field is being considered for a future platform update.