Kubernetes requests vs usage cost: you pay for requests, not usage
Your Kubernetes bill follows what your pods request, not what they use. The scheduler places each pod only on a node with enough unreserved CPU and memory to cover its resources.requests, and Cluster Autoscaler or Karpenter adds nodes when pending pods don’t fit, so node count tracks requested capacity (Kubernetes documentation, "Resource Management for Pods and Containers" and "Node Autoscaling").
On EC2-backed EKS node groups, AKS node pools and GKE Standard, you pay per node-hour however idle the nodes are. On EKS Fargate and GKE Autopilot the link is even more direct: each pod is sized and billed from its requests.
Why is my Kubernetes cluster bill so high when pods aren’t using much?
Your cloud provider bills for nodes, not pods. An EC2 node costs the same per hour whether its pods use all of its CPU or almost none. If a pod requests 2 CPU and uses 0.3, the scheduler still sets aside 2 CPU of that node’s allocatable capacity, and no other pod can be placed in that space.
When the reserved space runs out, new pods go Pending and Cluster Autoscaler or Karpenter launches another node. Over-requested pods buy you nodes, even when every node’s CPU graph looks quiet.
How do I compare Kubernetes requests to actual usage?
For a first look, compare what each node has reserved with what it is using. kubectl describe nodes shows reserved CPU and memory under "Allocated resources". kubectl top shows live usage and needs metrics-server installed.
kubectl describe nodes | grep -A 8 "Allocated resources"
kubectl top nodes
kubectl top pods -A --sort-by=cpu
kubectl top is a snapshot, so use it to find candidates, not to size them. For sizing, use Prometheus with kube-state-metrics (requests) and the kubelet’s cAdvisor metrics (usage). These queries give requested divided by used, per namespace:
sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})
/
sum by (namespace) (rate(container_cpu_usage_seconds_total{container!="", image!=""}[5m]))
sum by (namespace) (kube_pod_container_resource_requests{resource="memory"})
/
sum by (namespace) (container_memory_working_set_bytes{container!="", image!=""})
A ratio near 1 means requests match usage. Anything well above 1 is capacity you pay for and nobody uses. kube_pod_container_resource_requests also counts pods that have finished but still exist, such as completed Jobs. In batch-heavy namespaces, keep only Pending and Running pods before summing:
sum by (namespace) (
kube_pod_container_resource_requests{resource="cpu"}
* on (namespace, pod) group_left()
max by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Running"} == 1)
)
Size from peaks, not snapshots. With at least a week of Prometheus retention, this gives each container’s peak memory:
max by (namespace, container) (
max_over_time(container_memory_working_set_bytes{container!="", image!=""}[7d])
)
For CPU, use a high percentile of the 5-minute rate over the same week:
max by (namespace, container) (
quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{container!="", image!=""}[5m])[7d:5m])
)
Check your own bill in 2 minutes: https://revelations.consulting/checker?utm_source=blog&utm_medium=organic&utm_content=blog-kubernetes-requests-vs-usage-cost
How do I right-size Kubernetes requests without breaking services?
Start with the Vertical Pod Autoscaler (VPA) in recommendation mode. In recommendation mode, only the recommender’s output is used; the updater and admission controller leave Off objects alone.
On EKS, install VPA from the kubernetes/autoscaler repository (vertical-pod-autoscaler/hack/vpa-up.sh) or its Helm chart. On AKS, enable the managed add-on with az aks update --resource-group <rg> --name <cluster> --enable-vpa. On GKE Standard, run gcloud container clusters update <cluster> --enable-vertical-pod-autoscaling; GKE Autopilot has VPA on already.
Create one VerticalPodAutoscaler per workload with updateMode: "Off". Quote the "Off"; unquoted, some YAML parsers read it as the boolean false.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: checkout-api
namespace: shop
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: checkout-api
updatePolicy:
updateMode: "Off"
Let the recommender watch at least a full week, including your weekly peak, batch runs or month-end jobs. Then read the recommendations:
kubectl get vpa -A
kubectl get vpa checkout-api -n shop -o jsonpath='{.status.recommendation.containerRecommendations}'
Each container gets a lowerBound, target, upperBound and uncappedTarget under status.recommendation.containerRecommendations. Treat target as where you are heading, not a value to apply in one jump. Lower requests part of the way toward target, one Deployment at a time. Make the change in the manifest in source control so the next deploy does not revert it, and wait through a peak before the next step. Cut CPU before memory: a CPU mistake usually shows up as latency first, while a memory mistake shows up as OOMKills.
After each change, look for OOMKills and restarts:
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount,LAST:.status.containerStatuses[*].lastState.terminated.reason' | grep OOMKilled
sum by (namespace, pod, container) (increase(kube_pod_container_status_restarts_total[1h])) > 0
And for CPU throttling, in seconds and as the share of CFS periods throttled:
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_seconds_total{container!="", image!=""}[5m]))
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/
sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
Throttling comes from CPU limits, not requests. Once a container uses its limit’s quota for a CFS period, the kernel pauses it until the next period. If throttled seconds rise after a change, raise or remove the CPU limit; raising the request alone does not stop throttling. If OOMKills appear, roll the memory request and limit back to the last values that held.
Cutting CPU requests too far causes contention instead. On a busy node, CPU is shared in proportion to requests, so an under-requested pod slows down, and if its liveness probe times out the kubelet restarts it. Cutting memory requests too far packs more pods onto each node. When that node runs short, the kubelet evicts pods or the kernel OOM-kills them. Size memory requests from the peak of container_memory_working_set_bytes, not the average, and keep memory limits above that peak.
Change one value at a time. If requests and limits were equal, lowering only the request moves the pod from Guaranteed to Burstable QoS, which makes it an earlier eviction candidate under node pressure.
Check two more things before the first cut. A HorizontalPodAutoscaler measures CPU and memory utilization as a percentage of requests, so lower requests mean higher utilization and more replicas; adjust the HPA target at the same time. Smaller requests also only shrink the bill when nodes leave. Confirm Cluster Autoscaler scale-down or Karpenter consolidation is on, and that PodDisruptionBudgets, cluster-autoscaler.kubernetes.io/safe-to-evict: "false" or karpenter.sh/do-not-disrupt are not pinning half-empty nodes.
Track the result in Cost Explorer. Filter Service to EC2-Instances, group by Instance type, and filter on your node tag, for example eks:cluster-name. Activate that tag first under Billing and Cost Management, Cost allocation tags.
What it usually looks like
Check your own cluster for a few common patterns. One is requests copied from a template or a Helm chart’s defaults, so unrelated services all ask for the same CPU and memory. Another is requests sized for a launch-day peak or a load test and never revisited. A third is sidecars, such as mesh proxies and log shippers, whose small requests repeat on every replica.
Java and other fixed-heap runtimes need care. The JVM can grow toward -Xmx at any time, so a memory request or limit below heap plus overhead ends in OOMKills even if last week’s working set looked small. Check heap flags before you trust a memory recommendation for those services.
I cut a roughly $1M-a-year cloud bill by 26% at a former employer. The order I trust for request right-sizing on any cluster is the one in this article: read the VPA recommendations, cut CPU before memory in small steps, and watch restarts and throttling through a peak after every change.
If your pods request far more than they use, you are paying for nodes that sit reserved and idle. The fix is measurement and small steps, not a new tool: read the VPA recommendations, lower requests gradually, watch restarts and throttling, and confirm your node count actually falls.
If you want a second opinion on where your cluster is over-requested, book a free 30-minute consult: https://revelations.consulting/?utm_source=blog&utm_medium=organic&utm_content=blog-kubernetes-requests-vs-usage-cost#contact
Run your own numbers: Kubernetes Requests Right-Sizer.
Frequently asked questions
Can I see Kubernetes pod requests in AWS Cost Explorer?
No. AWS Cost Explorer shows EKS worker nodes as EC2-Instances running hours, with usage types such as BoxUsage:m6i.large (prefixed by region outside us-east-1, for example USW2-BoxUsage:m6i.large), and has no per-pod line. To tie EKS cost to pod requests, enable split cost allocation data for Amazon EKS in your Cost and Usage Report, or compare kube_pod_container_resource_requests in Prometheus with node-hours.
What’s the difference between requests and limits in Kubernetes?
Kubernetes requests set how much CPU and memory the scheduler reserves for a container on a node, so requests drive node count and cost. Kubernetes limits cap what a container can use at runtime: a container over its memory limit is OOMKilled, and one at its CPU limit is throttled. Limits shape performance and stability, not node count.
Can I use VPA to automatically update my deployments?
Yes, but carefully. VPA’s Recreate and Auto modes apply recommendations by evicting pods and recreating them with new requests, which interrupts single-replica workloads and conflicts with an HPA scaling on CPU or memory. Start with updateMode: "Off", apply VPA recommendations through your manifests, and automate only workloads with several replicas and a PodDisruptionBudget.
Why does my Kubernetes cluster need so many nodes when CPU usage is low?
Kubernetes fits pods onto nodes by their requests against each node’s allocatable capacity, which is smaller than the instance size because of system reservations and DaemonSets. If each pod requests 1.5 CPU, an 8-vCPU node holds at most 4 or 5 of them even when each uses 0.2 CPU, and the autoscaler adds nodes for the rest.
How do I check memory usage in Prometheus?
Use container_memory_working_set_bytes to measure container memory in Prometheus; it excludes inactive page cache and is what the kubelet uses for memory eviction decisions. Compare its peak over at least a week with kube_pod_container_resource_requests{resource="memory"}. Avoid container_memory_usage_bytes for sizing, because that metric includes reclaimable cache and overstates usage.
Run the numbers on your own bill: How much of your cluster is reserved and idle?. Free, in your browser.
Want this kind of thing weekly? Get Cloud Bill Weekly. Want us to find it in your bill? We're taking three founding clients for a free Cost Assessment.