Monitoring your deployment
The Cloud Monitoring deployment dashboard (EU production · EU staging · US production · US staging) is the primary tool for monitoring your deployment’s resource usage and traffic. It covers CPU and memory usage, replica health, HPA status, and both ingress and service-level traffic metrics.
Both dashboards have Namespace, Deployment, Pod, and Container filters at the top. Always set at least namespace and deployment — leaving them on defaults shows cluster-wide data across all deployments and makes the charts unreadable.
The Cloud Monitoring filter dropdowns enumerate all values across the cluster before rendering — they are slower than Grafana’s variable selectors. Type your value directly rather than scrolling.
- During or just after a deploy: last 30–60 minutes to confirm the new pods look healthy
- Investigating an incident: match the time range to when the issue was reported, plus buffer before and after
- Right-sizing resources: at least 2–3 days for a representative picture; a full week if your traffic has weekly patterns (e.g. drops on weekends). Cloud Monitoring handles longer time ranges well — 1–2 weeks loads quickly, so there is no reason to limit yourself to a short window.
Three panels stacked in the left column:
Pods available — count of available replicas over time. A drop here during a deploy is expected; a drop outside of deploys warrants investigation.
Replicas (updated / available / unavailable) — shows all three replica states on one chart. During a rolling deploy you will see updated climb while available briefly dips. If unavailable stays elevated after a deploy completes, something is wrong with the new pods.
Updated replicas — tracks how many replicas are running the latest version. Useful for confirming a rollout has fully propagated.
CPU usage vs request (%) / Memory usage vs request (%)
First panels to check for resource sizing. They show actual usage as a percentage of your configured request across all app containers (istio-proxy excluded). Two reference lines:
- too low (30%) — your request is more than 3× your actual usage. Reducing it would free up capacity for the rest of the cluster.
- too high (120%) — you are using more than your request. For CPU this means throttling risk; for memory it means approaching OOM territory.
Ideal range is roughly 50–90%.
CPU — Used / Requests / Limits / Memory — Used / Requests / Limits
Plots all three values on the same axis. A healthy deployment shows used comfortably below requests, with limits as a ceiling above peaks. If used regularly approaches limits, you are at risk of CPU throttling or OOM kills. See Resource details for sizing guidance.
CPU Usage per container (2m avg) / Memory Usage per container (MiB)
Usage broken out per individual pod and container, labelled pod-container. Use these to:
- Spot a single misbehaving pod among otherwise healthy replicas
- Check that a specific sidecar is not consuming more than expected
- Confirm that all containers are behaving consistently after a deploy
The Pod and Container filters at the top of the dashboard let you narrow to a specific container.
CPU p95 / p99 — avg across replicas per container (24h)
95th and 99th percentile CPU usage averaged across replicas over the last 24 hours. These are a convenient starting point when applying the P95 × 1.2 sizing rule described in Resource details.
CPU Usage spikes per container (5m rolling max)
Shows rolling maximum CPU usage per container. Use this if you suspect short bursts are causing throttling — burst activity that averages out over 2 minutes will be invisible on the standard usage chart but will show up here.
CPU throttling per container (2m avg)
Percentage of CPU scheduling periods that were throttled. Anything consistently above 10–20% warrants raising the CPU limit (not the request — see Resource details for the distinction). This is the most direct signal that your CPU limit is too low.
HPA Replica Ratio (current/max) — current replica count as a fraction of your configured maximum. A ratio consistently near 1.0 means your HPA is hitting its ceiling; consider raising maxReplicas if the pods are genuinely under load.
HPA Scaling Inactive — flags when the HPA condition ScalingActive=false is set. This means autoscaling has been suspended — usually because there are no valid metrics to scale on. If you are using CPU-based autoscaling and this fires, check whether metrics are being collected correctly.
Both the Ingress Traffic and Service Traffic (Istio) sections are available at the bottom of the Cloud Monitoring deployment dashboard. They measure traffic at different points in the request path and are complementary — if you see errors in one but not the other, it helps narrow down where the problem is.
Metrics from the nginx ingress controller — traffic as it enters the cluster from outside. This is the external-facing view.
Requests — request rate by HTTP status class (2xx, 3xx, 4xx, 5xx). A sudden drop in 2xx or spike in 5xx here means external traffic is being affected.
Latency — P50, P90, P95, P99 response time as seen by the ingress controller. Latency here includes time waiting for your application to respond.
Errors — 4xx and 5xx counts. Use this alongside Requests to distinguish between a volume increase and an error rate increase.
Metrics from the Istio service mesh — traffic between services inside the cluster. This covers both external traffic that has passed through the ingress and internal service-to-service calls.
Requests — request rate by response code class.
Latency — P50, P90, P95, P99. These are measured in milliseconds at the Istio proxy level. Comparing this to the ingress latency can help identify whether latency is introduced at the ingress layer or within the service itself.
Errors — 4xx and 5xx counts from Istio telemetry.
| Scenario | What it means |
|---|---|
| Errors in Ingress, not in Service Traffic | Problem likely at the ingress layer — routing, TLS, or the ingress controller itself |
| Errors in Service Traffic, not in Ingress | Internal service call failing, not external traffic |
| Latency high in Ingress, low in Service Traffic | Overhead at the ingress/network layer |
| Latency high in both | Application response time is the likely cause |
| Data in Service Traffic but not Ingress | Service is receiving internal traffic only (no external ingress configured) |
If you need to inspect the istio-proxy sidecar specifically — for example when investigating unexpected latency that cannot be explained from your application metrics — the Deployment Istio dashboard (EU prod · EU staging · US prod · US staging) shows resource usage and Istio-level metrics for the proxy container. See Managing resources for Istio proxy if you need to adjust the proxy’s resource allocation.
- Resource details — how to set and right-size resource requests and limits
- Kubernetes resource concepts — CPU throttling, OOM kills, requests vs limits explained
- Auto scaling