Post Migration Alert Validation

This document lists the things every team should verify after their alerts were automatically migrated from google-application-platform-alerts to gap-registry. The migration was done by tooling — a human review is still required to confirm correctness before the alerts go live.

In the old google-application-platform-alerts format, every alert rule had a labels: block that typically included both a severity and a namespace label:

# OLD format
labels:
  severity: warning
  namespace: my-namespace

In gap-registry the chart automatically injects only severity into the rule’s labels. The namespace label is no longer set manually — it is derived from the Kubernetes namespace the rule is deployed into.

What to check:

• If your alertmanager routing or Grafana dashboards match on namespace= labels, verify they still work. The label is still present at the Prometheus level (injected by GMP/Thanos), but it is no longer set explicitly in the alert rule definition.
• If you relied on any other custom labels (e.g. team, service) for routing or filtering, add them back using the new labels: field in your alerts.yaml:

# gap-registry format — add custom labels alongside severity
rules:
  custom:
    - alert: MyAlert
      severity: warning
      labels:
        team: my-team
        pagerduty_service: my-service

The old EU cluster alerts were written for self-managed Prometheus (Prometheus Operator + ServiceMonitor), which prefixed certain kube-state-metrics labels with exported_ to avoid collisions (exported_pod, exported_container, exported_namespace). Since all clusters now use Google Managed Prometheus (GMP), these prefixes no longer exist — labels are exposed under their original names.

The migration rewrote these automatically:

What to check:

• Open your migrated alerts.yaml and inspect any custom: alert queries that reference kube_pod_container_status_last_terminated_reason, kube_pod_container_resource_limits, container_memory_working_set_bytes, or container_cpu_usage_seconds_total.
• Validate the rewritten query in Metrics Explorer (see section 4) to confirm it returns data.
• Pay special attention to label_replace() calls — if the source label argument was exported_pod or exported_container, it was rewritten to pod/container. Verify the join still produces the expected result.

Alerts most likely to be affected:

• ContainerTerminatedOOMKilled / ContainerOOMKilled custom variants
• Memory usage percentage alerts (container_memory_working_set_bytes / kube_pod_container_resource_limits)
• CPU usage percentage alerts (container_cpu_usage_seconds_total / kube_pod_container_resource_limits)
• Any alert using kube_pod_labels with a group_left join on pod or namespace

All alerts that could not be mapped to a standard common alert type were placed in the custom: section with their PromQL carried over verbatim (after exported_* rewrites). These are the most likely to need manual adjustment.

Check the alerts.yaml.renamewhenvalidated file in your namespace in gap-registry for any entries under custom: and validate each query.

The following metric types were present in the old alerts but may not be available in all GMP clusters — verify before enabling:

Some namespaces (e.g. relational-data, segmentation) had CPU and memory percentage alerts using label_replace with exported_pod or exported_container as the source label. After the rewrite these labels may still not resolve correctly if the join logic depended on the old label collision behaviour. Validate these queries return data before enabling.

Before renaming the .renamewhenvalidated files to make the alerts live, validate your custom PromQL queries in the Metrics Explorer for the relevant cluster.

How to validate:

1. Open the Metrics Explorer for your target cluster.
2. Switch to the PromQL query mode.
3. Paste your query: value from alerts.yaml.
4. Confirm the query returns a non-empty result with the expected label names.
5. If the query references exported_pod, exported_container, or exported_namespace and returns no data, the label rewrite may be incomplete — update the query manually.

The migrated files are currently named with a .renamewhenvalidated suffix to prevent them from being picked up before review. Once you have validated your alerts:

1. Rename alerts.yaml.renamewhenvalidated → alerts.yaml
2. Rename alertmanager.yaml.renamewhenvalidated → alertmanager.yaml
3. Open separate PRs for each file to gap-registry
4. After merge, watch for <namespace>-alerts-<instance> ArgoCD apps to appear, the alertmanager config will be deployed automatically in another application.
5. Sync the apps in ArgoCD
6. Verify rules appear in k9s under your namespace’s Rules object

For full documentation on the alert format and available common alert types, see the alerting configuration guide.