Migrating Alerts to Gap Registry

This document is for people who want to do the alert migrations themself (e.g. for a new namespace). If you have the alerts automatically migrated already, you are most likely looking for Validating migrated alerts instead.

This guide explains how to manually migrate alert rules from the google-application-platform-alerts repository into gap-registry. The migration doc will assume you are in a root directory containing both repos as subdirectories.

The old repo stores raw Prometheus groups/rules YAML that is compiled into PrometheusRule Kubernetes manifests. The new format in gap-registry uses a higher-level abstraction — common alert types are declared by name with parameters, and custom PromQL expressions are kept for anything that doesn’t map to a built-in type.

The files you need to create per namespace with separate PRs are:

gap-registry/namespaces/<namespace>/
  alertmanager.yaml   # PagerDuty routing key
  alerts.yaml         # Alert rules

Full documentation on the target format is at alerting

In google-application-platform-alerts, each namespace has up to three subdirectories:

<namespace>/
  common/       # alerts that apply to both production and staging
  production/   # production-only alerts
  staging/      # staging-only alerts
  alert-config.json   # optional: lists globally-excluded alert names

List all YAML files for your namespace:

find google-application-platform-alerts/<namespace> -name "*.yaml" | sort

Create gap-registry/namespaces/<namespace>/alertmanager.yaml in a PR on its own.

Find the routing key by searching the source alertmanager config:

grep -A 5 "name: '<namespace>'" \
  google-application-platform-alerts/.platform/production/alertmanager.yaml \
  google-application-platform-alerts/.platform/staging/alertmanager.yaml

The routing_key value on the matching pagerduty_configs block is what you need.

If there are separate high/low receivers (e.g. tooling-high and tooling-low), the old repo routed them by severity. In the new format, model this with routes under the receiver — the namespace catch-all (routingKey) handles anything not matched by a sub-receiver:

# namespaces/<namespace>/alertmanager.yaml
routingKey: "<warning-routing-key>"   # catch-all (low priority)
routes:
  - receiver: <namespace>-critical
    routingKey: "<critical-routing-key>"
    match:
      severity: critical

If there is only one receiver, use a plain string with no routes:

# namespaces/<namespace>/alertmanager.yaml
routingKey: "<routing-key>"

Some namespaces in the old repo had per-application routing — different PagerDuty keys for specific services within the same namespace (e.g. smart-insight-alttab, smart-insight-experiment). Use routes with match (exact) or matchRegex for this:

# namespaces/smart-insight/alertmanager.yaml
routingKey: "si-default-key"   # catch-all for the namespace
routes:
  - receiver: smart-insight-alttab         # exact label match
    routingKey: "si-alttab-key"
    match:
      label_application_name: si-alttab
  - receiver: smart-insight-customer-registry
    routingKey: "si-customer-registry-key"
    match:
      label_application_name: si-customer-registry
  - receiver: smart-insight-experiment     # regex match
    routingKey: "si-experiment-key"
    matchRegex:
      label_application_name: ^experiment.*

Routes are evaluated in order, top to bottom. The first matching route wins; any alert not matched by a sub-receiver falls through to the namespace catch-all (routingKey).

Open gap-registry/namespaces/<namespace>/alerts.yaml (create it if it doesn’t exist) and add a rules: block. Start with a comment pointing back to the source so reviewers can trace the migration:

# Migrated from google-application-platform-alerts/<namespace>/
# See: https://github.com/emartech/google-application-platform-alerts/tree/master/<namespace>
rules:
  group: <namespace>-alerts
  interval: 1m
  common:
    ...
  custom:
    ...

For each alert in the source files, decide whether it maps to a common built-in type or needs to go into custom.

These alert names map directly to pre-built abstractions. Add them under rules.common:

Everything else goes into rules.custom — see Custom alerts below.

• severity — read from labels.severity in the source rule
• duration — read from for: in the source rule (omit if absent)
• aggregationLabels — if the source expr filters by deployment, ingress, or job name (e.g. deployment=~"my-app.*"), extract those label matchers into aggregationLabels

The old format expresses the success ratio as a value the rate must stay above (e.g. < 0.90 means more than 10% errors). The new format uses a plain percentage of errors:

threshold (%) = (1 - source_ratio) × 100

Examples:

Compare the same alert across common/, production/, and staging/ subdirectories.

• Same alert in both prod and staging with identical values — write it once with a plain value:

  deploymentReplicas:
    severity: warning
    duration: 5m
  

• Different values between prod and staging — use a per-env map:

  deploymentReplicas:
    severity:
      prod: critical
      stage: warning
    duration:
      prod: 15m
      stage: 5m
  

• Alert only exists in production/ — add excludeFrom: [stage]:

  ingressRequestErrors4xx:
    threshold: 65
    severity: warning
    excludeFrom:
      - stage
  

• Alert only exists in staging/ — add excludeFrom: [prod].

Alert queries written for EU clusters (p-eu1-01, s-eu1-01) use exported_pod, exported_container, and exported_namespace labels. These labels do not exist in GMP (the US instances) and must be rewritten.

Why they exist in EU: Prometheus Operator’s ServiceMonitor scrape pipeline detects a label collision between its own scrape-target labels (pod, namespace, container) and the same-named labels emitted by kube-state-metrics, and resolves this by prefixing the metric labels with exported_. GMP’s PodMonitoring pipeline does not do this — labels flow through unchanged.

Rename them in every query: you write:

GMP enforces rule isolation: every rule deployed in namespace <ns> has namespace="<ns>" automatically injected into its query. If a source alert’s expr filters on a different namespace (e.g. namespace="gmp-system"), placing it in rules.custom will cause an ArgoCD sync error:

admission webhook "validate.rules.gmp-operator..." denied the request:
isolating rules failed: conflicting label matcher namespace="gmp-system" found

Fix: move such alerts to clusterRules instead, which is not subject to namespace isolation. Add a comment explaining the reason:

clusterRules:
  group: <namespace>-cluster-alerts
  rules:
    # source: <namespace>/common/gmp_collector_rules.yaml
    # Moved to clusterRules: namespace-scoped rules cannot query gmp-system (GMP isolation enforcement)
    - alert: CollectorExcessiveRestarts
      query: increase(kube_pod_container_status_restarts_total{namespace="gmp-system", ...}[1h]) > 10
      severity: warning

Any alert that doesn’t match a common type and doesn’t query a foreign namespace goes under rules.custom. Copy the expr from the source as query (applying the exported_* rewrites above), and carry over summary, description, for → duration, and severity:

  custom:
    - alert: MyCustomAlert
      summary: "..."
      description: "..."
      query: <original expr, with exported_* labels rewritten>
      duration: 5m
      severity: warning

⚠️ If the query references GCP-specific metrics (e.g. stackdriver_redis_instance_*) or metrics from other namespaces, verify they are available in the new environment using Metrics Explorer before relying on the alert.

Add a # source: comment above every entry pointing to the file(s) it came from. This makes PR review easier and provides a paper trail:

  common:
    # source: <namespace>/common/deployment_rules.yaml
    deploymentReplicas:
      severity: warning
      duration: 2m

    # source: <namespace>/production/ingress_rules.yaml, <namespace>/staging/ingress_rules.yaml
    ingressRequestErrors5xx:
      threshold: 10
      severity:
        prod: critical
        stage: warning
      duration: 1m

  custom:
    # source: <namespace>/common/my_custom_rules.yaml
    - alert: MyCustomAlert
      query: ...
      severity: warning

List all files comma-separated on a single line when an entry was merged from multiple sources.

If the source namespace has an alert-config.json, check its global.exclude section:

{
  "global": {
    "exclude": {
      "production": { "HPAReachesMaxReplicas": ".*" },
      "staging":    { "HPAReachesMaxReplicas": ".*" }
    }
  }
}

Any alert listed there was intentionally suppressed globally. Omit the corresponding common entry from your alerts.yaml rather than adding it with an excludeFrom — the intent was to disable it entirely.

Below is the result of migrating the cloud-platform namespace for reference:

clusterRules:
  group: cloud-platform-cluster-alerts
  interval: 1m
  rules:
    # ... existing cluster-level rules ...

    # source: cloud-platform/common/gmp_collector_rules.yaml
    # Moved to clusterRules: namespace-scoped rules cannot query gmp-system (GMP isolation enforcement)
    - alert: CollectorExcessiveRestarts
      summary: "Collector Pod {{ $labels.pod }} in gmp-system namespace keeps restarting"
      description: "Collector Pod {{ $labels.pod }} restarted {{ $value }} times"
      query: increase(kube_pod_container_status_restarts_total{namespace="gmp-system", container="prometheus", pod=~"collector-.+"}[1h]) > 10
      severity: warning

# Migrated from google-application-platform-alerts/cloud-platform/
# See: https://github.com/emartech/google-application-platform-alerts/tree/master/cloud-platform
rules:
  group: cloud-platform-alerts
  interval: 1m

  common:
    # source: cloud-platform/common/deployment_rules.yaml
    deploymentReplicas:
      severity: warning
      duration: 2m

    # source: cloud-platform/common/job_rules.yaml
    jobStatusFailed:
      severity: warning

    # source: cloud-platform/common/pod_rules.yaml
    containerOOMKilled:
      severity: warning

    # source: cloud-platform/production/ingress_rules.yaml, cloud-platform/staging/ingress_rules.yaml
    ingressRequestErrors5xx:
      threshold: 10
      severity:
        prod: critical
        stage: warning
      duration: 1m

    # source: cloud-platform/production/ingress_rules.yaml
    ingressRequestErrors4xx:
      threshold: 65
      severity: warning
      duration: 1m
      excludeFrom:
        - stage

    # source: cloud-platform/production/request_error_rate_rules.yaml
    highInbound4xxErrorRate:
      threshold: 5
      duration: 10m
      severity: warning
      excludeFrom:
        - stage

    # source: cloud-platform/production/request_error_rate_rules.yaml, cloud-platform/staging/request_error_rate_rules.yaml
    highInbound5xxErrorRate:
      threshold: 5
      duration: 1m
      severity: warning

  custom:
    # source: cloud-platform/common/overseer_request_error.yaml
    - alert: OverseerRequestError
      summary: "Overseer request errors"
      description: "Overseer request errors detected to {{ $labels.destination_service }}"
      query: sum by (destination_service) (increase(istio_requests_total{source_workload="overseer",response_code!~"2.*|3.*"}[5m])) / on() group_left() count(kube_node_info) > 0
      duration: 1m
      severity: warning

1. Open a pull request to gap-registry with your changes, separately for alertmanager.yaml and alerts.yaml
2. After the merges, the <namespace>-alerts-p-us1-01 and <namespace>-alerts-s-us1-01 ArgoCD applications will appear automatically, the alertmanager config will be deployed automatically in another application.
3. Sync those two applications in ArgoCD
4. Verify the rules are applied by checking the Rules object in your namespace with k9s (shift+:, then type Rules)