Alerting

This guide explains how to configure the alerts and pagerduty configs for your namespace.

If you have the alerts automatically migrated already, you are most likely looking for Validating migrated alerts after you read this document.

If you are looking for how to manually migrate alerts, go to here after you read this document.

ℹ️ Note: Reworking the document is in progress, to cover EU alerting as well, not just s-us1-01 and p-us1-01 instances. A different document tries to cover migration from the old google-application-platform-alerts repo to the new gap-registry repo.

1. Set up the alertmanager receiver first — alerts won’t route anywhere without it
2. Create alert rules

Place your files under your team’s directory in the gap-registry:

namespaces/
├── <your-namespace-name>/
│   ├── alerts.yaml            # Alert rules
│   └── alertmanager.yaml      # Alertmanager receiver

Create namespaces/<your-namespace-name>/alertmanager.yaml to register your PagerDuty receiver.

# namespaces/<your-namespace-name>/alertmanager.yaml
routingKey:
  prod: "my-team-prod-key"
  stage: "my-team-stage-key"

routingKey can also be a plain string if the same key is used everywhere: routingKey: "my-key".

⚠️ Prerequisite: Make sure you have set up the alertmanager receiver before creating alert rules, please create a PR for it separately.

Create namespaces/<your-namespace-name>/alerts.yaml to define alert rules for your namespace.

Fields like severity, duration, threshold accept either a plain value or an env/instance map. When using a map, the chart resolves the value at render time: exact instance key → env tier → plain value.

• severity — one of critical (high priority), error, warning (low priority), info
• duration — Prometheus duration string, e.g. 30s, 5m, 1h
• interval — same format as duration; controls the rule evaluation interval
• threshold — number or string, e.g. 0, 5, 0.95

Things to pay attention to:

• All rules (common and custom) are automatically scoped to your namespace.
• Error-rate alerts (ingressRequestErrors5xx, ingressRequestErrors4xx, highInbound4xxErrorRate, highInbound5xxErrorRate) use percentages, e.g. 5 means 5%.

Pre-built alerts you can enable by adding them under rules.common. Use aggregationLabels to narrow the PromQL query to specific resources. Only severity is required.

Each entry in aggregationLabels is a raw PromQL label matcher, there are some example labels listed in the table below, but the exact label values for your rules will need to be observed for the specific metric in the Metrics Explorer for now, the base common alert expressions can be seen here.

Example values:

aggregationLabels:
  - 'deployment="my-deployment"'            # exact match
  - 'deployment=~"api|worker"'              # regex match
  - 'deployment!="test-app"'                # not equal
  - 'deployment!~"test-.*"'                 # negative regex

Since aggregationLabels is a list, you can use the same label multiple times:

aggregationLabels:
  - 'deployment!~"my-deployment-.*"'
  - 'deployment!~"my-other-deployment-.*"'

Expressions hidden behind the common alert abstractions can be seen here.

* required

Define any PromQL-based alert under rules.custom. Required fields: alert, description, summary, query, severity.

You can build your PromQL expressions and try them out in the Metrics Explorer.

More on how to use the code editor from above to build your PromQL, along with a cheat sheet of examples, can be found in this doc.

You can also move over your PromQL alert expressions from the google-application-platform-alerts repo, provided that you have confirmed that they work in the above Metrics Explorer.

rules:
  group: <your-namespace-name>-alerts   # required: name for the alert group
  interval: 1m                          # optional: evaluation interval, e.g. 30s, 1m, 1h (default: 30s)

  common:
    # Fires when a deployment has unavailable replicas (default threshold: 0)
    deploymentReplicas:
      aggregationLabels:
        - 'deployment=~"api|worker"'  # optional: regex match multiple deployments
      duration:
        prod: 15m
        p-us1-01: 10m
        stage: 5m
      severity:
        prod: error
        stage: warning

    # ── jobStatusFailed ──
    # Fires when a job has failed more than threshold times (default: 0).
    jobStatusFailed:
      severity:
        prod: critical
        p-us1-01: warning
        stage: warning

    # ── containerOOMKilled ──
    # Fires when a container is OOM killed more than threshold times (default: 0).
    containerOOMKilled:
      severity:
        prod: critical
        stage: warning

    # ── hpaMaxReplicas ──
    # Fires when an HPA's desired replicas equals its max replicas.
    hpaMaxReplicas:
      duration: 10m
      severity:
        prod: warning
        stage: warning

    # ── ingressRequestErrors5xx ──
    # Fires when the 5xx error rate (%) on an ingress exceeds the threshold.
    # Default threshold: 5 (percent).
    ingressRequestErrors5xx:
      aggregationLabels:
        - 'ingress=~"my-api|my-web"'  # scope to specific ingresses
      threshold: 5
      duration: 1m
      severity:
        prod: critical
        stage: warning

    # ── ingressRequestErrors4xx ──
    # Fires when the 4xx error rate (%) on an ingress exceeds the threshold.
    # Default threshold: 25 (percent).
    ingressRequestErrors4xx:
      aggregationLabels:
        - 'ingress="my-api"'
      threshold: 25
      duration: 1m
      severity:
        prod: critical
        stage: warning

    # ── highInbound4xxErrorRate ──
    # Fires when the istio inbound 4xx error rate (%) exceeds the threshold.
    # Default threshold: 5 (percent).
    highInbound4xxErrorRate:
      threshold: 5
      duration: 1m
      severity: warning

    # ── highInbound5xxErrorRate ──
    # Fires when the istio inbound 5xx error rate (%) exceeds the threshold.
    # Default threshold: 5 (percent).
    highInbound5xxErrorRate:
      threshold: 5
      duration: 1m
      severity: warning

  custom:
    # ── Custom PromQL alert ──
    # Required: alert, description, summary, query, severity
    # Optional: duration, excludeFrom
    - alert: CronJobInactivity
      description: "CronJob {{ $labels.cronjob }} in {{ $labels.namespace }} has not run recently"
      summary: "CronJob {{ $labels.cronjob }} inactive"
      query: time() - kube_cronjob_status_last_schedule_time{namespace="<your-namespace-name>",cronjob="my-cronjob"} > 7200
      duration:
        prod: 7m
        stage: 5m
      severity:
        prod: critical
        stage: warning

Exclude rules from specific instances or environments. Works on both common and custom rules.

ℹ️ Note: If an exclusion for an already deployed alert on an instance, be it a rule level or a global exclusion, means that there is no more alert rule to be deployed on that instance, the related <your-namespace-name>-alerts-<the-instance> Application will need to be deleted on ArgoCD.

rules:
  common:
    deploymentReplicas:
      excludeFrom:
        - stage          # skip this rule on all staging instances
      # - prod           # skip this rule on all production instances
      # - p-us1-01       # skip this rule on the p-us1-01 instance
      # ...

Top-level excludeFrom excludes all rules, can be used in conjunction with rule level exclusion:

rules:
  excludeFrom:
    - s-us1-01         # skip all rules on this instance
  # ...
  common:
    # ...

1. Please create a separate PR for your receiver(s) (namespaces/<your-namespace-name>/alertmanager.yaml), the file will be automatically deployed.

2. After that please create another PR for your (namespaces/<your-namespace-name>/alerts.yaml) file, once it’s merged to the gap-registry repo, in a few moments the <your-namespace-name>-alerts-<instance> ArgoCD apps will show up and you will need to sync them in order to deploy your alert rules.

Once your <your-namespace-name>-alerts-<instance> apps are applied, you can look for the Rules object in your namespace (usually shift+:, then write Rules on top) with e.g your k9s, where you can see the final PromQL expressions and rule definitions that get applied to the instance(s).

Use the Metrics Explorer for the project matching the instance you’re building or validating a query for: