Notification Events & Payloads

Gameplane emits notifications for critical cluster events — server health transitions, backup and restore completions — and routes them to your chosen sinks. Every event is strongly typed with structured fields, making it straightforward to filter, parse, and route.

Event catalog

The notification system emits six event types across three resource categories. Each event fires once on transition; state is never re-emitted.

Payload structure

Webhook sinks receive the full structured event as JSON. Discord, Slack, ntfy, and SMTP format the same data into their respective message shapes.

Common fields

Every payload contains these fields:

{
  "type": "server.unhealthy",
  "ts": "2026-07-04T10:12:00Z",
  "kind": "GameServer",
  "name": "survival-1",
  "namespace": "gameplane-games",
  "instance": "prod",
  "reason": "Liveness probe failure",
  "message": "Pod has been in an unhealthy state for 5 minutes",
  "test": false
}

Delivery by sink type

Each sink kind formats the event data for its platform while preserving the core fields. The delivery behavior (retries, observability) is consistent across all types.

Retry policy

When delivery fails, the system retries twice:

• Initial attempt: immediate
• First retry: 2 seconds
• Second retry: 8 seconds

Only retried on network errors (timeouts, DNS failures, connection refused) and 5xx server errors (temporary service unavailability). 4xx responses (bad token, revoked webhook, authentication failure) are not retried — they indicate permanent misconfiguration.

Events in the delivery queue are best-effort: if the queue fills, new events are dropped and counted at /metrics under gameplane_notify_deliveries_total{result="dropped"}. A slow or down sink never blocks reconciliation, API requests, or other operations.

Test sends

Test a sink without waiting for a real event:

curl -X POST https://<dashboard>/api/admin/notifications/sinks/{name}/test \
  -H "Authorization: Bearer <token>"

Returns one of:

• 200 {"delivered":true} — success
• 404 — sink not found
• 422 — sink has no credential Secret configured
• 502 — delivery failed (detailed error in response body)

The endpoint is rate-limited (~12 per minute per IP) to prevent abuse.

Sink configuration

Sinks are stored in the database but their credentials are always stored in Kubernetes Secrets, never in the database. Each sink references a Secret by name (configRef).

You can pre-create a Secret with kubectl and wire it up manually:

# Pre-create a Discord sink Secret
kubectl -n gameplane-system create secret generic team-alerts \
  --from-literal=url='https://discord.com/api/webhooks/...'

# Label it so the API can use it
kubectl -n gameplane-system label secret team-alerts \
  gameplane.local/notification-sink=true

# Then configure the sink via the dashboard or API
PUT /admin/config/notifications
{
  "sinks": [
    {
      "name": "team-alerts",
      "kind": "discord",
      "configRef": "team-alerts",
      "enabled": true
    }
  ]
}

Or use the dashboard form to create it directly — the API stores the credential as a Secret automatically.

Monitoring delivery health

Watch the delivery metrics at /metrics under gameplane_notify_deliveries_total:

gameplane_notify_deliveries_total{kind="discord|slack|smtp|webhook|ntfy",
                                  result="sent|failed|dropped|skipped_no_secret"}

• sent — successful delivery
• failed — delivery attempted but failed (check logs for details)
• dropped — event dropped because the queue was full
• skipped_no_secret — sink enabled but configRef Secret does not exist

A growing failed or dropped delta means alerts are not arriving. Check the API logs and verify each sink’s credential Secret exists and contains the correct keys for its kind.

Restart behavior

Notifications only fire for transitions observed while the API pod is running. If a backup starts and completes entirely while the API is down, no notification is sent (the event is still visible in the dashboard, kubectl, and Prometheus alerts like GameplaneBackupFailed). There is no watermark to replay missed events across restarts.

Next: Platform Settings & Telemetry