Runway Observability

Runway supports observability for a service by integrating with the monitoring stack. This includes both service-level metrics and load balancer observability across AWS, GKE, and Cloud Run environments.

Runway provides unified load balancer observability across all cloud environments using normalized runway_lb_* metrics.

The observability pipeline uses provider-native exporters with OpenTelemetry normalization:

GCP (GKE):

• Stackdriver Exporter deployed to designated clusters via ArgoCD
• Collects metrics from Cloud Load Balancing API
• OTel Gateway normalizes to runway_lb_* schema

AWS:

• CloudWatch Exporter deployed to EKS clusters via ArgoCD
• Collects metrics from CloudWatch API
• OTel Gateway normalizes to runway_lb_* schema

All metrics are exported to Mimir with X-Scope-OrgID: runway.

Request rate by runtime:

sum by (runtime) (rate(runway_lb_request_count{env="production"}[5m]))

Backend latency p99 across all clouds:

histogram_quantile(
  0.99,
  sum by (le, runtime) (
    rate(runway_lb_backend_latency_milliseconds_bucket{env="production"}[5m])
  )
)

Request drop rate by runtime (GKE only - EKS does not have backend_request_count):

sum by (runtime) (
  rate(runway_lb_request_count{env="production", runtime="gke"}[5m])
) - sum by (runtime) (
  rate(runway_lb_backend_request_count{env="production", runtime="gke"}[5m])
)

Request rate by HTTP status class:

sum by (response_code_class) (
  rate(runway_lb_response_code_count{runtime="eks", env="production"}[5m])
)

Backend latency statistics:

runway_lb_backend_latency_milliseconds{runtime="eks", env="production", statistic=~"average|maximum"}

Total latency p99 (GKE):

histogram_quantile(
  0.99,
  sum by (le, forwarding_rule_name) (
    rate(runway_lb_total_latency_milliseconds_bucket{runtime="gke", env="production"}[5m])
  )
)

The following informational dashboards are available in Grafana for collectively viewing load balancer stats across all Runway services and runtimes:

• Runway Load Balancer Metrics - Main - Unified cross-cloud dashboard with runtime comparison
• Runway Load Balancer Metrics - EKS - AWS ALB/NLB specific metrics
• Runway Load Balancer Metrics - GKE - GKE load balancer metrics
• Runway Load Balancer Metrics - CloudRun - Cloud Run load balancer metrics

Service-level metrics for Kubernetes (GKE and EKS) services are collected via OpenTelemetry collectors deployed to each cluster.

Runway Kubernetes services can be integrated with the runbooks observability stack to get out-of-the-box:

• Service overview Grafana dashboard — apdex, error rate, RPS, saturation panels
• SLO violation alerts — apdex, error rate, and traffic cessation
• Kubernetes saturation alerts — CPU, memory, HPA utilization

For full background on the metrics catalog, refer to the metrics-catalog README.

Step 1 — Add a service catalog entry

Add your service to services/service-catalog.yml in the runbooks repository. Ensure primary_grafana_dashboard points to <type>-main/<type>-overview.

Step 2 — Create a metrics catalog entry

Create metrics-catalog/services/my-service-gke.jsonnet using runway-k8s-archetype:

local k8sArchetype = import 'service-archetypes/runway-k8s-archetype.libsonnet';
local metricsCatalog = import 'servicemetrics/metrics.libsonnet';

metricsCatalog.serviceDefinition(
  k8sArchetype(
    type='my-service-gke',   // must match your runway_service_id
    team='my_team',
    featureCategory='my_feature_category',
    runtime='gke',
  )
)

Step 3 — Register in all.jsonnet

Add an import to metrics-catalog/services/all.jsonnet:

import 'my-service-gke.jsonnet',

Step 4 — Generate and commit

make generate

Commit all generated files. After the MR is merged, your dashboard will be available at https://dashboards.gitlab.net/d/my-service-gke-main.

The archetype automatically generates a frontend LB SLI using a url_map_name regex derived from your service type:

gkegw1-[^-]+-<type>-.*

This works for most services. If it does not match, find your exact url_map_name values by querying the Mimir - Runway datasource in Grafana Explore:

count by (url_map_name) (runway_lb_request_count{runtime="gke"})

Then override lbSelector explicitly:

k8sArchetype(
  type='my-service-gke',
  team='my_team',
  runtime='gke',
  lbSelector={
    url_map_name: { oneOf: [
      'gkegw1-l52v-my-service-gke-...',  // staging
      'gkegw1-ltbu-my-service-gke-...',  // production
    ] },
  },
)

Expose your service’s own Prometheus metrics; Runway collects them with OpenTelemetry and pushes them to Mimir, where you can query them and reference them as custom SLIs in the metrics catalog alongside the default LB-level SLIs.

The only difference between Runway v1 and v2 is how you declare the scrape target, in Expose the metrics for scraping below; instrumentation (next) and consumption are shared.

For Go services, use LabKit v2. The metrics package owns a Prometheus registry, and httpserver exposes it at /-/metrics on the probe port (:9090 by default):

import (
    "gitlab.com/gitlab-org/labkit/v2/httpserver"
    "gitlab.com/gitlab-org/labkit/v2/metrics"
)

m, err := metrics.New()
if err != nil {
    // handle error
}
srv := httpserver.NewWithConfig(&httpserver.Config{
    Addr:      ":8080", // your application
    ProbeAddr: ":9090", // serves /-/liveness, /-/readiness, /-/metrics
    Metrics:   m,
    Handler:   mux,
})
// srv.ListenAndServe() or similar to start it

Register your counters and histograms on m; the probe server serves them at /-/metrics.

Both versions render a Kubernetes Service named metrics; Runway’s OpenTelemetry collector discovers and scrapes services with ports named metrics-*.

Runway v2 (services with a .runway/fairway.yaml) — declare each endpoint under spec.metrics, then enable a ServiceMonitor in values.yaml:

spec:
  metrics:
    - port: 9090            # served at /-/metrics by LabKit v2
    # - port: 9091
    #   path: /metrics      # override the path per endpoint

metrics:
  prometheus:
    serviceMonitor:
      enabled: true

Enable the ServiceMonitor in values.yaml (applied to both staging and production), not in fairway.yaml’s spec.values.

Runway v1 — list each target under spec.observability.scrape_targets (the host:port your service serves metrics on):

spec:
  observability:
    scrape_targets:
      - "localhost:9090"

Your metrics are available in the Mimir - Runway datasource in Grafana.

Example query:

http_requests_total{env="production", kubernetes_namespace="my-service-gke"}

Once your metrics are available in Mimir, you can add them as SLIs in the metrics catalog:

local k8sArchetype = import 'service-archetypes/runway-k8s-archetype.libsonnet';
local metricsCatalog = import 'servicemetrics/metrics.libsonnet';
local rateMetric = metricsCatalog.rateMetric;

local type = 'my-service-gke';
local featureCategory = 'my_feature_category';

metricsCatalog.serviceDefinition(
  k8sArchetype(
    type=type,
    team='my_team',
    featureCategory=featureCategory,
    runtime='gke',
  )
  // Custom application-level SLIs on top of archetype defaults
  {
    serviceLevelIndicators+: {
      my_server: {
        userImpacting: true,
        featureCategory: featureCategory,
        requestRate: rateMetric(
          counter='http_requests_total',
          selector={ type: type },
        ),
        errorRate: rateMetric(
          counter='http_requests_total',
          selector={ type: type, status: '5xx' },
        ),
        significantLabels: ['handler', 'status'],
      },
    },
  }
)

See the metrics-catalog README for more details on defining SLIs.

Service-level observability via the metrics catalog is supported for Cloud Run services.

To get a service overview dashboard and SLO alerts:

1. Create a new entry in the service catalog in the expected format.
2. Create a new entry in the metrics catalog:

local runwayArchetype = import 'service-archetypes/runway-archetype.libsonnet';
local metricsCatalog = import 'servicemetrics/metrics.libsonnet';

metricsCatalog.serviceDefinition(
  runwayArchetype(
    type='my_service',
    team='my_team',
  )
)

3. Run make generate and commit all generated content.

After approval and merging, you can view the newly generated service overview dashboard.

By default, a dashboard is generated with:

• Default SLIs (e.g. runway_ingress)
• Default Saturation Details (e.g. runway_container_memory_utilization)

The dashboard is checked into version control and can be extended with custom SLIs. Optionally, you can use the general Runway Service Metrics dashboard.

Default metrics are reported under the stackdriver_cloud_run_* namespace in Mimir, even without a service catalog entry:

stackdriver_cloud_run_revision_run_googleapis_com_request_count{job="runway-exporter",env="gprd",service_name="my_service"}

To learn more, refer to Cloud Run metrics documentation.

Custom metrics can be reported using Prometheus text-based exposition format. When scrape targets are present, Runway deploys a sidecar OpenTelemetry Collector preconfigured to scrape your ingress container at the specified port(s):

spec:
  observability:
    scrape_targets:
      - "localhost:8082"
    metrics_path: "/foo" # defaults to /metrics

These custom metrics will be available under the Mimir - Runway data source in Grafana.

To learn more, refer to the Prometheus exporters documentation.

Alerts are generated automatically for any service with a metrics catalog entry. By default the following SLO violation alerts are created:

• Apdex SLO violation
• Error SLO violation
• Traffic absent / traffic cessation

Alerts are routed via Alertmanager to Slack and incident.io.

• Alerts default to the #feed_alerts-general Slack channel — this channel is very noisy. It is strongly recommended to route alerts to a channel you actively monitor.
• Only S1 or S2 severity alerts page the on-call SRE. S4 (default) alerts are reported to Slack only.
• To route alerts to a team Slack channel, specify a valid team in your metrics catalog entry.
• For full routing configuration, refer to the alert routing documentation.

To override alert thresholds, set the following fields in your metrics catalog entry:

Before setting S1 or S2 severity, your service must complete a production readiness review.

Runway application logs are available in Grafana via ClickHouse. You can query logs by filtering on ServiceName (your runway_service_id). Please refer to the Logging documentation.

Runway Kubernetes services (GKE and EKS) can export OpenTelemetry traces to Google Cloud Trace via the in-cluster OTel collector, using LabKit v2. Please refer to the Distributed Tracing documentation.