Distributed Tracing with Cloud Trace

Runway provides built-in distributed tracing for Kubernetes services using LabKit v2 and OpenTelemetry (OTLP). Spans are exported to the in-cluster OpenTelemetry Collector and forwarded to Google Cloud Trace in the shared Runway tenant GCP project (gitlab-runway-production or gitlab-runway-staging), where you filter by your service.name.

Runway injects the OTLP connection string automatically — you do not set it yourself:

GITLAB_TRACING=otlp://agent-otelcol-collector.monitoring.svc.cluster.local:4318?service_name=<runway_service_id>

From there, tracing is a two-part contract:

1. Your application builds a tracer from GITLAB_TRACING (automatic when you use the LabKit app.App framework; a one-liner otherwise).
2. You hand that tracer to the LabKit components you use (database, Redis, HTTP server/client, feature flags). Those components then emit spans automatically. A component only produces spans when it is given the tracer — LabKit does not auto-instrument globally.

app.New / app.NewWithConfig initializes the tracer for you from GITLAB_TRACING — you do not call trace.New. Retrieve it with a.Tracer() and pass it into each component’s config:

import (
    "gitlab.com/gitlab-org/labkit/v2/app"
    "gitlab.com/gitlab-org/labkit/v2/postgres"
    "gitlab.com/gitlab-org/labkit/v2/redis"
    "gitlab.com/gitlab-org/labkit/v2/httpserver"
)

a, err := app.New(ctx) // reads GITLAB_TRACING, builds the tracer + OTLP exporter

// Postgres — every query emits a "db SELECT" / "db INSERT" span.
pg, err := postgres.NewWithConfig(&postgres.Config{DSN: dsn, Tracer: a.Tracer()})

// Redis — every command emits a "db SET" / "db GET" span.
rdb, err := redis.NewWithConfig(&redis.Config{Addrs: addrs, Tracer: a.Tracer()})

// HTTP server — one server span per request; incoming W3C trace context is
// extracted automatically. The tracing middleware is wired for you.
srv := httpserver.NewWithConfig(&httpserver.Config{
    Addr: ":8080", Handler: mux, Tracer: a.Tracer(), Logger: a.Logger(),
})

a.Register(pg)
a.Register(rdb)
a.Register(srv)

Build the tracer yourself and pass it the same way:

import "gitlab.com/gitlab-org/labkit/v2/trace"

tracer, shutdown, err := trace.New(ctx) // reads GITLAB_TRACING
defer shutdown(context.Background())     // flushes buffered spans on exit

pg, err := postgres.New(ctx, postgres.WithTracer(tracer))

For a non-LabKit HTTP server (e.g. the standard library), use otelhttp and pass the provider explicitly — LabKit does not register a global TracerProvider:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

server := &http.Server{
    Addr:    ":8080",
    Handler: otelhttp.NewHandler(mux, "my-service", otelhttp.WithTracerProvider(tracer.Provider())),
}

The default sample rate is 1% (defaultSampleRate = 0.01). Sampling is decided head-based in the SDK; the collector does not re-sample.

To change it, set SampleRate in the trace config. Runway still supplies the collector endpoint via GITLAB_TRACING — the env var only fills fields left at their zero value, so you provide the endpoint from the environment and the sample rate from config:

// With the app framework:
a, err := app.NewWithConfig(ctx, &app.Config{
    Trace: &trace.Config{SampleRate: 0.05}, // sample 5%
})

// Standalone:
tracer, shutdown, err := trace.NewWithConfig(ctx, &trace.Config{SampleRate: 0.05})

See the LabKit tracing configuration for all options.

1. Agent collector (agent-otelcol, a DaemonSet) receives your OTLP spans on :4318, enriches them with k8sattributes (pod, namespace, deployment), and forwards to the gateway.
2. Gateway collector (gateway-otelcol) exports to Cloud Trace via the googlecloud exporter, targeting your tenant project gitlab-runway-{staging,production}.

1. Open the Cloud Trace console.
2. Select the tenant project — gitlab-runway-production or gitlab-runway-staging.
3. Filter by service.name matching your runway_service_id.

• LabKit v2 trace package
• Google Cloud Trace documentation
• Runway Observability reference

Reach out to the Runway team in the #f_runway Slack channel.