Skip to content

Kubernetes Secrets Management

Secrets Management for Kubernetes Workloads

Section titled “Secrets Management for Kubernetes Workloads”

This guide explains how to securely manage and expose secrets to your Runway Kubernetes workloads (GKE and EKS). Runway uses GitLab’s Vault installation to securely store secrets, which are then automatically made available as environment variables to your containers.

Runway uses a flexible approach to secrets management using Vault as the source of truth:

  1. Secrets are stored in Vault and referenced by your Runway configuration
  2. Multiple secret keys can be defined within a single Vault secret
  3. Each key in the secret becomes an environment variable in your container
  • Secret keys (which become environment variable names) should follow standard environment variable naming conventions (uppercase, letters/numbers/underscores only)

To manage secrets, you need access to the GitLab Vault installation. Follow these steps:

  1. Create an Access Request to provision an Okta group (Example)
  2. Create a Merge Request to set secret policy in Vault (Example)

When complete, service owners will only be granted access to secrets for their services.

Recommendation: Step 1 can take ~2 weeks for IT due to change management, so start the process as soon as you anticipate the need for secrets management.

  1. Determine your Runway service ID

    This is the ID defined in the Runway workload configuration file:

  2. Log into Vault

    If you do not have access to Vault, follow the instructions here. The path you need access to is runway/env/$environment/service/$runway_service_id.

  3. Go to your workload’s secrets path

    Navigate to runway/env/$environment/service/$runway_service_id where $environment is the environment (either staging or production) and $runway_service_id is your service ID. A service without any secrets will have a single empty secret called .placeholder.

  4. Create a secret with multiple keys

    Click on the Create secret button. At Path for this secret, enter a descriptive name for your secret collection (e.g., env-vars or database-credentials).

    Under Secret data, add multiple key-value pairs:

    • Each key will become an environment variable name in your container
    • Keys should be in uppercase and contain only letters, numbers, and underscores
    • Values should be the actual secret values you want to expose

    For example:

    API_KEY: "your-api-key-here"
    DATABASE_PASSWORD: "your-database-password"
    JWT_SECRET: "your-jwt-secret"
  5. Save the secret

    Click Save to store your secret in Vault.

Runway on Kubernetes deploys secrets separately from code deployments. This differs from the Runway on Cloud Run behavior.

  • Secret synchronization may take a few days to propagate to your Kubernetes cluster
  • If you need secrets deployed urgently, contact the #f_runway Slack channel to request expedited deployment
  1. Log into Vault and go to the secret path for your environment/service
  2. Click on the existing secret you wish to change
  3. Click on Create new version in the top right
  4. Update the key-value pairs as needed
  5. Click Save

The new secret values will be synchronized to your Kubernetes cluster according to the deployment schedule.

  1. Log into Vault and go to the secret path for your environment/service
  2. Click on the ... next to the secret, and choose Permanently delete

Alternatively, you can remove individual keys from a secret by editing it and deleting the specific key-value pairs.

First, you need to list all secrets your application needs to access in the Runway configuration file, .runway/gke-service.yaml:

# Example Runway configuration
example_service_id:
vaultSecrets:
- name: env-vars
mount: runway
path: env/staging/service/example_service_id/env-vars

This will make the secrets available in the cluster as Kubernetes Secret resources.

Next, tell the system to make the secret accessible as an environment variable. This is done by creating or adding the secret to the secretEnvFrom list in the .runway/gke-service.yaml configuration:

# Example Runway configuration
example_service_id:
vaultSecrets:
- name: env-vars
mount: runway
path: env/staging/service/example_service_id/env-vars
secretEnvFrom:
- env-vars

This will inject all keys from the env-vars secret as environment variables in your container.

Once deployed, all keys from the referenced secrets will be available as environment variables in your container. You can access them using your programming language’s standard method for reading environment variables.

  1. Group related secrets: Use descriptive names for your secret collections and group related secrets together
  2. Limit access: Only grant Vault access to team members who need to manage secrets
  3. Use secure values: Generate strong, random values for secrets
  4. Rotate regularly: Establish a routine schedule for rotating sensitive credentials