Kubernetes Secrets Management
Secrets Management for Kubernetes Workloads
Section titled “Secrets Management for Kubernetes Workloads”This guide explains how to securely manage and expose secrets to your Runway Kubernetes workloads (GKE and EKS). Runway uses GitLab’s Vault installation to securely store secrets, which are then automatically made available as environment variables to your containers.
Overview
Section titled “Overview”Runway uses a flexible approach to secrets management using Vault as the source of truth:
- Secrets are stored in Vault and referenced by your Runway configuration
- Multiple secret keys can be defined within a single Vault secret
- Each key in the secret becomes an environment variable in your container
Constraints
Section titled “Constraints”- Secret keys (which become environment variable names) should follow standard environment variable naming conventions (uppercase, letters/numbers/underscores only)
Setup Access to Vault
Section titled “Setup Access to Vault”To manage secrets, you need access to the GitLab Vault installation. Follow these steps:
- Create an Access Request to provision an Okta group (Example)
- Create a Merge Request to set secret policy in Vault (Example)
When complete, service owners will only be granted access to secrets for their services.
Recommendation: Step 1 can take ~2 weeks for IT due to change management, so start the process as soon as you anticipate the need for secrets management.
Managing Secrets in Vault
Section titled “Managing Secrets in Vault”Adding Secrets
Section titled “Adding Secrets”-
Determine your Runway service ID
This is the ID defined in the Runway workload configuration file:
-
Log into Vault
If you do not have access to Vault, follow the instructions here. The path you need access to is
runway/env/$environment/service/$runway_service_id
. -
Go to your workload’s secrets path
Navigate to
runway/env/$environment/service/$runway_service_id
where$environment
is the environment (eitherstaging
orproduction
) and$runway_service_id
is your service ID. A service without any secrets will have a single empty secret called.placeholder
. -
Create a secret with multiple keys
Click on the
Create secret
button. AtPath for this secret
, enter a descriptive name for your secret collection (e.g.,env-vars
ordatabase-credentials
).Under
Secret data
, add multiple key-value pairs:- Each key will become an environment variable name in your container
- Keys should be in uppercase and contain only letters, numbers, and underscores
- Values should be the actual secret values you want to expose
For example:
API_KEY: "your-api-key-here"DATABASE_PASSWORD: "your-database-password"JWT_SECRET: "your-jwt-secret" -
Save the secret
Click
Save
to store your secret in Vault.
Secret Deployment Timing
Section titled “Secret Deployment Timing”Runway on Kubernetes deploys secrets separately from code deployments. This differs from the Runway on Cloud Run behavior.
- Secret synchronization may take a few days to propagate to your Kubernetes cluster
- If you need secrets deployed urgently, contact the
#f_runway
Slack channel to request expedited deployment
Rotating Secrets
Section titled “Rotating Secrets”- Log into Vault and go to the secret path for your environment/service
- Click on the existing secret you wish to change
- Click on
Create new version
in the top right - Update the key-value pairs as needed
- Click
Save
The new secret values will be synchronized to your Kubernetes cluster according to the deployment schedule.
Removing Secrets
Section titled “Removing Secrets”- Log into Vault and go to the secret path for your environment/service
- Click on the
...
next to the secret, and choosePermanently delete
Alternatively, you can remove individual keys from a secret by editing it and deleting the specific key-value pairs.
Accessing Secrets in Your Application
Section titled “Accessing Secrets in Your Application”First, you need to list all secrets your application needs to access in the Runway configuration file, .runway/gke-service.yaml
:
# Example Runway configurationexample_service_id: vaultSecrets: - name: env-vars mount: runway path: env/staging/service/example_service_id/env-vars
This will make the secrets available in the cluster as Kubernetes Secret
resources.
Environment Variables
Section titled “Environment Variables”Next, tell the system to make the secret accessible as an environment variable.
This is done by creating or adding the secret to the secretEnvFrom
list in the .runway/gke-service.yaml
configuration:
# Example Runway configurationexample_service_id: vaultSecrets: - name: env-vars mount: runway path: env/staging/service/example_service_id/env-vars secretEnvFrom: - env-vars
This will inject all keys from the env-vars
secret as environment variables in your container.
Once deployed, all keys from the referenced secrets will be available as environment variables in your container. You can access them using your programming language’s standard method for reading environment variables.
Best Practices
Section titled “Best Practices”- Group related secrets: Use descriptive names for your secret collections and group related secrets together
- Limit access: Only grant Vault access to team members who need to manage secrets
- Use secure values: Generate strong, random values for secrets
- Rotate regularly: Establish a routine schedule for rotating sensitive credentials