Getting Started

What is Runway?

Runway is a platform to streamline the deployment of projects outside of the main GitLab codebase. If you have a separate codebase containing a service or a web application that can be distributed as a docker container, Runway can help you easily deploy it to staging and production environments. It’s in development right now but is already usable for simple deployment scenarios.

How do I start using Runway?

Add service to inventory

File an MR to add a service to the inventory.yml file in the provisioner project.

- name: example-workload
  project_id: 123456789 # GitLab Project ID
  members:
    - gitlab-username
  groups:
    - gitlab-group
  regions:
    - asia-northeast3
    - us-east1

- name: foo-frontend
  project_id: 123456789
  members:
    - gitlab-username
  groups:
    - gitlab-group
- name: foo-backend
  project_id: 123456789
  members:
    - gitlab-username
  groups:
    - gitlab-group

There are two ways the service project could be set up:

Retrieve name of generated deployment project

Once inventory changes are merged, the following happens:

1. A deployment GitLab project is created in the Deployments group, as well as a service account for executing deployments.
2. Deployment details are located in the downstream pipeline in the created project. This pipeline should be triggered by the CI/CD pipeline of the project.

For the next step you will need to get the Runway service name of your service, which is the name of the project you can find in Deployments. The format will be {name}-{hash}. For example, the service id of a service created in a previous step could be eval-code-viewer-78do0s.

Update job token permissions

The following steps will allow your deployment project access to your service repository required for Runway to function (see docs).

1. Open Settings for your service project
2. Go to CI-CD
3. Open the Job token permissions drop-down
4. Click on Add group or project
5. Enter your generated deployment project (from the previous step)

Create Runway configuration files

In the project that holds your code (known as the service repository), you will need to create the following files:

runway.yml

Format

The contents of this file depends on the type of workload you are deploying:

• RunwayService: this type of workload deploys a Cloud Run Service. You can configure the service with an external and/or internal load balancer.

  Example: woodhouse, which listens for Slack API requests and performs actions on them.

• RunwayJob: this type of workload deploys a Cloud Run Job, which can be configured to trigger on a schedule or on-demand via CI.

  Example: woodhouse-handover, which creates an EOC handover issue at set times of the day.

Location

Reconciler will first check if the path .runway/<runway service id>/ exists. If it does, it will look for config files (e.g., runway.yml) under this path. If the path does not exist, then it will search under .runway/.

Example tree structure for running a single deployment:

Example tree structure for running multiple deployments:

In the example above, when running a deployment for Runway service ID foo-backend, it will use the config files under the foo-backend path, however deployments for any other Runway service ID will use the config files under .runway/.

Alternatively, to be explicit, you could also make it look like the following and it will have the same effect with the exception that deployments for any other Runway service ID will fail because there is no runway.yml (nor env-*.yml) file in the .runway/ directory:

Once you have your config files in the right places, scroll down to read the section on updating your project’s .gitlab-ci.yml to trigger multiple deployments.

Stage-specific runway.yml

If you only have a single runway.yml file then it will be used for all stages. You can, alternatively, create different Runway config files for each stage by defining runway-<stage>.yml.

Example tree structure might look like this:

• In the foo-frontend service above, the production deployment would use runway.yml (as there is no runway-production.yml) while staging would use runway-staging.yml.

• In the foo-backend service above, we are being explicit by defining a config file for each stage (runway-staging.yml and runway-production.yml). Runway currently supports two stages (staging and production). It is your choice whether to be explicit or not, however we do recommend being explicit purely from a future-proof perspective so that if Runway ever supports more stages, they will not accidentally use your runway.yml file.

env-staging.yml and env-production.yml

Inside these two files, which can live in .runway/ or .runway/<runway service ID>/ depending on whether you are deploying a single (former) or multiple services (latter), is where you define any environment variables that you wish to make available to the containers running in either the staging or production environments.

Given that these files are cleartext and stored in your project repository, you must not place any secrets inside these files. See the documentation on secrets management for more info.

Example contents of one of these files might be:

FASTAPI_API_PORT: "8080"
GITLAB_URL: "https://staging.gitlab.com"
GITLAB_API_URL: "https://staging.gitlab.com/api/v4/"

Trigger Runway deployments

stages:
  - validate
  - runway_staging
  - runway_production

include:
  - project: 'gitlab-com/gl-infra/platform/runway/runwayctl'
    file: 'ci-tasks/service-project/runway.yml'
    ref: v2.21.0
    inputs:
      runway_service_id: eval-code-viewer-78do0s
      image: "$CI_REGISTRY_IMAGE/deploy:$CI_COMMIT_SHORT_SHA"
      runway_version: v2.21.0

stages:
  - validate
  - runway_staging
  - runway_production

include:
  - project: 'gitlab-com/gl-infra/platform/runway/runwayctl'
    file: 'ci-tasks/service-project/runway.yml'
    ref: v2.47.1
    inputs:
      runway_service_id: foo-frontend
      image: "$CI_REGISTRY_IMAGE/deploy:$CI_COMMIT_SHORT_SHA"
      runway_version: v2.47.1 # minimum version which supports multi-deployment
  - project: 'gitlab-com/gl-infra/platform/runway/runwayctl'
    file: 'ci-tasks/service-project/runway.yml'
    ref: v2.47.1
    inputs:
      runway_service_id: foo-backend
      image: "$CI_REGISTRY_IMAGE/deploy:$CI_COMMIT_SHORT_SHA"
      runway_version: v2.47.1 # minimum version which supports multi-deployment

Ensure you have your config files in the right places by reading the section above.

Check the deployment

Once the code above is integrated into the main branch of your project, you can check Pipelines section and see if it was successfully deployed.

If you expand the jobs on the downstream pipelines, you will see a number jobs:

Each of the downstream pipelines corresponds to a deployment to a stage. You can see the stage name is part of almost all jobs (e.g., Deploy production). You can also appreciate that the runway_staging stage (in GitLab terminology) comes before runway_production meaning that the staging downstream pipeline has to pass for production to start. As part of the downstream pipeline jobs, there is a monitor job that runs after the deploy, which performs basic error rate checks to determine if the service is healthy after the deploy. If the staging monitor job passes, then the production deployment will automatically start.

Accessing your environment

The best place to find the links to access your environments (once they have been deployed) is through Operate → Environments in your service project. In there you should see two environments, staging and production. Clicking on the Open button will take you to the URL to access that environment:

Keeping up with Runway releases

Renovate is a tool that checks for new dependency versions and, if found, will periodically create MRs to upgrade them to the newest version. It is used extensively within “Infrastructure”.

The easiest way to keep up with Runway releases is to enable Renovate CI triggered merge requests on your service project by doing the following:

1. Add @glrenovatebot as a Developer for your project under Manage > Members.
2. Add runway-workloads as a topic in your project under Settings > General > Naming, description, topics.
3. Set RENOVATE_DISABLED to true as a CI/CD variable if you are currently getting MRs filed by renovate-bot.yml from common-ci-tasks

If you’re not already using Renovate, once you’ve added the runway-workloads topic to your service project, Renovate CI will be able to discover your project and onboard it by opening a merge request like this one.

If you have an existing renovate.json file, Renovate will not update it for you so you will need to update the file yourself and add Runway’s Renovate config preset to the list of base configs (i.e. the extends keyword). For example:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "gitlab>gitlab-com/gl-infra/common-ci-tasks:renovate-common",
    "gitlab>gitlab-com/gl-infra/platform/runway/runwayctl:renovate-runway"
  ]
}

This will look for patterns like runway_version: v2.40.0 in your .gitlab-ci.yml file and keep the version up to date.

Troubleshooting

• If your deployment fails for any reason, check the deployment logs.
• If the downstream pipeline failed to be executed, there’s probably a permission mismatch.

Currently, a person triggering a deployment must have “Developer” permissions in the runwayctl project and “Maintainer” permissions in the generated deployment project (example). To grant additional permissions, update the inventory.yml file in the provisioner repo from step 1.

Frequently Asked Questions

How do I roll back the deployment?

At the moment the designed way to roll back a deployment is roll back a commit on the main branch, which will trigger a new deployment to Runway with the reverted code.

How do I view metrics?

By default, metrics are reported for all Runway services. To view metrics, you can use service filter on Runway Service Metrics dashboard.

Recommendation: For additional features, such as custom service overview dashboard, alerts, and capacity planning, refer to the observability page.

How do I view logs?

By default, application container logs can be viewed in Cloud Logging UI by filtering resource.labels.service_name to your runway_service_id.

To learn more, refer to the observability page.

How do I fix Terraform state error?

Insufficient permissions

When a pipeline is triggered with insufficient permissions, failed job includes following error message:

Terraform has been successfully initialized!
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: HTTP remote state endpoint invalid auth
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵
Error: failed to execute terraform command: exit status 1

To fix, there are two potential options:

1. Job can be retried by member with the required permissions, or
2. Add your GitLab username to members attribute in inventory.yml and retry the job yourself (example MR).