Deploying from ops.gitlab.net

Background

Runway workloads that could critically impact gitlab.com’s availability can be deployed from ops.gitlab.net. This decouples the workload’s deployment process from the health of gitlab.com and allows for rollbacks/rollforwards during outages.

Summary of the process:

As a Runway service owner, you need to ensure your service project is mirrored to ops.gitlab.net and file the provisioner MR to add/update your workload.
As a Runway maintainer, you need to review (but not merge) the provisioner change. You also need to file the CR for performing the Terraform state migration.
As a Runway maintainer, once the CR is approved, you need to follow the steps in the CR.

As a Runway service owner

This process assumes that your service repository is currently managed (or will be managed) by infra-mgmt, which is a repository (powered by Terraform) that manages projects and groups across gitlab.com and ops.gitlab.net. It makes setting up and managing projects and groups, mirrors and Vault configuration very easy.

The following steps will create (or manage) your canonical service repository and configure push mirroring to ops.gitlab.net:

Create a GitLab project on ops.gitlab.net for the service project

This repository will be the mirror for your canonical service repository.

You can create this repository using the infra-mgmt tooling by creating a project_*.tf file in the ops-gitlab-net directory. Refer to topology-service-deployer.
Create (or manage) your service repository on gitlab.com

If your canonical service repository is already managed by infra-mgmt then you can skip this step.

For reference, take a look at how topology-service is configured. You will need to create a project_*.tf file similar to this and tweak it to your liking.

If your canonical service repository already exists and you want infra-mgmt to manage it, you can set use_existing_project = true.

Configure push mirroring

In the same file that you edited in the previous step, you need to add a few lines to configure the push mirror settings. We will use a Personal Access Token (PAT) from the ops-gitlab-net user for this, which we can lookup in Vault using a data source.

The configuration should look like this:

data "vault_kv_secret_v2" "mirror-token-ops-deployed-runway-workload" {
  mount = "ci"
  name  = "${var.vault_secrets_base_path}/mirror-tokens/ops-deployed-runway-workloads"
}

module "project_ops-<YOUR PROJECT NAME>-mirror" {
  source  = "ops.gitlab.net/gitlab-com/project/gitlab//modules/push-mirror"
  version = "5.9.1"

  project_id                = module.<YOUR_CANONICAL_PROJECT>.id
  mirror_project            = { path = module.<YOUR_CANONICAL_PROJECT>.path_with_namespace }
  mirror_branch_protections = local.<YOUR_CANONICAL_PROJECT>_mirror_branch_protections
  only_protected_branches   = true
  keep_divergent_refs       = true

  custom_mirror_token = {
    token    = data.vault_kv_secret_v2.mirror-token-ops-deployed-runway-workload.data.token
    username = data.vault_kv_secret_v2.mirror-token-ops-deployed-runway-workload.data.username
    user_id  = data.gitlab_user.gl_infra_bot_ops["ops-gitlab-net"].user_id
  }

  providers = {
    gitlab.mirror = gitlab.ops-gitlab-net
  }
}

Add (or update) your Runway workload to provisioner

Open an MR on the provisioner project that either adds (if it does not exist) or updates your Runway workload entry with the deploy_from_ops field:
```
- name: test-runway-service
  project_id: 840         # this should be the service project id on gitlab.com
  deploy_from_ops:
    enabled: true
    project_id: 123       # this should be the service project id on ops.gitlab.net
```
This MR will need to be reviewed and merged by a Runway maintainer (see next section).

As a Runway maintainer

Runway maintainers need to review the provisioner MR for creating or transitioning a Runway workload to ops.gitlab.net.

The maintainer will perform the migration of Terraform state from gitlab.com to ops.gitlab.net as part of a change management process.

Review provisioner MR to set up ops deployment dependencies

Ensure deploy_from_ops is set with enabled: true and validate the service project IDs. There should be a project_id setting above deploy_from_ops that references the service project ID on gitlab.com, and another project_id nested below deploy_from_ops that references the service project ID on ops.gitlab.net.

Set up a local environment for running the migration script

Create a temporary folder in a path of your choosing containing 2 files (.env.sh and main.tf). The .terraform folder will be created in this folder during the migration process.
Create a Project Access Token (PrAT) for both deployment projects with a short expiry (long enough to get the CR executed). The token should have api scope and maintainer access level.

Populate an .env.sh file like the following ensuring you fill out the values:

# Get the project ids from the deployment project on gitlab.com and ops.gitlab.net
export SOURCE_PROJECT_ID=
export TARGET_PROJECT_ID=

# PrAT for the deployment project on gitlab.com and ops.gitlab.net
export SOURCE_TF_PASSWORD=
export TARGET_TF_PASSWORD=

# The address is used by terraform to connect to a remote http backend.
# Replace gitlab.example.com with the correct url, either gitlab.com/ops.gitlab.net
export SOURCE_TF_ADDRESS=https://gitlab.example.com/api/v4/projects/${SOURCE_PROJECT_ID}/terraform/state
export TARGET_TF_ADDRESS=https://gitlab.example.com/api/v4/projects/${TARGET_PROJECT_ID}/terraform/state

Prepare a main.tf for bootstrapping the initialization.

terraform {
  required_version = ">= 1.7"
}

terraform {
  backend "http" {}
}

Open a Change Management issue

You need to create an issue in the production issue tracker using the change_management template.

For the change steps, you may use the following as a guideline:

- [ ] Check with the service owner to ensure that no MRs are merged during the migration window.
- [ ] Lock the terraform states, this prevents any accidental deployments. See [example](https://gitlab.com/gitlab-com/gl-infra/platform/runway/deployments/schin1-ai-assist-1579xx/-/terraform)
- [ ] Merge the provisioner MR.
- [ ] Check if the new deployment project is found in ops.gitlab.net/gitlab-com/gl-infra/platform/runway/deployments/.
- [ ] Run: `cd /path/to/your/temporary/directory` (path that contains the `.env.sh` file)
- [ ] **STAGING**
    - [ ] Unlock the `staging` Terraform state on the canonical deployment project.
    - [ ] Run `/path/to/your/provisioner/repo/scripts/migrate_deployment_project_tfstate.sh staging`
    - [ ] Check the `staging` Terraform state on ops.gitlab.net.
    - [ ] Lock the `staging` terraform state on the canonical deployment project.
    - [ ] Remove the `.terraform` folder.
- [ ] **PRODUCTION**
    - [ ] Unlock the `production` Terraform state on the canonical deployment project.
    - [ ] Run `/path/to/your/provisioner/repo/scripts/migrate_deployment_project_tfstate.sh production`
    - [ ] Check the `production` Terraform state on ops.gitlab.net.
    - [ ] Lock the `production` terraform state on the canonical deployment project.
    - [ ] Remove the `.terraform` folder.

Refer to the change management page in the handbook for information on change criticalities, approvals needed, etc.

Update infra-mgmt

Add the Runway service id into the local.projects_deploying_from_ops list.

This will create a project access token for the ops.gitlab.net deployment project which enables the gitlab.com deployment project to perform dry-run jobs.