BYO GCP Folder

What does BYO GCP Folder mean?

BYO (Bring Your Own) GCP (Google Cloud Platform) Folder is a Runway feature that allows you to deploy your Runway workloads to your own GCP folder. By default, Runway will deploy to gitlab-runway-<environment> and these projects live in the GitLab.com organization, however you can also configure Runway to deploy to your own GCP folder, which may live in a different organization.

Process

1. Ensure the role roles/billing.user is granted to the provisioner SA on the billing account

   You need to ensure the provisioner SA ([email protected]) has the role roles/billing.user for the billing_account that you will define in inventory.yml.

2. Create GCP folder where Runway will create projects

   You need to create the folder(s) where you would like Runway to create projects. In Terraform, it would look like this:

   resource "google_folder" "runway_services_folder" {
     display_name = "Runway"
     parent       = google_folder.your-parent-folder.name
   }

3. Grant necessary roles to the provisioner at the GCP folder level

   For the folder(s) you created in the previous step, ensure the roles roles/resourcemanager.projectCreator and roles/billing.projectManager have been granted to the provisioner ([email protected]).

   In Terraform, it would look like this:

   locals {
     runway_provisioner_sa = "[email protected]"
   
     runway_folder_roles = [
       "roles/resourcemanager.projectCreator",
       "roles/billing.projectManager"
     ]
   }
   
   resource "google_folder" "runway_services_folder" {
     display_name = "Runway"
     parent       = google_folder.your-parent-folder.name
   }
   
   resource "google_folder_iam_member" "runway-folder-roles" {
     for_each = toset(local.runway_folder_roles)
   
     folder = google_folder.runway_services_folder.folder_id
     role   = each.key
     member = "serviceAccount:${local.runway_provisioner_sa}"
   
     depends_on = [
       google_org_policy_policy.allowed_policy_member_domains
     ]
   }

4. Ensure Cloud Run services can allow unauthenticated requests

   For each Runway service, we define an IAM policy that allows allUsers access to the roles/run.invoker role. Terraform will fail to apply this policy if you have DRS (Domain Restricted Sharing) enabled with an error similar to:

   Error 400: One or more users named in the policy do not belong to a permitted customer,  perhaps due to an organization policy

   There are a few ways to fix this, but the way we recommend is to conditionally enforce DRS as follows:

   1. Add a tag binding to the Runway GCP folder (example)
   2. Update your DRS organizational policy such that it will exclude resources with specified tags from DRS (part of the MR above)
   3. Bind roles/resourcemanager.tagAdmin to the SA applying Terraform. This allows it to create new tags (example).
   4. Bind roles/resourcemanager.tagUser to the SA applying Terraform. This allows it to bind tags (example).

   See this blog post for more info on this approach.

5. Configure gcp_project_groups in provisioner’s inventory.yml

   You are now ready to configure the provisioner. Find the gcp_project_groups key in the inventory.yml file and add the following:

   - name: services
     envs:
       staging:
         folder_id: "<folder ID>"
         billing_account: "<billing account ID>"
         exporter_sa_email: "<exporter SA email>" # optional

   Go ahead and merge this change and wait for it to be provisioned. Ensure you get a clean pipeline run with no errors.

   If the exporter_sa_email field is empty in inventory.yml, the default exporter SA (runway-exporter-k8s@gitlab-{production/staging-1}.iam.gserviceaccount.com) will be granted the roles/monitoring.viewer role for the GCP project. Next, update runway-exporter to include the new GCP project id in the list of --google.project-id for runway-exporter to scrape stackdriver metrics.

   Alternatively, you may provide your own exporter_sa_email if you want to scrape metrics using a different set of runway-exporters.

6. Create your Runway workload

   You should now be able to add your workload to the inventory.yml. It is very similar to how you would add any other workload with the addition of gcp_project_group:

   - name: your-workload
     project_id: <your service project ID>
     gcp_project_group: services

Limitations

1. We have VPC peering set-up between gitlab-production and gitlab-runway-production (and similarly for staging), which allows Rails to reach internal-only facing Runway workloads. By deploying to your own GCP folder, if you need to access your workloads via an internal-only load balancer, you will need to configure VPC peering yourself. Runway does support creating VPC + subnetworks for internal load balancers. See the schema for more info.

2. Deleting an entry from gcp_project_groups, applying it and attempting to add it again (with the same name) will not work. This is because once you apply the MR to delete the GCP project group, it will destroy all its resources including its IAM workload identity pool. These pools can be undeleted for up to 30 days before they are permanently deleted. Until then, you cannot reuse the name, so you cannot simply recreate the same GCP project group. You can either pick a new name OR you can manually undelete the workload identity pool and import the resources (example).