Kubernetes Deployment Overview

For our EKS and GKE Runtimes (currently in development), we have processes that dynamically generate two key components: inventory and resource manifests.

Inventory Generation

A set of “Build/Verify/Upload Inventory” jobs run from our provisioner repository and they generate Flux objects based on the contents of the workloads.yml file. The output is similar to the following:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: example-service-gke
  namespace: runway-provisioner
spec:
  interval: 5m0s
  url: oci://registry.gitlab.com/gitlab-com/gl-infra/platform/runway/deployments/example-service-gke/deploy
  ref:
    tag: production
  secretRef:
    name: registry-reader
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: example-service-gke
  namespace: runway-provisioner
spec:
  interval: 5m0s
  sourceRef:
    kind: OCIRepository
    name: example-service-gke
    namespace: runway-provisioner
  prune: true

The above two objects are repeated for every workload (in this example for the workload example-service-gke).

Once the inventory YAML is generated, it is pushed using flux push artifact ... to oci://registry.gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/<runtime>-inventory:<environment>.

All GKE and EKS clusters are configured to monitor/apply changes using the following:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: ${cloud_provider}-inventory
  namespace: runway-provisioner
spec:
  interval: 5m0s
  url: oci://registry.gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/${cloud_provider}-inventory
  ref:
    tag: "${environment}"
  secretRef:
    name: inventory-reader
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: ${cloud_provider}-inventory
  namespace: runway-provisioner
spec:
  interval: 10m0s
  path: "./${environment}"
  prune: true
  sourceRef:
    kind: OCIRepository
    name: ${cloud_provider}-inventory

This covers how our GKE/EKS clusters become aware of new workloads, but what objects get deployed for each workload?

Resource Manifests Generation

Service repositories will typically have an include similar to the following:

include:
  - project: 'gitlab-com/gl-infra/platform/runway/runwayctl'
    file: 'ci-tasks/service-project/helm-chart.yml'
    inputs:
      runway_service_id: example-service-gke
      runway_version: v3.73.0

The following diagram illustrates how your service repository and the deployment repository work together to create the resource manifests for your workload:

Resource Manifests Generation

Once the resource manifests are uploaded, Flux will periodically check for an updated deploy artifact and reconcile any changes.