Hermes

# GitLab Runners

GitLab's built-in CI/CD tool called GitLab Runner handles continuous deployment (CD) for most Solarix GitLab repositories.

# Runner Manager

An auto-scaling GitLab Runner Manager EC2 instance (srn:ec2:solarix:core:gitlab::instance/runner-manager) manages all Solarix GitLab Runner tasks. The Runner Manager is actually itself a shared runner. When a new deployment pipeline is triggered anywhere in the Solarix GitLab instance the Runner Manager makes a request for an available t3a.xlarge EC2 Spot Instance. Once a reservation is confirmed the Manager creates a new Runner process on the instance, which executes all pipeline stages within a Docker container, typically one container per stage.

After the current pipeline completes the newly-reserved spot instance is still available for up to 60 minutes. If a new pipeline trigger occurs the Runner Manager will dispatch a new Runner to any existing, idle reserved instances. However, if all current instances are in use, the Runner Manager will automatically reserve a second instance, so there's as little downtime as possible.

After 60 minutes the spot instance reservation is terminated and the instance is terminated.

WARNING

If no spot instance is idling at the time of deployment then the first deployment will take ~90 seconds longer than expected for the initial spot instance reservation and instance propagation. All subsequent deployments using the same instance will not require that overhead.

# Configuration

The GitLab Runner Manager is entirely configured via its config.toml file. Below is the configuration powering the Solarix Runner Manager:

# /etc/gitlab-runner/config.toml
# ---
# This file should be placed on the machine with GitLab Runner CLI, Docker, and Docker Machine installed.
# This configuration behaves as a GitLab Runner manager by spinning up spot EC2 instances to perform actual jobs.
# ---

# Limits how many jobs globally can be run concurrently
concurrent = 10
# Defines the interval length, in seconds, between new jobs check
check_interval = 3

[[runners]]
  name = "solarix-gitlab-runner-manager"
  url = "https://gitlab.solarixdigital.com/"
  # solarix-gitlab-runner-manager - shared
  token = "<token>"
  executor = "docker+machine"
  # Limit how many jobs can be handled concurrently by this token.
  limit = 20
  [runners.docker]
    image = "alpine:latest"
    privileged = true
    disable_cache = true
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "<AccessKey>"
      SecretKey = "<SecretKey>"
      BucketName = "solarix-gitlab-runner-cache"
      BucketLocation = "us-west-2"
  [runners.machine]
    # Intended to be number of idle machines ready for work.
    # However, ongoing bug may require either +1 maximum or non-limit (0)
    # See: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2251
    # PR: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1624
    IdleCount = 0
    IdleTime = 1800
    MaxBuilds = 100
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = [
      # Ubuntu 18.04 AMI
      "amazonec2-ami=ami-003634241a8fcdec0",
      "amazonec2-access-key=<amazonec2-access-key>",
      "amazonec2-secret-key=<amazonec2-secret-key>",
      "amazonec2-region=us-west-2",
      "amazonec2-vpc-id=vpc-04e2b51f6b88610c0",
      "amazonec2-subnet-id=subnet-0639a3f4059868fe6",
      # Must match subnet
      "amazonec2-zone=a",
      "amazonec2-tags=runner-manager-name,solarix-gitlab-runner-manager,gitlab,true,gitlab-runner-autoscale,true",
      # Default security group allows all VPC instances to accept incoming connections from these temp runner instances
      "amazonec2-security-group=default",
      "amazonec2-instance-type=t3a.xlarge",
      # Use spot instances and set maximum price and block duration, minimum 60 min
      "amazonec2-block-duration-minutes=60",
      "amazonec2-request-spot-instance",
      "amazonec2-spot-price=0.09"
    ]

See the official documentation for more information on configuration. See the full config.toml file in the repo here.

# Low Costs

A reserved instance (srn:ec2:solarix:core:gitlab::instance/runner-manager) is the only constant requirement to handle all Solarix CI/CD, which runs on a t2.micro instance at a cost of ~$8.47 / month. Each t3a.xlarge 60-minute Spot reservation is $0.0451 / hour, which is approximately 1/3rd of the normal On-Demand cost. Since most days only see a handful of deployments the cost is typically $0.10 - $0.20 / day, or $3.00 - $6.00 / month. Running just a single t3a.xlarge 24/7 On-Demand would be well over $100 / month, and we have the advantage of on-demand scaling when a lot of deployments are happening during the day.

# Initial Docker Machine Instantiation

WARNING

This section only applies for the creation of an entirely new Runner Manager, such as for use within a new GitLab instance.

GitLab Runner's docker machine executor has trouble during first-time creation as multiple processes fight with one another (see docs).

For this reason, it is recommended to manually create an appropriate docker machine instance with the same options used in config.toml:

  • Bash
  • PowerShell 
docker-machine create --driver amazonec2 \
    --amazonec2-ami ami-003634241a8fcdec0 \
    --amazonec2-access-key <amazonec2-access-key> \
    --amazonec2-secret-key <amazonec2-secret-key> \
    --amazonec2-region us-west-2 \
    --amazonec2-vpc-id vpc-04e2b51f6b88610c0 \
    --amazonec2-subnet-id subnet-0639a3f4059868fe6 \
    --amazonec2-zone a \
    --amazonec2-tags runner-manager-name,solarix-gitlab-runner-manager,gitlab,true,gitlab-runner-autoscale,true \
    --amazonec2-security-group default \
    --amazonec2-instance-type t3a.xlarge \
    --amazonec2-block-duration-minutes 60 \
    --amazonec2-request-spot-instance \
    --amazonec2-spot-price 0.09 \
    gitlab-docker-machine-1234567890

After this is successfully run for the first time the instance can be safely removed (by deleting the spot request). Future automated creations should work as normal.

← Releases →