GitHub Actions Runner Platform

Problem

GitHub-hosted runners were fast and convenient, but the compute cost didn't scale with our usage. We needed a self-hosted alternative.

As our Paved Road scaled to more users, those self-hosted runners started showing cracks:

• Long queue times
• Infrastructure instability
• Runners were too lightweight, leading to slow dependency and artifact downloads, which led to even longer queue times
• Not enough runners, too many runners, not enough runners, too many runners

We were dependent on on-prem self-hosted machines, which gave us good compute per cost — but because of network issues and intermittent outages, we'd run out of runners entirely.

At that point, the very users we had successfully onboarded onto the Paved Road were now experiencing delays and outages — and the support channel noise let us know.

Those of us lucky enough to be on-call felt it usually at 2am EST.

My Role

This was a team effort. I focused on two core problems:

• Runner startup times were slow because every ephemeral runner had to redownload dependencies and artifacts.
• Runner capacity constantly swung between overprovisioned (wasted compute) and underprovisioned (long queue times).

At the time, I was still relatively new to Kubernetes and Terraform, so I spent the first couple of weeks understanding how Actions Runner Controller worked internally, why the existing architecture behaved the way it did, and where the scaling bottlenecks actually were.

What I Built

• Runner startup times were slow because every ephemeral runner had to redownload dependencies and artifacts.

The answer was hidden in the problem itself. Every runner started from a minimal base image, so each workflow spent time reinstalling the same tools before any application code could run.

I built a custom runner image on top of our base image that pre-installed the most commonly used tooling, closely mirroring GitHub-hosted runners without unnecessarily bloating the image. This shifted installation work from every job execution to the image build process, reducing startup time while preserving the flexibility of ephemeral runners.

I then updated our Helm charts so runner pods used the custom image alongside a Docker-in-Docker (DinD) sidecar, allowing every newly provisioned runner to start with the required tooling already available.

• Runner capacity constantly swung between overprovisioned (wasted compute) and underprovisioned (long queue times)

I led the migration from the original RunnerDeployment model to the newer RunnerScaleSet architecture in ARC v2. Although the newer model is conceptually simpler, it relies on an entirely different scaling mechanism, so the migration required redesigning how runners were provisioned.

Instead of relying on Kubernetes HPA to guess capacity, ARC began watching GitHub's job queue and provisioning ephemeral runners only when work existed. That allowed us to eliminate idle runners, scale all the way down to zero when inactive, and significantly reduce queue times while improving availability. But also scale all the way up at peak time should we need more runners.

Challenges

• How did you migrate without interrupting thousands of developers?
• How did you validate the new scaling behavior?
• Did you have to redesign Helm charts?
• Were there authentication changes?
• Did you run both systems side-by-side?
• What metrics told you the migration was successful?

Impact

...

...

Lessons

What I'd change...

Tradeoffs...

Architecture...