Skip to main content

Self-Hosted GitHub Runner (decommissioned)

Historical record — instance terminated

The shared aiqlick-gha-runner EC2 (i-0b13fda708cf21e54, t3.medium) no longer exists. All workflows in aiqlick-backend, background-tasks, aiqlick-frontend, and documentation now run on GitHub-hosted ubuntu-latest runners with OIDC against github-actions-role for AWS access — see CI/CD Pipelines. The Phase 7 ECS migration removed the SSH-deploy step that needed the same-VPC runner for low-latency SSH access, so the ~30 s deploy advantage no longer materially beats ubuntu-latest (current backend deploys land in ~3-4 min, BG in ~3 min — dominated by Docker image push time, which the persistent runner cache helped with but is no longer worth the maintenance burden).

The page below is preserved for context — useful if we ever need to spin up a self-hosted runner again, or to understand decisions in older deploy logs.

All four AIQLick repos route their GitHub Actions workloads to runners on a single shared EC2 instance instead of GitHub-hosted ubuntu-latest. This bypasses the org-wide GitHub Actions spending cap entirely (CI billing is now AWS-only) and brings deploys down to ~30 s end-to-end versus 3–5 min on the hosted runners.

Why we switched​

In April 2026 the AiQlickProject org hit GitHub's monthly Actions spending cap mid-day and every queued deploy started failing with:

The job was not started because recent account payments have failed or your spending limit needs to be increased.

Raising the cap is a UI-only operation on github.com (no API to lift it from the CLI). To remove the dependency entirely, every workflow now runs on our own EC2 — same workflow YAML otherwise, just runs-on: [self-hosted, <label>] instead of runs-on: ubuntu-latest.

Architecture​

One EC2 hosts four runner processes, each registered to its own repo with a unique label. Concurrent deploys across repos run in parallel; concurrent pushes within a single repo serialise on that repo's runner.

Runner inventory​

RepoWorkflow usesRunner dirService name
background-tasksruns-on: [self-hosted, aiqlick-bg]/home/ec2-user/actions-runner/actions.runner.AiQlickProject-background-tasks.aiqlick-runner-1
aiqlick-backendruns-on: [self-hosted, aiqlick-backend]/home/ec2-user/actions-runner-backend/actions.runner.AiQlickProject-aiqlick-backend.aiqlick-runner-backend
aiqlick-frontendruns-on: [self-hosted, aiqlick-frontend]/home/ec2-user/actions-runner-frontend/actions.runner.AiQlickProject-aiqlick-frontend.aiqlick-runner-frontend
documentationruns-on: [self-hosted, aiqlick-docs]/home/ec2-user/actions-runner-docs/actions.runner.AiQlickProject-documentation.aiqlick-runner-docs

The two-label form ([self-hosted, <repo-label>]) ensures each job only lands on its intended runner — no risk of cross-repo scheduling.

Host properties​

PropertyValue
Instanceaiqlick-gha-runner (i-0b13fda708cf21e54)
Typet3.medium (2 vCPU, 4 GB RAM) — upgraded from t3.small after the third runner was added
AMIAmazon Linux 2023 (ami-0a0823e4ea064404d)
Disk30 GB gp3
VPC / Subnetvpc-0aa980c32ac6870d2 / subnet-076d83db3eda5e363 (eu-north-1a) — same as the prod app instances for fast ECR access
Security groupgha-runner-sg (sg-07a76013bca16da0a) — outbound only, no inbound
IAM instance profileEC2-SSM-Profile (just SSM agent — workflow auth uses GitHub OIDC)
Runner userec2-user (member of docker group)
Runner versionactions-runner-linux-x64-2.334.0 (auto-upgraded by GitHub)

The runners authenticate to AWS the same way GitHub-hosted runners did — via OIDC against the github-actions-role. The EC2 instance profile is not used for workflow AWS calls; it only exists so we can SSM into the host for maintenance. This means moving workflow steps between hosted and self-hosted runners requires no changes to AWS IAM.

Capacity​

At idle, all four runners use ~470 MB combined. RAM headroom for builds:

Concurrent buildsStatus
1 (typical)Comfortable — ~3 GB free for Docker / npm / yarn
2Safe
3Tight; close to limit on heavy builds
4At capacity — risk of OOM on simultaneous Docker builds

If sustained concurrency >= 3 becomes the norm, jump to t3.large (8 GB RAM, ~$60/mo). Today the workload is well within t3.medium budget.

Bootstrap a new repo​

Adding a fifth repo (or rebuilding any of the existing runners) follows the same pattern:

# 1. Get a one-shot registration token for the target repo
REG_TOKEN=$(gh api repos/AiQlickProject/<REPO>/actions/runners/registration-token \
  -X POST --jq '.token')

# 2. Provision dirs + register via SSM (one shot)
aws ssm send-command --profile Administrator-842697652860 --region eu-north-1 \
  --instance-ids i-0b13fda708cf21e54 \
  --document-name AWS-RunShellScript \
  --parameters "commands=[
    'sudo -u ec2-user bash -c \"mkdir -p /home/ec2-user/actions-runner-<LABEL> && cd /home/ec2-user/actions-runner-<LABEL> && curl -sL -o /tmp/r.tar.gz https://github.com/actions/runner/releases/download/v2.334.0/actions-runner-linux-x64-2.334.0.tar.gz && tar xzf /tmp/r.tar.gz\"',
    'sudo -u ec2-user bash -c \"cd /home/ec2-user/actions-runner-<LABEL> && ./config.sh --url https://github.com/AiQlickProject/<REPO> --token ${REG_TOKEN} --name aiqlick-runner-<LABEL> --labels self-hosted,linux,x64,aiqlick-<LABEL> --work _work --unattended --replace\"',
    'cd /home/ec2-user/actions-runner-<LABEL> && ./svc.sh install ec2-user && ./svc.sh start'
  ]"

# 3. Switch the repo's workflow YAML
#    runs-on: ubuntu-latest  →  runs-on: [self-hosted, aiqlick-<LABEL>]
caution
installdependencies.sh doesn't recognise AL2023

The runner ships an installdependencies.sh helper, but it bails on Amazon Linux 2023 with Can't detect current OS type based on /etc/os-release. The .NET 6 runtime needs libicu krb5-libs zlib openssl-libs — install them once with dnf install -y libicu krb5-libs zlib openssl-libs and skip the helper. Existing runners on this host already have these.

Operational commands​

# Status of all runners
sudo systemctl list-units --state=active --no-pager | grep actions.runner

# Status of one runner
sudo systemctl status actions.runner.AiQlickProject-<repo>.<name>

# Restart (rare — only after upgrading the runner binary or rotating tokens)
sudo systemctl restart actions.runner.AiQlickProject-<repo>.<name>

# Live logs during a build
sudo journalctl -u actions.runner.AiQlickProject-<repo>.<name> -f

# Cross-repo runner inventory (from a workstation with `gh` auth)
for repo in background-tasks aiqlick-backend aiqlick-frontend documentation; do
  echo "--- $repo ---"
  gh api repos/AiQlickProject/$repo/actions/runners \
    --jq '.runners[] | "\(.name): \(.status) [\(.labels | map(.name) | join(","))]"'
done

Deploy time benchmarks (background-tasks)​

After the cutover and a couple of related cleanups (tighter health-check polling, parallel docker pull + docker image prune -f), end-to-end deploy times look like this:

ConfigurationDEV deployPROD deploy
GitHub-hosted runner (historical)~3–5 min~3–5 min
Self-hosted runner — initial cut-over1m 19s1m 42s
+ health-check poll 3s → 1s1m 12s1m 9s
+ parallel prune+pull, drop prune -af32 s30 s

The biggest single win was dropping docker image prune -af from the critical path. The -a flag was a hangover from the old 8 GB GitHub-hosted disk; on our 30 GB volume it both burns 5–10 s every deploy and was a real rollback hazard (it could remove the :previous tag we set seconds earlier for failover).

The same health-check tightening was also applied to aiqlick-backend, where the deploy is dominated by the Prisma-aware deploy step (still ~30 s in the SSH session because it does container restart + db-push-when-changed + readiness loop). Backend prod deploy lands at ~57 s end-to-end on the new runner.

Cost​

ItemPer month
t3.medium on-demand (730 hr × $0.0432/hr)~$31.50
30 GB gp3 EBS~$2.80
Data transfer (all in-region: ECR, prod EC2s)$0
Total~$34/mo

That's shared across all four repos — no per-repo cost. Compared to the GitHub Actions per-org spending cap that just bit us, this is unbounded capacity at a known, fixed price.

Tradeoffs vs. GitHub-hosted runners​

Self-hosted (this setup)GitHub-hosted
BillingAWS only (~$34/mo, all 4 repos)GitHub Actions minutes, per-org cap
Spending capsNonePer-org monthly cap
Cold-startNone — runners persist between jobs (Docker layer cache, ECR creds, SSH known_hosts all stay warm)Fresh VM every job
NetworkSame VPC as ECR + target EC2s — fast pushesOutbound from GitHub-hosted region
SecurityOutbound-only SG, OIDC for AWS auth, IAM scoped via github-actions-roleSame OIDC pattern
MaintenanceRunner binary self-updates; we patch the OS via SSM dnf upgradeFully managed
ConcurrencyBounded by t3.medium RAM (≈2 heavy builds)Effectively unlimited

The maintenance overhead is small (the runners self-update and there's no inbound surface area), and the speedup from a warm Docker cache alone justifies the switch even ignoring the billing block.