DEVOPS
Choosing GitHub Runners vs Self-Hosted Runners
November 8, 2022
In order to ensure efficiency and quality, modern software development relies heavily on automation tools, such as code repositories and continuous integration/continuous deployment (CI/CD) pipelines.
Of all the different solutions available today, one of the most popular for code repositories is GitHub, which comes with a CI/CD solution known as GitHub Actions. In this blog we explore GitHub Actions, runners, and the merits and pitfalls of self-hosting GitHub Actions runners.
GitHub Actions is a CI/CD platform that lets you run pipelines to build, test, lint, and deploy your code, among other commands. Runners are the apps that run a pipeline (also called a job). In other words, when you have a pipeline building your code, it is being built by a GitHub Actions runner.
When a pipeline is triggered by an event, such as a push to a branch, a runner will be assigned the job and execute it. A GitHub Actions setup can have multiple runners so that jobs can be executed in parallel.
Having self-hosted runners means that the runner app is hosted in your environment of choice—an on-premises server or hosted via your selected cloud provider. For example, you could be running your runner on AWS EC2 machines, Azure Virtual Machines, or GCP compute engines, directly in your company’s AWS/GCP account.
On the other hand, if you choose to use a GitHub-hosted runner, your workflow will run on the virtual machines offered by GitHub.
In the end, a tradeoff needs to be made between the pros and cons of each method. We’ll discuss the benefits of both in the following sections.
Having self-hosted runners brings many benefits that generally revolve around customization and control for users. For starters, when using a GitHub-hosted solution, you will be limited to using GitHub’s VM for running CI jobs, but there are other advantages to consider as well.
A big issue with running your pipeline on a GitHub-hosted machine is that you are locked into the OS and hardware selection that GitHub provides for its virtual machines. At the moment of writing this article, the available hardware can still be limiting for many users as it:
For the OS selection, Windows Server (2022 and 2019), Linux (Ubuntu 22.04 and 20.04), and macOS (12 and 11 ) are available. Linux only comes with the Ubuntu distribution, which can be limiting for those who want to use things like Pacman for packages or other tools not shipped with Ubuntu.
Finally, if you want to run a pipeline for a legacy application that can’t run on these latest OS, you might reconsider and go for a self-hosted type. For example, think of a legacy Windows application using an x86 32-bit CPU library that can hardly compile on these latest Windows servers, at least natively.
The complete list of OS and hardware for GitHub-hosted runners is available here.
While writing this post, Github announced larger action runner machines, up to 64-cores for Windows and Linux and up to 256 GB of RAM. While this still does not appear to address the OS issues, it certainly addresses the hardware limitation that causes slow build time and is a major advantage for GitHub-hosted runner users.
Note that the larger action runner machines feature is still in beta and not yet available to the general public.
As seen in the previous section, CPU cores for GitHub-hosted runners are limited to two. For multithreaded compilation and testing, this can become a bottleneck. Many compilation tools use the advantage of multiple CPUs to speed things up.
Having a shorter pipeline completion time possibly means less time wasted waiting by your team members.
Another benefit of having self-hosted runners is that you have better observability. By having Actions running on your self-hosted machine, you have better visibility into and control over logs and debug tooling. For example, you can log in via SSH to the machine and troubleshoot any issues (hardware, networking, etc.).
When using GitHub-hosted runners, you will have some visibility via logs in the Actions panel. This can help you understand why a pipeline has failed, is stuck, or is having any sort of trouble, but it is also more limiting.
GitHub-hosted runners are virtual machines hosted on Microsoft’s Azure cloud service. This means your Actions are dependent on the health of Azure’s servers, so when they have downtime, you might be unable to run your pipeline.
Depending on your self-hosting strategy and redundancy, this can mean less or more downtime for your Actions runners.
Depending on your setup, your self-hosted runner servers could be only accessible via a VPN or even only local area for in-office usage. This can provide more security, as the machine is not—or at least less—exposed to hackers and malicious intruders and requests. This kind of setup can be seen as a way to implement the least privilege principle when it comes to accessing source code.
Having self-hosted runners brings some disadvantages as well. These primarily involve more manual work and self-management, which can result in higher costs.
When using a GitHub-hosted runner, all you need to do is point out what type of instance (OS) your pipeline needs to run on, and GitHub performs its magic behind the scenes.
But when you are using a self-hosted runner, you will need to spend time provisioning and setting up the servers (cloud-based or on-premises). Then you need to install the runner application and configure it. This includes desired behavior for autoscaling, monitoring, possibly establishing a proxy server, etc.
These things provide a high level of control and customization but also take time and expertise.
As with any self-managed server, your team will have to maintain and manage self-hosted runners. This includes but is not limited to:
Running with your own server can be more costly, although this very much depends on your selected hardware and software.
Also, since self-hosted runners require more management and maintenance, the total cost can be higher since you need to pay engineers and IT and DevOps specialists to do these jobs.
Choosing between self-hosting or using GitHub’s hosting service for your GitHub Actions runners is a matter of tradeoffs and depends on your specific use case. Below are some tips and recommendations that can help you decide which solution best fits your needs.
If your code base takes a long time to build and/or test and involves processes that are very multithreaded (e.g., you build your code with 12 CPU cores with make -j 12), or if you need a specific Linux distribution or even an old Windows version, you would be better off with a self-hosted solution. This is because you will otherwise be blocked by GitHub’s limited hardware and OS selection.
If your pipeline and code include sensitive data or workloads, having a self-hosted runner will let you have more control over the network and security. That can be a plus for your team or it can be a risk if you don’t know what you are doing.
If you are working in a small team of developers, with possibly no dedicated DevOps engineer, you should consider a GitHub-hosted solution, as it simplifies a lot of the work, giving your team more time to work on software development.
If you would prefer using a self-hosted cloud solution (e.g., installing the GitHub runner on an EC2 instance), you should consider using spot instances to lower your cost since your runner will not be busy running jobs most of the time.
Another way to simplify self-hosted cloud deployments is to use a managed container service such as ECS/EKS (optionally with Fargate, a serverless compute engine that runs containerized workloads on ECS/EKS). Using these containerized services can minimize maintenance work on the runners and may reduce costs.
Choosing between GitHub-hosted runners and self-hosted runners is mostly a matter of tradeoff between customization and simplicity.
On the one hand, a self-hosted runner lets you (with minimal limitation) choose the OS, hardware, networking solution, redundancy setup, and observability solution. These can lead to faster pipeline time, better observability, and less downtime, amongst other benefits.
On the other hand, a GitHub-hosted pipeline will save you a lot of configuration and maintenance work by your team.
The decision must be made relative to the needs and requirements of a company and the team using the service, but general guidelines suggest that a self-hosted solution is better suited for teams with large or sensitive workloads and teams with specific/non-standard OS and hardware requirements.
It also suggests that a small team with no dedicated DevOps/IT engineer should probably go for a GitHub-hosted solution.
