Guide to GitLab Runner

1. Overview

GitLab Runner is a powerful tool that works in conjunction with GitLab CI/CD to run jobs and deliver results back to GitLab. In other words, it’s an essential component of the GitLab continuous integration and deployment pipeline.

As an agent that runs on a separate machine or container, GitLab Runner’s primary purpose is to execute the jobs from the GitLab CI/CD configuration. When we push code to a GitLab repository, the runner picks up the job, runs the specified commands, and reports the results back to GitLab.

Thus, by using GitLab Runner, we can automate the build, test, and deployment processes, saving time and reducing the risk of manual errors.

In this tutorial, we’ll look at how to set up and configure GitLab Runner, and its various features and capabilities.

2. Getting Started

Before we use GitLab Runner, we need to set it up and configure it to work with a GitLab instance. In this section, we’ll walk through the process of installing GitLab Runner, registering it with GitLab, and catering the setup to the current needs.

2.1. Installing GitLab Runner

To get started, we install GitLab Runner on the machine. To be clear, GitLab Runner supports various operating systems, including Linux, macOS, and Windows. The installation process varies slightly depending on the operating system (OS) we’re using.

For example, on a Debian-based Linux distribution, we can install GitLab Runner by adding the official GitLab repository and installing the gitlab-runner package using a native package manager like apt:

$ curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
$ sudo apt-get install gitlab-runner

This short snippet downloads the installation script for the GitLab Runner repository and executes it using bash with superuser privileges. Further, it sets up the repository and adds the necessary configuration to the system.

Additionally, similar installation instructions are available for other operating systems in the GitLab Runner documentation.

2.2. Registering a Runner

Once we install GitLab Runner, the next step is to register it with а GitLab instance. Runner registration establishes a connection between the runner and GitLab, enabling them to communicate and execute jobs.

However, to register a runner, we need to obtain a registration token from a GitLab project or group settings. Specifically, in GitLab, we navigate to the project or group where we want to use the runner, then we go to Settings > CI/CD and expand the Runners section. There, we should find the registration token.

With the registration token at hand, we can use the gitlab-runner register command to register the runner. The command prompts for various details, such as the GitLab instance URL, the registration token, a description for the runner, and the executor type (e.g., shell, Docker, Kubernetes).

Let’s see an example of registering a runner using the shell execution:

$ sudo gitlab-runner register

Then, we enter the required information, and upon completion, the runner is registered and ready to accept jobs from GitLab.

2.3. Configuring a Runner

However, after registering a runner, we may need to configure it further to match the requirements of a given project. The runner’s configuration is stored in the config.toml file, typically located at /etc/gitlab-runner/config.toml on Linux systems.

There, we can specify various settings, such as the concurrent job limit, cache settings, and environment variables.

For example, if we want to limit the number of concurrent jobs a runner can handle, we can add the concurrent option:

concurrent = 4

Additionally, we can also define custom environment variables available to jobs:

[[
runners
]]
  environment = ["MY_VARIABLE=value"]

These are just a few examples of the configuration options available.

Furthermore, with a runner installed, registered, and configured, we’re ready to start defining jobs in the .gitlab-ci.yml file, so we can let GitLab Runner just handle the execution.

3. GitLab CI/CD Configuration

Now that we have our GitLab Runner set up and ready to go, let’s dive into configuring a GitLab CI/CD pipeline. In this section, we’ll explore how we can create a .gitlab-ci.yml file, define jobs and stages, and use variables and secrets to customize the pipeline.

3.1. Creating a .gitlab-ci.yml File

The .gitlab-ci.yml file is the heart of a GitLab CI/CD configuration. Specifically, it lives in the root directory of a repository and contains the instructions for GitLab Runner to execute our jobs.

So, let’s start by creating a new file named .gitlab-ci.yml in the project’s root directory.

3.2. Defining Jobs and Stages

Inside the .gitlab-ci.yml file, we define jobs and stages. A job represents a specific task or command that we want to run, while stages define the order in which jobs are executed.

Let’s look at a simple example of a .gitlab-ci.yml file with two jobs and two stages:

stages:
  - build
  - test

build-job:
  stage: build
  script:
    - echo "Building the project..."
    - npm install
    - npm run build

test-job:
  stage: test
  script:
    - echo "Running tests..."
    - npm run test

In the example above, we have two stages:

• build
• test

The build-job is assigned to the build stage and runs the commands to install dependencies and build the project. Moreover, the test-job belongs to the test stage and runs the project’s tests.

3.3. Using Variables and Secrets

GitLab CI/CD enables us to use variables and secrets to store sensitive information or configure different aspects of the pipeline. Variables can be defined at the project, group, or instance level, and they can be accessed within our .gitlab-ci.yml file in any job scripts.

In detail, to define a variable, we can navigate to a project’s Settings > CI/CD and expand the Variables section. we can then add a new variable by providing a key value.

For example, let’s say we have a variable named API_URL that holds the URL of our API server. Consequently, we can access this variable in the .gitlab-ci.yml file using the syntax $API_URL:

test-job:
  script:
    - echo "Running tests against $API_URL"
    - npm run test --api-url $API_URL

Secrets, on the other hand, store sensitive information like passwords or API tokens. They are stored securely and masked in job logs. However, to use a secret, we can define it in the same way as variables and access it using the $SECRET_VARIABLE syntax.

Moreover, by leveraging variables and secrets, we can make a pipeline more flexible and secure, enabling us to more easily configure different environments or pass sensitive information to jobs.

Additionally, with the .gitlab-ci.yml file set up and our variables and secrets defined, we’re ready to let GitLab Runner execute jobs.

4. Running Jobs with GitLab Runner

At this point, with a GitLab CI/CD pipeline configured, it’s time to see GitLab Runner in action. Here, we explore how GitLab Runner executes jobs, how we can run shell commands within jobs, and how we leverage Doker for isolated and reproducible job environments.

4.1. Understanding the Runner Execution Process

When we push changes to the repository or create a merge request, GitLab triggers the pipeline defined in the .gitlab-ci.yml file. GitLab Runner which is constantly polling GitLab for new jobs, picks up the job and starts executing it based on the defined stages and job configuration.

The runner follows several steps:

1. clones the repository: clones the repository at the specified commit or branch
2. prepares the environment: set up the job environment based on the specified executor (e.g., shell, Docker)
3. executes the job: runs the commands defined in the job’s script section
4. reports the result: sends the job status (success, failure, or canceled) back to GitLab after the job finishes

This is the runner execution process.

4.2. Executing Shell Commands

Moreover, one of the most common ways to define job steps is by using shell commands. GitLab Runner executes these commands in the job’s environment, enabling us to run scripts, build tools, and test suites.

Let’s look at an example of a job that executes shell commands:

test-job:
  script:
    - echo "Running tests..."
    - npm install
    - npm run test

In the code snippet above, the test-job runs three shell commands:

• printing a message
• installing dependencies using npm install
• running tests with npm run test

GitLab Runner executes these commands sequentially, and if any command fails (returns a non-zero exit code), the job is marked as failed.

4.3. Using Docker with GitLab Runner

Additionally, Docker is a popular choice for running jobs in isolated and reproducible environments. GitLab Runner has built-in support for Docker, enabling us to define job steps that run inside Docker containers.

To use Docker with GitLab Runner, we need to specify the image keyword in the job definition:

test-job:
  image: node:14
  script:
    - npm install
    - npm run test

In the code snippet above, the test-job runs inside a Docker container based on the node:14 image. GitLab Runner automatically pulls the specified image and starts a new container for each job execution.

5. Conclusion

In this article, we’ve explored the power and flexibility of GitLab Runner and how it can revolutionize a CI/CD pipeline from installation and configuration to running jobs with GitLab Runner.

To conclude, we saw how to install GitLab Runner, integrate it into an existing CI/CD setup, and run jobs.