Solve the 503 Service Unavailable error reported by gitlab runner in docker mode

After gitlab runner is installed and registered, it cannot be pulled at the first startup time gitlab-runner-helper error:

 Running with gitlab-runner 15.3.0 (bbcb5aba)
  on debian-docker CHvjghoP
  feature flags: FF_USE_FASTZIP:true
Preparing the "docker" executor
00:02
Using Docker executor with image ruby:3.0 ...
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-bbcb5aba ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://registry.gitlab.com/v2/gitlab-org/gitlab-runner/gitlab-runner-helper/manifests/x86_64-bbcb5aba": received unexpected HTTP status: 503 Service Unavailable (manager.go:235:1s)
ERROR: Job failed: failed to pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-bbcb5aba" with specified policies [always]: Error response from daemon: Head "https://registry.gitlab.com/v2/gitlab-org/gitlab-runner/gitlab-runner-helper/manifests/x86_64-bbcb5aba": received unexpected HTTP status: 503 Service Unavailable (manager.go:235:1s)

Analysis of the error message found that the runner first wanted to pull a registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-bbcb5aba image, the role of the image should be to speed up the download speed, secure links, set environment variables and add accessories to the software, etc. , because the default helper image is located in gitlab In the official docker repository, and a 503 error occurred when accessing this repository.

solution

The gitlab runner needs a helper, which is a must. And the current problem is: this registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-bbcb5aba has a 503 (probably due to authentication).

So what we have to do is: re-designate a gitlab-runner-helper for gitlab runner that does not require permission authentication. Fortunately, docker officially provides this image.

reset helper

We call the server where the gitlab runner is located, and after logging in, first check the version of our gitlab runner:

 root@debian-docker:/etc/gitlab-runner# gitlab-runner -v
Version:      15.3.0
Git revision: bbcb5aba
Git branch:   15-3-stable
GO version:   go1.17.9
Built:        2022-08-19T22:41:11+0000
OS/Arch:      linux/amd64

Then find the configuration file config.toml :

1. When running Gitlab Runner on a *nix system as root, the path is /etc/gitlab-runner/
2. When running Gitlab Runner on a *nix system as well as a non-root user, the path is at ~/.gitlab-runner/
3. Other systems, in ./

Then add the following information:

 [runners.docker]
    ...
    helper_image="gitlab/gitlab-runner-helper:x86_64-${CI_RUNNER_VERSION}"

For GitLab Runner version less than 13.2, add the following information:

 [runners.docker]
    ...
    helper_image="gitlab/gitlab-runner-helper:x86_64-${CI_RUNNER_REVISION}"

After the configuration is changed, run the job again, check the log again, and find the following errors:

 ERROR: Job failed: failed to pull image "gitlab/gitlab-runner-helper:x86_64-15.3.0" with specified policies [always]: Error response from daemon: manifest for gitlab/gitlab-runner-helper:x86_64-15.3.0 not found: manifest unknown: manifest unknown (manager.go:235:3s)

At this point, it is no longer a 504 error, and the requested image address has also become the one we configured itlab/gitlab-runner-helper:x86_64-15.3.0 , indicating that the configuration just now took effect. But unfortunately there is no x86_64-15.3.0 version on docker hub:

That being the case, then I think it shouldn't be a problem to use the latest version. The latest version is x86_64-v14.10.2

So I modified the configuration file again config.toml and changed the version of helper to x86_64-v14.10.2

 [runners.docker]
    ...
    helper_image="gitlab/gitlab-runner-helper:x86_64-v14.10.2"

Rerun the job, and the pull success screen is here:

Summarize

Searching on google for this error did not find a fool's answer, so the direction to solve the problem still depends on the error message and the principle guess. From the official documentation or to the helper, it is necessary to install the image, and then it is found from the address that reports the error that it cannot pull the image normally. So I finally put the key point of the solution: manually specify the address of the image.