Duplicate mount point of builds when using Docker executor
Summary
Error happens when run any job with setting persistent storage of /builds in Docker executor after update runner to 12.10.1.
Error message:
ERROR: Job failed (system failure): Error response from daemon: Duplicate mount point: /buildsSteps to reproduce
The gitlab-runner configuration:
/etc/gitlab-runner/config.toml
[[runners]]
name = "DockerRunner"
output_limit = 65536
url = "---"
token = "---"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "ubuntu:18.04"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = [
"/var/run/docker.sock:/var/run/docker.sock",
"/mnt/working/gitlab-runner/docker/:/builds/",
"/mnt/working/gitlab-runner/docker/cache:/cache/"
]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.custom]
run_exec = ""When the following volume mapping exist, the runner will fail when run the job.
volumes = [
"/mnt/working/gitlab-runner/docker/:/builds/",
"/mnt/working/gitlab-runner/docker/cache:/cache/"
]After removing the volume mapping of /builds, it works again.
Expected behavior
The config.toml works fine in gitlab-runner 12.9.x.
Relevant logs and/or screenshots
job log
[0KRunning with gitlab-runner 12.10.1 (ce065b93)
[0;m[0K on Veridandi-Docker oRHVaM97
[0;msection_start:1587610561:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m
[0;m[0KUsing Docker executor with image neszt/cppcheck-docker ...
[0;m[0KPulling docker image neszt/cppcheck-docker ...
[0;m[0KUsing docker image sha256:de5586948de68e92fc11f84491b54011b5ffa9b21617fa891679a9f102ebf08b for neszt/cppcheck-docker ...
[0;msection_end:1587610564:prepare_executor
[0Ksection_start:1587610564:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;msection_end:1587610564:prepare_script
[0Ksection_start:1587610564:upload_artifacts_on_failure
[0K[0K[36;1mUploading artifacts for failed job[0;m
[0;msection_end:1587610564:upload_artifacts_on_failure
[0K[31;1mERROR: Job failed (system failure): Error response from daemon: Duplicate mount point: /builds (docker.go:788:0s)
[0;mEnvironment description
- Self-host GitLab web site
- gitalb-runner 12.10.1 on Ubuntu 18
Docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 84
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-72-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 1.475TiB
Name: Veridandi
ID: PFZ5:JOO7:V2YN:Y55R:BCB2:BWO2:5MZA:T2RF:A52L:QOLB:OHCT:GBZO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
hades.nchc.org.tw:5100
yggdrasil.nchc.org.tw:5100
127.0.0.0/8
Live Restore Enabled: true
WARNING: API is accessible on http://0.0.0.0:2376 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: No swap limit supportUsed GitLab Runner version
Version: 12.10.1
Git revision: ce065b93
Git branch: 12-10-stable
GO version: go1.13.8
Built: 2020-04-22T21:29:52+0000
OS/Arch: linux/amd64Root cause analysis
This was working as expected in v12.9.0 by "accident" because of how --volumes-from work. Before !1989 (merged) was merged we used to get the volume detention from the previous container and then add the user bindings. In v12.9.0 we still used to create the /builds volume but when using --volumes-from and specify your own binding docker silently overrides any duplicate bindings that it finds, meaning it was silently overriding the $VOLUME:/builds with /src:/builds binding.
We can see this by running simple docker commands. This is what GitLab Runner was doing in `v12.9.0`
# Create volumes (GitLab Runner used to do it a complicated way with containers)
$ docker volume create cache
$ docker volume create builds
# Mount those volumes to the container
$ docker run -v cache:/cache -v builds:/builds alpine:3.11 echo "hello"
hello
# Get container ID
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
14f9b1cf3438 alpine:3.11 "echo hello" 4 seconds ago Exited (0) 3 seconds ago sleepy_northcutt
# As we can see the `cache` volume is mount to `/cache` and `buidls` is mounted to `/builds`
$ docker inspect 14f9b1cf3438 | jq '.[].Mounts'
[
{
"Type": "volume",
"Name": "cache",
"Source": "/var/lib/docker/volumes/cache/_data",
"Destination": "/cache",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "volume",
"Name": "builds",
"Source": "/var/lib/docker/volumes/builds/_data",
"Destination": "/builds",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
]
# Create a new container using `--volumes-from` using the ID of the previous container
$ docker run --volumes-from=14f9b1cf3438 alpine:3.11 echo "hello2"
$ docker inspect 14f9b1cf3438 | jq '.[].Mounts'
[
{
"Type": "volume",
"Name": "cache",
"Source": "/var/lib/docker/volumes/cache/_data",
"Destination": "/cache",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "volume",
"Name": "builds",
"Source": "/var/lib/docker/volumes/builds/_data",
"Destination": "/builds",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
]
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fad2222a9463 alpine:3.11 "echo hello2" 5 seconds ago Exited (0) 5 seconds ago quirky_almeida
14f9b1cf3438 alpine:3.11 "echo hello" About a minute ago Exited (0) About a minute ago sleepy_northcutt
# We can see that it took those mounts correctly!
$ docker inspect fad2222a9463 | jq '.[].Mounts'
[
{
"Type": "volume",
"Name": "builds",
"Source": "/var/lib/docker/volumes/builds/_data",
"Destination": "/builds",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "volume",
"Name": "cache",
"Source": "/var/lib/docker/volumes/cache/_data",
"Destination": "/cache",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
]
# Create a new container using the volumes from the previous container however override the `/builds` to something host mounted, it still works! This is was GitLab Runner was doing!
$ docker run --volumes-from=14f9b1cf3438 -v /tmp/builds:/builds alpine:3.11 echo "hello3"
hello3
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a9aa1f217128 alpine:3.11 "echo hello3" 4 seconds ago Exited (0) 3 seconds ago gracious_sutherland
fad2222a9463 alpine:3.11 "echo hello2" 4 minutes ago Exited (0) 4 minutes ago quirky_almeida
14f9b1cf3438 alpine:3.11 "echo hello" 5 minutes ago Exited (0) 5 minutes ago sleepy_northcutt
# We can see the the `/builds` directory was silently overridden to be `/tmp/builds:/builds` instead of using the volume
$ docker inspect a9aa1f217128 | jq '.[].Mounts'
[
{
"Type": "volume",
"Name": "cache",
"Source": "/var/lib/docker/volumes/cache/_data",
"Destination": "/cache",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/tmp/builds",
"Destination": "/builds",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
]
We had to stop using --volumes-from because those containers were being removed and it resulted into errors like #4450 (closed) as we can see in #4450 (comment 316034514)
Edited by Steve Xuereb