Regression: Docker 4.19.0 permission denied for /var/run/docker.sock when using Docker in Docker

[ericvn]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

Using Docker in Docker should work the same as in prior versions, like 4.18.0

Actual behavior

Docker 4.19.0:

docker run -v /var/run/docker.sock.raw:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

Docker 4.18.0:

docker run -v /var/run/docker.sock.raw:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
6aecfb8cf30a        docker              "docker-entrypoint.s…"   3 seconds ago       Up 2 seconds                            jovial_shirley

Information

• macOS Version: 13.3.1
• Intel chip or Apple chip: Intel
• Docker Desktop Version: 4.19.0

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Steps to reproduce the behavior

1. docker run -v /var/run/docker.sock.raw:/var/run/docker.sock -u501: -it --rm docker sh
2. Inside container: docker ps

[ericvn]

Removing the -u501: does allow things to work as before.

[christophermclellan]

Hi @ericvn , can I close this issue ? Or is there anything we can help with here ?

[djs55]

I tried with build 4.18.0 (104112) but I couldn't get it to work then either:

% docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
/ $ ls -l /var/run/docker.sock
srwxr-xr-x    1 root     root             0 May  2 09:16 /var/run/docker.sock

Possibly the change is in the docker image itself? Could you check which versions you're running?

[ericvn]

I will try and install an older version over 4.19 (is that supported) to verify that 4.18 and/or 4.17 works. Unfortunately, I don't have a laptop to fresh install to verify. Note that this also happened back in the 2.3 days: #4755

[jeffrozica]

I just created an issue for this too. Here is my notes of my issue. Same problem on MAC and Windows.

Docker desktop version 4.19.0 broke my -v /var/run/docker.sock.raw /var/run/docker.sock. About 2 years ago, this solved my docker socket permission denied error (See belwo).... It was working fine up through version 4.18, but after the v4.19 upgrade, it broke again. Getting this error again. Had to revert back to version 4.18. Same issue on Windows and Mac versions.

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

Again, no issues until I went to docker desktop version 4.19 and its the same issue on MAC and WINDOWS. The docker.sock.raw was the only viable option for me to fix this.

Mac Version: Ventura 13.3
Windows Version: Windows 11

[ericvn]

I tried with build 4.18.0 (104112) but I couldn't get it to work then either:

% docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
/ $ ls -l /var/run/docker.sock
srwxr-xr-x    1 root     root             0 May  2 09:16 /var/run/docker.sock

Possibly the change is in the docker image itself? Could you check which versions you're running?

Actually to get it to work on 4.18 or earlier you need to specify the -v /var/run/docker.sock.raw:/var/run/docker.sock If you look in my earlier PR, the .raw was needed after 2.6.? maybe. It had worked until 4.19.

[ncouse]

Prior to 4.19.0, a non-root user could be used, but the user had to be in the root group. Also, you needed to mount /var/run/docker.sock.raw, as others have mentioned.

With 4.19.0, only the root user seems to work. I have had to downgrade back to 4.18.0 to get this working again.

There are lots of reasons to run as a non-root user. In fact that should be the norm.

In one use case I have, I use a container for building, and run as a host user, so that files can be written to bind mounted volumes. Part of the build process involves creating images, so it needs Docker access also.

[thaJeztah]

First of all, this scenario is not "Docker in Docker"; the docker container in this case is not running a docker daemon, "Docker Outside of Docker"; it's running the docker CLI inside the container, but connecting to the Docker Daemon on the host (host being the VM in Docker Desktop's case). If that's your intended use-case, I would also recommend using the docker:cli (docker:23-cli) image, instead of the docker:dind image; the docker:cli image does not contain the Docker Engine, containerd, and runc binaries (none of which would be needed for this use-case).

In regular setups (plain, non-Docker-Desktop "Linux" installation), access to the docker socket should be protected, as having access to the docker engine API (through /var/run/docker.sock) is equivalent to having root access on the host; see https://docs.docker.com/go/attack-surface/.

On Docker Desktop, the security model is somewhat different, as the VM acts as an extra security boundary; while the Docker Engine runs as root inside the VM (and as such, having access to the Engine API would allow escalating to gain root access on the host), the VM itself runs as a non-privileged user, so escalating to a root user would only allow for root access within the VM (but as a non-privileged user on the host (macOS / Windows)). In addition, "Hardened Desktop" allows restricting access within the VM in addition to restricting access on the host.

For reasons above, permissions of the docker socket on the host (macOS, Windows) can be more permissive than on a Linux machine, allowing non-root users to connect to the Docker API.

Possibly the change is in the docker image itself? Could you check which versions you're running?

I don't think this would be related to changes in the docker image, but just to be sure, I verified this with older versions of the docker image (docker:20.10.0-dind), and the result was the same.

Now the questions are;

1. What's the expected behavior?
2. What changed between Docker Desktop 4.18 and 4.19?

Looking at permissions on a Linux machine:

ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Dec 10  2020 /var/run/docker.sock

cat /etc/group | grep docker
docker:x:999:sebastiaan

(I added myself to the docker group on this machine)

Or (showing numeric UID/GID), on this machine the docker GID is 999;

ls -ln /var/run/docker.sock
srw-rw---- 1 0 999 0 Dec 10  2020 /var/run/docker.sock

Comparing this to permissions of the socket inside the Docker Desktop VM, I see

ls -l /var/run/docker.sock
srwxr-xr-x 1 root root 0 May  5 00:39 /var/run/docker.sock

cat /etc/group | grep docker
# no result

Differences worth noting here;

• Inside the Docker Desktop VM, the socket's group is root, and not docker.
• There is no docker group present.
• Inside the Docker Desktop VM, the permissions are difference (srwxr-xr-x vs srw-rw----).
• ☝️ Note that:
  • execute (x) permissions should not be needed for the socket
  • the socket currently is "world readable" and "world executable"; this should not be the case. While it's not a real issue (socket requires rw permissions to be usable on Linux), it should probably not be world-accessible.
  • (Also worth noting that BSD-based systems, such as macOS may treat socket permissions differently)

On a Linux machine, that also means I won't be able to access the socket as a non-root user, as it's protected by its Linux permissions;

Using the UID/GID I have on the host (1000:1000);

id
uid=1000(sebastiaan) gid=1000(sebastiaan) groups=1000(sebastiaan),999(docker)

docker run -v /var/run/docker.sock:/var/run/docker.sock -u1000 -it --rm docker version
Client:
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:12:25 2023
 OS/Arch:           linux/amd64
 Context:           default
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied

But, running with the docker group (999), or adding the group as auxiliary group (--group-add), I can:

docker run -v /var/run/docker.sock:/var/run/docker.sock -u1000:999 -it --rm docker version

Client:
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:12:25 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.4
...

docker run -v /var/run/docker.sock:/var/run/docker.sock -u1000:1000 --group-add 999 -it --rm docker version
Client:
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:12:25 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.4
...

So, what changed?

Not entirely sure.

• Docker Desktop 4.19 ships with Docker Engine 23.0.0, and more current versions of containerd and runc. I know there's been some bugfixes around handling of auxiliary groups, which could be related. That said, I verified the behavior on a fresh install of Docker Engine 20.10 on Linux, and that gave the same result.
• Did permissions change inside the Docker Desktop VM? Looking at the output of Regression: Docker 4.19.0 permission denied for /var/run/docker.sock when using Docker in Docker #6823 (comment), permissions look the same on older versions of Docker Desktop.
• Is it possible that older versions of Docker Desktop connected to the socket in a different way? (Don't think it's possible, but could it have connected to another socket / proxied socket? @djs55 ?)
• Can VirtioFS play a role in this in any way? (don't expect that to be relevant)

Things to look into

I think we should look into the permissions of the socket inside the VM, and the missing docker group;

• what do we want the permissions of the socket within the VM to be? "permissive" (like on the macOS/Windows host) or "default" (no access, unless root, or part of the docker group)?
• make sure the permissions of the socket(s) within the VM are as expected (rw-rw----)
• make sure there's a docker group within the VM, and that the socket's group-ownership is docker

With that, I think access to the socket within the VM would still be protected "by default", but it would be possible to access the socket by passing the docker GID as auxiliary group;

# run with the same UID and GID as the user on the host, but add the docker group's GID as auxiliary group
docker run -v /var/run/docker.sock:/var/run/docker.sock -u"$(id -u):$(id -g)" --group-add 999 -it --rm docker version

[ncouse]

The socket permissions in the VM seem to have changed between 4.18.0 and 4.19.0

4.18.0

$ nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock <<< "ls -lL /var/run/docker.sock"
/ # ^[[47;5Rls -lL /var/run/docker.sock
srw-rw----    1 root     root             0 May  7 15:03 /var/run/docker.sock

4.19.0

$  nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock <<< "ls -lL /var/run/docker.sock"
/ # ^[[47;5Rls -lL /var/run/docker.sock
srwxr-xr-x    1 root     root             0 May  7 21:17 /var/run/docker.sock

It seems to no longer have write permissions for the root group.

I was previously able to access when the root group id was used (which I usually did with --group-add 0). I didn't particularly like using the root group, but this seems to be the only way to work. On Linux, I would use the docker group directly, but the Docker Desktop setup is quite different.

Now the the 4.19.0 changes, it looks like being the root user is the only way to access the docker socket.

[ncouse]

This is still broken in 4.20.0.

---

As a reminder, I would expect this command to work, as it did prior to 4.19.0:

docker run -ti --rm -v /var/run/docker.sock.raw:/var/run/docker.sock -u $(id -u):$(id -g) --group-add 0 library/docker:cli docker info

This runs the container using the host user's id/group, but adds it to the root group also. From 4.19.0, you need to be the root user, which breaks a lot of use cases.

[dgageot]

Hi everyone. FYI, a fix is being tested.