support cgroup v2 (unified hierarchy)

[sols1]

cgroup v2 (unified hierarchy) is now official in 4.5:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34a9304a96d6351c2d35dcdc9293258378fc0bd8

cgroup v2 should have more sensible behavior:

https://www.youtube.com/watch?v=PzpG40WiEfM

moby/moby#16238

[cyphar]

cgroupv2 still doesn't support many of the cgroup controllers we need for runc. The most important one is the device "cgroup", which is a hard requirement for security. As far as I can see, CPU still hasn't been implemented either. Also, many of the other cgroups provide us with protections against other resource exhaustion attacks.

[sols1]

It is possible to do cgroup v2 for some controllers and cgroup v1 for others, which are still not available for cgroup v2.

Memory is the most difficult resource to manage and that's what is fixed in cgroup v2.

The device cgroup seems to be fairly straightforward to convert to cgroup v2: add device permissions to existing single hierarchy.

[cyphar]

The other issue is that we need to be running on a distribution which supports cgroupv2 as the default setup with systemd (which is essentially none of them). We can't really use cgroupv2 otherwise because it would require either:

• Moving all of the processes in the system to the v2 equivalent. But because of the internal node (and threadgroup) constraints this won't be pretty and we'd be changing distro policy.
• Moving just the subtree to the v2 equivalent. While this is technically allowed, the documentation makes it clear that it's a development tool and shouldn't be used for production purposes.

For me, one of the biggest benefits of cgroupv2 is that cgroup namespaces make more sense on v2. Unfortunately, cgroup namespaces don't implement features that would make them useful at the moment (see #774 and #781). So there's that.

And yes, we can use both v2 and v1 at the same time, but that doesn't make the implementation any nicer (now we'd have to use two managers with two different "cgroup paths").

[rodionos]

For context, Ubuntu 16 LTS is on kernel version 4.4
https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Linux_kernel_4.4

[sols1]

Not sure I understand all the issues related to cgroup namespaces. It would be nice to resolve all conceptual issues before doing this but for practical production use of containers resource management is a big issue and memory is the most difficult resource to manage because of its "non-renewable" nature so to speak.

For example, Paralles/Virtuozzo used containers in production for 10+ years and they ended up back porting memory cgroup v2 to the old kernel that they used (RHEL6, if I'm not mistaken).

Also, as far as I understand Google used containers in production for a long time and they had some kernel patches to deal with memory accounting and management.

[cyphar]

@sols1

Not sure I understand all the issues related to cgroup namespaces. It would be nice to resolve all conceptual issues before doing this but ...

cgroup namespaces was a benefit of cgroupv2 😉. The general issue with cgroupv2 is that there just aren't enough controllers enabled for us to be able to use it properly (at a minimum, we'd need the freezer and device cgroups), and using both cgroupv2 and cgroupv1 together will make the implementation more complicated than it needs to be. On the plus side, we don't need the net_* controllers in cgroupv2 (they won't ever be added to cgroupv2) because you can now specify iptable rules by cgroup path (which AFAIK is namespaced by cgroups).

I'd be happy to work on kernel patches to add support for the controllers, but I'd recommend pushing upstream to get more controllers enabled for cgroupv2 -- they just aren't feature complete for us right now and I don't feel good about adding hacks to our cgroup management implementation to deal with cgroupv2's shortcomings.

but for practical production use of containers resource management is a big issue and memory is the most difficult resource to manage because of its "non-renewable" nature so to speak.

I understand, but there's also the problem that I'm not sure how we could test our use of cgroupv2 because systemd uses the cgroupv1 hierarchy on almost every distribution (I tried to switch to cgroupv2 on my laptop while my system was running -- it did not end well).

[justincormack]

@cyphar we are in the merge window for 4.9 which will be next LTS, so it is getting quite late to get support in for the next few years for most distros - any chance of looking at the kernel patches?

I am happy to help testing, it should be fairly easy on Alpine Linux as it does not use systemd so can change more easily.

[sols1]

RancherOS (https://github.com/rancher/os) is another option. It does not use systemd and even systemd emulation was removed AFAIK.

[cyphar]

I haven't really had a chance to work on kernel patches recently. However, I did try a few months ago to implement freezer so it worked with cgroupv2 -- as far as I can tell it's not really that trivial to do. Namely there are some edge cases that made the handling non-clear. And I looked at the devices code but its quite a bit more complicated than the freezer code.

I might take look sometime next month, but I can't really guarantee anything (I've been swamped quite recently).

[hustcat]

Buffer io throttle is another biggest benefits of cgroupv2.

[rhatdan]

Rawhide just moved to CgroupV2. Causing docker/runc to blow up.

https://bugzilla.redhat.com/show_bug.cgi?id=1411286

docker run -ti fedora bash
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay/e1432a26e33bebbc27619c9802d9218f3da8938b7f1696ca9be0890a2e75ac65/merged\\\\\\\" at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"no subsystem for mount\\\\\\\"\\\"\"\n".

uname -r

4.10.0-0.rc2.git4.1.fc26.x86_64

[webczat]

isn't cpu controller merged for 4.14 already?