rootless: enable resource limitation (requires cgroup v2, systemd)

[AkihiroSuda]

Proposed Changes

rootless: enable resource limitation (requires cgroup v2, systemd)

Types of Changes

New Feature

Verification

• Install Ubuntu 20.10

• Add systemd.unified_cgroup_hierarchy=1 to kernel cmdline (/etc/default/grub) to enable cgroup v2

• Enable systemd delegation (https://rootlesscontaine.rs/getting-started/common/cgroup2/)

$ sudo mkdir -p /etc/systemd/system/[email protected]
$ cat <<EOF | sudo tee /etc/systemd/system/[email protected]/delegate.conf
[Service]
Delegate=cpu cpuset io memory pids
EOF
$ sudo systemctl daemon-reload

• Copy k3s-rootless.service to ~/.config/systemd/user/k3s-rootless.service

• Run systemctl --user daemon-reload

• Run systemctl --user enable --now k3s-rootless

• Run KUBECONFIG=~/.kube/k3s.yaml kubectl get pods -A. Make sure the pods are running.

• Run a test pod like this:

apiVersion: v1
kind: Pod
metadata:
  name: test-limits
spec:
  containers:
  - name: test-limits
    image: alpine
    command: ["top"]
    resources:
      limits:
        cpu: 420m
        memory: 42Mi

$ cat /proc/41309/cgroup 
0::/user.slice/user-1001.slice/[email protected]/k3s-rootless.service/kubepods/podd0eb6921-c81a-4214-b36c-d3b9bb212fac/671b393f08437d7e89e234d9d395aecef155104c7c241f5701844360ffebacac

$ cat /sys/fs/cgroup/user.slice/user-1001.slice/[email protected]/k3s-rootless.service/kubepods/podd0eb6921-c81a-4214-b36c-d3b9bb212fac/671b393f08437d7e89e234d9d395aecef155104c7c241f5701844360ffebacac/cpu.max 
42000 100000

$ cat /sys/fs/cgroup/user.slice/user-1001.slice/[email protected]/k3s-rootless.service/kubepods/podd0eb6921-c81a-4214-b36c-d3b9bb212fac/671b393f08437d7e89e234d9d395aecef155104c7c241f5701844360ffebacac/memory.max 
44040192

Linked Issues

Issue #3004

Further Comments

Now rootless mode can be used with cgroup v2 resource limitations.

A pod is executed in a cgroup like /user.slice/user-1001.slice/[email protected]/k3s-rootless.service/kubepods/podd0eb6921-c81a-4214-b36c-d3b9bb212fac/63b5a253a1fd4627da16bfce9bec58d72144cf30fe833e0ca9a6d60ebf837475.

This is accomplished by running kubelet in a cgroup namespace, and enabling cgroupfs driver for the cgroup hierarchy delegated by systemd.

To enable cgroup v2 resource limitation, k3s server --rootless needs to be launched as systemctl --user service.
Please see the comment lines in k3s-rootless.service for the usage.

Running k3s server --rootless via a terminal is not supported and results in disabling cgroup.

To run k3s server --rootless on a terminal forcibly, systemd-run --user -p Delegate --tty needs to be prepended to create a systemd scope.

[AkihiroSuda]

@dweomer @ibuildthecloud PTAL

[AkihiroSuda]

Thanks for merging. I'd like to add CI for this after #2887 (or its equivalent) gets merged.

[eduncan911]

Is there public documentation for this somewhere, that steps people through the process?

[brandond]

Rootless is still kinda experimental and not very well documented. @AkihiroSuda would you mind taking a shot at the docs? I think this bit in particular could use a writeup:

To enable cgroup v2 resource limitation, k3s server --rootless needs to be launched as systemctl --user service.
Please see the comment lines in k3s-rootless.service for the usage.

Running k3s server --rootless via a terminal is not supported and results in disabling cgroup.

To run k3s server --rootless on a terminal forcibly, systemd-run --user -p Delegate --tty needs to be prepended to create a systemd scope.

[AkihiroSuda]

I added some basic docs to https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental

Let me know if I have to add more