Graceful node shutdown does not work #13

New Issue

lauri · 2023-08-15T19:01:30Z

lauri commented

2023-08-15 19:01:30 +00:00

Not sure what is missing from https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown

Basically what has been tried:

Edited kubelet configmap in kube-system namespace but this config change doesn't seem to be propagated to existing nodes. I guess it only affects newly joinet Kubernetes nodes
Added Ansible playbook to reconfigure existing nodes f9393fd0da

Tried systemctl daemon-reload and node reboot but still node immediately shuts down without marking it not ready and allowing time to drain the pods

The file /etc/systemd/logind.conf.d/99-kubelet.conf on nodes shows:

[Login]
InhibitDelayMaxSec=300

But systemd-inhibit --list doesn't show anything kubelet related

Nodes are Ubuntu 22.04 at the moment, but this looks like more reasons to move to Flatcar

Not sure what is missing from https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown Basically what has been tried: * Edited kubelet configmap in kube-system namespace but this config change doesn't seem to be propagated to existing nodes. I guess it only affects newly joinet Kubernetes nodes * Added Ansible playbook to reconfigure existing nodes https://git.k-space.ee/k-space/kube/commit/f9393fd0dac12a5e27eaf195d27cabcf376d9701 Tried `systemctl daemon-reload` and node reboot but still node immediately shuts down without marking it not ready and allowing time to drain the pods The file `/etc/systemd/logind.conf.d/99-kubelet.conf` on nodes shows: ```# Kubelet logind override [Login] InhibitDelayMaxSec=300 ``` But `systemd-inhibit --list` doesn't show anything kubelet related Nodes are Ubuntu 22.04 at the moment, but this looks like more reasons to move to Flatcar

arti commented

2023-08-15 20:11:05 +00:00

Ubuntu by default has unattended-upgrades installed that contains /lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf file with InhibitDelayMaxSec config option.

Kubelet writes its DelayMaxSec into file called 99-kubelet.conf

systemd-logind sorts all the files in its config directories and will pick the config value from last file.

Files in the *.conf.d/ configuration subdirectories are sorted by their filename in lexicographic order, regardless of in which of the subdirectories they reside. When multiple files specify the same option, for options which accept just a single value, the entry in the file sorted last takes precedence, and for options which accept a list of values, entries are collected as they occur in the sorted files.

This means that unattended-upgrades-logind-maxdelay.conf will overwrite 99-kubelet.conf InhibitDelayMaxSec.

Fix is to rename 99-kubelet.conf to zz-kubelet.conf so that is loaded last https://github.com/kubernetes/kubernetes/issues/107043#issuecomment-1023756350

Ubuntu by default has `unattended-upgrades` installed that contains `/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf` file with `InhibitDelayMaxSec` config option. [Kubelet writes its DelayMaxSec](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodeshutdown/systemd/inhibit_linux.go#L175) into file called `99-kubelet.conf` systemd-logind sorts all the files in its config directories and will pick the config value from last file. > Files in the *.conf.d/ configuration subdirectories are sorted by their filename in lexicographic order, regardless of in which of the subdirectories they reside. When multiple files specify the same option, for options which accept just a single value, the entry in the file sorted last takes precedence, and for options which accept a list of values, entries are collected as they occur in the sorted files. This means that `unattended-upgrades-logind-maxdelay.conf` will overwrite `99-kubelet.conf` InhibitDelayMaxSec. Fix is to rename 99-kubelet.conf to zz-kubelet.conf so that is loaded last https://github.com/kubernetes/kubernetes/issues/107043#issuecomment-1023756350

Sign in to join this conversation.