Created attachment 1507 [details] adjust Linux out-of-memory killer In some out-of-memory situations, the Linux kernel will look for a process to kill, employing some heuristics to try to guess what will help. It doesn't always get this right and can occasionally end up killing innocent bystanders (though as noted in the referenced bug log it's possible to tweak this to be more accurate). It is useful to instruct the OOM killer never to kill sshd, since almost everyone wants it to keep on running so that they have a chance of dealing with the problem remotely. Originally I implemented this in an init script, by getting sshd's pid and writing to /proc/$pid/oom_adj, but Vaclav Ovsik pointed out in http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=480020 that that ends up immortalising child processes too. The attached patch is based on Vaclav's, though I tidied it up and moved chunks of it into openbsd-compat/port-linux. The use of an environment variable for configuration is a bit odd. I didn't feel good about introducing a port-specific configuration file key, and the values you write into oom_adj have a pretty bizarre syntax (documented in http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/proc.txt;hb=HEAD) which I think is unlikely to be portable to other systems. I'd appreciate any better ideas here.
some feedback on the diff: Isn't /proc/self/oom_adj just an integer? I don't see any bizarre syntax in the referenced document, but the available docs are seriously deficient... If it is just an integer then why not just read and write it using stdio? I don't think that it should be controlled by an environment variable - sshd should just set the "never kill me" flag for the master process unconditionally. Failure to open() /proc/self/oom_adj shouldn't throw an logit() - this will just spam older Linux systems that lack the control or don't have /proc mounted. The port-linux.c code is incorrect: it doesn't handle EINTR errors on read()/write() - it should use atomicio() if it can't use stdio.
Why is the oom_adj value being passed in as an environment variable? I would have expected it to be a flag in /etc/ssh/sshd_config. Is it bad form to have OS specific sshd_config flags?
Created attachment 1712 [details] Revised patch for Linux OOM killer Updated the previous patch based on Damien's feedback in Comment #1. Limited testidng indicates that the patch works. The one oddity is that the message logged by verbose() when restoring the original oom_adj value shows up three times in /var/log/syslog.
Add to list for 5.4
Created attachment 1740 [details] openssh-linux-oom_kill.patch Use the platform_* hooks to avoid sprinkling more #ifdefs into the main code. Move the saved value to port-linux.c. Add LINUX_ to the define since it is Linux-specific.
Created attachment 1741 [details] openssh-linux-oom_kill.patch Create a platform_pre_listen hook and use that for the oom adjust.
Created attachment 1742 [details] openssh-linux-oom_kill.patch Don't try to restore a value that we did not save.
(In reply to comment #3) > The one oddity is that > the message logged by verbose() when restoring the original oom_adj > value shows up three times in /var/log/syslog. I think I can explain that: #1: Despite what's implied by the message oom_adjust_setup() actually logs the saved value not the set value: + verbose("Set %s to %d", + OOM_ADJ_PATH, oom_adj_save); #2: oom_adjust_setup() gets called a second time when sshd re-execs itself to randomize its address space. #3: the real call to oom_adjust_restore()
Comment on attachment 1742 [details] openssh-linux-oom_kill.patch >Index: openbsd-compat/port-linux.c >=================================================================== >RCS file: /var/cvs/openssh/openbsd-compat/port-linux.c,v >retrieving revision 1.6 >diff -u -p -r1.6 port-linux.c >--- openbsd-compat/port-linux.c 24 Oct 2009 04:04:13 -0000 1.6 >+++ openbsd-compat/port-linux.c 7 Dec 2009 06:06:11 -0000 >@@ -27,8 +27,15 @@ > #include <stdarg.h> > #include <string.h> > >-#ifdef WITH_SELINUX >+#if defined(LINUX_OOM_ADJUST) || defined(WITH_SELINUX) > #include "log.h" >+#endif >+ >+#ifdef LINUX_OOM_ADJUST >+#include <stdio.h> >+#endif >+ I wouldn't bother slicing and dicing the header inclusion based on preprocessor symbols. There is little cost to including them unconditionally, or perhaps conditionally on the union of all supported symbols for this file. >+#ifdef LINUX_OOM_ADJUST >+#define OOM_ADJ_PATH "/proc/self/oom_adj" >+#define OOM_ADJ_NOKILL -17 /* magic value to disable OOM killer */ FYI, -17 is documented in Documentation/filesystems/proc.txt in the Linux source. A stable URL for this is http://lxr.linux.no/#linux+v2.6.32/Documentation/filesystems/proc.txt if you want to include it.
(In reply to comment #9) > perhaps conditionally on the union of all supported > symbols for this file. Done. > FYI, -17 is documented in Documentation/filesystems/proc.txt in the > Linux source. A stable URL for this is > http://lxr.linux.no/#linux+v2.6.32/Documentation/filesystems/proc.txt > if you want to include it. I know, I read it :-). Reference added. Thanks all, the patch has been applied and will be in 5.4p1.
The patch as applied has one flaw that I can see. Apparently some virtualisation containers (vserver/OpenVZ) don't allow processes to write to /proc/self/oom_adj, and will return an error code if they try. It would be a shame for sshd to unconditionally log an error on such systems; I think this was probably the main benefit of having it controlled by an environment variable, so that they could turn this feature off. How about just lowering errors from writing to /proc/self/oom_adj to debug1(), rather than logit()?
(In reply to comment #11) > The patch as applied has one flaw that I can see. Apparently some > virtualisation containers (vserver/OpenVZ) don't allow processes to > write to /proc/self/oom_adj, and will return an error code if they try. > It would be a shame for sshd to unconditionally log an error on such > systems; I think this was probably the main benefit of having it > controlled by an environment variable, so that they could turn this > feature off. > > How about just lowering errors from writing to /proc/self/oom_adj to > debug1(), rather than logit()? I've lowered them to verbose(), same as the other calls.
Thanks, that should do the job.
With the release of 5.4p1, this bug is now considered closed.