While testing the upcoming release of Ubuntu (15.10 Wily Warewolf), I ran over a bug that renders the kernel crash dump mechanism unusable by default :
LP: #1496317 : kexec fails with OOM killer with the current crashkernel=128 value
The root cause of this bug is that the initrd.img file that is used by kexec to reboot into a new kernel when the original one panics is getting bigger with kernel 4.2 on Ubuntu. Hence, it is using too much of the reserved crashkernel memory (default: 128Mb). This triggers the “Out Of Memory (OOM)” killer and the kernel dump capture cannot complete.
One workaround for this issue is to increase the amount of reserved memory to a higher value. 150Mb seems to be sufficient but you may need to increase it to a higher value. While one solution to this problem could be to increase the default crashkernel= value, it is only pushing the issue forward until we hit this limit once again.
Reduce the size of initrd.img
update-initramfs has an option in its configuration file ( /etc/initramfs-tools/initramfs.conf) that let us modify the modules that are included in the initrd.img file. Our current default is to add most of the modules :
# MODULES: [ most | netboot | dep | list ] # # most - Add most filesystem and all harddrive drivers. # # dep - Try and guess which modules to load. # # netboot - Add the base modules, network modules, but skip block devices. # # list - Only include modules from the 'additional modules' list # MODULES=most
By changing this configuration to MODULES=dep, we can sensibly reduce the size of the initrd.img :
MODULES=most : initrd.img-4.2.0-16-generic = 30Mb
MODULES=dep :initrd.img-4.2.0-16-generic = 12Mb
Identifying this led to a discussion with the Ubuntu Kernel team about using a custom crafted initrd.img for kdump. This would keep the file to a sensible size and avoid triggering the OOM killer.
Implementation
The current implementation of kdump-tools already provides a mechanism to specify which vmlinuz and initrd.img files to use when settting up kexec (from /etc/default/kdump-tools) :
# --------------------------------------------------------------- # Kdump Kernel: # KDUMP_KERNEL - A full pathname to a kdump kernel. # KDUMP_INITRD - A full pathname to the kdump initrd (if used). # If these are not set, kdump-config will try to use the current kernel # and initrd if it is relocatable. Otherwise, you will need to specify # these manually. #KDUMP_KERNEL= #KDUMP_INITRD=
If we use those variables, defined to point to a generic value that can be adapted according to the running kernel version, we have a way to specify a smaller initrd.img for kdump.
Building a smaller initrd.img
Kernel package hooks already exists in /etc/kernel/postinst.d and /etc/kernel/postrm.d to create the initrd.img. Using those as templates, we created new hooks that will create smaller images in /var/lib/kdump and clean them up if the kernel version they pertain to is removed.
In order to create that smaller initrd.img, the content of the /etc/initramfs-tools directory needs to be replicated in /var/lib/kdump. This is done each time that the hook is executed to assure that the content matches the original source. Otherwise, their content may diverge if the content of the original directory gets modified.
Each time a new kernel package is installed, the hook will create a kdump specific initrd.img using MODULES=dep. and store it in /var/lib/kdump. When the kernel package is removed, the corresponding file is removed.
Using the smaller initrd.img
As we outlined previously, the /etc/default/kdump-tools file can be used to point to a specific initrd.img/vmlinuz pair. So we can do :
KDUMP_KERNEL=/var/lib/kdump/vmlinuz KDUMP_INITRD=/var/lib/kdump/initrd.img
When kexec will be loaded by kdump-config, it will find the appropriate files and load them in memory for future use. But for that to happen, those new parameter needs to point to the correct file. Here we use symbolic links to achieve our goal.
Linking to the smaller initrd.img
Using the hooks to create the proper symbolic links turns out to be overly complex. But since kdump-config runs at each boot, we can ask this script to be responsible for doing symlink maintenance.
Symlink creation follow this simple flowchart
This will assure that the symbolic links always point to the file with the version of the running kernel.
One drawback of this method is that, in the remote eventuality that the running kernel breaks the kernel crash dump functionality, we cannot automatically revert to the previous kernel in order to use a known configuration.
A future evolution of the kdump-config tool will add a function to specify which kernel version to use to create the symbolic link. In the meantime, the links can be created manually with those simple commands :
$ export wanted_version="some version"
$ rm -f /var/lib/kdump/initrd.img
$ ln -s /var/lib/kdump/initrd.img-${wanted_version} /var/lib/kdump/initrd.img
$ rm -f /var/lib/kdump/vmlinuz
$ ln -s /boot/vmlinuz-${wanted_version} /var/lib/kdump/vmlinuz
For those of you interested in nitty-gritty details, you can find the modifications in the following GIT branch :
Update: New git branch with cleanup commit history
https://github.com/karibou/makedumpfile-next/tree/smaller_initrd_final