August 3, 2019

Kernel Panic: Automatic reboot a Server Linux and Send email to the system Administrator

Kernel Panic is the same as the famous blue screen of Death (BSOD) in windows, only with one difference. The Linux server won’t reboot automatic and keeps in the Kernel Panic for days if nobody notifies that the server is Offline. So when you get noticed that you have a Kernel Panic you can drive to the DataCenter (most of the time it is not next to the door) and reboot your Linux Server. If you have a managed server contract your better of and can simply open a support ticket to reboot the server. The reason of a Kernel Panic can be a wrong write to the memory, bad software instructions or failing hardware. Therefore, to make the life of a system administrator a little easier we can set our server to Automatically reboot when we have a Kernel Panic. Afterwards we do want to know that our server had a Automatically reboot, so we instruct our Linux Server to email the system administrator.

Force Linux to reboot even you could not start a shell via SSH If the server is too busy, creating the shell via SSH may also fail even though.

If the server is too busy, locked down on a process, creating the shell via SSH may also fail even though ssh is alive. There are times, you get lucky that you can execute remote some commands by ssh directly. You may try to make use of the magical SysRq to force Linux to restart.

ssh [email protected]_home \
'echo 1 > /proc/sys/kernel/sysrq; echo b > /proc/sysrq-trigger'

If this command works, you will find your server disappear from the network, while it may be rebooting itself. Wait for a while and it may come back.

Make Linux automatically reboot after a kernel panic and notify the  system administrator

Some times, you are in bad luck that there is a real kernel panic. Almost everything including the network stop working and you can not connect to the server anymore. That is not good but may not be too bad if we did our home work before by configuring Linux to reboot itself after kernel panics.

Linux has a nice feature that reboots itself after a certain timeout if a kernel panic happened. Normally, this is disabled by default. We can turn it on as we are “lazy system admins”.  It can be enabled by setting the kernel.panic kernel parameter.

For a running system:

echo 20 >/proc/sys/kernel/panic

sudo nano /etc/default/grub
add kernel.panic = 20 to /etc/sysctl.conf

Here, 20 is the number of seconds before the kernel goes into reboots. 0 means this feature is disabled. To make the configuration definitive, you have at least 2 choices: Add the kernel parameter panic=20 to your bootloader (grub or grub2).

I prefer the second method that writes the configuration to /etc/sysctrl.conf.

nano /etc/sysctl.conf
kernel.panic = 20

Here, 20 is the number of seconds before the kernel reboots. 0 means this feature is disabled.

Email notifications after Linux reboot.
Auto reboot is a good thing. We want our servers with the maximum up time. It will be better that the server also notifies the system admins after a reboot.

It makes use of the @reboot cron jobs and mailx by adding an entry like

crontab -e
@reboot date | mailx -S smtp=smtp://smtp.example.com -s "`hostname` started" -r [email protected] [email protected]


 

 

 

Share

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *