NMI Watchdog is something you would have heard of. NMI or Non Maskable
Interrupt, is a type of interrupt which the CPU cores canʼt ignore. Other type of interrupts, can be routed to different CPUs depending on its workload, while a NMI interrupt canʼt be passed on and has to be handled by the CPU core which got the interrupt.

Historically, NMI Watchdog used to interrupt CPUs whenever there is a critical hardware error, but recently it is also used to do a soft reset whenever there is a kernel panic. If the interrupts are not increasing (5 sec interval ?), it means that the system is hung and the NMI initiates a reboot or kernel panic and generates a debug log.

However there could be situations where NMI watchdog is disabled. If NMI is disabled, you may get 0 values in the NMI row of /proc/interrupts. Benefit of NMI is that it can trigger a panic, but with a crash dump so that we can later analyze the situation which caused the crash.

What if, I want to kill some process whenever the load rises above certain limit and bring the system under control, automatically ? We used to run this using a cron script, but often times, I have noticed that cron jobs donʼt run whenever there is a heavy load. It just skips.

The hangwatch daemon will invoke pre-configured sysrq triggers when system load average exceeds a certain threshold. Hangwatch periodically polls /proc/loadavg, and echos a user-defined set of characters into /proc/sysrq-trigger if a user-defined load threshold is exceeded.

Unlike in cron, here with hangwatch, the system is passed a magic sysrq to the /proc/sysrq-trigger which will do the needful, including passing SIGTERM to all processes running.

https://github.com/jumanjiman/hangwatch/

You can set the configurations in the file /etc/sysconfig/hangwatch. A copy of the settings can be found at

github.com/jumanjiman/hangwatch/blob/master/src/etc/sysconfig/hangwatch

About+

Hire a service+

Tech Stack+

SupportSages

Posts by Abhilash R

About

Hire a service

Tech Stack