OP5 Monitor - How to use event handlers to restart services

Introduction

This article will step you through using an event handler in OP5 Monitor to restart services, processes, and daemons that have stopped responding. We will focus on Windows targets (hosts) using the NSClient++ agent, also known as NCSP.

Our demonstration example will restart Microsoft's Internet Information Service (IIS) on a Windows Server. Rather than make you bounce from link to link, we will cover all the steps in order: from installing the NSCP agent in OP5 mode to getting automated confirmation that the process restarted. You can of course use event handlers to initiate any kind of action upon a state-change of a host or a service, such as writing an entry to a log or power cycling an appliance server. As you get comfortable with the main steps, you can review our more advanced steps in the section below to add support for additional arguments from the OP5 server side.

Please note: This document is in transition -

1. NSCP has improved dramatically since this article's first publication in 2013. Thus we are revising and simplifying the steps to match its new configuration approach and modern implementation for multiple scripting languages. You can read a lot more in the NSCP article about external scripts.
2. We also provide more Unix-centric steps [in an article to be linked later]. However, NSCP also supports Linux and has a superior authentication setup than the traditional NRPE agent. Thus we will improve this document over time to include Linux and Windows steps. OP5 would like its customers to feel as comfortable as possible with NSCP as their agent of choice, as our preferred agent, and as the simplest way to get big and small server farm deployments underway and in compliance.

Overview - the three main parts

Upon a state change of a host or service, an event handler can be executed. Every host and service object has three related advanced settings that control this behavior:
event_handler ? the command to call
event_handler_args ? the arguments to pass along to the command
event_handler_enabled ? the on-off switch for the event handler

The event handler that you define will be executed whenever a host or service state change occurs. An event handler is a command just like your other check-commands - a configuration object with a name and a command_line that may be configured to handle arguments. The typical event handler is a reference to a script.

The script that the event handler calls should be able to perform more than one action, depending on which kind of state change just occurred. It's a good idea to include logging in to your event handler script. A typical action performed by an event handler script is to call an NRPE (Nagios Remote Plugin Executor) command on a remote server. The remote command can for example execute a local script that restarts a service or process.

The IIS example

In this example, we assume that we already have a service configured that tests the availability of a website. If this service goes critical, we want to use an event handler to restart the IIS service on the remote server.

Important files, commands, and addresses

On the Windows target

First, we script and test the restart action for the IIS service.

1. Create a script that stops and starts the IIS service. Save your script to the following path and filename: C:\Program Files\NSClient++\scripts\custom\. This script must meet the standards for a Nagios plugin response, such as including an explicit 'OK' reply upon completion and a return value. Here is an example that meets the NRPE and OP5 criteria:
   

    

   restartiis.bat

   @echo on
   C:\Windows\System32\iisreset.exe
   @echo OK: This Windows web service is back on
   @exit 0

    

2. Create or edit custom.ini located in "C:\Program Files\NSClient++\".
3. Add the modules section just as below to turn on the External Scripts module in NSCP, then add a reference to a reference like the one below to make a tag that holds the NSCP-based path for the batch script:
   
   custom.ini

   [/modules]CheckExternalScripts = enabled
   [/settings/external scripts/scripts]
   restart_iis = scripts\custom\restartiis.bat

4. Restart the NSClient++ system service to reload the configuration.
5. Test your restart action end-to-end from the command line interface at your OP5 server (via ssh or directly at the console):
   

   /opt/plugins/check_nrpe -H 192.168.1.8 -c restart_iis

6. If your test successfully restarted the IIS service, you can now continue with the scripting and configuration on the OP5 Monitor server.

On the OP5 Monitor server

First, create a script to which your new event handler command can point. The script must be able to handle different state-types and have some kind of logging. An example follows:

#!/bin/bash

## Event handler script for executing an NRPE command on a given host when a service
# is in CRITICAL and HARD state.

state=$1 # the four states are "OK","CRITICAL,"WARNING", and "UNKNOWN"
statetype=$2 # this value will either be SOFT or HARD
host=$3 # IP address of the host target that will execute the NRPE command
command=$4 # NRPE command to execute
logfile=/opt/monitor/var/eventhandler.log # log file to store executions by this event handler

# Date format: "2016-03-29 13:10 CEST"
date=`date +"%Y-%m-%d %H:%M %Z"`

case "$1" in (CRITICAL)
 if [ "$statetype" = "HARD" ] ; then
 /bin/echo -en "$date | ${0##*/} Got state: <$state> and statetype: <$statetype> with command <$command> for execution on host <$host>\n" >> $logfile
 /opt/plugins/check_nrpe -H $host -c $command >> $logfile
 fi
esac
exit 0

This simple script executes an NRPE command on a remote server if it is called with a couple of arguments where the first one is CRITICAL and the second one is HARD. It logs an informational line plus the output from the execution to a log file.

Now place this script in /opt/plugins/custom/.

Change the permissions to 755:

chmod 755 /opt/plugins/custom/restart_winsvc.sh

Use Configure in OP5 Monitors and add a new "command" (Configure -> Commands).
Configure your new command like this:

The macros above are just examples of what you may want to pass to your script. If you also want to check whether the object is in a downtime, so that the script can exit immediately in such a situation, you may want to include the $HOSTDOWNTIME$ and $SERVICEDOWNTIME$ macros, which will resolve to a number. If this number is greater than zero, the object is in a downtime.

When you have saved your new command, you edit the host where you want to use your new event handler command and select the service where you want to use it. Click the Advanced link, set the parameter "event_handler" to "restart_winsvc", and Save your configuration. You should now have a working event handler.

Adding support for arguments

The above configuration can be modified to accept arguments, such as which service should be restarted. This will allow you to pass different services as tags to control SQL Server's service or any other process.

It's important to understand that adding support for arguments involves modifying the whole transport of data between the OP5 Monitor server and the target Windows server.

The configuration changes

On the Windows target

Add the following to 'custom.ini' to allow NSCP's NRPE module commands t:

allow_arguments=1

This makes NRPE able to process requests with arguments.

Make a copy of restartiis.bat to restartservice.bat and change it to process a supplied argument.

Change your previously configured command definition in custom.ini to:

restart_service="C:\Program Files\NSClient++\scripts\restartservice.bat" $ARG1$

Restart the NSCP service to load the changes.

On the OP5 Monitor server

First, we need to modify the event handler-script. An example follows.

#!/bin/bash
## Event handler script for executing an NRPE command on a given host when a service# is in CRITICAL and HARD state.

state=$1 # eg. "OK","CRITICAL,"WARNING","UNKNOWN"
statetype=$2 # eg. "SOFT","HARD"
host=$3 # hostaddress of where to execute nrpe command
command=$4 # nrpe command to execute
cmdarg=$5
logfile=/opt/monitor/var/eventhandler.log # logfile to store executions by this eventhandler

# Date format: "2016-03-29 13:10 CEST"
date=`date +"%Y-%m-%d %H:%M %Z"`
case "$1" in (CRITICAL)
 if [ "$statetype" = "HARD" ] ; then
 /bin/echo -en "$date | ${0##*/} Got state: <$state> and statetype: <$statetype> with command <$command> and argument <$cmdarg> for execution on host <$host>\n" >> $logfile
 /opt/plugins/check_nrpe -H $host -c $command -a $cmdarg >> $logfile
 fi
esac
exit 0

The following has been added to the event handler-script:

cmdarg=$5

We have also changed the line where we execute check_nrpe.

Now we need to edit our check_command (Configure -> Commands) and edit the command restart_winsvc.

The new command-line should now look like this:

$USER1$/custom/restart_winsvc.sh $SERVICESTATE$ $SERVICESTATETYPE$ $HOSTADDRESS$ restart_service $ARG1$

Here we have added $ARG1$ and changed which NRPE command we call.

Finally, we need to change which arguments are supplied by the event_handler_args in the service-definition (Configure -> select host -> Services for host -> select service -> Advanced):

You should now have a working configuration where you can supply as an event handler argument, which Windows service name to restart upon problems.