Custom snmpd extension for port checking

As weird as it sounds, recently I had a task to accomplish port checks without access to the LAN on which daemons listen for connections. Speaking of a monitoring solution, the obvious choice was SNMP, which is the most widespread means of getting health information from network-attached devices, anyway. We perform an “indirect” port check, meaning that it’s sufficient for us to know that a process is listening on a given port without trying to communicate with it.

I found three alternative methods to achieve such a port check via SNMP. They have one thing in common: they utilize Net-SNMP server and/or client binaries.

  • The snmpnetsat utility. A straightforward answer to our question. Provides a netstat-like listing of active connections on the remote host. But it’s got its flaw. It queries the older MIB-II (RFC1213) objects tcpConnectionTable and udpTable, which don’t support IPv6. As a result, server processes listening on all IPs on an IPv6-enabled system doesn’t show up in the list.
  • tcpListenerTable and udpEndpointTable objects. These two were defined as part of TCP-MIB (RFC4022) and UDP-MIB (RFC4113) respectively. As their names suggest, these structures hold all listener processes on the system. Apparently indexes cannot be queried directly, so one needs to walk through the whole table to find a specific port. At least this is the case in Net-SNMP v5.5, but I didn’t investigate any further.
  • Custom snmpd extension. Of course we don’t want to develop a new MIB module for the sake of this single task. Several other ways exist to extend snmpd functionality – they are all described in snmpd.conf man page. The basic idea is to collect the listener data with a script and let snmpd transfer it to the client in a simple form.

So we have two viable options here. One is to let the client (monitoring system) extract the information it needs from tcpListenerTable and udpEndpointTable. The other is to leave decision making to the server (monitored host) and let it serve the other party with a simple OK/NOT-OK value. Needless to say, former was the winner in this competition because of the smaller impact on host configurations.

But, surprise-surprise, I created a working POC solution for the other one, too. In memoriam…

Extending snmpd

As I already mentioned, my idea of an extension was to run an external script from the daemon and provide the client with a single value it needed. An SNMP-aware solution (denoted as a MIB-Specific Extension Command by the documentation) is what fits. In this scenario the daemon acts lazy and delegates all work to the external command: it transfers object ID and request type (GET, GETNEXT, SET) and expects an object value or a report on the outcome.

All we have to do is assign an unused portion of the MIB-tree to the script with a pass directive and grant view rights to the client (“systemview” is “public” by default).

pass .1.3.6.1.4.1.2021.51 /path/to/script.sh
view systemview included .1.3.6.1.4.1.2021.51

So it’s now fully up to us, what happens when an object in this subtree is requested. Our script serves objects which have numeric OIDs of the form “ROOT.PORT”, where ROOT is the subtree’s relative root and PORT can be any valid port number. The object value returned corresponds to the number of processes bound to that port. E.g. suppose ROOT is .1.3.6.1.4.1.2021.51, if a GET request is received on .1.3.6.1.4.1.2021.51.22, then the response will hold the number of processes listening on port 22. Obviously the value is 0, if there’s no such process. When responding to GETNEXT requests, only those ports are taken into consideration which are bound to a process.

#!/bin/sh

# Find ports bound to a single IP address. (Empty means ALL.)
IP=

# OID of the subtree root object assigned to this extension (with leading dot).
ROOT=.1.3.6.1.4.1.2021.51

# Paths to binaries
LSOF=/usr/sbin/lsof
GREP=/bin/grep
SORT=/bin/sort
SED=/bin/sed
HEAD=/usr/bin/head
WC=/usr/bin/wc

OID=$2

# Check wether OID is valid (it's a direct descendant of ROOT or ROOT itself)
echo $OID | $GREP -E ^$ROOT\.?[0-9]*$ > /dev/null || exit

# Extracting the last portion of the OID
[ "$OID" != "$ROOT" ] && port=`echo $OID | $GREP -o "[0-9]*$"` || port=0

[ "$IP" != "" ] && IP="@$IP"

case $1 in

	"-g")

		# Checking if port is in valid range
		[ "$port" -gt "65535" ] && exit

		[ "$port" = "0" ] && echo -e "$OID\nstring\n$0" && exit
	;;

	"-n")

		(( port = $port + 1 ))

		# Checking if port is in valid range
		[ "$port" -gt "65535" ] && exit

		# Searching next listener port on the system
		port=`($LSOF -i$IP:$port-65535 -sTCP:LISTEN -Fnp -P; $LSOF -iUDP$IP:$port-65535 -Fnp -P) | $GREP "^n" | $GREP -o "[0-9]*$" | $SORT -n | $HEAD -1`
		[ "$port" == "" ] && exit
		OID="$ROOT.$port"
	;;

	"-s")

		# Refusing SET requests
		echo not-writable; exit
	;;

	*)
		exit
	;;
esac

# Output for snmpd (number of processes)
echo -e "$OID\ninteger"
($LSOF -iTCP$IP:$port -sTCP:LISTEN -Fp; $LSOF -iUDP$IP:$port -Fp) | $SORT -u | $WC -l

Just an example on how this can be used from the client side. Let’s add a command to Nagios configuration.

define command{
        command_name    check_snmp_port
        command_line    $USER1$/check_snmp -H $HOSTADDRESS$ -C public -c 1: -o .1.3.6.1.4.1.2021.51.$ARG1$
}

Where $USER1$ is a user macro holding the path to the Nagios plugin directory, where check_snmp resides. Obviously the first argument ($ARG1$) is the port number, so the check_command directive in a service definition should be something like this: check_snmp_port!22.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s