OP5 Monitor - How to use ITRS Support Tools (ist, ist-collect, and ist-diagnose).

What are the ITRS Support Tools?

ITRS Support Tools (or henceforth IST) is a collection of tools that can help with diagnostics and log collection. There are three individual components of IST: ist, ist-collect, and ist-diagnose.

ist

A wrapper to launch ist-collect and ist-diagnose as the proper user and with the correct niceness.

ist-collect

A tool that collects various log files, configuration files and other system information from the local node or from some or all nodes in the cluster. ist-collect has its own documentation.

ist-diagnose

A tool to quickly self-diagnose a server running OP5 Monitor (EL7 only). ist-diagnose is a much simpler program to use with few options apart from the option -n which produces Naemon compatible output to be used as an OP5 Monitor plugin.

Installation

Enable the OP5 Support repository:

# yum install op5-support-release

Install the 'ist' package:

# yum install ist

The package ist will install ist-collect and ist-diagnose as dependencies.

How do I use these tools together?

Both ist-collect and ist-diagnose should always be launched through the wrapper ist. Before opening a support ticket to troubleshoot a problem, start by running ist diagnose, and see if the output can guide you in the right direction. A sample output could be the following:

root@master01:~# ist diagnose
1 Warning:
--------------------------------------------------------------------------------
Name: MYSQL max_connections

Current value: 214
Minimum recommended value: 1024

1 Error:
--------------------------------------------------------------------------------
Name: mon check spool /opt/monitor/var/spool/perfdata/

A problem was encountered, see output:

CRITICAL: 1308 files too old|old_files=1308;5;10;;

In the output above I can see 1 Warning and 1 Error:

• The max_connections variable in MySQL is set too low.
• I have old perfdata files in /opt/monitor/var/spool/perfdata/.

If any of these two seem to be related to the problem I'm experiencing, then I can try to fix it to see if it solves my main problem. If a problem persists after ist diagnose gives all OK, then it would be a good idea to open up a support ticket.

This is where ist collect will come into play. ist-collect can collect all kinds of data that will be useful for the Client Services Analyst who will help you with the troubleshooting. Performing a general collection from the current node is as simple as running ist collect. But there are many options to consider when using ist-collect, and it can be hard to know where to start.

Is the problem spread over multiple nodes?

Then consider using the -a option, or use the -N option if you know that it only affects a few of the nodes in the cluster. -a collects data from all nodes, but -N allows you to select the nodes that you want to collect data from.

Consider a cluster with the nodes master01, master02, and multiple pollers. If I know that the problem only affects the masters, then I could issue:

root@master01:~# ist collect -N master2

Since the command is launched from master01 it will collect data from master01 and master02.

If I also know that the problem is related to the component merlin, then I could add that as a module at the end of the command as such:

root@master01:~# ist collect -N master2 merlin

This will only collect merlin related files from the nodes master01 and master02. The produced file would then be a useful attachment when creating a new support ticket.

When in doubt, ask the Client Services Analyst for guidance on what information to collect.