The main tools for troubleshooting are the rmc recorders and the TR adsmon and adhmon.
NOTE: To alleviate the issue of faulty shared memory, the memory testing was made optional in GA3.0.12. Unfortunately this created some instability. This was finally solved in GA3.0.21. Any versions in between these should be avoided if you are using the RMC plugin
The rmc plugin reads information out of the shared memory on boxes running TREP components like ADH, ADS, ATS, DACS, RRCP and RMON. The plugin runs an rmcTest followed by an rcmGet on success.
The ADH, ADS and ATS write to shared memory by default. DACS comes with a dacs_trans process that will do the same for DACS. RRCP and RMON can write to shared memory using the TR RMS tools available from Thomson Reuters Customer Zone. All of these are 64 bit.
The default shared memory segments are:
The way to check if the process is publishing to shared memory is to run ipcs -a from command line and check for the semaphore keys
Reading it with the RMC plugin
Within the shared memory the individual variables takes a certain syntax, such as:
In the plugin we would refer to this as:
Where the instance is taken from the sampler configuration. The host variable comes from the shell.
How to check if the value is there using ITRS or TR tools
- rmcRecorder (available from our downloads page)
./rmcRecorder_<os> -k <key>
This will produce a mapping of the shared memory segment followed by updates received within the run period. A run period of approx. 5 minutes piped into a file should give you a good amount of recurring data showing that the component is running and reporting. Always use the bit version matching the writing process (ADS etc.) The default is now 64 bit. If you are looking for a particular key, combine this with grep.
- adsmon or adhmon. These are default placed in /opt/thomsonreuters/ads/bin/ or /opt/thomsonreuters/adh/bin. Navigating these vi style you can drill into the specific elements showing updating values all the way down to the actual variable behind it.
For other components that don't have a 'mon' process you can run, you can run up either adsmon or adhmon using the syntax
./adsmon -mob -key <key>
This allows you to browse the structure of the shared memory segment. You can also use the -print option to output the entire structure to a file.
- Probe crashes on startup with segmentation failure:
- Key not available:
- Values are truncated or individual values missing:
It has been seen that the rmcTest will fail if the process has had a 'dirty' restart, ie. kill -9 etc. If this is done, the shared memory will not clear out before restarting. This can be resolved by turning of the Shared Memory test in the sampler.
One way of testing the memory without relying on the rmcTest is to use a timing record. Each component published a heartbeat every 5 seconds. The default variable is: