The main tools for troubleshooting are the RMC recorders and the Refinitiv adsmon and adhmon.
NOTE: To alleviate the issue of faulty shared memory, the memory testing was made optional in GA3.0.12. Unfortunately this created some instability. This was finally solved in GA3.0.21. Any versions in between these should be avoided if you are using the RMC-Interface plugin
The RMC-Interface plugin reads information out of the shared memory on boxes running RTDS components like ADH, ADS, ATS, DACS, RRCP and RMON. The plugin runs an rmcTest followed by an rmcGet on success.
The ADH, ADS and ATS write to shared memory by default. DACS comes with a dacs_trans process that will do the same for DACS. RRCP and RMON can write to shared memory using the TR RMS tools available from MyRefinitiv Customer Zone. All of these are 64 bit.
The default shared memory segments are:
The way to check if the process is publishing to shared memory is to run ipcs -a from command line and check for the semaphore keys
Reading it with the RMC plugin
Within the shared memory the individual variables takes a certain syntax, such as:
In the plugin we would refer to this as:
Where the instance is taken from the sampler configuration. The host variable comes from the shell.
How to check if the value is there using ITRS or TR tools
- rmcRecorder (available from our downloads page)
./rmcRecorder_<os> -k <key>
This will produce a mapping of the shared memory segment followed by updates received within the run period. A run period of approx. 5 minutes piped into a file should give you a good amount of recurring data showing that the component is running and reporting. Always use the bit version matching the writing process (ADS etc.) The default is now 64 bit. If you are looking for a particular key, combine this with grep.
- adsmon or adhmon. These are default placed in /opt/refinitiv/ads/bin/, /opt/refinitiv/adh/bin or /opt/rtds/ads/bin/, /opt/rtds/adh/bin . Navigating these vi style you can drill into the specific elements showing updating values all the way down to the actual variable behind it.
For other components that don't have a 'mon' process you can run, you can run up either adsmon or adhmon using the syntax
./adsmon -mob -key <key>
This allows you to browse the structure of the shared memory segment. You can also use the -print option to output the entire structure to a file.
- Probe crashes on startup with segmentation failure:
- Key not available:
- Values are truncated or individual values missing:
It has been seen that the rmcTest will fail if the process has had a 'dirty' restart, ie. kill -9 etc. If this is done, the shared memory will not clear out before restarting. This can be resolved by turning of the Shared Memory test in the sampler.
- Probe error message - Failed loadGatewayParameter()
This is due to the plugin not being able to resolve a setting. This is normally due to a variable being set in the one of the plugin options which cannot be resolved. This can be checked with the Gateway Setup Editor and running the verify option which will report an error if a variable cannot be resolved.
One way of testing the memory without relying on the rmcTest is to use a timing record. Each component published a heartbeat every 5 seconds. The default variable is: