The following FAQ will discuss how to use webmon and extractor when samplerDataTransfer is enabled so that a webpages contents can be monitored by Geneos.
In this example we will be monitoring and extracting information from the following website: https://www.treasurydirect.gov/TA_WS/securities/announced?format=html
Summary of Steps (Please see below for more detail)
1. Find out Request Header information
To ensure that webmon requests the page in the same format that we see in a normal browser, a good site to get this information is: https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending
You should see a website displaying your web browsers request header details, which will look like the following:
Make sure to note the details for: ACCEPT/ ACCEPT_ENCODING/ ACCEPT_LANGUAGE / USER_AGENT
2. Adding obtained Request Header information in Webmon
The main section where we will force webmon to parse the downloaded webpage from JSON into text/html is the Request data section. This can be found under the following:
Please see the below screenshot on where this can be found:
The Request Data area, is where additional headers can be sent up to the server as part of a request when attempting to access a webpage.
The headers we have used are as follows:
The important one is the Content-Type field where we force the download in the following format:
Content-Type text/html; charset=utf-8
However, ALL the other Request Headers would need to be modified, as the information obtained differs when sending headers from different web browsers. For example looking at the below screenshot you can see different information is gained when using Chrome vs Firefox
3. Enabling Webmon Data Transfer
Having downloaded the page in webmon, we then need pass it as a data stream to extractor. To enable this you will need to implement the following setting:
By specifying the name of the data transfer source, this setup can be seen in the below screenshot:
4. Setting the source of information for the extractor plugin
Moving on to the setup of the extractor plugin, we will need to specify the source of data in which the plugin will be reading from. In this case it will be data transferred from the webmon plugin.
This can be enabled as follows:
And give the name of the Data transfer implemented in webmon, in this example it would be called securities2
5. Extracting the html tags
After setting up the source to come from the webmon plugin, we can then extract individual or recursive xml tags as we would normally do within the extractor plugin.
A good way to view the xml structure of the webpage would be to download First Object. This tool, will allow you to parse the xml/html structure of the website to allow users to easily navigate through the websites structure and locate the relevant html tags to implement in the sampler. The download for this can be found here: http://www.firstobject.com/dn_editor.htm
After downloading First Object, on the webpage you want to extract data from do the following:
For example this would give you a layout as follows:
You can find the xml include attached at the bottom of the FAQ (Webmon_&_Extractor_Include.xml) in order to play around with config. The file contains the setup used in this example to parse JSON > HTML
We can see that the source of the information being passed to the extractor plugin is now in an html format by:
- Right clicking on the dataview in the state tree > debug > show source
You will see the following output as html:
VS - the below output which appears as JSON, which, as you can see cannot be displayed in the extractor plugin
NOTE: This is a workaround on how to parse JSON > HTML for the use of the extractor plugin a pebl(PLI-2229) has been raised for an enhancement on this.