Monitoring Geospatial Data Services: Grafana and GeoHealthCheck
Not all map services that use or serve APIs necessarily report when those APIs are not functioning. It might take days or weeks to hear about the issues from the service’s users, and to determine the cause you might need to manually sift through error reports and logs. When there are dozens or even hundreds of APIs, these issues can accumulate quickly. Ideally, the owner of the API would know about problems before the users do and act accordingly. In this article, we introduce two solutions that make the work of those serving geospatial data APIs easier.
Grafana — Almost Everything for Almost Everyone
Grafana is a multi-platform open-source web application. The basic principle is simple: it queries a database and from the results it plots charts as an interactive visualization on a web application. Users can either utilize Grafana’s ready-made queries or write their own. The data and visualizations provided by Grafana can be viewed in a web browser. Visualization options include various types of charts such as line, bar, and pie charts.
The primary Grafana program within the ecosystem is used for visualizing data, but other components are also included like the Prometheus database, where data is stored and queried by Grafana. In Prometheus the retention period for data can be defined. Other databases can also be used. Additionally, Grafana includes various backend functionalities for a wide range of needs, including those beyond geospatial data services.
Setting up Grafana and Prometheus can be done conveniently with container solutions like Docker. For those familiar with the subject matter, the setup can be quite quick. However, as always, the more complex the task, the more time it takes. Nevertheless, Grafana can be used to monitor pretty much anything, and setting up and using more advanced tools (such as Elastic Stack) is even more complex.
Container Monitoring and Server Cost Savings
Why should anyone be interested in Grafana? If, for example, there are multiple servers in use and you want to know if each one is necessary, Grafana can show their load and determine if some servers are idle. Removing even a few unnecessary instances can result in significant cost savings.
The software can also monitor service availability: if, for example, no data is received from a server for a minute, Grafana can alert the user via email, port, or web address as specified by the user. This enables quick reaction if a server is down or stuck for any reason. Users can also set up alerts for other threshold violations.
We have used Grafana in our client projects. Recently, one of our clients had containers on a server from which they wanted data. For this purpose, we used a combination of Grafana, Prometheus and C-advisor. First, we set up the C-advisor program on the server, which monitored the containers and published the data to a desired port like a web server. Prometheus then read this port at regular intervals — in this case once every 15 seconds. Grafana read the data from Prometheus’ database and plotted curves based on the queries. Eventually, Grafana displayed many parameters for each container — from memory consumption to processing time.
Why Grafana?
Of course, Grafana is not the only software that can perform these processes. In theory, Microsoft’s PowerBI could be used to read from C-advisor. The data provided by C-advisor is ultimately just names and numbers, such as “C-usage 2.3,” where the numeric value varies over different time intervals.
Compared to PowerBI and other software, Grafana’s advantage is that it is open-source (AGPL3.0 licence) and does not require any monthly fees (although there is also a hosted, monthly charged version available). It can be run on your own instance, and installation is quite straightforward.
Another significant advantage is that if the containers or other objects being monitored are behind a firewall where external access is not possible, Grafana can be placed in the same instance. This way, everything remains secure, and no changes need to be made to the firewall for data transfer.
Although we use Grafana for processes related to geospatial data, the software can be connected to almost anything that provides some numerical output. Grafana demo page allows you to explore its functionalities, such as tracking logins, payment amounts, and other variables over different time periods.
Grafana demo can be found here: https://play.grafana.org/
GeoHealthCheck — Simple Monitoring for Geospatial services
GeoHealthCheck is a browser-based software that monitors API services intended for serving geospatial data. Like Grafana, GeoHealthCheck monitors service availability, draws diagrams, and checks how quickly the service responds to various queries. The software is remarkably easy to set up and can be described roughly as a one-trick-pony: it does one thing and shows the results in one way.
Users can define how thoroughly the software examines the API. Geospatial data services have their own logic, and GeoHealthCheck can be used, for example, to check the GetCapabilities document from a WFS API and a few layers or features from there. Queries made to the API are displayed under the “Probes” section on the site, and the history section shows the latest errors. The site also provides a full report of the queries.
Just a Server and a Little Effort
While Grafana requires a lot of technical knowledge about the services being monitored (such as which port is open to the server, etc.), GeoHealthCheck only needs the address of the desired API service.
Setting up the software requires a server and a bit of effort — and may we emphasize: not a huge effort, just a bit. As the software is lightweight and does not require much, it can work on various different platforms.
GeoHealthCheck is best set up on a server using container deployment, which has been made relatively straightforward for the user. The most adept users can run the software in a container on their own machine. After this, the APIs to be monitored can either be entered manually or, if there are many APIs, imported as JSON file.
The software also allows for the creation of different users and granting them rights to monitor service status. This allows a whole team to use the same GeoHealthCheck instance for service monitoring. (Another option would be to set up a separate instance of the software for each user.) User management enables restricting the visibility of services so that some information can be displayed publicly, some only to logged-in users, or even different services to different users.
As a side note, allowing registration requires some tweaking because then a machine capable of sending emails must be available to send confirmation emails. However, using such a machine is sensible anyway, as GeoHealthCheck can then send alert messages directly to email. Otherwise, alerts can be set to be sent to, for example, a desired website.
Like Grafana, GeoHealthCheck is open-source and allows for custom extensions. The software is constantly being developed, and although it already works well, it is not yet perfect. For example, if a user wants to change their password in a situation where the software does not have a machine capable of sending emails, they have to deal with unnecessarily difficult JSON operations. Fixing issues like this and minor development are things that we at Gispo do for our clients.
Easy Benefits
The benefits of the software are clear: the status of APIs can be easily seen, and if problems arise, alerts can be received in the desired manner. Reports can be exported as CSV, JSON, or PNG files. Testing existing APIs and monitoring their availability is easy. The software also allows for live testing, which is helpful when making adjustments to an API, updating GeoServer, or releasing a new API.
Thanks to alerts, problems can be responded to more quickly, and the amount of manual work required to troubleshoot decreases when errors don’t need to be manually extracted from the GetCapabilities report but are neatly displayed in the browser interface. Naturally, the more APIs are available, the more beneficial the software becomes.
The advantage of GeoHealthCheck lies in its simplicity and clarity, as well as its easily understandable user interface. If you need to monitor the things GeoHealthCheck offers, there’s no need to unnecessarily complicate matters by setting up Grafana to do the same. GeoHealthCheck is specifically designed for spatial data use, but it can also monitor regular websites.
GeoHealthCheck’s demo can be found here: https://demo.geohealthcheck.org/
This article was initially published at www.gispo.fi by Juho Rekilä (Gispo Ltd.).
