Health Monitoring

Aggregates diagnostics data from devices, synchronizes them with the cloud, and aggregates them for easy monitoring. On the robot the capability collects diagnostics from the ROS /diagnostics topic as well as some hard-coded OS level metrics (ping timings, CPU temperature, GPU temperature (NVIDIA only), and available disk space).

Works with both ROS 1 and ROS 2, even simulatenously if desired.

Built into the capability itself, i.e., not requiring you to publish to /diagnostics, is a simple ping-monitor. It pings one of our servers 30 times a minute and reports aggregate results for each minute:

• percentage of successful pings
• min/avg/max ping timings

The capability stores historic values in ClickHouse, shows them as in-line spark-lines in the UI and allows you to download them as JSON. Clicking the sparkline plot opens it in Grafana.

You have full access to the health data you collect from your devices and can build your own Grafana dashboards to visualize it, e.g., showing the health level history as colored bars.

This capability provides a Grafana Alert template that you can use to get notified when your robots are unhealthy (or some other condition of your choice). To do so, log into your Grafana account (using the instructions on your Security page), create a copy of the alert template, and set a contact point, e.g., Slack, PagerDuty, Telegram, or your own AWS SNS (for email, SMS). To learn more, please see the Grafana Alerting documentation.

• ros-DISTRO-diagnostic-msgs for the ROS distro you use
• iputils-ping for the ping monitor to work
• lm-sensors for temperate sensors to work