Monitoring
The Data Flow monitoring architecture helps to deliver application metrics critical to the health and performance of the server infrastructure and the deployed stream and task pipelines.
Designed around the Micrometer library the Data Flow monitoring supports some of the most popular monitoring systems, such as Prometheus, Wavefront and InfluxDB.
Wavefront is a high-performance streaming analytics platform that supports 3D observability (metrics, histograms, traces/spans). It scales to very high data ingestion rates and query loads while also collecting data many services and sources across your entire application stack.
Prometheus is a popular pull-based Time Series Database that pulls the metrics from the target applications with pre-configured endpoints and provides a query language to select and aggregate time series data in real time.
The Data Flow architecture employs the Prometheus-RSocket-Proxy to provide an uniform support of both long-lived (streams) and short-lived (tasks) applications.
InfluxDB is a popular open-source push-based Time Series Database. It supports downsampling, automatically expiring and deleting unwanted data, and backup and restore. Analysis of data is done through an SQL-like query language.
The Data Flow allows you to declaratively select and configure which monitoring system to use. Just apply the Spring Boot metrics configuration to enable and configure the Data Flow monitoring support. Usually you will add the following configurations to your Data Flow and Skipper server configurations:
management.metrics.export.<your-meter-registry>.enabled=true
management.metrics.export.<your-meter-registry>.<meter-specific-properties>=... - Replace the
<your-meter-registry>with either,influx,wavefrontorprometheus. (Note: in case of prometheus enable the prometheus-rsocket-proxy as well:management.metrics.export.prometheus.rsocket.enabled=true). - Use the Spring Boot/Micrometer meter specific configurations provided for Wavefront, InfluxDB and Prometheus/RSocket-Proxy.
By default the Micrometer configuration is reused for both the server infrastructure and the data pipeline monitoring.
To help you get started, Data Flow provides Grafana and Wavefront dashboards that you can customize.
You can also opt for the Wavefront Data Flow SaaS tile.
For detailed information on how to set up the monitoring infrastructure follow the following feature guides:
The following image shows Data Flow with enabled monitoring and Grafana buttons:
The following image shows the Stream applications view in the Grafana dashboard:
Here's the Tasks & Batch applications view in Grafana dashboard:
Here's Wavefront Stream applications dashboard:
Next: visit the Servers Monitoring Feature Guide, Stream Monitoring Feature Guide and Task Monitoring Feature Guide guides for further information on how to set up the Data Flow monitoring infrastructure.