Monitoring

The Data Flow monitoring architecture helps to deliver application metrics critical to the health and performance of the server infrastructure and the deployed stream and task pipelines.

Designed around the Micrometer library the Data Flow monitoring supports some of the most popular monitoring systems, such as Prometheus, Wavefront and InfluxDB.

Data Flow Servers, Streams & Tasks Monitoring Architecture

Wavefront is a high-performance streaming analytics platform that supports 3D observability (metrics, histograms, traces/spans). It scales to very high data ingestion rates and query loads while also collecting data many services and sources across your entire application stack.

Prometheus is a popular pull-based Time Series Database that pulls the metrics from the target applications with pre-configured endpoints and provides a query language to select and aggregate time series data in real time.

The Data Flow architecture employs the Prometheus-RSocket-Proxy to provide an uniform support of both long-lived (streams) and short-lived (tasks) applications.

InfluxDB is a popular open-source push-based Time Series Database. It supports downsampling, automatically expiring and deleting unwanted data, and backup and restore. Analysis of data is done through an SQL-like query language.

The Data Flow allows you to declaratively select and configure which monitoring system to use. Just apply the Spring Boot metrics configuration to enable and configure the Data Flow monitoring support. Usually you will add the following configurations to your Data Flow and Skipper server configurations:

management.metrics.export.<your-meter-registry>.enabled=true 
management.metrics.export.<your-meter-registry>.<meter-specific-properties>=... 
  • Replace the <your-meter-registry> with either, influx, wavefront or prometheus. (Note: in case of prometheus enable the prometheus-rsocket-proxy as well: management.metrics.export.prometheus.rsocket.enabled=true).
  • Use the Spring Boot/Micrometer meter specific configurations provided for Wavefront, InfluxDB and Prometheus/RSocket-Proxy.

By default the Micrometer configuration is reused for both the server infrastructure and the data pipeline monitoring.

To help you get started, Data Flow provides Grafana and Wavefront dashboards that you can customize.

You can also opt for the Wavefront Data Flow SaaS tile.

For detailed information on how to set up the monitoring infrastructure follow the following feature guides:

The following image shows Data Flow with enabled monitoring and Grafana buttons:

Two stream definitions

The following image shows the Stream applications view in the Grafana dashboard:

Grafana Streams Dashboard

Here's the Tasks & Batch applications view in Grafana dashboard:

Grafana Tasks Dashboard

Here's Wavefront Stream applications dashboard:

Wavefront Stream Application Dashboard

Next: visit the Servers Monitoring Feature Guide, Stream Monitoring Feature Guide and Task Monitoring Feature Guide guides for further information on how to set up the Data Flow monitoring infrastructure.