Monitoring

The Data Flow metrics architecture is designed around the Micrometer library, which is a vendor-neutral application metrics facade. It provides a simple facade over the instrumentation clients for the most popular monitoring systems. The Micrometer instrumentation library powers the delivery of application metrics from Spring Boot and includes metrics for message rates and errors, which is critical to the monitoring of deployed streams.

The prebuilt applications are configured to support two of the most popular monitoring systems, Prometheus and InfluxDB. You can declaratively select which monitoring system to use.

To help you get started monitoring Streams and Tasks, Data Flow provides Grafana Dashboards that you can customize for your needs.

The following image shows the general architecture of how the Stream applications are monitored:

Data Flow Stream Monitoring Architecture

And similarly for the Task monitoring architecture:

Data Flow Task Monitoring Architecture

The following image shows the creation of two streams defined with the time | filter | log DSL expression:

Two stream definitions

The following image shows the Stream applications view in the Grafana dashboard:

Grafana Streams Dashboard

Here's the Tasks & Batch applications view in Grafana dashboard:

Grafana Tasks Dashboard

The Stream Monitoring Feature Guide and Task Monitoring Feature Guide contains detailed information on how to set up the monitoring infrastructure.