Task and Batch Monitoring with Prometheus and InfluxDB

This section describes how to monitor the applications that were deployed as part of a Task definition in Data Flow. The setup for each platform is different, but the general architecture is the same across the platforms.

The Data Flow metrics architecture is designed around the Micrometer library, which is a vendor-neutral application metrics facade. It provides a simple facade over the instrumentation clients for the most popular monitoring systems. See the Micrometer documentation for the list of supported monitoring systems. Micrometer is the instrumentation library that powers the delivery of application metrics from Spring Boot. Spring Batch provides additional integration to expose metrics around task durations, rates, and errors, which are critical to the monitoring of deployed batch-jobs.

We focus on using three time series databases: Wavefront, Prometheus, and InfluxDB.

Wavefront is a high-performance streaming analytics platform that supports 3D observability (metrics, histograms, and traces and spans). It scales to very high data ingestion rates and query loads while also collecting data from many services and sources across your entire application stack.

Prometheus is a popular pull-based time series database that pulls the metrics from the target applications with pre-configured endpoints and provides a query language to select and aggregate time series data in real time.

InfluxDB is a popular open-source push-based time series database. It supports downsampling, automatically expiring and deleting unwanted data, and backup and restore. Analysis of data is done through an SQL-like query language.

Enable Task Metrics

To enable Task metrics integration with Data Flow, you must add the spring-boot-starter-actuator to your task application and import the spring-cloud-dependencies BOM, as follows:

<dependencies>
  <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>2021.0.5</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Then you need to include the desired Micrometer registry as the dependency in the Task POM:

<dependency>
	<groupId>io.micrometer.prometheus</groupId>
	<artifactId>prometheus-rsocket-spring</artifactId>
    <version>1.5.0</version>
</dependency>

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-wavefront</artifactId>
</dependency>

<dependency>
    <groupId>com.wavefront</groupId>
    <artifactId>wavefront-sdk-java</artifactId>
</dependency>

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-influx</artifactId>
</dependency>

Build Docker Image

To build a Docker image, run the following:

mvn spring-boot:build-image

To help you get started monitoring tasks, Data Flow provides Grafana dashboards that you can install and customize for your needs.

The following image shows the general architecture of how task applications are monitored:

Spring Cloud Task Metric Tags

To allow aggregating metrics per application type and per instance ID or per task name, the Spring Cloud Task applications are configured to use the following Micrometer tags:

• task.name: The name of the Task that contains the applications that send the metrics.
• task.execution.id: The instance ID of the executed task.
• task.external.execution.id: The external Task ID, as present on the target platform (such as Cloud Foundry or Kubernetes) The type (Source, Processor, or Sink) of the application that reports the metrics.
• task.parent.execution.id: The parent task ID used to identify task that executes another task or tasks.

If the Data Flow server is started with the spring.cloud.dataflow.metrics.dashboard.url property pointing to your Grafana URL, the Grafana feature is enabled and the Data Flow UI provides you with Grafana buttons that can open a particular dashboard for a given task.

Installing Wavefront, Prometheus, and InfluxDB differs, depending on the platform on which you run. Links to installation instructions are provides in each of the following sections.

Local

This section describes how to view application metrics for tasks by using either Prometheus or InfluxDB as the metrics store on your local machine.

app register --name myTask --type task --uri https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow-samples/main/dataflow-website/feature-guides/batch/monitoring/prometheus-task-demo-metrics-0.0.1-SNAPSHOT.jar

task create --name task1 --definition "myTask";

task create --name task2 --definition "myTask";

task launch --name task1

task launch --name task2

app register --name myTask --type task --uri https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow-samples/main/dataflow-website/feature-guides/batch/monitoring/influx-task-demo-metrics-0.0.1-SNAPSHOT.jar

task create --name task1 --definition "myTask";

task create --name task2 --definition "myTask";

task launch --name task1

task launch --name task2

Kubernetes

This section describes how to view application metrics for task by using Prometheus or InfluxDB as the metrics store on Kubernetes.

minikube service --url grafana
http://192.168.99.100:31595

dataflow:>app register --name myTask --type task --uri docker://<your task app>

dataflow:>task create --name task1 --definition "myTask"
dataflow:>task create --name task2 --definition "myTask"

dataflow:>task launch --name task1
dataflow:>task launch --name task2
dataflow:>task launch --name task1
dataflow:>task launch --name task2
dataflow:>task launch --name task1
dataflow:>task launch --name task2

minikube service --url scdf-server
http://192.168.99.100:32121

management:
  metrics:
    export:
      wavefront:
        enabled: true
        api-token: <YOUR API-KEY>
        uri: <YOUR WAVEFRONT-URI>
        source: demo-scdf-source

Cloud Foundry

This section describes how to view application metrics for streams by using Prometheus and InfluxDB as the metrics store on Cloud Foundry.

dataflow:>app register --name myTask --type task --uri https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow-samples/main/dataflow-website/feature-guides/batch/monitoring/prometheus-task-demo-metrics-0.0.1-SNAPSHOT.jar

dataflow:>task create --name task1 --definition "myTask"
dataflow:>task create --name task2 --definition "myTask"

dataflow:>task launch --name task1
dataflow:>task launch --name task2