Stream Monitoring with Prometheus and InfluxDB

This section describes how to monitor the applications that were deployed as part of a Stream. The setup for each platform is different, but the general architecture is the same across the platforms.

The Data Flow 2.x metrics architecture is designed around the Micrometer library, which is a vendor-neutral application metrics facade. It provides a simple facade over the instrumentation clients for the most popular monitoring systems. See the Micrometer documentation for the list of supported monitoring systems. Starting with Spring Boot 2.0, Micrometer is the instrumentation library that powers the delivery of application metrics from Spring Boot. Spring Integration provides additional integration to expose metrics around message rates and errors, which is critical to the monitoring of deployed streams.

All the Spring Cloud Stream App Starters are configured to support two of the most popular monitoring systems, Prometheus and InfluxDB. You can declaratively select which monitoring system to use. If you do not use Prometheus or InfluxDB, you can customize the App starters to use a different monitoring system as well as include your preferred Micrometer monitoring system library in your own custom applications. To help you get started monitoring streams, Data Flow provides Grafana Dashboards that you can install and customize for your needs.

The following image shows the general architecture of how applications are monitored:

To allow aggregating metrics per application type and per instance or per stream, the Spring Cloud Stream Application Starters are configured to use the following Micrometer tags:

• stream.name: The name of the Stream that contains the applications that send the metrics
• application.name: The name or label of the application that reports the metrics
• application.type: The type (Source, Processor, or Sink) of the application that reports the metrics
• application.guid: Unique instance identifier of the application instance that reports the metrics
• application.index: Application instance ID (when available)

If the Data Flow server is started with the spring.cloud.dataflow.grafana-info.url property pointing to your Grafana URL, the Grafana feature is enabled and Data Flow UI provides you with Grafana-buttons that can open a particular dashboard for a given stream, application, or application instance. The following screenshots illustrate these buttons:

As setting up Prometheus and InfluxDB is different depending on the platform on which you run, we provide instructions for each platform. In Spring Cloud Data Flow 2.x, local server and Kubernetes instructions have been provided.

Local

This section describes how to set up Prometheus and InfluxDB for a local machine.

Prometheus

Prometheus is a popular pull-based time series database that pulls the metrics from the target applications from preconfigured endpoints. Prometheus needs to be regularly provided with the URLs of the target applications to monitor. This task is delegated to a component called Service Discovery, which is usually a platform-specific implementation. The Spring Cloud Data Flow server provides a standalone Service Discovery service that uses the Data Flow /runtime/apps REST endpoint to compute the URL locations of the deployed application’s Prometheus endpoints.

To enable Micrometer’s Prometheus meter registry for Spring Cloud Stream application starters, set the following properties.

management.metrics.export.prometheus.enabled=true
management.endpoints.web.exposure.include=prometheus

Disable the application’s security, which allows for a simple Prometheus configuration to scrape monitoring information by setting the following property:

spring.cloud.streamapp.security.enabled=false

The next steps is to start up Prometheus, Grafana, and the local service discovery application.

First, clone the Spring Cloud Data Flow Github repository for the tagged release and change to the prometheus/docker folder, as follows:

cd ./src/grafana/prometheus/docker

Next, set the SCDF_HOST_IP environment variable to the IP address of your local host, as follows:

export SCDF_HOST_IP=<YOUR locahost IP address>

Use the real IP address and not localhost/127.0.0.1. You can use ifconfig to find out your IP address.

In many cases, the find_host_ip.sh script (included in the distribution) can give you the IP address. To use it, run the following command:

source ./find_host_ip.sh

Now you can start Prometheus and Grafana + Service-Discovery by using docker-compose, as follows:

docker-compose up -d --build

You should check that the containers have started by using the docker ps command, as the following example shows (with output):

docker ps
CONTAINER ID IMAGE              ...  PORTS                    NAMES
2b8b6a442365 tzolov/spring-...  ...  0.0.0.0:8181->8181/tcp   service-discovery
bff63c4902d5 docker_prometheus  ...  0.0.0.0:9090->9090/tcp   prometheus
40190da6aa4b docker_grafana     .... 0.0.0.0:3000->3000/tcp   grafana

To validate the setup, you can log in to those containers by using the following commands:

docker exec -it service-discovery /bin/sh
docker exec -it prometheus /bin/sh
docker exec -it grafana /bin/bash

Then, on the prometheus and service-discovery containers, you can check the content of the targets.json file by running cat /tmp/scdf-targets/targets.json.

You can reach the Prometheus UI at http://localhost:9090/graph and http://localhost:9090/targets

You can reach the Grafana dashboard at http://localhost:3000 with the following credentials:

• User: admin
• Password: admin

It comes with two provisioned dashboards:

• Streams: http://localhost:3000/d/scdf-streams/streams?refresh=10s
• Applications: http://localhost:3000/d/scdf-applications/applications?refresh=10s

Start the Skipper server. Then start the Data Flow server with the following properties:

--spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.prometheus.enabled=true
--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.streamapp.security.enabled=false
--spring.cloud.dataflow.applicationProperties.stream.management.endpoints.web.exposure.include=prometheus,info,health
--spring.cloud.dataflow.grafana-info.url=http://localhost:3000

Now you can deploy a simple stream that uses Kafka, as follows:

dataflow:>app import --uri https://dataflow.spring.io/kafka-maven-latest --force
dataflow:>stream create stream2 --definition "time --fixed-delay=10 --time-unit=MILLISECONDS | filter --expression=payload.contains('3') | log" --deploy

You should see dashboards similar to those shown in the following image:

You can destroy all containers by running the following command:

docker-compose down

InfluxDB

InfluxDB is a popular open-source push-based time series database. It supports downsampling, automatically expiring and deleting unwanted data, and backup and restore. Analysis of data is done through an SQL-like query language.

To enable Micrometer’s Influx meter registry for Spring Cloud Stream application starters, set the following property:

management.metrics.export.influx.enabled=true

In the docker setup provided later in this section, the InfluxDB server runs on localhost:8086. If you use a different InfluxDB server, setting Common Application Properties for Influx is a convenient way to have all deployed applications configured to send metrics to Influx. The property to set is management.metrics.export.influx.uri. Alternatively, when deploying a stream, you can pass this as a deployment property (app.\*.management.metrics.export.influx.uri={influxdb-server-url}). The Micrometer influx documentation shows the full list of Spring Boot properties to configure sending metrics to Influx.

To start Influx and Grafana:

1. Clone the Data Flow Github repository for tagged release and change to the influxdb/docker folder, as follows:

   Copy

   cd ./src/grafana/influxdb/docker

2. Start Influx and Grafna by using docker-compose, as follows:

   Copy

   docker-compose up -d --build

3. Check that the containers have started by running the docker ps command, as follows (shown with its output):

docker ps
CONTAINER ID        IMAGE               PORTS                    NAMES
1b7633c63ba1        docker_influxdb     0.0.0.0:8086->8086/tcp   influxdb
2f42e88f0606        docker_grafana      0.0.0.0:3000->3000/tcp   grafana

To validate the setup, you can log in to those containers by using the following commands:

docker exec -it influxdb /bin/sh
docker exec -it grafana /bin/bash

Then you can check the content of InfluxDB by running the following commands:

root:/# influx
> show databases
> use myinfluxdb
> show measurements
> select * from spring_integration_send limit 10

You can reach the Grafana dashboard at http://localhost:3000 with the following credentials:

• User: admin
• password: admin

It comes with two provisioned dashboards.

• Streams: http://localhost:3000/d/scdf-streams/streams?refresh=10s
• Applications: http://localhost:3000/d/scdf-applications/applications?refresh=10s

Now you can start the Skipper server. Then start the Data Flow server with the following properties:

--spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.enabled=true
--spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.db=myinfluxdb
--spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.uri=http://localhost:8086
--spring.cloud.dataflow.grafana-info.url=http://localhost:3000

Now you can deploy a simple stream that uses Kafka, such as the following:

dataflow:>app import --uri https://dataflow.spring.io/kafka-maven-latest --force

dataflow:>stream create stream2 --definition "time --fixed-delay=10 --time-unit=MILLISECONDS | filter --expression=payload.contains('3') | log" --deploy

You should see dashboards similar to those shown in the following image:

Kubernetes

This section describes how to set up Prometheus and InfluxDB for Kubernetes.

Prometheus

Prometheus is a popular pull-based time series database that pulls metrics from the target applications from a pre-configured endpoint. When running in Kubernetes, Prometheus "scrapes" metrics from target applications that have a specific pod-level annotation. The endpoint to scrape is provided by Spring Boot, under the default path of /actuator/prometheus.

Out of the box, each binder middleware configuration file defines attributes to enable metrics and their supporting properties. You can find settings in: src/kubernetes/server/server-config.yaml. The main point of interest is the following configuration section:

applicationProperties:
  stream:
    management:
      metrics:
        export:
          prometheus:
            enabled: true
      endpoints:
        web:
          exposure:
            include: 'prometheus,info,health'
    spring:
      cloud:
        streamapp:
          security:
            enabled: false
grafana-info:
  url: 'https://grafana:3000'

In this configuration, Prometheus metrics are enabled, along with the appropriate endpoints and security settings.

With Prometheus, Grafana, Spring Cloud Data Flow, and any other services as defined in the Getting Started - Kubernetes section up and running, you are ready to collect metrics.

To obtain the URL of the Grafana UI when it is deployed to Minikube, run the following command:

$ minikube service --url grafana
http://192.168.99.100:31595

In the preceding example, you can reach the Grafana dashboard at http://192.168.99.100:31595. The default credentials are as follows:

• User name: admin
• Password: password

The Grafana instance is pre-provisioned with two dashboards:

• Streams: http://192.168.99.100:31595/d/scdf-streams/streams?refresh=10s
• Applications: http://192.168.99.100:31595/d/scdf-applications/applications?refresh=10s

You can collect metrics on a per-application, per-stream basis or apply metrics collection to all deployed applications globally.

To deploy a single stream with metrics enabled, enter the following into the Spring Cloud Data Flow shell:

dataflow:>stream create metricstest --definition "time --fixed-delay=10 --time-unit=MILLISECONDS | filter --expression=payload.contains('3') | log"
dataflow:>stream deploy --name metricstest --properties "deployer.*.kubernetes.podAnnotations=prometheus.io/path:/actuator/prometheus,prometheus.io/port:8080,prometheus.io/scrape:true"

The preceding example creates a stream definition and sets the podAnnotations property on each application in the stream. The annotations applied to the pod indicate to Prometheus that it should be scraped for metrics by using the provided endpoint path and the port.

As a global setting, to deploy all streams with metrics enabled, you can append the following podAnnotations entry to the configuration in either src/kubernetes/skipper/skipper-config-rabbit.yaml (when using RabbitMQ) or src/kubernetes/skipper/skipper-config-kafka.yaml (when using Kafka):

data:
  application.yaml: |-
    spring:
      cloud:
        skipper:
          server:
            platform:
              kubernetes:
                accounts:
                  myaccountname:
                    podAnnotations: 'prometheus.io/path:/actuator/prometheus,prometheus.io/port:8080,prometheus.io/scrape:true'

All streams and their containing applications then have the appropriate pod annotations applied, which instructs Prometheus to scrape metrics. The shell command to deploy the same stream shown earlier, for example becomes the following:

dataflow:>stream create metricstest --definition "time --fixed-delay=10 --time-unit=MILLISECONDS | filter --expression=payload.contains('3') | log" --deploy

Either way, metrics are enabled. After deploying a stream, you can visit the Grafana UI and see dashboard graphs similar to those shown in the following image: