Customizing Docker Compose

This section covers how to customize the Docker Compose installation by editing the docker-compose.yml file.

The Docker Compose file uses Apache Kafka for the messaging middleware and Prometheus for monitoring. If you want to use RabbitMQ or InfluxDB instead, this guide shows you the changes to make in the docker compose file.

Also, when doing development of custom applications, you need to enable the Docker container that runs the Data Flow Server to see your local file system. This guide shows you how to do that as well.

Using RabbitMQ Instead of Kafka

You can use RabbitMQ rather than Kafka for communication. To do so:

1. Delete the following configuration under the services: section:

   Copy

   kafka:
    image: confluentinc/cp-kafka:5.2.1
    ...
   zookeeper:
    image: confluentinc/cp-zookeeper:5.2.1
    ....

2. Insert the following:

   Copy

   rabbitmq:
    image: rabbitmq:3.7
    expose:
      - '5672'

3. In the dataflow-server services configuration block, add the following environment entry:

   Copy

   - spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.host=rabbitmq

4. Delete the following:

   Copy

   depends_on:
    - kafka

5. Insert the following:

   Copy

   depends_on:
    - rabbitmq

6. Modify the app-import service definition command attribute to replace https://dataflow.spring.io/kafka-maven-latest with https://dataflow.spring.io/rabbitmq-maven-latest.

Using InfluxDB Instead of Prometheus

You can use InfluxDB rather than Prometheus to monitor time-series database. To do so:

1. Delete the following configuration under the services section:

   Copy

   prometheus:
    image: springcloud/spring-cloud-dataflow-prometheus-local:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set! Use 'export DATAFLOW_VERSION=dataflow-version'}
    container_name: 'prometheus'
    volumes:
      - 'scdf-targets:/etc/prometheus/'
    ports:
      - '9090:9090'
    depends_on:
      - service-discovery
   
   service-discovery:
    image: springcloud/spring-cloud-dataflow-prometheus-service-discovery:0.0.3
    container_name: 'service-discovery'
    volumes:
      - 'scdf-targets:/tmp/scdf-targets/'
    expose:
      - '8181'
    ports:
      - '8181:8181'
    environment:
      - metrics.prometheus.target.refresh.cron=0/20 * * * * *
      - metrics.prometheus.target.discovery.url=http://localhost:9393/runtime/apps
      - metrics.prometheus.target.file.path=/tmp/targets.json
    depends_on:
      - dataflow-server

2. Insert the following:

   Copy

   influxdb:
    image: influxdb:1.7.4
    container_name: 'influxdb'
    ports:
      - '8086:8086'

3. In the dataflow-server services configuration block, delete the following environment entries:

   Copy

   - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.prometheus.enabled=true
   - spring.cloud.dataflow.applicationProperties.stream.spring.cloud.streamapp.security.enabled=false
   - spring.cloud.dataflow.applicationProperties.stream.management.endpoints.web.exposure.include=prometheus,info,health

4. Insert the following:

   Copy

   - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.enabled=true
   - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.db=myinfluxdb
   - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.uri=http://influxdb:8086

5. Modify the grafana service definition image attribute to replace spring-cloud-dataflow-grafana-prometheus with spring-cloud-dataflow-grafana-influxdb.

Accessing the Host File System

If you develop custom applications on your local machine, you need to register them with Spring Cloud Data Flow. Since Spring Cloud Data Flow runs inside of a Docker container, you need to configure the Docker container to access to your local file system to resolve the registration reference. Also in order to deploy those custom applications, the Skipper Server in turn needs to access them from within its own Docker container using exactly the same path definitions configured in the Data Flow server configuration.

You can enable local disk access by mounting the host folders to the dataflow-server and skipper-server containers. For example, if the my-app.jar is in the /thing1/thing2/apps folder on your host machine, add the following volumes block to the dataflow-server and skipper-server service definitions:

dataflow-server:
  image: springcloud/spring-cloud-dataflow-server:${DATAFLOW_VERSION}
  # ...
  volumes:
    - /thing1/thing2/apps:/root/apps

and mount exactly the same volume to the skipper-server service definition:

skipper-server:
  image: springcloud/spring-cloud-skipper-server:${SKIPPER_VERSION:?SKIPPER_VERSION is not set!}
  # ...
  volumes:
    - /thing1/thing2/apps:/root/apps

This configuration provides access to the /thing1/thing2/apps directory that contains your my-app.jar from within dataflow-server and skipper-server containers' /root/apps/ folder. See the compose-file reference for for further configuration details.

Once you mount the host folder, you can register the app starters (from /root/apps), with the Data Flow Shell or Dashboard by using the file:// URI schema. The following example shows how to do so:

app register --type source --name my-app --uri file://root/apps/my-app-1.0.0.RELEASE.jar

Maven Local Repository Mounting

To access the host’s local maven repository from Spring Cloud Data Flow you must mount the host maven local repository to a dataflow-server and skipper-server volume called /root/.m2/. The Maven Local Repository location defaults to ~/.m2 for OSX and Linux and C:\Documents and Settings\{your-username}\.m2 for Windows.

For MacOS or Linux host machines, this looks like the following listing:

dataflow-server:
.........
  volumes:
    - ~/.m2:/root/.m2

 skipper-server:
 ........
   volumes:
    - ~/.m2:/root/.m2

Now you can use the maven:// URI schema and Maven coordinates to resolve jars installed in the host’s maven repository, as the following example shows:

app register --type processor --name pose-estimation --uri maven://org.springframework.cloud.stream.app:pose-estimation-processor-rabbit:2.0.2.BUILD-SNAPSHOT --metadata-uri maven://org.springframework.cloud.stream.app:pose-estimation-processor-rabbit:jar:metadata:2.0.2.BUILD-SNAPSHOT

This approach lets you use applications that are built and installed on the host machine (for example, by using mvn clean install) directly with the Spring Cloud Data Flow server.

You can also pre-register the apps directly in the docker-compose instance. For every pre-registered app starer, add an additional wget statement to the app-import block configuration, as the following example shows:

app-import:
  image: alpine:3.7
  command: >
    /bin/sh -c "
      ....
      wget -qO- 'https://dataflow-server:9393/apps/source/my-app' --post-data='uri=file:/root/apps/my-app.jar&metadata-uri=file:/root/apps/my-app-metadata.jar';
      echo 'My custom apps imported'"

See the Data Flow REST API for further details.