Installing by Using Docker Compose

Spring Cloud Data Flow provides a Docker Compose file to let you quickly bring up Spring Cloud Data Flow, Skipper, and MySQL, Apache Kafka. Follow the customizing guide to switch to RabbitMQ or to a different database. Prometheus or InfluxDB, along with Grafana can be configured for monitoring the stream and task applications.

Upgrade to latest version of Docker

We recommended that you upgrade to the latest version of Docker before running the docker-compose command. We have tested with Docker Engine version 18.09.2.

Downloading the Docker Compose File

To download the Spring Cloud Data Flow Server Docker Compose file, run the following command:

wget https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/master/spring-cloud-dataflow-server/docker-compose.yml

If wget is unavailable, you can use curl or another platform-specific utility. Click here to download the docker-compose.yml file.

The Docker Compose file starts instances of the following products:

  1. Spring Cloud Data Flow Server
  2. Spring Cloud Skipper Server
  3. MySQL
  4. Apache Kafka

Starting Docker Compose

In the directory where you downloaded docker-compose.yml, start the system, by running the following commands:

export DATAFLOW_VERSION=2.3.0.BUILD-SNAPSHOT
export SKIPPER_VERSION=2.2.0.BUILD-SNAPSHOT
docker-compose up

If you use Windows, you can define environment variables by using the set command. To start the system on Windows, enter the following commands:

C:\ set DATAFLOW_VERSION=2.3.0.BUILD-SNAPSHOT
C:\ set SKIPPER_VERSION=2.2.0.BUILD-SNAPSHOT
C:\ docker-compose up

Spring Cloud Data Flow is ready for use once the docker-compose command stops emitting log messages.

The docker-compose.yml file defines DATAFLOW_VERSION and SKIPPER_VERSION variables, so that you can easily change those values. The preceding commands first set the DATAFLOW_VERSION and SKIPPER_VERSION to use and then start docker-compose.

You can also use a shorthand version that exposes only the DATAFLOW_VERSION and SKIPPER_VERSION variables to the docker-compose process (rather than setting it in the environment), as follows:

DATAFLOW_VERSION=2.3.0.BUILD-SNAPSHOT SKIPPER_VERSION=2.2.0.BUILD-SNAPSHOT docker-compose up

By default, the latest GA releases of Stream and Task applications are imported automatically. Use the STREAM_APPS_URI and TASK_APPS_URI to alter the default imported Stream and Task applications. For example to override the default maven apps (https://dataflow.spring.io/kafka-maven-latest) with docker based distribution (https://dataflow.spring.io/kafka-docker-latest) run: STREAM_APPS_URI=https://dataflow.spring.io/kafka-docker-latest docker-compose up

By default, Docker Compose uses locally available images. For example, when using the latest tag, you can run docker-compose pull prior to docker-compose up to ensure the latest image is downloaded.

Some stream applications may open a port (for example, http --server.port=). By default, a port range of 9000-9010 is exposed from the container to the host. If you need to change this range, you can modify the ports block of the dataflow-server service in thedocker-compose.yml file.

Now that docker compose is up, you can access the Spring Cloud Data Flow Dashboard. In your browser, navigate to the Spring Cloud Data Flow Dashboard URL.

Stopping Spring Cloud Data Flow

When you want to shut down Spring Cloud Data Flow, you can use the docker-compose down command. To do so:

  1. Open a new terminal window.

  2. Change directory to the directory in which you started (where the docker-compose.yml file is located).

  3. Run the following command:

DATAFLOW_VERSION=2.3.0.BUILD-SNAPSHOT SKIPPER_VERSION=2.2.0.BUILD-SNAPSHOT docker-compose down

You need to specify the DATAFLOW_VERSION and the SKIPPER_VERSION environment variables if you are running the command in a separate terminal window from which you started Docker Compose. The export commands you used earlier set the variables for only that terminal window, so those values are not found in the new terminal window.

If all else fails, you can shut down Docker Compose by pressing Ctrl+C.

Shell

For convenience and as an alternative to using the Spring Cloud Data Flow Dashboard, you can use the Spring Cloud Data Flow Shell as an alternative to the UI. The shell supports tab completion for commands and stream and task application configuration properties.

If you have started Spring Cloud Data Flow by using Docker Compose, the shell is also included in the springcloud/spring-cloud-dataflow-server Docker image. To use it, open another console window and type the following:

docker exec -it dataflow-server java -jar shell.jar

If you have started the Data Flow server with java -jar, you can download and start the shell. To download the Spring Cloud Data Flow Shell application, run the following command:

wget https://repo.spring.io/snapshot/org/springframework/cloud/spring-cloud-dataflow-shell/2.3.0.BUILD-SNAPSHOT/spring-cloud-dataflow-shell-2.3.0.BUILD-SNAPSHOT.jar

Monitoring

By default, the Data Flow docker-compose does not enable the monitoring functionality for Stream and Task applications. To enable that, follow the Monitoring with Prometheus and Grafana or Monitoring with InfluxDB and Grafana sections for how to configure Prometheus or InfluxDB based monitoring set up for Spring Cloud Data Flow.

To learn more about the monitoring experience in Spring Cloud Data Flow with Prometheus and InfluxDB, see the Stream Monitoring feature guide.