Migrating from 2.x to 3.x Stream Applications

Recently the Spring team redesigned the pre-packaged stream applications. These will be released in bulk using calendar-based release names, such as 2020.0.0-M2. We also refer to the 3.x (+) version, since stream applications included in these releases start with version 3.0.0. Along with many important enhancements, we introduced some changes that are incompatible with the 2.x line. This means that existing Spring Cloud Data Flow stream definitions based on 2.x applications will not translate directly to 3.x applications. The following is a summary of changes that you need to be aware of when migrating 2.x Spring Cloud Stream applications to 3.x:

  • Some applications have been renamed
  • In some cases, applications have been retired, replaced with equivalent functionality, or combined into a single application
  • Property prefixes have changed
  • In some cases, properties have been removed or added
  • All pre-built sources now support function composition

We do not support pipelines that mix 2.x and 3.x applications. They use different versions of the Spring Cloud Stream binder, which has been known to cause problems in some cases. If you have an existing data pipeline that requires an app that is not available in 3.x, we recommend that you do not upgrade.

Application changes

The following applications have been retired, replaced, or renamed in the 3.x line. None in the replacement column indicates that the application will be discontinued unless there is sufficient interest in the community.

Retired Replacement
counter-processor None
counter-sink analytics-sink
gemfire-cq-source geode-source with cq option
gemfire-source geode-source
gemfire-sink geode-sink
groovy-filter-processor groovy-processor
groovy-transform-processor groovy-processor
grpc-processor None
hdfs-sink None
httpclient-processor http-request-processor
loggregator-source None
pmml-processor None
pose-estimation-processor None
python-http-processor Use Polyglot recipe
python-jython-processor Use Polyglot recipe
redis-pubsub-sink redis-sink
scriptable-transform-processor script-processor
sftp-dataflow-source sftp-source with function composition
task-launcher-dataflow-sink tasklauncher-sink
tasklaunchrequest-transform-processor None
tcp-client-processor None
tcp-client-source None
tensorflow-processor Custom app using tensorflow-common function library
trigger-source time-source
triggertask-source time-source
twitter-sentiment-processor None
twitterstream-source twitter-stream-source

Property names

This following table provides a detailed comparison of property keys used for the 2.x applications which have been ported to the 3.x line, using the same name. For some of these apps, all the properties are identical, but have been included here for reference.

Sources Processors Sinks
cdc-debezium-source aggregator-processor cassandra-sink
file-source filter-processor file-sink
ftp-source header-enricher-processor ftp-sink
http-source image-recognition-processor jdbc-sink
jdbc-source object-detection-processor log-sink
jms-source splitter-processor mondodb-sink
load-generator-source mqtt-sink
mail-source pgpcopy-sink
mongodb-source rabbit-sink
mqtt-source router-sink
rabbit-source s3-sink
s3-source sftp-sink
sftp-source tcp-sink
syslog-source throughput-sink
tcp-source websocket-sink
time-source

Function Composition

The 3.x stream applications are function-based, meaning that most of the logic is provided in a dependent artifact that exposes a functional interface. This design lends itself to function-composition. In fact, all of the pre-packaged sources include the necessary dependencies to support composition with useful functions to perform filtering, transformation, header enrichment, and the creation of task launch requests within the source app itself. Some examples are shown in the time-source tests, and the sftp-source tests.