Migrating from 2.x to 3.x Stream Applications
Recently, the Spring team redesigned the pre-packaged stream applications.
These will be released in bulk using calendar-based release names, such as 2020.0.0-RC2.
We also refer to the 3.x(+) version, since stream applications included in these releases start with version 3.0.0.
Along with many important enhancements, we introduced some changes that are incompatible with the 2.x line. This means that existing Spring Cloud Data Flow stream definitions based on 2.x applications do not translate directly to 3.x applications. The following is a summary of changes that you need to be aware of when migrating 2.x Spring Cloud Stream applications to 3.x:
- Some applications have been renamed.
- In some cases, applications have been retired, replaced with equivalent functionality, or combined with other applications.
- Property prefixes have changed.
- In some cases, properties have been removed or added.
- All pre-built sources now support function composition.
We do not support pipelines that mix 2.x and 3.x applications.
They use different versions of the Spring Cloud Stream binder. Trying to combine them has been known to cause problems in some cases. If you have an existing data pipeline that requires an app that is not available in 3.x, we recommend that you do not upgrade.
Application changes
The following applications have been retired, replaced, or renamed in the 3.x line.
None in the replacement column indicates that the application will be discontinued, unless there is sufficient interest in the community.
| Retired | Replacement |
|---|---|
| counter-processor | None |
| counter-sink | analytics-sink |
| gemfire-cq-source | geode-source with cq option |
| gemfire-source | geode-source |
| gemfire-sink | geode-sink |
| groovy-filter-processor | groovy-processor |
| groovy-transform-processor | groovy-processor |
| grpc-processor | None |
| hdfs-sink | None |
| httpclient-processor | http-request-processor |
| loggregator-source | None |
| pmml-processor | None |
| pose-estimation-processor | None |
| python-http-processor | Use Polyglot recipe |
| python-jython-processor | Use Polyglot recipe |
| redis-pubsub-sink | redis-sink |
| scriptable-transform-processor | script-processor |
| sftp-dataflow-source | sftp-source with function composition |
| task-launcher-dataflow-sink | tasklauncher-sink |
| tasklaunchrequest-transform-processor | None |
| tcp-client-processor | None |
| tcp-client-source | None |
| tensorflow-processor | Custom app using tensorflow-common function library |
| trigger-source | time-source |
| triggertask-source | time-source |
| twitter-sentiment-processor | None |
| twitterstream-source | twitter-stream-source |
Property names
The following table provides a detailed comparison of property keys used for the 2.x applications that have been ported to the 3.x line with the same name. For some of these apps, all the properties are identical but have been included here for reference.
Function Composition
The 3.x stream applications are function-based, meaning that most of the logic is provided in a dependent artifact that exposes a functional interface.
This design lends itself to function composition. In fact, all of the pre-packaged sources include the necessary dependencies to support composition of functions to perform filtering, transformation, header enrichment, and the creation of task launch requests within the source app itself. Some examples are shown in the time source tests, and the sftp source tests.