Migrating from 2.x to 3.x Stream Applications
Recently the Spring team redesigned the pre-packaged stream applications.
These will be released in bulk using calendar-based release names, such as 2020.0.0-M2.
We also refer to the 3.x (+) version, since stream applications included in these releases start with version 3.0.0.
Along with many important enhancements, we introduced some changes that are incompatible with the 2.x line. This means that existing Spring Cloud Data Flow stream definitions based on 2.x applications will not translate directly to 3.x applications. The following is a summary of changes that you need to be aware of when migrating 2.x Spring Cloud Stream applications to 3.x:
- Some applications have been renamed
- In some cases, applications have been retired, replaced with equivalent functionality, or combined into a single application
- Property prefixes have changed
- In some cases, properties have been removed or added
- All pre-built sources now support function composition
We do not support pipelines that mix 2.x and 3.x applications.
They use different versions of the Spring Cloud Stream binder, which has been known to cause problems in some cases. If you have an existing data pipeline that requires an app that is not available in 3.x, we recommend that you do not upgrade.
Application changes
The following applications have been retired, replaced, or renamed in the 3.x line.
None in the replacement column indicates that the application will be discontinued unless there is sufficient interest in the community.
| Retired | Replacement |
|---|---|
| counter-processor | None |
| counter-sink | analytics-sink |
| gemfire-cq-source | geode-source with cq option |
| gemfire-source | geode-source |
| gemfire-sink | geode-sink |
| groovy-filter-processor | groovy-processor |
| groovy-transform-processor | groovy-processor |
| grpc-processor | None |
| hdfs-sink | None |
| httpclient-processor | http-request-processor |
| loggregator-source | None |
| pmml-processor | None |
| pose-estimation-processor | None |
| python-http-processor | Use Polyglot recipe |
| python-jython-processor | Use Polyglot recipe |
| redis-pubsub-sink | redis-sink |
| scriptable-transform-processor | script-processor |
| sftp-dataflow-source | sftp-source with function composition |
| task-launcher-dataflow-sink | tasklauncher-sink |
| tasklaunchrequest-transform-processor | None |
| tcp-client-processor | None |
| tcp-client-source | None |
| tensorflow-processor | Custom app using tensorflow-common function library |
| trigger-source | time-source |
| triggertask-source | time-source |
| twitter-sentiment-processor | None |
| twitterstream-source | twitter-stream-source |
Property names
This following table provides a detailed comparison of property keys used for the 2.x applications which have been ported to the 3.x line, using the same name. For some of these apps, all the properties are identical, but have been included here for reference.
Function Composition
The 3.x stream applications are function-based, meaning that most of the logic is provided in a dependent artifact that exposes a functional interface.
This design lends itself to function-composition. In fact, all of the pre-packaged sources include the necessary dependencies to support composition with useful functions to perform filtering, transformation, header enrichment, and the creation of task launch requests within the source app itself. Some examples are shown in the time-source tests, and the sftp-source tests.