Spring Cloud Data Flow
FeaturesDocumentationGetting StartedCommunityBlog
Search

Debugging Stream Applications

Applications should be runnable as a standard Java JAR, invoked via java -jar. All dependencies such as databases and messaging middleware should be available and tested manually for connection issues. Running and debugging applications is independent of deploying via SCDF and running on a platform.

Project Build Errors

Yes
No
Errors in IDE
Run using Maven CLI
Success?
Fix IDE errors
Fix code errors

Fixing errors in the IDE will be dependent on which IDE is used. By verifying the build is successful outside the IDE, this helps rule out project specific issues.

Application Startup Exceptions

Yes
Yes
No
Application Exceptions
Review stacktrace
General Coding Errors
Fix Code
Connection Exceptions
Valid configuration?
Test manually & apply any changes
Fix settings

When the application starts up, exceptions may occur due to general coding errors or connectivity issues. Review the application log file to find the exceptions and fix as appropriate. Ensure all external services are reachable, correct credentials are provided and any other required information.

General Application Debugging

Review stacktrace
Enable JDWP for debugging
Enable Spring Integration debug logging

Applications can be debugged as normal java processes as well as by utilizing verbose log output via the DEBUG logging level.

To debug your application using a debugger, add the following JVM argument when starting your application, for example:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

By adding this parameter, your application will wait for a debugger to connect on port 5005 to start debugging. This can be any port you choose that is not already in use.

Enabling DEBUG level logging can also be a useful aid. One package of interest is Spring Integration (SI): org.springframework.integration. To enable debug logging of this package, add the following JVM argument when starting your application, for example:

-Dlogging.level.org.springframework.integration=DEBUG

Data Loss

Yes
No
Yes
Yes
No
No
No
Data is lost
Middleware Accessible?
Valid configuration?
TestBinder in build test scope?
Are messages in middleware?
Move to test scope
Test manually / fix settings
Verify configuration
Verify Binder Configuration
Enable SI debug logging

Sources, processors and sinks communicate via Binder's over the chosen middleware. Ensure the middleware is available and all configuration is correct. Configuration includes credentials, queue/topic names, hosts, etc. By default, if no binder dependency is included in the project's classpath, the TestBinder will be used.

Sources

Yes
No
Yes
No
Not Producing Data
Verify External Service Accessibility
EnableBinding Annotated?
Handler Method Defined?
Enable SI logging for output channel

Sources obtain data from an input and send data to an output channel for downstream processing. Ensure the proper class level annotation @EnableBinding(Source.class) is present and the handler method is implemented. The handler method should use an output channel of Source.OUTPUT.

Processors - Input

Yes
No
Yes
No
No Input Data
EnableProcessor Annotated?
StreamListener Annotated?
Enable SI logging for input channel

Processors obtain data, manipulate and return that data for further downstream processing. Ensure the proper class level annotation @EnableBinding(Processor.class) is present and the handler method is implemented. The handler method should use an input channel of Source.INPUT.

Processors - Output

Yes
No
Yes
No
No Output Data
EnableProcessor Annotated?
SendTo Annotated?
Enable SI logging for input channel

Processors obtain data, process and return that data for further downstream processing. Ensure the proper class level annotation @EnableBinding(Processor.class) is present and the handler method is implemented. The handler method should use an output channel of Source.OUTPUT.

Sinks

Yes
No
Yes
No
Yes
No
Not Receiving Data
EnableSink Annotated?
StreamListener Annotated?
External Service Accessible?
Enable SI logging for input channel

Sinks obtain data from an input channel and for example store the that data in an external repository. Ensure the proper class level annotation @EnableBinding(Sink.class) is present and the handler method is implemented. The handler method should use an output channel of Source.INPUT.

Debugging Cloud Foundry Deployments

No
Yes
Yes
No
Verify Manifest
Verify environment variables
Services to bind created?
Errors in application log files?
Resolve exceptions
Enable SI logging for appropriate channels

If an application that runs locally, but fails when deployed to Cloud Foundry, first inspect the deployment manifest (manifest.yml) for correctness. This includes any environment variables that must be set, services to bind to and those services created. Inspect the application startup log for any exceptions to resolve.

Debugging Kubernetes Deployments

No
Yes
No
Yes
No
Yes
Yes
Verify Deployment files
Verify environment variables
External services accessible?
Pod in RUNNING state?
kubectl describe POD_NAME
Errors in event table?
Resolve errors
kubectl logs POD_NAME
Application exceptions?
Resolve errors

If an application that runs locally, but fails when deployed to Kubernetes, first inspect the deployment files for correctness. This includes any environment variables that must be set, properties for services to connect to and those services available. Describe the application pod's event table to see issues creating the pod such as image pull causes, health check failures, etc. Inspect the application startup log for any exceptions to resolve.

Containers

No
Yes
Yes
No
Yes
No
Create Container
Launch Container using 'docker run' with env variables and args
Successful?
Update container creation
Launch Container using 'docker run'
Create K8s pod spec yaml for container
Apply pod spec yaml in K8s instance
Successful?
Problem Resolved
Update Pod Spec Yaml
Success
Open Git Issue for Data Flow

As discussed before the application may run as expected from your local platform however, it still fails when launched from Spring Cloud Data Flow. This could be because of how the container is created by the tool that is being used to create the container, for example: DockerFile, Spring Boot container Plugin, Jib, etc. The chart above provides a guide on how to trouble shoot why apps may fail to launch because of container related issues.

Data Flow uses the Common application properties and the Spring Boot Common Tags to apply common metrics tags to all deployed Stream applications. Those properties often use Spring placeholders to resolve their values, for example: | management.metrics.tags.application: ${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown} | Some image containers may not be able to parse this correctly and will replace the embedded properties with empty strings which may cause stream apps to fail. If you are not using metrics you may disable these fields by setting the spring.cloud.dataflow.applicationProperties.streamResource to a non-existent file. For example: spring.cloud.dataflow.applicationProperties.streamResource=classpath:fake.yml

Edit this page on GitHub
Content

© 2013-2021 VMware, Inc. or its affiliates.

Spring Cloud Data Flow is under the Apache 2.0 license.