Debugging Streams in Data Flow

If a stream fails to deploy:

  • Ensure that the latest GA of a particular release version is being used
  • Ensure that your platform of choice meets at least the minimum supported version
  • Before deploying stream applications in SCDF, ensure that the application has been successfully run as a standalone application. See Debugging Stream Applications before continuing.

Skipper is responsible for deploying streams. Stream deployment failure messages can typically be found in the Skipper application logs.

The rest of this guide is divided in to sections for each platform:

  • Local
  • Cloud Foundry
  • Kubernetes

Local

This section describes how to debug common deployment problems in a local environment.

The following diagram shows a typical debugging process for deployment problems in a local environment:

No
Yes
Yes
No
Yes
No
Start Locally?
Exceptions in SCDF or Skipper logs?
Resolve exceptions
Exceptions in app logs?
Resolve exceptions
Raise log levels
Enable debug

You can inspect application log files on a per-application basis. To aggregate logs from all applications into one, you can set the inheritLogging=true deployer property. See "Is it possible to aggregate Local deployments into a single log?" for more information and "How do I enable DEBUG logs for platform deployments?" for how to enable more log output.

You can debug applications with JDWP by setting the debugPort deployer property. See "How do I remote debug deployed applications?" for more information.

Docker Compose - Startup

The following diagram shows a typical debugging process for Docker compose startup problems in a local environment:

No
Yes
No
DATAFLOW_VERSION defined?
SKIPPER_VERSION defined?

The DATAFLOW_VERSION and SKIPPER_VERSION environment variables must be available in the current terminal environment through export or by prefixing the docker-compose command. See Starting Docker Compose for more information.

Docker Compose - Runtime

The following diagram shows a typical debugging process for Docker compose runtime problems in a local environment:

Yes
No
Yes
Low Resources?
Increase Docker Resources
Exceptions in app logs?
Resolve exceptions

By default, the amount of memory allocated to Docker may be too low. The recommended amount of memory to allocate is 8GB. The docker stats command can provide useful information about resource usage. If applications fail to deploy due to resource constraints, increase resource allocations. See the Docker documentation for your platform.

As streams are deployed through Skipper, applications that are part of that stream are launched as Java processes on the Skipper container. For every part of a stream, an application is deployed. The overall resource allocation (memory, CPU, etc) provided to Docker should account for the number of deployed applications. See Viewing Stream Logs for more information on viewing log files.

Cloud Foundry

This section describes how to debug common deployment problems on Cloud Foundry.

Startup Failures

The following diagram shows a typical debugging process for startup failures on Cloud Foundry:

No
Yes
No
Verify Manifest
Verify environment variables
Service instances running?
SCDF and Skipper Running?
View logs and resolve errors

Application Failures

The following diagram shows a typical debugging process for application failures on Cloud Foundry:

No
No
No
Applications Started?
Errors in SCDF logs?
Errors in Skipper logs?
Errors in app logs?
Resolve

When debugging deployment issues, raising deployer and Cloud Foundry related log levels may help. See "How do I enable DEBUG logs for platform deployments?" for more information.

Kubernetes

This section describes how to debug common deployment problems on Kubernetes.

Distributed Deployment Files

The following diagram shows a typical debugging process for distributed deployment files on Kubernetes:

No
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
All files applied including RBAC?
External Services Running
Exceptions in SCDF pod logs?
Exceptions in Skipper pod logs?
Exceptions in app pod logs?
Resolve

Helm Chart

The following diagram shows a typical debugging process for the Helm chart on Kubernetes:

No
Yes
No
Yes
Yes
Yes
Yes
Chart found?
helm repo update
Expected Services Running?
Exceptions in SCDF pod logs?
Exceptions in Skipper pod logs?
Resolve

General

The following diagram shows a typical debugging process for general problems on Kubernetes:

Yes
Yes
Errors in pod events table?
Resolve
Exceptions in app pod logs?

When describing a pod, the events table section provides useful information when debugging and can be invoked by the following:

kubectl describe po/pod_name

For example, the events from a successfully deployed time application would look similar to the following:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m    default-scheduler  Successfully assigned default/ticktock-time-v16-869947b6b9-brfc4 to minikube
  Normal  Pulled     2m    kubelet, minikube  Container image "springcloudstream/time-source-rabbit:2.1.0.RELEASE" already present on machine
  Normal  Created    2m    kubelet, minikube  Created container
  Normal  Started    2m    kubelet, minikube  Started container

Application logs can be tailed to watch logs as they arrive with the following command (where pod_name is the name of your pod):

kubectl logs -f po/pod_name

Troubleshooting Help

If none of these troubleshooting techniques helped and if you still need help, reach out to us on StackOverflow with the relevant details (see: Wiki). We actively monitor the forum threads.