Helm Installation
The Helm project has ended support for Helm 2 in November of 2020. As of Spring Cloud Data Flow 2.7.0 the chart will be based on Helm 3, dropping support for Helm 2.
Migration steps from Helm 2 to Helm 3 are required. In preparation for the migration, it is advised to read the Helm v2 to v3 Migration Guide for more information. Additionally, some helpful tips on data migration and upgrades can be found in the post migration issues article.
As of Spring Cloud Data Flow 2.6.1, the Bitnami team maintains the Helm chart. To report bugs and/or feature requests please do so using the Bitnami Issue Tracker.
Spring Cloud Data Flow offers a Helm Chart for deploying the Spring Cloud Data Flow server and its required services to a Kubernetes Cluster.
The following sections cover how to initialize Helm
and install Spring Cloud Data Flow on a Kubernetes cluster.
If using Minikube, see Setting Minikube Resources for details on CPU and RAM resource requirements.
Installing the Spring Cloud Data Flow Server and Required Services
It is important to review the following documentation and adjust any parameter customizations that have been made for your environment or how they may differ from the legacy official Helm chart. Value names, defaults, and so on may have changed during the Bitnami chart migration. More information can be found in the Parameter tables, Upgrading, and Notable Changes sections.
Spring Cloud Data Flow
Spring Cloud Data Flow is a microservices-based Streaming and Batch data processing pipeline in Cloud Foundry and Kubernetes.
TL;DR
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-release bitnami/spring-cloud-dataflow
Introduction
This chart bootstraps a Spring Cloud Data Flow deployment on a Kubernetes cluster using the Helm package manager.
Bitnami charts can be used with Kubeapps for deployment and management of Helm Charts in clusters.
Prerequisites
- Kubernetes 1.12+
- Helm 3.0-beta3+
- PV provisioner support in the underlying infrastructure
Installing the Chart
To install the chart with the release name my-release
:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-release bitnami/spring-cloud-dataflow
These commands deploy Spring Cloud Data Flow on the Kubernetes cluster with the default configuration. The parameters section lists the parameters that can be configured during installation.
Tip: List all releases using
helm list
Uninstalling the Chart
To uninstall/delete the my-release
chart:
helm uninstall my-release
Parameters
The following tables lists the configurable parameters of the Spring Cloud Data Flow chart and their default values per section/component:
Global parameters
Parameter | Description | Default |
---|---|---|
global.imageRegistry |
Global Docker image registry | nil |
global.imagePullSecrets |
Global Docker registry secret names as an array | [] (does not add image pull secrets to deployed pods) |
global.storageClass |
Global storage class for dynamic provisioning | nil |
Common parameters
Parameter | Description | Default |
---|---|---|
nameOverride |
String to partially override scdf.fullname | nil |
fullnameOverride |
String to fully override scdf.fullname | nil |
clusterDomain |
Default Kubernetes cluster domain | cluster.local |
deployer.resources.limits |
Streaming applications resource limits | { cpu: "500m", memory: "1024Mi" } |
deployer.resources.requests |
Streaming applications resource requests | {} |
deployer.resources.readinessProbe |
Streaming applications readiness probes requests | Check values.yaml file |
deployer.resources.livenessProbe |
Streaming applications liveness probes requests | Check values.yaml file |
deployer.nodeSelector |
Streaming applications nodeSelector | "" |
deployer.tolerations |
Streaming applications tolerations | {} |
deployer.volumeMounts |
Streaming applications extra volume mounts | {} |
deployer.volumes |
Streaming applications extra volumes | {} |
deployer.environmentVariables |
Streaming applications environment variables | "" |
deployer.podSecurityContext |
Streaming applications Security Context. | {runAsUser: 1001} |
Dataflow Server parameters
Parameter | Description | Default |
---|---|---|
server.image.registry |
Spring Cloud Dataflow image registry | docker.io |
server.image.repository |
Spring Cloud Dataflow image name | bitnami/spring-cloud-dataflow |
server.image.tag |
Spring Cloud Dataflow image tag | {TAG_NAME} |
server.image.pullPolicy |
Spring Cloud Dataflow image pull policy | IfNotPresent |
server.image.pullSecrets |
Specify docker-registry secret names as an array | [] (does not add image pull secrets to deployed pods) |
server.composedTaskRunner.image.registry |
Spring Cloud Dataflow Composed Task Runner image registry | docker.io |
server.composedTaskRunner.image.repository |
Spring Cloud Dataflow Composed Task Runner image name | bitnami/spring-cloud-dataflow-composed-task-runner |
server.composedTaskRunner.image.tag |
Spring Cloud Dataflow Composed Task Runner image tag | {TAG_NAME} |
server.configuration.streamingEnabled |
Enables or disables streaming data processing | true |
server.configuration.batchEnabled |
Enables or disables bath data (tasks and schedules) processing | true |
server.configuration.accountName |
The name of the account to configure for the Kubernetes platform | default |
server.configuration.trustK8sCerts |
Trust K8s certificates when querying the Kubernetes API | false |
server.configuration.containerRegistries |
Container registries configuration | {} (check values.yaml for more information) |
server.configuration.metricsDashboard |
Endpoint to the metricsDashboard instance | nil |
server.existingConfigmap |
Name of existing ConfigMap with Dataflow server configuration | nil |
server.extraEnvVars |
Extra environment variables to be set on Dataflow server container | {} |
server.extraEnvVarsCM |
Name of existing ConfigMap containing extra env vars | nil |
server.extraEnvVarsSecret |
Name of existing Secret containing extra env vars | nil |
server.replicaCount |
Number of Dataflow server replicas to deploy | 1 |
server.strategyType |
Deployment Strategy Type | RollingUpdate |
server.podAffinityPreset |
Dataflow server pod affinity preset. Ignored if server.affinity is set. Allowed values: soft or hard |
"" |
server.podAntiAffinityPreset |
Dataflow server pod anti-affinity preset. Ignored if server.affinity is set. Allowed values: soft or hard |
soft |
server.nodeAffinityPreset.type |
Dataflow server node affinity preset type. Ignored if server.affinity is set. Allowed values: soft or hard |
"" |
server.nodeAffinityPreset.key |
Dataflow server node label key to match Ignored if server.affinity is set. |
"" |
server.nodeAffinityPreset.values |
Dataflow server node label values to match. Ignored if server.affinity is set. |
[] |
server.affinity |
Dataflow server affinity for pod assignment | {} (evaluated as a template) |
server.nodeSelector |
Dataflow server node labels for pod assignment | {} (evaluated as a template) |
server.tolerations |
Dataflow server tolerations for pod assignment | [] (evaluated as a template) |
server.priorityClassName |
Controller priorityClassName | nil |
server.podSecurityContext |
Dataflow server pods' Security Context | { fsGroup: "1001" } |
server.containerSecurityContext |
Dataflow server containers' Security Context | { runAsUser: "1001" } |
server.resources.limits |
The resources limits for the Dataflow server container | {} |
server.resources.requests |
The requested resources for the Dataflow server container | {} |
server.podAnnotations |
Annotations for Dataflow server pods | {} |
server.livenessProbe |
Liveness probe configuration for Dataflow server | Check values.yaml file |
server.readinessProbe |
Readiness probe configuration for Dataflow server | Check values.yaml file |
server.customLivenessProbe |
Override default liveness probe | nil |
server.customReadinessProbe |
Override default readiness probe | nil |
server.service.type |
Kubernetes service type | ClusterIP |
server.service.port |
Service HTTP port | 8080 |
server.service.nodePort |
Service HTTP node port | nil |
server.service.clusterIP |
Dataflow server service clusterIP IP | None |
server.service.externalTrafficPolicy |
Enable client source IP preservation | Cluster |
server.service.loadBalancerIP |
loadBalancerIP if service type is LoadBalancer |
nil |
server.service.loadBalancerSourceRanges |
Address that are allowed when service is LoadBalancer | [] |
server.service.annotations |
Annotations for Dataflow server service | {} |
server.ingress.enabled |
Enable ingress controller resource | false |
server.ingress.certManager |
Add annotations for cert-manager | false |
server.ingress.hostname |
Default host for the ingress resource | dataflow.local |
server.ingress.annotations |
Ingress annotations | [] |
server.ingress.extraHosts[0].name |
Additional hostnames to be covered | nil |
server.ingress.extraHosts[0].path |
Additional hostnames to be covered | nil |
server.ingress.extraTls[0].hosts[0] |
TLS configuration for additional hostnames to be covered | nil |
server.ingress.extraTls[0].secretName |
TLS configuration for additional hostnames to be covered | nil |
server.ingress.secrets[0].name |
TLS Secret Name | nil |
server.ingress.secrets[0].certificate |
TLS Secret Certificate | nil |
server.ingress.secrets[0].key |
TLS Secret Key | nil |
server.initContainers |
Add additional init containers to the Dataflow server pods | {} (evaluated as a template) |
server.sidecars |
Add additional sidecar containers to the Dataflow server pods | {} (evaluated as a template) |
server.pdb.create |
Enable/disable a Pod Disruption Budget creation | false |
server.pdb.minAvailable |
Minimum number/percentage of pods that should remain scheduled | 1 |
server.pdb.maxUnavailable |
Maximum number/percentage of pods that may be made unavailable | nil |
server.autoscaling.enabled |
Enable autoscaling for Dataflow server | false |
server.autoscaling.minReplicas |
Minimum number of Dataflow server replicas | nil |
server.autoscaling.maxReplicas |
Maximum number of Dataflow server replicas | nil |
server.autoscaling.targetCPU |
Target CPU utilization percentage | nil |
server.autoscaling.targetMemory |
Target Memory utilization percentage | nil |
server.jdwp.enabled |
Enable Java Debug Wire Protocol (JDWP) | false |
server.jdwp.port |
JDWP TCP port | 5005 |
server.extraVolumes |
Extra Volumes to be set on the Dataflow Server Pod | nil |
server.extraVolumeMounts |
Extra VolumeMounts to be set on the Dataflow Container | nil |
Dataflow Skipper parameters
Parameter | Description | Default |
---|---|---|
skipper.enabled |
Enable Spring Cloud Skipper component | true |
skipper.image.registry |
Spring Cloud Skipper image registry | docker.io |
skipper.image.repository |
Spring Cloud Skipper image name | bitnami/spring-cloud-dataflow |
skipper.image.tag |
Spring Cloud Skipper image tag | {TAG_NAME} |
skipper.image.pullPolicy |
Spring Cloud Skipper image pull policy | IfNotPresent |
skipper.image.pullSecrets |
Specify docker-registry secret names as an array | [] (does not add image pull secrets to deployed pods) |
skipper.configuration.accountName |
The name of the account to configure for the Kubernetes platform | default |
skipper.configuration.trustK8sCerts |
Trust K8s certificates when querying the Kubernetes API | false |
skipper.existingConfigmap |
Name of existing ConfigMap with Skipper server configuration | nil |
skipper.extraEnvVars |
Extra environment variables to be set on Skipper server container | {} |
skipper.extraEnvVarsCM |
Name of existing ConfigMap containing extra env vars | nil |
skipper.extraEnvVarsSecret |
Name of existing Secret containing extra env vars | nil |
skipper.replicaCount |
Number of Skipper server replicas to deploy | 1 |
skipper.strategyType |
Deployment Strategy Type | RollingUpdate |
skipper.podAffinityPreset |
Skipper pod affinity preset. Ignored if skipper.affinity is set. Allowed values: soft or hard |
"" |
skipper.podAntiAffinityPreset |
Skipper pod anti-affinity preset. Ignored if skipper.affinity is set. Allowed values: soft or hard |
soft |
skipper.nodeAffinityPreset.type |
Skipper node affinity preset type. Ignored if skipper.affinity is set. Allowed values: soft or hard |
"" |
skipper.nodeAffinityPreset.key |
Skipper node label key to match Ignored if skipper.affinity is set. |
"" |
skipper.nodeAffinityPreset.values |
Skipper node label values to match. Ignored if skipper.affinity is set. |
[] |
skipper.affinity |
Skipper affinity for pod assignment | {} (evaluated as a template) |
skipper.nodeSelector |
Skipper node labels for pod assignment | {} (evaluated as a template) |
skipper.tolerations |
Skipper tolerations for pod assignment | [] (evaluated as a template) |
skipper.priorityClassName |
Controller priorityClassName | nil |
skipper.podSecurityContext |
Skipper server pods' Security Context | { fsGroup: "1001" } |
skipper.containerSecurityContext |
Skipper server containers' Security Context | { runAsUser: "1001" } |
skipper.resources.limits |
The resources limits for the Skipper server container | {} |
skipper.resources.requests |
The requested resources for the Skipper server container | {} |
skipper.podAnnotations |
Annotations for Skipper server pods | {} |
skipper.livenessProbe |
Liveness probe configuration for Skipper server | Check values.yaml file |
skipper.readinessProbe |
Readiness probe configuration for Skipper server | Check values.yaml file |
skipper.customLivenessProbe |
Override default liveness probe | nil |
skipper.customReadinessProbe |
Override default readiness probe | nil |
skipper.service.type |
Kubernetes service type | ClusterIP |
skipper.service.port |
Service HTTP port | 8080 |
skipper.service.nodePort |
Service HTTP node port | nil |
skipper.service.clusterIP |
Skipper server service clusterIP IP | None |
skipper.service.externalTrafficPolicy |
Enable client source IP preservation | Cluster |
skipper.service.loadBalancerIP |
loadBalancerIP if service type is LoadBalancer |
nil |
skipper.service.loadBalancerSourceRanges |
Address that are allowed when service is LoadBalancer | [] |
skipper.service.annotations |
Annotations for Skipper server service | {} |
skipper.initContainers |
Add additional init containers to the Skipper pods | {} (evaluated as a template) |
skipper.sidecars |
Add additional sidecar containers to the Skipper pods | {} (evaluated as a template) |
skipper.pdb.create |
Enable/disable a Pod Disruption Budget creation | false |
skipper.pdb.minAvailable |
Minimum number/percentage of pods that should remain scheduled | 1 |
skipper.pdb.maxUnavailable |
Maximum number/percentage of pods that may be made unavailable | nil |
skipper.autoscaling.enabled |
Enable autoscaling for Skipper server | false |
skipper.autoscaling.minReplicas |
Minimum number of Skipper server replicas | nil |
skipper.autoscaling.maxReplicas |
Maximum number of Skipper server replicas | nil |
skipper.autoscaling.targetCPU |
Target CPU utilization percentage | nil |
skipper.autoscaling.targetMemory |
Target Memory utilization percentage | nil |
skipper.jdwp.enabled |
Enable Java Debug Wire Protocol (JDWP) | false |
skipper.jdwp.port |
JDWP TCP port | 5005 |
skipper.extraVolumes |
Extra Volumes to be set on the Skipper Pod | nil |
skipper.extraVolumeMounts |
Extra VolumeMounts to be set on the Skipper Container | nil |
externalSkipper.host |
Host of a external Skipper Server | localhost |
externalSkipper.port |
External Skipper Server port number | 7577 |
RBAC parameters
Parameter | Description | Default |
---|---|---|
serviceAccount.create |
Enable the creation of a ServiceAccount for Dataflow server and Skipper server pods | true |
serviceAccount.name |
Name of the created serviceAccount | Generated using the scdf.fullname template |
rbac.create |
Weather to create & use RBAC resources or not | true |
Metrics parameters
Parameter | Description | Default |
---|---|---|
metrics.metrics |
Enable the export of Prometheus metrics | false |
metrics.image.registry |
Prometheus Rsocket Proxy image registry | docker.io |
metrics.image.repository |
Prometheus Rsocket Proxy image name | bitnami/prometheus-rsocket-proxy |
metrics.image.tag |
Prometheus Rsocket Proxy image tag | {TAG_NAME} |
metrics.image.pullPolicy |
Prometheus Rsocket Proxy image pull policy | IfNotPresent |
metrics.image.pullSecrets |
Specify docker-registry secret names as an array | [] (does not add image pull secrets to deployed pods) |
metrics.kafka.service.httpPort |
Prometheus Rsocket Proxy HTTP port | 8080 |
metrics.kafka.service.rsocketPort |
Prometheus Rsocket Proxy Rsocket port | 8080 |
metrics.kafka.service.annotations |
Annotations for Prometheus Rsocket Proxy service | Check values.yaml file |
metrics.serviceMonitor.enabled |
if true , creates a Prometheus Operator ServiceMonitor (also requires metrics.enabled to be true ) |
false |
metrics.serviceMonitor.namespace |
Namespace in which Prometheus is running | nil |
metrics.serviceMonitor.interval |
Interval at which metrics should be scraped. | nil (Prometheus Operator default value) |
metrics.serviceMonitor.scrapeTimeout |
Timeout after which the scrape is ended | nil (Prometheus Operator default value) |
Init Container parameters
Parameter | Description | Default |
---|---|---|
waitForBackends.enabled |
Wait for the database and other services (such as Kafka or RabbitMQ) used when enabling streaming | true |
waitForBackends.image.registry |
Init container wait-for-backend image registry | docker.io |
waitForBackends.image.repository |
Init container wait-for-backend image name | bitnami/kubectl |
waitForBackends.image.tag |
Init container wait-for-backend image tag | {TAG_NAME} |
waitForBackends.image.pullPolicy |
Init container wait-for-backend image pull policy | IfNotPresent |
waitForBackends.image.pullSecrets |
Specify docker-registry secret names as an array | [] (does not add image pull secrets to deployed pods) |
waitForBackends.resources.limits |
Init container wait-for-backend resource limits | {} |
waitForBackends.resources.requests |
Init container wait-for-backend resource requests | {} |
Database parameters
Parameter | Description | Default |
---|---|---|
mariadb.enabled |
Enable/disable MariaDB chart installation | true |
mariadb.architecture |
MariaDB architecture (standalone or replication ) |
standalone |
mariadb.auth.database |
Database name to create | dataflow |
mariadb.auth.username |
Username of new user to create | dataflow |
mariadb.auth.password |
Password for the new user | change-me |
mariadb.auth.rootPassword |
Password for the MariaDB root user |
random 10 character alphanumeric string |
mariadb.initdbScripts |
Dictionary of initdb scripts | Check values.yaml file |
externalDatabase.driver |
The fully qualified name of the JDBC Driver class | "" |
externalDatabase.scheme |
The scheme is a vendor-specific or shared protocol string that follows the "jdbc:" of the URL | "" |
externalDatabase.host |
Host of the external database | localhost |
externalDatabase.port |
External database port number | 3306 |
externalDatabase.password |
Password for the above username | "" |
externalDatabase.existingPasswordSecret |
Existing secret with database password | "" |
externalDatabase.dataflow.url |
JDBC URL for dataflow server. Overrides external scheme, host, port, database, and jdbc parameters. | "" |
externalDatabase.dataflow.username |
Existing username in the external db to be used by Dataflow server | dataflow |
externalDatabase.dataflow.database |
Name of the existing database to be used by Dataflow server | dataflow |
externalDatabase.skipper.url |
JDBC URL for skipper. Overrides external scheme, host, port, database, and jdbc parameters. | "" |
externalDatabase.skipper.username |
Existing username in the external db to be used by Skipper server | skipper |
externalDatabase.skipper.database |
Name of the existing database to be used by Skipper server | skipper |
externalDatabase.hibernateDialect |
Hibernate Dialect used by Dataflow/Skipper servers | "" |
RabbitMQ chart parameters
Parameter | Description | Default |
---|---|---|
rabbitmq.enabled |
Enable/disable RabbitMQ chart installation | true |
rabbitmq.auth.username |
RabbitMQ username | user |
rabbitmq.auth.password |
RabbitMQ password | random 40 character alphanumeric string |
externalRabbitmq.enabled |
Enable/disable external RabbitMQ | false |
externalRabbitmq.host |
Host of the external RabbitMQ | localhost |
externalRabbitmq.port |
External RabbitMQ port number | 5672 |
externalRabbitmq.username |
External RabbitMQ username | guest |
externalRabbitmq.password |
External RabbitMQ password | guest |
externalRabbitmq.vhost |
External RabbitMQ virtual host | / |
externalRabbitmq.existingPasswordSecret |
Existing secret with RabbitMQ password | "" |
Kafka chart parameters
Parameter | Description | Default |
---|---|---|
kafka.enabled |
Enable/disable Kafka chart installation | false |
kafka.replicaCount |
Number of Kafka brokers | 1 |
kafka.offsetsTopicReplicationFactor |
Kafka Secret Key | 1 |
kafka.zookeeper.enabled |
Enable/disable Zookeeper chart installation | nil |
kafka.zookeeper.replicaCount |
Number of Zookeeper replicas | 1 |
Specify each parameter using the --set key=value[,key=value]
argument to helm install
. For example,
helm install my-release --set server.replicaCount=2 bitnami/spring-cloud-dataflow
The above command install Spring Cloud Data Flow chart with 2 Dataflow server replicas.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
helm install my-release -f values.yaml bitnami/spring-cloud-dataflow
Tip: You can use the default values.yaml
Configuration and installation details
Rolling VS Immutable tags
It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.
Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.
Production configuration
This chart includes a values-production.yaml
file where you can find some parameters oriented to production configuration in comparison to the regular values.yaml
. You can use this file instead of the default one.
- Enable Pod Disruption Budget for Server and Skipper:
- server.pdb.create: false
+ server.pdb.create: true
- skipper.pdb.create: false
+ skipper.pdb.create: true
- Enable exposing Prometheus Metrics via Prometheus Rsocket Proxy:
- metrics.enabled: false
+ metrics.enabled: true
- Force users to specify a password and mount secrets as volumes instead of using environment variables on MariaDB:
- mariadb.auth.forcePassword: false
+ mariadb.auth.forcePassword: true
- mariadb.auth.usePasswordFiles: false
+ mariadb.auth.usePasswordFiles: true
Features
If you only need to deploy tasks and schedules, streaming and Skipper can be disabled:
server.configuration.batchEnabled=true
server.configuration.streamingEnabled=false
skipper.enabled=false
rabbitmq.enabled=false
If you only need to deploy streams, tasks and schedules can be disabled:
server.configuration.batchEnabled=false
server.configuration.streamingEnabled=true
skipper.enabled=true
rabbitmq.enabled=true
NOTE: Both server.configuration.batchEnabled
and server.configuration.streamingEnabled
should not be set to false
at the same time.
Messaging solutions
There are two supported messaging solutions in this chart:
- RabbitMQ (default)
- Kafka
To change the messaging layer to Kafka, use the the following parameters:
rabbitmq.enabled=false
kafka.enabled=true
Only one messaging layer can be used at a given time.
Using an external database
Sometimes you may want to have Spring Cloud components connect to an external database rather than installing one inside your cluster, e.g. to use a managed database service, or use run a single database server for all your applications. To do this, the chart allows you to specify credentials for an external database under the externalDatabase
parameter. You should also disable the MariaDB installation with the mariadb.enabled
option. For example with the following parameters:
mariadb.enabled=false
externalDatabase.scheme=mariadb
externalDatabase.host=myexternalhost
externalDatabase.port=3306
externalDatabase.password=mypassword
externalDatabase.dataflow.user=mydataflowuser
externalDatabase.dataflow.database=mydataflowdatabase
externalDatabase.dataflow.user=myskipperuser
externalDatabase.dataflow.database=myskipperdatabase
NOTE: When using the indidual propertes (scheme, host, port, database, an optional jdbcParameters) this chart will format the JDBC URL as jdbc:{scheme}://{host}:{port}/{database}{jdbcParameters}
. The URL format follows that of the MariaDB database drive but may not work for other database vendors.
To use an alternate database vendor (other than MariaDB) you can use the externalDatabase.dataflow.url
and externalDatabase.skipper.url
properties to provide the JDBC URLs for the dataflow server and skipper respectively. If these properties are defined, they will take precedence over the individual attributes. As an example of configuring an external MS SQL Server database:
mariadb.enabled=false
externalDatabase.password=mypassword
externalDatabase.dataflow.url=jdbc:sqlserver://mssql-server:1433
externalDatabase.dataflow.user=mydataflowuser
externalDatabase.skipper.url=jdbc:sqlserver://mssql-server:1433
externalDatabase.skipper.user=myskipperuser
externalDatabase.hibernateDialect=org.hibernate.dialect.SQLServer2012Dialect
NOTE: If you disable MariaDB per above you MUST supply values for the externalDatabase
connection.
Adding extra flags
In case you want to add extra environment variables to any Spring Cloud component, you can use XXX.extraEnvs
parameter(s), where XXX is placeholder you need to replace with the actual component(s). For instance, to add extra flags to Spring Cloud Data Flow, use:
server:
extraEnvs:
- name: FOO
value: BAR
Using custom Dataflow configuration
This helm chart supports using custom configuration for Dataflow server.
You can specify the configuration for Dataflow server setting the server.existingConfigmap
parameter to an external ConfigMap with the configuration file.
Using custom Skipper configuration
This helm chart supports using custom configuration for Skipper server.
You can specify the configuration for Skipper server setting the skipper.existingConfigmap
parameter to an external ConfigMap with the configuration file.
Sidecars and Init Containers
If you have a need for additional containers to run within the same pod as Dataflow or Skipper components (e.g. an additional metrics or logging exporter), you can do so via the XXX.sidecars
parameter(s), where XXX is placeholder you need to replace with the actual component(s). Simply define your container according to the Kubernetes container spec.
server:
sidecars:
- name: your-image-name
image: your-image
imagePullPolicy: Always
ports:
- name: portname
containerPort: 1234
Similarly, you can add extra init containers using the XXX.initContainers
parameter(s).
server:
initContainers:
- name: your-image-name
image: your-image
imagePullPolicy: Always
ports:
- name: portname
containerPort: 1234
Ingress
This chart provides support for ingress resources. If you have an ingress controller installed on your cluster, such as nginx-ingress or traefik you can utilize the ingress controller to serve your Spring Cloud Data Flow server.
To enable ingress integration, please set server.ingress.enabled
to true
Hosts
Most likely you will only want to have one hostname that maps to this Spring Cloud Data Flow installation. If that's your case, the property server.ingress.hostname
will set it. However, it is possible to have more than one host. To facilitate this, the server.ingress.extraHosts
object is can be specified as an array. You can also use server.ingress.extraTLS
to add the TLS configuration for extra hosts.
For each host indicated at server.ingress.extraHosts
, please indicate a name
, path
, and any annotations
that you may want the ingress controller to know about.
For annotations, please see this document. Not all annotations are supported by all ingress controllers, but this document does a good job of indicating which annotation is supported by many popular ingress controllers.
Setting Pod's affinity
This chart allows you to set your custom affinity using the XXX.affinity
parameter(s). Find more information about Pod's affinity in the kubernetes documentation.
As an alternative, you can use of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common chart. To do so, set the XXX.podAffinityPreset
, XXX.podAntiAffinityPreset
, or XXX.nodeAffinityPreset
parameters.
Troubleshooting
Find more information about how to deal with common errors related to Bitnami’s Helm charts in this troubleshooting guide.
Upgrading
If you enabled RabbitMQ chart to be used as the messaging solution for Skipper to manage streaming content, then it's necessary to set the rabbitmq.auth.password
and rabbitmq.auth.erlangCookie
parameters when upgrading for readiness/liveness probes to work properly. Inspect the RabbitMQ secret to obtain the password and the Erlang cookie, then you can upgrade your chart using the command below:
To 2.0.0
On November 13, 2020, Helm v2 support was formally finished, this major version is the result of the required changes applied to the Helm Chart to be able to incorporate the different features added in Helm v3 and to be consistent with the Helm project itself regarding the Helm v2 EOL.
What changes were introduced in this major version?
- Previous versions of this Helm Chart use
apiVersion: v1
(installable by both Helm 2 and 3), this Helm Chart was updated toapiVersion: v2
(installable by Helm 3 only). Here you can find more information about theapiVersion
field. - Move dependency information from the requirements.yaml to the Chart.yaml
- After running
helm dependency update
, a Chart.lock file is generated containing the same structure used in the previous requirements.lock - The different fields present in the Chart.yaml file has been ordered alphabetically in a homogeneous way for all the Bitnami Helm Charts
Considerations when upgrading to this version
- If you want to upgrade to this version from a previous one installed with Helm v3, you shouldn't face any issues
- If you want to upgrade to this version using Helm v2, this scenario is not supported as this version doesn't support Helm v2 anymore
- If you installed the previous version with Helm v2 and wants to upgrade to this version with Helm v3, please refer to the official Helm documentation about migrating from Helm v2 to v3
Useful links
- https://docs.bitnami.com/tutorials/resolve-helm2-helm3-post-migration-issues/
- https://helm.sh/docs/topics/v2_v3_migration/
- https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/
v0.x.x
helm upgrade my-release bitnami/spring-cloud-dataflow --set mariadb.rootUser.password=[MARIADB_ROOT_PASSWORD] --set rabbitmq.auth.password=[RABBITMQ_PASSWORD] --set rabbitmq.auth.erlangCookie=[RABBITMQ_ERLANG_COOKIE]
v1.x.x
helm upgrade my-release bitnami/spring-cloud-dataflow --set mariadb.auth.rootPassword=[MARIADB_ROOT_PASSWORD] --set rabbitmq.auth.password=[RABBITMQ_PASSWORD] --set rabbitmq.auth.erlangCookie=[RABBITMQ_ERLANG_COOKIE]
Notable changes
v1.0.0
MariaDB dependency version was bumped to a new major version that introduces several incompatilibites. Therefore, backwards compatibility is not guaranteed unless an external database is used. Check MariaDB Upgrading Notes for more information.
To upgrade to 1.0.0
, you will need to reuse the PVC used to hold the MariaDB data on your previous release. To do so, follow the instructions below (the following example assumes that the release name is dataflow
):
NOTE: Please, create a backup of your database before running any of those actions.
Obtain the credentials and the name of the PVC used to hold the MariaDB data on your current release:
export MARIADB_ROOT_PASSWORD=$(kubectl get secret --namespace default dataflow-mariadb -o jsonpath="{.data.mariadb-root-password}" | base64 --decode)
export MARIADB_PASSWORD=$(kubectl get secret --namespace default dataflow-mariadb -o jsonpath="{.data.mariadb-password}" | base64 --decode)
export MARIADB_PVC=$(kubectl get pvc -l app=mariadb,component=master,release=dataflow -o jsonpath="{.items[0].metadata.name}")
export RABBITMQ_PASSWORD=$(kubectl get secret --namespace default dataflow-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)
export RABBITMQ_ERLANG_COOKIE=$(kubectl get secret --namespace default dataflow-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)
Upgrade your release (maintaining the version) disabling MariaDB and scaling Data Flow replicas to 0:
$ helm upgrade dataflow bitnami/spring-cloud-dataflow --version 0.7.4 \
--set server.replicaCount=0 \
--set skipper.replicaCount=0 \
--set mariadb.enabled=false \
--set rabbitmq.auth.password=$RABBITMQ_PASSWORD \
--set rabbitmq.auth.erlangCookie=$RABBITMQ_ERLANG_COOKIE
Finally, upgrade you release to 1.0.0 reusing the existing PVC, and enabling back MariaDB:
$ helm upgrade dataflow bitnami/spring-cloud-dataflow \
--set mariadb.primary.persistence.existingClaim=$MARIADB_PVC \
--set mariadb.auth.rootPassword=$MARIADB_ROOT_PASSWORD \
--set mariadb.auth.password=$MARIADB_PASSWORD \
--set rabbitmq.auth.password=$RABBITMQ_PASSWORD \
--set rabbitmq.auth.erlangCookie=$RABBITMQ_ERLANG_COOKIE
You should see the lines below in MariaDB container logs:
$ kubectl logs $(kubectl get pods -l app.kubernetes.io/instance=dataflow,app.kubernetes.io/name=mariadb,app.kubernetes.io/component=primary -o jsonpath="{.items[0].metadata.name}")
...
mariadb 12:13:24.98 INFO ==> Using persisted data
mariadb 12:13:25.01 INFO ==> Running mysql_upgrade
...
Expected output
After issuing the helm install
command, you should see output similar to the following:
NAME: my-release
LAST DEPLOYED: Sun Nov 22 21:12:29 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
Spring Cloud Data Flow chart was deployed enabling the following components:
- Spring Cloud Data Flow server
- Spring Cloud Skipper server
Spring Cloud Data Flow can be accessed through the following DNS name from within your cluster:
my-release-spring-cloud-dataflow-server.default.svc.cluster.local (port 8080)
To access Spring Cloud Data Flow dashboard from outside the cluster execute the following commands:
1. Get the Data Flow dashboard URL by running these commands:
export SERVICE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].port}" services my-release-spring-cloud-dataflow-server)
kubectl port-forward --namespace default svc/my-release-spring-cloud-dataflow-server ${SERVICE_PORT}:${SERVICE_PORT} &
echo "http://127.0.0.1:${SERVICE_PORT}/dashboard"
2. Open a browser and access the Data Flow dashboard using the obtained URL.
If you prefer, the Spring Cloud Data Flow service type may be changed by passing the following set
argument to helm install
:
--set server.service.type=ServiceType
Where ServiceType
would be a valid service name, for example LoadBalancer
, NodePort
, etc.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of the server by running kubectl get svc -w my-release-spring-cloud-dataflow-server
If your using Minikube without load balancer support, you can use the following command to get the URL for the server:
minikube service --url my-release-spring-cloud-dataflow-server
You have just created a new release in the default namespace of your Kubernetes cluster.
It takes a couple of minutes for the application and its required services to start.
You can check on the status by issuing a kubectl get pod -w
command.
You need to wait for the READY
column to show 1/1
for all pods.
When all pods are ready, you can access the Spring Cloud Data Flow dashboard by accessing http://<SERVICE_ADDRESS>/dashboard
where <SERVICE_ADDRESS>
is the address returned by either the kubectl
or minikube
commands above.
Version Compatibility
The following listing shows Spring Cloud Data Flow’s version compatibility with the respective Helm Chart releases:
Deprecated chart mappings from official Helm repository:
SCDF Version | Chart Version |
---|---|
SCDF-K8S-Server 1.7.x | 1.0.x |
SCDF-K8S-Server 2.0.x | 2.2.x |
SCDF-K8S-Server 2.1.x | 2.3.x |
SCDF-K8S-Server 2.2.x | 2.4.x |
SCDF-K8S-Server 2.3.x | 2.5.x |
SCDF-K8S-Server 2.4.x | 2.6.x |
SCDF-K8S-Server 2.5.x | 2.7.x |
SCDF-K8S-Server 2.6.x | 2.8.x |
Bitnami chart mappings:
SCDF Version | Chart Version |
---|---|
SCDF-K8S-Server 2.6.x | 1.1.x |
SCDF-K8S-Server 2.7.x | 2.0.x |
Register prebuilt applications
All the prebuilt streaming applications:
- Are available as Apache Maven artifacts or Docker images.
- Use RabbitMQ or Apache Kafka.
- Support monitoring via Prometheus and InfluxDB.
- Contain metadata for application properties used in the UI and code completion in the shell.
Applications can be registered individually using the app register
functionality or as a group using the app import
functionality.
There are also dataflow.spring.io
links that represent the group of prebuilt applications for a specific release which is useful for getting started.
You can register applications using the UI or the shell. Even though we are only using two prebuilt applications, we will register the full set of prebuilt applications.
The easiest way to install Data Flow on Kubernetes is using the Helm chart that uses RabbitMQ as the default messaging middleware. The command to import the Kafka version of the applications is
dataflow:>app import --uri https://dataflow.spring.io/kafka-docker-latest
Change kafka
to rabbitmq
in the above URL if you set kafka.enabled=true
in the helm chart or followed the manual kubectl
based installation instructions for installing Data Flow on Kubernetes and chose to use Kafka as the messaging middleware.
Only applications registered with a --uri
property
pointing to a Docker resource are supported by the Data Flow Server
for Kubernetes. However, we do support Maven resources for the
--metadata-uri
property, which is used to list the properties
supported by each application. For example, the following application
registration is valid:
app register --type source --name time --uri docker://springcloudstream/time-source-rabbit:{docker-time-source-rabbit-version} --metadata-uri maven://org.springframework.cloud.stream.app:time-source-rabbit:jar:metadata:{docker-time-source-rabbit-version}
Any application registered with a Maven, HTTP, or File resource or the executable jar (by using a --uri
property prefixed with
maven://
, http://
or file://
) is not supported.
Application and Server Properties
This section covers how you can customize the deployment of your applications. You can use a number of properties to influence settings for the applications that are deployed. Properties can be applied on a per-application basis or in the appropriate server configuration for all deployed applications.
Properties set on a per-application basis always take precedence over properties set as the server configuration. This arrangement lets you override global server level properties on a per-application basis.
Properties to be applied for all deployed Tasks are defined in the
src/kubernetes/server/server-config.yaml
file and for Streams
in src/kubernetes/skipper/skipper-config-(binder).yaml
. Replace
(binder)
with the messaging middleware you are using — for example,
rabbit
or kafka
.
Memory and CPU Settings
Applications are deployed with default memory and CPU settings. If
needed, these values can be adjusted. The following example shows how to
set Limits
to 1000m
for CPU
and 1024Mi
for memory and Requests
to 800m
for CPU and 640Mi
for memory:
deployer.<app>.kubernetes.limits.cpu=1000m
deployer.<app>.kubernetes.limits.memory=1024Mi
deployer.<app>.kubernetes.requests.cpu=800m
deployer.<app>.kubernetes.requests.memory=640Mi
Those values results in the following container settings being used:
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 800m
memory: 640Mi
You can also control the default values to which to set the cpu
and
memory
globally.
The following example shows how to set the CPU and memory for streams and tasks:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
limits:
memory: 640mi
cpu: 500m
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
limits:
memory: 640mi
cpu: 500m
The settings we have used so far only affect the settings for the container. They do not affect the memory setting for the JVM process in the container. If you would like to set JVM memory settings, you can provide an environment variable to do so. See the next section for details.
Environment Variables
To influence the environment settings for a given application, you can
use the spring.cloud.deployer.kubernetes.environmentVariables
deployer
property. For example, a common requirement in production settings is to
influence the JVM memory arguments. You can do so by using the
JAVA_TOOL_OPTIONS
environment variable, as the following example
shows:
deployer.<app>.kubernetes.environmentVariables=JAVA_TOOL_OPTIONS=-Xmx1024m
The environmentVariables
property accepts a comma-delimited string.
If an environment variable contains a value which is also a
comma-delimited string, it must be enclosed in single quotation marks — for example,
spring.cloud.deployer.kubernetes.environmentVariables=spring.cloud.stream.kafka.binder.brokers='somehost:9092, anotherhost:9093'
This overrides the JVM memory setting for the desired <app>
(replace
<app>
with the name of your application).
Liveness and Readiness Probes
The liveness
and readiness
probes use paths called /health
and
/info
, respectively. They use a delay
of 10
for both and a
period
of 60
and 10
respectively. You can change these defaults
when you deploy the stream by using deployer properties. Liveness and
readiness probes are only applied to streams.
The following example changes the liveness
probe (replace <app>
with
the name of your application) by setting deployer properties:
deployer.<app>.kubernetes.livenessProbePath=/health
deployer.<app>.kubernetes.livenessProbeDelay=120
deployer.<app>.kubernetes.livenessProbePeriod=20
You can declare the same as part of the server global configuration for streams, as the following example shows:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
livenessProbePath: /health
livenessProbeDelay: 120
livenessProbePeriod: 20
Similarly, you can swap liveness
for readiness
to override the
default readiness
settings.
By default, port 8080 is used as the probe port. You can change the
defaults for both liveness
and readiness
probe ports by using
deployer properties, as the following example shows:
deployer.<app>.kubernetes.readinessProbePort=7000
deployer.<app>.kubernetes.livenessProbePort=7000
You can declare the same as part of the global configuration for streams, as the following example shows:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
readinessProbePort: 7000
livenessProbePort: 7000
By default, the liveness
and readiness
probe paths use Spring Boot
2.x+ actuator endpoints. To use Spring Boot 1.x actuator endpoint
paths, you must adjust the liveness
and readiness
values, as the
following example shows (replace <app>
with the name of your
application):
deployer.<app>.kubernetes.livenessProbePath=/health
deployer.<app>.kubernetes.readinessProbePath=/info
To automatically set both liveness
and readiness
endpoints on a
per-application basis to the default Spring Boot 1.x paths, you can set
the following property:
deployer.<app>.kubernetes.bootMajorVersion=1
You can access secured probe endpoints by using credentials stored in a
Kubernetes
secret. You
can use an existing secret, provided the credentials are contained under
the credentials
key name of the secret’s data
block. You can
configure probe authentication on a per-application basis. When enabled,
it is applied to both the liveness
and readiness
probe endpoints by
using the same credentials and authentication type. Currently, only
Basic
authentication is supported.
To create a new secret:
-
Generate the base64 string with the credentials used to access the secured probe endpoints.
Basic authentication encodes a username and password as a base64 string in the format of
username:password
.The following example (which includes output and in which you should replace
user
andpass
with your values) shows how to generate a base64 string:echo -n "user:pass" | base64 dXNlcjpwYXNz
-
With the encoded credentials, create a file (for example,
myprobesecret.yml
) with the following contents:apiVersion: v1 kind: Secret metadata: name: myprobesecret type: Opaque data: credentials: GENERATED_BASE64_STRING
- Replace
GENERATED_BASE64_STRING
with the base64-encoded value generated earlier. -
Create the secret by using
kubectl
, as the following example shows:kubectl create -f ./myprobesecret.yml secret "myprobesecret" created
-
Set the following deployer properties to use authentication when accessing probe endpoints, as the following example shows:
deployer.<app>.kubernetes.probeCredentialsSecret=myprobesecret
Replace
<app>
with the name of the application to which to apply authentication.
Using SPRING_APPLICATION_JSON
You can use a SPRING_APPLICATION_JSON
environment variable to set Data
Flow server properties (including the configuration of maven repository
settings) that are common across all of the Data Flow server
implementations. These settings go at the server level in the container
env
section of a deployment YAML. The following example shows how to
do so:
env:
- name: SPRING_APPLICATION_JSON
value: |-
{
"maven": {
"local-repository": null,
"remote-repositories": {
"repo1": {
"url": "https://repo.spring.io/libs-snapshot"
}
}
}
}
Private Docker Registry
You can pull Docker images from a private registry on a per-application basis. First, you must create a secret in the cluster. Follow the Pull an Image from a Private Registry guide to create the secret.
Once you have created the secret, you can use the imagePullSecret
property to set the secret to use, as the following example shows:
deployer.<app>.kubernetes.imagePullSecret=mysecret
Replace <app>
with the name of your application and mysecret
with
the name of the secret you created earlier.
You can also configure the image pull secret at the global server level.
The following example shows how to do so for streams and tasks:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
imagePullSecret: mysecret
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
imagePullSecret: mysecret
Replace mysecret
with the name of the secret you created earlier.
Volume Mounted Secretes
Data Flow uses the application metadata stored in a container image label. To access the metadata labels in a private registry, you have to extend the Data Flow deployment configuration and mount the registry secrets as a Secrets PropertySource:
spec:
containers:
- name: scdf-server
...
volumeMounts:
- name: mysecret
mountPath: /etc/secrets/mysecret
readOnly: true
...
volumes:
- name: mysecret
secret:
secretName: mysecret
Annotations
You can add annotations to Kubernetes objects on a per-application
basis. The supported object types are pod Deployment
, Service
, and
Job
. Annotations are defined in a key:value
format, allowing for
multiple annotations separated by a comma. For more information and use
cases on annotations, see
Annotations.
The following example shows how you can configure applications to use annotations:
deployer.<app>.kubernetes.podAnnotations=annotationName:annotationValue
deployer.<app>.kubernetes.serviceAnnotations=annotationName:annotationValue,annotationName2:annotationValue2
deployer.<app>.kubernetes.jobAnnotations=annotationName:annotationValue
Replace <app>
with the name of your application and the value of your
annotations.
Entry Point Style
An entry point style affects how application properties are passed to the container to be deployed. Currently, three styles are supported:
exec
(default): Passes all application properties and command line arguments in the deployment request as container arguments. Application properties are transformed into the format of--key=value
.shell
: Passes all application properties and command line arguments as environment variables. Each of the application and command line argument properties is transformed into an uppercase string and.
characters are replaced with_
.boot
: Creates an environment variable calledSPRING_APPLICATION_JSON
that contains a JSON representation of all application properties. Command line arguments from the deployment request are set as container args.
In all cases, environment variables defined at the server-level configuration and on a per-application basis are set onto the container as is.
You can configure applications as follows:
deployer.<app>.kubernetes.entryPointStyle=<Entry Point Style>
Replace <app>
with the name of your application and
<Entry Point Style>
with your desired entry point style.
You can also configure the entry point style at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
entryPointStyle: entryPointStyle
The following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
entryPointStyle: entryPointStyle
Replace entryPointStye
with the desired entry point style.
You should choose an Entry Point Style of either exec
or shell
, to
correspond to how the ENTRYPOINT
syntax is defined in the container’s
Dockerfile
. For more information and uses cases on exec
versus
shell
, see the
ENTRYPOINT
section of the Docker documentation.
Using the boot
entry point style corresponds to using the exec
style
ENTRYPOINT
. Command line arguments from the deployment request are
passed to the container, with the addition of application properties
being mapped into the SPRING_APPLICATION_JSON
environment variable
rather than command line arguments.
When you use the boot
Entry Point Style, the deployer.<app>.kubernetes.environmentVariables
property must not
contain SPRING_APPLICATION_JSON
.
Deployment Service Account
You can configure a custom service account for application deployments
through properties. You can use an existing service account or create a
new one. One way to create a service account is by using kubectl
, as
the following example shows:
kubectl create serviceaccount myserviceaccountname
serviceaccount "myserviceaccountname" created
Then you can configure individual applications as follows:
deployer.<app>.kubernetes.deploymentServiceAccountName=myserviceaccountname
Replace <app>
with the name of your application and
myserviceaccountname
with your service account name.
You can also configure the service account name at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
deploymentServiceAccountName: myserviceaccountname
The following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
deploymentServiceAccountName: myserviceaccountname
Replace myserviceaccountname
with the service account name to be
applied to all deployments.
Image Pull Policy
An image pull policy defines when a Docker image should be pulled to the local registry. Currently, three policies are supported:
IfNotPresent
(default): Do not pull an image if it already exists.Always
: Always pull the image regardless of whether it already exists.Never
: Never pull an image. Use only an image that already exists.
The following example shows how you can individually configure applications:
deployer.<app>.kubernetes.imagePullPolicy=Always
Replace <app>
with the name of your application and Always
with your
desired image pull policy.
You can configure an image pull policy at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
imagePullPolicy: Always
The following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
imagePullPolicy: Always
Replace Always
with your desired image pull policy.
Deployment Labels
You can set custom labels on objects related to
Deployment.
See
Labels
for more information on labels. Labels are specified in key:value
format.
The following example shows how you can individually configure applications:
deployer.<app>.kubernetes.deploymentLabels=myLabelName:myLabelValue
Replace <app>
with the name of your application, myLabelName
with
your label name, and myLabelValue
with the value of your label.
Additionally, you can apply multiple labels, as the following example shows:
deployer.<app>.kubernetes.deploymentLabels=myLabelName:myLabelValue,myLabelName2:myLabelValue2
NodePort
Applications are deployed using a Service
type of ClusterIP which is the default Kubernetes Service
type if not defined otherwise.
ClusterIP
services are only reachable from within the cluster itself.
To expose the deployed application to be available externally, one option is to use NodePort
.
See the NodePort documentation for more information.
The following example shows how you can individually configure applications using Kubernetes assigned ports:
deployer.<app>.kubernetes.createNodePort=true
Replace <app>
with the name of your application.
Additionally, you can define the port to use for the NodePort
Service
as shown below:
deployer.<app>.kubernetes.createNodePort=31101
Replace <app>
with the name of your application and the value of 31101
with your desired port.
When defining the port manually, the port must not already be in use and within the defined NodePort
range.
Per NodePort the default port range is 30000-32767.
Monitoring
To learn more about the monitoring experience in Data Flow using Prometheus running on Kubernetes, please refer to the Stream Monitoring or Task Monitoring guides.