Skip to main content
Use this destination when downstream consumers read from Kafka and you want the messages validated before they are published. The Kafka destination writes messages to a Kafka topic. It supports keyed messages for partition control, structured and raw payload formats, and SASL/PLAIN authentication.

​
Message Format

The payload format depends on the source schema:
  • Structured data — When the schema contains regular typed columns, each row is serialized as a JSON object and sent as the message value.
  • JSON or XML — When the schema contains a single _json or _xml field, the raw string value is sent directly as the message value without additional wrapping.

​
Message Key

Set keyField to the name of a column whose value should be used as the Kafka message key. Messages with the same key are routed to the same partition, preserving ordering for that key. If keyField is omitted, messages are distributed across partitions using the default partitioner.

​
Authentication

Connection details are retrieved from Vault using the secret name specified in kafkaProducerSecretName. When the secret carries both username and password, the destination authenticates to the Kafka cluster using SASL/PLAIN (SASL_PLAINTEXT); without them it connects unauthenticated. The secret contains:

​
Bootstrap Server Override

To override the bootstrap servers from the Vault secret on a per-pipeline basis, set overrideBootstrapServers in the destination configuration.

​
Configuration Example

Kafka credentials (bootstrapServers, username, password) are configured globally in application.yaml via secrets.kafkaProducerSecretName. See Configuration Reference for Vault secret format.

​
Field Reference

​
Completion Notification

A pipeline notification is published to ActiveMQ on completion. See Notifications for details.