Confluent Avro Support

The Kafka connector supports the use of Apache Avro-formatted messages. Avro is a data serialization system that provides a compact binary format and relies on schemas to define data structures. By integrating with a Confluent Schema Registry, the connector can handle serialization (Produce) and deserialization (Consume/Listen) of messages.

Avro support is available for the following operations

Produce
Consume
Listen

Wire format modes

The Kafka connector supports two formats for Avro messages on a Kafka topic:

Format	Bytes on the topic	Used by
Confluent wire format	`[0x00][4-byte schema ID][Avro binary]`	Confluent stack consumers
Plain Avro binary	`[Avro binary]`	Consumers that already know the schema out-of-band

Producer behavior is controlled by the Produce Avro message in Confluent Wire Format checkbox on the Produce operation. The default is unchecked, which emits plain Avro binary.
Consumer behavior is automatic. The Consume and Listen operations can process both formats on the same topic.

Prerequisites

To implement Avro support within your Kafka operations, ensure you have the following.

Connectivity from the Boomi runtime to your Confluent Schema Registry.
Registered Avro schemas in the Schema Registry for the specific topics you intend to use.
Necessary credentials (Username/Password or API Key/Secret) if the Schema Registry requires authentication.
Existing Kafka topics are configured to process Avro data.

Connection configuration

To enable Avro support, you must configure the Schema Registry settings within your Kafka connection component:

Schema Registry URL — Enter the full URL for your registry (e.g., https://schema-registry.example.com:8081).
Authentication Method — Basic
Username — Enter the username for your registry.
Password — Enter the password for your registry.

note

These credentials must have sufficient permissions to read schemas for all operations using this connection.
When the Schema Registry URL is specified, the Test Connection validates that the connection component can successfully connect to both Kafka as well as the schema registry.

Enabling Avro in Operations

The Kafka connector utilizes dynamic configuration to manage Avro serialization.

During Browse, in the Import wizard, select Dynamic Topic and Use Avro Schema checkboxes. The Kafka connector lists all the schemas in the Object Type dropdown. The user can select the schema from this list. The connector retrieves the schema and generates a corresponding JSON profile.

The user can specify the Topic Name in the Topic Name Browse property or Topic Name Operation Property. Alternatively, the Topic Name can be passed as a Dynamic Operation Property.

Produce operation: This is a wire format selection. After import, the Produce operation displays a Produce Avro message in Confluent Wire Format checkbox. The default is unchecked. See Producing in Confluent wire format for guidance on when to enable it.

Consume and Listen operations: No extra configuration is required. Both operations automatically detect Confluent wire format and plain Avro binary on a per-message basis. The same operation can process both formats from the same topic without any change.

Schema Updates: If a schema changes in the Registry, you must re-import the object in the operation to update the JSON profile.

Produce Operation

The Produce operation converts JSON input documents into Avro binary format before sending them to Kafka.

The input JSON document must strictly match the structure of the imported Avro schema. Discrepancies in data types or missing required fields will result in serialization errors.

Producing in Confluent wire format

When you import a Produce operation against a Schema Registry subject, the operation properties include the following checkbox:

Operation property	Default	Description
Produce Avro message in Confluent Wire Format	Unchecked	When checked, each produced message is prefixed with the Confluent magic byte (`0x00`) and the 4-byte schema ID of the imported subject, followed by the Avro-binary-encoded payload. When unchecked, the connector emits plain Avro binary with no prefix.

When to enable the checkbox - Enable the checkbox if the consumer of your topic is part of the Confluent stack or expects the Confluent wire format.

When to leave the checkbox unchecked - Leave the checkbox unchecked if your consumers are configured to expect plain Avro binary.

Consume and Listen Operations

Consume and Listen operations read Avro binary data from Kafka and deserialize it into JSON documents.

Schema evolution

The connector supports Avro schema evolution only on the Confluent wire-format path, where each message carries the schema ID used by the producer:

The connector uses the writer schema (resolved by the schema ID in the message) and the reader schema (imported on the operation) together. Avro's GenericDatumReader reconciles the two and produces a record that conforms to the reader schema.
Supported evolution patterns follow standard Avro rules — added fields with defaults, removed optional fields.
On the plain-Avro fallback path (messages without the magic byte), the connector decodes using the reader schema only. There is no writer schema, so no evolution occurs — the message must conform to the imported schema exactly, or deserialization fails.

note

When to re-import the operation: if the registered schema has changed in a way that you want reflected in the Boomi profile (for example, a new field that downstream shapes should reference), re-import the operation.

Error handling and batch semantics

The underlying Kafka client deserializes all records in a fetched batch before returning them. There is no per-record error handling at the client deserialization layer, which has the following consequences:

Operation	Behavior when one record in a batch fails to deserialize
Consume	The entire poll fails. The failure is surfaced on the current document; subsequent documents in the same execution are still attempted.
Listen	The polling task reports the failure to Boomi and closes the consumer. The connection's standard retry sequence (`0, 10, 30, 60, 120, 300 seconds`, up to six retries) applies.

Common causes of deserialization failure

Message has the 0x00 magic byte but the schema ID is not present in the Schema Registry — verify the producer is publishing to the same registry the connection is configured against.
Message does not have the magic byte and does not conform to the imported reader schema — fallback is plain-Avro path.
Message body is corrupted or truncated. This corrupted message needs to be handled manually removing it from the topic or need to consume via non Avro consumer operation.

If non-Avro messages may be published to the same topic, route them to a separate topic — there is no per-record fall-through to a raw-bytes path.

Supported Data Types

The Kafka connector supports primitive types in AVRO messages: Primitive datatypes include string, int, long, float, double, boolean, bytes, and null.

Troubleshooting

Issue	Resolution
Schema Not Found	Verify the Schema Registry URL, credentials, and ensure the schema is present in the Confluent schema registry.
Serialization Errors (Produce)	Ensure the input JSON matches the Avro schema structure and all required fields are present.
Deserialization Errors (Consume/Listen)	Ensure the input JSON matches the Avro schema structure and all required fields are present.
Schema evolution (Produce/Consume/Listen)	Check for schema incompatibility or re-import the object if the registry schema has been updated.
"Use Avro Schema" missing	Ensure Allow Dynamic Topics is selected in during Browse
Confluent consumer cannot decode messages produced by Boomi Likely Cause: The Produce Avro message in Confluent Wire Format checkbox is unchecked, so the producer is emitting plain Avro binary without the magic byte and schema ID prefix.	Enable the checkbox on the Produce operation and redeploy.
Consume or Listen fails after a producer-side schema change Likely Cause: The new writer schema ID is not resolvable. The imported reader schema may be too old to be compatible.	Verify the new schema is registered in the Schema Registry the connection points to. If the Boomi profile must reflect new fields, re-import the operation.
One bad message stops the entire poll Likely Cause: This is a Kafka client constraint — the deserializer runs across the whole batch before any record is returned.	Ensure only Avro-compatible messages are published to the topic; route incompatible payloads to a separate topic.
`GET /schemas/ids/{id}` returns `401/403` in the Atom logs Likely Cause: Schema Registry credentials on the connection lack read permission on the subject.	Grant the registry user read access to the affected subjects.