Record Transcode Produce

The Record Transcode Produce policy converts an already schema-validated record value into a different serialization format before Event Gateway produces it to the backend cluster. It runs on a record that was already parsed by a Schema Validation Produce policy, so you must nest it under that policy.

This policy supports converting between JSON and Avro.

Use this policy to change the format of a record before Event Gateway writes it to the backend cluster. This change is irreversible, because the cluster never receives the record in its original format. To convert the format of a record only for the client that reads it, and keep the stored record unchanged, use the Record Transcode Consume policy instead.

Common use cases for the Record Transcode Produce policy:

This policy runs in the produce phase, after schema validation parses the record.

1. A Kafka client produces a record and sends it to Event Gateway.
2. The parent Schema Validation policy parses the record value.

3. The Record Transcode Produce policy converts the record value into the configured output_format.

   If the record value already matches output_format, the policy leaves it unchanged.

4. Event Gateway sends the record to the backend cluster, which stores the converted value.

 
sequenceDiagram
  autonumber
  participant client as Client
  participant egw as Event Gateway
  participant broker as Event broker

  client->>egw: produce record
  egw->>egw: parse record (Schema Validation)
  egw->>egw: convert record value to output_format

  egw->>broker: send record with converted value
  

The parent Schema Validation policy determines which formats a client can produce. A schema registry with a fixed schema_type only accepts records in that one format.

To accept records in any supported format, set schema_type to any v1.3+ on the schema registry that the parent policy uses. This is useful while producers migrate from JSON to Avro, or when different producers send different formats to the same topics.

With any, Event Gateway resolves the format of each record individually:

• Values that carry a schema reference, in the Confluent wire format or in a record header, are decoded using the registry.
• Any other value is parsed as JSON without a schema, so Event Gateway doesn’t validate it.

For example, with output_format: avro, clients can produce both JSON and Avro records to the same topic. The policy transcodes the JSON records to Avro, so the backend cluster only ever stores Avro.

Set output_format to json or avro. Converting to Avro requires schema_source, because Avro needs a schema to serialize the record value. Converting to JSON can optionally use schema_source to validate the converted value.

schema_source accepts one of the following types:

Leave schema_source unset if you don’t need a schema for the output data, for example when converting to JSON without validating the result.

Recording the schema reference

schema_ref_destination determines whether and how Event Gateway records a reference to the output schema on the converted record:

The Record Transcode Produce policy supports the following failure modes:

Conversion fails when the record value can’t be converted to output_format, or when the converted value doesn’t satisfy the schema referenced by schema_source.

This policy needs the parsed record value, so nest it directly under the Schema Validation Produce policy that parses it.

This policy doesn’t support nested policies of its own.

See the reference for nested policies for more detail.