Record Transcode Consume

The Record Transcode Consume policy converts an already schema-validated record value into a different serialization format before Event Gateway returns it to the consumer. It runs on a record that was parsed by a Schema Validation Consume policy, so you must nest it under that policy.

This policy supports converting between JSON and Avro.

Use this policy to change the format of a record only for the client that reads it. The record in the backend cluster doesn’t change, so systems that are permitted to see the original format keep their access to it. To convert the format of a record before Event Gateway writes it to the backend cluster, use the Record Transcode Produce policy instead.

Common use cases for the Record Transcode Consume policy:

This policy runs in the consume phase, after schema validation parses the record.

1. A Kafka client sends a request to consume records to Event Gateway.
2. The backend cluster returns the records, which hold the original values.
3. The parent Schema Validation policy parses the record value.

4. The Record Transcode Consume policy converts the record value into the configured output_format.

   If the record value already matches output_format, the policy leaves it unchanged.

5. Event Gateway returns the converted record to the client. The stored record doesn’t change.

 
sequenceDiagram
  autonumber
  participant client as Client
  participant egw as Event Gateway
  participant broker as Event broker

  client->>egw: consume request
  egw->>broker: fetch records
  broker->>egw: return records with original values
  egw->>egw: parse record (Schema Validation)
  egw->>egw: convert record value to output_format

  egw->>client: return record with converted value
  

Set output_format to json or avro. Converting to Avro requires schema_source, because Avro needs a schema to serialize the record value. Converting to JSON can optionally use schema_source to validate the converted value.

schema_source accepts one of the following types:

Leave schema_source unset if you don’t need a schema for the output data, for example when converting to JSON without validating the result.

Recording the schema reference

schema_ref_destination determines whether and how Event Gateway records a reference to the output schema on the converted record:

The Record Transcode Consume policy supports the following failure modes:

Conversion fails when the record value can’t be converted to output_format, or when the converted value doesn’t satisfy the schema referenced by schema_source.

Use error sparingly, because an error on a batch makes clients stop at the problematic offset, and an operator must intervene to skip it.

This policy needs the parsed record value, so nest it directly under the Schema Validation Consume policy that parses it.

This policy doesn’t support nested policies of its own.

See the reference for nested policies for more detail.