Mask Fields Consume

The Mask Fields Consume policy redacts string fields of Kafka records during the consume phase. It runs on a record that a Schema Validation Consume policy parsed, so you must nest it under that policy.

Use this policy to redact a value only for the client that reads it. The record in the backend cluster doesn’t change, so systems that are permitted to see the original value keep their access to it. To redact a value before Event Gateway writes it to the cluster, use the Mask Fields Produce policy instead.

Select the fields to redact as a static list of paths, or as a single expression that returns a list of paths. Each entry pairs a set of paths with the strategy that redacts them.

Common use cases for the Mask Fields Consume policy:

This policy runs in the consume phase, after schema validation parses the record.

1. A Kafka client sends a request to consume records to Event Gateway.
2. The backend cluster returns the records, which hold the original values.
3. The parent Schema Validation policy parses the record value.
4. The Mask Fields Consume policy replaces the value of each selected field with a masked value.
5. Event Gateway returns the masked record to the client. The stored record doesn’t change.

 
sequenceDiagram
  autonumber
  participant client as Client
  participant egw as Event Gateway
  participant broker as Event broker

  client->>egw: consume request
  egw->>broker: fetch records
  broker->>egw: return records with original values
  egw->>egw: parse record (Schema Validation)
  egw->>egw: mask selected fields

  egw->>client: return record with masked value
  

This policy masks string fields only:

• A selected field that is not present in the record is ignored.
• A selected field that is not a string is a policy failure. The failure_mode setting controls what happens in this situation.

Warning: Masking can break client-side validation. For example, a client that validates the format of an SSN can reject a masked value. Make sure your clients accept masked values before you enable this policy.

Each entry in mask_fields uses one of the following strategies:

The Mask Fields Consume policy supports the following failure modes:

Set the parent Schema Validation Consume policy to skip or error to use this policy.

The parent policy must not use passthrough or mark. With these failure modes, the parent delivers the record without parsing it. The Mask Fields Consume policy then never runs, and the client receives the fields unmasked.

Instead of listing paths one by one, the paths entry can take a single expression that reads the record’s schema metadata and returns the tagged field paths. For example, with a Confluent Schema Registry, an expression can return every field path that carries a PII tag:

paths: 'record.value.schema.metadata.tags["PII"]'

This is the recommended way to mask fields. The mask follows the schema, so when the schema evolves and a new field gets tagged, it is masked without any policy change.

In this case, the schema becomes the input that decides which fields are masked, so anyone who can edit the schema tags can change the masking. Keep schema registry write access restricted to trusted roles.

See the full walkthrough in Mask Kafka message fields selected by schema registry tags.

This policy needs the original value, so the policy order is important when a field is also encrypted. Let the Decrypt Fields policy decrypt the fields before you mask them.

See the reference for nested policies for more detail.