Non-sampled logs (for example, access logs) are excellent sources of usage information. Combined with the OpenTelemetry open standard for log forwarding, you can extract usage information from your logs and forward them to Metering & Billing.
OpenTelemetry Collector
Related Documentation
Incompatible with
on-prem
Related Resources
Prerequisites
You will need an OpenTelemetry-compatible log-forwarding solution. You can get started with the OpenTelemetry Collector.
Configuration
First, create a new YAML file for the collector configuration. Use the otel_log Redpanda Connect input:
input:
otel_log:
# Point your log forwarder to this address using the OTLP gRPC protocol.
address: 127.0.0.1:4317
Copied!
Note: This is a custom input plugin that is not part of the official Redpanda Connect distribution. You can find the source code of the plugin on GitHub.
Next, configure the mapping from your log schema to CloudEvents using bloblang:
pipeline:
processors:
- mapping: |
root = {
"id": uuid_v4(),
"specversion": "1.0",
"type": "api-calls",
"source": "otlp-log",
"time": this.record.attributes.time,
"subject": this.record.attributes.subject,
"data": {
"method": this.record.attributes.method,
"path": this.record.attributes.path,
"region": this.record.attributes.region,
"zone": this.record.attributes.zone,
"duration_ms": this.record.attributes.duration,
},
}
Copied!
About log attributes:
this.record.attributescontains the log attributes extracted by theotel_loginput plugin.
Finally, configure the output:
output:
label: 'openmeter'
drop_on:
error: false
error_patterns:
- Bad Request
output:
http_client:
url: '${OPENMETER_URL:https://us.api.konghq.com}/v3/openmeter/events'
verb: POST
headers:
Authorization: 'Bearer $KONNECT_SYSTEM_ACCESS_TOKEN'
Content-Type: 'application/json'
timeout: 30s
retry_period: 15s
retries: 3
max_retry_backoff: 1m
max_in_flight: 64
batch_as_multipart: false
drop_on:
- 400
batching:
count: 100
period: 1s
processors:
- metric:
type: counter
name: openmeter_events_sent
value: 1
- archive:
format: json_array
dump_request_log_level: DEBUG
Copied!
Replace $KONNECT_SYSTEM_ACCESS_TOKEN with your own system access token.
Example use case
A fully working example is available on GitHub.
Installation
The Metering & Billing Collector (a custom Redpanda Connect distribution) is available via the following distribution strategies:
- Binaries can be downloaded from the GitHub Releases page.
- Container images are available on ghcr.io.
- A Helm chart is also available on GitHub Packages.