Skip to content

feat(codecs): Support more avro types #24774

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
omwbennett wants to merge 11 commits into
base: master
Choose a base branch
from
Open

feat(codecs): Support more avro types #24774

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
43dfd88
Support avro date values
omwbennett Feb 26, 2026
8f2d0fd
Add changelog fragment
omwbennett Feb 26, 2026
6ea80e7
Run make generate-component-docs
omwbennett Mar 3, 2026
b41b736
Fix avro fixture generation
omwbennett Apr 13, 2026
490291a
Make avro encoder aware of schemas
omwbennett Apr 22, 2026
8fb61ef
Support Avro Fixed, TimeMillis, TimestampNanos, LocalTimestampNanos t…
omwbennett Apr 22, 2026
53a9071
Update changelog
omwbennett Apr 22, 2026
20815d0
Add conversion of vrl int to avro float/double
omwbennett Apr 23, 2026
05045a9
Refactor to_avro to coerce_logical_types
omwbennett Apr 26, 2026
88a117b
fix(codecs): coerce logical types through schema refs
pront Jul 27, 2026
63ff827
test(codecs): add named Avro record reference fixture
pront Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog.d/24773_support_more_avro_types.fix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
The Avro codec now supports encoding and decoding Date, Fixed, TimeMillis, TimestampNanos, LocalTimestampNanos values.

authors: omwbennett
2 changes: 1 addition & 1 deletion lib/codecs/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ vector-config-macros = { path = "../vector-config-macros", default-features = fa
vector-core = { path = "../vector-core", default-features = false, features = ["vrl"] }
vector-vrl-functions.workspace = true
toml = { version = "0.9.8", optional = true }
uuid.workspace = true

[dev-dependencies]
criterion.workspace = true
Expand All @@ -74,7 +75,6 @@ similar-asserts = "1.7.0"
vector-core = { path = "../vector-core", default-features = false, features = ["vrl", "test"] }
rstest = "0.26.1"
tracing-test = "0.2.6"
uuid.workspace = true
vrl.workspace = true

[features]
Expand Down
25 changes: 6 additions & 19 deletions lib/codecs/src/decoding/format/avro.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,11 +91,8 @@ impl From<&AvroDeserializerOptions> for AvroSerializerOptions {
pub struct AvroDeserializerOptions {
/// The Avro schema definition.
/// **Note**: The following [`apache_avro::types::Value`] variants are *not* supported:
/// * `Date`
/// * `Decimal`
/// * `Duration`
/// * `Fixed`
/// * `TimeMillis`
#[configurable(metadata(
docs::examples = r#"{ "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] }"#,
docs::additional_props_description = r#"Supports most avro data types, unsupported data types includes
Expand Down Expand Up @@ -189,10 +186,8 @@ pub fn try_from(value: AvroValue) -> vector_common::Result<VrlValue> {
Ok(VrlValue::Array(vector))
}
AvroValue::Boolean(boolean) => Ok(VrlValue::from(boolean)),
AvroValue::Bytes(bytes) => Ok(VrlValue::from(bytes)),
AvroValue::Date(_) => Err(vector_common::Error::from(
"AvroValue::Date is not supported",
)),
AvroValue::Bytes(bytes) => Ok(VrlValue::Bytes(Bytes::from(bytes))),

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve non-UTF-8 Avro byte payloads

When an Avro bytes field contains arbitrary non-UTF-8 data, converting it to VrlValue::Bytes here makes the encoder's later apache_avro::to_value(log) pass serialize it through VRL's lossy string representation. For example, [0xff, 0x00] is silently re-encoded as [0xef, 0xbf, 0xbd, 0x00]; the new Fixed branch has the same problem and can instead fail when replacement bytes change the fixed length. Avro binary values are not restricted to UTF-8, so the serializer must preserve VRL bytes through a schema-aware conversion rather than the lossy Serde path.

Useful? React with 👍 / 👎.

@pront pront Jul 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @omwbennett, apologies for the delay. I spend a little time on this PR but this is actually an important issue. If you are still interesting in completing this, happy to help reviewing. Otherwise, I think we want to split this into smaller PRs.

The fix here is to do the following:

(VrlValue::Bytes(bytes), Schema::Bytes) => Ok(AvroValue::Bytes(bytes.to_vec())),

@omwbennett omwbennett Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for taking another look. I can split this up into smaller PRs.

@omwbennett omwbennett Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting with #26000

@omwbennett omwbennett Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next is the introduction of coerce_logical_types to support a few simpler logical types (date/timemillis): #26112
Will add support for the remaining logical types in a follow-up PR.

AvroValue::Date(days) => Ok(VrlValue::from(days)),

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Regenerate Splunk HEC's Avro support docs

For users configuring either Splunk HEC decoding path, website/cue/reference/components/sources/generated/splunk_hec.cue still says that Date, Fixed, and TimeMillis are unsupported at lines 110-114 and 654-658, even though this commit updates the same generated description in the other source pages. Regenerate this component page as well so the published options do not contradict the newly supported branches.

AGENTS.md reference: AGENTS.md:L212-L212

Useful? React with 👍 / 👎.

AvroValue::Decimal(_) => Err(vector_common::Error::from(
"AvroValue::Decimal is not supported",
)),
Expand All @@ -201,9 +196,7 @@ pub fn try_from(value: AvroValue) -> vector_common::Result<VrlValue> {
"AvroValue::Duration is not supported",
)),
AvroValue::Enum(_, string) => Ok(VrlValue::from(string)),
AvroValue::Fixed(_, _) => Err(vector_common::Error::from(
"AvroValue::Fixed is not supported",
)),
AvroValue::Fixed(_, bytes) => Ok(VrlValue::Bytes(Bytes::from(bytes))),
AvroValue::Float(float) => Ok(VrlValue::from_f64_or_zero(float as f64)),
AvroValue::Int(int) => Ok(VrlValue::from(int)),
AvroValue::Long(long) => Ok(VrlValue::from(long)),
Expand All @@ -220,9 +213,7 @@ pub fn try_from(value: AvroValue) -> vector_common::Result<VrlValue> {
.map(|v| VrlValue::Object(v.into_iter().collect())),
AvroValue::String(string) => Ok(VrlValue::from(string)),
AvroValue::TimeMicros(time_micros) => Ok(VrlValue::from(time_micros)),
AvroValue::TimeMillis(_) => Err(vector_common::Error::from(
"AvroValue::TimeMillis is not supported",
)),
AvroValue::TimeMillis(time_millis) => Ok(VrlValue::from(time_millis)),
AvroValue::TimestampMicros(ts_micros) => Ok(VrlValue::from(ts_micros)),
AvroValue::TimestampMillis(ts_millis) => Ok(VrlValue::from(ts_millis)),
AvroValue::Union(_, v) => try_from(*v),
Expand All @@ -232,12 +223,8 @@ pub fn try_from(value: AvroValue) -> vector_common::Result<VrlValue> {
AvroValue::BigDecimal(_) => Err(vector_common::Error::from(
"AvroValue::BigDecimal is not supported",
)),
AvroValue::TimestampNanos(_) => Err(vector_common::Error::from(
"AvroValue::TimestampNanos is not supported",
)),
AvroValue::LocalTimestampNanos(_) => Err(vector_common::Error::from(
"AvroValue::LocalTimestampNanos is not supported",
)),
AvroValue::TimestampNanos(ts_nanos) => Ok(VrlValue::from(ts_nanos)),
AvroValue::LocalTimestampNanos(ts_nanos) => Ok(VrlValue::from(ts_nanos)),
}
}

Expand Down
Loading
Loading