feat(codecs): Support more avro types by omwbennett · Pull Request #24774 · vectordotdev/vector

omwbennett · 2026-02-26T16:07:50Z

Summary

Addresses #24773 by adding support for decoding Avro Date values, in addition to Fixed, TimeMillis, TimestampNanos, LocalTimestampNanos types.
Due to the way VRL stores integers, an extra step was added to the avro encoder path to coerce Avro types based on the schema field's LogicalType.

Vector configuration

sources:
  kafka_avro:
    type: kafka
    bootstrap_servers: "kafka:9092"
    group_id: "vector"
    topics:
      - test_topic
    decoding:
      codec: avro
      avro:
        schema: |
          {
            "type": "record",
            "name": "test",
            "fields": [
              {
                "name": "date_field",
                "type": { "type": "int", "logicalType": "date" }
              }
            ]
          }
        strip_schema_id_prefix: false

sinks:
  console_out:
    type: console
    inputs:
      - kafka_avro
    encoding:
      codec: json

Also used similar configs for the other Avro data types.

How did you test this PR?

Change Type

Bug fix
[x New feature
Dependencies
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Closes #24773

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

github-actions · 2026-02-26T16:08:06Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

omwbennett · 2026-02-26T16:25:04Z

I have read the CLA Document and I hereby sign the CLA

omwbennett · 2026-04-02T13:23:53Z

Hey @vectordotdev/vector, appreciate if you could review this PR, thanks!

pront · 2026-04-02T15:37:39Z

@codex review

Hi @omwbennett, thanks for contributing to Vector. I enabled the workflows and requested a first pass from codex.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 857b4b3903

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb2cbd735b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5336c62721

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

pront · 2026-04-24T20:29:51Z

Hi @omwbennett, happy to review this once all codex comments are addressed (feel free to decline those with a justification if they do not make sense)

omwbennett · 2026-04-27T09:28:54Z

Hi @omwbennett, happy to review this once all codex comments are addressed (feel free to decline those with a justification if they do not make sense)

Thanks @pront, I have addressed all the codex review comments now, appreciate if you could take another look.

…ypes

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05045a98ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-27T17:45:57Z

+                None => Ok(value),
+            }
+        }
+        (value, _) => Ok(value),


Resolve named schemas before coercing logical fields

When a record containing a date or time-millis field is reused through an Avro named-type reference, its field schema reaches this fallback as Schema::Ref, so the nested VRL integer remains AvroValue::Long. The subsequent schema resolution then rejects it (for example, a second field of type "Inner" fails with Expected Value::Date or Value::Int, got: Long(...)), meaning valid schemas using named records cannot encode the newly supported logical types. Resolve references during this recursive pass, using the root schema's names map, before descending into the referenced schema.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-27T17:45:57Z

-        AvroValue::Date(_) => Err(vector_common::Error::from(
-            "AvroValue::Date is not supported",
-        )),
+        AvroValue::Bytes(bytes) => Ok(VrlValue::Bytes(Bytes::from(bytes))),


Preserve non-UTF-8 Avro byte payloads

When an Avro bytes field contains arbitrary non-UTF-8 data, converting it to VrlValue::Bytes here makes the encoder's later apache_avro::to_value(log) pass serialize it through VRL's lossy string representation. For example, [0xff, 0x00] is silently re-encoded as [0xef, 0xbf, 0xbd, 0x00]; the new Fixed branch has the same problem and can instead fail when replacement bytes change the fixed length. Avro binary values are not restricted to UTF-8, so the serializer must preserve VRL bytes through a schema-aware conversion rather than the lossy Serde path.

Useful? React with 👍 / 👎.

Hi @omwbennett, apologies for the delay. I spend a little time on this PR but this is actually an important issue. If you are still interesting in completing this, happy to help reviewing. Otherwise, I think we want to split this into smaller PRs.

The fix here is to do the following:

(VrlValue::Bytes(bytes), Schema::Bytes) => Ok(AvroValue::Bytes(bytes.to_vec())),

Hi, thanks for taking another look. I can split this up into smaller PRs.

Starting with #26000

Next is the introduction of coerce_logical_types to support a few simpler logical types (date/timemillis): #26112
Will add support for the remaining logical types in a follow-up PR.

chatgpt-codex-connector · 2026-07-27T17:45:57Z

+                    };
+                    Ok((name, value))
+                })
+                .collect::<vector_common::Result<_>>()?;


Avoid rebuilding every Avro event during coercion

Although the final version no longer rebuilds ResolvedSchema, it still sends every event through this branch: apache_avro::to_value(log) produces a map for log events, and this collect allocates a replacement map after walking every field, recursively rebuilding nested maps and arrays even when the schema contains no Date or TimeMillis. Nullable unions additionally deep-clone their values before the final resolution traverses them again. In high-throughput pipelines with ordinary schemas or large messages, this adds per-event allocations and copies unrelated to the feature; skip the pass when the schema needs no coercion or mutate containers in place.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63ff8274c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-27T18:17:58Z

+        let resolved =
+            apache_avro::schema::ResolvedSchema::try_from(&self.schema).map_err(|error| {
+                vector_common::Error::from(format!("Failed resolving Avro schema: {error}"))
+            })?;


Cache Avro name resolution across encoded events

In high-throughput Avro sinks, this rebuilds the schema's name map for every event, and value.resolve(&self.schema) immediately afterward constructs another ResolvedSchema, so each encoding traverses the schema twice before writing any data. Fresh evidence since the earlier thread is that the current diff has reintroduced ResolvedSchema::try_from(&self.schema) in encode; resolve or cache the names when constructing AvroSerializer, then reuse that state for coercion and final resolution.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-27T18:17:58Z

-            "AvroValue::Date is not supported",
-        )),
+        AvroValue::Bytes(bytes) => Ok(VrlValue::Bytes(Bytes::from(bytes))),
+        AvroValue::Date(days) => Ok(VrlValue::from(days)),


Regenerate Splunk HEC's Avro support docs

For users configuring either Splunk HEC decoding path, website/cue/reference/components/sources/generated/splunk_hec.cue still says that Date, Fixed, and TimeMillis are unsupported at lines 110-114 and 654-658, even though this commit updates the same generated description in the other source pages. Regenerate this component page as well so the published options do not contradict the newly supported branches.

AGENTS.md reference: AGENTS.md:L212-L212

Useful? React with 👍 / 👎.

omwbennett requested a review from a team as a code owner February 26, 2026 16:07

omwbennett force-pushed the support-avro-date-values branch 2 times, most recently from 40d0d6a to aa24899 Compare February 26, 2026 16:28

omwbennett requested a review from a team as a code owner March 3, 2026 14:59

github-actions Bot added the domain: external docs Anything related to Vector's external, public documentation label Mar 3, 2026

rtrieu self-assigned this Mar 3, 2026

rtrieu approved these changes Mar 3, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Apr 2, 2026

View reviewed changes

Comment thread lib/codecs/tests/data/avro/generated/date.avsc Outdated

github-actions Bot added the work in progress label Apr 22, 2026

chatgpt-codex-connector Bot reviewed Apr 22, 2026

View reviewed changes

Comment thread lib/codecs/src/encoding/format/avro.rs Outdated

github-actions Bot added the domain: ci Anything related to Vector's CI environment label Apr 23, 2026

chatgpt-codex-connector Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread lib/codecs/src/encoding/format/avro.rs Outdated

omwbennett changed the title ~~fix(codecs): Support avro date values~~ feat(codecs): Support more avro types Apr 27, 2026

omwbennett added 9 commits July 27, 2026 13:30

Support avro date values

43dfd88

Add changelog fragment

8f2d0fd

Run make generate-component-docs

6ea80e7

Fix avro fixture generation

b41b736

Make avro encoder aware of schemas

490291a

Support Avro Fixed, TimeMillis, TimestampNanos, LocalTimestampNanos t…

8fb61ef

…ypes

Update changelog

53a9071

Add conversion of vrl int to avro float/double

20815d0

Refactor to_avro to coerce_logical_types

05045a9

pront force-pushed the support-avro-date-values branch from e1dc4da to 05045a9 Compare July 27, 2026 17:33

github-actions Bot added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Jul 27, 2026

github-actions Bot removed the domain: ci Anything related to Vector's CI environment label Jul 27, 2026

pront removed the work in progress label Jul 27, 2026

chatgpt-codex-connector Bot reviewed Jul 27, 2026

View reviewed changes

pront added 2 commits July 27, 2026 14:02

fix(codecs): coerce logical types through schema refs

88a117b

test(codecs): add named Avro record reference fixture

63ff827

chatgpt-codex-connector Bot reviewed Jul 27, 2026

View reviewed changes

pront added the meta: awaiting author Pull requests that are awaiting their author. label Jul 27, 2026

Uh oh!

Conversation

omwbennett commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

github-actions Bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

omwbennett commented Feb 26, 2026

Uh oh!

omwbennett commented Apr 2, 2026

Uh oh!

pront commented Apr 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pront commented Apr 24, 2026

Uh oh!

omwbennett commented Apr 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

pront Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

omwbennett Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

omwbennett Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

omwbennett Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

omwbennett commented Feb 26, 2026 •

edited

Loading

github-actions Bot commented Feb 26, 2026 •

edited

Loading