Project Proposal: Audit Logging SIG

[mlenkeit]

This PR contains a project proposal for an Audit Logging SIG as discussed on Slack.

We are aware that the project proposal still has several tbd's especially with regard to staffing and timeline that need to be defined before the SIG can start working.

We will approach other vendors directly with this proposal to identify additional contributors. Of course, anyone who comes across this proposal here on GitHub is invited to contribute.

While we do have some ideas about a potential timeline for semantic conventions, OTEL SDK/API and collector adjustments respectively, we would like to align this with other contributors first before publishing.

Any feedback from the community on the proposed scope of the SIG is highly appreciated!

Open topics

The following items reference topics from the PR discussion that are still open:

• immutability, tamper-proof logs, signing - in scope?
  • Project Proposal: Audit Logging SIG #2409 (comment)
  • Project Proposal: Audit Logging SIG #2409 (comment)
  • Project Proposal: Audit Logging SIG #2409 (comment)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: mlenkeit / name: Maximilian Lenkeit (65ae32e, 5094fb1, 3876a31, 776b821, d7e265f, a5ef343, 75f2c57, 087865c, 2ec002d, 9337b7f, 70cbac4, 711dc46, f81c2f4, 0adb8e5, a6b34f1, 066501b, 6bc9a5e, 8b38626, 6dd519d, e03dcd8, 41329f3, 405ddb5, 86bd77a)
• ✅ login: hilmarf / name: Hilmar Falkenberg (2f9813f, 03d1de1)

[mtwo]

Suggested change

	- no semantic conventions for audit logs in OTEL
	- There aren't currently any semantic conventions designed specifically for audit logs in OTEL

[mtwo]

Are there any requirements around signing logs / detecting tampering? I've heard that mentioned before in the context of audit logs, but I don't know how common of a requirement it is

[reyang]

Good points! In addition, these are something we might want to consider:

1. Audit logs might be considered as a critical part of the business, which could result in a different API design strategy - for example, audit logging might require a different API behavior, if the information provided by the caller is invalid, the API might throw exception instead of failing silently and move on.
2. Audit log might require some sensitive information without redaction due to the regulation requirements (e.g. user identity and client IP address).
3. The data path could require higher level of access control or privilege.

[mlenkeit]

@reyang thanks for mentioning these points.

Especially the API behavior is something that we had thought about initially. However, when we first pitched audit logging on Slack, we received the following comment from Ted Young:

As a rule, the OpenTelemetry API never throws an exception. I understand why you might want this, though it is not present in many audit logging systems, which use regular loggers. So a strong case would have to be made on this particular point.

Based on this initial feedback, we decided to file this SIG proposal without proposing such API changes.

[reyang]

I think we need prototype in two parts:

1. API/SDK - this is where we need three programming languages IIRC.
2. OTel Collector - higher guarantee on data delivery (completeness, integrity, latency, etc.), data path security.

[mlenkeit]

Thanks for pointing this out! It's clear to us, but I'll work on making this clearer in the doc...

[mlenkeit]

Are there any requirements around signing logs / detecting tampering? I've heard that mentioned before in the context of audit logs, but I don't know how common of a requirement it is

@mtwo for all I know, immutability of audit logs is a common requirement although not all audit logging systems/use cases that I've seen address this requirement with technical measures but sometimes also organizational measures. However, given the flexibility of OTel processing queues (i.e. different topologies of collectors), having a technical solution in OTel would be favorable.

@reyang what is your opinion on this?

[svrnm]

Suggested change

	- no semantic conventions for audit logs in OTel
	- no semantic conventions for audit logs in OTel

Can you provide some examples of what would be part of such semantic conventions? My knowledge on audit logs is very limited, so it would help to understand the problem much better.

[mlenkeit]

@svrnm our experience has shown that in order to analyze audit logs at scale, it is important to define an (extensible) event catalog. The event catalog standardizes audit log events across workloads/produces. For example, our internal event catalog currently consists of 50+ such events. Ideally, such a catalog would be part of semantic conventions.

To make this more tangible, I've added some examples to the appendix of the document:
https://github.com/open-telemetry/community/pull/2409/files#diff-736e6b0ae9ae655b78d9ba007d08592071abb6cc1ef64d7893ff81642c8ec734R115-R192

[renewelches]

another examples from the security world is https://github.com/ocsf.

[svrnm]

thanks @mlenkeit. Makes it much clearer

The metadata looks like attributes that would be covered by other semantic conventions (e.g. there is a log.record.id for the metadata.id, the timestamp of course and some of the other ones (e.g. for k8sCluster we have k8s.cluster.name. So I would assume here it is more about re-using and extending certain other domains that are not unique to "audit logs"

For the event and data examples you gave, I would argue that they are not "semantic conventions for audit logs" but "semantic conventions for log types that typically require the strict requirements of auditing". What do I mean by that: if we talk about "semantic conventions for audit logs" I think about a namespace called audit. that holds attributes that are specific to the business logic of audit logging, like a signature that helps to tamperproof the log line, or maybe even meta information under which regulation this log is required to be an "audit log"
In contrast "semantic conventions for log types that typically require the strict requirements of auditing" are their own namespaces like the "UserLoginFailure" example would fall into a "authentication" or "auth" namespace, with "auth.login.method" or "auth.login.failureReason" as potential attributes, event.name being set to auth.login.failure or something.

I am just making those things up to exemplify the difference, they will probably take a different form or shape eventually, so to make a long story short, here is a suggestion to rephrase:

Suggested change

	- no semantic conventions for audit logs in OTel
	- no semantic conventions for audit logs in OTel
	- no semantic conventions for log types that typically rquire the strict requirements of auditing, like authentication, authorization and data changes

@renewelches thanks for calling out OCSF, if I remember correctly there were conversations in the past between OTel and OCSF, cc @lmolkova

[mlenkeit]

Regarding metadata, I fully agree: Most of these attributes are already covered by semconv. We may identify additional attributes in SIG meetings though, depending on the experience/requirement of other contributors/companies.

I understand how "semantic conventions for audit logs" can be misleading. To me, the suggestion that you made has a notion of particularly describe logs that are "already there" (e.g. events emitted by a K8s cluster) and can be considered relevant for audit purposes. Especially in enterprise software, it's common that applications produce logs that are specifically mean to be audit logs (and nothing else). To me, it' s important that we find wording that covers these two types that we do have.

How about the following?

Suggested change

	- no semantic conventions for audit logs in OTel
	- no semantic conventions for representing and identifying audit trail-relevant events in OTel (like authentication, authorization or modification of

[renewelches]

As mentioned in another comment, this all depends on what attributes are changeable or must be immutable. As of my understanding an attribute could be altered by a processor in the collector. Which is something we would want to avoid or want to prevent in cases of audit logs. If we conclude that we can or should only guarantee immutability for the log itself then we must live with replication/doublication. Otherwise we might have to add the constrain that also certain attributes must be immutable.

[tigrannajaryan]

+1 to looking into OCSF for security events and borrowing relevant semantic conventions from there.

[trisch-me]

@mlenkeit security semconv SIG is also working on introducing security related attributes to the semconv, we have currently discussion around user authentication. I feel that this SIG and security SIG might have some common tasks/discussions

[hilmarf]

Component donation: signingprocessor - open-telemetry/opentelemetry-collector-contrib#50079

[MarkovianProtocol]

We published a pinned canonical form and hash for OTel spans with conformance vectors: https://github.com/MarkovianProtocol/otel-canonical-profile at 4f8c4c5. An outside implementer ran both validators and got 18/18 in Python and Node. Spans rather than log records, so not a drop-in — I can produce the equivalent vectors for the fields the processor canonicalizes if that's useful.

[Levaj2000]

A datapoint for the open scope question ("immutability, tamper-proof logs, signing — in scope?"): OCSF 1.9.0, released last week, ships the same primitive that this proposal's draft data model and the signingprocessor donation (open-telemetry/opentelemetry-collector-contrib#50079) describe — a signature over a canonical form of the record, plus a hash link to the previous record. Disclosure: I contributed the OCSF side (the attestation object and record_integrity profile).

Many pipelines will end up touching both — OTel as transport, OCSF as the envelope in the SIEM — so here is the field mapping while both vocabularies are still cheap to align:

Audit Logging draft	OCSF 1.9.0
canonical form: RFC 8785 (JCS)	digital_signature.serialization_id (enum: JCS, JWS, COSE, DSSE)
audit.integrity.algorithm	digital_signature.algorithm_id / algorithm
audit.integrity.certificate	digital_signature.certificate
audit.integrity.value (signature bytes)	no landed equivalent yet — proposed in ocsf/ocsf-schema#1709
audit.sequence.prev_hash	attestation.prev_event.fingerprint
audit.sequence.number	no direct equivalent (prev_event.uid links by id, not counter)
—	attestation.chain_uid, attestation.authority_uid — no draft equivalent

Two observations from having shipped the OCSF shape, offered as input rather than objections:

1. sequence.number + prev_hash give continuity within a stream, but nothing names the stream or says who vouches for it. We added chain_uid/authority_uid after hitting exactly this in multi-tenant verification: records signed under different keys interleave in one pipeline, and the verifier needs to know which chain a record claims membership in before it can check the link.
2. Worth deciding early where the signature is produced. A Collector-side processor attests custody from the pipeline onward; producer-side signing attests from the point of action. Both are legitimate tiers — but the record should say which one a consumer is getting.

Happy to contribute the mapping as a doc, or conformance vectors in either direction, if the SIG forms and wants them.

[hilmarf]

Two observations from having shipped the OCSF shape...

Thanks a lot @Levaj2000 for your inputs and experience sharing with OCSF. I've tried to incorporate them with apeirora/opentelemetry-specification@578b930. Happy to get your feedback.

Given your hands-on experience with OCSF, it would be great to have you as a supporter of this SIG — your perspective on the intersection of OCSF and OTel would be really valuable as we shape the audit logging signal. Would you be open to joining us?

[Levaj2000]

Thanks @hilmarf — happy to join as a supporter; please add me. The corner I can usefully hold is the OCSF↔OTel intersection: we run the OCSF side of this in production (hash-chained, per-event-signed audit exports under the record_integrity profile that landed in OCSF 1.9 via ocsf/ocsf-schema#1661), and my interest is that the two specs end up mutually verifiable rather than parallel.

I read through 578b930 — both additions are the right calls. audit.sequence.stream_id resolves the demultiplexing problem cleanly, and the audit.integritysigner language that producer and collector signatures "MUST NOT be treated interchangeably" is exactly the discipline this needs: point-of-action attestation and custody attestation answer different questions, and we learned to keep them separate the slow way.

A few notes from having shipped this shape, in roughly descending order of how much round-trip they might save you:

A chain pointer needs a resolvable record identifier, not only a hash. A predecessor hash proves linkage but can't locate the predecessor across storage, sharding, or retention boundaries. OCSF's merged shape ended up requiring every event to carry metadata.uid and the previous-record pointer to reference it (prev_event.uid), with the hash binding the reference to content. I'd recommend the data model require a per-record id within the stream and have the chain pointer carry both id and hash.

Define genesis explicitly: the first record of a stream omits the previous-record pointer entirely — never a sentinel. We shipped the sentinel version once (a literal "GENESIS" string inside a hash-valued field) and have it documented as an anti-pattern: a hash field whose value isn't a hash breaks every schema-driven verifier. One sentence of spec text here saves every implementer that round trip.

Signatures and chain hashes are only third-party-verifiable if the record declares what bytes were hashed. I'd recommend a required companion attribute naming the canonicalization for audit.integrity.value — RFC 8785/JCS, flat bytes, or a named producer scheme. OCSF landed on an enum plus a free-text sibling precisely so producers whose serialization isn't JCS can say so honestly instead of misclaiming; without it, "verify the signature" quietly becomes "trust the producer."

Offer: a written OCSF↔OTel crosswalk. The mappings look near-lossless from here — audit.sequence.stream_id ↔ attestation.chain_uid, the producer/collector signer split ↔ producer signatures vs custody attestations with authority_uid naming the attesting party, the chain pointer ↔ prev_event. If we keep that mapping tight, one producer can emit both shapes (or transform between them) without divergent integrity constructs. Happy to draft it as a doc for the spec repo if useful.

We can also contribute test vectors for the verification-procedures section: a production OCSF export (236-event chain, per-event ECDSA-P256 signatures verifiable against a public JWKS, no secrets required) and a stdlib-only worked sample with fully recomputable fingerprints — both public:

https://github.com/levaj2000/ai-identity/tree/main/docs/cosai-ws4-ocsf-mapping/ocsf-log-reference-bundle
https://github.com/levaj2000/ai-identity/tree/main/docs/cosai-ws4-ocsf-mapping/trust-base-inventory-sample

Looking forward to it. Cheers

[atoulme]

Offer: a written OCSF↔OTel crosswalk. The mappings look near-lossless from here — audit.sequence.stream_id ↔ attestation.chain_uid, the producer/collector signer split ↔ producer signatures vs custody attestations with authority_uid naming the attesting party, the chain pointer ↔ prev_event. If we keep that mapping tight, one producer can emit both shapes (or transform between them) without divergent integrity constructs. Happy to draft it as a doc for the spec repo if useful.

Interested in this. I dabbled into mapping logs to OCSF events here: open-telemetry/opentelemetry-collector-contrib#47461

[surpradhan]

I maintain otel-agent-audit, a signed-audit Collector component for AI agent traces, listed in the OTel Registry: canonical serialization, hash chaining, Ed25519 signing, and a standalone verifier. Most of the draft data model is something I've implemented and had to operate, so I'd rather contribute that experience than duplicate the work.

Three findings that bear on the draft.

Arrival order can't be the chain order. Collector batching and concurrency mean arrival order isn't reproducible. I order by a content-derived key before hashing, so a verifier can re-derive the same chain from the same records. Anything chaining on arrival sequence is reproducible only for the process that wrote it.

Sealing needs an explicit completeness boundary. I chain per trace, and a trace seals when its root span arrives. If the root arrives before its children, those children are dropped: the chain is internally valid but incomplete. I needed a separate buffering processor upstream to hold this together. Discrete events may not hit this, but anything with a parent/child or session boundary will.

Rotation is the gap I'd most like the spec to close. With audit.sequence.prev_hash and now stream_id, a chain crossing a key rotation is linked structurally, but that link is signed by the incoming key. Nothing signed by the outgoing key attests to the handover, so a legitimate rotation and a truncation with a rotation-shaped hole are hard to tell apart. I filed it against my own implementation here and don't have a good answer yet. It seems worth settling at spec level rather than per implementation.

Two things I could contribute if useful: signed checkpoints over chain tips, for cross-stream completeness rather than only per-record continuity; and verifier tooling, since for an audit format the verifier is arguably the product and it's the thinner half right now.

@hilmarf @svrnm I'd like to join as a supporter if you're still taking them. The corner I can usefully hold is AI agent traces, including the bridge to the GenAI SIG's in-flight guardrail work (open-telemetry/semantic-conventions-genai#262, open-telemetry/semantic-conventions-genai#427), where a policy decision is exactly the kind of event that should land in a verifiable record.

[Levaj2000]

@atoulme

Great — the draft is already underway: OCSF↔OTel audit crosswalk + derived test vectors. Field-level mapping between the audit data model draft and OCSF 1.9's record_integrity shape, the places it isn't 1:1 with what a transform must do there, and OTel-shaped vectors derived from our production OCSF export — chain linkage re-verifies across the transform (235/235 links on the 236-event export), and the signatures survive byte-for-byte, verifiable against the same public JWKS.

The connector angle is exactly why it's worth writing down: with the mapping explicit, collector-contrib#47461-style translation becomes a mechanical transform instead of a design exercise, and the integrity constructs survive rather than being rebuilt divergently on each side. One finding worth flagging for a connector specifically: a signature is bound to the canonical bytes of its origin shape — a transform can carry the proof (origin digest + declared canonicalization) but can never re-sign, which is what "lossless" can and can't mean here.

Would value your review against what the connector actually needed. @hilmarf — the doc ends with four one-attribute spec asks that would make the mapping fully lossless (three OTel-side, one already filed on the OCSF side); happy to bring it into the spec repo in whatever form suits.

Cheers, Jeff

[hilmarf]

@surpradhan Thanks a lot for your inputs!

I've added the rotation and sealing problems as open questions in the OTEP. apeirora/opentelemetry-specification@165dd2b

Regarding arrival order: I think stream_id (relatively new - apeirora/opentelemetry-specification@578b930) + sequence.number should be sufficient to reconstruct the chain deterministically, regardless of arrival order. What do you think?

When multiple SDK instances fan-in (different stream_ids) - there's no global ordering across streams, true. Is this something you ran into?

Unfortunately this SIG hasn't started yet, due to other higher priorities from OTel. The more I'm happy to have you onboard with your valuable experience.

[surpradhan]

Thanks @hilmarf, and good to see the sealing and rotation points land as open questions in the OTEP.

stream_id and sequence.number are enough to reconstruct the chain for source-emitted streams. When a single AuditLogger assigns the sequence and owns the stream_id, arrival order at the Collector doesn't matter: a verifier demuxes by stream_id, sorts by sequence.number, and checks prev_hash. The caveat I raised only applies when records are derived downstream from telemetry that arrived out of order, which is my case: building records from agent spans in the Collector. There's no source-assigned counter there, so I order by a content-derived key (start_time_unix_nano, span_id) before hashing, and a verifier re-derives the same chain from the same records. For that derived case, the order has to come from record content rather than arrival order, or two honest rebuilds can disagree. That's probably its own profile rather than something the base signal needs, since a source-emitted stream never encounters it.

A dropped stream leaves no trace in the streams that survive. Within a stream, sequence.number catches gaps, but a whole stream that's lost or truncated leaves every other stream intact, so per-record continuity can't detect it. What I've leaned on is periodic signed checkpoints that form their own hash chain: a stream's tip {stream_id, tip hash, entry count} is committed when it seals, batched every N seals, with each checkpoint linking to the previous one. A verifier replays that chain and re-derives every committed tip from the log, so a stream that reached a checkpoint can't be dropped or truncated without causing a tip mismatch or a break in the checkpoint chain. Honest bounds: a stream dropped before its checkpoint is written, or one that never seals, isn't covered. So this narrows the completeness window to the checkpoint interval and also serves as a batched seal record, complementary to the terminal-record idea rather than replacing it.

Only a co-signed transition record makes the outgoing key attest to a rotation. Of the three options, that's the one that actually closes the gap. A per-record key_id and a JWKS pointer let a verifier map records to keys, but neither has the outgoing key sign the handover, so a legitimate rotation still looks like a rotation-shaped truncation. I don't have this solved in my own implementation yet, but that's the direction I'd back, and I'm happy to prototype it against a live chain and report what breaks.

[hilmarf]

... the doc ends with four one-attribute spec asks that would make the mapping fully lossless (three OTel-side, one already filed on the OCSF side)

Hi @Levaj2000, I hope this (apeirora/opentelemetry-specification@7cac2e1) does address the three open points for OTel. Happy to hear your thoughts.

[MarkovianProtocol]

Rotation and the checkpoint idea are the same gap. sequence.number, prev_hash and the signature all come from one key, so a producer can keep two sets of books under one stream_id — one chain served to you, another to whoever checks later — and every record in both verifies. What separates them isn't in the record. It's a second observer.

So the checkpoint signature has to come from outside the producer. Cosigners that remember the last tip they saw for a stream, and refuse a new one inconsistent with it, make the second book uncollectable — one fork can't get signed. Rotation falls out of the same move: a cosigner holding the pre-rotation tip won't sign across the boundary unless the outgoing key signed the handover.

It attests the stream is singular and append-only, not that any record is true, and covers nothing before a stream's first checkpoint. Running live: curl https://log.markovianprotocol.com/checkpoint — RFC 6962, tree size 7897, eight independent witnesses.

Happy to write it against the OTEP's audit.sequence attributes, with vectors.

[Levaj2000]

Thanks @hilmarf — read through 7cac2e1 and the follow-ups on the auditing branch. All three asks land:

1. audit.integrity.canonicalization — jcs as the assumed default plus the explicit-set recommendation is the right split. Verifiers no longer guess, and the section text names the failure modes honestly.
2. The resolvable chain pointer — and since audit.record.id is MUST, the locator half rests on a required field, which is exactly what a transform across storage, sharding, and retention boundaries needs.
3. Genesis by omission — this came out stronger than my ask: MUST omit, and receivers MUST NOT require a magic constant. Good.

One note for the thread: I reviewed at the branch tip, where these now read previous_hash / previous_record_id after the clarity renames — flagging in case anyone lands on this comment from 7cac2e1's diff and wonders about the names.

Also noticed audit.sequence.end landing today — that's a graceful answer to the sealing question @surpradhan raised: a defined completeness boundary for clean shutdowns, with crash truncation honestly "unknown" rather than silently passing. I'll check whether the crosswalk needs an end-signal row and update the doc.

One more observation on the open rotation question below: the thread is converging usefully — @surpradhan's outgoing-key-signed handover plus @MarkovianProtocol's framing of rotation and checkpoints as one fork-resistance gap, with the checkpoint cosigner sitting outside the producer. That last part is the load-bearing move: one producer's keys can sign both books. Happy to review the OTEP text when it lands.

Cheers, Jeff

[surpradhan]

@MarkovianProtocol, you're right about my checkpoints. The same Ed25519 key signs the entries and the checkpoints, so a producer holding it can serve two consistent chains and both verify. My threat model §2 already lists split-view as undefended with witnesses as the upgrade path, so it isn't news on my side, but better said here than letting the checkpoint idea sound stronger than it is. The narrower thing they do buy, and only against an adversary who doesn't hold the key, is detecting a whole stream going missing, which per-record continuity can't, since a dropped stream leaves every surviving stream intact. Singularity needs the outside observer.

Agreed too that a cosigner won't sign across a rotation unless the outgoing key signed the handover. The part I'd add is that the dependency only runs one way. The attacker in my rotation issue holds the incoming key and not the outgoing one, and cuts across a boundary so it looks like a routine rotation. A co-signed transition record stops that on its own, with no witness anywhere in the picture. So option (a) under open question 8 can be settled now on its own merits, rather than waiting on whatever checkpoint and witness design lands later, and it makes that design easier when it arrives.

On your span profile: it pins a different canonical form from mine by design, so running its vectors against my form as-is would just disagree on every byte. What's actually useful is implementing your profile alongside my own serialization and checking I reproduce your hashes and your reject cases, then writing up where the two forms deliberately differ. Your choice to string-encode nanosecond timestamps is the same 2^53 boundary I've just filed against the signingprocessor as open-telemetry/opentelemetry-collector-contrib#50881, where nanosecond timestamps and int64 attributes are currently bare JSON numbers going through JCS. Happy to do that and report back.

[MarkovianProtocol]

@surpradhan

You're right about the direction of the dependency, and it changes the ordering.
The attacker in your issue 19 holds the incoming key and not the outgoing one, so
a transition record signed by the outgoing key refuses that case on its own, with
no witness anywhere in it. Option (a) under open question 8 stands on its own
merits and does not need to wait on a checkpoint design.

Your narrowing of what checkpoints buy is the accurate one. Against an adversary
who does not hold the signing key, a checkpoint over {stream_id, count, tip}
detects a whole stream going missing, which per-record continuity cannot, because
a dropped stream leaves every surviving stream intact. Non-equivocation is the
part that needs the second observer.

On running our profile alongside your serialization: spec, pass and reject
vectors, and two dependency-free validators (Python stdlib, Node) are at
https://github.com/MarkovianProtocol/otel-canonical-profile. The places the two
forms will disagree are deliberate, so they are worth naming before you spend
time deriving them:

• start_time_unix_nano and end_time_unix_nano are decimal strings, and a JSON
  number there is a reject (TIMESTAMP_NOT_STRING) rather than a coercion. Same
  2^53 boundary you filed as collector-contrib#50881.
• int64 attributes are decimal strings inside a typed wrapper, {"i": "-12"}.
  Every attribute value carries a one-key type tag, so {"i":"1"} and {"s":"1"}
  are different bytes instead of relying on JSON's own types to separate them.
• Attributes are folded in behind a second-level digest, attributes_sha256, so
  a verifier whose store rewrote attribute bytes can still check the span core.
  It also means our two forms can disagree on attributes without disagreeing on
  the core.
• Span fields are an allowlist; status.message is outside hash scope.
• Duplicate attribute keys, NaN and +/-Infinity doubles, and uppercase or
  wrong-length ids are rejects, not normalizations.

One practical note so it doesn't read as a hole in the vector set:
NONFINITE_DOUBLE has no JSON vector, because NaN and Infinity are not
representable in JSON transport. It is reachable only through protobuf ingestion,
and the spec says implementations must still enforce it there.

Worth having the write-up cover the cases where our form is the one that is
wrong, not only where they differ.

Drafted with AI assistance.