feat(observability): agent-level trace collection via supervisor OTLP relay

[rhuss]

Problem Statement

Agents running inside network-isolated sandboxes cannot export traces to external collectors. The sandbox's egress policy blocks direct OpenTelemetry Protocol (OTLP) export, and requesting per-sandbox policy exceptions for collector endpoints is operationally impractical and a security concern. As a result, agent developers lose visibility into tool calls, LLM invocations, and reasoning steps when running inside OpenShell sandboxes.

This gap affects three personas:

1. Agent developers cannot debug reasoning failures because their LLM observability tools (MLflow, Langfuse) receive no traces from sandboxed agents
2. Platform operators can see supervisor network activity (via #2508) but cannot correlate it with the agent's behavior that caused it
3. Workspace administrators in multi-tenant deployments cannot route agent traces per tenant

The supervisor already sits between the agent process and the outside world, handling network interception, process supervision, and policy enforcement. It is the natural place to relay agent telemetry out of the sandbox.

Persona Workflows

Agent developer: seeing tool calls and LLM invocations without platform-specific code

A developer runs a LangChain agent inside an OpenShell sandbox. The agent makes tool calls to external APIs and LLM invocations to OpenAI. The developer wants to see these in their LLM observability tool (MLflow, Langfuse, or similar) to debug a reasoning failure. Today, the sandbox is network-isolated, so the agent's OTel SDK cannot reach the external collector. The developer either gives up on tracing or hacks an egress allowlist exception for the collector endpoint, which the security team will not approve.

How this issue enables the workflow: The supervisor automatically sets OTEL_EXPORTER_OTLP_ENDPOINT in the agent's environment. The agent's OTel SDK exports to a local address. The supervisor enriches the spans with sandbox context (sandbox ID, policy, user, workspace) and forwards them to the gateway, which relays them to the configured agent trace backend. The developer sees their traces in MLflow/Langfuse with full infrastructure context, without writing any OpenShell-specific code or requesting egress exceptions. This also works for short-lived agents (CI tasks, one-shot scripts) that exit before their OTel SDK's batch exporter would normally flush, because the supervisor outlives the agent process and drains any buffered spans before the sandbox tears down.

Platform operator: investigating repeated API rate limiting from a sandbox

An operator notices elevated 429 (rate limit) responses on outbound connections from sandbox sb-xyz in the supervisor's network spans. The supervisor traces show repeated CONNECT attempts to api.openai.com with 429 responses, but the operator cannot tell what the agent is doing that triggers this volume. The supervisor's infrastructure traces show the network activity but not the agent's reasoning.

How this issue enables the workflow: The operator filters spans in Tempo by openshell.sandbox.id = sb-xyz and finds the agent's traces alongside the supervisor's network spans. The agent trace reveals a call_llm tool invocation in a retry loop with no backoff. The operator reports the finding to the agent developer, who fixes the retry logic. Both the agent's tool call spans and the supervisor's network spans carry the same openshell.sandbox.id attribute, so the operator can correlate them without needing to know anything about the agent's internals.

Workspace administrator: routing traces per tenant

A workspace administrator manages a multi-tenant deployment. Team A's agent traces should go to their MLflow instance, Team B's agent traces should go to Langfuse. Today, all traces go to the same OTLP endpoint and the administrator has no way to separate them.

How this issue enables the workflow: The openshell.workspace.id resource attribute is attached to every span during enrichment. The administrator configures collector-side routing rules based on this attribute, or (as a future extension) configures per-workspace OTLP endpoints at the gateway. Each team sees only their own traces in their own backend.

Security/compliance reviewer: centralized OCSF event collection

A security reviewer needs to audit all network deny events across sandboxes for a compliance report. OCSF JSONL events are generated inside each sandbox but stay local. The reviewer has to SSH into each sandbox to collect them, which does not scale.

How this issue enables the workflow: The log relay extends the existing PushSandboxLogs mechanism to carry OCSF events from sandboxes to the gateway. The reviewer queries their centralized log aggregator and sees all deny events across all sandboxes, enriched with sandbox and workspace context.

Proposed Design

OTLP relay

The supervisor acts as a telemetry sidecar for the isolated sandbox. It listens on standard OTLP ports inside the sandbox, accepts spans from any OpenTelemetry (OTel)-instrumented agent framework, enriches them with sandbox context, and forwards them through the gateway to an external collector.

graph TD
    A["Agent process<br/>(inside sandbox)"] -->|"OTLP gRPC+HTTP<br/>(ports 4317/4318)"| B["Supervisor<br/>(telemetry relay)"]
    B -->|"Session protocol<br/>(traces, OCSF logs)"| C["Gateway<br/>(relay)"]
    C -->|"OTLP/gRPC"| D["Trace Collector"]
    D --> E["Agent trace backend"]

Loading

The supervisor:

1. Accepts OTLP spans from the agent process (gRPC on 4317, HTTP on 4318)
2. Buffers in memory (bounded, with explicit drop semantics)
3. Enriches spans with sandbox resource attributes (see below)
4. Forwards to the gateway over the session protocol
5. Flushes buffered spans before shutdown, surviving agent process termination

Point 5 matters for short-lived agents. A CI agent that runs for 10 seconds and exits may not flush its OTel SDK's batch exporter before the process terminates. The supervisor outlives the agent process and drains any buffered spans before the sandbox tears down.

The gateway receives spans from supervisors via the session protocol and relays them to the configured external OTLP endpoint alongside its own spans. It can apply head or tail sampling when configured.

For agents, the experience is zero-config: OTEL_EXPORTER_OTLP_ENDPOINT is set automatically in the sandbox environment, pointing at the supervisor's OTLP receiver. Any OTel-instrumented framework (LangChain, CrewAI, AutoGen) works without OpenShell-specific code.

OTLP receiver reachability per driver

The supervisor creates an internal workload network namespace for the agent process, connected to the supervisor's namespace via a veth pair. The OTLP receiver follows the same pattern as the proxy: bind to the veth host IP (netns.host_ip()) that is routable from inside the workload netns.

Driver	Network relationship	OTLP receiver binding
Docker	Agent enters separate workload netns via setns()	netns.host_ip():4318
Podman	Identical to Docker	Same as Docker
K8s (embedded)	Supervisor side-loaded into agent container	Same as Docker
K8s (sidecar)	Sidecar + agent share pod network namespace	localhost:4318 directly
VM	Supervisor is PID 1 inside libkrun microVM	Same as Docker (within VM)

No new networking plumbing is needed. The OTLP receiver reuses the same infrastructure the proxy already uses to make supervisor-hosted services reachable from the agent process.

Span enrichment

When forwarding agent-emitted spans, the supervisor attaches sandbox context as resource attributes:

Attribute	Source	Example
openshell.sandbox.id	Sandbox metadata	sb-abc123
openshell.sandbox.policy	Active policy name	default-policy
openshell.sandbox.user	Authenticated user	user@example.com
openshell.sandbox.image	Container image	ubuntu:22.04
openshell.sandbox.driver	Compute driver type	kubernetes
openshell.workspace.id	Workspace/tenant identifier	ws-prod-team-a

Enrichment is configurable and can be disabled for pass-through forwarding. The primary correlation mechanism between agent traces and infrastructure traces is openshell.sandbox.id. Both the agent's spans and the supervisor's network spans carry this attribute, so an operator can query "show me all spans for sandbox sb-xyz" and see both domains together.

The supervisor can optionally add span links from agent root spans to the gateway's sandbox.create span for one-click navigation in backends that support it (Jaeger, Tempo). These are a convenience layer, not the primary correlation mechanism.

Trace context propagation on agent egress

When an agent makes an outbound HTTP request, the supervisor's egress proxy intercepts it. The proxy propagates W3C traceparent context with inject-if-missing behavior:

• If the agent's request already carries a traceparent header (because the agent's OTel SDK instrumented the HTTP client), the supervisor passes it through unchanged
• If the request has no traceparent, the supervisor injects one from its own network span context

OTel-instrumented agents keep their trace continuity. Non-instrumented agents get trace context for free, making their API calls visible in the supervisor's trace.

Separation of concerns

The relay creates a clean separation between three roles:

• Agent developer: Exports to a fixed, auto-injected endpoint. Never thinks about collector topology, authentication, transport encryption, or routing. The same agent code works in every sandbox, every deployment, every workspace.
• Workspace administrator: Decides where their workspace's telemetry goes without touching agent configuration.
• Global administrator: Configures the default OTLP endpoint and platform-wide policies like rate limits and enrichment.

Visibility domains

Two distinct visibility domains flow through the relay pipeline:

1. Infrastructure traces (operator-only): supervisor network/process/middleware spans from inside the sandbox, locked as operator-only per #2508
2. Agent traces (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks

Both domains share the same relay transport. Initially, both go to the same configured OTLP endpoint, and the platform integrator separates them at the collector level using resource attributes. As a future extension, the gateway could support per-domain OTLP endpoints so it can route directly.

Multi-tenant observability

Multi-tenant deployments need per-workspace observability isolation. The openshell.workspace.id resource attribute enables collector-side routing for deployments that use a shared collector with attribute-based routing.

Per-workspace OTLP endpoint routing requires a workspace-level configuration surface that does not exist today. The Workspace resource is immutable after creation with no configuration fields (PR #2243), and the settings cascade has only two tiers (global > sandbox) with no workspace tier. A workspace settings tier (making the cascade global > workspace > sandbox) is a cross-cutting infrastructure feature that should be covered by a dedicated specification.

Precedence: sandbox OTLP setting > workspace OTLP setting > global [openshell.gateway.otlp] endpoint.

Spike: OTLP relay validation

Before implementation, a spike should validate the relay design:

1. Implement a minimal OTLP receiver in the supervisor that accepts ExportTraceServiceRequest on the veth host IP
2. Forward spans to the gateway over the session protocol
3. Measure: latency overhead per span, memory usage under sustained load, behavior when the gateway is unreachable
4. Test across Docker, Podman, and K8s sidecar topologies
5. Stress test: 100 spans/second sustained for 10 minutes, measure drop rate and gateway memory growth

Log relay for centralized OCSF collection

The supervisor generates Open Cybersecurity Schema Framework (OCSF) events describing agent workload behavior (network decisions, L7 enforcement, SSH authentication, process lifecycle). These events stay local to the sandbox today. The log relay extends the existing PushSandboxLogs mechanism to carry OCSF JSONL events from sandboxes to the gateway for centralized auditing.

A log push mechanism already exists: the supervisor streams tracing events to the gateway via the PushSandboxLogs client-streaming RPC. Extending it to include OCSF events means either adding OCSF records to the existing SandboxLogLine format (with a source field to distinguish them) or adding a dedicated OCSF push alongside the existing log push.

Agent stdout/stderr is explicitly out of scope. Agent sandboxes are interactive sessions with terminal UIs and multiplexed SSH channels. That output stream is user-facing interaction, not structured log data.

OCSF is evolving toward AI agent support (v1.8 introduced ai_operation, v1.9 adds ai_agent with delegation and prompt/response fields, OWASP AOS maps agent activities to OCSF event classes). If agent frameworks adopt OCSF emission, the log relay carries those events unchanged since it is transport-agnostic.

Alternatives Considered

Use host.openshell.internal for agent trace collection: Sandboxes can reach host-side services via host.openshell.internal (#2478). An agent could export OTLP directly to a collector on the gateway host. The supervisor relay is preferred as the default for five reasons:

• Security: Direct export has no per-sandbox authentication. Any sandbox can write to the collector port. The relay authenticates through the existing sandbox-scoped session.
• Multi-tenancy: Direct export sends all sandboxes to the same collector endpoint with no per-workspace routing. The relay lets the gateway route by workspace.
• Reliability: The SSRF engine blocks host.openshell.internal on newer gateway versions (#2478), and direct export requires per-sandbox policy exceptions. The relay uses the existing session protocol with no policy changes.
• Enrichment: Direct export sends raw spans with no sandbox context. The relay attaches openshell.sandbox.id, openshell.workspace.id, and other resource attributes that make cross-domain correlation possible.
• Separation of concerns: Direct export requires the agent developer to know the collector address, which varies by deployment. The relay auto-injects a fixed local endpoint.

These approaches are not mutually exclusive. The relay is the zero-config default, and users can override OTEL_EXPORTER_OTLP_ENDPOINT to any endpoint if they prefer direct export.

Gateway-owned root span with TRACEPARENT: Rejected. Long-lived parent spans are an OTel antipattern for unpredictable sandbox lifetimes. Resource attribute enrichment (openshell.sandbox.id on every span) provides correlation without duration coupling.

Direct OTLP export from the sandbox: Rejected. Requires an egress allowlist hole for the collector endpoint in every sandbox. Routing through the gateway keeps the sandbox egress policy closed.

Agent Investigation

• The supervisor already creates the workload network namespace and veth pair for the proxy (crates/openshell-sandbox/)
• The proxy binds to netns.host_ip() to intercept agent traffic; the OTLP receiver follows the same pattern
• PushSandboxLogs client-streaming RPC and LogPushLayer already exist in crates/openshell-supervisor-process/src/log_push.rs
• The session protocol between supervisor and gateway provides the transport path
• #2508 tracks supervisor-emitted spans; this issue tracks agent-emitted traces relayed through the supervisor. They share transport infrastructure (see sub-issue below).
• Dapr's sidecar model is the closest prior art. The three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design.
• The OpenTelemetry GenAI semantic conventions define the emerging standard for agent trace attributes.

Related: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission), #1922 (Portable sandbox log collection)