Configure cloud and BYOC log routing

[rguliyev]

Summary

• Replaces sandbox/build-log LaunchDarkly routing with independent environment controls.
• CLICKHOUSE_LOGS_READ_ENABLED=true reads persistent logs from ClickHouse; false/default reads Loki.
• CLICKHOUSE_LOGS_WRITE_ONLY=true writes only to the dedicated local /logs/clickhouse endpoint.
• False/default mode sends exactly once to LOGS_COLLECTOR_ADDRESS; logs-collector owns the Loki + ClickHouse fan-out.
• Removes dynamic per-write routing and all application-level ClickHouse shadow writers.

Behavior

Read control	Write control	Persistent reads	External writes
false / unset	false / unset	Loki	Collector-owned Loki + ClickHouse fan-out
true	false / unset	ClickHouse	Collector-owned Loki + ClickHouse fan-out
false / unset	true	Loki	ClickHouse only
true	true	ClickHouse	ClickHouse only

There is no read fallback. Missing or unavailable ClickHouse causes enabled persistent reads to fail. Temporary live builder logs remain independent and continue working without a persistent ClickHouse reader.

The write control reaches API, orchestrator, template-manager, and hyperloop/envd forwarding. False/default mode has one application destination—the collector—so the companion Terraform collector shadow cannot duplicate ClickHouse rows. True mode has one application destination—the local ClickHouse writer—and never falls back to Loki.

Staged rollout

The companion e2b-dev/terraform#1368 change is intentionally deployed first:

1. Terraform installs the ClickHouse receiver, retains Loki, enables the bounded collector shadow, and provides managed-cloud environment values.
2. Older binaries ignore the new values, so Terraform-first preparation remains Loki-read and collector-dual-write.
3. After compatible binaries from this PR deploy, managed-cloud true values select ClickHouse-only reads and writes.
4. BYOC/non-target deployments retain false/default values: Loki reads and collector-owned dual writes.
5. Disabling the managed-cloud Loki sink is a later, separate Terraform stage after all producers are upgraded and verified.

Deployment defaults

• Generic/self-host/BYOC application defaults are Loki reads and one write to logs-collector; the deployment collector owns dual writing.
• AWS BYOC explicitly sets both controls false for API and write-only false for orchestrator/template-manager.
• Terraform Don't assume an orchestrator port number, but pull it from nomad #1368 explicitly sets both controls true for staging, Foxtrot, Juliett, and Tango, including Nomad orchestrator-ee.

Scope

• Removes only LaunchDarkly code used by sandbox/build-log routing; unrelated LaunchDarkly functionality remains.
• Uses the existing http://127.0.0.1:4321/logs/clickhouse contract; generic /logs is untouched.
• Changes only AWS BYOC IaC in this repository; no GCP apply, image publication, DDL, migration, or deployment is included.

Validation

• Targeted go test -race -count=1 passed for shared logger/sandbox and orchestrator hyperloop handlers.
• Targeted repository-configured golangci-lint passed with zero issues.
• gofmt and git diff --check passed.
• Existing AWS Terraform fmt, backendless init, and validate checks passed.
• Tests cover strict defaults/parsing, independent read/write combinations, no read fallback, exactly one selected write destination, primary failure propagation, and no application-level shadow.
• Searches returned zero stale NewDualHTTPWriter, shadowEnabled, sandboxClickhouseAddr, or dynamic resolver identifiers.

GATE-TRIGGER: customer-log persistence/read routing and staged removal of Loki.

GATE-APPROVED: human user, 2026-08-19, implement the cloud/BYOC matrix in existing PR #3589 only—cloud defaults to ClickHouse-only reads/writes; BYOC deployment defaults to Loki reads and dual writes; CLICKHOUSE_LOGS_READ_ENABLED enables BYOC ClickHouse reads; CLICKHOUSE_LOGS_WRITE_ONLY switches BYOC from dual writes to ClickHouse-only; no LaunchDarkly, no deployment.

GATE-APPROVED: human user, 2026-08-19, update and publish PRs #1368 and #3589 for Terraform-first preparation, collector-owned BYOC dual writes, and managed-cloud ClickHouse-only application routing.

Architecture reference: architect-harness@a6f21d900e8f2ed714256b64aa4575e62dcee4a0 (stale; refresh was blocked by unrelated dirty DES-020 work).

[codecov]

Codecov Report

❌ Patch coverage is 38.77551% with 30 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/orchestrator/pkg/factories/run.go	0.00%	11 Missing ⚠️
packages/api/main.go	0.00%	6 Missing ⚠️
packages/shared/pkg/logger/sandbox/logger.go	55.55%	4 Missing ⚠️
packages/api/internal/handlers/store.go	0.00%	2 Missing ⚠️
...ackages/orchestrator/pkg/hyperloopserver/server.go	0.00%	2 Missing ⚠️
packages/shared/pkg/logger/exporter.go	33.33%	1 Missing and 1 partial ⚠️
packages/api/internal/clusters/cluster.go	0.00%	1 Missing ⚠️
packages/api/internal/clusters/clusters_sync.go	0.00%	1 Missing ⚠️
.../orchestrator/pkg/hyperloopserver/handlers/logs.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!