Skip to content

orchestrator: include current/max counts in ResourceExhausted error messages - #3584

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/orchestrator-resource-exhausted-error-context
Open

orchestrator: include current/max counts in ResourceExhausted error messages#3584
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/orchestrator-resource-exhausted-error-context

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Fixes #3583

Problem

When the orchestrator returns a ResourceExhausted gRPC error and the API logs "Node exhausted, trying another node", the attached error string carries no count information. All three ResourceExhausted sites were either missing the current value or missing both current and max:

// only max, no current:
"max number of running sandboxes on node reached (%d), please retry"

// no numbers at all:
"too many sandboxes starting on this node, please retry"
"too many sandboxes resuming on this node, please retry"

Changes

  • packages/shared/pkg/utils/resizable_semaphore.go: Add Current() int64 and Limit() int64 accessors to AdjustableSemaphore
  • packages/orchestrator/pkg/server/sandboxes.go: Include current=N, max=M in both running-sandboxes and starting-sandboxes error messages
  • packages/orchestrator/pkg/server/utils.go: Include current=N, max=M in the resuming-sandboxes error message

After this change

"max number of running sandboxes on node reached: current=50, max=50, please retry"
"too many sandboxes starting on this node: current=10, max=10, please retry"
"too many sandboxes resuming on this node: current=10, max=10, please retry"

This makes the "Node exhausted, trying another node" log line actionable — operators can now tell at a glance whether the node is fully saturated or just throttled on in-flight starts.

Test plan

  • Verify go build ./packages/shared/... passes
  • Verify go vet ./packages/shared/pkg/utils/... passes
  • Manually trigger a sandbox create against a node at its MaxSandboxesPerNode limit and confirm the log line now includes current/max counts

All three ResourceExhausted returns in the orchestrator server were
either missing the current count entirely or only reporting the max
limit. This made the "Node exhausted, trying another node" log line
on the API side carry no actionable information about how exhausted
the node actually was.

Add Current() and Limit() accessors to AdjustableSemaphore, then use
them to format all three error messages as current=N, max=M.

Fixes e2b-dev#3583
@AdaAibaby
AdaAibaby force-pushed the fix/orchestrator-resource-exhausted-error-context branch from 999c1f7 to 8de62ee Compare August 18, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

orchestrator: ResourceExhausted error messages don't include current vs max counts

2 participants