Skip to content

Add retry-with-backoff to has_repo_permission check in route job on transient 5xx errors #6558

Description

@fullsend-ai-retro

What happened

PR #130 was a docs-only PR opened from fork BATMAN-JD/rosa-cora-agent. On the initial opened event (run 32039699116), the route job's has_repo_permission function called the GitHub collaborator API to check whether the fork author had write access. The API returned HTTP 503: Permission API call failed for BATMAN-JD: gh: No server is currently available to service your request. The function returned failure (exit code 1), the route job treated the user as lacking write permission, and the result was 'No stage matched -- skipping dispatch.' No retry was attempted. All subsequent workflow runs for this PR also skipped agents (for other reasons: synchronize not routable for forks, reviews not changes_requested), so the PR received zero agent review across its entire lifecycle.

What could go better

The has_repo_permission function should distinguish between 'user lacks permission' (a definitive answer) and 'API is temporarily unavailable' (a transient failure). Currently, both cases result in the same outcome: permission denied. A transient 503 from GitHub should trigger a retry before concluding the user lacks access. This is a high-confidence finding — the 503 error and its consequence are clearly visible in the run logs. The practical impact is moderate: GitHub 503s are uncommon but not rare, and when they occur during the narrow window of PR open/routing, a legitimate contributor's PR silently loses all agent coverage with no feedback. Existing retry issues (#6194 for workflow_dispatch calls, #3902 for harness resource fetching) cover other API call sites but not this specific permission-check path in the route job.

Proposed change

In the route job's has_repo_permission function (likely in the route script sourced by the fullsend.yaml reusable workflow), add retry-with-backoff for HTTP 5xx responses from the GitHub collaborator API. Specifically: (1) Detect 5xx status codes from the gh api call to /repos/{owner}/{repo}/collaborators/{username}/permission. (2) Retry up to 2 times with exponential backoff (e.g., 2s, 4s). (3) Only treat the check as failed (permission denied) if all retries are exhausted or the API returns a definitive non-5xx error. (4) Log a warning when retries are needed so transient GitHub instability is observable in CI logs. This is consistent with the retry patterns already proposed in #6194 and #3902 for other API call sites.

Validation criteria

  1. When the GitHub collaborator API returns a transient 5xx during routing, the route job retries before falling through to 'no stage matched'. Observable in CI logs as a retry warning followed by a successful permission check. 2. When the API returns a definitive 404 or 403, the function still treats it as 'no permission' without retrying. 3. The next fork PR from a contributor with write access that encounters a transient 503 should still get routed to the appropriate stage (review/code) rather than being silently skipped.

Generated by retro agent from https://github.com/openshift-online/rosa-cora-agent/pull/130

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions