control_plane: retry transient postgres errors in control plane jobs

[Mwea]

Problem

A production cluster (cdip-diskless-prod-spoke-*) threw a burst of FetchError / Read future failed exceptions. Every trace bottoms out in the same PostgreSQL condition on the read replica that serves find_batches_v2, in two escalating forms:

1. Statement cancellation

   org.postgresql.util.PSQLException: ERROR: canceling statement due to conflict with recovery
     Detail: User was holding shared buffer pin for too long.
   

2. Connection termination (escalation of the same conflict)

   org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery
     Detail: User was holding shared buffer pin for too long.
   

   which then surfaces downstream as server conn crashed?, Cannot commit transaction, and a suppressed Connection is closed / Cannot get autoCommit.

Both are hot-standby recovery conflicts: a read query holds a shared buffer pin while WAL replay needs the same page. They are transient and the server rolls the transaction back — but JobUtils had no retry (only a // TODO add retry with backoff), so a single conflict propagated all the way up as ControlPlaneException("Error finding batches") → FindBatchesException and FetchHandler returned UNKNOWN_SERVER_ERROR for every partition in the fetch.

Note: the DB-side mitigations (hot_standby_feedback = on, raised max_standby_streaming_delay) are already enabled on this cluster. This PR is the application-side resilience for the conflicts that still occur under replay pressure.

Change

Implement the missing retry in JobUtils: retry the whole job with jittered exponential backoff (3 attempts, ~50 ms → ~100 ms with 0.2 jitter, capped at 1 s; via the injected Time so it is fully virtual in tests).

Exceptions handled — retry is keyed on rollback-guaranteed PostgreSQL SQLStates:

SQLState	Meaning	Why retriable
40001	serialization_failure	PostgreSQL raises this (ERRCODE_T_R_SERIALIZATION_FAILURE) for both recovery-conflict forms above — the statement cancellation and the connection termination. Server guarantees rollback.
40P01	deadlock_detected	Victim transaction is rolled back.

Key design points:

• Retry wraps the whole transaction, not the inner lambda. Because the control-plane DSLContexts are Hikari-pool-backed, a retry pulls a fresh connection — this is what recovers from the connection-terminated form where the previous connection is dead.
• Full cause chain + suppressed throwables are scanned. The retriable PSQLException is buried under ControlPlaneException → DataAccessException; in the connection-killed form it appears as a suppressed throwable (the failed rollback), exactly as in the traces.
• Generic connection-loss states (08*, 57P01) are intentionally NOT retried — a drop during commit leaves a write in-doubt, so blindly replaying could double-apply. The recovery-conflict family that caused this incident is fully covered by the rollback-guaranteed states, so the retry is safe for both read and write jobs and can live in the shared helper.
• Error contract unchanged: on exhaustion or a non-retriable failure, the original exception is rethrown and mapped to ControlPlaneException / wrapped RuntimeException as before.

Hot-path safety

This retry sits on the fetch hot path shared by all 16 control-plane jobs, so it is designed to stay safe under a conflict storm rather than amplify one:

• Metrics stay accurate. measureDurationMs fires the duration callback in a finally, so timing each attempt directly would record N samples per run and pollute the latency/rate percentiles with failed-attempt durations. Only the decisive attempt (success, or the final failure) is reported to the callback → exactly one sample per run().
• Jittered backoff (ExponentialBackoff, reused from clients) prevents many jobs failing against the same struggling standby from retrying in lockstep and re-colliding on the same recovery-conflict window.
• Interrupt-aware. Utils.sleep restores the interrupt flag but returns normally; the loop checks Thread.isInterrupted() after backoff and aborts (keeping the flag set) rather than spinning through remaining attempts and delaying broker shutdown.
• One terminal log on exhaustion, so the final user-visible failure is greppable instead of appearing only as intermediate "retrying" warnings.

Tests

JobUtilsTest (pure unit — no DB container, uses MockTime): happy path, recovery-conflict-then-succeed (reproduces the incident with the exact wrapping from the traces), deadlock retry, max-attempts exhaustion preserving ControlPlaneException, non-retriable SQLState (23505) and plain runtime exceptions not retried, suppressed-throwable scanning, self-referential cause chain (terminates, no StackOverflow), plus: jittered backoff timing asserted via MockTime deltas, a single duration-callback per run (guards the metrics contract), no sleep on non-retriable failures, and interrupt-driven abort.

Follow-up (not in this PR)

• When retries are exhausted the client still gets UNKNOWN_SERVER_ERROR. Surfacing exhausted transient failures as a retriable Kafka error would be a separate change in the FetchHandler error-mapping path.
• The connection-terminated form relies on pgjdbc surfacing SQLState 40001; there is a residual driver-level race where a socket EOF beating the FATAL message could downgrade to 08006. Confirming this end-to-end warrants a hot-standby integration test.

🤖 Generated with Claude Code

[viktorsomogyi]

If it's a transient error, maybe the warn level is a bit too harsh. I think this ideally should be debug or maybe info level. The final attempt is fair to be warn though.

[viktorsomogyi]

The comment seems to be a bit too verbose. Perhaps:

Suggested change

	/**
	* PostgreSQL {@code SQLState}s that are safe to retry because they guarantee the transaction did
	* <em>not</em> commit (the server rolled it back), so replaying the whole transaction cannot
	* double-apply a write.
	*
	* <ul>
	* <li>{@code 40001} — {@code serialization_failure}. PostgreSQL also raises this
	* ({@code ERRCODE_T_R_SERIALIZATION_FAILURE}) for hot-standby recovery conflicts, in both
	* observed forms: the statement cancellation
	* ("canceling statement due to conflict with recovery") <em>and</em> the connection
	* termination ("terminating connection due to conflict with recovery"). The latter closes
	* the JDBC connection, but because {@code readJooqCtx}/{@code writeJooqCtx} are backed by a
	* connection pool, the retry transparently obtains a fresh connection.</li>
	* <li>{@code 40P01} — {@code deadlock_detected}. The victim transaction is rolled back.</li>
	* </ul>
	*
	* <p>We deliberately do <b>not</b> retry generic connection-loss states (class {@code 08*},
	* {@code 57P01}) here: a connection dropped <em>during commit</em> leaves the transaction
	* in-doubt, so blindly replaying a write job could apply it twice. The recovery-conflict family
	* that motivated this retry logic is fully covered by the rollback-guaranteed states above.
	*/
	/**
	* PostgreSQL failures that guarantee transaction rollback and are safe to retry:
	* {@code 40001} for serialization or recovery conflicts, and {@code 40P01} for deadlocks.
	* Connection-loss states are excluded because the commit outcome may be unknown.
	*/

[viktorsomogyi]

Would it make sense to take a look at the whole chain by iterating getNextException()?

[viktorsomogyi]

Can't we just catch SQLException here?

[viktorsomogyi]

The parameter could be narrowed down to SQLException based on its behavior and the javadoc on the function too.