[Testing] Fix races and bugs found by flaky tests - #8633
Open
janezpodhostnik wants to merge 5 commits into
Open
Conversation
Contributor
📝 WalkthroughWalkthroughThe changes address concurrent state updates, cache lookup and record ownership, gRPC shutdown handling, atomic registry startup state, asynchronous test behavior, and slashing message-type handling. ChangesOptimistic sync pipeline
Shared cache and record safety
Server and registry lifecycle handling
Slashing offense logging
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 3, 2026
janezpodhostnik
changed the title
Base automatically changed from
janez/flaky-test-followup-evm-registers
to
master
August 4, 2026 15:25
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@network/alsp/internal/cache.go`:
- Around line 126-129: Replace the panic call in the error handler that logs
"unexpected error while getting spam record from cache" with
irrecoverable.NewExceptionf(...) instead, using the same descriptive error
message. This ensures backend faults in the cache are handled consistently
through Flow's exception mechanism rather than bypassing it with a raw panic.
In `@network/p2p/scoring/registry.go`:
- Around line 192-196: Update the startup guard around silencePeriodStartTime to
use CompareAndSwap(nil, &now) as an atomic one-shot initialization; if the swap
fails, return immediately from the current worker instead of calling
parent.Throw or continuing toward ready(). Preserve the timestamp initialization
and startup behavior for the winning worker.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5bbe8b0d-3d8c-4daa-a2a0-b6dd5de7c161
📒 Files selected for processing (13)
module/executiondatasync/optimistic_sync/pipeline/pipeline.gomodule/executiondatasync/optimistic_sync/pipeline/pipeline_functional_test.gomodule/executiondatasync/optimistic_sync/pipeline/pipeline_test.gomodule/executiondatasync/optimistic_sync/pipeline/pipeline_test_utils.gomodule/grpcserver/server.gomodule/mempool/herocache/backdata/cache.gomodule/mempool/herocache/backdata/cache_test.gonetwork/alsp/internal/cache.gonetwork/p2p/inspector/validation/control_message_validation_inspector.gonetwork/p2p/scoring/internal/appSpecificScoreCache.gonetwork/p2p/scoring/internal/subscriptionCache.gonetwork/p2p/scoring/registry.gonetwork/slashing/consumer.go
Comment on lines
+126
to
+129
| if err != nil { | ||
| // the Run closure above never returns an error; an error here indicates a bug in the | ||
| // backend implementation. | ||
| panic(fmt.Errorf("unexpected error while getting spam record from cache: %w", err)) |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)cache\.go$|module/component/component\.go|irrecoverable|docs/agents/(CodingConventions|GoDocs)\.md' || true
echo "== target snippet =="
if [ -f network/alsp/internal/cache.go ]; then
sed -n '1,220p' network/alsp/internal/cache.go | nl -ba | sed -n '1,220p'
fi
echo "== irrecoverable usage examples =="
rg -n "irrecoverable|panic\(fmt\.Errorf|panic\(" -S --glob '*.go' . | head -200Repository: onflow/flow-go
Length of output: 1148
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== read docs guidelines =="
for f in docs/agents/CodingConventions.md docs/agents/GoDocs.md; do
if [ -f "$f" ]; then
echo "--- $f ---"
cat -n "$f"
else
echo "missing $f"
fi
doneRepository: onflow/flow-go
Length of output: 28678
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target file =="
awk '{printf "%6d\t%s\n", NR, $0}' network/alsp/internal/cache.go | sed -n '1,240p'
echo "== irrecoverable package =="
awk '{printf "%6d\t%s\n", NR, $0}' module/irrecoverable/exception.go | sed -n '1,240p'
awk '{printf "%6d\t%s\n", NR, $0}' module/irrecoverable/irrecoverable.go | sed -n '1,240p'
echo "== irrecoverable import/usages in network/alsp/internal/cache.go and neighboring alsp cache files =="
rg -n 'irrecoverable|panic\(fmt\.Errorf|fmt\.Errorf|No error returns are expected|Unexpected error' network/alsp -S network/alsp -S || true
echo "== component interface =="
awk '{printf "%6d\t%s\n", NR, $0}' module/component/component.go | sed -n '1,240p'Repository: onflow/flow-go
Length of output: 34983
Use irrecoverable exceptions for backend faults.
This cache documents returned errors as irrecoverable bugs, but panic(fmt.Errorf(...)) bypasses Flow’s exception mechanism. Use irrecoverable.NewExceptionf(...) here or refactor the cache methods to return exceptions consistently.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@network/alsp/internal/cache.go` around lines 126 - 129, Replace the panic
call in the error handler that logs "unexpected error while getting spam record
from cache" with irrecoverable.NewExceptionf(...) instead, using the same
descriptive error message. This ensures backend faults in the cache are handled
consistently through Flow's exception mechanism rather than bypassing it with a
raw panic.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes bugs discovered while stress-running the full unit test suite (500+ full-suite runs plus per-test
-racestress runs, continuing the work from #8626).Changes
Cache.getstopped scanning a bucket when it hit a slot whose value was ejected but whose stale 32-bit id prefix collided with the queried key, reporting a present entity as missing. Now continues scanning. Includes a deterministic regression test (verified to fail before the fix).Rununconditionally stored the initial parent state and could overwrite a concurrently deliveredOnParentStateUpdated, losing the update and deadlocking the pipeline (it would never observeStateCompleteand never persist).Runnow initializes with CompareAndSwap fromStatePending, andOnParentStateUpdateduses an unconditional store so updates cannot be silently dropped. Includes the pipeline test harness fixes this change interlocks with.ready()andServe,ServereturnsErrServerStopped, which was thrown as an irrecoverable error during a normal shutdown. Now excluded explicitly (gRPC returns nil when stopped while serving, so the exclusion is precise).SpamRecordCache:Getcopied record fields andAdjustWithInitread the adjusted penalty after the backend lock was released, racing in-place record mutations. Both now read under the lock.logOffensemutated the caller'sViolation(defaultingMsgType), racing concurrent use of the same violation. Uses a local variable now.silencePeriodStartTime(multi-wordtime.Time) was written by the startup worker while the score function read it concurrently, now an atomic pointer.AppSpecificScoreCacheandSubscriptionRecordCachemutated records in place whileGetreads fields outside the lock, both use copy-on-write now.inspectRpcPublishMessagesshuffled the live RPC's publish message slice on worker goroutines while libp2p pubsub still reads the same RPC. Now samples a shallow clone.All fixes verified with 20-100x
-racestress runs per affected package.network/alsp/...,network/p2p/scoring/..., andconsensus/hotstuff/integrationare now fully race-clean.Related: #8626, #8629
Follow-up PRs will add dedicated regression tests for paths that still lack deterministic coverage (grpcserver shutdown race, slashing consumer, pipeline initialization race).
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit