[WIP] OCPBUGS-105240: fix bootstrap race by gating TNF on EtcdBootstrapMemberRemoved

[Neilhamza]

Summary

Fix a race condition in the TNF bootstrap gate that can deadlock cluster installation ~10% of the time.

The done() function in waitforceo.go previously checked only EtcdRunningInCluster to decide when bootstrap was complete. However, removeBootstrap() in the bootstrap teardown controller sets EtcdRunningInCluster=True before calling MemberRemove(), creating a race window where pacemaker can start while the etcd-bootstrap member still exists. When the podman-etcd resource agent sees 3 etcd members instead of the expected 2, it refuses to set the learner_node attribute, deadlocking master-1's etcd startup indefinitely.

Changes

• Add EtcdBootstrapMemberRemoved as an additional gate condition in done() (AND with EtcdRunningInCluster)
• EtcdBootstrapMemberRemoved is set after MemberRemove() succeeds, ensuring the bootstrap member is actually gone before TNF setup proceeds
• Improve the waiting log message to show both condition values for easier debugging
• Add unit tests for the done() function covering all condition combinations

Why AND rather than EtcdBootstrapMemberRemoved alone

The ordering of the two conditions is not consistent across removeBootstrap() code paths (Path 1 sets removed-first, Path 3 sets running-first). Requiring both is a defensive invariant: "etcd is running in-cluster AND the bootstrap member is confirmed gone."

Scope

This fix targets the initial-install path only. Post-transition (HasExternalEtcdCompletedTransition() is true), the gate is skipped entirely on CEO restarts — that path is unaffected.

Latency caveat

setSuccessfulBootstrapRemovalStatus() can fail after MemberRemove() succeeds (raft settling). In that case, EtcdBootstrapMemberRemoved is set on the next controller sync (~1 min). Net effect: TNF setup may start up to one sync cycle later than today. No functional impact — the member is already gone.

Bug

https://issues.redhat.com/browse/OCPBUGS-105240

Test plan

• Unit tests for done() covering all condition combinations (5 cases)
• Existing TestRemoveBootstrap and TestCanRemoveEtcdBootstrap tests pass
• Reproduce with artificial delay PR (inject time.Sleep between Step 1 and Step 3 in removeBootstrap()) to confirm deadlock, then verify this fix blocks TNF setup until member is actually removed

Summary by CodeRabbit

• Bug Fixes

  • Etcd bootstrap now reports success only after both the cluster is running and the bootstrap member has been removed.
  • Waiting status messages now accurately reflect both required conditions.

• Tests

  • Added coverage for successful, incomplete, missing, and explicitly failed bootstrap-removal states.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@Neilhamza: This pull request references Jira Issue OCPBUGS-105240, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Fix a race condition in the TNF bootstrap gate that can deadlock cluster installation ~10% of the time.

The done() function in waitforceo.go previously checked only EtcdRunningInCluster to decide when bootstrap was complete. However, removeBootstrap() in the bootstrap teardown controller sets EtcdRunningInCluster=True before calling MemberRemove(), creating a race window where pacemaker can start while the etcd-bootstrap member still exists. When the podman-etcd resource agent sees 3 etcd members instead of the expected 2, it refuses to set the learner_node attribute, deadlocking master-1's etcd startup indefinitely.

Changes

• Add EtcdBootstrapMemberRemoved as an additional gate condition in done() (AND with EtcdRunningInCluster)
• EtcdBootstrapMemberRemoved is set after MemberRemove() succeeds, ensuring the bootstrap member is actually gone before TNF setup proceeds
• Improve the waiting log message to show both condition values for easier debugging
• Add unit tests for the done() function covering all condition combinations

Why AND rather than EtcdBootstrapMemberRemoved alone

The ordering of the two conditions is not consistent across removeBootstrap() code paths (Path 1 sets removed-first, Path 3 sets running-first). Requiring both is a defensive invariant: "etcd is running in-cluster AND the bootstrap member is confirmed gone."

Scope

This fix targets the initial-install path only. Post-transition (HasExternalEtcdCompletedTransition() is true), the gate is skipped entirely on CEO restarts — that path is unaffected.

Latency caveat

setSuccessfulBootstrapRemovalStatus() can fail after MemberRemove() succeeds (raft settling). In that case, EtcdBootstrapMemberRemoved is set on the next controller sync (~1 min). Net effect: TNF setup may start up to one sync cycle later than today. No functional impact — the member is already gone.

Bug

https://issues.redhat.com/browse/OCPBUGS-105240

Test plan

• Unit tests for done() covering all condition combinations (5 cases)
• Existing TestRemoveBootstrap and TestCanRemoveEtcdBootstrap tests pass
• Reproduce with artificial delay PR (inject time.Sleep between Step 1 and Step 3 in removeBootstrap()) to confirm deadlock, then verify this fix blocks TNF setup until member is actually removed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 38154668-200c-4d7f-9314-33fc4ee27555

📥 Commits

Reviewing files that changed from the base of the PR and between d2195e3 and 01e467c.

📒 Files selected for processing (1)

• pkg/operator/bootstrapteardown/waitforceo_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/operator/bootstrapteardown/waitforceo_test.go

---

Walkthrough

The etcd bootstrap completion check now requires both EtcdRunningInCluster and EtcdBootstrapMemberRemoved. Tests cover successful, partial, missing, and explicitly false condition states.

Changes

Etcd bootstrap completion

Layer / File(s)	Summary
Completion check and validation pkg/operator/bootstrapteardown/waitforceo.go, pkg/operator/bootstrapteardown/waitforceo_test.go	done requires both etcd conditions and logs both states while waiting. Table-driven tests cover the supported condition combinations.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: flavianmissi, p0lyn0mial

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the bootstrap race fix and the required EtcdBootstrapMemberRemoved gate.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The added subtests use static condition-based names via t.Run; no dynamic pod, node, namespace, timestamp, UUID, IP, or generated values appear in test titles.
Test Structure And Quality	✅ Passed	The local table-driven test covers all condition combinations, uses no cluster resources or waits, has useful failure messages, and matches nearby standard testing patterns.
Microshift Test Compatibility	✅ Passed	The changes add only a standard Go TestDone unit test using testing and Etcd status types; no Ginkgo e2e tests, MicroShift-incompatible APIs, namespaces, or assumptions were added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The added test is a standard Go TestDone unit test using testing.T; it adds no Ginkgo e2e test or multi-node/SNO assumption.
Topology-Aware Scheduling Compatibility	✅ Passed	The PR changes only etcd bootstrap condition gating, logging, and unit tests; it adds no affinity, topology spread, replica, node selector, toleration, or PDB scheduling constraints.
Ote Binary Stdout Contract	✅ Passed	The PR adds no process-level stdout writes. OTE configures GinkgoWriter to os.Stderr, and vendored klog defaults to stderr; changed klog calls are in helper/test execution code.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR adds a standard Go testing.T unit test for done; it adds no Ginkgo e2e tests, IPv4 assumptions, or external connectivity.
No-Weak-Crypto	✅ Passed	The PR changes only etcd bootstrap condition logic, logging, and unit tests; the added lines contain no weak crypto primitives, custom crypto, or secret comparisons.
Container-Privileges	✅ Passed	The PR changes only Go source and test files. The patch adds no container or Kubernetes manifest settings for privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeEscalation.
No-Sensitive-Data-In-Logs	✅ Passed	The changed log emits only two boolean condition values; no passwords, tokens, keys, PII, session IDs, hostnames, or customer data are logged.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign benluddy for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/operator/bootstrapteardown/waitforceo_test.go`:
- Around line 38-44: Add table-driven cases in the existing wait-force condition
test alongside “EtcdRunningInCluster true but EtcdBootstrapMemberRemoved false”
for EtcdRunningInCluster=false with EtcdBootstrapMemberRemoved=true and false,
covering both missing-completion outcomes while preserving the expected results
defined by the test’s behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2f21c59c-bdaf-4902-b41c-ddb70538f721

📥 Commits

Reviewing files that changed from the base of the PR and between 2f256f2 and d2195e3.

📒 Files selected for processing (2)

• pkg/operator/bootstrapteardown/waitforceo.go
• pkg/operator/bootstrapteardown/waitforceo_test.go

[openshift-ci]

@Neilhamza: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/configmap-scale	01e467c	link	false	/test configmap-scale

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.