NO-ISSUE: Add fix TC 63866 failing in CI job

[ptalgulk01]

Added fix for below failure

[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO alerts [PolarionID:63866][OTP] MCDPivotError alert [Disruptive] expand_less	3m47s
{  fail [github.com/openshift/machine-config-operator/test/extended-priv/mco_alerts.go:421]: The worker MCP is not reporting the right error message
In resource <Kind: mcp, Name: worker, Namespace: >, the following condition field 'NodeDegraded.message' failed to satisfy matcher.
Expected
    <string>: Node ip-10-0-61-195.us-west-2.compute.internal is reporting: "Node ip-10-0-61-195.us-west-2.compute.internal upgrade failure. unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\n", Node ip-10-0-61-195.us-west-2.compute.internal is reporting: "unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\n"
to match regular expression
    <string>: Node ip-10-0-61-195\.us-west-2\.compute\.internal .*[Ff]ailed to update OS to quay\.io/mcoqe/layering@sha256:aeae0ae63b1dc0dfeeff4eb618db3c6573029524172e32ca75a717420267433d
{"lastTransitionTime":"2026-08-06T06:13:08Z","message":"Node ip-10-0-61-195.us-west-2.compute.internal is reporting: \"Node ip-10-0-61-195.us-west-2.compute.internal upgrade failure. unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\\n\", Node ip-10-0-61-195.us-west-2.compute.internal is reporting: \"unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\\n\"","reason":"1 nodes are reporting degraded status on sync","status":"True","type":"NodeDegraded"}}

Summary by CodeRabbit

• Tests
  • Expanded coverage for update scenarios involving an already mounted system update command.
  • Improved validation of failed system pivots and recovery behavior.
  • Added clearer checks that replacement commands return the expected failure status and correctly forward other operations.
  • Strengthened testing of fallback behavior during update and recovery flows.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

Walkthrough

The test helper now unmounts /usr/bin/rpm-ostree before installing a replacement. The replacement shim returns status 255 for rebase and forwards other commands to /var/tmp/rpm-ostree.

Changes

rpm-ostree replacement handling

Layer / File(s)	Summary
Replacement mount and command shim test/extended-priv/node.go, test/extended-priv/testdata/files/rpm-ostree-force-pivot-error.sh	ReplaceRpmOstree lazily unmounts the existing rpm-ostree binary before copying and bind-mounting the replacement. The shim fails rebase with status 255 and forwards other commands to /var/tmp/rpm-ostree.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: isabella-janssen, pablintino

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The commit changes only mount logic and a shell fixture. It adds no Ginkgo It, Describe, Context, or When title, and the related MCDPivotError title is static.
Test Structure And Quality	✅ Passed	The PR changes only a node helper and shell fixture; it adds no Ginkgo It block, cluster wait, resource setup, cleanup, or assertion.
Microshift Test Compatibility	✅ Passed	The commit changes only an existing helper and shell fixture; it adds no Ginkgo test, OpenShift API reference, namespace use, or MicroShift-incompatible assumption.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR changes only ReplaceRpmOstree and an existing shell test shim; it adds no Ginkgo It/Describe/Context/When test, so the SNO check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The change only updates test utilities for rpm-ostree replacement; it adds no deployment, controller, manifest, replica, affinity, topology spread, selector, toleration, or PDB scheduling constraints.
Ote Binary Stdout Contract	✅ Passed	The PR only changes a test helper and shim; OTE main has no process-level stdout writes, and ReplaceRpmOstree is called from test cases.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The commit modifies only an existing helper and a local shell shim; it adds no Ginkgo test and no new IPv4 assumption or external connectivity requirement.
No-Weak-Crypto	✅ Passed	The HEAD diff adds only mount handling and shell forwarding; scans found no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret comparisons.
Container-Privileges	✅ Passed	The PR changes only a Go test helper and a shell shim. The diff adds no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, root, or allowPrivilegeEscalation manifest setting.
No-Sensitive-Data-In-Logs	✅ Passed	The patch adds no logging or sensitive literals. Its only new command suppresses unmount stderr; the shim preserves existing rpm-ostree output forwarding.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the CI fix for test case 63866, which matches the pull request objectives and changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended-priv/node.go`:
- Line 1621: Update the command assembled by ReplaceRpmOstree so the chmod,
umount, and cp steps remain in a single && chain, preserving failures through
the final status. Handle only the expected “path already unmounted” result
explicitly, and propagate all other umount errors; do not suppress error
returns.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8fe2a1fa-6891-4f2e-af78-40b093abb428

📥 Commits

Reviewing files that changed from the base of the PR and between 13ea39a and d4de2a5.

📒 Files selected for processing (2)

• test/extended-priv/node.go
• test/extended-priv/testdata/files/rpm-ostree-force-pivot-error.sh

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[sergiordlr]

/test bootstrap-unit

/lgtm

[openshift-merge-bot]

Pipeline controller notification

No second-stage tests were triggered for this PR.

This can happen when:

• The changed files don't match any pipeline_run_if_changed patterns
• All files match pipeline_skip_if_only_changed patterns
• No pipeline-controlled jobs are defined for the main branch

Use /test ? to see all available tests.

[ptalgulk01]

/verified later by @ptalgulk01

[openshift-ci-robot]

@ptalgulk01: Only users can be targets for the /verified later command.

Details

In response to this:

/verified later by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: isabella-janssen, ptalgulk01, sergiordlr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [isabella-janssen,ptalgulk01,sergiordlr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@ptalgulk01: This pull request explicitly references no jira issue.

Details

In response to this:

Added fix for below failure

[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO alerts [PolarionID:63866][OTP] MCDPivotError alert [Disruptive] expand_less	3m47s
{  fail [github.com/openshift/machine-config-operator/test/extended-priv/mco_alerts.go:421]: The worker MCP is not reporting the right error message
In resource <Kind: mcp, Name: worker, Namespace: >, the following condition field 'NodeDegraded.message' failed to satisfy matcher.
Expected
   <string>: Node ip-10-0-61-195.us-west-2.compute.internal is reporting: "Node ip-10-0-61-195.us-west-2.compute.internal upgrade failure. unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\n", Node ip-10-0-61-195.us-west-2.compute.internal is reporting: "unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\n"
to match regular expression
   <string>: Node ip-10-0-61-195\.us-west-2\.compute\.internal .*[Ff]ailed to update OS to quay\.io/mcoqe/layering@sha256:aeae0ae63b1dc0dfeeff4eb618db3c6573029524172e32ca75a717420267433d
{"lastTransitionTime":"2026-08-06T06:13:08Z","message":"Node ip-10-0-61-195.us-west-2.compute.internal is reporting: \"Node ip-10-0-61-195.us-west-2.compute.internal upgrade failure. unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\\n\", Node ip-10-0-61-195.us-west-2.compute.internal is reporting: \"unexpected on-disk state validating against rendered-worker-8d04bb6279730408aa1553d6c36a1d91: error running rpm-ostree kargs: exit status 127\\n/usr/bin/rpm-ostree: line 9: /tmp/rpm-ostree: No such file or directory\\n\"","reason":"1 nodes are reporting degraded status on sync","status":"True","type":"NodeDegraded"}}

Summary by CodeRabbit

• Tests
• Expanded coverage for update scenarios involving an already mounted system update command.
• Improved validation of failed system pivots and recovery behavior.
• Added clearer checks that replacement commands return the expected failure status and correctly forward other operations.
• Strengthened testing of fallback behavior during update and recovery flows.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ptalgulk01]

/verified later by @ptalgulk01

[openshift-ci-robot]

@ptalgulk01: Only users can be targets for the /verified later command.

Details

In response to this:

/verified later by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ptalgulk01]

/verified later @ptalgulk01

[openshift-ci-robot]

@ptalgulk01: This PR has been marked to be verified later by @ptalgulk01.

Details

In response to this:

/verified later @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD cd2a4c6 and 2 for PR HEAD 543dfae in total

[openshift-ci]

@ptalgulk01: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.