Skip to content

OCPBUGS-105315: Check secret before mutating vSphere templates - #6382

Closed
djoshy wants to merge 1 commit into
openshift:mainfrom
djoshy:vsphere-secret-bug
Closed

OCPBUGS-105315: Check secret before mutating vSphere templates#6382
djoshy wants to merge 1 commit into
openshift:mainfrom
djoshy:vsphere-secret-bug

Conversation

@djoshy

@djoshy djoshy commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

- What I did
vSphere's reconciler swapped the VM template in vCenter before validating the user-data secret via upgradeStubIgnitionIfRequired. If that secret was broken, the error only surfaced after vCenter was already mutated, so the next reconcile saw a matching version and cleared BootImageUpdateDegraded, even though the secret was still broken. Other platforms don't hit this since none of them involve mutating objects native to their environment for performing updates.

To fix this, additionals call for upgradeStubIgnitionIfRequired were added before the mutating create/swap in both resolveExistingTemplateVM and createNewVMTemplate, so a bad secret fails fast and the error persists as expected. Kept the existing call in platform_helpers.go for the name-divergence-only patch path.

- How to verify it
The targeted test for this failure mode,[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO Bootimages [PolarionID:80436][OTP] Bootimage secret doesn't exist error upgrading stub ignition to spec 3 should now pass.

Existing boot image tests should also continue to pass.

Summary by CodeRabbit

  • Bug Fixes
    • Added preflight validation before creating, recovering, or replacing vSphere templates.
    • Improved ignition-stub upgrade checks during template operations.
    • Prevented unnecessary OVA downloads and vSphere changes when validation or upgrade checks fail.
    • Reduced the risk of incomplete or outdated templates being created during boot image updates.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 071890ae-fcfb-4179-a1a6-97618e63a695

📥 Commits

Reviewing files that changed from the base of the PR and between d690173 and 488c2a3.

📒 Files selected for processing (1)
  • pkg/controller/bootimage/vsphere_helpers.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/controller/bootimage/vsphere_helpers.go

Walkthrough

The vSphere controller validates and upgrades the referenced ignition stub before creating a template, restoring a rollback template, or replacing an outdated template. Errors stop processing before OVA downloads or vSphere mutations.

Changes

vSphere ignition stub preflight

Layer / File(s) Summary
Preflight validation for template operations
pkg/controller/bootimage/vsphere_helpers.go
Template creation, rollback recovery, and outdated-template reconciliation validate and upgrade the ignition stub first. Validation errors return before OVA downloads, VM renames, imports, or other vSphere changes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TemplateReconciliation
  participant IgnitionStub
  participant VSphereTemplateOperations
  TemplateReconciliation->>IgnitionStub: Validate and upgrade referenced stub
  alt validation succeeds
    IgnitionStub-->>TemplateReconciliation: Return success
    TemplateReconciliation->>VSphereTemplateOperations: Create, restore, or replace template
  else validation fails
    IgnitionStub-->>TemplateReconciliation: Return error
    TemplateReconciliation-->>TemplateReconciliation: Stop before vSphere operations
  end
Loading

Suggested reviewers: pablintino, yuqi-zhang

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the bug and summarizes the main change: validating the secret before vSphere template mutations.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The commit changes only vSphere helper logic and adds no Ginkgo test declarations or test-title expressions.
Test Structure And Quality ✅ Passed The pull request changes only pkg/controller/bootimage/vsphere_helpers.go and adds no Ginkgo tests or It blocks; the listed test-structure requirements are not applicable.
Microshift Test Compatibility ✅ Passed The HEAD commit changes only pkg/controller/bootimage/vsphere_helpers.go and adds no Ginkgo e2e test declarations or test files, so MicroShift test compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed HEAD changes only pkg/controller/bootimage/vsphere_helpers.go (+19 lines); no test files or Ginkgo declarations were added, so SNO compatibility checks do not apply.
Topology-Aware Scheduling Compatibility ✅ Passed The commit changes only vSphere template validation flow; it adds no manifests or scheduling constraints such as affinity, topology spread, selectors, tolerations, replicas, or PDBs.
Ote Binary Stdout Contract ✅ Passed The patch changes only vSphere reconciliation helpers and adds validation calls; it adds no stdout writes, logging setup, or OTE process-level code.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Only pkg/controller/bootimage/vsphere_helpers.go changed (+19 lines); no Ginkgo test or test file was added, so this compatibility check is not applicable.
No-Weak-Crypto ✅ Passed The HEAD patch adds only ignition-validation calls and comments; it introduces no weak algorithms, custom crypto, or secret/token comparisons. Existing SHA-256 usage is not flagged.
Container-Privileges ✅ Passed The pull request changes only one Go file. Added lines define ignition validation control flow and contain no privileged, host namespace, SYS_ADMIN, or privilege-escalation manifest settings.
No-Sensitive-Data-In-Logs ✅ Passed The PR adds only validation calls and comments. It adds no logging, and the existing helper logs only Ignition version and secret name, not secret contents, passwords, tokens, or API keys.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from pablintino and yuqi-zhang August 6, 2026 17:47
@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2026
@djoshy djoshy changed the title vsphere: check secret before mutating template OCPBUGS-105315: Check secret before mutating vSphere templates Aug 6, 2026
@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Aug 6, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@djoshy: This pull request references Jira Issue OCPBUGS-105315, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did
vSphere's reconciler swapped the VM template in vCenter before validating the user-data secret via upgradeStubIgnitionIfRequired. If that secret was broken, the error only surfaced after vCenter was already mutated, so the next reconcile saw a matching version and cleared BootImageUpdateDegraded, even though the secret was still broken. Other platforms don't hit this since none of them involve mutating objects native to their environment for performing updates.

To fix this, additionals call for upgradeStubIgnitionIfRequired were added before the mutating create/swap in both resolveExistingTemplateVM and createNewVMTemplate, so a bad secret fails fast and the error persists as expected. Kept the existing call in platform_helpers.go for the name-divergence-only patch path.

- How to verify it
The targeted test for this failure mode,[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO Bootimages [PolarionID:80436][OTP] Bootimage secret doesn't exist error upgrading stub ignition to spec 3 should now pass.

Existing boot image tests should also continue to pass.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@djoshy: This pull request references Jira Issue OCPBUGS-105315, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

- What I did
vSphere's reconciler swapped the VM template in vCenter before validating the user-data secret via upgradeStubIgnitionIfRequired. If that secret was broken, the error only surfaced after vCenter was already mutated, so the next reconcile saw a matching version and cleared BootImageUpdateDegraded, even though the secret was still broken. Other platforms don't hit this since none of them involve mutating objects native to their environment for performing updates.

To fix this, additionals call for upgradeStubIgnitionIfRequired were added before the mutating create/swap in both resolveExistingTemplateVM and createNewVMTemplate, so a bad secret fails fast and the error persists as expected. Kept the existing call in platform_helpers.go for the name-divergence-only patch path.

- How to verify it
The targeted test for this failure mode,[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO Bootimages [PolarionID:80436][OTP] Bootimage secret doesn't exist error upgrading stub ignition to spec 3 should now pass.

Existing boot image tests should also continue to pass.

Summary by CodeRabbit

  • Bug Fixes
  • Added preflight validation before creating or replacing vSphere templates.
  • Updated ignition stubs before template operations.
  • Prevented OVA downloads and vSphere changes when validation or preparation fails.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
pkg/controller/bootimage/vsphere_helpers.go (1)

365-370: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift

Thread context.Context through boot-image reconciliation.

upgradeStubIgnitionIfRequired uses context.TODO() for both Secret requests. Introduce a reconciliation context and pass it through syncMAPIMachineSet, checkMachineSet, the provider helpers, and createNewVMTemplate. Use it for both Get and Update, and update all six callers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/bootimage/vsphere_helpers.go` around lines 365 - 370, Thread
the reconciliation context through syncMAPIMachineSet, checkMachineSet, the
provider helper methods, and createNewVMTemplate, updating all six callers and
their signatures. Change upgradeStubIgnitionIfRequired to accept that context
and use it for both Secret Get and Update operations instead of context.TODO().

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/bootimage/vsphere_helpers.go`:
- Around line 751-758: The validation in createNewVMTemplate must occur before
resolveExistingTemplateVM renames the existing oldVM; move
upgradeStubIgnitionIfRequired ahead of the rollback rename and ensure every path
validates the Secret, including when the restored template already matches
release. Add a regression test covering an invalid Secret with an existing
mco-old-* VM, asserting no vCenter template mutation and preservation of
BootImageUpdateDegraded.

---

Nitpick comments:
In `@pkg/controller/bootimage/vsphere_helpers.go`:
- Around line 365-370: Thread the reconciliation context through
syncMAPIMachineSet, checkMachineSet, the provider helper methods, and
createNewVMTemplate, updating all six callers and their signatures. Change
upgradeStubIgnitionIfRequired to accept that context and use it for both Secret
Get and Update operations instead of context.TODO().
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d7d7d86b-bace-4e34-b0a1-57693f4b8d18

📥 Commits

Reviewing files that changed from the base of the PR and between d690173 and 9a69788.

📒 Files selected for processing (1)
  • pkg/controller/bootimage/vsphere_helpers.go

Comment thread pkg/controller/bootimage/vsphere_helpers.go
@djoshy
djoshy force-pushed the vsphere-secret-bug branch from 9a69788 to 488c2a3 Compare August 6, 2026 18:35
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@djoshy

djoshy commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-vsphere-mco-bootimages e2e-vsphere-multi-vcenter-mco-bootimages

@djoshy

djoshy commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-vsphere-mco-bootimages e2e-vsphere-multi-vcenter-mco-bootimages

@openshift-ci

openshift-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@djoshy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/perfscale-control-plane-6nodes 488c2a3 link false /test perfscale-control-plane-6nodes
ci/prow/e2e-vsphere-multi-vcenter-mco-bootimages 488c2a3 link false /test e2e-vsphere-multi-vcenter-mco-bootimages
ci/prow/e2e-vsphere-mco-bootimages 488c2a3 link false /test e2e-vsphere-mco-bootimages

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@djoshy

djoshy commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

closing in favor of #6387 (comment)

@djoshy djoshy closed this Aug 11, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@djoshy: This pull request references Jira Issue OCPBUGS-105315. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

Details

In response to this:

- What I did
vSphere's reconciler swapped the VM template in vCenter before validating the user-data secret via upgradeStubIgnitionIfRequired. If that secret was broken, the error only surfaced after vCenter was already mutated, so the next reconcile saw a matching version and cleared BootImageUpdateDegraded, even though the secret was still broken. Other platforms don't hit this since none of them involve mutating objects native to their environment for performing updates.

To fix this, additionals call for upgradeStubIgnitionIfRequired were added before the mutating create/swap in both resolveExistingTemplateVM and createNewVMTemplate, so a bad secret fails fast and the error persists as expected. Kept the existing call in platform_helpers.go for the name-divergence-only patch path.

- How to verify it
The targeted test for this failure mode,[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO Bootimages [PolarionID:80436][OTP] Bootimage secret doesn't exist error upgrading stub ignition to spec 3 should now pass.

Existing boot image tests should also continue to pass.

Summary by CodeRabbit

  • Bug Fixes
  • Added preflight validation before creating, recovering, or replacing vSphere templates.
  • Improved ignition-stub upgrade checks during template operations.
  • Prevented unnecessary OVA downloads and vSphere changes when validation or upgrade checks fail.
  • Reduced the risk of incomplete or outdated templates being created during boot image updates.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@djoshy
djoshy deleted the vsphere-secret-bug branch August 11, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants