-
Notifications
You must be signed in to change notification settings - Fork 170
feat: add cluster upgrade partial and complete SLO observability #5752
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
openshift-merge-bot
merged 1 commit into
Azure:main
from
machi1990:control-plane-upgrade-slo-alert
Aug 10, 2026
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
57 changes: 57 additions & 0 deletions
57
observability/alerts/userJourneyClusterUpgradeMonitor-prometheusRule.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| apiVersion: monitoring.coreos.com/v1 | ||
|
machi1990 marked this conversation as resolved.
|
||
| kind: PrometheusRule | ||
| metadata: | ||
| labels: | ||
| app.kubernetes.io/name: kube-prometheus | ||
| app.kubernetes.io/part-of: kube-prometheus | ||
| prometheus: k8s | ||
| role: alert-rules | ||
| name: user-journey-cluster-upgrade-alerts | ||
| namespace: monitoring | ||
| spec: | ||
| groups: | ||
| # Per-upgrade stuck alerts for cluster upgrade SLOs. Sev 4. | ||
| # At current upgrade volumes, alert directly on duration instead of fleet burn-rate. | ||
| # | ||
| # The ControlPlaneActiveVersion controller derives upgrade Partial/Completed state from | ||
| # HostedCluster version history (preferring status.controlPlaneVersion.history, else | ||
| # status.version.history), which feeds backend_cluster_version_info and these alerts. | ||
| # Control plane version history reports for versions >= 4.20, so completeness tracks the | ||
| # control plane independently of the data plane. While 4.19 is still around, those | ||
| # clusters fall back to status.version.history, where upgrade completeness depends on | ||
| # data plane state — e.g. no worker nodes means the upgrade will not reach completed; | ||
| # missing console component in external auth prevents CVO from marking the upgrade | ||
| # complete. The complete alert may therefore fire for 4.19 clusters due to missing | ||
| # data plane configuration rather than control plane failure. | ||
| - name: rp-user-journey-cluster-upgrade-monitor-rules | ||
|
|
||
| rules: | ||
|
machi1990 marked this conversation as resolved.
|
||
| - alert: userJourneyClusterUpgradeStuckInDesired | ||
| expr: | | ||
| hosted_control_plane_upgrade:duration_in_desired:seconds > 1200 | ||
| for: 5m | ||
| labels: | ||
|
machi1990 marked this conversation as resolved.
|
||
| severity: "4" | ||
| slo: cluster-upgrade-reach-partial | ||
|
machi1990 marked this conversation as resolved.
|
||
| component: slo | ||
| annotations: | ||
| summary: "{{ $labels.cluster }}: Cluster upgrade on {{ $labels.resource_id }} to version {{ $labels.version }} not progressing" | ||
| description: | | ||
| Cluster upgrade target version {{ $labels.version }} has been in desired for over 20 minutes without reaching partial. With the alert pending for 5 minutes, paging starts after ~25 minutes. Partial is when the target version becomes active (recognized as active) on the HCP cluster; the upgrade is in progress but not yet complete. Investigate backend_cluster_version_info on the affected cluster. | ||
| Service Cluster: {{ $labels.cluster }} | ||
| runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade | ||
| correlationId: "userJourneyClusterUpgradeStuckInDesired/{{ $labels.cluster }}{{ $labels.resource_id }}/{{ $labels.version }}" | ||
| - alert: userJourneyClusterUpgradeStuckInProgress | ||
| expr: | | ||
| hosted_control_plane_upgrade:duration_in_progress:seconds > 1800 | ||
| for: 5m | ||
| labels: | ||
|
machi1990 marked this conversation as resolved.
|
||
| severity: "4" | ||
| slo: cluster-upgrade-complete | ||
| component: slo | ||
| annotations: | ||
| summary: "{{ $labels.cluster }}: Cluster upgrade on {{ $labels.resource_id }} to version {{ $labels.version }} stuck" | ||
| description: | | ||
| Cluster upgrade target version {{ $labels.version }} has been in progress for over 30 minutes without reaching completed. With the alert pending for 5 minutes, paging starts after ~35 minutes. Completed is when the target version upgrade has finished and is reported as complete on the HCP cluster. Investigate backend_cluster_version_info on the affected cluster. | ||
| Service Cluster: {{ $labels.cluster }} | ||
| runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade | ||
| correlationId: "userJourneyClusterUpgradeStuckInProgress/{{ $labels.cluster }}{{ $labels.resource_id }}/{{ $labels.version }}" | ||
130 changes: 130 additions & 0 deletions
130
observability/alerts/userJourneyClusterUpgradeMonitor-prometheusRule_test.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,130 @@ | ||
| rule_files: | ||
| - userJourneyClusterUpgrade-recordingRule.yaml | ||
| - userJourneyClusterUpgradeMonitor-prometheusRule.yaml | ||
| evaluation_interval: 1m | ||
| tests: | ||
| # ======================================== | ||
| # Reach partial: successful — no alert | ||
| # ======================================== | ||
| - interval: 1m | ||
| input_series: | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/1/rg/c1", subscription_id="sub-1", cluster_uuid="uuid-1", version="4.19.19", state="completed"}' | ||
| values: "1+0x40" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/1/rg/c1", subscription_id="sub-1", cluster_uuid="uuid-1", version="4.19.20", state="desired"}' | ||
| values: "1x10 0x25" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/1/rg/c1", subscription_id="sub-1", cluster_uuid="uuid-1", version="4.19.20", state="partial"}' | ||
| values: "0x10 1x25" | ||
| alert_rule_test: | ||
| - eval_time: 25m | ||
| alertname: userJourneyClusterUpgradeStuckInDesired | ||
| exp_alerts: [] | ||
| - eval_time: 40m | ||
| alertname: userJourneyClusterUpgradeStuckInDesired | ||
| exp_alerts: [] | ||
| # ======================================== | ||
| # Reach partial: stuck in desired >20m — alert pending for 5m, fires after ~25m | ||
| # ======================================== | ||
| - interval: 1m | ||
| input_series: | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/2/rg/c2", subscription_id="sub-2", cluster_uuid="uuid-2", version="4.19.20", state="completed"}' | ||
| values: "1+0x50" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/2/rg/c2", subscription_id="sub-2", cluster_uuid="uuid-2", version="4.19.21", state="desired"}' | ||
| values: "1+0x50" | ||
| alert_rule_test: | ||
| - eval_time: 20m | ||
| alertname: userJourneyClusterUpgradeStuckInDesired | ||
| exp_alerts: [] | ||
| - eval_time: 30m | ||
| alertname: userJourneyClusterUpgradeStuckInDesired | ||
| exp_alerts: | ||
| - exp_labels: | ||
| alertname: userJourneyClusterUpgradeStuckInDesired | ||
| cluster: "svc-1" | ||
| cluster_uuid: "uuid-2" | ||
| resource_id: "/sub/2/rg/c2" | ||
| subscription_id: "sub-2" | ||
| version: "4.19.21" | ||
| state: "desired" | ||
| severity: "4" | ||
| slo: cluster-upgrade-reach-partial | ||
| component: "slo" | ||
| exp_annotations: | ||
| summary: "svc-1: Cluster upgrade on /sub/2/rg/c2 to version 4.19.21 not progressing" | ||
| description: | | ||
| Cluster upgrade target version 4.19.21 has been in desired for over 20 minutes without reaching partial. With the alert pending for 5 minutes, paging starts after ~25 minutes. Partial is when the target version becomes active (recognized as active) on the HCP cluster; the upgrade is in progress but not yet complete. Investigate backend_cluster_version_info on the affected cluster. | ||
| Service Cluster: svc-1 | ||
| runbook_url: "https://aka.ms/arohcp-runbook-cluster-upgrade" | ||
| correlationId: "userJourneyClusterUpgradeStuckInDesired/svc-1/sub/2/rg/c2/4.19.21" | ||
| # ======================================== | ||
| # Reach partial: install only — not upgrade eligible, no alert | ||
| # ======================================== | ||
| - interval: 1m | ||
| input_series: | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/3/rg/c4", subscription_id="sub-3", cluster_uuid="uuid-4", version="4.19.20", state="desired"}' | ||
| values: "1+0x50" | ||
| alert_rule_test: | ||
| - eval_time: 50m | ||
| alertname: userJourneyClusterUpgradeStuckInDesired | ||
| exp_alerts: [] | ||
| # ======================================== | ||
| # Upgrade complete: successful — no alert | ||
| # ======================================== | ||
| - interval: 1m | ||
| input_series: | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/1/rg/c1", subscription_id="sub-1", cluster_uuid="uuid-1", version="4.19.19", state="completed"}' | ||
| values: "1+0x40" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/1/rg/c1", subscription_id="sub-1", cluster_uuid="uuid-1", version="4.19.20", state="desired"}' | ||
| values: "1x10 0x25" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/1/rg/c1", subscription_id="sub-1", cluster_uuid="uuid-1", version="4.19.20", state="completed"}' | ||
| values: "0x10 1x25" | ||
| alert_rule_test: | ||
| - eval_time: 40m | ||
| alertname: userJourneyClusterUpgradeStuckInProgress | ||
| exp_alerts: [] | ||
| # ======================================== | ||
| # Upgrade complete: stuck in progress >30m — alert pending for 5m, fires after ~35m | ||
| # ======================================== | ||
| - interval: 1m | ||
| input_series: | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/2/rg/c2", subscription_id="sub-2", cluster_uuid="uuid-2", version="4.19.20", state="completed"}' | ||
| values: "1+0x50" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/2/rg/c2", subscription_id="sub-2", cluster_uuid="uuid-2", version="4.19.21", state="desired"}' | ||
| values: "1x10 0x50" | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/2/rg/c2", subscription_id="sub-2", cluster_uuid="uuid-2", version="4.19.21", state="partial"}' | ||
| values: "0x10 1x50" | ||
| alert_rule_test: | ||
| - eval_time: 30m | ||
| alertname: userJourneyClusterUpgradeStuckInProgress | ||
| exp_alerts: [] | ||
| - eval_time: 40m | ||
| alertname: userJourneyClusterUpgradeStuckInProgress | ||
| exp_alerts: | ||
| - exp_labels: | ||
| alertname: userJourneyClusterUpgradeStuckInProgress | ||
| cluster: "svc-1" | ||
| cluster_uuid: "uuid-2" | ||
| resource_id: "/sub/2/rg/c2" | ||
| subscription_id: "sub-2" | ||
| version: "4.19.21" | ||
| state: "partial" | ||
| severity: "4" | ||
| slo: cluster-upgrade-complete | ||
| component: "slo" | ||
| exp_annotations: | ||
| summary: "svc-1: Cluster upgrade on /sub/2/rg/c2 to version 4.19.21 stuck" | ||
| description: | | ||
| Cluster upgrade target version 4.19.21 has been in progress for over 30 minutes without reaching completed. With the alert pending for 5 minutes, paging starts after ~35 minutes. Completed is when the target version upgrade has finished and is reported as complete on the HCP cluster. Investigate backend_cluster_version_info on the affected cluster. | ||
| Service Cluster: svc-1 | ||
| runbook_url: "https://aka.ms/arohcp-runbook-cluster-upgrade" | ||
| correlationId: "userJourneyClusterUpgradeStuckInProgress/svc-1/sub/2/rg/c2/4.19.21" | ||
| # ======================================== | ||
| # Upgrade complete: install only — not upgrade eligible, no alert | ||
| # ======================================== | ||
| - interval: 1m | ||
| input_series: | ||
| - series: 'backend_cluster_version_info{cluster="svc-1", resource_id="/sub/3/rg/c4", subscription_id="sub-3", cluster_uuid="uuid-4", version="4.19.20", state="desired"}' | ||
| values: "1+0x50" | ||
| alert_rule_test: | ||
| - eval_time: 50m | ||
| alertname: userJourneyClusterUpgradeStuckInProgress | ||
| exp_alerts: [] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.