feat: add MC AKS node health metrics to gather-observability

[Gerd Oberlechner (geoberle)]

Summary

Jira: https://issues.redhat.com/browse/ARO-29285

• Add new "MC AKS Metrics" panel to gather-observability Spyglass artifacts
• Six charts for management cluster user node health visibility:
  1. Node count by phase (Ready/NotReady/Joining) — faceted stacked area per MC, distinguishes genuinely unhealthy nodes from nodes still joining (age ≤10m)
  2. Unhealthy node conditions — faceted stacked area per MC, shows only active bad conditions (DiskPressure, MemoryPressure, KubeletProblem, etc.), zero noise when healthy
  3. CPU usage — max across user nodes per MC
  4. Memory usage — max across user nodes per MC
  5. Disk usage — max root filesystem across user nodes per MC
  6. Pod capacity — max running/capacity ratio across user nodes per MC
• All queries filter to management clusters (cluster=~".*-mgmt-.*") and exclude system/infra pool nodes
• Handles HA Prometheus replica dedup and dual kube-state-metrics instance dedup

Test plan

• go test ./test/cmd/aro-hcp-tests/gather-observability/... passes (validates queries.yaml parsing)
• Run gather-observability against a personal dev environment and verify charts render with data
• Verify node count matches actual user node count (no double-counting)
• Verify unhealthy conditions chart is empty on healthy clusters

[Copilot]

Pull request overview

Adds an “MC AKS Metrics” observability panel for management-cluster node health and resource usage.

Changes:

• Adds node phase and unhealthy-condition charts.
• Adds CPU, memory, disk, and pod-capacity charts.
• Applies management-cluster filtering and replica deduplication.

Suppressed comments (5)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:344

• As with the CPU query, instance is a node-exporter scrape address rather than the Kubernetes node name, so this selector does not remove system/infra nodes. The resulting maximum memory value can therefore be driven by an excluded pool and does not satisfy the chart's user-node scope. Map the series through kubelet_node_name or otherwise apply the pool filter using the actual node label before taking the max.

          and on (node, cluster)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:357

• This instance filter has the same identity mismatch: node-exporter exposes the scrape endpoint in instance, not the AKS node name. System/infra root filesystems therefore remain in the max, so a full system disk can falsely report user-node disk pressure. Apply the pool exclusion after mapping the node-exporter series to the Kubernetes node (for example via kubelet_node_name).

        ),

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:254

• Because this PR adds a metrics visualization, the PR description must include the required before/after screenshots (with annotations if needed) and a linked Jira/GitHub tracking issue or an explanation for why no issue exists, per the Pull Request Standards. Please update the description before approval.

      )

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:254

• This PR changes metrics visualizations by adding six charts. CONTRIBUTING.md requires before/after screenshots for graph, dashboard, and metrics-visualization changes; the current PR description does not include them. Please attach the required screenshots (and annotate any subtle changes) before approval.

      )

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:254

• The PR description does not reference a Jira/GitHub tracking issue. CONTRIBUTING.md requires every PR to include a ticket link or explain why no ticket exists; please add that reference or explanation before approval.

      )

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:437

• kubelet_running_pods is keyed by the kubelet scrape target instance, while kube_node_status_capacity is keyed by Kubernetes node. Relabeling the capacity node into instance does not join these series (the repository's existing KubeletTooManyPods rule uses kubelet_node_name for this mapping), so this query will normally return no pod-capacity data. Join running pods to kubelet_node_name on (cluster, instance), then divide by capacity on (cluster, node) and apply the user-node filter there.

          label_replace(
            kube_node_status_capacity{cluster=~".*-mgmt-.*", node!~"aks-system-.*|aks-infra.*-.*", resource="pods"},
            "instance", "$1", "node", "(.*)"

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:323

• The test plan is entirely unchecked, so the PR provides no evidence that the required parser test or the chart/data smoke checks passed. Please run the listed validation and update the checklist, or document any unrelated/flaky failure, before requesting approval.

- title: "MC AKS Metrics"

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:395

• Filtering by instance does not select Kubernetes node names in this Prometheus setup. The existing node-exporter rules map each target to node via (cluster, namespace, pod), and repository fixtures show endpoint-style instances such as 10.0.0.4:9100; consequently this matcher does not exclude system/infra nodes and the cluster maximum can report a non-user node. Join to the node label before filtering and aggregating (or use an equivalent recording rule).

            rate(node_cpu_seconds_total{cluster=~".*-mgmt-.*", instance!~"aks-system-.*|aks-infra.*-.*", mode="idle"}[5m])

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:409

• These memory operands apply the pool exclusion to instance, but node-exporter instance is the scrape target identity rather than the Kubernetes node label in this deployment. System/infra nodes can therefore contribute to the maximum despite the chart claiming to show user nodes. Map the node-exporter target to node first, then filter/aggregate by that label.

          max by (instance, cluster) (node_memory_MemAvailable_bytes{cluster=~".*-mgmt-.*", instance!~"aks-system-.*|aks-infra.*-.*"})
          /
          max by (instance, cluster) (node_memory_MemTotal_bytes{cluster=~".*-mgmt-.*", instance!~"aks-system-.*|aks-infra.*-.*"})

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:422

• The root-filesystem operands have the same incorrect pool filter on scrape-target instance instead of Kubernetes node. This can include system/infra filesystems in the cluster maximum and make the user-node disk chart inaccurate. Attach the node label before filtering and aggregate by node.

          max by (instance, cluster) (node_filesystem_avail_bytes{cluster=~".*-mgmt-.*", instance!~"aks-system-.*|aks-infra.*-.*", mountpoint="/", fstype!="tmpfs"})
          /
          max by (instance, cluster) (node_filesystem_size_bytes{cluster=~".*-mgmt-.*", instance!~"aks-system-.*|aks-infra.*-.*", mountpoint="/", fstype!="tmpfs"})

[Rael Garcia (raelga)]

/lgtm
/approve

[Ben Vesel (bennerv)]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bennerv, geoberle, raelga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/OWNERS [bennerv,geoberle,raelga]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 61af7a5 and 2 for PR HEAD 5dfc8bf in total

[openshift-ci]

Gerd Oberlechner (@geoberle): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	5dfc8bf	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD c81c1b6 and 1 for PR HEAD 5dfc8bf in total