feat: add etcd gRPC latency Grafana dashboard

[Shaheen Sayyed (shaheen0b111)]

Summary

• Add new Grafana dashboard hcp-etcd-grpc-latency.json for monitoring etcd gRPC request latency (P99/P95) across HCP clusters
• Dashboard includes fleet summary stat tiles, per-cluster timeseries, SLO violation tables, raw histogram panels, and breakdown by gRPC service/method/type
• Template variables for datasource, cluster, and namespace filtering (cascading selection)
• Uses recording rules from PR feat: add etcd gRPC latency recording rules and MWMBR alerts #6351 (etcd:grpc_server_handling:*) and raw grpc_server_handling_seconds histogram with $__rate_interval
• SLO threshold at 500ms P99 latency (500ms headroom before KAS 1s SLO boundary)
• Dashboard info panel with etcd gRPC service and KV method reference tables

Depends on

• feat: add etcd gRPC latency recording rules and MWMBR alerts #6351 (etcd gRPC latency recording rules and alerts)
• feat: add grpc_server_handling_seconds to etcd SRE metrics set #6488 (etcd metric enablement - merged)

ARO-26896

Test plan

• Import dashboard JSON into dev Grafana Scratchpad folder
• Verify datasource variable populates with Managed_Prometheus_hcps-* sources
• Verify cluster and namespace dropdowns populate and cascade correctly
• Verify stat tiles, timeseries, and tables show data for clusters with ETCD_METRICS=extensive
• Verify raw histogram panels render with $__rate_interval
• Verify breakdown panels show data by service, method, and type
• Run grafanactl verify dashboards — passes

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shaheen0b111
Once this PR has been reviewed and has the lgtm label, please assign sudobrendan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• observability/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Adds a Grafana dashboard for monitoring HCP etcd gRPC latency across fleets and clusters.

Changes:

• Adds P95/P99 latency panels and SLO violation tables.
• Adds raw histogram and gRPC service/method/type breakdowns.
• Adds datasource, cluster, and namespace filtering variables.

Suppressed comments (9)

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1575

• This request-rate query sums a counter from every HA Prometheus replica. Managed Prometheus can expose the same scrape with different prometheus_replica labels, so the panel will report inflated req/s (often 2x) when multiple replicas are selected. Collapse that label before the outer aggregation, as done by other dashboards such as backend/backend.json:209.

          "expr": "sum by (grpc_service, grpc_method) (\n  rate(grpc_server_handling_seconds_count{\n    namespace=~\"$namespace\",\n    cluster=~\"$cluster\"\n  }[$__rate_interval])\n)",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:163

• This write-latency tile has the same multi-datasource issue: lastNotNull reduces each regional response independently, so it does not produce one fleet-wide worst value when $datasource is All. Merge the Mixed frames and reduce with max, matching the fleet-stat pattern used by etcd-availability.json.

        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:233

• This P95 read tile also reduces the per-workspace Mixed results with lastNotNull, so it cannot reliably display the worst value across all regional datasources selected by default. Merge the frames and reduce with max before displaying the stat.

        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:303

• This P95 write tile has the same problem: lastNotNull is applied to separate regional Mixed responses, not to one merged fleet result. Merge the datasource frames and reduce with max so the tile matches its fleet-wide description.

        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:953

• The write SLO table has the same multi-datasource problem: it only organizes each Mixed response and does not merge the regional frames, so it cannot reliably show all fleet violations together. Add a merge transformation before organize, as with the existing etcd fleet tables.

      "transformations": [
        {
          "id": "organize",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1287

• With Datasource=All, this breakdown is evaluated independently in each regional workspace. The same grpc_service is therefore returned once per workspace, but the legend contains only grpc_service, producing duplicate unlabeled service series instead of an unambiguous fleet view. Include a region/data-source identity in the series or scope the panel to one datasource; apply the same treatment to the method/type/rate breakdowns.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_service, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "legendFormat": "{{ grpc_service }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1387

• With Datasource=All, this breakdown is evaluated independently in each regional workspace. The same grpc_method is therefore returned once per workspace, but the legend contains only grpc_method, producing duplicate unlabeled method series instead of an unambiguous fleet view. Include a region/data-source identity in the series or scope the panel to one datasource.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_method, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      grpc_service=\"etcdserverpb.KV\",\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "legendFormat": "{{ grpc_method }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1576

• With Datasource=All, this request-rate breakdown is evaluated independently in each regional workspace. The legend contains only service and method, so identical operations from different workspaces appear as duplicate unlabeled series rather than an unambiguous fleet view. Include a region/data-source identity in the series or scope the panel to one datasource.

          "expr": "sum by (grpc_service, grpc_method) (\n  rate(grpc_server_handling_seconds_count{\n    namespace=~\"$namespace\",\n    cluster=~\"$cluster\"\n  }[$__rate_interval])\n)",
          "legendFormat": "{{ grpc_service }}/{{ grpc_method }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1482

• With Datasource=All, this breakdown is evaluated independently in each regional workspace. The same grpc_type is therefore returned once per workspace, but the legend contains only grpc_type, producing duplicate unlabeled type series instead of an unambiguous fleet view. Include a region/data-source identity in the series or scope the panel to one datasource.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_type, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "legendFormat": "{{ grpc_type }}",

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (10)

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:93

• Because $datasource is a multi-select variable, this max(...) is evaluated independently in each regional Prometheus workspace. The stat's lastNotNull reduction does not compute the maximum across those returned series, so the tile can show one workspace's value instead of the fleet worst value despite its title. Make this an instant query and add a cross-datasource max reduction, following the aggregation pattern in etcd-availability.json.

          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:163

• Because $datasource is a multi-select variable, this max(...) is evaluated independently in each regional Prometheus workspace. The stat's lastNotNull reduction does not compute the maximum across those returned series, so the tile can show one workspace's value instead of the fleet worst value despite its title. Make this an instant query and add a cross-datasource max reduction, following the aggregation pattern in etcd-availability.json.

          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:233

• Because $datasource is a multi-select variable, this max(...) is evaluated independently in each regional Prometheus workspace. The stat's lastNotNull reduction does not compute the maximum across those returned series, so the tile can show one workspace's value instead of the fleet worst value despite its title. Make this an instant query and add a cross-datasource max reduction, following the aggregation pattern in etcd-availability.json.

          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:303

• Because $datasource is a multi-select variable, this max(...) is evaluated independently in each regional Prometheus workspace. The stat's lastNotNull reduction does not compute the maximum across those returned series, so the tile can show one workspace's value instead of the fleet worst value despite its title. Make this an instant query and add a cross-datasource max reduction, following the aggregation pattern in etcd-availability.json.

          "calcs": [
            "lastNotNull"
          ],

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1

• The repository PR standard requires screenshots for dashboard/metrics visualization changes (CONTRIBUTING.md:95-98), but the PR description currently has no rendered dashboard screenshot and all validation steps remain unchecked. Please import the dashboard, complete the verification, and attach the required Grafana screenshot(s) before treating this PR as ready.

{

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:954

• The write violations table has the same multi-datasource issue as the read table: with datasource=All, the instant query produces regional frames, but this chain has no merge, so the displayed table may not contain all regions' violating clusters. Add merge before organize, as in the existing etcd fleet dashboard.

      "transformations": [
        {
          "id": "organize",
          "options": {

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1739

• The datasource variable is multi-valued and defaults to All, but the dependent cluster and namespace variables below issue Prometheus label_values(...) queries through $datasource. Grafana variable queries require a single datasource; when $datasource expands to multiple sources/Mixed, these dropdown queries cannot be evaluated reliably, so the default dashboard will have empty/erroring filters and the cascade will not work. Keep the variable datasource single-select or redesign how cluster/namespace options are collected across regions.

        "includeAll": true,
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "regex": "^Managed_Prometheus_hcps-.*$",
        "type": "datasource",
        "multi": true

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:38

• The info panel presents Compact as a KV gRPC method and maintenance operation, but the etcd protobuf RPC label used by the method breakdown is Compaction on etcdserverpb.KV (the Go client method is Compact). Operators will not find a Compact series in the raw grpc_method breakdown, so this reference is misleading; use the actual RPC name and service/category.

        "content": "### HCP etcd gRPC Latency \u2014 Fleet Summary\n\nThis dashboard tracks **etcd gRPC request latency** (P99/P95) for HCP clusters across all regional Prometheus workspaces.\n\n- **SLO threshold**: 500ms P99 latency (gives 500ms headroom before KAS 1s SLO boundary)\n- **Metric source**: `grpc_server_handling_seconds` histogram (requires `ETCD_METRICS=extensive`)\n- **Alerts**: `userJourneyEtcdReadLatencyP99*` / `userJourneyEtcdWriteLatencyP99*` (MWMBR, RP lane)\n\n#### etcd gRPC Services\n\n| Service | Description |\n|---|---|\n| `etcdserverpb.KV` | Key-value store operations \u2014 primary data path for KAS reads/writes |\n| `etcdserverpb.Watch` | Watch streams for key change notifications (used by KAS informers) |\n| `etcdserverpb.Lease` | Lease management for TTL-based key expiration (leader election, endpoints) |\n| `etcdserverpb.Cluster` | Cluster membership operations (member add/remove/list) |\n| `etcdserverpb.Maintenance` | Maintenance operations (defrag, snapshot, alarm) |\n| `etcdserverpb.Auth` | Authentication and authorization (user/role management) |\n\n#### KV Methods (etcdserverpb.KV)\n\n| Method | Type | Description |\n|---|---|---|\n| `Range` | Read | Primary KAS read path \u2014 fetches keys/ranges from etcd store |\n| `Txn` | Write | Primary KAS write path \u2014 atomic compare-and-swap transactions |\n| `Put` | Write | Direct key-value writes (less common from KAS, used internally) |\n| `DeleteRange` | Write | Delete keys by range (garbage collection, resource deletion) |\n| `Compact` | Maintenance | History compaction \u2014 reclaims storage from old revisions |",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1299

• This panel description names Compact, but the grpc_method label for the etcd KV protobuf RPC is Compaction; the current text therefore refers to a method that will not appear in this breakdown. Update the description to use the actual RPC name.

      "description": "P99 latency broken down by gRPC method within etcdserverpb.KV (Range, Txn, Compact, etc.).",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:631

• When datasource=All, this instant table query returns one frame per regional Prometheus datasource. Unlike the existing fleet SLO tables in etcd-availability.json, this transformation chain has no merge, so regional rows are not reliably combined into one complete violations table. Add a merge transformation before organize.

This issue also appears on line 951 of the same file.

      "transformations": [
        {
          "id": "organize",
          "options": {

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (8)

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1871

• Repository PR standards require screenshots for dashboard and metrics changes (CONTRIBUTING.md:95-98), but this PR description does not include a rendered dashboard screenshot. Please import the dashboard and attach the required before/after evidence (or explain why a before state is unavailable for this new dashboard) before approval.

  "title": "HCP etcd gRPC Latency",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:106

• These primary summary, timeseries, and SLO panels depend entirely on the etcd:grpc_server_handling:* recording rules introduced by #6351, but that dependency is currently still open/blocked. If this dashboard is imported before those rules are deployed, the main SLO views will be blank while only the raw-histogram section can render. Please gate the dashboard rollout on #6351 or provide a raw-metric fallback.

          "expr": "max(etcd:grpc_server_handling:read_latency_p99:rate5m{cluster=~\"$cluster\",namespace=~\"$namespace\"})",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1766

• The P50/P99 table has the same multi-datasource ambiguity: both instant queries group only by service/method, and the following merge has no cluster, namespace, or datasource key to distinguish identical rows returned by different regional workspaces. With the default All selection, rows can be duplicated or combined without identifying their source. Preserve those dimensions in the result, or restrict/document this table as a single-workspace view.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_service, grpc_method, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:106

• This stat target does not set instant/range: false, so Grafana requests the full dashboard time range. Because the following transformations reduce with max, this tile reports the highest latency observed during the selected window rather than the current worst cluster. Make the target instant (as in etcd-availability.json:212-215) so the reduction only combines the selected workspaces.

          "expr": "max(etcd:grpc_server_handling:read_latency_p99:rate5m{cluster=~\"$cluster\",namespace=~\"$namespace\"})",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:195

• This stat target does not set instant/range: false, so Grafana requests the full dashboard time range. Because the following transformations reduce with max, this tile reports the highest latency observed during the selected window rather than the current worst cluster. Make the target instant (as in etcd-availability.json:212-215) so the reduction only combines the selected workspaces.

          "expr": "max(etcd:grpc_server_handling:write_latency_p99:rate5m{cluster=~\"$cluster\",namespace=~\"$namespace\"})",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:284

• This stat target does not set instant/range: false, so Grafana requests the full dashboard time range. Because the following transformations reduce with max, this tile reports the highest latency observed during the selected window rather than the current worst cluster. Make the target instant (as in etcd-availability.json:212-215) so the reduction only combines the selected workspaces.

          "expr": "max(etcd:grpc_server_handling:read_latency_p95:rate5m{cluster=~\"$cluster\",namespace=~\"$namespace\"})",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:373

• This stat target does not set instant/range: false, so Grafana requests the full dashboard time range. Because the following transformations reduce with max, this tile reports the highest latency observed during the selected window rather than the current worst cluster. Make the target instant (as in etcd-availability.json:212-215) so the reduction only combines the selected workspaces.

          "expr": "max(etcd:grpc_server_handling:write_latency_p95:rate5m{cluster=~\"$cluster\",namespace=~\"$namespace\"})",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1370

• With the default datasource=All, this target is evaluated independently in each regional Prometheus workspace. Because the aggregation and legend retain only grpc_service, identical service series from different workspaces are rendered with the same name rather than as one fleet aggregate or source-qualified series. The method/type/rate and P50/P99 panels use the same pattern; please either add source identity to these views or constrain them to one datasource.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_service, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1393

• This description also claims that the KV-method panel covers Compact, but the query at line 1480 filters grpc_service="etcdserverpb.KV", so Compact can never appear in this panel. Remove Compact from the description or add a separate Maintenance-service panel.

      "description": "P99 latency broken down by gRPC method within etcdserverpb.KV (Range, Txn, Compact, etc.).",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:38

• Please add the required rendered dashboard screenshot(s) to the PR description before approval. CONTRIBUTING.md requires screenshots for dashboard/metrics visualization changes; this new dashboard currently has no screenshot demonstrating the panels and variable behavior.

        "content": "### HCP etcd gRPC Latency — Fleet Summary\n\nThis dashboard tracks **etcd gRPC request latency** (P99/P95) for HCP clusters across all regional Prometheus workspaces.\n\n- **SLO threshold**: 500ms P99 latency (gives 500ms headroom before KAS 1s SLO boundary)\n- **Metric source**: `grpc_server_handling_seconds` histogram (requires `ETCD_METRICS=extensive`)\n- **Alerts**: `userJourneyEtcdReadLatencyP99*` / `userJourneyEtcdWriteLatencyP99*` (MWMBR, RP lane)\n\n#### etcd gRPC Services\n\n| Service | Description |\n|---|---|\n| `etcdserverpb.KV` | Key-value store operations — primary data path for KAS reads/writes |\n| `etcdserverpb.Watch` | Watch streams for key change notifications (used by KAS informers) |\n| `etcdserverpb.Lease` | Lease management for TTL-based key expiration (leader election, endpoints) |\n| `etcdserverpb.Cluster` | Cluster membership operations (member add/remove/list) |\n| `etcdserverpb.Maintenance` | Maintenance operations (defrag, snapshot, alarm) |\n| `etcdserverpb.Auth` | Authentication and authorization (user/role management) |\n\n#### KV Methods (etcdserverpb.KV)\n\n| Method | Type | Description |\n|---|---|---|\n| `Range` | Read | Primary KAS read path — fetches keys/ranges from etcd store |\n| `Txn` | Write | Primary KAS write path — atomic compare-and-swap transactions |\n| `Put` | Write | Direct key-value writes (less common from KAS, used internally) |\n| `DeleteRange` | Write | Delete keys by range (garbage collection, resource deletion) |\n| `Compact` | Maintenance | History compaction — reclaims storage from old revisions |",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1844

• The datasource variable is multi-valued and defaults to All (line 1833), but Grafana's multi-datasource support is implemented for panel queries through MixedDataSource; a query variable's datasource field still cannot query every selected datasource. Referencing $datasource here therefore cannot populate cluster when multiple workspaces are selected, and the dependent namespace variable has the same problem below, so the promised cascading dropdowns will be empty or error by default. Use a single datasource for variable queries or redesign how these values are aggregated.

        "datasource": {
          "type": "prometheus",
          "uid": "$datasource"
        },
        "definition": "label_values(grpc_server_handling_seconds_bucket{namespace=~\"ocm-.*\"},cluster)",

[Shaheen Sayyed (shaheen0b111)]

/retest-required

[Shaheen Sayyed (shaheen0b111)]

/test image-updater-images

[Shaheen Sayyed (shaheen0b111)]

/test image-updater-images

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1380

• When $datasource is left at its default All, Grafana executes this query independently in each regional workspace. sum by (grpc_service, le) removes cluster/namespace identity and the legend contains only grpc_service, so the panel shows indistinguishable same-named regional series rather than a clearly identified fleet result. Preserve a workspace/cluster identity in the output or explicitly label/document this as a per-region breakdown.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_service, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1

• Before this PR is marked ready, please attach a screenshot of the imported dashboard and complete the unchecked Grafana verification in the test plan. CONTRIBUTING.md:95-98 requires screenshots for dashboard/metrics changes, and CONTRIBUTING.md:105-108 requires non-Tide checks to be green before review.

{

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:38

• Compact is not an etcdserverpb.KV RPC; it belongs to the Maintenance service. The method panel below filters to grpc_service="etcdserverpb.KV", so this entry is documented under a table where it can never appear. Move it to a Maintenance-method reference or remove it from the KV table.

        "content": "### HCP etcd gRPC Latency — Fleet Summary\n\nThis dashboard tracks **etcd gRPC request latency** (P99/P95) for HCP clusters across all regional Prometheus workspaces.\n\n- **SLO threshold**: 500ms P99 latency (gives 500ms headroom before KAS 1s SLO boundary)\n- **Metric source**: `grpc_server_handling_seconds` histogram (requires `ETCD_METRICS=extensive`)\n- **Alerts**: `userJourneyEtcdReadLatencyP99*` / `userJourneyEtcdWriteLatencyP99*` (MWMBR, RP lane)\n\n#### etcd gRPC Services\n\n| Service | Description |\n|---|---|\n| `etcdserverpb.KV` | Key-value store operations — primary data path for KAS reads/writes |\n| `etcdserverpb.Watch` | Watch streams for key change notifications (used by KAS informers) |\n| `etcdserverpb.Lease` | Lease management for TTL-based key expiration (leader election, endpoints) |\n| `etcdserverpb.Cluster` | Cluster membership operations (member add/remove/list) |\n| `etcdserverpb.Maintenance` | Maintenance operations (defrag, snapshot, alarm) |\n| `etcdserverpb.Auth` | Authentication and authorization (user/role management) |\n\n#### KV Methods (etcdserverpb.KV)\n\n| Method | Type | Description |\n|---|---|---|\n| `Range` | Read | Primary KAS read path — fetches keys/ranges from etcd store |\n| `Txn` | Write | Primary KAS write path — atomic compare-and-swap transactions |\n| `Put` | Write | Direct key-value writes (less common from KAS, used internally) |\n| `DeleteRange` | Write | Delete keys by range (garbage collection, resource deletion) |\n| `Compact` | Maintenance | History compaction — reclaims storage from old revisions |",

[Shaheen Sayyed (shaheen0b111)]

/test e2e-parallel

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (9)

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1381

• With the default datasource selection set to All, this query runs once per regional Prometheus datasource. Grouping only by grpc_service and using that alone as the legend discards cluster/namespace (and datasource) context, so the panel renders duplicate same-named lines representing separate regional P99s rather than a distinguishable fleet view. Preserve cluster/namespace in the grouping and legend, or explicitly reduce/document one series per region.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_service, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "legendFormat": "{{ grpc_service }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1481

• This method breakdown has the same cross-datasource ambiguity: when datasource=All, each workspace returns a P99 per method, but cluster/namespace are removed before the legend is rendered. The resulting duplicate grpc_method series cannot be tied to a region or HCP. Keep the cluster/namespace labels in the aggregation and legend, or restrict this panel to one datasource.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_method, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      grpc_service=\"etcdserverpb.KV\",\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "legendFormat": "{{ grpc_method }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1576

• When all regional datasources are selected, this type breakdown also produces one grpc_type series per workspace while discarding the labels that distinguish those workspaces. Grafana will show same-named lines with different regional P99s, making the breakdown misleading. Include cluster/namespace in both the aggregation and legend, or make the panel single-datasource.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_type, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "legendFormat": "{{ grpc_type }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1670

• With datasource=All, this request-rate query fans out across workspaces, but its grouping and legend omit cluster/namespace. The chart therefore overlays separate regional rates under identical service/method names instead of providing distinguishable fleet data. Include the cluster/namespace dimensions in the grouping and legend, or explicitly select a single datasource.

          "expr": "sum by (grpc_service, grpc_method) (\n  rate(grpc_server_handling_seconds_count{\n    namespace=~\"$namespace\",\n    cluster=~\"$cluster\"\n  }[$__rate_interval])\n)",
          "legendFormat": "{{ grpc_service }}/{{ grpc_method }}",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1769

• The P50 and P99 targets in this table likewise aggregate away cluster/namespace. Under the default multi-datasource All selection, the merged table contains duplicate service/method rows from different regional workspaces without any source context, so the displayed comparison is ambiguous. Preserve those dimensions and expose them in the table, or make this panel single-datasource.

          "expr": "histogram_quantile(0.5,\n  sum by (grpc_service, grpc_method, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "P50"

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1

• CONTRIBUTING.md requires before/after screenshots for dashboard, graph, and metrics changes, but the PR description includes none. Add the required annotated Grafana screenshot(s) showing the imported dashboard before requesting approval.

{

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1380

• This aggregation includes streaming RPCs in a panel labeled as request latency. grpc_server_handling_seconds measures the full handling duration, so Watch/Lease bidi streams can dominate the P99 with connection lifetimes rather than request latency. Filter this query to grpc_type="unary", or separate/relabel the streaming-duration view.

          "expr": "histogram_quantile(0.99,\n  sum by (grpc_service, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1669

• grpc_server_handling_seconds_count counts completed handling observations; for Watch/Lease bidi streams it increments when a stream closes, not once per request. With no grpc_type filter, this panel can report stream-completion rate as request rate. Filter to unary RPCs or rename the panel to describe completed stream/handling rate.

          "expr": "sum by (grpc_service, grpc_method) (\n  rate(grpc_server_handling_seconds_count{\n    namespace=~\"$namespace\",\n    cluster=~\"$cluster\"\n  }[$__rate_interval])\n)",

observability/grafana-dashboards/sre/user-journey/hcp-etcd-grpc-latency.json:1765

• The P50/P99 table also aggregates all service/method histograms without a grpc_type filter, so Watch/Lease rows represent stream lifetimes alongside unary method latency and can distort tail-latency diagnosis. Add grpc_type="unary" to both targets, or label this table as handling/stream duration rather than request latency.

          "expr": "histogram_quantile(0.5,\n  sum by (grpc_service, grpc_method, le) (\n    rate(grpc_server_handling_seconds_bucket{\n      namespace=~\"$namespace\",\n      cluster=~\"$cluster\"\n    }[$__rate_interval])\n  )\n)",

[Shaheen Sayyed (shaheen0b111)]

This is the established pattern used in the existing etcd-availability.json dashboard (same multi-value datasource with includeAll: true, same regex). The label_values() query runs against whichever datasource Grafana resolves first — this is a known Grafana limitation, not a bug in this dashboard. The panel targets use -- Mixed -- datasource which correctly fans out queries to all selected workspaces. The template variables serve as convenience filters; the data completeness comes from the mixed-datasource panel queries, not from the variable dropdown. No change needed.