ARO-26057: reconcile Grafana ADX integration fabrics by swiencki · Pull Request #4878 · Azure/ARO-HCP

Simon Wiencki (swiencki) · 2026-04-14T21:56:30Z

What

Replaces the regional direct Grafana datasource approach with optional Managed Grafana integration-fabric reconciliation under the existing single global GrafanaManage owner.

This change:

Adds disabled-by-default gates for Kusto access and integration-fabric reconciliation.
Tags Kusto clusters for environment and geography-scoped discovery.
Grants the global Grafana identity database-level Viewer access to ServiceLogs only.
Passes typed ADX integration settings through the global Grafana pipelines and templatize runner.
Requires an explicit complete geography set before reconciliation can run.
Documents nonproduction validation, dry-run review, rollout order, and rollback behavior.

Why

The Managed Grafana resource provider owns datasource lifecycle and may remove datasources created directly through Grafana's HTTP API. The global integration-fabric design avoids competing regional reconcilers and keeps ownership with the resource provider.

Safety and rollout

Both gates default to false. Rollout is staged:

Deploy discovery tags.
Enable Kusto Viewer access and deploy every target geography.
Validate the integration-fabric scenario, target semantics, datasource behavior, and ownership behavior in nonproduction.
Run grafanactl manage reconcile --dry-run and confirm there are no unexpected delete operations.
Enable integration-fabric reconciliation in the global pipeline.

The configuration schema prevents integration reconciliation unless Kusto access is enabled and the geography set is non-empty.

Dependencies

grafanactl: reconcile ADX integration fabrics ARO-Tools#228 contains the required pipeline and grafanactl API.
This draft temporarily uses root go.work replacements to the reviewed Remove temporary/now unused genuuid code #228 fork commit so CI can test the coordinated changes before Remove temporary/now unused genuuid code #228 merges.
After Remove temporary/now unused genuuid code #228 merges, remove the replacements and update the ARO-Tools versions in test, tooling/grafanactl, tooling/helmtest, tooling/pipeline-documentation, and tooling/templatize.
feat(kusto): add entity-groups pipeline step and retain deployer Admin #6294 should merge first. Resolve the expected overlap where both changes add Kusto discovery tags.
The previous sdp-pipelines PR 15393796 is abandoned because it packages the superseded design. A fresh downstream PR will be generated after both GitHub changes merge.

The standard sdp-pipelines EV2 check currently stops at the expected stale ARO-Tools schema. Rebuilding the generator with the #228 commit passes the full Global entrypoint in resolve mode.

Testing

Full make lint
make validate-config-pipelines
go test ./tooling/templatize/pkg/pipeline/... ./tooling/helmtest/... -count=1
make verify
Kusto Bicep compilation
Coordinated sdp-pipelines Global EV2 manifest generation in resolve mode
Coordinated multi-pass pre-push review with no remaining blockers

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds ARO-HCP support to provision and clean up Azure Data Explorer (Kusto) data sources in the shared Managed Grafana workspace, integrated into the existing geography rollout flow.

Changes:

Extend Kusto IaC to optionally grant Grafana’s managed identity DB-level Viewer access and output the cluster URI.
Add geography pipeline steps to fetch global Grafana outputs and provision/update ADX Grafana datasources via Azure CLI.
Introduce configuration flags/schema + docs, and update the Kusto delete cleanup to remove the matching Grafana datasource.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/monitoring.md	Documents ADX/Kusto datasource provisioning gates, naming, validation, and teardown behavior.
dev-infrastructure/templates/kusto.bicep	Threads optional Grafana resource ID into the Kusto module and exposes `kustoUri` output.
dev-infrastructure/modules/logs/kusto/main.bicep	Grants Grafana MI Viewer on `ServiceLogs` when provided and surfaces cluster URI output.
dev-infrastructure/modules/logs/kusto/cluster.bicep	Exposes Kusto cluster `uri` as an output.
dev-infrastructure/geography-pipeline.yaml	Adds global output step and a shell step to create/update ADX Grafana datasources.
dev-infrastructure/configurations/kusto.tmpl.bicepparam	Adds `grafanaResourceId` parameter placeholder for pipeline substitution.
dev-infrastructure/cleanup/delete.kusto.instance.sh	Extends cleanup to also delete the corresponding Grafana datasource.
dev-infrastructure/cleanup/delete.kusto.instance.pipeline.yaml	Wires Grafana resource ID from global outputs into the cleanup script run.
config/rendered/dev/swft/uksouth.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/prow/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/pers/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/perf/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/dev/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/cspr/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/config.yaml	Introduces new monitoring defaults for ADX provisioning flags.
config/config.schema.json	Defines schema/validation for the new monitoring configuration knobs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Jan-Hendrik Boll (janboll) · 2026-04-16T07:26:50Z

/hold
please use https://github.com/Azure/ARO-Tools/blob/main/tools/grafanactl/cmd/modify/cmd.go#L32
for doing modifications to grafana. Ideally we should update grafana ONCE in the pipeline, cause running an update on the instance will block it for several minutes.

Simon Wiencki (swiencki) · 2026-04-16T16:06:29Z

Thanks for the feedback, I'll move this to grafanactl in Azure/ARO-Tools.

On the "update once" concern: the existing modify datasource reconcile runs globally because UpdateGrafanaIntegrations is an ARM-level mutation that blocks the Grafana instance for several minutes. ADX datasources go through the Grafana REST API directly (POST/PUT /api/datasources), which is a lightweight config write, no ARM-level instance lock. So per-geography invocation is safe here. Is this good with you?

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Simon Wiencki (swiencki) · 2026-04-28T21:19:54Z

/unhold

Updated to use grafanactl.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…peline Per PR #4878 review. Drops the new GrafanaDatasources call from geography-pipeline.yaml and extends the existing add-grafana-datasource in region-pipeline.yaml with the ADX block. clusterUrl sources from regional kusto-lookup output. AzureMonitor reconcile preserved via the ARO-Tools default-true. Triple-AND gate keeps the three feature flags agreeing by construction. No Go/Bicep/config/test changes.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

…peline Per PR #4878 review. Drops the new GrafanaDatasources call from geography-pipeline.yaml and extends the existing add-grafana-datasource in region-pipeline.yaml with the ADX block. clusterUrl sources from regional kusto-lookup output. AzureMonitor reconcile preserved via the ARO-Tools default-true. Triple-AND gate keeps the three feature flags agreeing by construction. No Go/Bicep/config/test changes.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

openshift-ci · 2026-06-09T19:55:38Z

Simon Wiencki (@swiencki): The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/cspr	`8d14303`	link	true	`/test cspr`
ci/prow/mega-linter	`4d47166`	link	true	`/test mega-linter`
ci/prow/images	`4d47166`	link	true	`/test images`
ci/prow/e2e-parallel	`4d47166`	link	true	`/test e2e-parallel`
ci/prow/config-change-detection	`4d47166`	link	true	`/test config-change-detection`
ci/prow/bicep-lint	`4d47166`	link	true	`/test bicep-lint`
ci/prow/test-unit	`4d47166`	link	true	`/test test-unit`
ci/prow/lint	`4d47166`	link	true	`/test lint`
ci/prow/verify	`4d47166`	link	true	`/test verify`
ci/prow/e2e-images	`4d47166`	link	true	`/test e2e-images`
ci/prow/image-updater-images	`4d47166`	link	true	`/test image-updater-images`
ci/prow/periodic-healthcheck-images	`4d47166`	link	true	`/test periodic-healthcheck-images`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Mariusz Mazur (mmazur) · 2026-06-17T15:29:11Z

+      defaultDatabase:
+        configRef: kusto.serviceLogsDatabase
+      datasourceName:
+        value: "kusto-{{ .environmentName }}-{{ .azureGeoShortId }}"


This should reflect the actual source name.

Rael Garcia (raelga) · 2026-08-02T11:39:24Z

Simon Wiencki (@swiencki) is this PR still needed?

Simon Wiencki (swiencki) · 2026-08-04T15:26:51Z

Simon Wiencki (@swiencki) is this PR still needed?

Rael Garcia (@raelga) Still needed and plan to work on it shortly after #6294

Copilot-Session: d00ea40f-616b-4f7b-89c5-014b59c12169

openshift-ci · 2026-08-04T18:16:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: swiencki
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI review requested due to automatic review settings April 14, 2026 21:56

openshift-ci Bot requested review from Gerd Oberlechner (geoberle) and Jan-Hendrik Boll (janboll) April 14, 2026 21:56

Copilot AI reviewed Apr 14, 2026

View reviewed changes

Comment thread dev-infrastructure/geography-pipeline.yaml Outdated

Comment thread dev-infrastructure/geography-pipeline.yaml Outdated

Comment thread dev-infrastructure/configurations/kusto.tmpl.bicepparam Outdated

Comment thread dev-infrastructure/cleanup/delete.kusto.instance.sh Outdated

Simon Wiencki (swiencki) changed the title ~~ARO-22279: add ADX Grafana datasource provisioning~~ ARO-26057: add ADX Grafana datasource provisioning Apr 15, 2026

openshift-ci Bot added the do-not-merge/hold label Apr 16, 2026

openshift-ci Bot added the needs-rebase label Apr 22, 2026

Copilot AI review requested due to automatic review settings April 28, 2026 18:45

Simon Wiencki (swiencki) force-pushed the adx-grafana-datasource-upstream branch from 8d14303 to a80a076 Compare April 28, 2026 18:45

openshift-ci Bot removed the needs-rebase label Apr 28, 2026

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Comment thread dev-infrastructure/geography-pipeline.yaml

Copilot AI review requested due to automatic review settings April 28, 2026 19:51

Copilot started reviewing on behalf of Simon Wiencki (swiencki) April 28, 2026 19:52 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

openshift-ci Bot added needs-rebase and removed do-not-merge/hold labels Apr 28, 2026

Jan-Hendrik Boll (janboll) requested changes Apr 30, 2026

View reviewed changes

Comment thread dev-infrastructure/geography-pipeline.yaml Outdated

Copilot AI review requested due to automatic review settings April 30, 2026 21:02

Simon Wiencki (swiencki) force-pushed the adx-grafana-datasource-upstream branch from 5d069ae to c091e9c Compare April 30, 2026 21:02

openshift-ci Bot removed the needs-rebase label Apr 30, 2026

Copilot started reviewing on behalf of Simon Wiencki (swiencki) April 30, 2026 21:02 View session

Simon Wiencki (swiencki) mentioned this pull request Apr 30, 2026

grafanactl: reconcile ADX integration fabrics Azure/ARO-Tools#228

Open

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Comment thread tooling/templatize/pkg/pipeline/grafana_datasources_test.go Outdated

Copilot AI review requested due to automatic review settings May 8, 2026 15:56

Copilot started reviewing on behalf of Simon Wiencki (swiencki) May 8, 2026 15:57 View session

Copilot AI reviewed May 8, 2026

View reviewed changes

Comment thread docs/monitoring.md Outdated

Comment thread docs/monitoring.md

openshift-ci Bot added the do-not-merge/hold label May 28, 2026

Copilot AI review requested due to automatic review settings June 3, 2026 17:33

Copilot started reviewing on behalf of Simon Wiencki (swiencki) June 3, 2026 17:33 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread tooling/templatize/pkg/pipeline/grafana_datasources.go Outdated

Copilot AI review requested due to automatic review settings June 4, 2026 19:50

Copilot started reviewing on behalf of Simon Wiencki (swiencki) June 4, 2026 19:50 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Gerd Oberlechner (geoberle) reviewed Jun 9, 2026

View reviewed changes

Comment thread dev-infrastructure/region-pipeline.yaml Outdated

Comment thread dev-infrastructure/region-pipeline.yaml Outdated

Comment thread dev-infrastructure/modules/logs/kusto/grant-access.bicep Outdated

Comment thread dev-infrastructure/templates/kusto-lookup.bicep Outdated

openshift-ci Bot added the needs-rebase label Jun 9, 2026

Copilot AI review requested due to automatic review settings June 9, 2026 18:54

Simon Wiencki (swiencki) force-pushed the adx-grafana-datasource-upstream branch from 6131235 to 4d47166 Compare June 9, 2026 18:54

openshift-ci Bot removed the needs-rebase label Jun 9, 2026

Copilot started reviewing on behalf of Simon Wiencki (swiencki) June 9, 2026 18:54 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Comment thread dev-infrastructure/region-pipeline.yaml Outdated

Comment thread dev-infrastructure/region-pipeline.yaml Outdated

Comment thread dev-infrastructure/region-pipeline.yaml Outdated

Simon Wiencki (swiencki) requested a review from Gerd Oberlechner (geoberle) June 9, 2026 19:56

Mariusz Mazur (mmazur) reviewed Jun 17, 2026

View reviewed changes

openshift-ci Bot added the needs-rebase label Jul 3, 2026

ARO-26057: reconcile Grafana ADX integration fabrics

cc2fd2e

Copilot-Session: d00ea40f-616b-4f7b-89c5-014b59c12169

Simon Wiencki (swiencki) changed the title ~~ARO-26057: add ADX Grafana datasource provisioning~~ ARO-26057: reconcile Grafana ADX integration fabrics Aug 4, 2026

Simon Wiencki (swiencki) marked this pull request as draft August 4, 2026 18:15

openshift-ci Bot added the do-not-merge/work-in-progress label Aug 4, 2026

Simon Wiencki (swiencki) force-pushed the adx-grafana-datasource-upstream branch from 4d47166 to cc2fd2e Compare August 4, 2026 18:15

openshift-ci Bot removed the needs-rebase label Aug 4, 2026

openshift-ci Bot added the needs-rebase label Aug 9, 2026

openshift-ci Bot removed the needs-rebase label Aug 17, 2026

Uh oh!

Conversation

Simon Wiencki (swiencki) commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Safety and rollout

Dependencies

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jan-Hendrik Boll (janboll) commented Apr 16, 2026

Uh oh!

Simon Wiencki (swiencki) commented Apr 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Simon Wiencki (swiencki) commented Apr 28, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci Bot commented Jun 9, 2026

Uh oh!

Mariusz Mazur (mmazur) Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Rael Garcia (raelga) commented Aug 2, 2026

Uh oh!

Simon Wiencki (swiencki) commented Aug 4, 2026

Uh oh!

openshift-ci Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Simon Wiencki (swiencki) commented Apr 14, 2026 •

edited

Loading