feat(kusto): add entity-groups pipeline step and retain deployer Admin

[swiencki]

Replaces #6150 (reverted in #6258) and #6229.

What

Adds the Kusto entity-groups pipeline step to the geography pipeline, enabling cross-cluster entity group sync on all three databases:

• AllServiceLogs:ServiceLogs
• AllHostedControlPlaneLogs:HostedControlPlaneLogs
• AllMonitoringEvents:MonitoringEvents

This enables entity_group KQL queries across all regional Kusto clusters without manually listing each cluster URI.

Why

#6150 failed in INT because explicit PrincipalAssignment resources (hardcoded name globalMSI-admin) conflicted with existing auto-named assignments created by the Kusto script deployment flow (RetainPermissionOnScriptCompletion). This v3 takes a different approach: instead of adding PrincipalAssignment resources, remove the removePermission module so the deploying identity retains Database Admin after table-creation scripts complete.

Permission model

The global MSI retains Database Admin on all three databases (ServiceLogs, HostedControlPlaneLogs, MonitoringEvents). Previously this was revoked after each deploy. The retained privilege is required for the entity-groups pipeline step and is lesser than the ARM Contributor the global MSI already holds on the Kusto cluster.

Changes

• Remove removePermission module from kusto/main.bicep (retain Admin for entity-groups on all DBs)
• Remove globalMSIId parameter wiring (no longer needed)
• Remove PrincipalAssignment resources (no longer needed)
• Add aroHCPPurpose: 'logs' tag to Kusto cluster (kustoctl discovery)
• Add entity-groups pipeline step to geography-pipeline.yaml and geography-pipeline-stg.yaml (all 3 databases)
• Add KustoEntityGroups runner in templatize (kusto_entity_groups.go + run.go case)
• Register entity-groups subcommand in tooling/kustoctl

Dependencies

• sdp-pipelines PR 16614060: KustoEntityGroupsStep EV2RA adapter (must merge first)
• ARO-Tools #267 (merged): kustoctl entity-groups sync command

Jira: ARO-27830
Closes #6229

[swiencki]

/hold

[Copilot]

Pull request overview

This PR enables cross-cluster Kusto entity group synchronization by adding a new KustoEntityGroups pipeline step (backed by ARO-Tools kustoctl) and adjusting the Kusto deployment so the deployer identity retains Database Admin on the logs databases required for .create-or-alter entity_group.

Changes:

• Add a KustoEntityGroups step runner to tooling/templatize and wire it into RunStep.
• Update Kusto infra Bicep to retain deployer Database Admin on ServiceLogs and HostedControlPlaneLogs, while still revoking it for MonitoringEvents.
• Add aroHCPPurpose: 'logs' tagging on Kusto clusters and add entity-groups steps to both geography pipelines (prod/int + stg variant).

Reviewed changes

Copilot reviewed 12 out of 14 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tooling/templatize/pkg/pipeline/run.go	Adds dispatch for the new KustoEntityGroups pipeline step type.
tooling/templatize/pkg/pipeline/kusto_entity_groups.go	Implements the runner that calls ARO-Tools entitygroups.DefaultSyncOptions().Run().
tooling/templatize/go.mod	Adds dependency on ARO-Tools tools/kustoctl.
tooling/templatize/go.sum	Records new transitive deps pulled in by ARO-Tools kustoctl.
tooling/kustoctl/main.go	Registers the entity-groups subcommand from ARO-Tools in the local wrapper CLI.
tooling/kustoctl/go.mod	Adds ARO-Tools tools/kustoctl dependency and its transitive deps.
tooling/kustoctl/go.sum	Updates sums for the new ARO-Tools dependencies.
dev-infrastructure/templates/kusto.bicep	No functional change (whitespace-only in diff).
dev-infrastructure/modules/logs/kusto/main.bicep	Removes “remove permission” cleanup for all DBs and adds selective cleanup only for MonitoringEvents.
dev-infrastructure/modules/logs/kusto/cluster.bicep	Tags Kusto clusters with aroHCPPurpose: 'logs' for kustoctl discovery.
dev-infrastructure/geography-pipeline.yaml	Adds global output step (for globalMSIId) and a new entity-groups step after Kusto deploy.
dev-infrastructure/geography-pipeline-stg.yaml	Same as above for the STG-global pipeline variant.
dev-infrastructure/configurations/kusto.tmpl.bicepparam	No functional change (whitespace-only in diff).
dev-infrastructure/configurations/kusto-stg.tmpl.bicepparam	No functional change (whitespace-only in diff).

[Copilot]

Pull request overview

Copilot reviewed 12 out of 14 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 16 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 15 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 15 changed files in this pull request and generated 2 comments.

[swiencki]

/unhold

[Copilot]

Pull request overview

Copilot reviewed 12 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

dev-infrastructure/configurations/kusto.tmpl.bicepparam:30

• This change is whitespace-only (adds a trailing blank line) and doesn’t affect the template’s behavior. To keep the PR focused and diffs minimal, please drop the extra blank line.

param enableAutoScale = {{ .kusto.enableAutoScale }}

dev-infrastructure/configurations/kusto-stg.tmpl.bicepparam:34

• This change is whitespace-only (adds a trailing blank line) and doesn’t affect the template’s behavior. To keep the PR focused and diffs minimal, please drop the extra blank line.

param enableAutoScale = {{ .kusto.enableAutoScale }}

[openshift-ci]

New changes are detected. LGTM label has been removed.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

dev-infrastructure/modules/logs/kusto/main.bicep:189

• This module-level comment is overly specific by calling out the deploying identity as “(global MSI)”. The effective behavior is that whatever identity executes the scripts retains Database Admin due to RetainPermissionOnScriptCompletion, which can include local dev callers as well; keeping it generic avoids misleading future readers.

// The deploying identity (global MSI) retains Database Admin on all databases

tooling/templatize/go.mod:44

• tooling/templatize is already pinned to the ARO-Tools multi-module commit v0.0.0-20260729121920-58ceb447fbb3 for config/pipelines/cmdutils/etc, but tools/kustoctl is pinned to an older commit (…202607201…ca2cea2cb707). Mixing different ARO-Tools commits in one build can cause hard-to-debug transitive version skews; consider bumping tools/kustoctl to the same pseudo-version commit as the other ARO-Tools modules here and regenerating go.sum.

	github.com/Azure/ARO-Tools/tools/kustoctl v0.0.0-20260720175808-ca2cea2cb707

[swiencki]

/retest

[swiencki]

/retest

[raelga]

/test e2e-parallel

[raelga]

Ran the EV2 resolve-mode render check locally against this branch (Global entrypoint, aro ev2 manifests test --output-format resolve) and it passes. The KustoEntityGroups step, identityFrom, and the kusto bicep changes all resolve cleanly, and sdp-pipelines main's toolchain already parses/renders the step type. That only validates manifest generation, not the runtime EV2RA adapter, so 16614060 still needs to land first.

Approach looks good, retaining Admin instead of the PrincipalAssignment resources is a cleaner fix for the INT conflict. A few non-blocking things:

1. The entity-groups step has no automatedRetry while kusto-infra/deploy does. It hits every regional Kusto cluster over the network with a 10m timeout, so a transient failure fails the geo rollout. Can we add automatedRetry, or does kustoctl retry internally and is the sync idempotent on re-run?
2. Can you add a line confirming Database Admin is the minimal role entity-group sync needs? Keeps someone from later trying to scope it down and breaking the step.
3. Worth a small unit test on the new run.go dispatch case to cover the timeout parse.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: janboll, swiencki

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• dev-infrastructure/OWNERS [janboll]
• test/OWNERS [janboll]
• tooling/OWNERS [janboll]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 11 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tooling/templatize/pkg/pipeline/kusto_entity_groups_test.go:65

• This unit test invokes runKustoEntityGroupsStep(), which calls opts.Run(ctx) and can make real Azure calls if credentials are available in the test environment. That makes the test non-hermetic and potentially flaky/slow (and could mutate real entity groups). Consider refactoring runKustoEntityGroupsStep to separate/return the parsed timeout (or inject the runner) so the timeout-parse behavior can be tested without executing opts.Run.

			// runKustoEntityGroupsStep will fail at opts.Run (no Azure creds)
			// but we're testing the timeout parse path before that
			err := runKustoEntityGroupsStep(graph.Identifier{}, step, context.Background())

[openshift-ci]

@swiencki: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-unit	5b003bf	link	true	/test test-unit
ci/prow/e2e-parallel	5b003bf	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.