feat(kusto): add KustoEntityGroups pipeline step for cross-cluster sync

[swiencki]

Re-lands #6150 (reverted in #6258) and adds the pipeline step that actually runs the entity group sync.

What

From #6150 (re-landed):

• aroHCPPurpose tag on Kusto clusters
• Global MSI Database Admin role assignment
• Pipeline wiring for kustoctl

New in this PR:

• Templatize pipeline runner for the KustoEntityGroups action type (calls entitygroups.DefaultSyncOptions().Run() from ARO-Tools kustoctl)
• entity-groups steps in geography-pipeline.yaml and geography-pipeline-stg.yaml, running after Kusto deploy with the global MSI identity
• Syncs two entity groups:
  • AllServiceLogs:ServiceLogs
  • AllHostedControlPlaneLogs:HostedControlPlaneLogs

Why

Without this step, the infrastructure deploys (tag + MSI role) but the cross-cluster entity groups never get created.

Dependencies

• ARO-Tools #267 (merged): kustoctl entity-groups sync command
• #6150 (merged, then reverted in Revert "feat(kusto): add aroHCPPurpose tag, global MSI role, and kustoctl entity-groups" #6258): infrastructure prerequisites (re-landed here)
• sdp-pipelines PR #16614060: pipeline step config that consumes this runner

Jira: ARO-27830

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: swiencki
Once this PR has been reviewed and has the lgtm label, please assign geoberle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• dev-infrastructure/OWNERS
• test/OWNERS
• tooling/templatize/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Adds first-class support in the tooling/templatize pipeline runner for a new KustoEntityGroups step type, and wires that step into the geography infrastructure pipelines so cross-cluster Kusto entity group sync actually runs after Kusto deployment (using the global MSI identity set up in #6150).

Changes:

• Add KustoEntityGroups step dispatch in the templatize pipeline runner, calling kustoctl’s entitygroups.DefaultSyncOptions().Run().
• Add a dedicated runner implementation (runKustoEntityGroupsStep) including optional timeout parsing and entity group list wiring.
• Extend geography pipelines (prod/int and stg-global v2) to run the entity group sync after the Kusto deploy step, using globalMSIId.

Reviewed changes

Copilot reviewed 6 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tooling/templatize/pkg/pipeline/run.go	Dispatches the new KustoEntityGroups step type during pipeline execution.
tooling/templatize/pkg/pipeline/kusto_entity_groups.go	Implements the runner that invokes kustoctl entity-groups sync via library options.
tooling/templatize/go.mod	Adds the kustoctl module dependency required by the new runner.
tooling/templatize/go.sum	Records new/updated module checksums pulled in by the kustoctl dependency.
test/go.mod	Updates workspace test module indirect deps due to the overall workspace dependency graph changes.
test/go.sum	Records new checksums for updated indirect deps in the test module.
dev-infrastructure/geography-pipeline.yaml	Adds an entity-groups step to run the cross-cluster entity group sync after Kusto deploy.
dev-infrastructure/geography-pipeline-stg.yaml	Adds the same entity-groups step for the STG-global v2 geography pipeline.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 8 changed files in this pull request and generated no new comments.

[swiencki]

/retest

[swiencki]

/hold

[Copilot]

Pull request overview

Copilot reviewed 6 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

dev-infrastructure/geography-pipeline.yaml:18

• This pipeline change removes several previously-defined steps/resourceGroups (e.g., svc + subscription-lookup, kusto-lookup, action-groups, kusto-monitoring, and detach-legacy-monitoring-stack) in addition to adding entity-groups. The PR description says it’s only adding the entity group sync step; if these removals are unintentional, this could stop deploying Kusto monitoring/action group resources.

resourceGroups:
- name: global
  resourceGroup: '{{ .global.rg }}'
  subscription: '{{ .global.subscription.key }}'
  steps:
  - name: output
    action: ARM
    template: templates/output-global.bicep
    parameters: configurations/output-global.tmpl.bicepparam
    deploymentLevel: ResourceGroup
    outputOnly: true
- name: kusto-infra
  resourceGroup: '{{ .kusto.rg }}'
  subscription: '{{ .global.subscription.key }}'
  executionConstraints:

dev-infrastructure/geography-pipeline-stg.yaml:45

• Same as geography-pipeline.yaml: this STG V2 pipeline now only deploys kusto-infra + runs entity-groups, but the diff also deletes prior steps like svc/subscription-lookup, kusto-lookup, action-groups, and kusto-monitoring. This is a material behavioral change that isn’t described in the PR metadata.

$schema: pipeline.schema.v1
rolloutName: Geography Rollout (STG V2)
serviceGroup: Microsoft.Azure.ARO.HCP.Geography.StgGlobal
resourceGroups:
- name: global
  resourceGroup: '{{ .global.rg }}'
  # Hardcoded on purpose — see global-pipeline-stg.yaml. Removed at decommission.
  subscription: 'hcp-stg-global'
  steps:
  - name: output
    action: ARM
    template: templates/output-global.bicep
    parameters: configurations/output-global-stg.tmpl.bicepparam
    deploymentLevel: ResourceGroup
    outputOnly: true
- name: kusto-infra
  resourceGroup: '{{ .kusto.rg }}'
  # Hardcoded on purpose — see global-pipeline-stg.yaml. Removed at decommission.
  subscription: 'hcp-stg-global'
  executionConstraints:
  - clouds:
    - public
    environments:
    - stg
    regions:
    - uksouth
  steps:
  - name: deploy
    action: ARM
    template: templates/kusto.bicep
    parameters: configurations/kusto-stg.tmpl.bicepparam
    deploymentLevel: ResourceGroup

tooling/templatize/pkg/pipeline/run.go:971

• The new KustoEntityGroups step runner doesn’t appear to honor the pipeline YAML identityFrom (e.g., globalMSIId). runKustoEntityGroupsStep is called with only (id, step, ctx) and there’s no resolution of an identity input or passing a credential/clientID into kustoctl, so this step will likely run under whatever ambient identity the process already has (which can fail if it’s not the global MSI with Database Admin).

	case *types.KustoEntityGroupsStep:
		if err := runKustoEntityGroupsStep(id, step, ctx); err != nil {
			return nil, nil, fmt.Errorf("error running Kusto Entity Groups Step: %w", err)
		}
		return nil, nil, nil

[swiencki]

/unhold