Add LogsDB Columnar support for system tests

[andrewkroh]

Summary

Adds opt-in LogsDB Columnar support to elastic-package system tests so integration teams can validate data streams against logsdb_columnar semantics on supported snapshot stacks.

Closes #3700.

Assisted-by: Cursor (gpt-5.5 for planning, gpt-5.3-codex-high for implementation)

What changed

• Added new stack profile setting: stack.logsdb_columnar_enabled.
• Updated stack rendering to configure Elasticsearch when enabled on supported versions:
  • cluster.logsdb_columnar.enabled: true
  • -Des.columnar_index_mode_feature_flag_enabled=true
• Added new system-test flag: elastic-package test system --logsdb-columnar.
• Updated system runner to:
  • check ES create-index capability columnar_index_modes
  • apply index.mode: logsdb_columnar via scoped logs @custom templates before package install
  • restore/delete prior template state on teardown
• Added nested-field detection and skip behavior for incompatible logs streams under columnar mode.
• Treated logsdb_columnar as synthetic-source-like in system test validation.
• Normalized dotted live mapping keys before comparison so columnar mappings validate against nested simulated index-template mappings.
• Added test coverage for stack rendering, template lifecycle, skip behavior, and synthetic mode detection.

Behavior and compatibility

• No behavior change unless users opt in.
• Columnar mode is gated by ES capability checks and stack version checks.
• Setup/no-provision/tear-down flows preserve template state across phases.

Usage examples

1) Start stack with columnar enabled

elastic-package stack up -d -v --version 9.5.0-SNAPSHOT \
  -U stack.logsdb_columnar_enabled=true

2) Run system tests in columnar mode for a package

elastic-package -C /path/to/integrations/packages/apache \
  test system -v --data-streams access --logsdb-columnar

3) Setup/tests/teardown split flow in columnar mode

elastic-package test system --setup --config-file <config.yml> --logsdb-columnar
elastic-package test system --no-provision --logsdb-columnar
elastic-package test system --tear-down --logsdb-columnar

Validation performed

• go test ./internal/stack/... ./internal/testrunner/runners/system/... ./internal/elasticsearch/... ./cmd/...
• make build format lint licenser gomod update
• Manual proof on 9.5.0-SNAPSHOT for direct logsdb_columnar index creation + indexing.
• End-to-end apache/access system test with --logsdb-columnar passing with scoped template lifecycle logs.

[infra-vault-gh-plugin-prod]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 45bf8de

Failed CI Steps

• :go: Integration test: mysql_input_otel

[github-actions]

TL;DR

The mysql_input_otel integration step did not fail because of the PR changes. The job exited during the post-test safe-log upload when gcloud storage cp could not refresh its auth token (invalid_grant: ID Token ... is stale to sign-in).

Remediation

• Retry the Buildkite job after the GCP auth/token issue is resolved or refreshed on the agent.
• If retries keep failing at the same point, investigate the Buildkite oblt-google-auth-buildkite-plugin credential lifetime/refresh path for safe-log uploads.

Investigation details

Root Cause

This is an infrastructure failure in the Buildkite safe-log upload path, not a source-code/test failure from this PR.

The job log shows the package cleanup finished successfully and then the script reached safe-log upload:

• .buildkite/scripts/integration_tests.sh:173-176 runs make SERVERLESS=... PACKAGE_UNDER_TEST=... ${TARGET} with set +e and stores the result in testReturnCode.
• .buildkite/scripts/integration_tests.sh:179-180 runs upload_package_test_logs before checking testReturnCode.
• .buildkite/scripts/integration_tests.sh:75 prints --- Uploading safe logs to GCP bucket ${JOB_GCS_BUCKET_INTERNAL}.
• .buildkite/scripts/tooling.sh:109 performs the upload with gcloud storage cp ....

The failure occurs exactly at that upload step.

Evidence

• Build: https://buildkite.com/elastic/elastic-package/builds/8289
• Job/step: :go: Integration test: mysql_input_otel
• Command: ./.buildkite/scripts/integration_tests.sh -t test-check-packages-parallel -p mysql_input_otel
• Key log excerpt:

+ exit 0

--- Uploading safe logs to GCP bucket ecosystem-ci-internal

ERROR: (gcloud.storage.cp) There was a problem refreshing your current auth tokens: ('Error code invalid_grant: ID Token issued at 1782877145 is stale to sign-in.', '{"error":"invalid_grant","error_description":"ID Token issued at 1782877145 is stale to sign-in."}')
...
🚨 Error: The command exited with status 1
user command error: exit status 1

The log then shows Buildkite artifact upload succeeding, including test result XML and coverage artifacts, which further points to a post-test upload/auth issue rather than a package test assertion failure.

Verification

• Reviewed the failed job summary and the provided job log under /tmp/gh-aw/buildkite-logs/.
• Checked the PR changed files; the PR does not modify .buildkite/scripts/integration_tests.sh, .buildkite/scripts/tooling.sh, or the Buildkite auth/upload path.
• Checked existing PR comments and review threads; no duplicate detective comment was present.

Follow-up

If this recurs across unrelated PRs, the durable fix is likely in the Buildkite GCP auth plugin configuration or token refresh timing, not in elastic-package test code.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[andrewkroh]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#19907