Skip to content

Test elastic-package#3725 - DO NOT MERGE#19907

Draft
elastic-vault-github-plugin-prod[bot] wants to merge 4 commits into
mainfrom
test-elastic-package-pr-3725
Draft

Test elastic-package#3725 - DO NOT MERGE#19907
elastic-vault-github-plugin-prod[bot] wants to merge 4 commits into
mainfrom
test-elastic-package-pr-3725

Conversation

@elastic-vault-github-plugin-prod

@elastic-vault-github-plugin-prod elastic-vault-github-plugin-prod Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Update elastic-package reference to elastic/elastic-package@45bf8de.
Automated by Buildkite build

Relates: elastic/elastic-package#3725

@elastic-vault-github-plugin-prod elastic-vault-github-plugin-prod Bot mentioned this pull request Jul 1, 2026
Draft
@github-actions

This comment has been minimized.

Sign in to view
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@andrewkroh

andrewkroh commented Jul 1, 2026

Copy link
Copy Markdown
Member

I reproduced the hashicorp_vault 0-hit failure locally with the LogsDB Columnar test path and found the ingest rejection reason.

Summary:

  • The input was producing data. The Elastic Agent event log showed hashicorp_vault.audit events being published for logs-hashicorp_vault.audit-43464.

  • Elasticsearch never created the target data stream because the bulk request was rejected with HTTP 400.

  • The rejection reason was:

    field [event.original] cannot reconstruct _source from doc values; every field must be reconstructable from doc values in index using [logsdb_columnar] index mode

  • All three failing hashicorp_vault log system configs set preserve_original_event: true, which populates event.original. This likely explains these 0-hit failures, and may explain other LogsDB Columnar failures in packages that preserve event.original.

I did not patch elastic-package to expose this. I reproduced with --defer-cleanup and inspected the independent Elastic Agent container log at /usr/share/elastic-agent/state/data/logs/elastic-agent-20260701.ndjson.

Full Elastic Agent error JSON 
{"log.level":"error","@timestamp":"2026-07-01T17:34:35.220Z","message":"failed to index document","resource":{"service.instance.id":"56ac4a15-5a49-4c48-9a03-195300de3593","service.name":"/usr/share/elastic-agent/data/elastic-agent-1885ca/components/elastic-otel-collector","service.version":"9.5.0"},"otelcol.component.id":"elasticsearch/_agent-component/default","otelcol.signal":"logs","index":"","http.response.status_code":400,"log.origin.stack_trace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.154.0/bulkindexer.go:425\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*syncBulkIndexerSession).Flush\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.154.0/bulkindexer.go:252\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*sessionList).Flush.func1\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.154.0/exporter.go:666\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.21.0/errgroup/errgroup.go:93","otelcol.component.kind":"exporter","error.type":"illegal_argument_exception","error.reason":"field [event.original] cannot reconstruct _source from doc values; every field must be reconstructable from doc values in index using [logsdb_columnar] index mode","ecs.version":"1.6.0","log.origin":{"file.line":425,"file.name":"elasticsearchexporter@v0.154.0/bulkindexer.go","function":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer"},"ecs.version":"1.6.0"}

@andrewkroh

andrewkroh commented Jul 1, 2026

Copy link
Copy Markdown
Member

The ECS event.original value is defined with doc_values=false and index=false.

https://github.com/elastic/ecs/blob/b85f7573f0b48c03fddf90dad06fc743876f1b33/generated/elasticsearch/composable/component/event.json#L60-L64

which makes it incompatible with columnar index modes because of

Disabling doc values: Mapped fields cannot disable doc values. Setting doc_values to false leads to mapping errors. The only exception is multi-fields, where it's often desirable to store doc values for just one and use different index configurations for the rest.

Disabling _source: Setting "_source": {"enabled": false} is not allowed.

@github-actions

This comment has been minimized.

Sign in to view
elastic-vault-github-plugin-prod and others added 4 commits July 15, 2026 09:47
@andrewkroh
Test elastic-package from PR 3725 - 45bf8deca7695b8e0094d8207af5f87d5…
f45adf5 
…f72ce56
@andrewkroh
ci: run PR-19907 package tests with logsdb_columnar
65b4fa8 
Add a one-off Buildkite path for PR 19907 that forces full package coverage on 9.5.0-SNAPSHOT with LogsDB and LogsDB Columnar enabled. This ensures the normal integrations test flow exercises the elastic-package columnar implementation under realistic CI conditions.
@andrewkroh
update elastic-package to fix mapping validation
fc2fea5
@andrewkroh
Update elastic-package to remove nested-field skip workaround.
47887e1 
Point the test branch at elastic-package commit b2124167, which drops the
LogsDB Columnar nested-mapping skip logic now that Elasticsearch supports
nested fields in columnar mode.

Assisted-By: Composer
@andrewkroh
andrewkroh force-pushed the test-elastic-package-pr-3725 branch from 0c9ad7b to 47887e1 Compare July 15, 2026 14:28
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown

💔 Build Failed

Failed CI Steps

History

cc @andrewkroh

@github-actions

This comment has been minimized.

Sign in to view
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

TL;DR

The four security_detection_engine ECH OOM-test jobs in Buildkite build 46066 failed for a separate infrastructure reason: each timed out while writing its QA state to Elasticsearch. This is distinct from the LogsDB Columnar package-matrix issue already reported above.

Remediation

  • Retry the four ECH jobs/build after confirming the target Elasticsearch deployment is reachable and responsive.
  • If the timeout reproduces, investigate the ECH deployment or Elasticsearch availability/latency rather than package code.
Investigation details

Root Cause

All four appex-qa-stateful-security-prebuilt-rules-ftr-oom-testing-elastic-cloud-create-deployment logs fail in qaf.persistence.kvstore.elasticsearch.persist_entry while calling the Elasticsearch index API. The terminal exception is ConnectionTimeout: Connection timed out.

Evidence

Verification

The prefetched logs provide the same timeout signature for all four jobs; no package source changes are implicated by these failures.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@andrewkroh andrewkroh

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrewkroh