[BUG] Trigger liquid clustering in Delta Lake integration tests [databricks]

[nartal1]

Fixes #15195.

Description

The tests in delta_lake_liquid_clustering_test.py create Delta tables with CLUSTER BY, but liquid clustering is not triggered automatically. Delta performs liquid clustering only when OPTIMIZE is run.

As a result, these tests were validating writes against tables with liquid-clustering metadata without actually clustering the written data.

This PR adds a shared helper that:

• Runs OPTIMIZE after creating or modifying each liquid-clustered table.
• Covers CTAS, RTAS, append, overwrite, DML, merge, and DataFrame write paths.
• Collects the OPTIMIZE command result so command failures fail the test.

Other liquid-clustering test modules already run OPTIMIZE, so no changes are required there.

Validation with Apache Spark 4.0.0 and Delta Lake 4.0.0:

• delta_lake_liquid_clustering_test.py::test_delta_ctas_sql_liquid_clustering: passed
• delta_lake_liquid_clustering_test.py::test_delta_append_sql_liquid_clustering: passed

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR fixes liquid-clustering tests that were checking Delta log metadata without ever triggering actual data clustering. A new optimize_liquid_clustered_table helper runs OPTIMIZE (via .collect() so failures propagate) after every write operation, and a delta_path_identifier helper encapsulates the delta.\path`` quoting needed for path-based table references.

• optimize_liquid_clustered_table is added once and called consistently after CTAS, RTAS, INSERT, overwrite, DML, merge, and DataFrame write paths, covering the full test matrix.
• Path vs. name consistency: CTAS-style tests (no catalog name) use delta_path_identifier(path), while catalog-registered tables use the table name directly — both are correct.
• [databricks] tag is present in the PR title, which auto-triggers Databricks pre-merge CI for these integration-test changes.

Confidence Score: 5/5

Safe to merge — all OPTIMIZE calls use .collect() so failures propagate, identifier formats are consistent with the write patterns they follow, and the [databricks] tag is present in the PR title.

The change is a focused, mechanical addition of OPTIMIZE calls after each write in liquid-clustering tests. The two helpers are straightforward. Every call site correctly pairs the identifier style (path-based vs. catalog name) with how the table was originally created.

No files require special attention.

Important Files Changed

Filename	Overview
integration_tests/src/main/python/delta_lake_liquid_clustering_test.py	Adds delta_path_identifier and optimize_liquid_clustered_table helpers, then inserts an OPTIMIZE call after each write in every liquid-clustering test (CTAS, RTAS, append, overwrite, DML, merge, DF paths). Fixes tests that were verifying clustered metadata without actually clustering the data.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[write_func / modify_table called by test framework] --> B{Table referenced by}
    B -- catalog name --> C["spark.sql(OPTIMIZE table_name).collect()"]
    B -- path only --> D["delta_path_identifier(path)"]
    D --> E["spark.sql(OPTIMIZE delta.path).collect()"]
    C --> F[Clustering physically applied to Delta files]
    E --> F
    F --> G[Read lambda compares GPU vs CPU results]
    G --> H[assert_liquid_clustering_delta_logs_equivalent]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[write_func / modify_table called by test framework] --> B{Table referenced by}
    B -- catalog name --> C["spark.sql(OPTIMIZE table_name).collect()"]
    B -- path only --> D["delta_path_identifier(path)"]
    D --> E["spark.sql(OPTIMIZE delta.path).collect()"]
    C --> F[Clustering physically applied to Delta files]
    E --> F
    F --> G[Read lambda compares GPU vs CPU results]
    G --> H[assert_liquid_clustering_delta_logs_equivalent]

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'main' of github.com:NVIDIA..." | Re-trigger Greptile}

[nartal1]

build

[nartal1]

build

[nartal1]

build

[nartal1]

build