doc(spark-version): document Spark 3.5 LTS as the target; Spark 4 future-tracked#15
Open
estebanzimanyi wants to merge 1 commit into
Open
Conversation
…ure-tracked Adds doc/spark-version.md as the canonical position statement on Spark-version targeting: - MobilitySpark targets Apache Spark 3.5.x (LTS through 2026). - Spark 4.0 (May 2025 GA) is early-adoption, not production-default. - The MEOS-C-library-version axis is orthogonal and handled inside JMEOS (one JMEOS jar per MEOS version; MobilitySpark pins one JMEOS jar). - Five trigger signals enumerated for when to commit to Spark 4 support; commit when two or more fire. - The future Spark 4 effort, if/when needed, will mirror MobilityDuck's multi-DuckDB-version foundation (Maven profile per Spark version, CI matrix, per-version jar artefacts). README updated to bump the Spark requirement from the stale 3.4.0 line to 3.5.x with a pointer to the new doc. The position is asymmetric to MobilityDuck's multi-DuckDB-version work: DuckDB v1.4/v1.5 are both production with split adopter base today, whereas Spark 3.5 is the production line and 4.0 is the early-adoption line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
doc/spark-version.mdas the canonical position statement on Spark-version targeting, plus a README bump from the staleSpark 3.4.0line toSpark 3.5.xwith a pointer to the new doc.Position
Trigger signals (documented in the doc)
Commit to Spark 4 when two or more of these fire:
None have fired today.
Asymmetry vs MobilityDuck
The DuckDB v1.4 vs v1.5 work in MobilityDuck #166 / #167 is urgent because both DuckDB versions are production with a split adopter base. Spark 3.5 vs 4.0 is asymmetric — 3.5 is the production line, 4.0 is the early-adoption line — so this PR documents the position rather than expanding the build matrix.
The future Spark 4 effort, if/when needed, will mirror MobilityDuck's foundation pattern: Maven profile per Spark version, CI matrix dimension, per-version jar artefacts.