diff --git a/site/versioning-policy.html b/site/versioning-policy.html index 97727c04f1..1278909557 100644 --- a/site/versioning-policy.html +++ b/site/versioning-policy.html @@ -164,19 +164,32 @@

Spark versions

Each Spark release will be versioned: [MAJOR].[FEATURE].[MAINTENANCE]

+ +

Each feature release will have a merge window where new patches can be merged, a QA window when only fixes can be merged, then a final period where voting occurs on release candidates. These windows will be announced immediately after the previous feature release to give people plenty -of time, and over time, we might make the whole release process more regular (similar to Ubuntu). -

  • MAINTENANCE: Maintenance releases will occur more frequently and depend on specific patches -introduced (e.g. bug fixes) and their urgency. In general these releases are designed to patch bugs. -However, higher level libraries may introduce small features, such as a new algorithm, provided -they are entirely additive and isolated from existing code paths. Spark core may not introduce -any features.
    • +of time.

    +

    Alpha components

    @@ -274,9 +287,10 @@

    Alternatives to breaking an API

    Release cadence

    -

    The branch is cut every January and July, so feature (“minor”) releases occur about every 6 months in general. -Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. Maintenance releases happen as needed -in between feature releases. Major releases do not happen according to a fixed schedule.

    +

    Starting with Spark 4.3, feature releases occur quarterly (every 3 months), containing new features, +improvements, and bug fixes. Major releases occur annually (every 12 months), allowing breaking +changes and dependency upgrades. Maintenance releases happen as needed in between for critical +bug fixes and security patches.

    Spark 4.2 release window

    @@ -303,16 +317,78 @@

    Spark 4.2 release window

    +

    Illustrative transition: 2026 and 2027

    + +

    The calendar below is an example to show how the community expects to bootstrap the faster cadence; +exact dates stay subject to the usual release-discussion and voting process.

    + + +

    Maintenance releases and EOL

    -

    Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months. -For example, branch 2.3.x is no longer considered maintained as of September 2019, 18 months after the release -of 2.3.0 in February 2018. No more 2.3.x releases should be expected after that point, even for bug fixes.

    +

    The following table summarizes the maintenance window for each release type:

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Release TypeCadenceMaintenance Window
    Major (x.0.0)Annually6 months
    Feature (x.[1–2].0)Every 3 months6 months
    LTS (x.3.0)Every 3 months (last feature of the major line)18 months
    Maintenance (x.y.z, z ≥ 1)Ad hocN/A (patches only)
    + +

    Release branches other than LTS will, generally, be maintained with bug fix releases for a period of +6 months (see the Major and Feature rows in the table above).

    + +

    Under the quarterly cadence, the third feature release after each major—x.3.0—is designated +as the LTS (Long-Term Support) release for that major line and is maintained for 18 months (for example +Apache Spark 5.3.0 and 6.3.0). The x.0.0, x.1.0, and x.2.0 trains are maintained for 6 months +each as usual. LTS releases provide a stable target for ecosystem projects and downstream vendors to +standardize around.

    + +

    As an exception while the project transitions into this cadence, Spark 4.x still ships 4.3.0 as the +first quarterly release on the new train, but the 4.x LTS will be Apache Spark 4.5.0 (the last +planned 4.x feature release) rather than 4.3.0. 4.5.x therefore receives the 18-month LTS +window in the same role as x.3.x on newer major lines.

    -

    The last minor release within a major release will typically be maintained for longer as an “LTS” release. -For example, 3.5.0 was released on September 13th 2023 and would normally be maintained for 31 months until April 12th 2026.

    +

    Critical security patches will be backported to all actively maintained branches. Critical bug fixes +(e.g., correctness issues) that may introduce behavior changes will be evaluated by the community +on a case-by-case basis.

    -

    As an exception from the normal versioning policy, version 3.5.x has an “extended” LTS period to allow for migrations to be completed. This extended LTS period will end November 2027. During the 3.5.x extended LTS period, we will only include security fixes. This extend LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won’t fix or release the new version without Java 8 support.

    +

    As an exception from the normal versioning policy, version 3.5.x has an “extended” LTS period to allow for migrations to be completed. This extended LTS period will end November 2027. During the 3.5.x extended LTS period, we will only include security fixes. This extended LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won’t fix or release the new version without Java 8 support.

    diff --git a/versioning-policy.md b/versioning-policy.md index cce0805b08..7d5dc91105 100644 --- a/versioning-policy.md +++ b/versioning-policy.md @@ -17,19 +17,27 @@ These small differences account for Spark's nature as a multi-module project. Each Spark release will be versioned: `[MAJOR].[FEATURE].[MAINTENANCE]` -- **MAJOR**: All releases with the same major version number will have API compatibility. -Major version numbers will remain stable over long periods of time. For instance, 1.X.Y may last -1 year or more. -- **FEATURE**: Feature releases will typically contain new features, improvements, and bug fixes. +- **MAJOR**: Major releases occur annually (every 12 months) as **x.0.0** releases. These releases may +include **breaking changes**, third-party dependency upgrades, and other changes that are not +compatible within the same major line. Within a major line, all **x.y.z** releases share API compatibility +as described in the **FEATURE** and **MAINTENANCE** bullets below. +- **FEATURE**: Feature releases occur quarterly (every 3 months) and contain new features, performance +improvements, API additions, and bug fixes. To ensure safe and predictable upgrades for downstream +projects, feature releases have the following compatibility: + - No third-party dependency upgrades (e.g. Parquet, Arrow, ORC, Hadoop, Netty) by default. Upgrades + required to address **security** issues may be allowed; any other exception is decided case by case by + the release managers and the community. + - No behavior or semantic changes (e.g. SQL semantics, execution behavior, optimizer behavior, + configuration defaults). Exceptions might occur case by case (e.g., security issues). + - Public APIs may be added but not changed or removed. + Each feature release will have a merge window where new patches can be merged, a QA window when only fixes can be merged, then a final period where voting occurs on release candidates. These windows will be announced immediately after the previous feature release to give people plenty -of time, and over time, we might make the whole release process more regular (similar to Ubuntu). -- **MAINTENANCE**: Maintenance releases will occur more frequently and depend on specific patches -introduced (e.g. bug fixes) and their urgency. In general these releases are designed to patch bugs. -However, higher level libraries may introduce small features, such as a new algorithm, provided -they are entirely additive and isolated from existing code paths. Spark core may not introduce -any features. +of time. +- **MAINTENANCE**: Maintenance releases (**x.y.z** with **z ≥ 1**) will occur on an ad hoc basis depending on specific patches +introduced (e.g. critical bug fixes and security patches) and their urgency. They contain bug fixes and +security patches only; they do not introduce new features.

    Alpha components

    @@ -96,9 +104,10 @@ In cases where there is a "Bad API", but where the cost of removal is also high,

    Release cadence

    -The branch is cut every January and July, so feature ("minor") releases occur about every 6 months in general. -Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. Maintenance releases happen as needed -in between feature releases. Major releases do not happen according to a fixed schedule. +Starting with Spark 4.3, feature releases occur quarterly (every 3 months), containing new features, +improvements, and bug fixes. Major releases occur annually (every 12 months), allowing breaking +changes and dependency upgrades. Maintenance releases happen as needed in between for critical +bug fixes and security patches.

    Spark 4.2 release window

    @@ -108,13 +117,48 @@ in between feature releases. Major releases do not happen according to a fixed s | Mid May 2026 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| | Late May 2026 | Release candidates (RC), voting, etc. until final release passes| +

    Illustrative transition: 2026 and 2027

    + +The calendar below is an **example** to show how the community expects to bootstrap the faster cadence; +exact dates stay subject to the usual release-discussion and voting process. + +- **2026**: Ship **Apache Spark 4.2.0** on the timeline above (the last *large* feature drop before the + transition). Treat **4.2.x** like other non-LTS branches (6 month maintenance window). After **4.2.0** + is generally available, plan **Apache Spark 4.3.0** as the first quarterly feature release on the new + train (for example, roughly **three months** after the **4.2.0** GA date for the start of the **4.3** merge/RC + cycle—this is not a fixed rule, only an illustration of quarterly feature releases). +- **2027**: Ship **Apache Spark 5.0.0** as the next **annual** major. Follow with quarterly **5.1.0**, + **5.2.0**, and **5.3.0** feature releases; **5.3.0** is the **5.x** LTS per the **x.3.0** rule above + (for example targeting calendar quarters **2027 Q1** through **Q4** if the **5.0.0** major lands early + in the year). +

    Maintenance releases and EOL

    -Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months. -For example, branch 2.3.x is no longer considered maintained as of September 2019, 18 months after the release -of 2.3.0 in February 2018. No more 2.3.x releases should be expected after that point, even for bug fixes. +The following table summarizes the maintenance window for each release type: + +| Release Type | Cadence | Maintenance Window | +| ----- | ----- | ----- | +| Major (x.0.0) | Annually | 6 months | +| Feature (x.[1–2].0) | Every 3 months | 6 months | +| LTS (x.3.0) | Every 3 months (last feature of the major line) | 18 months | +| Maintenance (x.y.z, z ≥ 1) | Ad hoc | N/A (patches only) | + +Release branches **other than LTS** will, generally, be maintained with bug fix releases for a period of +6 months (see the **Major** and **Feature** rows in the table above). + +Under the quarterly cadence, the **third** feature release after each major—**x.3.0**—is designated +as the LTS (Long-Term Support) release for that major line and is maintained for **18 months** (for example +Apache Spark 5.3.0 and 6.3.0). The **x.0.0**, **x.1.0**, and **x.2.0** trains are maintained for **6 months** +each as usual. LTS releases provide a stable target for ecosystem projects and downstream vendors to +standardize around. + +As an exception while the project transitions into this cadence, **Spark 4.x** still ships **4.3.0** as the +first quarterly release on the new train, but the **4.x** LTS will be **Apache Spark 4.5.0** (the last +planned **4.x** feature release) rather than **4.3.0**. **4.5.x** therefore receives the **18**-month LTS +window in the same role as **x.3.x** on newer major lines. -The last minor release within a major release will typically be maintained for longer as an "LTS" release. -For example, 3.5.0 was released on September 13th 2023 and would normally be maintained for 31 months until April 12th 2026. +Critical security patches will be backported to all actively maintained branches. Critical bug fixes +(e.g., correctness issues) that may introduce behavior changes will be evaluated by the community +on a case-by-case basis. -As an exception from the normal versioning policy, version 3.5.x has an "extended" LTS period to allow for migrations to be completed. This extended LTS period will end *November 2027*. During the 3.5.x extended LTS period, we will only include security fixes. This extend LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won't fix or release the new version without Java 8 support. +As an exception from the normal versioning policy, version 3.5.x has an "extended" LTS period to allow for migrations to be completed. This extended LTS period will end *November 2027*. During the 3.5.x extended LTS period, we will only include security fixes. This extended LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won't fix or release the new version without Java 8 support.