Core Space Merging Algorithm | Feature Request

[kishore-s-15]

Add Core Space Merge Method

Summary

This PR adds the Core Space merge method to mergekit, implementing the algorithm from "Accurate and Efficient Low-Rank Model Merging in Core Space" (Panariello et al., NeurIPS 2025).

Core Space enables efficient merging of LoRA-adapted models by operating in a compact, SVD-aligned subspace, providing significant computational savings while guaranteeing information preservation.

Motivation

LoRA (Low-Rank Adaptation) has become a popular technique for parameter-efficient fine-tuning. However, existing merge methods in mergekit operate in full parameter space, which:

• Wastes computational resources when merging low-rank adaptations
• Doesn't account for misaligned LoRA subspaces
• Scales poorly with model dimension

Core Space addresses these issues by:

• Operating in compact core space (rank² vs dimension²)
• Using SVD to align LoRA subspaces before merging
• Guaranteeing lossless information preservation
• Supporting heterogeneous ranks across adapters

Changes Made

New Files

1. mergekit/merge_methods/core_space.py (~300 lines)

Complete implementation of the Core Space algorithm:

Classes:

• CoreSpaceMerge(MergeMethod) - Main merge method class
• CoreSpaceTask(Task[torch.Tensor]) - Task execution class

Key Methods:

• _compute_reference_bases() - Computes U_B and V_A via SVD on concatenated LoRA matrices
• _extract_lora_matrices() - Extracts or approximates A and B matrices from models
• _core_space_merge() - Projects to core space, merges, and reconstructs
• _weighted_average() - Fallback for non-LoRA weights

Algorithm:

# 1. Concatenate LoRA matrices
B_concat = torch.cat([B_1, B_2, ..., B_n], dim=1)  # Horizontal
A_concat = torch.cat([A_1, A_2, ..., A_n], dim=0)  # Vertical

# 2. Compute reference bases via SVD
U_B, _, _ = SVD(B_concat)
_, _, V_A = SVD(A_concat)

# 3. Project to core space
Core_i = U_B^T @ B_i @ A_i @ V_A

# 4. Merge in core space
Core_merged = mean(Core_1, Core_2, ..., Core_n)

# 5. Reconstruct
ΔW_merged = U_B @ Core_merged @ V_A^T
W_final = W_base + ΔW_merged

2. tests/test_core_space.py (10 tests)

Comprehensive test suite covering:

• Method initialization and configuration
• SVD reference basis computation
• LoRA weight detection logic
• Weighted averaging fallback
• Low-rank approximation via SVD
• Core space projection and reconstruction
• Parameter handling
• Multi-model merge workflow
• Comparison with naive merging

All tests are passing.

3. examples/core_space.yml

Example configuration demonstrating basic usage with multiple LoRA adapters.

Configuration

Basic usage:

models:
  - model: gpt2
    parameters:
      weight: 0.5
  - model: gpt2
    parameters:
      weight: 1.0

merge_method: core_space
base_model: gpt2
dtype: float32

References

Paper:

@inproceedings{panariello2025accurate,
  title = {Accurate and Efficient Low-Rank Model Merging in Core Space},
  author = {Panariello, Aniello and Marczak, Daniel and Magistri, Simone and 
            Porrello, Angelo and Twardowski, Bart{\l}omiej and 
            Bagdanov, Andrew D. and Calderara, Simone and van de Weijer, Joost},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2025}
}

Original Implementation: https://github.com/apanariello4/core-space-merging

---

Note

Adds a new SVD-aligned LoRA merging method that operates in a compact core subspace, plus docs, example, tests, and registry wiring.

• Introduces core_space merge method (mergekit/merge_methods/core_space.py) with low-rank SVD projection, core-space averaging, and fallback to weighted deltas
• Registers method in merge_methods/registry.py; no changes to existing methods
• Updates docs: README method table and detailed guide (docs/merge_methods.md), including usage, parameters, and reference
• Adds runnable example examples/core_space.yml
• Adds unit tests tests/test_core_space.py covering SVD bases, low-rank approximation, projection/reconstruction, parameter handling

^{Written by Cursor Bugbot for commit 58d750b. This will update automatically on new commits. Configure here.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[kishore-s-15]

I have read the CLA Document and I hereby sign the CLA

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

Unused _is_lora_weight method is dead code

Low Severity

The _is_lora_weight method is defined but never called anywhere in the codebase. It always returns False with a comment "For now, treat all as full weights". Since no code path invokes this method, it adds unnecessary complexity and could confuse future maintainers about the intended design.

 

[kishore-s-15]

Hi @cg123,
Let me know if you have any questions or doubts while integrating the Core Space Merging algorithm with MergeKit. I’d love to see it integrated into MergeKit and am happy to help if needed.