Skip to content

fix: share forward pass / loss in DFlash / DSpark#134

Merged
yubofredwang merged 3 commits into
lightseekorg:mainfrom
Dogacel:dspark-refactoring
Jun 30, 2026
Merged

fix: share forward pass / loss in DFlash / DSpark#134
yubofredwang merged 3 commits into
lightseekorg:mainfrom
Dogacel:dspark-refactoring

Conversation

@Dogacel

@Dogacel Dogacel commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Overview

  1. Share backbone forward pass in DSpark / DFlash.
  2. Add L1 loss support (1/2 * TV) to DFlash training.
  3. Update HF exporter to export vLLM / SGLang compatible keys directly

L1 loss (Same as TV) 0.5, CE loss 0.5 test run on 500 samples:
image

Also trained some internal checkpoints using 0.5 L1 / 0.5 CE loss and they slightly outperform CE only.

For HF exporting, I have re-exported existing DSpark endpoint (huggingface.co/Dogacel/Qwen3-8B-DSpark) and run,

vllm serve Qwen/Qwen3-8B --speculative_config '{"method":"dspark","model":"deepseek-ai/dspark_qwen3_8b_block7","num_speculative_tokens":7, "attention_backend":"FLASH_ATTN", "draft_sample_method": "probabilistic"}'

On the vllm branch: (https://github.com/Dogacel/vllm/tree/dspark-local).

It got ~2.3 acceptance length on SPEED-bench.

Future Work: Potentially moving D-PACE in DSpark's CE objective to get better acceptance, however the anchoring offset is defined differently in DSpark and DFlash, need to look closer.

Dogacel added 3 commits June 29, 2026 23:21
Signed-off-by: Doğaç Eldenk <dogacel@gmail.com>
Signed-off-by: Doğaç Eldenk <dogacel@gmail.com>
Signed-off-by: Doğaç Eldenk <dogacel@gmail.com>
@Dogacel Dogacel marked this pull request as ready for review June 30, 2026 15:00
@yubofredwang yubofredwang merged commit f1c59f1 into lightseekorg:main Jun 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants