[LoRA] Investigate LoRA training inference mismatch for Qwen3.6 MoE and GLM-4.7-Flash

[CharlieFRuan]

See:

• [script] Add Qwen3.6-35b LoRA DAPO script #1775

Had similar observation in GLM-4.7-Flash and other Megatron + vLLM RL frameworks (during time of testing).

After fixing, re-run DAPO to see if reward/eval curve improves

[benrhodes26]

@CharlieFRuan: Relatedly, I've found that multimodal qwen3.5 and 3.6 models are not learning anything when using the Tinker entrypoint.

My ablations showed that lora_rank=0 works fine, but lora_rank=32 learns nothing, and the KL divergence between fsdp model and vllm model explodes.

This patch fixed it however:

        old=(
            "        if persist:\n"
            "            # TODO(tyler): For LoRA, only save the adapters instead of the full merged model\n"
            "            with tempfile.TemporaryDirectory() as temp_dir:\n"
            '                hf_dir = os.path.join(temp_dir, "model")\n'
            '                self._dispatch.save_hf_model(model="policy", export_dir=hf_dir, tokenizer=self._tokenizer)\n'
            "                self._create_tar_from_directory(hf_dir, output_path)\n"
            '            logger.info(f"Saved sampler checkpoint for {model_id} to {output_path}")\n'
        ),
        new=(
            "        if persist:\n"
            "            lora_cfg = self._cfg.trainer.policy.model.lora\n"
            "            is_lora = lora_cfg and lora_cfg.rank > 0\n"
            "            if is_lora:\n"
            "                lora_sync_path = lora_cfg.lora_sync_path\n"
            "                self._create_tar_from_directory(lora_sync_path, output_path)\n"
            '                logger.info(f"Saved LoRA adapter checkpoint for {model_id} to {output_path}")\n'
            "            else:\n"
            "                with tempfile.TemporaryDirectory() as temp_dir:\n"
            '                    hf_dir = os.path.join(temp_dir, "model")\n'
            '                    self._dispatch.save_hf_model(model="policy", export_dir=hf_dir, tokenizer=self._tokenizer)\n'
            "                    self._create_tar_from_directory(hf_dir, output_path)\n"
            '                logger.info(f"Saved sampler checkpoint for {model_id} to {output_path}")\n'
        ),