[tinker] cascading distributed barrier timeouts with multi-tenant LoRA

[nikhilbarhate99]

Summary

When running Tinker with the Megatron backend and multiple remote vLLM inference engines, concurrent clients calling save_weights_for_sampler() can fail during LoRA adapter loading on one inference engine. After the first failed LoRA load, other concurrent requests appear to fail later inside torch.distributed.barrier() with RuntimeError: Application timeout caused pair closure.

This looks like a server-side concurrency/failure-handling issue around broadcast_to_inference_engines() / _save_lora_adapters_and_sync().

Setup

• SHA hash: cbfb68e391fb4cdaa97e2958e298dcf9c8daf669
• Using the following command to start the server:

uv run --no-sync --extra tinker --extra megatron -m skyrl.tinker.api \
    --host 0.0.0.0 \
    --port 8000 \
    --base-model "Qwen/Qwen3.5-4B" \
    --backend megatron \
    --backend-config '{
        "strategy": "megatron",
        "trainer.use_sample_packing": false,
        "trainer.placement.policy_num_gpus_per_node": 4,
        "trainer.placement.policy_num_nodes": 1,
        "trainer.placement.colocate_all": false,
        "trainer.policy.megatron_config.tensor_model_parallel_size": 4,
        "trainer.policy.megatron_config.pipeline_model_parallel_size": 1,
        "trainer.policy.megatron_config.lora_config.merge_lora": false,
        "trainer.micro_train_batch_size_per_gpu": 16,
        "trainer.micro_forward_batch_size_per_gpu": 16,
        "generator.inference_engine.num_engines": 4,
        "generator.inference_engine.tensor_parallel_size": 1,
        "generator.inference_engine.gpu_memory_utilization": 0.5,
        "trainer.policy.model.lora.max_loras": 4,
        "trainer.policy.model.lora.max_cpu_loras": 4,
        "trainer.logprobs_chunk_size": null
    }'

Failure mode

One request fails while loading a LoRA adapter onto one of the vLLM servers:

ERROR - skyrl: Error processing request <request_id>:
ray::MegatronPolicyWorkerBase.broadcast_to_inference_engines()
  File "/tmp/ray/.../skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py", line <line>, in broadcast_to_inference_engines
    await self._save_lora_adapters_and_sync(lora_sync_path, inference_engine_client, lora_name=lora_name)
  File "/tmp/ray/.../skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py", line <line>, in _save_lora_adapters_and_sync
    await inference_engine_client.load_lora_adapter(lora_name, lora_sync_path)
  File "/tmp/ray/.../skyrl/backends/skyrl_train/inference_engines/remote_inference_client.py", line <line>, in load_lora_adapter
    raise RuntimeError(
RuntimeError: Failed to load LoRA adapter 'model_<id>' from /tmp/skyrl_lora_sync/model_<id> on 1/4 server(s): ClientConnectorError: Cannot connect to host <vllm-server>:<port> ssl:default [Connect call failed ('<vllm-server>', <port>)]

Other concurrent requests then fail in the distributed barrier:

ERROR - skyrl: Error processing request <request_id>:
ray::MegatronPolicyWorkerBase.broadcast_to_inference_engines()
  File "/tmp/ray/.../skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py", line <line>, in broadcast_to_inference_engines
    await self._save_lora_adapters_and_sync(lora_sync_path, inference_engine_client, lora_name=lora_name)
  File "/tmp/ray/.../skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py", line <line>, in _save_lora_adapters_and_sync
    torch.distributed.barrier()
  File "/workspace/SkyRL/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 5037, in barrier
    work.wait()
RuntimeError: Application timeout caused pair closure

A second request shows the same barrier failure:

ERROR - skyrl: Error processing request <request_id>:
ray::MegatronPolicyWorkerBase.broadcast_to_inference_engines()
  File "/tmp/ray/.../skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py", line <line>, in broadcast_to_inference_engines
    await self._save_lora_adapters_and_sync(lora_sync_path, inference_engine_client, lora_name=lora_name)
  File "/tmp/ray/.../skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py", line <line>, in _save_lora_adapters_and_sync
    torch.distributed.barrier()
  File "/workspace/SkyRL/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 5037, in barrier
    work.wait()
RuntimeError: Application timeout caused pair closure