`nccl_api_->winPut` Error Using RMA Windows

[BenBrock]

Should I expect the RMA window interface to work with NCCLX (or any of the other backends), or is it still WIP?

I get the following error on issuing a win.put(...):

F0202 09:40:35.775741 120495 TorchCommWindowNCCLX.cpp:130] Check failed: nccl_api_->winPut( tensor.data_ptr(), tensor.numel(), torch_comm_->getNcclDataType(tensor), dstRank, targetOffsetNelems, win_, stream) == ncclSuccess (5 vs. 0)

Setting NCCL_DEBUG=WARN produces an error: NCCL ERROR Invalid window handle in ncclPut.

I've installed TorchComms using CUDA 12.9 and the pip nightly wheels, although I get the same results when building from source. I'm running on a system with four H200s fully connected by NVLink.

Note I opened another issue #410 about which interface should be used for creating an RMA window.

Here's my minimal program:

#!/usr/bin/env python3
import os
import torch
import torchcomms

def main():
    # Initialize TorchComm with NCCLX backend
    device = torch.device("cuda")
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")

    # Get rank and world size
    rank = comm.get_rank()
    world_size = comm.get_size()

    # Calculate device ID
    num_devices = torch.cuda.device_count()
    device_id = rank % num_devices
    target_device = torch.device(f"cuda:{device_id}")

    print(f"Rank {rank}/{world_size}: Running on device {device_id}")

    size = 1024
    dtype = torch.float32

    allocator = torchcomms.get_mem_allocator(comm.get_backend())

    pool = torch.cuda.MemPool(allocator)

    with torch.cuda.use_mem_pool(pool):
        win_buf = torch.ones(
            [size], dtype=dtype, device=device
        )

    print(comm.get_backend())
    print(allocator)

    print(win_buf)

    comm.barrier(False)

    win = comm.new_window()
    win.tensor_register(win_buf)
    comm.barrier(False)

    # Program

    if rank == 1:
        send_data = torch.zeros((10,), dtype=dtype, device=device)
        win.put(send_data, dst_rank=0, target_offset_nelems=0, async_op=False)

    comm.barrier(False)

    if rank == 0:
        print(win_buf)

    comm.barrier(False)
    win.tensor_deregister()

    # Cleanup
    comm.finalize()

if __name__ == "__main__":
    main()

And the full NCCL_DEBUG=WARN output:

(torchcomms-pip-2025-02-02) bash-5.2$ NCCL_DEBUG=WARN torchrun --nproc_per_node=2 ./put_01.py
W0202 09:44:44.157000 120687 site-packages/torch/distributed/run.py:851]
W0202 09:44:44.157000 120687 site-packages/torch/distributed/run.py:851] *****************************************
W0202 09:44:44.157000 120687 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0202 09:44:44.157000 120687 site-packages/torch/distributed/run.py:851] *****************************************
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0202 09:44:46.515873 120755 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0202 09:44:46.515873 120756 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0202 09:44:46.515902 120755 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0202 09:44:46.515903 120756 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0202 09:44:46.515919 120756 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0202 09:44:46.515923 120755 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0202 09:44:46.782441 120756 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 2 CUDA devices
I0202 09:44:46.782469 120756 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:1
I0202 09:44:46.794304 120755 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 2 CUDA devices
I0202 09:44:46.794325 120755 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:0
[E202 09:44:47.148137560 TCPStore.cpp:277] [c10d] The server socket on 29500 has failed to bind. TORCHELASTIC_USE_AGENT_STORE is enabled so ignoring the error.
I0202 09:44:47.079793 120755 CudaWrap.cc:267] cudaDriverVersion 13000
I0202 09:44:47.079841 120755 CudaWrap.cc:269] CUDART_VERSION 12090
I0202 17:44:47.079884548.079884 120755 showVersion:653] pcl-kini02:120755:120755 [0][main] NCCL INFO NCCL version 2.28.9x-git-107bbda+cuda12.9
I0202 09:44:47.082183 120756 CudaWrap.cc:267] cudaDriverVersion 13000
I0202 09:44:47.082224 120756 CudaWrap.cc:269] CUDART_VERSION 12090
W0202 17:44:47.289372953.289372 120755 DataSink.h:26] pcl-kini02:120755:120755 [0][main] NCCL WARN Empty sink for dataset: nccl_profiler_slow_rank. No logging will be done.
W0202 17:44:47.289491478.289491 120755 DataSink.h:26] pcl-kini02:120755:120755 [0][main] NCCL WARN Empty sink for dataset: nccl_profiler_algo. No logging will be done.
W0202 17:44:47.289638639.289638 120755 DataTable.cc:40] pcl-kini02:120755:120755 [0][main] NCCL WARN Failed to create scuba file: filesystem error: cannot create directories: Permission denied [/logs]
W0202 17:44:47.289708714.289708 120755 DataTable.cc:40] pcl-kini02:120755:120755 [0][main] NCCL WARN Failed to create scuba file: filesystem error: cannot create directories: Permission denied [/logs]
W0202 17:44:47.289803979.289803 120755 DataTableWrapper.cc:89] pcl-kini02:120755:120755 [0][main] NCCL WARN Could not find table name: nccl_structured_logging
W0202 17:44:47.289803979.289803 120755 DataTableWrapper.cc:89] pcl-kini02:120755:120755 [0][main] NCCL WARN
W0202 17:44:47.289833013.289833 120755 ncclIbInit:852] pcl-kini02:120755:120755 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/transport/net_ib.cc:852 -> 3
W0202 17:44:47.293188688.293188 120756 DataSink.h:26] pcl-kini02:120756:120756 [1][main] NCCL WARN Empty sink for dataset: nccl_profiler_slow_rank. No logging will be done.
W0202 17:44:47.293288186.293288 120756 DataSink.h:26] pcl-kini02:120756:120756 [1][main] NCCL WARN Empty sink for dataset: nccl_profiler_algo. No logging will be done.
W0202 17:44:47.293428325.293428 120756 DataTable.cc:40] pcl-kini02:120756:120756 [1][main] NCCL WARN Failed to create scuba file: filesystem error: cannot create directories: Permission denied [/logs]
W0202 17:44:47.293485834.293485 120756 DataTable.cc:40] pcl-kini02:120756:120756 [1][main] NCCL WARN Failed to create scuba file: filesystem error: cannot create directories: Permission denied [/logs]
W0202 17:44:47.293589336.293589 120756 DataTableWrapper.cc:89] pcl-kini02:120756:120756 [1][main] NCCL WARN Could not find table name: nccl_structured_logging
W0202 17:44:47.293589336.293589 120756 DataTableWrapper.cc:89] pcl-kini02:120756:120756 [1][main] NCCL WARN
W0202 17:44:47.293627446.293627 120756 ncclIbInit:852] pcl-kini02:120756:120756 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/transport/net_ib.cc:852 -> 3
W0202 17:44:47.551827834.551827 120755 DataTableWrapper.cc:89] pcl-kini02:120755:120755 [0][main] NCCL WARN Could not find table name: nccl_memory_logging
W0202 17:44:47.551827834.551827 120755 DataTableWrapper.cc:89] pcl-kini02:120755:120755 [0][main] NCCL WARN
I0202 09:44:47.556586 120755 CommStateX.cc:297] CommStateX: set rankTopology with system
I0202 09:44:47.556583 120756 CommStateX.cc:297] CommStateX: set rankTopology with system
Rank 0/2: Running on device 0
Rank 1/2: Running on device 1
ncclx
<torch._C._CUDAPluggableAllocator object at 0x147ba4dcd230>
tensor([1., 1., 1.,  ..., 1., 1., 1.], device='cuda:0')
ncclx
<torch._C._CUDAPluggableAllocator object at 0x15429282d2f0>
tensor([1., 1., 1.,  ..., 1., 1., 1.], device='cuda:1')
E0202 17:44:48.003116073.003116 120756 rma.cc:19] pcl-kini02:120756:120756 [1][main] NCCL ERROR Invalid window handle in ncclPut
W0202 17:44:48.003340242.003340 120756 ncclPut:81] pcl-kini02:120756:120756 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/meta/rma/rma.cc:81 -> 5
F0202 09:44:48.003360 120756 TorchCommWindowNCCLX.cpp:130] Check failed: nccl_api_->winPut( tensor.data_ptr(), tensor.numel(), torch_comm_->getNcclDataType(tensor), dstRank, targetOffsetNelems, win_, stream) == ncclSuccess (5 vs. 0)
*** Check failure stack trace: ***
    @     0x15439ca88652  google::LogMessage::Fail()
    @     0x15439ca885b0  google::LogMessage::SendToLog()
    @     0x15439ca87f07  google::LogMessage::Flush()
    @     0x15439ca8b3fc  google::LogMessageFatal::~LogMessageFatal()
    @     0x15427e92819f  torch::comms::TorchCommWindowNCCLX::put()
    @     0x1543a28994e8  _ZNO8pybind116detail15argument_loaderIJRN5torch5comms15TorchCommWindowERKN2at6TensorEimbSt8optionalISt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESH_St4hashISH_ESt8equal_toISH_ESaISt4pairIKSH_SH_EEEESA_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEEEE4callIN3c1013intrusive_ptrINS3_9TorchWorkENS10_6detail34intrusive_target_default_null_typeIS12_EEEENS_18gil_scoped_releaseERZL20pybind11_init__commsRNS_7module_EEUlS5_S9_imbSR_SX_E_EENSt9enable_ifIXntsrSt7is_voidIT_E5valueES1E_E4typeEOT1_.isra.0
    @     0x1543a289be73  _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRN5torch5comms15TorchCommWindowERKN2at6TensorEimbSt8optionalISt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESJ_St4hashISJ_ESt8equal_toISJ_ESaISt4pairIKSJ_SJ_EEEESC_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEEE_N3c1013intrusive_ptrINS5_9TorchWorkENS11_6detail34intrusive_target_default_null_typeIS13_EEEEJS7_SB_imbST_SZ_EJNS_4nameENS_9is_methodENS_7siblingEA1071_cNS_3argES1C_S1C_S1C_NS_5arg_vES1D_NS_10call_guardIJNS_18gil_scoped_releaseEEEEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE_4_FUNES1U_
    @     0x1543a28bc36d  pybind11::cpp_function::dispatcher()
    @     0x55a30a252468  cfunction_call
    @     0x55a30a22923c  _PyObject_MakeTpCall.localalias
    @     0x55a30a233581  _PyEval_EvalFrameDefault
    @     0x55a30a2eb93f  PyEval_EvalCode
    @     0x55a30a3276ea  run_eval_code_obj
    @     0x55a30a322215  run_mod
    @     0x55a30a31f220  pyrun_file
    @     0x55a30a31eee6  _PyRun_SimpleFileObject.localalias
    @     0x55a30a31ec14  _PyRun_AnyFileObject.localalias
    @     0x55a30a31b7ae  Py_RunMain.localalias
    @     0x55a30a2d3877  Py_BytesMain
    @     0x1543e9f4230e  __libc_start_call_main
    @     0x1543e9f423c9  __libc_start_main_alias_2
    @     0x55a30a2d36c7  (unknown)
    @              (nil)  (unknown)
W0202 09:44:52.370000 120687 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 120755 closing signal SIGTERM
*** Aborted at 1770054292 (unix time) try "date -d @1770054292" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGTERM (@0xb901fc0001d76f) received by PID 120755 (TID 0x147cfbe9f740) from PID 120687; stack trace: ***
    @     0x147cfbee2b40 (unknown)
    @     0x147caf061274 (unknown)
    @     0x147caee1f183 (unknown)
    @     0x147caee5ddeb (unknown)
    @     0x147cafb9c8f7 (unknown)
    @     0x147cafb9cd45 (unknown)
    @     0x147caee2b9f4 (unknown)
    @     0x147caeef1fb3 (unknown)
    @     0x147cafb51765 (unknown)
    @     0x147caefa8027 (unknown)
    @     0x147caef9ef10 cuStreamSynchronize
    @     0x147ced8128b3 (unknown)
    @     0x147ced87ce70 cudaStreamSynchronize
    @     0x147bb54d39b5 at::native::nonzero_cuda_out_impl<>()
    @     0x147bb5487c06 at::native::nonzero_out_cuda()
    @     0x147bb5487f02 at::native::nonzero_cuda()
    @     0x147bb673511d c10::impl::wrap_kernel_functor_unboxed_<>::call()
    @     0x147c053db94d at::_ops::nonzero::call()
    @     0x147c04dbc7cc at::native::make_info()
    @     0x147c04db77b3 at::meta::structured_index_Tensor::meta()
    @     0x147bb6897ad4 at::(anonymous namespace)::wrapper_CUDA_index_out_Tensor_out()
    @     0x147bb6b0f326 at::native::masked_select_out_cuda_impl()
    @     0x147bb6b0f95e at::native::masked_select_cuda()
    @     0x147bb6747530 c10::impl::wrap_kernel_functor_unboxed_<>::call()
    @     0x147c056f2538 at::_ops::masked_select::redispatch()
    @     0x147c0809717b torch::autograd::VariableType::(anonymous namespace)::masked_select()
    @     0x147c08097572 c10::impl::wrap_kernel_functor_unboxed_<>::call()
    @     0x147c05786231 at::_ops::masked_select::call()
    @     0x147c187f63d2 torch::autograd::THPVariable_masked_select()
    @     0x55b2263e7468 cfunction_call
    @     0x55b2263be23c _PyObject_MakeTpCall.localalias
    @     0x55b2263c8581 _PyEval_EvalFrameDefault
E0202 09:44:52.584000 120687 site-packages/torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: -6) local_rank: 1 (pid: 120756) of binary: /data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/bin/python3.12
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/bin/torchrun", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/lib/python3.12/site-packages/torch/distributed/run.py", line 990, in main
    run(args)
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/lib/python3.12/site-packages/torch/distributed/run.py", line 981, in run
    elastic_launch(
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-02/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
========================================================
./put_01.py FAILED
--------------------------------------------------------
Failures:
[1]:
  time      : 2026-02-02_09:44:52
  host      : pcl-kini02.sc.intel.com
  rank      : 0 (local_rank: 0)
  exitcode  : -15 (pid: 120755)
  error_file: <N/A>
  traceback : Signal 15 (SIGTERM) received by PID 120755
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-02-02_09:44:52
  host      : pcl-kini02.sc.intel.com
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 120756)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 120756
========================================================

[tianfengfrank]

Hi @BenBrock, can you please set NCCL_DEBUG=INFO instead of WARN to get more diagnostic information for further debugging? And also, did you turn on NCCL_CTRAN_ENABLE=1? Currently window API implementation requires Ctran support.

[BenBrock]

Hi @tianfengfrank, thanks for the response.

With NCCL_CTRAN_ENABLE=1, I get different errors, about failing to load hostname and topology from /etc/fbwhoami (is this a Meta system–internal thing?). Full errors, without and with NCCL_DEBUG=INFO below.

With NCCL_CTRAN_ENABLE=1:

(torchcomms-pip-2025-02-13) bash-5.2$ NCCL_CTRAN_ENABLE=1 torchrun --rdzv-endpoint=localhost:2503 --nproc_per_node=3 ./rma_test.py
W0213 09:48:55.579000 344209 site-packages/torch/distributed/run.py:851]
W0213 09:48:55.579000 344209 site-packages/torch/distributed/run.py:851] *****************************************
W0213 09:48:55.579000 344209 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0213 09:48:55.579000 344209 site-packages/torch/distributed/run.py:851] *****************************************
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0213 09:48:58.518404 344312 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0213 09:48:58.518406 344311 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0213 09:48:58.518409 344313 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0213 09:48:58.518440 344312 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0213 09:48:58.518440 344311 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0213 09:48:58.518441 344313 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0213 09:48:58.518453 344313 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0213 09:48:58.518453 344312 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0213 09:48:58.518453 344311 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0213 09:48:59.379000 344312 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 3 CUDA devices
I0213 09:48:59.379014 344312 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:1
I0213 09:48:59.389258 344313 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 3 CUDA devices
I0213 09:48:59.389273 344313 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:2
I0213 09:48:59.416944 344311 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 3 CUDA devices
I0213 09:48:59.416957 344311 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:0
[E213 09:48:59.998060318 TCPStore.cpp:277] [c10d] The server socket on 2503 has failed to bind. TORCHELASTIC_USE_AGENT_STORE is enabled so ignoring the error.
I0213 09:48:59.833614 344311 CudaWrap.cc:267] cudaDriverVersion 13000
I0213 09:48:59.833657 344311 CudaWrap.cc:269] CUDART_VERSION 12090
I0213 09:48:59.836304 344313 CudaWrap.cc:267] cudaDriverVersion 13000
I0213 09:48:59.836343 344313 CudaWrap.cc:269] CUDART_VERSION 12090
I0213 09:48:59.836356 344312 CudaWrap.cc:267] cudaDriverVersion 13000
I0213 09:48:59.836391 344312 CudaWrap.cc:269] CUDART_VERSION 12090
I0213 09:49:00.951281 344312 CommStateX.cc:297] CommStateX: set rankTopology with system
I0213 09:49:00.951287 344313 CommStateX.cc:297] CommStateX: set rankTopology with system
I0213 09:49:00.951298 344311 CommStateX.cc:297] CommStateX: set rankTopology with system
E0213 09:49:00.951353 344312 Topology.cc:111] Failed to load hostname (DEVICE_NAME) from /etc/fbwhoami
E0213 09:49:00.951357 344313 Topology.cc:111] Failed to load hostname (DEVICE_NAME) from /etc/fbwhoami
E0213 09:49:00.951362 344311 Topology.cc:111] Failed to load hostname (DEVICE_NAME) from /etc/fbwhoami
E0213 09:49:00.951364 344312 CommStateX.cc:106] Check failed: false - Failed to load topology from /etc/fbwhoami
E0213 09:49:00.951366 344313 CommStateX.cc:106] Check failed: false - Failed to load topology from /etc/fbwhoami
E0213 09:49:00.951371 344311 CommStateX.cc:106] Check failed: false - Failed to load topology from /etc/fbwhoami
E0213 09:49:00.951936 344313 TorchCommNCCLX.cpp:77] [TC][name=main_comm] TorchCommNCCLX main_comm was not finalized before destruction
E0213 09:49:00.951973 344311 TorchCommNCCLX.cpp:77] [TC][name=main_comm] TorchCommNCCLX main_comm was not finalized before destruction
I0213 09:49:00.952019 344313 TorchCommNCCLXUtils.cpp:487] [TC][name=main_comm] Detaching memory hook comm=0x55a0e65197f0
E0213 09:49:00.952024 344312 TorchCommNCCLX.cpp:77] [TC][name=main_comm] TorchCommNCCLX main_comm was not finalized before destruction
I0213 09:49:00.952064 344311 TorchCommNCCLXUtils.cpp:487] [TC][name=main_comm] Detaching memory hook comm=0x55ff170607f0
I0213 09:49:00.952105 344312 TorchCommNCCLXUtils.cpp:487] [TC][name=main_comm] Detaching memory hook comm=0x56016d27f740
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 63, in <module>
    main()
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 9, in main
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to initialize NCCL communicator: internal error - please report this issue to the NCCL developers
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 63, in <module>
    main()
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 9, in main
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to initialize NCCL communicator: internal error - please report this issue to the NCCL developers
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 63, in <module>
    main()
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 9, in main
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to initialize NCCL communicator: internal error - please report this issue to the NCCL developers
W0213 09:49:01.384000 344209 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 344311 closing signal SIGTERM
W0213 09:49:01.385000 344209 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 344312 closing signal SIGTERM
E0213 09:49:01.549000 344209 site-packages/torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 2 (pid: 344313) of binary: /data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/bin/python3.12
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/bin/torchrun", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/run.py", line 990, in main
    run(args)
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/run.py", line 981, in run
    elastic_launch(
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
./rma_test.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2026-02-13_09:49:01
  host      : pcl-kini02.sc.intel.com
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 344311)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2026-02-13_09:49:01
  host      : pcl-kini02.sc.intel.com
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 344312)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-02-13_09:49:01
  host      : pcl-kini02.sc.intel.com
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 344313)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

With NCCL_CTRAN_ENABLE=1 and NCCL_DEBUG=INFO:

W0213 09:51:48.938000 344359 site-packages/torch/distributed/run.py:851]
W0213 09:51:48.938000 344359 site-packages/torch/distributed/run.py:851] *****************************************
W0213 09:51:48.938000 344359 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0213 09:51:48.938000 344359 site-packages/torch/distributed/run.py:851] *****************************************
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0213 09:51:51.848022 344461 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0213 09:51:51.848024 344462 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0213 09:51:51.848024 344460 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0213 09:51:51.848059 344461 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0213 09:51:51.848060 344462 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0213 09:51:51.848060 344460 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0213 09:51:51.848073 344461 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0213 09:51:51.848073 344462 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0213 09:51:51.848073 344460 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0213 09:51:52.708536 344462 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 3 CUDA devices
I0213 09:51:52.708551 344462 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:2
I0213 09:51:52.722155 344461 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 3 CUDA devices
I0213 09:51:52.722169 344461 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:1
I0213 09:51:52.734738 344460 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 3 CUDA devices
I0213 09:51:52.734753 344460 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:0
[E213 09:51:53.309099127 TCPStore.cpp:277] [c10d] The server socket on 2503 has failed to bind. TORCHELASTIC_USE_AGENT_STORE is enabled so ignoring the error.
I0213 17:51:53.143454273.143454 344460 GlobalHints.cc:106] pcl-kini02:344460:344460 [0][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.crashOnAsyncError is registered.
I0213 17:51:53.143491982.143491 344460 GlobalHints.cc:106] pcl-kini02:344460:344460 [0][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.crashOnTimeout is registered.
I0213 17:51:53.143498458.143498 344460 GlobalHints.cc:106] pcl-kini02:344460:344460 [0][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.timeoutMs is registered.
I0213 17:51:53.143503875.143503 344460 GlobalHints.cc:106] pcl-kini02:344460:344460 [0][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.comm.useCtran is registered.
I0213 17:51:53.143508600.143508 344460 GlobalHints.cc:106] pcl-kini02:344460:344460 [0][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.comm.algo_reducescatter is registered.
I0213 17:51:53.144058017.144058 344460 openPluginLib:107] pcl-kini02:344460:344460 [0][main] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
I0213 17:51:53.144088215.144088 344460 setCpuStackSize:128] pcl-kini02:344460:344460 [0][main] NCCL INFO Stack size limit (unlimited) is unsafe; will use 8192KB for newly launched threads
I0213 17:51:53.144331799.144331 344460 findInterfaces:194] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL_SOCKET_IPADDR_PREFIX , current addrString 192.168.7.4
I0213 17:51:53.144345732.144345 344460 bootstrapNetInit:142] pcl-kini02:344460:344460 [0][main] NCCL INFO Bootstrap: Using ibs20:192.168.7.4<0>
I0213 17:51:53.144505358.144505 344460 initOnceFunc:279] pcl-kini02:344460:344460 [0][main] NCCL INFO cudaDriverVersion 13000
I0213 09:51:53.145026 344460 CudaWrap.cc:267] cudaDriverVersion 13000
I0213 09:51:53.145053 344460 CudaWrap.cc:269] CUDART_VERSION 12090
I0213 17:51:53.145075870.145075 344460 showVersion:655] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL version 2.28.9x-git-bae29e6+cuda12.9
I0213 17:51:53.145931554.145931 344460 ncclCommInitRankDev:2292] pcl-kini02:344460:344460 [0][main] NCCL INFO CommInit comm 0x55e8750d7010 commHash 0x10cb34dcc234bc6c commDesc main_comm useCtran 1 usePatAvg 0: NCCL_CTRAN_ENABLE=true, GlobalHint ncclx.comm.useCtran=false NCCL_REDUCESCATTER_PAT_AVG_ENABLE=false, GlobalHint ncclx.comm.algo_reducescatter=(not set)
I0213 17:51:53.145956294.145956 344460 ncclxParseCommConfig:2018] pcl-kini02:344460:344460 [0][main] NCCL INFO Comm config Comm description set to main_comm
I0213 17:51:53.145971171.145971 344460 envConfigOverride:1890] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL_NCHANNELS_PER_NET_PEER -1 is too low, leaving it set at -2147483648
I0213 17:51:53.145981182.145981 344460 envConfigOverride:1904] pcl-kini02:344460:344460 [0][main] NCCL INFO Comm config nvlinkCentricSched reset to NCCL_NVLINK_UTIL_CENTRIC_SCHED_ENABLE=0
I0213 17:51:53.145995017.145995 344460 envConfigOverride:1942] pcl-kini02:344460:344460 [0][main] NCCL INFO Comm config collnetEnable reset to NCCL_COLLNET_ENABLE=0
I0213 17:51:53.146003102.146003 344460 envConfigOverride:1944] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0.
I0213 17:51:53.146364628.146364 344461 GlobalHints.cc:106] pcl-kini02:344461:344461 [1][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.crashOnAsyncError is registered.
I0213 17:51:53.146384654.146384 344462 GlobalHints.cc:106] pcl-kini02:344462:344462 [2][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.crashOnAsyncError is registered.
I0213 17:51:53.146397716.146397 344461 GlobalHints.cc:106] pcl-kini02:344461:344461 [1][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.crashOnTimeout is registered.
I0213 17:51:53.146404420.146404 344461 GlobalHints.cc:106] pcl-kini02:344461:344461 [1][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.timeoutMs is registered.
I0213 17:51:53.146410129.146410 344461 GlobalHints.cc:106] pcl-kini02:344461:344461 [1][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.comm.useCtran is registered.
I0213 17:51:53.146414786.146414 344461 GlobalHints.cc:106] pcl-kini02:344461:344461 [1][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.comm.algo_reducescatter is registered.
I0213 17:51:53.146418756.146418 344462 GlobalHints.cc:106] pcl-kini02:344462:344462 [2][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.crashOnTimeout is registered.
I0213 17:51:53.146425607.146425 344462 GlobalHints.cc:106] pcl-kini02:344462:344462 [2][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.colltrace.timeoutMs is registered.
I0213 17:51:53.146431294.146431 344462 GlobalHints.cc:106] pcl-kini02:344462:344462 [2][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.comm.useCtran is registered.
I0213 17:51:53.146436096.146436 344462 GlobalHints.cc:106] pcl-kini02:344462:344462 [2][main] NCCL INFO GLOBAL-HINTS: Hint key ncclx.comm.algo_reducescatter is registered.
I0213 17:51:53.146940556.146940 344461 openPluginLib:107] pcl-kini02:344461:344461 [1][main] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
I0213 17:51:53.146960795.146960 344462 openPluginLib:107] pcl-kini02:344462:344462 [2][main] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
I0213 17:51:53.146982146.146982 344461 initOnceFunc:279] pcl-kini02:344461:344461 [1][main] NCCL INFO cudaDriverVersion 13000
I0213 17:51:53.147003876.147003 344462 initOnceFunc:279] pcl-kini02:344462:344462 [2][main] NCCL INFO cudaDriverVersion 13000
I0213 09:51:53.147838 344461 CudaWrap.cc:267] cudaDriverVersion 13000
I0213 09:51:53.147861 344461 CudaWrap.cc:269] CUDART_VERSION 12090
I0213 17:51:53.147887972.147887 344461 setCpuStackSize:128] pcl-kini02:344461:344461 [1][main] NCCL INFO Stack size limit (unlimited) is unsafe; will use 8192KB for newly launched threads
I0213 09:51:53.147886 344462 CudaWrap.cc:267] cudaDriverVersion 13000
I0213 09:51:53.147914 344462 CudaWrap.cc:269] CUDART_VERSION 12090
I0213 17:51:53.147942246.147942 344462 setCpuStackSize:128] pcl-kini02:344462:344462 [2][main] NCCL INFO Stack size limit (unlimited) is unsafe; will use 8192KB for newly launched threads
I0213 17:51:53.148077668.148077 344461 findInterfaces:194] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL_SOCKET_IPADDR_PREFIX , current addrString 192.168.7.4
I0213 17:51:53.148084701.148084 344462 findInterfaces:194] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL_SOCKET_IPADDR_PREFIX , current addrString 192.168.7.4
I0213 17:51:53.148091660.148091 344461 bootstrapNetInit:142] pcl-kini02:344461:344461 [1][main] NCCL INFO Bootstrap: Using ibs20:192.168.7.4<0>
I0213 17:51:53.148099302.148099 344462 bootstrapNetInit:142] pcl-kini02:344462:344462 [2][main] NCCL INFO Bootstrap: Using ibs20:192.168.7.4<0>
I0213 17:51:53.148102130.148102 344461 showVersion:655] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL version 2.28.9x-git-bae29e6+cuda12.9
I0213 17:51:53.148110321.148110 344462 showVersion:655] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL version 2.28.9x-git-bae29e6+cuda12.9
I0213 17:51:53.149007817.149007 344461 ncclCommInitRankDev:2292] pcl-kini02:344461:344461 [1][main] NCCL INFO CommInit comm 0x560c44709010 commHash 0x10cb34dcc234bc6c commDesc main_comm useCtran 1 usePatAvg 0: NCCL_CTRAN_ENABLE=true, GlobalHint ncclx.comm.useCtran=false NCCL_REDUCESCATTER_PAT_AVG_ENABLE=false, GlobalHint ncclx.comm.algo_reducescatter=(not set)
I0213 17:51:53.149033806.149033 344461 ncclxParseCommConfig:2018] pcl-kini02:344461:344461 [1][main] NCCL INFO Comm config Comm description set to main_comm
I0213 17:51:53.149047274.149047 344461 envConfigOverride:1890] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL_NCHANNELS_PER_NET_PEER -1 is too low, leaving it set at -2147483648
I0213 17:51:53.149040226.149040 344462 ncclCommInitRankDev:2292] pcl-kini02:344462:344462 [2][main] NCCL INFO CommInit comm 0x563d9b6ef010 commHash 0x10cb34dcc234bc6c commDesc main_comm useCtran 1 usePatAvg 0: NCCL_CTRAN_ENABLE=true, GlobalHint ncclx.comm.useCtran=false NCCL_REDUCESCATTER_PAT_AVG_ENABLE=false, GlobalHint ncclx.comm.algo_reducescatter=(not set)
I0213 17:51:53.149056837.149056 344461 envConfigOverride:1904] pcl-kini02:344461:344461 [1][main] NCCL INFO Comm config nvlinkCentricSched reset to NCCL_NVLINK_UTIL_CENTRIC_SCHED_ENABLE=0
I0213 17:51:53.149063833.149063 344462 ncclxParseCommConfig:2018] pcl-kini02:344462:344462 [2][main] NCCL INFO Comm config Comm description set to main_comm
I0213 17:51:53.149070080.149070 344461 envConfigOverride:1942] pcl-kini02:344461:344461 [1][main] NCCL INFO Comm config collnetEnable reset to NCCL_COLLNET_ENABLE=0
I0213 17:51:53.149078041.149078 344461 envConfigOverride:1944] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0.
I0213 17:51:53.149078067.149078 344462 envConfigOverride:1890] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL_NCHANNELS_PER_NET_PEER -1 is too low, leaving it set at -2147483648
I0213 17:51:53.149087936.149087 344462 envConfigOverride:1904] pcl-kini02:344462:344462 [2][main] NCCL INFO Comm config nvlinkCentricSched reset to NCCL_NVLINK_UTIL_CENTRIC_SCHED_ENABLE=0
I0213 17:51:53.149101866.149101 344462 envConfigOverride:1942] pcl-kini02:344462:344462 [2][main] NCCL INFO Comm config collnetEnable reset to NCCL_COLLNET_ENABLE=0
I0213 17:51:53.149110165.149110 344462 envConfigOverride:1944] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0.
I0213 17:51:53.346788754.346788 344460 openPluginLib:107] pcl-kini02:344460:344460 [0][main] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
W0213 17:51:53.347043265.347043 344460 DataSink.h:26] pcl-kini02:344460:344460 [0][main] NCCL WARN Empty sink for dataset: nccl_profiler_slow_rank. No logging will be done.
W0213 17:51:53.347133284.347133 344460 DataSink.h:26] pcl-kini02:344460:344460 [0][main] NCCL WARN Empty sink for dataset: nccl_profiler_algo. No logging will be done.
W0213 17:51:53.347258976.347258 344460 ScubaFileUtils.cc:39] pcl-kini02:344460:344460 [0][main] NCCL WARN Failed to create scuba file /logs/dedicated_log_structured_json.perfpipe_nccl_slow_coll.Rank_0.20260213-095153-3364384216424942091.scribe: filesystem error: cannot create directories: Permission denied [/logs]
W0213 17:51:53.347325327.347325 344460 ScubaFileUtils.cc:39] pcl-kini02:344460:344460 [0][main] NCCL WARN Failed to create scuba file /logs/dedicated_log_structured_json.perfpipe_nccl_collective_stats.Rank_0.20260213-095153-3364384216424942091.scribe: filesystem error: cannot create directories: Permission denied [/logs]
W0213 17:51:53.347441142.347441 344460 DataTableWrapper.cc:89] pcl-kini02:344460:344460 [0][main] NCCL WARN Could not find table name: nccl_structured_logging
W0213 17:51:53.347441142.347441 344460 DataTableWrapper.cc:89] pcl-kini02:344460:344460 [0][main] NCCL WARN
W0213 17:51:53.347468951.347468 344460 ncclIbInit:852] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/transport/net_ib.cc:852 -> 3
I0213 17:51:53.347480030.347480 344460 ncclNetPluginInit:225] pcl-kini02:344460:344460 [0][main] NCCL INFO Failed to initialize NET plugin IB
I0213 17:51:53.347654710.347654 344460 findInterfaces:194] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL_SOCKET_IPADDR_PREFIX , current addrString 192.168.7.4
I0213 17:51:53.347693408.347693 344460 ncclNetSocketInit:76] pcl-kini02:344460:344460 [0][main] NCCL INFO NET/Socket : Using [0]ibs20:192.168.7.4<0>
I0213 17:51:53.347712864.347712 344460 ncclNetPluginInit:187] pcl-kini02:344460:344460 [0][main] NCCL INFO Initialized NET plugin Socket
I0213 17:51:53.347739274.347739 344460 ncclNetPluginAssignToComm:244] pcl-kini02:344460:344460 [0][main] NCCL INFO Assigned NET plugin Socket to comm
I0213 17:51:53.347748095.347748 344460 commAlloc:466] pcl-kini02:344460:344460 [0][main] NCCL INFO Using network Socket
I0213 17:51:53.348460961.348460 344461 openPluginLib:107] pcl-kini02:344461:344461 [1][main] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
W0213 17:51:53.348679491.348679 344461 DataSink.h:26] pcl-kini02:344461:344461 [1][main] NCCL WARN Empty sink for dataset: nccl_profiler_slow_rank. No logging will be done.
W0213 17:51:53.348763570.348763 344461 DataSink.h:26] pcl-kini02:344461:344461 [1][main] NCCL WARN Empty sink for dataset: nccl_profiler_algo. No logging will be done.
W0213 17:51:53.348895729.348895 344461 ScubaFileUtils.cc:39] pcl-kini02:344461:344461 [1][main] NCCL WARN Failed to create scuba file /logs/dedicated_log_structured_json.perfpipe_nccl_slow_coll.Rank_1.20260213-095153-9854319518851315868.scribe: filesystem error: cannot create directories: Permission denied [/logs]
W0213 17:51:53.348964417.348964 344461 ScubaFileUtils.cc:39] pcl-kini02:344461:344461 [1][main] NCCL WARN Failed to create scuba file /logs/dedicated_log_structured_json.perfpipe_nccl_collective_stats.Rank_1.20260213-095153-9854319518851315868.scribe: filesystem error: cannot create directories: Permission denied [/logs]
W0213 17:51:53.349052274.349052 344461 DataTableWrapper.cc:89] pcl-kini02:344461:344461 [1][main] NCCL WARN Could not find table name: nccl_structured_logging
W0213 17:51:53.349052274.349052 344461 DataTableWrapper.cc:89] pcl-kini02:344461:344461 [1][main] NCCL WARN
W0213 17:51:53.349077542.349077 344461 ncclIbInit:852] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/transport/net_ib.cc:852 -> 3
I0213 17:51:53.349088593.349088 344461 ncclNetPluginInit:225] pcl-kini02:344461:344461 [1][main] NCCL INFO Failed to initialize NET plugin IB
I0213 17:51:53.349218871.349218 344461 findInterfaces:194] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL_SOCKET_IPADDR_PREFIX , current addrString 192.168.7.4
I0213 17:51:53.349254725.349254 344461 ncclNetSocketInit:76] pcl-kini02:344461:344461 [1][main] NCCL INFO NET/Socket : Using [0]ibs20:192.168.7.4<0>
I0213 17:51:53.349263459.349263 344461 ncclNetPluginInit:187] pcl-kini02:344461:344461 [1][main] NCCL INFO Initialized NET plugin Socket
I0213 17:51:53.349287232.349287 344461 ncclNetPluginAssignToComm:244] pcl-kini02:344461:344461 [1][main] NCCL INFO Assigned NET plugin Socket to comm
I0213 17:51:53.349295872.349295 344461 commAlloc:466] pcl-kini02:344461:344461 [1][main] NCCL INFO Using network Socket
I0213 17:51:53.349565046.349565 344462 openPluginLib:107] pcl-kini02:344462:344462 [2][main] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
W0213 17:51:53.349796021.349796 344462 DataSink.h:26] pcl-kini02:344462:344462 [2][main] NCCL WARN Empty sink for dataset: nccl_profiler_slow_rank. No logging will be done.
W0213 17:51:53.349865621.349865 344462 DataSink.h:26] pcl-kini02:344462:344462 [2][main] NCCL WARN Empty sink for dataset: nccl_profiler_algo. No logging will be done.
W0213 17:51:53.350004735.350004 344462 ScubaFileUtils.cc:39] pcl-kini02:344462:344462 [2][main] NCCL WARN Failed to create scuba file /logs/dedicated_log_structured_json.perfpipe_nccl_slow_coll.Rank_2.20260213-095153-15366219575773999094.scribe: filesystem error: cannot create directories: Permission denied [/logs]
W0213 17:51:53.350070444.350070 344462 ScubaFileUtils.cc:39] pcl-kini02:344462:344462 [2][main] NCCL WARN Failed to create scuba file /logs/dedicated_log_structured_json.perfpipe_nccl_collective_stats.Rank_2.20260213-095153-15366219575773999094.scribe: filesystem error: cannot create directories: Permission denied [/logs]
W0213 17:51:53.350175082.350175 344462 DataTableWrapper.cc:89] pcl-kini02:344462:344462 [2][main] NCCL WARN Could not find table name: nccl_structured_logging
W0213 17:51:53.350175082.350175 344462 DataTableWrapper.cc:89] pcl-kini02:344462:344462 [2][main] NCCL WARN
W0213 17:51:53.350201242.350201 344462 ncclIbInit:852] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/transport/net_ib.cc:852 -> 3
I0213 17:51:53.350211835.350211 344462 ncclNetPluginInit:225] pcl-kini02:344462:344462 [2][main] NCCL INFO Failed to initialize NET plugin IB
I0213 17:51:53.350332508.350332 344462 findInterfaces:194] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL_SOCKET_IPADDR_PREFIX , current addrString 192.168.7.4
I0213 17:51:53.350365628.350365 344462 ncclNetSocketInit:76] pcl-kini02:344462:344462 [2][main] NCCL INFO NET/Socket : Using [0]ibs20:192.168.7.4<0>
I0213 17:51:53.350374368.350374 344462 ncclNetPluginInit:187] pcl-kini02:344462:344462 [2][main] NCCL INFO Initialized NET plugin Socket
I0213 17:51:53.350398273.350398 344462 ncclNetPluginAssignToComm:244] pcl-kini02:344462:344462 [2][main] NCCL INFO Assigned NET plugin Socket to comm
I0213 17:51:53.350406774.350406 344462 commAlloc:466] pcl-kini02:344462:344462 [2][main] NCCL INFO Using network Socket
I0213 17:51:54.016853632.016853 344460 ncclCommInitRankFunc:1726] pcl-kini02:344460:344460 [0][main] NCCL INFO ncclCommInitRankConfig comm 0x55e8750d7010 rank 0 nranks 3 cudaDev 0 nvmlDev 0 busId ac000 commId 0x10cb34dcc234bc6c - Init START
I0213 17:51:54.213565228.213565 344462 ncclCommInitRankFunc:1726] pcl-kini02:344462:344462 [2][main] NCCL INFO ncclCommInitRankConfig comm 0x563d9b6ef010 rank 2 nranks 3 cudaDev 2 nvmlDev 2 busId d4000 commId 0x10cb34dcc234bc6c - Init START
I0213 17:51:54.221233718.221233 344461 ncclCommInitRankFunc:1726] pcl-kini02:344461:344461 [1][main] NCCL INFO ncclCommInitRankConfig comm 0x560c44709010 rank 1 nranks 3 cudaDev 1 nvmlDev 1 busId c0000 commId 0x10cb34dcc234bc6c - Init START
I0213 17:51:54.221601516.221601 344461 bootstrapInit:833] pcl-kini02:344461:344461 [1][main] NCCL INFO Bootstrap timings total 0.000315 (create 0.000028, send 0.000073, recv 0.000086, ring 0.000039, delay 0.000001)
I0213 17:51:54.221610666.221610 344462 bootstrapInit:833] pcl-kini02:344462:344462 [2][main] NCCL INFO Bootstrap timings total 0.007986 (create 0.000029, send 0.000093, recv 0.000051, ring 0.000032, delay 0.000001)
I0213 17:51:54.221650224.221650 344461 transportExt.h:28] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL_CHANNEL_METADATA_LOCATION=2
I0213 17:51:54.221619658.221619 344460 bootstrapInit:833] pcl-kini02:344460:344460 [0][main] NCCL INFO Bootstrap timings total 0.204697 (create 0.000030, send 0.000089, recv 0.204432, ring 0.000021, delay 0.000001)
I0213 17:51:54.221671906.221671 344462 transportExt.h:28] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL_CHANNEL_METADATA_LOCATION=2
I0213 17:51:54.221690073.221690 344460 transportExt.h:28] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL_CHANNEL_METADATA_LOCATION=2
I0213 17:51:54.222949600.222949 344461 ncclTopoGetSystem:1427] pcl-kini02:344461:344461 [1][main] NCCL INFO NCCL_TOPO_FILE set by environment to /var/run/nvidia-topologyd/virtualTopology.xml
I0213 17:51:54.222957478.222957 344462 ncclTopoGetSystem:1427] pcl-kini02:344462:344462 [2][main] NCCL INFO NCCL_TOPO_FILE set by environment to /var/run/nvidia-topologyd/virtualTopology.xml
I0213 17:51:54.222959793.222959 344460 ncclTopoGetSystem:1427] pcl-kini02:344460:344460 [0][main] NCCL INFO NCCL_TOPO_FILE set by environment to /var/run/nvidia-topologyd/virtualTopology.xml
I0213 17:51:54.222972278.222972 344461 ncclTopoGetXmlFromFile:391] pcl-kini02:344461:344461 [1][main] NCCL INFO Could not open XML topology file /var/run/nvidia-topologyd/virtualTopology.xml : No such file or directory
I0213 17:51:54.222978340.222978 344462 ncclTopoGetXmlFromFile:391] pcl-kini02:344462:344462 [2][main] NCCL INFO Could not open XML topology file /var/run/nvidia-topologyd/virtualTopology.xml : No such file or directory
I0213 17:51:54.222980080.222980 344460 ncclTopoGetXmlFromFile:391] pcl-kini02:344460:344460 [0][main] NCCL INFO Could not open XML topology file /var/run/nvidia-topologyd/virtualTopology.xml : No such file or directory
I0213 17:51:54.236838903.236838 344460 ncclTopoGetCpuAffinity:1773] pcl-kini02:344460:344460 [0][main] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 75-119,195-239. (GPU affinity = 60-119,180-239 ; CPU affinity = 75-119,195-239).
I0213 17:51:54.236880922.236880 344460 ncclNvlsInit:205] pcl-kini02:344460:344460 [0][main] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
I0213 17:51:54.237278910.237278 344462 ncclTopoGetCpuAffinity:1773] pcl-kini02:344462:344462 [2][main] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 75-119,195-239. (GPU affinity = 60-119,180-239 ; CPU affinity = 75-119,195-239).
I0213 17:51:54.237319521.237319 344462 ncclNvlsInit:205] pcl-kini02:344462:344462 [2][main] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0)
I0213 17:51:54.237521717.237521 344461 ncclTopoGetCpuAffinity:1773] pcl-kini02:344461:344461 [1][main] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 75-119,195-239. (GPU affinity = 60-119,180-239 ; CPU affinity = 75-119,195-239).
I0213 17:51:54.237561961.237561 344461 ncclNvlsInit:205] pcl-kini02:344461:344461 [1][main] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0)
I0213 17:51:54.241984714.241984 344461 initTransportsRank:1286] pcl-kini02:344461:344461 [1][main] NCCL INFO comm 0x560c44709010 rank 1 nRanks 3 nNodes 1 localRanks 3 localRank 1 MNNVL 0
I0213 17:51:54.241983901.241983 344460 initTransportsRank:1286] pcl-kini02:344460:344460 [0][main] NCCL INFO comm 0x55e8750d7010 rank 0 nRanks 3 nNodes 1 localRanks 3 localRank 0 MNNVL 0
I0213 17:51:54.241984505.241984 344462 initTransportsRank:1286] pcl-kini02:344462:344462 [2][main] NCCL INFO comm 0x563d9b6ef010 rank 2 nRanks 3 nNodes 1 localRanks 3 localRank 2 MNNVL 0
I0213 17:51:54.242011606.242011 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 00/16 : 0 1 2
I0213 17:51:54.242020338.242020 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 01/16 : 0 1 2
I0213 17:51:54.242017768.242017 344461 initTransportsRank:1342] pcl-kini02:344461:344461 [1][main] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] -1/-1/-1->1->2 [3] -1/-1/-1->1->2 [4] 2/-1/-1->1->-1 [5] 0/-1/-1->1->-1 [6] -1/-1/-1->1->0 [7] 0/-1/-1->1->2 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] -1/-1/-1->1->2 [11] -1/-1/-1->1->2 [12] 2/-1/-1->1->-1 [13] 0/-1/-1->1->-1 [14] -1/-1/-1->1->0 [15] 0/-1/-1->1->2
I0213 17:51:54.242019398.242019 344462 initTransportsRank:1342] pcl-kini02:344462:344462 [2][main] NCCL INFO Trees [0] -1/-1/-1->2->1 [1] -1/-1/-1->2->1 [2] 1/-1/-1->2->0 [3] 1/-1/-1->2->0 [4] 0/-1/-1->2->1 [5] -1/-1/-1->2->0 [6] 0/-1/-1->2->-1 [7] 1/-1/-1->2->-1 [8] -1/-1/-1->2->1 [9] -1/-1/-1->2->1 [10] 1/-1/-1->2->0 [11] 1/-1/-1->2->0 [12] 0/-1/-1->2->1 [13] -1/-1/-1->2->0 [14] 0/-1/-1->2->-1 [15] 1/-1/-1->2->-1
I0213 17:51:54.242028275.242028 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 02/16 : 0 2 1
I0213 17:51:54.242031564.242031 344461 computeBuffSizes:837] pcl-kini02:344461:344461 [1][main] NCCL INFO P2P Chunksize set to 524288
I0213 17:51:54.242032582.242032 344462 computeBuffSizes:837] pcl-kini02:344462:344462 [2][main] NCCL INFO P2P Chunksize set to 524288
I0213 17:51:54.242035895.242035 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 03/16 : 0 2 1
I0213 17:51:54.242043567.242043 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 04/16 : 0 1 2
I0213 17:51:54.242051283.242051 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 05/16 : 0 1 2
I0213 17:51:54.242058776.242058 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 06/16 : 0 2 1
I0213 17:51:54.242066153.242066 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 07/16 : 0 2 1
I0213 17:51:54.242073724.242073 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 08/16 : 0 1 2
I0213 17:51:54.242081491.242081 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 09/16 : 0 1 2
I0213 17:51:54.242089279.242089 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 10/16 : 0 2 1
I0213 17:51:54.242096900.242096 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 11/16 : 0 2 1
I0213 17:51:54.242104548.242104 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 12/16 : 0 1 2
I0213 17:51:54.242112159.242112 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 13/16 : 0 1 2
I0213 17:51:54.242119855.242119 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 14/16 : 0 2 1
I0213 17:51:54.242127561.242127 344460 dumpLine:27] pcl-kini02:344460:344460 [0][main] NCCL INFO Channel 15/16 : 0 2 1
I0213 17:51:54.242128258.242128 344461 openPluginLib:107] pcl-kini02:344461:344461 [1][main] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
I0213 17:51:54.242135809.242135 344462 openPluginLib:107] pcl-kini02:344462:344462 [2][main] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
I0213 17:51:54.242139070.242139 344461 ncclTransportCheckP2pType:128] pcl-kini02:344461:344461 [1][main] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
I0213 17:51:54.242143642.242143 344460 initTransportsRank:1342] pcl-kini02:344460:344460 [0][main] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 2/-1/-1->0->-1 [3] 2/-1/-1->0->-1 [4] -1/-1/-1->0->2 [5] 2/-1/-1->0->1 [6] 1/-1/-1->0->2 [7] -1/-1/-1->0->1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 2/-1/-1->0->-1 [11] 2/-1/-1->0->-1 [12] -1/-1/-1->0->2 [13] 2/-1/-1->0->1 [14] 1/-1/-1->0->2 [15] -1/-1/-1->0->1
I0213 17:51:54.242146764.242146 344462 ncclTransportCheckP2pType:128] pcl-kini02:344462:344462 [2][main] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
I0213 17:51:54.242156619.242156 344460 computeBuffSizes:837] pcl-kini02:344460:344460 [0][main] NCCL INFO P2P Chunksize set to 524288
I0213 17:51:54.242253132.242253 344460 openPluginLib:107] pcl-kini02:344460:344460 [0][main] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
I0213 17:51:54.242264188.242264 344460 ncclTransportCheckP2pType:128] pcl-kini02:344460:344460 [0][main] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
I0213 17:51:54.242263483.242263 344493 ncclProxyService:1609] pcl-kini02:344462:344493 [0][main] NCCL INFO [Proxy Service] Device 2 CPU core 195
I0213 17:51:54.242254972.242254 344492 ncclProxyService:1609] pcl-kini02:344461:344492 [0][main] NCCL INFO [Proxy Service] Device 1 CPU core 205
I0213 17:51:54.242293126.242293 344495 ncclProxyServiceUDS:1803] pcl-kini02:344461:344495 [0][main] NCCL INFO [Proxy Service UDS] Device 1 CPU core 206
I0213 17:51:54.242298824.242298 344494 ncclProxyServiceUDS:1803] pcl-kini02:344462:344494 [0][main] NCCL INFO [Proxy Service UDS] Device 2 CPU core 197
I0213 17:51:54.242361294.242361 344496 ncclProxyService:1609] pcl-kini02:344460:344496 [0][main] NCCL INFO [Proxy Service] Device 0 CPU core 198
I0213 17:51:54.242396918.242396 344497 ncclProxyServiceUDS:1803] pcl-kini02:344460:344497 [0][main] NCCL INFO [Proxy Service UDS] Device 0 CPU core 199
I0213 17:51:54.242561563.242561 344461 EventMgrHelperTypes.cc:61] pcl-kini02:344461:344461 [1][main] NCCL INFO Initialized NCCL_FILTER_MEM_LOGGING_BY_RANKS, globalRank: 1, isAllowed: false
I0213 17:51:54.242572601.242572 344461 EventMgrHelperTypes.cc:17] pcl-kini02:344461:344461 [1][main] NCCL INFO Empty rank list, skip NCCL_FILTER_MEM_REG_LOGGING_BY_RANKS initialization. isAllowed: true
I0213 17:51:54.242579454.242579 344462 EventMgrHelperTypes.cc:61] pcl-kini02:344462:344462 [2][main] NCCL INFO Initialized NCCL_FILTER_MEM_LOGGING_BY_RANKS, globalRank: 2, isAllowed: false
I0213 17:51:54.242590563.242590 344462 EventMgrHelperTypes.cc:17] pcl-kini02:344462:344462 [2][main] NCCL INFO Empty rank list, skip NCCL_FILTER_MEM_REG_LOGGING_BY_RANKS initialization. isAllowed: true
I0213 17:51:54.242637711.242637 344460 EventMgrHelperTypes.cc:61] pcl-kini02:344460:344460 [0][main] NCCL INFO Initialized NCCL_FILTER_MEM_LOGGING_BY_RANKS, globalRank: 0, isAllowed: true
I0213 17:51:54.242649670.242649 344460 EventMgrHelperTypes.cc:17] pcl-kini02:344460:344460 [0][main] NCCL INFO Empty rank list, skip NCCL_FILTER_MEM_REG_LOGGING_BY_RANKS initialization. isAllowed: true
W0213 17:51:54.242762122.242762 344460 DataTableWrapper.cc:89] pcl-kini02:344460:344460 [0][main] NCCL WARN Could not find table name: nccl_memory_logging
W0213 17:51:54.242762122.242762 344460 DataTableWrapper.cc:89] pcl-kini02:344460:344460 [0][main] NCCL WARN
I0213 17:51:54.251197478.251197 344461 openPluginLib:107] pcl-kini02:344461:344461 [1][main] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
I0213 17:51:54.251234033.251234 344461 ncclTopoTuneModel:566] pcl-kini02:344461:344461 [1][main] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 512 | 512
I0213 17:51:54.251244531.251244 344461 initTransportsRank:1481] pcl-kini02:344461:344461 [1][main] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
I0213 17:51:54.251557052.251557 344462 openPluginLib:107] pcl-kini02:344462:344462 [2][main] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
I0213 17:51:54.251594531.251594 344462 ncclTopoTuneModel:566] pcl-kini02:344462:344462 [2][main] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 512 | 512
I0213 17:51:54.251605293.251605 344462 initTransportsRank:1481] pcl-kini02:344462:344462 [2][main] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
I0213 17:51:54.253255680.253255 344460 openPluginLib:107] pcl-kini02:344460:344460 [0][main] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
I0213 17:51:54.253348994.253348 344460 ncclTopoTuneModel:566] pcl-kini02:344460:344460 [0][main] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 512 | 512
I0213 17:51:54.253364886.253364 344460 initTransportsRank:1481] pcl-kini02:344460:344460 [0][main] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
I0213 17:51:54.253790144.253790 344460 devCommSetup:592] pcl-kini02:344460:344460 [0][main] NCCL INFO CC Off, workFifoBytes 1048576
I0213 17:51:54.254181299.254181 344461 CollTraceWrapper.cc:307] pcl-kini02:344461:344461 [1][main] NCCL INFO CollTrace init - NCCL_COLLTRACE: [algostat: false, verbose: false, trace: false], NCCL_COLLTRACE_USE_NEW_COLLTRACE: true
I0213 17:51:54.254196470.254196 344460 CollTraceWrapper.cc:307] pcl-kini02:344460:344460 [0][main] NCCL INFO CollTrace init - NCCL_COLLTRACE: [algostat: false, verbose: false, trace: false], NCCL_COLLTRACE_USE_NEW_COLLTRACE: true
I0213 17:51:54.254194872.254194 344462 CollTraceWrapper.cc:307] pcl-kini02:344462:344462 [2][main] NCCL INFO CollTrace init - NCCL_COLLTRACE: [algostat: false, verbose: false, trace: false], NCCL_COLLTRACE_USE_NEW_COLLTRACE: true
I0213 09:51:54.254229 344461 CommStateX.cc:297] CommStateX: set rankTopology with system
I0213 09:51:54.254235 344460 CommStateX.cc:297] CommStateX: set rankTopology with system
I0213 09:51:54.254243 344462 CommStateX.cc:297] CommStateX: set rankTopology with system
I0213 17:51:54.254264353.254264 344461 createCommStateXFromNcclComm:68] pcl-kini02:344461:344461 [1][main] NCCL INFO CommStateX: initialize from ncclComm with system
I0213 17:51:54.254265714.254265 344460 createCommStateXFromNcclComm:68] pcl-kini02:344460:344460 [0][main] NCCL INFO CommStateX: initialize from ncclComm with system
I0213 17:51:54.254277477.254277 344462 createCommStateXFromNcclComm:68] pcl-kini02:344462:344462 [2][main] NCCL INFO CommStateX: initialize from ncclComm with system
E0213 09:51:54.254287 344461 Topology.cc:111] Failed to load hostname (DEVICE_NAME) from /etc/fbwhoami
E0213 09:51:54.254289 344460 Topology.cc:111] Failed to load hostname (DEVICE_NAME) from /etc/fbwhoami
E0213 09:51:54.254296 344461 CommStateX.cc:106] Check failed: false - Failed to load topology from /etc/fbwhoami
E0213 09:51:54.254297 344460 CommStateX.cc:106] Check failed: false - Failed to load topology from /etc/fbwhoami
E0213 09:51:54.254299 344462 Topology.cc:111] Failed to load hostname (DEVICE_NAME) from /etc/fbwhoami
E0213 09:51:54.254307 344462 CommStateX.cc:106] Check failed: false - Failed to load topology from /etc/fbwhoami
E0213 17:51:54.254375051.254375 344461 FactoryCommStateX.cc:156] pcl-kini02:344461:344461 [1][main] NCCL ERROR Failed to initialize CtranComm statex from ncclComm: Exception: Check failed: false - Failed to load topology from /etc/fbwhoami, rank: 1, commHash: 10cb34dcc234bc6c, desc: main_comm, result: commInternalError (commInternalError)
E0213 17:51:54.254375441.254375 344462 FactoryCommStateX.cc:156] pcl-kini02:344462:344462 [2][main] NCCL ERROR Failed to initialize CtranComm statex from ncclComm: Exception: Check failed: false - Failed to load topology from /etc/fbwhoami, rank: 2, commHash: 10cb34dcc234bc6c, desc: main_comm, result: commInternalError (commInternalError)
E0213 17:51:54.254374856.254374 344460 FactoryCommStateX.cc:156] pcl-kini02:344460:344460 [0][main] NCCL ERROR Failed to initialize CtranComm statex from ncclComm: Exception: Check failed: false - Failed to load topology from /etc/fbwhoami, rank: 0, commHash: 10cb34dcc234bc6c, desc: main_comm, result: commInternalError (commInternalError)
W0213 17:51:54.254478208.254478 344460 ncclCommInitRankFunc:1782] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:1782 -> 3
I0213 17:51:54.254489968.254489 344460 ncclAsyncJobMain:84] pcl-kini02:344460:344460 [0][main] NCCL INFO /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:84 -> 3 [Async thread]
W0213 17:51:54.254495617.254495 344462 ncclCommInitRankFunc:1782] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:1782 -> 3
W0213 17:51:54.254495055.254495 344461 ncclCommInitRankFunc:1782] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:1782 -> 3
I0213 17:51:54.254506377.254506 344462 ncclAsyncJobMain:84] pcl-kini02:344462:344462 [2][main] NCCL INFO /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:84 -> 3 [Async thread]
I0213 17:51:54.254508318.254508 344461 ncclAsyncJobMain:84] pcl-kini02:344461:344461 [1][main] NCCL INFO /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:84 -> 3 [Async thread]
W0213 17:51:54.254529325.254529 344460 groupLaunch:561] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:561 -> 3
W0213 17:51:54.254542275.254542 344462 groupLaunch:561] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:561 -> 3
W0213 17:51:54.254543024.254543 344461 groupLaunch:561] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:561 -> 3
W0213 17:51:54.254567599.254567 344460 ncclGroupEndInternal:769] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:769 -> 3
W0213 17:51:54.254576699.254576 344461 ncclGroupEndInternal:769] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:769 -> 3
W0213 17:51:54.254581314.254581 344462 ncclGroupEndInternal:769] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/group.cc:769 -> 3
W0213 17:51:54.254608798.254608 344460 ncclCommInitRankConfig:2480] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:2480 -> 3
W0213 17:51:54.254610772.254610 344461 ncclCommInitRankConfig:2480] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:2480 -> 3
W0213 17:51:54.254616203.254616 344462 ncclCommInitRankConfig:2480] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:2480 -> 3
W0213 17:51:54.254951836.254951 344460 PtrCheck:30] pcl-kini02:344460:344460 [0][main] NCCL WARN CommUserRank : comm argument is NULL
W0213 17:51:54.254992620.254992 344461 PtrCheck:30] pcl-kini02:344461:344461 [1][main] NCCL WARN CommUserRank : comm argument is NULL
W0213 17:51:54.255003912.255003 344460 CommCheck:37] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/misc/argcheck.cc:37 -> 4
W0213 17:51:54.255038063.255038 344461 CommCheck:37] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/misc/argcheck.cc:37 -> 4
W0213 17:51:54.255042342.255042 344460 ncclCommUserRank:3172] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:3172 -> 4
W0213 17:51:54.255038524.255038 344462 PtrCheck:30] pcl-kini02:344462:344462 [2][main] NCCL WARN CommUserRank : comm argument is NULL
E0213 09:51:54.254935 344460 TorchCommNCCLX.cpp:77] [TC][name=main_comm] TorchCommNCCLX main_comm was not finalized before destruction
W0213 17:51:54.255074440.255074 344460 PtrCheck:30] pcl-kini02:344460:344460 [0][main] NCCL WARN CommUserRank : comm argument is NULL
W0213 17:51:54.255073820.255073 344461 ncclCommUserRank:3172] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:3172 -> 4
W0213 17:51:54.255084276.255084 344462 CommCheck:37] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/misc/argcheck.cc:37 -> 4
E0213 09:51:54.254976 344461 TorchCommNCCLX.cpp:77] [TC][name=main_comm] TorchCommNCCLX main_comm was not finalized before destruction
W0213 17:51:54.255105521.255105 344461 PtrCheck:30] pcl-kini02:344461:344461 [1][main] NCCL WARN CommUserRank : comm argument is NULL
W0213 17:51:54.255111211.255111 344460 CommCheck:37] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/misc/argcheck.cc:37 -> 4
W0213 17:51:54.255120697.255120 344462 ncclCommUserRank:3172] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:3172 -> 4
E0213 09:51:54.255020 344462 TorchCommNCCLX.cpp:77] [TC][name=main_comm] TorchCommNCCLX main_comm was not finalized before destruction
W0213 17:51:54.255139275.255139 344461 CommCheck:37] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/misc/argcheck.cc:37 -> 4
W0213 17:51:54.255147203.255147 344460 ncclCommUserRank:3172] pcl-kini02:344460:344460 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:3172 -> 4
W0213 17:51:54.255149841.255149 344462 PtrCheck:30] pcl-kini02:344462:344462 [2][main] NCCL WARN CommUserRank : comm argument is NULL
I0213 09:51:54.255067 344460 TorchCommNCCLXUtils.cpp:487] [TC][name=main_comm] Detaching memory hook comm=0x55e8735a6840
W0213 17:51:54.255173404.255173 344461 ncclCommUserRank:3172] pcl-kini02:344461:344461 [1][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:3172 -> 4
W0213 17:51:54.255184527.255184 344462 CommCheck:37] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/misc/argcheck.cc:37 -> 4
I0213 09:51:54.255100 344461 TorchCommNCCLXUtils.cpp:487] [TC][name=main_comm] Detaching memory hook comm=0x560c42bd67d0
W0213 17:51:54.255218836.255218 344462 ncclCommUserRank:3172] pcl-kini02:344462:344462 [2][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/init.cc:3172 -> 4
I0213 09:51:54.255143 344462 TorchCommNCCLXUtils.cpp:487] [TC][name=main_comm] Detaching memory hook comm=0x563d99bbc8b0
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 63, in <module>
    main()
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 9, in main
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to initialize NCCL communicator: internal error - please report this issue to the NCCL developers
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 63, in <module>
    main()
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 9, in main
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to initialize NCCL communicator: internal error - please report this issue to the NCCL developers
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 63, in <module>
    main()
  File "/data/nfs_home/bbrock/src/torchcomms/./rma_test.py", line 9, in main
    comm = torchcomms.new_comm("ncclx", device, name="main_comm")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to initialize NCCL communicator: internal error - please report this issue to the NCCL developers
I0213 17:51:54.410960052.410960 344461 ncclEnvPluginUnload:74] pcl-kini02:344461:344461 [1][main] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
I0213 17:51:54.410967346.410967 344460 ncclEnvPluginUnload:74] pcl-kini02:344460:344460 [0][main] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
I0213 17:51:54.418928184.418928 344462 ncclEnvPluginUnload:74] pcl-kini02:344462:344462 [2][main] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
W0213 09:51:54.712000 344359 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 344460 closing signal SIGTERM
W0213 09:51:54.713000 344359 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 344462 closing signal SIGTERM
E0213 09:51:54.876000 344359 site-packages/torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 1 (pid: 344461) of binary: /data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/bin/python3.12
Traceback (most recent call last):
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/bin/torchrun", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/run.py", line 990, in main
    run(args)
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/run.py", line 981, in run
    elastic_launch(
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/nfs_home/bbrock/.miniforge3/envs/torchcomms-pip-2025-02-13/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
./rma_test.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2026-02-13_09:51:54
  host      : pcl-kini02.sc.intel.com
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 344460)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2026-02-13_09:51:54
  host      : pcl-kini02.sc.intel.com
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 344462)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-02-13_09:51:54
  host      : pcl-kini02.sc.intel.com
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 344461)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

[BenBrock]

Hi @tianfengfrank, just wanted to check back to see if you have any comments on this second /etc/fbwhoami error?

[dolpm]

the mentioned pr should fix this.

[BenBrock]

@dolpm curious if there's been any progress on your PR or if there's a branch I should build for a workaround in the meantime?

[dolpm]

@dolpm curious if there's been any progress on your PR or if there's a branch I should build for a workaround in the meantime?

sorry, forgot about this. let me update to something you can use in the next few hours and try to get it merged in the next few days

[dolpm]

@dolpm curious if there's been any progress on your PR or if there's a branch I should build for a workaround in the meantime?

you can try https://github.com/dolpm/torchcomms/tree/export-D97240869