Using both NCCL and NCCLX in the same script

Hi,

I am trying to compare NCCL v/s NCCLX performance. However, I am unable to use both in the same script. I need to run each of them serially.

Probably this is due to the same library names for NCCLX as NCCL leading to conflicts?

This is leading to issues in other instances, for e.g. when I want to use NCCLX collectives in an existing infra that uses torch.distributed with NCCL backend for other distributed operations.

[d4l3k]

Can you share logs/stack traces of the error when you use them both? It should be fine -- we build them together in our CI

[d4l3k]

Repro:

tristanr@devvm5553 /d/u/t/f/f/c/torchcomms (default) [SIGABRT]> MASTER_ADDR=localhost MASTER_PORT=0 RANK=0 WORLD_SIZE=1 python -c "import torch; import torch.distributed as dist; import torchcomms; co
mm = torchcomms.new_comm('ncclx', torch.device('cuda'), store=None, name='1234'); t = torch.zeros(10, device=comm.get_device()); comm.all_reduce(t, torchcomms.ReduceOp.SUM, async_op=False); comm2=torc
hcomms.new_comm('nccl', torch.device('cuda'), name='foo')"
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0211 13:18:23.323849 1718364 TorchCommFactory.cpp:164] [TC] Backend ncclx is registered
I0211 13:18:23.323879 1718364 TorchCommFactory.cpp:164] [TC] Backend dummy is registered
I0211 13:18:23.323899 1718364 TorchCommNCCLXBootstrap.cpp:60] [TC] TORCHCOMM_NCCLX_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
I0211 13:18:23.424827 1718364 TorchCommNCCLXBootstrap.cpp:83] [TC] Found 4 CUDA devices
I0211 13:18:23.424863 1718364 TorchCommNCCLXBootstrap.cpp:86] [TC] User did not provide device ID; using device cuda:0
I0211 13:18:23.655603 1718364 CudaWrap.cc:267] cudaDriverVersion 12040
I0211 13:18:23.655668 1718364 CudaWrap.cc:269] CUDART_VERSION 12080
I0211 21:18:23.655698629.655698 1718364 showVersion:653] devvm5553:1718364:1718364 [0][main] NCCL INFO NCCL version 2.28.9x-git-69dc6db+cuda12.8
W0211 21:18:23.963663011.963663 1718364 DataSink.h:26] devvm5553:1718364:1718364 [0][main] NCCL WARN Empty sink for dataset: nccl_profiler_slow_rank. No logging will be done.
W0211 21:18:23.963843303.963843 1718364 DataSink.h:26] devvm5553:1718364:1718364 [0][main] NCCL WARN Empty sink for dataset: nccl_profiler_algo. No logging will be done.
W0211 21:18:23.964953808.964953 1718364 DataTableWrapper.cc:89] devvm5553:1718364:1718364 [0][main] NCCL WARN Could not find table name: nccl_structured_logging
W0211 21:18:23.964953808.964953 1718364 DataTableWrapper.cc:89] devvm5553:1718364:1718364 [0][main] NCCL WARN 
W0211 21:18:23.965002641.965002 1718364 ncclIbInit:852] devvm5553:1718364:1718364 [0][main] NCCL WARN /__w/torchcomms/torchcomms/meta-pytorch/torchcomms/comms/ncclx/v2_28/src/transport/net_ib.cc:852 -> 3
W0211 21:18:24.223664747.223664 1718364 DataTableWrapper.cc:89] devvm5553:1718364:1718364 [0][main] NCCL WARN Could not find table name: nccl_memory_logging
W0211 21:18:24.223664747.223664 1718364 DataTableWrapper.cc:89] devvm5553:1718364:1718364 [0][main] NCCL WARN 
I0211 13:18:24.247900 1718364 CommStateX.cc:297] CommStateX: set rankTopology with system
NCCL version 2.29.3+cuda12.9
*** Aborted at 1770844704 (unix time) try "date -d @1770844704" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 1718364 (TID 0x7fe0defda740) from PID 0; stack trace: ***
    @     0x7fe0dec3fc30 (unknown)
    @     0x7fe0a24c64d8 std::_Rb_tree_insert_and_rebalance()
    @     0x7fdf164c79c7 std::_Rb_tree<>::_M_emplace_hint_unique<>()
    @     0x7fdf164b79e6 torch::comms::TorchCommNCCLX::register_address()
    @     0x7fdf164cd5a7 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7fe09a8a1ea4 torch::comms::TorchCommNCCL::init()
    @     0x7fe09ba7663b torch::comms::TorchCommFactory::create_backend()
    @     0x7fe09ba70337 torch::comms::new_comm()
    @     0x7fe09b196e1d _ZNO8pybind116detail15argument_loaderIJRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEEE9call_implISt10shared_ptrIN5torch5comms9TorchCommEERZL20pybind11_init__commsRNS_7module_EEUlS9_SD_S9_SF_SL_SF_ST_S13_E_JLm0ELm1ELm2ELm3ELm4ELm5ELm6ELm7EENS0_9void_typeEEET_OT0_St16integer_sequenceImJXspT1_EEEOT2_.isra.0
    @     0x7fe09b1976eb _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE_clES1V_.constprop.0
    @     0x7fe09b1ba71d pybind11::cpp_function::dispatcher()
    @           0x597a4d cfunction_call
    @           0x55af84 _PyObject_MakeTpCall.localalias
    @           0x57adff _PyEval_EvalFrameDefault
    @           0x68e4d8 PyEval_EvalCode
    @           0x6281f8 run_mod
    @           0x6c28ca _PyRun_StringFlagsWithName.constprop.0
    @           0x6c246c _PyRun_SimpleStringFlagsWithName
    @           0x6c18a7 Py_RunMain.localalias
    @           0x681d79 Py_BytesMain
    @     0x7fe0dec2a610 __libc_start_call_main
    @     0x7fe0dec2a6c0 __libc_start_main_alias_2
    @           0x680fbe _start
    @                0x0 (unknown)
fish: Job 1, 'MASTER_ADDR=localhost MASTER_PO…' terminated by signal SIGSEGV (Address boundary error)

[d4l3k]

@rghadia do you mind sharing your use case? We haven't tested combining both nccl and ncclx in the same job

Hi @d4l3k

I was actually trying to use NCCL via torch.distributed, and NCCLX via torchcomms.

In general, will it be possible to replace selective torch.distributed comm-calls with torchcomm comm-calls?

For example, in my case, I am using the torchtitan framework, which uses torch.distributed for FSDP. However, I wish to use torchcomms with NCCLX backend for all-to-all, is it possible to do that?

Below is the test script I am using to test NCCL v/s NCCLX performance. I am unable to run the script with --backend=both

test_ncclx_a2a_ext.py

Thanks!

[d4l3k]

@rghadia can you share logs/stack traces?

Running torchcomms all-to-all for backend: nccl

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:09.256911 495420 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:09.256937 495420 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:09.256951 495420 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
[rank0]:[E212 11:55:09.192777418 TCPStore.cpp:277] [c10d] The server socket on 29500 has failed to bind. TORCHELASTIC_USE_AGENT_STORE is enabled so ignoring the error.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:09.437045 495422 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:09.437078 495422 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:09.437090 495422 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:09.601161 495426 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:09.601192 495426 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:09.601212 495426 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:10.105846 495427 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:10.105877 495427 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:10.105891 495427 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:10.274329 495421 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:10.274358 495421 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:10.274372 495421 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:10.685401 495425 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:10.685427 495425 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:10.685441 495425 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:11.009070 495423 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:11.009284 495423 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:11.009318 495423 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0212 11:55:11.178025 495424 TorchCommFactory.cpp:157] [TC] Backend nccl is registered
I0212 11:55:11.178047 495424 TorchCommFactory.cpp:157] [TC] Backend dummy is registered
I0212 11:55:11.178064 495424 TorchCommNCCLBootstrap.cpp:42] [TC] TORCHCOMM_NCCL_BOOTSTRAP_UNIQUEID_EXCHANGE_METHOD not set, defaulting to auto
/path/to/anaconda3/envs/torchcomms_env/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)
[rank0]:[W212 11:56:23.020264814 ProcessGroupNCCL.cpp:5178] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
/path/to/anaconda3/envs/torchcomms_env/lib/python3.10/site-packages/torch/profiler/profiler.py:217: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.
  _warn_once(

[nccl] avg_iter=1.032982s max_iter=1.180502s per_rank_bw=0.06 GB/s

/path/to/anaconda3/envs/torchcomms_env/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)


Running torchcomms all-to-all for backend: ncclx


I0212 11:56:54.934869 495420 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:54.934930 495420 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:54.942616 495421 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:54.942659 495421 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:54.944573 495427 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:54.944629 495427 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:54.953667 495422 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:54.953710 495422 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:55.056891 495423 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:55.056945 495423 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:55.057851 495424 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:55.061254 495424 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:55.058748 495426 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:55.064073 495426 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:55.200203 495425 CudaWrap.cc:267] cudaDriverVersion 12070
I0212 11:56:55.200282 495425 CudaWrap.cc:269] CUDART_VERSION 12060
I0212 11:56:59.708869 495426 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.713023 495427 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.720326 495421 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.722783 495420 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.722778 495424 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.724226 495425 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.724224 495423 CommStateX.cc:296] CommStateX: set rankTopology with system
I0212 11:56:59.724241 495422 CommStateX.cc:296] CommStateX: set rankTopology with system
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495422 (TID 0x7f735e3ae740) from PID 17; stack trace: ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495426 (TID 0x7f98cbc7e740) from PID 17; stack trace: ***
    @     0x7f98cbcc3520 (unknown)
    @     0x7f735e3f3520 (unknown)
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495425 (TID 0x7f0071b1e740) from PID 17; stack trace: ***
    @     0x7f0071b63520 (unknown)
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495421 (TID 0x7f4da75d1740) from PID 17; stack trace: ***
    @     0x7f4da7616520 (unknown)
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495420 (TID 0x7f715bb8c740) from PID 17; stack trace: ***
    @     0x7f715bbd1520 (unknown)
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495423 (TID 0x7f7cb1715740) from PID 17; stack trace: ***
    @     0x7f7cb175a520 (unknown)
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495424 (TID 0x7f1bc394b740) from PID 17; stack trace: ***
    @     0x7f1bc3990520 (unknown)
*** Aborted at 1770926234 (unix time) try "date -d @1770926234" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x11) received by PID 495427 (TID 0x7f76ce18d740) from PID 17; stack trace: ***
    @     0x7f76ce1d2520 (unknown)
    @     0x7f70c46484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f70c465bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f1b2c4484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f1b2c45bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f98752c74b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f98752dac45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f72c6e484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f72c6e5bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7effda6484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7effda65bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f7636c484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f7636c5bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f4d100484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f7c1a2484b0 torch::comms::TorchCommNCCL::register_address()
    @     0x7f4d1005bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f7c1a25bc45 torch::comms::CachingAllocatorHookImpl::registerComm()
    @     0x7f73785b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f70085b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f95785b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7efd1c5b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f186e5b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f7355de7ba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f76ccde7ba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f98ca7d0ba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f76ccde4b5b torch::comms::new_comm()
    @     0x7f7355de4b5b torch::comms::new_comm()
    @     0x7f6e085b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f4a545b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f98ca7cdb5b torch::comms::new_comm()
    @     0x7f795e5b7483 torch::comms::TorchCommNCCLX::init()
    @     0x7f007079bba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f7688c249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f98866249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f4d9efe7ba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f7cb0367ba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f7688c2f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f715a7e7ba6 torch::comms::TorchCommFactory::create_backend()
    @     0x7f0070798b5b torch::comms::new_comm()
    @     0x7f988662f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f4d9efe4b5b torch::comms::new_comm()
    @     0x7f7688c50d1f pybind11::cpp_function::dispatcher()
    @     0x7f7cb0364b5b torch::comms::new_comm()
    @     0x7f002c6249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f9886650d1f pybind11::cpp_function::dispatcher()
    @     0x7f1bc25c3ba6 torch::comms::TorchCommFactory::create_backend()
    @           0x4fe087 cfunction_call
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @     0x7f715a7e4b5b torch::comms::new_comm()
    @     0x7f4d620249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f7c6c2249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f002c62f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f71166249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f1bc25c0b5b torch::comms::new_comm()
    @     0x7f4d6202f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f7c6c22f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f7318e249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @     0x7f711662f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f1b7e4249c0 _ZZL20pybind11_init__commsRN8pybind117module_EENKUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceES9_St8optionalIbESE_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESF_SE_INSA_13intrusive_ptrIN4c10d5StoreENSA_6detail34intrusive_target_default_null_typeISO_EEEEESE_ISt13unordered_mapIS7_S7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIS8_S7_EEEEE4_clES9_SD_S9_SF_SL_SF_ST_S13_.constprop.0.isra.0
    @     0x7f4d62050d1f pybind11::cpp_function::dispatcher()
    @     0x7f7c6c250d1f pybind11::cpp_function::dispatcher()
    @           0x4fe087 cfunction_call
    @           0x4fe087 cfunction_call
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @     0x7f7318e2f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @     0x7f1b7e42f7f9 _ZZN8pybind1112cpp_function10initializeIZL20pybind11_init__commsRNS_7module_EEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN3c106DeviceESB_St8optionalIbESG_INSt6chrono8durationIlSt5ratioILl1ELl1000EEEEESH_SG_INSC_13intrusive_ptrIN4c10d5StoreENSC_6detail34intrusive_target_default_null_typeISQ_EEEEESG_ISt13unordered_mapIS9_S9_St4hashIS9_ESt8equal_toIS9_ESaISt4pairISA_S9_EEEEE4_St10shared_ptrIN5torch5comms9TorchCommEEJSB_SF_SB_SH_SN_SH_SV_S15_EJNS_4nameENS_5scopeENS_7siblingEA1203_cNS_3argES1G_S1G_NS_5arg_vES1H_S1H_S1H_S1H_EEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clES1V_.constprop.0
    @     0x7f7116650d1f pybind11::cpp_function::dispatcher()
    @           0x4fe087 cfunction_call
    @     0x7f7318e50d1f pybind11::cpp_function::dispatcher()
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @     0x7f002c650d1f pybind11::cpp_function::dispatcher()
    @           0x4fe087 cfunction_call
    @     0x7f1b7e450d1f pybind11::cpp_function::dispatcher()
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @           0x4fe087 cfunction_call
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe087 cfunction_call
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4fe087 cfunction_call
    @           0x4f77db _PyObject_MakeTpCall.localalias
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4f3d9b _PyEval_EvalFrameDefault
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x5960d7 PyEval_EvalCode
    @           0x5c72d7 run_eval_code_obj
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x5960d7 PyEval_EvalCode
    @           0x5c2420 run_mod
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x4598b9 pyrun_file.cold
    @           0x5c72d7 run_eval_code_obj
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x5c2420 run_mod
    @           0x5960d7 PyEval_EvalCode
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @           0x4598b9 pyrun_file.cold
    @           0x5c72d7 run_eval_code_obj
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x5c2420 run_mod
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @           0x4598b9 pyrun_file.cold
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x588fa9 Py_BytesMain
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x588fa9 Py_BytesMain
    @     0x7f1bc3977d90 (unknown)
    @           0x588fa9 Py_BytesMain
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x5960d7 PyEval_EvalCode
    @     0x7f4da75fdd90 (unknown)
    @           0x5c72d7 run_eval_code_obj
    @     0x7f1bc3977e40 __libc_start_main
    @           0x5c2420 run_mod
    @           0x4598b9 pyrun_file.cold
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @     0x7f7cb1741d90 (unknown)
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @     0x7f4da75fde40 __libc_start_main
    @           0x5960d7 PyEval_EvalCode
    @           0x5c72d7 run_eval_code_obj
    @     0x7f7cb1741e40 __libc_start_main
    @           0x5c2420 run_mod
    @           0x4598b9 pyrun_file.cold
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x588e5e (unknown)
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @           0x5960d7 PyEval_EvalCode
    @           0x5c72d7 run_eval_code_obj
    @           0x5c2420 run_mod
    @                0x0 (unknown)
    @           0x588fa9 Py_BytesMain
    @           0x588e5e (unknown)
    @           0x588e5e (unknown)
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x4fe4cf _PyFunction_Vectorcall
    @           0x588fa9 Py_BytesMain
    @                0x0 (unknown)
    @     0x7f715bbb8d90 (unknown)
    @                0x0 (unknown)
    @           0x5960d7 PyEval_EvalCode
    @           0x4ee7cf _PyEval_EvalFrameDefault
    @           0x596192 _PyEval_Vector
    @           0x5c72d7 run_eval_code_obj
    @     0x7f76ce1b9d90 (unknown)
    @           0x5c2420 run_mod
    @           0x4598b9 pyrun_file.cold
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @     0x7f715bbb8e40 __libc_start_main
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @     0x7f76ce1b9e40 __libc_start_main
    @           0x5960d7 PyEval_EvalCode
    @           0x5c72d7 run_eval_code_obj
    @           0x5c2420 run_mod
    @           0x4598b9 pyrun_file.cold
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x588fa9 Py_BytesMain
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @           0x588e5e (unknown)
    @           0x588e5e (unknown)
    @           0x588fa9 Py_BytesMain
    @     0x7f98cbcaad90 (unknown)
    @                0x0 (unknown)
    @                0x0 (unknown)
    @     0x7f98cbcaae40 __libc_start_main
    @     0x7f0071b4ad90 (unknown)
    @           0x588e5e (unknown)
    @     0x7f0071b4ae40 __libc_start_main
    @           0x588e5e (unknown)
    @                0x0 (unknown)
    @                0x0 (unknown)
    @           0x4598b9 pyrun_file.cold
    @           0x5bc9af _PyRun_SimpleFileObject.localalias
    @           0x5bc713 _PyRun_AnyFileObject.localalias
    @           0x5b94cd Py_RunMain.localalias
    @           0x588fa9 Py_BytesMain
    @     0x7f735e3dad90 (unknown)
    @     0x7f735e3dae40 __libc_start_main
    @           0x588e5e (unknown)
    @                0x0 (unknown)