gather_cxi_counters

Collect and aggregate HPE Slingshot CXI network counters across HPC job nodes

---

Table of Contents

• What it does
• Prerequisites
• Build
• Quick Start
• Environment Variables
• Multi-Node Usage
• Output Interpretation
• Advanced Features
• Troubleshooting
• Contributing
• License

---

What it does

HPE Slingshot uses the CXI (Cassini Network Interface) driver to expose hundreds of per-NIC performance and diagnostic counters through the Linux sysfs interface. On large clusters these counters are spread across many nodes and NICs, making manual collection impractical during a live job.

gather_cxi_counters is a C++ binary that wraps a user-supplied command, reading CXI counters before and after the command executes on every node of a SLURM job. It then aggregates the counter deltas across all nodes using a ZeroMQ chain topology and prints a summary report showing per-second rates across all participating NICs.

Typical use cases include:

• MPI/collective benchmarking — quantify the exact Slingshot traffic generated by all_reduce_perf, NCCL tests, or application runs
• Bottleneck identification — correlate high error counters with application slowdowns or MPI hangs
• Fabric health checks — verify counter activity before and after maintenance windows
• Performance characterization — measure bandwidth, packet rates, and error rates under real workloads

---

Prerequisites

Hardware

• One or more nodes equipped with HPE Slingshot 200G NICs (Cassini ASIC, also known as CXI). Devices appear as cxi0, cxi1, … under /sys/class/net/. The tool reads counters from /sys/class/cxi/cxi<N>/device/ports/0/counters/.

Operating System

• Linux kernel with the cxi_core kernel module loaded
• Verified: RHEL 9 (5.14.x kernel), HPE Cray OS

Build Dependencies

Component	Minimum Version	Notes
CMake	3.20	Required by CMakeLists.txt
C++ compiler	C++20 support	GCC 11+, Clang 14+ (Clang via Cray PE also works)
git	any	Required for CMake FetchContent (pulls libzmq automatically)
Internet access at build time	—	FetchContent clones github.com/zeromq/libzmq

libzmq is fetched automatically by CMake's FetchContent. You do not need to install it separately. The build produces a fully statically linked binary.

Non-fatal CMake warnings: If GnuTLS or AsciiDoctor are absent on your build host, CMake will print warnings about missing WSS support and documentation generation. Neither affects the binary's functionality.

Runtime Requirements

• SLURM (srun) for multi-node invocation
• CXI devices present and accessible on compute nodes
• No special privileges required for sysfs counter reads

---

Build

git clone <repo-url> gather_cxi_counters
cd gather_cxi_counters
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j8

The result is a single statically linked binary:

build/gather_cxi_counters   (ELF 64-bit, ~966 KB)

Copy or stage it to a path accessible from your compute nodes before running under SLURM.

---

Quick Start

gather_cxi_counters wraps any command you would normally pass to srun. It measures CXI counter activity for the duration of that command.

Single node

srun -N 1 -n 1 ./gather_cxi_counters sleep 2

Expected output (no network traffic from sleep; all deltas are zero and filtered from the table):

Slingshot CXI Counter Summary:
Counter                                                Min        (/s)        Mean        (/s)         Max        (/s)

Single node, JSON output

GATHER_CXI_JSON=1 srun -N 1 -n 1 ./gather_cxi_counters sleep 2

{
    "counters": {},
    "time_interval": 2.004058238
}

time_interval is the wall-clock duration of the wrapped command in seconds. counters is empty because sleep generates no Slingshot traffic.

Wrapping a real workload

Replace sleep 2 with any MPI program or collective benchmark:

srun -N 4 -n 4 --ntasks-per-node=1 ./gather_cxi_counters \
    mpirun -np 4 ./all_reduce_perf -b 8 -e 4G -f 2

When the workload generates network traffic, the counter table will be populated with rate columns.

---

Environment Variables

All configuration is through environment variables set before srun. There are no command-line flags (except the optional -e experiment name; see Advanced Features).

Variable	Default	Description
GATHER_CXI_JSON	0	Set to 1 to emit output as JSON instead of a text table
COUNTER_LEVEL	1	Output verbosity: 0=quiet, 1=default, 2=summary, 3=on_error, 4=all_on_error, 5=all counters
GATHER_CXI_COUNTERS	(all)	Comma-separated list of counter names to collect; filters the counter set at runtime
GATHER_CXI_LOG	0	Set to 1 to enable verbose logging of inter-node ZMQ communication
GATHER_CXI_DETAILED	0	Set to 1 to enable per-node, per-NIC detailed output (time-series)
GATHER_CXI_INTERVAL	100	Time-series sampling interval in milliseconds (used when GATHER_CXI_DETAILED=1)
GATHER_CXI_OS_METRICS	0	Set to 1 to enable Linux perf event metric collection alongside CXI counters
GATHER_CXI_NCCL_PROFILER	0	Set to 1 to enable the NCCL/RCCL proxy-step profiler plugin (requires librccl-profiler-gather.so in LD_LIBRARY_PATH)
GATHER_CXI_CPU_PIN	(none)	CPU affinity mask for the sidecar collection thread

COUNTER_LEVEL behavior note

COUNTER_LEVEL controls which counters are included in the output table, but the filter operates on delta values (final minus initial counter reading), not absolute raw values. Even at COUNTER_LEVEL=5 ("all counters"), if a workload generates zero Slingshot traffic the output table will be empty — because all deltas are zero. This is correct behavior. To see populated output, wrap a workload that generates real network activity.

---

Multi-Node Usage

Multi-node aggregation is automatic. No special flags are required. Run with srun -N <count> and the tool instances on each node will self-organize into a ZMQ chain topology, aggregate counter data, and produce a single summary on the first node (rank 0).

srun -N 2 -n 2 --ntasks-per-node=1 ./gather_cxi_counters sleep 2

Example output (2-node run; empty table expected for sleep):

Slingshot CXI Counter Summary:
Counter                                                Min        (/s)        Mean        (/s)         Max        (/s)

With GATHER_CXI_JSON=1:

{
    "counters": {},
    "time_interval": 2.003986692
}

How the chain topology works

• Nodes are ordered by SLURM_NODEID (0 = first/aggregator, N−1 = last).
• The last node collects its local counters, runs the command, then sends its data to the previous node via ZMQ.
• Middle nodes receive data from the node ahead of them, add their own counters, and forward the accumulated result.
• The first node (rank 0) receives the fully aggregated result, computes min/mean/max statistics, and prints the final summary.

This chain pattern minimizes network overhead and avoids a central coordinator.

One task per node: Use --ntasks-per-node=1. If multiple tasks per node are allocated, only SLURM_LOCALID=0 on each node participates in counter collection; other tasks run the wrapped command transparently.

---

Output Interpretation

Text table

Slingshot CXI Counter Summary:
Counter                                                Min        (/s)        Mean        (/s)         Max        (/s)
hni_tx_ok_pkts                                       1234       617.0        1456       728.0        1678       839.0

Column	Meaning
Counter	CXI sysfs counter name
Min	Minimum absolute delta across all nodes/NICs
(/s) (after Min)	Min delta divided by elapsed time (per-second rate)
Mean	Mean absolute delta
(/s) (after Mean)	Mean delta per second
Max	Maximum absolute delta across all nodes/NICs
(/s) (after Max)	Max delta per second

Counters with zero delta across all nodes are omitted from the table by default.

JSON output (GATHER_CXI_JSON=1)

{
    "counters": {
        "hni_tx_ok_pkts": {
            "min": 1234,
            "max": 1678,
            "mean": 1456.0,
            "min_per_sec": 617.0,
            "max_per_sec": 839.0,
            "mean_per_sec": 728.0
        }
    },
    "time_interval": 2.004058238
}

time_interval is the elapsed time of the wrapped command in seconds, as measured on the aggregating node.

---

Advanced Features

Experiment name label (-e)

Tag a run with an experiment name for downstream identification in CSV output:

srun -N 4 -n 4 --ntasks-per-node=1 ./gather_cxi_counters -e allreduce_4node \
    ./all_reduce_perf -b 8 -e 4G -f 2

Counter filtering (GATHER_CXI_COUNTERS)

Restrict collection to a specific subset of counters at runtime:

GATHER_CXI_COUNTERS=hni_tx_ok_pkts,hni_rx_ok_pkts \
    srun -N 1 -n 1 ./gather_cxi_counters ./my_benchmark

NCCL/RCCL profiler plugin

Enable the built-in NCCL/RCCL proxy-step profiler to correlate CXI counters with collective operation boundaries:

export GATHER_CXI_NCCL_PROFILER=1
export LD_LIBRARY_PATH=<path-to-nccl_profiler>:$LD_LIBRARY_PATH
srun -N 4 -n 4 --ntasks-per-node=1 ./gather_cxi_counters ./nccl_test

The nccl_profiler/ subdirectory in the source tree builds a librccl-profiler-gather.so shared library. Build the project normally with CMake — it is included automatically.

---

Troubleshooting

Empty counter table

Symptom: Output shows the table header but no counter rows.

Cause: All counter deltas are zero — the wrapped workload generated no Slingshot network traffic.

Fix: Wrap a workload that uses the network (MPI collectives, NCCL tests, etc.). Using sleep or a CPU-only program will always produce an empty table.

---

"No CXI devices found"

Symptom: Tool exits immediately or reports no devices.

Cause: The node does not have CXI NICs or the cxi_core module is not loaded.

Fix: Verify with ls /sys/class/cxi/ on the compute node. Confirm cxi_core is loaded with lsmod | grep cxi. Contact your system administrator if CXI devices are expected but absent.

---

ZMQ communication errors / hang on multi-node run

Symptom: Tool hangs or prints ZMQ error messages in multi-node mode.

Cause: Network routing between compute nodes is blocked for the ZMQ port, or nodes were not allocated contiguously.

Fix:

1. Enable verbose ZMQ logging: GATHER_CXI_LOG=1 srun ...
2. Confirm all allocated nodes can reach each other on the required ZMQ port.
3. Ensure --ntasks-per-node=1 so each node has exactly one aggregating task.

---

CMake FetchContent fails (no internet access)

Symptom: CMake fails during FetchContent_MakeAvailable(libzmq) with a network error.

Cause: The build host cannot reach github.com to clone libzmq.

Fix: Clone https://github.com/zeromq/libzmq.git manually on a machine with internet access, then copy it to your build host. Point FetchContent to the local path by setting FETCHCONTENT_SOURCE_DIR_LIBZMQ before running CMake:

cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DFETCHCONTENT_SOURCE_DIR_LIBZMQ=/path/to/local/libzmq

---

CMake warnings about GnuTLS or AsciiDoctor

These are non-fatal warnings printed by the libzmq build system. They indicate WebSocket Secure (WSS) support and documentation generation are unavailable. Neither affects the functionality of gather_cxi_counters.

---

Contributing

See CONTRIBUTING.md for build setup, coding standards, and the pull request process.

---

License

MIT License — see LICENSE.md.

Copyright (C) 2026 Hewlett Packard Enterprise Development LP