Supported Communication Protocols

Supported Communication Protocols#

Mooncake Transfer Engine supports multiple communication protocols for data transfer between nodes in a cluster. The protocol selection depends on your hardware capabilities and performance requirements.

Quick Reference#

Protocol

Hardware Required

Use Case

Python API Support

tcp

Standard network

General purpose, works everywhere

✅ Primary

rdma

RDMA-capable NIC

High-performance, low-latency

✅ Primary

efa

AWS EFA-capable instance

High-performance on AWS (libfabric SRD)

✅ Primary

nvmeof

NVMe-oF capable storage

Direct NVMe storage access

⚠️ Advanced

nvlink

NVIDIA MNNVL

Inter-node GPU communication

⚠️ Advanced

musa

Moore Threads GPU + MTLink

Intra-node GPU IPC/P2P

⚠️ Advanced

nvlink_intra

NVIDIA NVLink

Intra-node GPU communication

⚠️ Advanced

hip

AMD ROCm/HIP

AMD GPU communication

⚠️ Advanced

barex

RDMA-capable NIC

Bare-metal RDMA extension

⚠️ Advanced

cxl

CXL-capable hardware

Memory pooling and sharing

⚠️ Advanced

shm

None (POSIX shm, same host)

Same-host DRAM copies without NIC loopback

⚠️ Advanced

ascend

Huawei Ascend NPU

Ascend NPU communication

⚠️ Advanced

hylink

Hygon DCU + DTK

Hygon DCU HSL fabric communication

⚠️ Advanced

tpu

Google TPU (PJRT)

TPU KV-cache transfer via host-DRAM staging

🧪 Experimental (TENT)

xpu

Intel GPU + oneAPI SYCL

Host staging and same-node XPU P2P/IPC; TCP/RDMA between nodes

⚠️ Advanced (TENT)

mpcomm

RDMA-capable NIC(s)

Multi-NIC memory pooling with NIC/QP load balancing

⚠️ Advanced (TENT)

flagcx

RDMA-capable NIC(s)

Unified P2P transfer through FlagOS FlagCX

⚠️ Advanced

Commonly Used Protocols (Python API)#

TCP (Default)#

Description: Standard TCP/IP network protocol.

Use When:

  • No special hardware is available

  • Testing or development environments

  • Compatibility is more important than performance

Configuration:

# Python API
engine.initialize(
    hostname="localhost",
    metadata_server="P2PHANDSHAKE",
    protocol="tcp",  # No device_name needed
    device_name=""
)

# Environment variables
export MOONCAKE_PROTOCOL="tcp"

Advantages:

  • Works in all environments

  • No special hardware required

  • Simple setup

Limitations:

  • Lower throughput compared to RDMA

  • Higher CPU overhead

  • Higher latency

EFA (AWS Elastic Fabric Adapter)#

Description: AWS EFA transport using libfabric’s Scalable Reliable Datagram (SRD) protocol, providing high-bandwidth RDMA-like performance on AWS instances without traditional RDMA support.

Use When:

  • Running on AWS EFA-enabled instances (e.g., p5e.48xlarge, p6-b200.48xlarge, p4d.24xlarge)

  • High-performance networking is required on AWS

  • Traditional RDMA (ibverbs QP) is not supported by the hardware

Configuration:

# Python API
engine.initialize(
    hostname="localhost",
    metadata_server="P2PHANDSHAKE",
    protocol="efa",
    device_name=""
)

Build Requirements:

cmake .. -DUSE_EFA=ON -DUSE_CUDA=ON

Note: -DUSE_CUDA=ON is required when transferring GPU memory. Without it, fallback to TCP protocol will fail with “Bad address” errors on GPU buffers.

Advantages:

  • High throughput (~170 GB/s with 8 EFA devices, tuned)

  • Bypasses kernel network stack

  • Available on all AWS EFA-enabled instances

Limitations:

  • AWS-only

  • Software-emulated RDMA writes (higher CPU overhead than true RDMA)

  • ~88% of RoCE RDMA throughput

Documentation: See EFA Transport for build instructions, benchmarks, and tuning.

Advanced Protocols (C++ Transfer Engine)#

The following protocols are available at the C++ Transfer Engine level for specialized use cases. They are not commonly used through the Python API.

NVMe over Fabric (nvmeof)#

Description: Direct data transfer between NVMe storage and DRAM/VRAM using GPUDirect Storage, bypassing the CPU for zero-copy operations.

Use When:

  • Direct NVMe storage access is needed

  • Implementing multi-tier storage (DRAM/VRAM/NVMe)

  • Working with large datasets that don’t fit in memory

Requirements:

  • NVMe-oF capable storage

  • Properly mounted remote storage nodes

MUSA Transport (musa)#

Description: Moore Threads GPU IPC transport for P2P copies over the intra-node MTLink path. It reuses the NVLink transport’s transfer bookkeeping, but opens imported IPC memory and submits copies using MUSA-specific device context rules.

Requirements:

  • Moore Threads GPUs with peer access (validated on S5000)

  • MUSA SDK/runtime; MUSA 5.2 or newer enables the low-CPU transfer-batch API

  • Compiled with USE_MUSA=ON

Configuration:

# Use the same runtime-visible logical-device mapping in every peer process.
export MUSA_VISIBLE_DEVICES=0,1
export MC_FORCE_MUSA=1

# Safe defaults shown explicitly. Opt in to metadata after checking the
# visibility contract below; "default" rolls back to per-slice copies.
export MC_MUSA_IPC_OPEN_DEVICE=current
export MC_MUSA_COPY_API=auto

# Performance path after both peers use the same logical device mapping.
export MC_MUSA_IPC_OPEN_DEVICE=metadata

MTHREADS_VISIBLE_DEVICES is consumed by mt-container-toolkit when the container is created; Mooncake does not use it to infer the MUSA runtime’s logical device mapping. Buffer metadata uses runtime-visible logical ordinals such as musa:0. With metadata, every peer must map each logical ordinal to the same physical GPU. Mooncake validates that the advertised ordinal exists locally, but it cannot prove cross-peer identity from environment variables. Both peers must run a version that recognizes the musa protocol; rolling interoperability with an older peer advertising only nvlink is not supported. MC_MUSA_IPC_OPEN_DEVICE=current is the safe default; select metadata only when peers satisfy the logical mapping contract above. Python bindings that register the default wildcard location (*) resolve the owning MUSA device during registration; an older peer that still advertises * falls back to the current-device open path.

In auto mode, batches whose copies are at least 1 MiB use muMemoryTransferBatchAsync; set MC_MUSA_TRANSFER_BATCH_MIN_BYTES to tune the threshold, MC_MUSA_COPY_API=transfer_batch to force the API, or MC_MUSA_COPY_API=default to use the CUDA-compatible per-slice path.

HIP Transport (hip)#

Description: AMD ROCm/HIP transport for GPU communication using IPC handles or Shareable handles.

Use When:

  • Working with AMD GPUs

  • Need intra-node GPU communication on AMD hardware

Requirements:

  • AMD ROCm/HIP runtime

  • AMD GPUs

Barex Transport (barex)#

Description: Bare-metal RDMA extension protocol for specialized RDMA configurations.

Use When:

  • Advanced RDMA features are required

  • Custom RDMA configurations

Requirements:

  • RDMA-capable hardware

  • Specialized configuration

CXL Transport (cxl)#

Description: Compute Express Link for memory pooling and sharing across devices.

Use When:

  • CXL memory pooling is available

  • Memory disaggregation is needed

Requirements:

  • CXL-capable hardware

SHM Transport (shm)#

Description: Same-host DRAM copies over POSIX shared memory. Classic Transfer Engine maps the peer’s named shm object, relocates the peer virtual address into the local mapping, and memcpys. This is a first-class transport like HIP, not a replacement for CxlTransport (DAX offset addressing).

Use When:

  • Two processes on the same machine exchange DRAM buffers

  • You want to avoid RDMA/TCP loopback for that path

Requirements:

  • Linux POSIX shm (/dev/shm), or a writable hugetlbfs mount when allocating with SharedMemoryOptions.use_hugepage (2MB / 512MB / 1GB)

  • Buffers allocated with TransferEngine::allocateSharedMemory (ordinary malloc cannot be exported)

  • Runtime opt-in: MC_FORCE_SHM=1, or installTransport("shm"). With -DENABLE_MULTI_PROTOCOL=ON this adds SHM next to RDMA/TCP (rdma,shm / tcp,shm); without it, SHM is the only transport.

  • Same-host SHM and cross-host RDMA/TCP in one engine: build with -DENABLE_MULTI_PROTOCOL=ON (segment protocol becomes rdma,shm or tcp,shm)

Limitations:

  • Same host only. Without ENABLE_MULTI_PROTOCOL, MC_FORCE_SHM=1 (or installTransport("shm") after another transport) sets segment.protocol to shm and replaces RDMA/TCP routing; installTransport("shm") logs a WARNING when it overwrites a non-empty protocol. Coexistence needs -DENABLE_MULTI_PROTOCOL=ON.

  • registerLocalMemory must use the pointer from allocateSharedMemory (a shorter prefix is allowed). A sub-range or overflowing range returns an error instead of silently skipping. Ordinary malloc is still skipped so TCP/RDMA can register it.

  • Same-UID only: objects are created 0600. POSIX names are /mooncake_<pid>_xxxxxxxx; hugepage files are <hugetlbfs-mount>/mooncake_<pid>_xxxxxxxx. Creator and consumer must share a user; a hostname match does not imply a shared /dev/shm or hugetlbfs mount (for example Kubernetes hostNetwork pods).

  • Crash or SIGKILL can leave POSIX objects in /dev/shm and hugetlbfs files on the mount. There is no automatic reaper (wiping mooncake_* on start would hit live peers on the same mount). POSIX leftovers waste tmpfs until reboot; hugetlbfs leftovers keep hugepages reserved until the file is unlinked or the node reboots. After a crash, delete only mooncake_<pid>_* whose pid no longer exists, e.g. rm /dev/hugepages/mooncake_<dead-pid>_*.

  • Hugepage allocations do not fall back to tmpfs. length must be a multiple of the hugepage size; TE does not round up.

  • After freeSharedMemory + allocateSharedMemory, a peer that still has a cached mapping probes the object name before memcpy. An unlinked object is dropped and the segment descriptor is refetched once; a changed virtual address still requires the initiator to read the new BufferDesc.addr (relocate cannot guess a new offset). Background refresh remains optional via MC_TE_METADATA_REFRESH_INTERVAL_SECONDS.

  • Relocate caches at most 32 mmap’d peer objects per target. An in-flight copy pins its mapping so prune/cap cannot munmap it until memcpy returns; the cache may briefly exceed 32 while pins are held.

  • Default off because the path is not NUMA-aware

  • Mooncake Store segments are not shm-backed until a follow-up allocator change

Ascend Transport (ascend)#

Description: Huawei Ascend NPU communication using HCCL (Huawei Collective Communication Library) or direct transport.

Use When:

  • Working with Huawei Ascend NPUs

  • Distributed inference on Ascend hardware

Requirements:

  • Huawei Ascend NPU hardware

  • HCCL runtime

Documentation:

TPU Transport (tpu) — Experimental#

Description: Google TPU support in the TENT runtime. Because TPU HBM is not directly addressable by the NIC, transfers touching TPU memory are staged through host DRAM: the HBM ↔ host-DRAM hop is performed by a PJRT device-copy adapter, and the host ↔ host hop is carried by an existing transport (RDMA/TCP). The two stages are chained automatically by the TENT staging pipeline (ProxyManager), so no separate networked TPU transport is required.

Status: Experimental. The C++/TENT data path is gated behind -DUSE_TPU=ON (OFF by default). A serving-framework (JAX / PyTorch-XLA) integration layer is planned as a follow-up.

Use When:

  • Disaggregated prefill/decode serving on TPU hosts

  • KV-cache transfer between TPU nodes over RDMA/TCP

Requirements:

  • Built with -DUSE_TPU=ON -DUSE_TENT=ON

  • A PJRT device-copy adapter shared library exposing the mc_tpu_pjrt_* C ABI (see tpu_pjrt_abi.h). The adapter is resolved at runtime via dlopen; its path defaults to libmooncake_tpu_pjrt.so and can be overridden with the MC_TPU_PJRT_LIB environment variable. No PJRT/TPU SDK is required at build time.

  • An RDMA (or TCP) transport enabled for the host ↔ host hop.

Design notes:

  • TPU memory is reported as a distinct memory type (tpu:N locations); the staging policy routes the local HBM ↔ host copy to the TPU device-copy transport and the cross-node hop to RDMA/TCP.

  • DMA-mapped (pinned) staging buffers for true async device DMA are a planned performance follow-up.

Intel XPU Transport (xpu)#

Description: Intel GPU support in TENT through oneAPI SYCL. On the same node, the XPU transport copies between XPUs. Within a process, the copy runs directly over PCIe (or Xe Link) when the two devices have peer access, and is bounced through host memory otherwise. Between processes, it maps the peer’s buffer through a Level Zero IPC handle when the IPC requirements below are met. With rdma, registered XPU memory is accessed directly through dma-buf when available. Otherwise, the XPU transport performs the local device-to-host and host-to-device copies, and an existing TCP/RDMA transport carries the staged data between peers. See the Intel XPU notes in the RDMA section above.

Status: TENT only. xpu is a local transport (host staging and same-node XPU copies), not a network protocol to pass to Store setup. Select tcp or rdma for the network hop.

Use When:

  • Moving Intel GPU device buffers through TENT

  • Copying between Intel GPUs on one node, including between processes such as prefill and decode workers

  • Reading and writing PyTorch XPU tensors with the C++ Mooncake Store backend

Requirements:

  • Built with -DUSE_XPU=ON -DUSE_TENT=ON using the Intel oneAPI compilers (icpx for C++ is required; the build examples also use icx for C)

  • Compatible SYCL and Level Zero runtimes and access to the Intel GPU

  • MC_USE_TENT=1 before constructing the Store client or compatibility API engine

  • A TCP or RDMA transport for transfers between nodes, or when same-node IPC is unavailable

Configuration:

export ONEAPI_DEVICE_SELECTOR=level_zero:gpu
export MC_USE_TENT=1

Design notes:

  • Device memory uses xpu:N locations. Host/shared USM is treated as host memory.

  • PyTorch XPU device pointers are recognized through the platform default SYCL context. Synchronize producing work before passing a buffer to Mooncake, and keep registered allocations alive until transfers complete.

  • Same-node IPC is used only when the selector policy allows xpu; a policy with a non-empty transports list must name "xpu" ahead of "rdma" (see Transport Selector). Set transports/xpu/disable_ipc to true to turn IPC off.

Limitations:

  • No XPU support in the classic Transfer Engine

  • Transfers involving XPU memory require an XPU-platform initiator (see the RDMA section above)

  • Same-node IPC between processes requires both processes in the same PID namespace, peer access from the copying device to the device that holds the buffer, and ptrace permission over the exporting process: Yama ptrace_scope 0 or 1, or CAP_SYS_PTRACE for scope 2; scope 3 rejects the import even with that capability. Otherwise the transfer takes the network path (RDMA dma-buf or host staging); see the GPU-Direct note in the build guide.

Build instructions: See Build with Intel XPU.

MPComm Transport (mpcomm)#

Description: UCL-MPComm (Unified Communication Library - Memory Pool Communication) is an RDMA library for heterogeneous memory pooling, integrated as a TENT transport. It drives multiple RDMA NICs concurrently with two-level load balancing (across NICs, and across QPs within a NIC) and NUMA-aware worker placement, exposing one-sided put/get primitives. It is shortened to MPComm below.

Status: TENT only. There is no MPComm backend on the legacy Transfer Engine transport path, so it cannot be selected through MOONCAKE_PROTOCOL or transfer_engine_bench --protocol=.

Use When:

  • The host has several RDMA NICs and you want them saturated by a single transfer stream

  • Multi-NUMA hosts where NIC-to-NUMA affinity matters

Requirements:

  • Built with -DUSE_TENT=ON -DUSE_MPCOMM=ON -DMPCOMM_ROOT=<prefix>

  • MPComm installed, providing include/mpcomm.h and lib/libmpcomm.so (Tencent/UCL-MPComm)

  • libmpcomm.so reachable by the dynamic linker at run time

Enable:

{ "transports": { "mpcomm": { "enable": true } } }

See MPComm Transport for the full guide, including selection via transport policy, tuning environment variables, and troubleshooting.

FlagOS FlagCX Transport (flagcx)#

Description: FlagCX is the unified communication library in the FlagOS ecosystem for multi-vendor and cross-vendor deployments. Mooncake integrates the FlagCX P2P Engine as a classic Transfer Engine transport, allowing the existing Mooncake transfer workflow to use the accelerator and network backends provided by the local FlagCX build.

Use When:

  • Deploying Mooncake on a platform supported by FlagCX

  • Using FlagCX’s P2P Engine for accelerator memory transfers

  • Building a cross-vendor deployment around the FlagOS communication stack

Build Requirements:

cmake -S . -B build \
  -DUSE_FLAGCX=ON \
  -DFLAGCX_HOME=/path/to/FlagCX/build
cmake --build build -j

FLAGCX_HOME must contain include/flagcx_p2p.h and either lib/libflagcx.so or lib64/libflagcx.so. If it is omitted, Mooncake checks $FLAGCX_HOME and then $HOME/FlagCX/build.

Configuration:

engine.initialize(
    hostname="node1",
    metadata_server="P2PHANDSHAKE",
    protocol="flagcx",
    device_name=""
)

# Select the interface used for FlagCX bootstrap and endpoint advertisement.
export FLAGCX_SOCKET_IFNAME="eth0"

Current Scope:

  • Available through the classic Transfer Engine; it is not a TENT transport

  • Must be built from source with USE_FLAGCX=ON

  • Should be selected as the standalone flagcx protocol, not as part of a multi-protocol string

  • Buffers should be registered before the first transfer to a peer and remain registered while that peer connection is active

See FlagOS FlagCX Transport for dependency, build, benchmark, runtime configuration, and troubleshooting details.

Configuration Examples#

Configuration File (JSON)#

TCP Configuration:

{
    "local_hostname": "localhost",
    "metadata_server": "localhost:8080",
    "protocol": "tcp",
    "device_name": "",
    "master_server_address": "localhost:8081"
}

RDMA Configuration:

{
    "local_hostname": "node1",
    "metadata_server": "etcd://10.0.0.1:2379",
    "global_segment_size": "3GB",
    "local_buffer_size": "1GB",
    "protocol": "rdma",
    "device_name": "mlx5_0",
    "master_server_address": "10.0.0.1:8081"
}

Environment Variables#

# TCP (Default)
export MOONCAKE_PROTOCOL="tcp"

# RDMA with specific device
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="mlx5_0"

# RDMA with auto-discovery
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="auto-discovery"

# Other configuration
export MOONCAKE_MASTER="10.0.0.1:50051"
export MOONCAKE_TE_META_DATA_SERVER="P2PHANDSHAKE"
export MOONCAKE_LOCAL_HOSTNAME="node1"

Choosing the Right Protocol#

Scenario

Recommended Protocol

Notes

Development/Testing

tcp

Simple setup, no special hardware

Production Inference

rdma

Best performance and latency

AWS Cloud (EFA instances)

efa

High performance on p5e, p6-b200, p4d, etc.

Cloud Environments

tcp or rdma (if available)

Check cloud provider support

Multi-tier Storage

rdma + nvmeof

Combine protocols for different layers

AMD GPU Clusters

rdma + hip

Use HIP for local GPU communication

Cambricon MLU Clusters

rdma

Build with -DUSE_MLU=ON; MLU uses the normal RDMA protocol

Intel XPU Clusters

tcp or rdma (TENT)

Build with -DUSE_XPU=ON -DUSE_TENT=ON and set MC_USE_TENT=1; RDMA uses dma-buf when available, otherwise host staging; same-node XPUs copy over P2P/IPC

Ascend NPU Clusters

rdma + ascend

Use Ascend for NPU-specific operations

Hygon DCU Clusters

rdma + hylink

Build with -DUSE_HYLINK=ON; use hylink for HSL fabric communication

Multi-vendor or cross-vendor clusters

flagcx

Build with -DUSE_FLAGCX=ON; transfers use the FlagCX P2P Engine over RDMA-capable NICs

Troubleshooting#

RDMA Connection Issues#

  1. Check RDMA devices:

    ibv_devices
ibv_devinfo

  2. Verify network connectivity:

    # Test RDMA connectivity (requires rdma-core tools)
rping -s  # On server
rping -c -a <server_ip> -v  # On client

  3. Check permissions:

    • RDMA may require elevated permissions

    • Run with sudo if necessary

    • Configure proper udev rules for non-root access

  4. Firewall configuration:

    • Ensure RDMA ports are not blocked

    • Check InfiniBand subnet manager is running

Protocol Selection#

If a protocol fails to initialize:

  1. Verify hardware support

  2. Check that required drivers are installed

  3. Ensure compile-time flags are set correctly (for C++ protocols)

  4. Fall back to TCP for basic functionality

See Also#

Contents