Supported Communication Protocols

Supported Communication Protocols#

Mooncake Transfer Engine supports multiple communication protocols for data transfer between nodes in a cluster. The protocol selection depends on your hardware capabilities and performance requirements.

Quick Reference#

Protocol

Hardware Required

Use Case

Python API Support

tcp

Standard network

General purpose, works everywhere

✅ Primary

rdma

RDMA-capable NIC

High-performance, low-latency

✅ Primary

efa

AWS EFA-capable instance

High-performance on AWS (libfabric SRD)

✅ Primary

nvmeof

NVMe-oF capable storage

Direct NVMe storage access

⚠️ Advanced

nvlink

NVIDIA MNNVL

Inter-node GPU communication

⚠️ Advanced

musa

Moore Threads GPU + MTLink

Intra-node GPU IPC/P2P

⚠️ Advanced

nvlink_intra

NVIDIA NVLink

Intra-node GPU communication

⚠️ Advanced

hip

AMD ROCm/HIP

AMD GPU communication

⚠️ Advanced

barex

RDMA-capable NIC

Bare-metal RDMA extension

⚠️ Advanced

cxl

CXL-capable hardware

Memory pooling and sharing

⚠️ Advanced

shm

None (POSIX shm, same host)

Same-host DRAM copies without NIC loopback

⚠️ Advanced

ascend

Huawei Ascend NPU

Ascend NPU communication

⚠️ Advanced

tpu

Google TPU (PJRT)

TPU KV-cache transfer via host-DRAM staging

🧪 Experimental (TENT)

mpcomm

RDMA-capable NIC(s)

Multi-NIC memory pooling with NIC/QP load balancing

⚠️ Advanced (TENT)

flagcx

RDMA-capable NIC(s)

Unified P2P transfer through FlagOS FlagCX

⚠️ Advanced

Commonly Used Protocols (Python API)#

TCP (Default)#

Description: Standard TCP/IP network protocol.

Use When:

  • No special hardware is available

  • Testing or development environments

  • Compatibility is more important than performance

Configuration:

# Python API
engine.initialize(
    hostname="localhost",
    metadata_server="P2PHANDSHAKE",
    protocol="tcp",  # No device_name needed
    device_name=""
)

# Environment variables
export MOONCAKE_PROTOCOL="tcp"

Advantages:

  • Works in all environments

  • No special hardware required

  • Simple setup

Limitations:

  • Lower throughput compared to RDMA

  • Higher CPU overhead

  • Higher latency

EFA (AWS Elastic Fabric Adapter)#

Description: AWS EFA transport using libfabric’s Scalable Reliable Datagram (SRD) protocol, providing high-bandwidth RDMA-like performance on AWS instances without traditional RDMA support.

Use When:

  • Running on AWS EFA-enabled instances (e.g., p5e.48xlarge, p6-b200.48xlarge, p4d.24xlarge)

  • High-performance networking is required on AWS

  • Traditional RDMA (ibverbs QP) is not supported by the hardware

Configuration:

# Python API
engine.initialize(
    hostname="localhost",
    metadata_server="P2PHANDSHAKE",
    protocol="efa",
    device_name=""
)

Build Requirements:

cmake .. -DUSE_EFA=ON -DUSE_CUDA=ON

Note: -DUSE_CUDA=ON is required when transferring GPU memory. Without it, fallback to TCP protocol will fail with “Bad address” errors on GPU buffers.

Advantages:

  • High throughput (~170 GB/s with 8 EFA devices, tuned)

  • Bypasses kernel network stack

  • Available on all AWS EFA-enabled instances

Limitations:

  • AWS-only

  • Software-emulated RDMA writes (higher CPU overhead than true RDMA)

  • ~88% of RoCE RDMA throughput

Documentation: See EFA Transport for build instructions, benchmarks, and tuning.

Advanced Protocols (C++ Transfer Engine)#

The following protocols are available at the C++ Transfer Engine level for specialized use cases. They are not commonly used through the Python API.

NVMe over Fabric (nvmeof)#

Description: Direct data transfer between NVMe storage and DRAM/VRAM using GPUDirect Storage, bypassing the CPU for zero-copy operations.

Use When:

  • Direct NVMe storage access is needed

  • Implementing multi-tier storage (DRAM/VRAM/NVMe)

  • Working with large datasets that don’t fit in memory

Requirements:

  • NVMe-oF capable storage

  • Properly mounted remote storage nodes

MUSA Transport (musa)#

Description: Moore Threads GPU IPC transport for P2P copies over the intra-node MTLink path. It reuses the NVLink transport’s transfer bookkeeping, but opens imported IPC memory and submits copies using MUSA-specific device context rules.

Requirements:

  • Moore Threads GPUs with peer access (validated on S5000)

  • MUSA SDK/runtime; MUSA 5.2 or newer enables the low-CPU transfer-batch API

  • Compiled with USE_MUSA=ON

Configuration:

# Use the same runtime-visible logical-device mapping in every peer process.
export MUSA_VISIBLE_DEVICES=0,1
export MC_FORCE_MUSA=1

# Safe defaults shown explicitly. Opt in to metadata after checking the
# visibility contract below; "default" rolls back to per-slice copies.
export MC_MUSA_IPC_OPEN_DEVICE=current
export MC_MUSA_COPY_API=auto

# Performance path after both peers use the same logical device mapping.
export MC_MUSA_IPC_OPEN_DEVICE=metadata

MTHREADS_VISIBLE_DEVICES is consumed by mt-container-toolkit when the container is created; Mooncake does not use it to infer the MUSA runtime’s logical device mapping. Buffer metadata uses runtime-visible logical ordinals such as musa:0. With metadata, every peer must map each logical ordinal to the same physical GPU. Mooncake validates that the advertised ordinal exists locally, but it cannot prove cross-peer identity from environment variables. Both peers must run a version that recognizes the musa protocol; rolling interoperability with an older peer advertising only nvlink is not supported. MC_MUSA_IPC_OPEN_DEVICE=current is the safe default; select metadata only when peers satisfy the logical mapping contract above. Python bindings that register the default wildcard location (*) resolve the owning MUSA device during registration; an older peer that still advertises * falls back to the current-device open path.

In auto mode, batches whose copies are at least 1 MiB use muMemoryTransferBatchAsync; set MC_MUSA_TRANSFER_BATCH_MIN_BYTES to tune the threshold, MC_MUSA_COPY_API=transfer_batch to force the API, or MC_MUSA_COPY_API=default to use the CUDA-compatible per-slice path.

HIP Transport (hip)#

Description: AMD ROCm/HIP transport for GPU communication using IPC handles or Shareable handles.

Use When:

  • Working with AMD GPUs

  • Need intra-node GPU communication on AMD hardware

Requirements:

  • AMD ROCm/HIP runtime

  • AMD GPUs

Barex Transport (barex)#

Description: Bare-metal RDMA extension protocol for specialized RDMA configurations.

Use When:

  • Advanced RDMA features are required

  • Custom RDMA configurations

Requirements:

  • RDMA-capable hardware

  • Specialized configuration

CXL Transport (cxl)#

Description: Compute Express Link for memory pooling and sharing across devices.

Use When:

  • CXL memory pooling is available

  • Memory disaggregation is needed

Requirements:

  • CXL-capable hardware

SHM Transport (shm)#

Description: Same-host DRAM copies over POSIX shared memory. Classic Transfer Engine maps the peer’s named shm object, relocates the peer virtual address into the local mapping, and memcpys. This is a first-class transport like HIP, not a replacement for CxlTransport (DAX offset addressing).

Use When:

  • Two processes on the same machine exchange DRAM buffers

  • You want to avoid RDMA/TCP loopback for that path

Requirements:

  • Linux POSIX shm (/dev/shm)

  • Buffers allocated with TransferEngine::allocateSharedMemory (ordinary malloc cannot be exported)

  • Runtime opt-in: MC_FORCE_SHM=1, or installTransport("shm"). With -DENABLE_MULTI_PROTOCOL=ON this adds SHM next to RDMA/TCP (rdma,shm / tcp,shm); without it, SHM is the only transport.

  • Same-host SHM and cross-host RDMA/TCP in one engine: build with -DENABLE_MULTI_PROTOCOL=ON (segment protocol becomes rdma,shm or tcp,shm)

Limitations:

  • Same host only. Without ENABLE_MULTI_PROTOCOL, MC_FORCE_SHM=1 (or installTransport("shm") after another transport) sets segment.protocol to shm and replaces RDMA/TCP routing; installTransport("shm") logs a WARNING when it overwrites a non-empty protocol. Coexistence needs -DENABLE_MULTI_PROTOCOL=ON.

  • registerLocalMemory must use the pointer from allocateSharedMemory (a shorter prefix is allowed). A sub-range or overflowing range returns an error instead of silently skipping. Ordinary malloc is still skipped so TCP/RDMA can register it.

  • Same-UID only: objects are created 0600 with POSIX names /mooncake_<pid>_xxxxxxxx. Creator and consumer must share a user; a hostname match does not imply a shared /dev/shm (for example Kubernetes hostNetwork pods).

  • Crash or SIGKILL can leave objects in /dev/shm until reboot; there is no automatic reaper.

  • After freeSharedMemory + allocateSharedMemory, a peer that still has a cached mapping probes the POSIX name before memcpy. An unlinked object is dropped and the segment descriptor is refetched once; a changed virtual address still requires the initiator to read the new BufferDesc.addr (relocate cannot guess a new offset). Background refresh remains optional via MC_TE_METADATA_REFRESH_INTERVAL_SECONDS.

  • Relocate caches at most 32 mmap’d peer objects per target. An in-flight copy pins its mapping so prune/cap cannot munmap it until memcpy returns; the cache may briefly exceed 32 while pins are held.

  • Default off because the path is not NUMA-aware

  • Mooncake Store segments are not shm-backed until a follow-up allocator change

Ascend Transport (ascend)#

Description: Huawei Ascend NPU communication using HCCL (Huawei Collective Communication Library) or direct transport.

Use When:

  • Working with Huawei Ascend NPUs

  • Distributed inference on Ascend hardware

Requirements:

  • Huawei Ascend NPU hardware

  • HCCL runtime

Documentation:

TPU Transport (tpu) — Experimental#

Description: Google TPU support in the TENT runtime. Because TPU HBM is not directly addressable by the NIC, transfers touching TPU memory are staged through host DRAM: the HBM ↔ host-DRAM hop is performed by a PJRT device-copy adapter, and the host ↔ host hop is carried by an existing transport (RDMA/TCP). The two stages are chained automatically by the TENT staging pipeline (ProxyManager), so no separate networked TPU transport is required.

Status: Experimental. The C++/TENT data path is gated behind -DUSE_TPU=ON (OFF by default). A serving-framework (JAX / PyTorch-XLA) integration layer is planned as a follow-up.

Use When:

  • Disaggregated prefill/decode serving on TPU hosts

  • KV-cache transfer between TPU nodes over RDMA/TCP

Requirements:

  • Built with -DUSE_TPU=ON -DUSE_TENT=ON

  • A PJRT device-copy adapter shared library exposing the mc_tpu_pjrt_* C ABI (see tpu_pjrt_abi.h). The adapter is resolved at runtime via dlopen; its path defaults to libmooncake_tpu_pjrt.so and can be overridden with the MC_TPU_PJRT_LIB environment variable. No PJRT/TPU SDK is required at build time.

  • An RDMA (or TCP) transport enabled for the host ↔ host hop.

Design notes:

  • TPU memory is reported as a distinct memory type (tpu:N locations); the staging policy routes the local HBM ↔ host copy to the TPU device-copy transport and the cross-node hop to RDMA/TCP.

  • DMA-mapped (pinned) staging buffers for true async device DMA are a planned performance follow-up.

MPComm Transport (mpcomm)#

Description: UCL-MPComm (Unified Communication Library - Memory Pool Communication) is an RDMA library for heterogeneous memory pooling, integrated as a TENT transport. It drives multiple RDMA NICs concurrently with two-level load balancing (across NICs, and across QPs within a NIC) and NUMA-aware worker placement, exposing one-sided put/get primitives. It is shortened to MPComm below.

Status: TENT only. There is no MPComm backend on the legacy Transfer Engine transport path, so it cannot be selected through MOONCAKE_PROTOCOL or transfer_engine_bench --protocol=.

Use When:

  • The host has several RDMA NICs and you want them saturated by a single transfer stream

  • Multi-NUMA hosts where NIC-to-NUMA affinity matters

Requirements:

  • Built with -DUSE_TENT=ON -DUSE_MPCOMM=ON -DMPCOMM_ROOT=<prefix>

  • MPComm installed, providing include/mpcomm.h and lib/libmpcomm.so (Tencent/UCL-MPComm)

  • libmpcomm.so reachable by the dynamic linker at run time

Enable:

{ "transports": { "mpcomm": { "enable": true } } }

See MPComm Transport for the full guide, including selection via transport policy, tuning environment variables, and troubleshooting.

FlagOS FlagCX Transport (flagcx)#

Description: FlagCX is the unified communication library in the FlagOS ecosystem for multi-vendor and cross-vendor deployments. Mooncake integrates the FlagCX P2P Engine as a classic Transfer Engine transport, allowing the existing Mooncake transfer workflow to use the accelerator and network backends provided by the local FlagCX build.

Use When:

  • Deploying Mooncake on a platform supported by FlagCX

  • Using FlagCX’s P2P Engine for accelerator memory transfers

  • Building a cross-vendor deployment around the FlagOS communication stack

Build Requirements:

cmake -S . -B build \
  -DUSE_FLAGCX=ON \
  -DFLAGCX_HOME=/path/to/FlagCX/build
cmake --build build -j

FLAGCX_HOME must contain include/flagcx_p2p.h and either lib/libflagcx.so or lib64/libflagcx.so. If it is omitted, Mooncake checks $FLAGCX_HOME and then $HOME/FlagCX/build.

Configuration:

engine.initialize(
    hostname="node1",
    metadata_server="P2PHANDSHAKE",
    protocol="flagcx",
    device_name=""
)

# Select the interface used for FlagCX bootstrap and endpoint advertisement.
export FLAGCX_SOCKET_IFNAME="eth0"

Current Scope:

  • Available through the classic Transfer Engine; it is not a TENT transport

  • Must be built from source with USE_FLAGCX=ON

  • Should be selected as the standalone flagcx protocol, not as part of a multi-protocol string

  • Buffers should be registered before the first transfer to a peer and remain registered while that peer connection is active

See FlagOS FlagCX Transport for dependency, build, benchmark, runtime configuration, and troubleshooting details.

Configuration Examples#

Configuration File (JSON)#

TCP Configuration:

{
    "local_hostname": "localhost",
    "metadata_server": "localhost:8080",
    "protocol": "tcp",
    "device_name": "",
    "master_server_address": "localhost:8081"
}

RDMA Configuration:

{
    "local_hostname": "node1",
    "metadata_server": "etcd://10.0.0.1:2379",
    "global_segment_size": "3GB",
    "local_buffer_size": "1GB",
    "protocol": "rdma",
    "device_name": "mlx5_0",
    "master_server_address": "10.0.0.1:8081"
}

Environment Variables#

# TCP (Default)
export MOONCAKE_PROTOCOL="tcp"

# RDMA with specific device
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="mlx5_0"

# RDMA with auto-discovery
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="auto-discovery"

# Other configuration
export MOONCAKE_MASTER="10.0.0.1:50051"
export MOONCAKE_TE_META_DATA_SERVER="P2PHANDSHAKE"
export MOONCAKE_LOCAL_HOSTNAME="node1"

Choosing the Right Protocol#

Scenario

Recommended Protocol

Notes

Development/Testing

tcp

Simple setup, no special hardware

Production Inference

rdma

Best performance and latency

AWS Cloud (EFA instances)

efa

High performance on p5e, p6-b200, p4d, etc.

Cloud Environments

tcp or rdma (if available)

Check cloud provider support

Multi-tier Storage

rdma + nvmeof

Combine protocols for different layers

AMD GPU Clusters

rdma + hip

Use HIP for local GPU communication

Cambricon MLU Clusters

rdma

Build with -DUSE_MLU=ON; MLU uses the normal RDMA protocol

Ascend NPU Clusters

rdma + ascend

Use Ascend for NPU-specific operations

Multi-vendor or cross-vendor clusters

flagcx

Build with -DUSE_FLAGCX=ON; transfers use the FlagCX P2P Engine over RDMA-capable NICs

Troubleshooting#

RDMA Connection Issues#

  1. Check RDMA devices:

    ibv_devices
ibv_devinfo

  2. Verify network connectivity:

    # Test RDMA connectivity (requires rdma-core tools)
rping -s  # On server
rping -c -a <server_ip> -v  # On client

  3. Check permissions:

    • RDMA may require elevated permissions

    • Run with sudo if necessary

    • Configure proper udev rules for non-root access

  4. Firewall configuration:

    • Ensure RDMA ports are not blocked

    • Check InfiniBand subnet manager is running

Protocol Selection#

If a protocol fails to initialize:

  1. Verify hardware support

  2. Check that required drivers are installed

  3. Ensure compile-time flags are set correctly (for C++ protocols)

  4. Fall back to TCP for basic functionality

See Also#

Contents