FlagOS FlagCX Transport

FlagOS FlagCX Transport#

Overview#

FlagCX is the unified communication library in the FlagOS ecosystem for multi-vendor and cross-vendor deployments. Mooncake’s FlagCX transport connects the classic Transfer Engine to the FlagCX P2P Engine. The integration keeps Mooncake’s existing segment, memory-registration, batch, and completion APIs and maps them to FlagCX connections and P2P read/write requests.

The protocol name used by Mooncake configuration and APIs is flagcx. flagos is not a protocol name.

Classic Transfer Engine only. The FlagCX transport is implemented under mooncake-transfer-engine/src/transport/ and is not a TENT transport. Build with USE_FLAGCX=ON, select --backend=classic in tebench, and use flagcx as a standalone protocol.

Code Structure#

Unit

Responsibility

flagcx_transport.{h,cpp}

Creates the P2P Engine, advertises its endpoint, registers memory, opens peer connections, submits transfers, and polls completion

flagcx_transport_internal.h

Overflow-safe registered-range checks and descriptor-length conversion

mooncake-common/common.cmake

Locates the external FlagCX headers and library and defines FlagCX::flagcx

multi_transport.cpp

Instantiates FlagCxTransport when the protocol name is flagcx

Transfer Pipeline#

  1. Initialization: Mooncake creates a FlagCX P2P Engine and starts its RPC server.

  2. Advertisement: the transport obtains the FlagCX endpoint and publishes it in the local Mooncake segment descriptor.

  3. Registration: registerLocalMemory() registers each local range with FlagCX and publishes the corresponding Mooncake buffer descriptor.

  4. Connection: the first request to a remote segment reads its published FlagCX endpoint and opens a connection. FlagCX caches the connection for later requests.

  5. Submission: Mooncake groups requests with the same target and operation, creates remote descriptors, and calls the FlagCX vector read or write API.

  6. Completion: a Mooncake worker polls the returned FlagCX transfer ID and updates the original Mooncake task when the transfer becomes terminal.

Dependencies#

Use compatible FlagCX revisions on all nodes. Each node may build the accelerator backend that matches its local platform. The prefix passed as FLAGCX_HOME must contain:

  • include/flagcx_p2p.h

  • lib/libflagcx.so or lib64/libflagcx.so

  • The accelerator runtime, communication library, and network dependencies required by the chosen FlagCX backend

Choose the FlagCX backend that matches each node’s local platform; for example, USE_NVIDIA=1, USE_METAX=1, or USE_MUSA=1. See the FlagCX getting-started guide for the current backend list and platform prerequisites.

Build and Compile#

1. Build FlagCX#

git clone https://github.com/flagos-ai/FlagCX.git
cd FlagCX
git submodule update --init --recursive

# Replace USE_NVIDIA with the backend for the local platform.
make USE_NVIDIA=1 -j$(nproc)

The default build output is suitable for an in-tree Mooncake build:

FlagCX/build/include/flagcx_p2p.h
FlagCX/build/lib/libflagcx.so

Alternatively, install FlagCX under a dedicated prefix. Installing under /opt normally requires root privileges:

sudo make PREFIX=/opt/flagcx install

2. Build Mooncake#

From the Mooncake repository root:

cmake -S . -B build \
  -DUSE_FLAGCX=ON \
  -DFLAGCX_HOME=/path/to/FlagCX/build
cmake --build build -j$(nproc)

For the installed layout above, use -DFLAGCX_HOME=/opt/flagcx. If FLAGCX_HOME is not passed to CMake, Mooncake first checks the environment variable of the same name, then defaults to $HOME/FlagCX/build.

Configuration fails early if either flagcx_p2p.h or the FlagCX shared library cannot be found. A successful configuration includes a message similar to:

FlagCX transport enabled, include=/path/to/FlagCX/build/include, library=/path/to/FlagCX/build/lib/libflagcx.so

Make the shared library and its backend dependencies visible to the dynamic linker when they are not installed in a system search path:

export FLAGCX_HOME=/path/to/FlagCX/build
export LD_LIBRARY_PATH="$FLAGCX_HOME/lib:${LD_LIBRARY_PATH:-}"

If Mooncake also needs to allocate or identify device memory, enable its matching hardware option, such as USE_CUDA, USE_MACA, USE_MUSA, USE_HIP, USE_COREX, USE_HYGON, or USE_MLU. See the build guide for the options and SDK requirements.

Enabling the Transport#

Python API#

A Mooncake Python extension built from source with USE_FLAGCX=ON accepts flagcx through the existing initialization API:

engine.initialize(
    hostname="node1",
    metadata_server="P2PHANDSHAKE",
    protocol="flagcx",
    device_name="",
)

The Python binding disables automatic RDMA transport discovery for this protocol and installs the FlagCX transport explicitly. No new Python API is required.

C++ API#

Applications using the classic C++ Transfer Engine can install the transport by name after engine initialization. Keep automatic transport discovery disabled so that flagcx remains the standalone transport:

mooncake::TransferEngine engine(false);
if (engine.init("P2PHANDSHAKE", "node1") != 0) {
    return -1;
}

auto* transport = engine.installTransport("flagcx", nullptr);
if (transport == nullptr) {
    return -1;
}

Continue to use the standard Transfer Engine APIs for memory registration, segment discovery, batch submission, and status queries.

Run and Test#

Connectivity Test with tebench#

tebench can validate registration, endpoint exchange, connection setup, read/write transfers, and completion. Both peers must use the classic backend and the flagcx transport. An empty --target_seg_name starts the target; setting it starts the initiator.

Choose a network interface reachable by the other peer on both hosts:

export FLAGCX_SOCKET_IFNAME=eth0

Start the target first:

./build/mooncake-transfer-engine/benchmark/tebench \
  --backend=classic \
  --xport_type=flagcx \
  --metadata_type=p2p \
  --seg_type=DRAM \
  --total_buffer_size=1073741824

The target prints a command containing its segment name. Use that value on the initiator:

./build/mooncake-transfer-engine/benchmark/tebench \
  --backend=classic \
  --xport_type=flagcx \
  --metadata_type=p2p \
  --target_seg_name=<target-segment-name> \
  --seg_type=DRAM \
  --total_buffer_size=1073741824 \
  --op_type=read \
  --start_block_size=4096 \
  --max_block_size=67108864 \
  --start_batch_size=1 \
  --max_batch_size=1 \
  --start_num_threads=1 \
  --max_num_threads=1

Use --op_type=write to test the opposite direction. The classic tebench VRAM allocator currently supports CUDA builds. For that path, add -DUSE_CUDA=ON when building Mooncake and run with --seg_type=VRAM. Other accelerator backends should validate device buffers through their application integration until tebench provides a matching allocator.

Useful initialization messages include:

FlagCxTransport: engine up, endpoint=...
FlagCxTransport: install OK (direct submit)
tebench: FlagCX transport installed

Hardware-Free Checks#

The internal range and descriptor-boundary test does not require FlagCX hardware:

ctest --test-dir build -R '^flagcx_transport_internal_test$' --output-on-failure

The Python source integration check verifies that the binding recognizes and installs flagcx:

python3 -m unittest mooncake-wheel/tests/test_transfer_engine_flagcx_source.py

Runtime Configuration#

FlagCX owns network-device selection and most transport tuning. The variables most relevant to the Mooncake integration are:

Variable

Purpose

FLAGCX_SOCKET_IFNAME

Selects the interface used for socket bootstrap and endpoint advertisement; it does not select the RDMA HCA

FLAGCX_IB_HCA

Selects the InfiniBand/RoCE HCA or HCA set used by FlagCX

FLAGCX_P2P_TRANSPORT=accl

Selects the optional ACCL P2P implementation when the FlagCX build includes it; both peers must set it

LD_LIBRARY_PATH

Makes libflagcx.so and non-system backend libraries visible at run time

Set FLAGCX_SOCKET_IFNAME and FLAGCX_IB_HCA for each node’s local interfaces and devices. For interface filtering syntax, GID selection, retry settings, and backend-specific tuning, see the FlagCX environment-variable reference.

Important Notes#

Memory Registration Lifecycle#

The FlagCX connection handshake exchanges the peer’s registered-memory table, and the P2P Engine caches established connections. Use the following lifecycle for the current Mooncake integration:

  1. Register every buffer that a peer may access.

  2. Establish or use the peer connection only after registration is complete.

  3. Keep those buffers registered while the connection or its transfers are active.

  4. Stop submitting work and allow outstanding work to finish before unregistering or freeing the buffers.

Registering a new remote buffer after a peer has already cached its connection does not refresh the connection’s memory table. Reconnect both processes before using a changed registration set.

Deployment Scope#

  • Use the literal protocol name flagcx; do not use flagos.

  • Use flagcx as a standalone protocol. Multi-protocol selection and routing are outside the current integration scope.

  • The peers may use different accelerator backends, but they must use compatible FlagCX revisions and the same P2P network transport: default IBRC on both peers, or ACCL on both peers.

  • The transport relies on FlagCX’s endpoint exchange and completion behavior; it does not add a separate Mooncake cancellation API.

Security and Trust#

Treat the FlagCX endpoint like other RDMA-capable transport endpoints. Only expose it to trusted peers, restrict the selected interface with network policy or host firewall rules, and register only the memory required by the application. Mooncake segment metadata exposes the endpoint and buffer addresses, while the FlagCX handshake exchanges the registered-region table. Do not expose either channel to untrusted peers.

Troubleshooting#

CMake Cannot Find FlagCX#

USE_FLAGCX=ON but flagcx_p2p.h was not found

Check that FLAGCX_HOME/include/flagcx_p2p.h exists. If the header remains only in the FlagCX source tree, finish the FlagCX build so its public headers are copied to build/include, or run its install target and point FLAGCX_HOME at that prefix.

USE_FLAGCX=ON but the FlagCX library was not found

Check for FLAGCX_HOME/lib/libflagcx.so or FLAGCX_HOME/lib64/libflagcx.so and confirm that the same prefix is passed during CMake configuration.

The Executable Cannot Load libflagcx.so#

If startup reports that libflagcx.so or a platform backend library is missing, add their library directories to LD_LIBRARY_PATH or install them in a configured system loader path. This is a run-time linker issue, not a Mooncake protocol-selection issue.

The FlagCX Engine Advertises the Wrong Interface#

Set FLAGCX_SOCKET_IFNAME before starting each process. Use an interface whose advertised address is reachable from the peer. If InfiniBand/RoCE device selection is also ambiguous, set FLAGCX_IB_HCA to the appropriate local HCA set on each side.

installTransport(flagcx) Fails#

Confirm all of the following:

  • Mooncake was configured with USE_FLAGCX=ON.

  • The FlagCX shared library and its platform dependencies load successfully.

  • FLAGCX_SOCKET_IFNAME resolves to a usable local address.

  • The FlagCX log does not report a P2P Engine or RPC-server initialization error.

Connection or Remote-Descriptor Creation Fails#

Verify that the target process is still running, the endpoint printed in the target log is reachable, and both peers selected the same FlagCX P2P network transport. Check host firewall rules on the connection path. If the target’s registrations changed after the connection was first used, restart both processes and register all buffers before reconnecting.

Contents