Transfer Engine C++ API Reference

Transfer Engine C++ API Reference#

Overview#

This page summarizes the C++ APIs in mooncake-transfer-engine/include/transfer_engine.h. It follows the same narrative style as the design doc, while covering the full set of public C++ APIs.

For conceptual background, see Transfer Engine.

Core APIs vs Advanced APIs

  • Core APIs form the minimal path to move data: initialize the engine, register memory, open segments, submit transfers, and query status.

  • Advanced APIs are optional; they help with transport control, notifications, metadata/cache maintenance, topology discovery, and debugging.

Core APIs#

Core Usage Path (C++)#

Most integrations follow a short, repeatable path: initialize the engine, register local memory, open a target segment, submit a batch transfer, poll status, and finally free resources (batch, segment, memory). A minimal example is shown below.

#include "transfer_engine.h"

using mooncake::TransferEngine;
using mooncake::TransferRequest;
using mooncake::TransferStatus;

TransferEngine engine(true);
engine.init("etcd://127.0.0.1:2379", "node0");

void *local_addr = /* local buffer address */;
size_t length = /* bytes to transfer */;
uint64_t remote_addr = /* remote address or file offset */;

engine.registerLocalMemory(local_addr, length, "cpu:0", true);

auto segment = engine.openSegment("node1");
auto batch = engine.allocateBatchID(1);

TransferRequest req{};
req.opcode = TransferRequest::WRITE;
req.source = local_addr;
req.target_id = segment;
req.target_offset = remote_addr;
req.length = length;

engine.submitTransfer(batch, {req});

TransferStatus status{};
engine.getTransferStatus(batch, 0, status);

engine.freeBatchID(batch);
engine.closeSegment(segment);
engine.unregisterLocalMemory(local_addr);

Constructors#

TransferEngine(bool auto_discover = false);
TransferEngine(bool auto_discover, const std::vector<std::string>& filter);

Constructs a Transfer Engine instance. auto_discover enables topology discovery; filter constrains discovery or device selection with a whitelist.

Data Transfer#

TransferEngine::TransferRequest#

The core API provided by Mooncake Transfer Engine is submitting a group of asynchronous TransferRequest tasks through the submitTransfer interface, and querying their status through the getTransferStatus interface. Each TransferRequest specifies reading or writing a continuous data space of length starting from the local starting address source, to the position starting at target_offset in the segment corresponding to target_id.

The TransferRequest structure is defined as follows:

struct TransferRequest
{
    enum OpCode { READ, WRITE };
    enum Priority { PRIO_HIGH = 0, PRIO_MEDIUM = 1, PRIO_LOW = 2 };
    static constexpr uint64_t kNoTaskGroup = 0;

    OpCode opcode;
    void *source;
    SegmentID target_id; // The ID of the target segment, which may correspond to local or remote DRAM/VRAM/NVMeof, with the specific routing logic hidden
    uint64_t target_offset;
    size_t length;
    int advise_retry_cnt = 0;
    int transport_hint = 0;
    uint64_t task_group_id = kNoTaskGroup;
    int priority = PRIO_MEDIUM;
    int nic_hint = -1;
};

  • opcode takes the values READ or WRITE. READ indicates that data is copied from the target address indicated by <target_id, target_offset> to the local starting address source; WRITE indicates that data is copied from source to the address indicated by <target_id, target_offset>.

  • source represents the DRAM/VRAM buffer managed by the current TransferEngine, which must have been registered in advance by the registerLocalMemory interface.

  • target_id represents the segment ID of the transfer target. The segment ID is obtained using the openSegment interface. Segments are divided into the following types:

    • RAM space type, covering DRAM/VRAM. As mentioned earlier, there is only one segment under the same process (or TransferEngine instance), which contains various types of Buffers (DRAM/VRAM). In this case, the segment name passed to the openSegment interface is equivalent to the server hostname. target_offset is the virtual address of the target server.

    • NVMeOF space type, where each file corresponds to a segment. In this case, the segment name passed to the openSegment interface is equivalent to the unique identifier of the file. target_offset is the offset of the target file.

  • length represents the amount of data transferred. TransferEngine may further split this into multiple read/write requests internally.

  • nic_hint optionally prefers a local RNIC by its index in getLocalTopology()->getHcaList() in the classic RDMA transport. The default -1, an out-of-range index, or an unavailable device falls back to normal topology-based selection. The hint affects initial submission only and is not a hard pin for an in-flight slice.

TransferEngine::allocateBatchID#

BatchID allocateBatchID(size_t batch_size);

Allocates a BatchID. A maximum of batch_size TransferRequests can be submitted under the same BatchID.

  • batch_size: The maximum number of TransferRequests that can be submitted under the same BatchID;

  • Return value: If successful, returns a BatchID; on failure, returns an invalid handle (for example INVALID_BATCH_ID).

TransferEngine::submitTransfer#

Status submitTransfer(BatchID batch_id,
                      const std::vector<TransferRequest> &entries);

Submits new TransferRequest tasks to batch_id. The task is asynchronously submitted to the background thread pool. The total number of entries accumulated under the same batch_id should not exceed the batch_size defined at creation.

  • batch_id: The BatchID it belongs to;

  • entries: Array of TransferRequest;

  • Return value: If successful, returns an OK status; otherwise, returns a non-OK status.

TransferEngine::getTransferStatus#

enum TransferStatusEnum
{
  WAITING,   // In the transfer phase
  PENDING,   // Not supported
  INVALID,   // Invalid parameters
  CANCELED,  // Not supported
  COMPLETED, // Transfer completed
  TIMEOUT,   // Not supported
  FAILED     // Transfer failed even after retries
};
struct TransferStatus {
  TransferStatusEnum s;
  size_t transferred_bytes; // How much data has been successfully transferred (not necessarily an accurate value, but it is a lower bound)
};
Status getTransferStatus(BatchID batch_id, size_t task_id, TransferStatus &status)

Obtains the running status of the TransferRequest with task_id in batch_id.

  • batch_id: The BatchID it belongs to;

  • task_id: The sequence number of the TransferRequest to query;

  • status: Output Transfer status;

  • Return value: If successful, returns an OK status; otherwise, returns a non-OK status.

TransferEngine::getBatchTransferStatus#

Status getBatchTransferStatus(BatchID batch_id, TransferStatus& status);

Obtains the aggregated status of the batch and the total transferred bytes.

While the batch is still WAITING, RDMA-only batches that use the completion-counter fast path report status.transferred_bytes = 0. That is still a valid lower bound; the accurate total is filled in when the batch reaches COMPLETED. TCP, NVLink, mixed batches, failures, and MC_SLICE_TIMEOUT > 0 keep walking every task and still accumulate bytes while in flight.

  • batch_id: The BatchID it belongs to;

  • status: Output Transfer status;

  • Return value: If successful, returns an OK status; otherwise, returns a non-OK status.

TransferEngine::freeBatchID#

Status freeBatchID(BatchID batch_id);

Recycles BatchID, and subsequent operations on submitTransfer and getTransferStatus are undefined. If there are still TransferRequests pending completion in the BatchID, the operation is refused.

  • batch_id: The BatchID it belongs to;

  • Return value: If successful, returns an OK status; otherwise, returns a non-OK status.

Space Registration#

For the RDMA transfer process, the source pointer TransferRequest::source must be registered in advance as an RDMA readable/writable Memory Region space, that is, included as part of the RAM Segment of the current process. Therefore, the following functions are needed:

TransferEngine::registerLocalMemory#

int registerLocalMemory(void *addr,
                        size_t length,
                        const std::string& location = kWildcardLocation,
                        bool remote_accessible = true,
                        bool update_metadata = true);

Registers a space starting at address addr with a length of length on the local DRAM/VRAM.

  • addr: The starting address of the registration space;

  • length: The length of the registration space;

  • location: The device corresponding to this memory segment, such as cuda:0 indicating the GPU device, cpu:0 indicating the CPU socket, by matching with the network card priority order table (see installTransport), the preferred network card is identified. You can also use *, Transfer Engine will try to automatically recognize the device corresponding to addr, if it fails to recognize the device, it will print a WARNING level log and use all network cards, no preferred network cards.

  • remote_accessible: Indicates whether this memory can be accessed by remote nodes. For RDMA, false registers the buffer for local transfer use only: remote read/write permissions are not granted and no rkey is published.

  • update_metadata: Whether to publish the registration to the metadata service.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

TransferEngine::unregisterLocalMemory#

int unregisterLocalMemory(void *addr, bool update_metadata = true);

Unregisters the region.

  • addr: The starting address of the registration space;

  • update_metadata: Whether to publish the unregistration to the metadata service.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

TransferEngine::allocateSharedMemory#

void* allocateSharedMemory(size_t length);
void* allocateSharedMemory(size_t length, const SharedMemoryOptions& opt);
int freeSharedMemory(void* addr);

Allocates a shared-memory region that ShmTransport can export to same-host peers. Ordinary malloc cannot be advertised this way. Requires ShmTransport (MC_FORCE_SHM=1 or installTransport("shm")). Call registerLocalMemory on the returned pointer before remote access; a sub-range inside that allocation, or a length larger than the allocation, returns an error. Classic Transfer Engine only; TENT returns nullptr / ERR_NOT_IMPLEMENTED. Combining SHM with RDMA/TCP on one engine requires -DENABLE_MULTI_PROTOCOL=ON. Without that flag, installTransport("shm") logs a WARNING if it replaces an existing rdma/tcp segment protocol.

The no-options overload (and SharedMemoryOptions{}) creates a POSIX shm object in /dev/shm named /mooncake_<pid>_xxxxxxxx, mode 0600 (same UID). With opt.use_hugepage and hugepage_size of 2MB, 512MB, or 1GB, TE creates a file on a matching hugetlbfs mount (opt.hugetlbfs_path, or /dev/hugepages, /dev/hugepages-512M, /dev/hugepages-1G) named <mount>/mooncake_<pid>_xxxxxxxx. Hugepage length must already be a multiple of that page size; TE does not round up, and it does not fall back to tmpfs if the mount is missing or the page size mismatches. freeSharedMemory only accepts pointers returned by allocateSharedMemory; a malloc pointer is rejected without unregistering other transports.

freeSharedMemory and ShmTransport’s destructor unlink the object. Crash or SIGKILL skips that. POSIX leftovers sit in /dev/shm until reboot or manual shm_unlink. Hugetlbfs leftovers stay on the mount and keep hugepages reserved until the file is unlinked or the node reboots — worse than tmpfs leftovers. There is no automatic startup reaper: several processes share a mount, so deleting every mooncake_* on start would remove live peers’ files. After a crash, remove only files whose encoded pid is gone, for example rm /dev/hugepages/mooncake_<dead-pid>_* (and the 512MB/1GB mounts if used). Peers detect an unlinked object on the next transfer, drop the orphaned mmap, and refetch metadata once; if realloc changes the virtual address, the initiator must use the new BufferDesc.addr.

TransferEngine::registerLocalMemoryBatch#

int registerLocalMemoryBatch(const std::vector<BufferEntry>& buffer_list,
                             const std::string& location);

Registers multiple buffers in one call to reduce registration overhead.

  • buffer_list: A list of {addr, length} entries.

  • location: The device corresponding to these buffers.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

TransferEngine::unregisterLocalMemoryBatch#

int unregisterLocalMemoryBatch(const std::vector<void*>& addr_list);

Unregisters multiple buffers in one call.

  • addr_list: A list of buffer start addresses.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

Segment Management and Metadata Format#

TransferEngine provides the openSegment function, which obtains a SegmentHandle for subsequent Transport transfers.

SegmentHandle openSegment(const std::string& segment_name);

  • segment_name: The unique identifier of the segment. For RAM Segment, this needs to be consistent with the server_name filled in by the peer process when initializing the TransferEngine object.

  • Return value: If successful, returns the corresponding SegmentHandle; otherwise, returns an invalid handle.

int closeSegment(SegmentHandle segment_id);

  • segment_id: The unique identifier of the segment.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

TransferEngine::getSegmentBuffers#

struct SegmentBufferInfo {
    uint64_t addr;
    uint64_t length;
    std::string location;
};

int getSegmentBuffers(SegmentHandle handle,
                      std::vector<SegmentBufferInfo>& buffers);

Returns a snapshot of the memory buffers published in a segment, using the selected classic or TENT backend. This supports buffer inspection without accessing getMetadata(), which is unavailable under TENT.

  • handle: A segment handle returned by openSegment().

  • buffers: Replaces any existing contents with buffer addresses, lengths in bytes, and backend-reported location labels. Classic copies BufferDesc::name (some transports, such as TCP, use a server name); TENT copies the native location. Entries preserve the backend descriptor’s order, which is not guaranteed to be sorted by address. TENT internal buffers are excluded.

  • Return value: 0 on success, including a segment with no memory buffers; ERR_METADATA if the descriptor cannot be retrieved; ERR_NOT_IMPLEMENTED for file segments. On error, buffers is empty.

  • The call may fetch remote metadata and block. It does not close the handle, register memory, or keep the reported buffers alive; the caller must coordinate buffer lifetime with the owning peer.

TransferEngine::removeLocalSegment#

int removeLocalSegment(const std::string& segment_name);

Removes local segment metadata entries.

  • segment_name: The local segment name to remove.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

TransferEngine::CheckSegmentStatus#

Status CheckSegmentStatus(SegmentID sid);

Checks whether a segment is reachable and valid.

  • sid: The segment identifier.

  • Return value: If successful, returns an OK status; otherwise, returns a non-OK status.

Metadata Format 
// Used to find the communicable address and exposed rpc port based on server_name.
// Created: when calling TransferEngine::init().
// Deleted: when TransferEngine is destructed.
Key = mooncake/rpc_meta/[server_name]
Value = {
    'ip_or_host_name': 'node01'
    'rpc_port': 12345
}

// For segments, the key naming method of mooncake/[proto]/[segment_name] is used, and the segment name can use the Server Name.
// A segment corresponds to a machine, and a buffer corresponds to different segments of memory or different files or different disks on the machine. Different buffers of the same segment are in the same fault domain.

// RAM Segment, used by RDMA Transport to obtain transfer information.
// Created: command line tool register.py, at this time buffers are empty, only fill in the information that can be known in advance.
// Modified: TransferEngine at runtime through register / unregister to add or delete Buffer.
Key = mooncake/ram/[segment_name]
Value = {
    'server_name': server_name,
    'protocol': rdma,
    'devices': [
        { 'name': 'mlx5_2', 'lid': 17, 'gid': 'fe:00:...' },
        { 'name': 'mlx5_3', 'lid': 22, 'gid': 'fe:00:...' }
    ],
    'priority_matrix': {
        "cpu:0": [["mlx5_2"], ["mlx5_3"]],
        "cpu:1": [["mlx5_3"], ["mlx5_2"]],
        "cuda:0": [["mlx5_2"], ["mlx5_3"]],
    },
    'buffers': [
        {
            'name': 'cpu:0',
            'addr': 0x7fa16bdf5000,
            'length': 1073741824,
            'rkey': [1fe000, 1fdf00, ...], // The length is the same as the number of elements in the 'devices' field
        },
    ],
}

// Created: command line tool register.py, determine the file path that can be mounted.
// Modified: command line tool mount.py, add a mapping of the machine mounting the file to the file path on the mounting machine to the buffers.local_path_map.
Key = mooncake/nvmeof/[segment_name]
Value = {
    'server_name': server_name,
    'protocol': nvmeof,
    'buffers':[{
        'length': 1073741824,
        'file_path': "/mnt/nvme0" // The file path on this machine
        'local_path_map': {
            "node01": "/mnt/transfer_engine/node01/nvme0", // The machine mounting the file -> The file path on the mounting machine
            ...
        },
     },
     {
        'length': 1073741824,
        'file_path': "/mnt/nvme1",
        'local_path_map': {
            "node02": "/mnt/transfer_engine/node02/nvme1",
            ...
        },
     }
    ]
}

HTTP Metadata Server#

The HTTP server should implement three following RESTful APIs, while the metadata server configured to http://host:port/metadata as an example:

  1. GET /metadata?key=$KEY: Get the metadata corresponding to $KEY.

  2. PUT /metadata?key=$KEY: Update the metadata corresponding to $KEY to the value of the request body.

  3. DELETE /metadata?key=$KEY: Delete the metadata corresponding to $KEY.

For specific implementation, refer to the demo service implemented in Golang at mooncake-transfer-engine/example/http-metadata-server.

Initialization#

TransferEngine needs to be initialized by calling the init method before further actions:

TransferEngine();

int init(const std::string &metadata_conn_string,
         const std::string &local_server_name);

There is also an extended form to override RPC binding:

int init(const std::string &metadata_conn_string,
         const std::string &local_server_name,
         const std::string &ip_or_host_name,
         uint64_t rpc_port);

  • metadata_conn_string: Connecting string of metadata storage servers, i.e., the IP address/hostname of etcd/redis or the URI of the http service. The general form is [proto]://[hostname:port]. For example, the following metadata server addresses are legal:

    • Using etcd as a metadata storage service: "10.0.0.1:2379" or "etcd://10.0.0.1:2379".

    • Using redis as a metadata storage service: "redis://10.0.0.1:6379"

    • Using http as a metadata storage service: "http://10.0.0.1:8080/metadata"

  • local_server_name: The local server name, ensuring uniqueness within the cluster. It also serves as the name of the RAM Segment that other nodes refer to the current instance (i.e., Segment Name).

  • ip_or_host_name: Optional explicit bind address or hostname for the RPC service.

  • rpc_port: Optional explicit RPC port.

  ~TransferEngine();

Reclaims all allocated resources and also deletes the global metadata server information.

TransferEngine::freeEngine#

int freeEngine();

Releases resources early without waiting for destruction.

Advanced APIs#

These APIs are optional; use them when you need manual transport control, coordination notifications, metadata/cache refresh, topology discovery, or debugging. They are not required for the basic data path.

Multi-Transport Management#

The TransferEngine class internally manages multiple backend Transport classes. And it will discover the topology between CPU/CUDA and RDMA devices automatically (more device types are working in progress, feedbacks are welcome when the automatic discovery mechanism is not accurate), and it will install Transport automatically based on the topology.

TransferEngine::installTransport#

Transport* installTransport(const std::string& proto, void** args);

Installs a transport backend explicitly.

  • proto: Transport protocol name, such as rdma, tcp, nvmeof, or shm.

  • args: Transport-specific arguments.

Note: In TENT, this method is a compatibility no-op and returns a non-null handle for callers that check the classic success contract. The handle cannot submit or query transfers; TENT selects transports internally from its configuration.

TransferEngine::uninstallTransport#

int uninstallTransport(const std::string& proto);

Uninstalls a transport backend.

  • proto: Transport protocol name.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

TransferEngine::getTransport#

Transport* getTransport(const std::string& proto);

Returns a transport instance by protocol name, mainly for advanced inspection or debugging.

Notifications#

TransferEngine can send and receive lightweight notifications across segments to coordinate data movement.

Notifications are sent only when requested through the APIs below. For classic TE:

  • MC_RDMA_NOTIFY_ENABLED=1 (default) enables RDMA notifications for both rdma and rdma_twosided; set it to 0 before initialization to use TCP.

  • Ordinary rdma supports up to 65528 combined bytes of name and msg. Unsupported peers or larger messages fall back to TCP when MC_RDMA_NOTIFY_OOB_FALLBACK=1 (default); set it to 0 to return an error instead.

  • A successful ordinary RDMA send indicates local submission, without confirming remote receipt. Connection setup or backpressure may block; applications must ensure delivery before shutting down either engine.

  • Ordinary RDMA notifications use 32 MiB of registered host memory per endpoint.

TransferEngine::submitTransferWithNotify#

Status submitTransferWithNotify(BatchID batch_id,
                                const std::vector<TransferRequest>& entries,
                                TransferMetadata::NotifyDesc notify_msg);

Submits a batch transfer and delivers a notification upon completion.

  • notify_msg: A {name, msg} payload delivered to the receiver.

  • Typical use: signal that a transferred buffer or KV-cache slice is ready to consume on the receiver side.

TransferEngine::getNotifies#

int getNotifies(std::vector<TransferMetadata::NotifyDesc>& notifies);

Gets pending notifications from peers.

  • Return value: If successful, returns 0; otherwise, returns a negative value.

  • Typical use: receiver-side polling loop to trigger follow-up actions (e.g., attaching the transferred buffer to a scheduler or cache).

TransferEngine::sendNotifyByID#

int sendNotifyByID(SegmentID target_id, TransferMetadata::NotifyDesc notify_msg);

Sends a notification to a specific segment by ID.

  • Typical use: control-plane message such as “ready”, “invalidate”, or “retry” tied to a known segment handle.

TransferEngine::sendNotifyByName#

int sendNotifyByName(std::string remote_agent, TransferMetadata::NotifyDesc notify_msg);

Sends a notification to a remote agent by name.

  • Typical use: best-effort coordination when you only know the peer’s name (segment name), not the handle.

Segment Cache and Metadata Access#

TransferEngine::syncSegmentCache#

int syncSegmentCache(const std::string& segment_name = "");

Synchronizes local segment cache from the metadata service.

  • segment_name: If empty, refreshes all segments; otherwise, refreshes the specified segment.

  • Typical use: when peers dynamically register/unregister buffers and you need to refresh before opening or submitting transfers.

TransferEngine::getMetadata#

std::shared_ptr<TransferMetadata> getMetadata();

Returns the metadata subsystem instance for advanced workflows.

  • Typical use: advanced integration or debugging of metadata content beyond the standard APIs.

Topology and Discovery#

TransferEngine::setAutoDiscover#

void setAutoDiscover(bool auto_discover);

Enables or disables topology discovery.

  • Typical use: controlled environments or debugging where auto discovery is not desired.

TransferEngine::setWhitelistFilters#

void setWhitelistFilters(std::vector<std::string>&& filters);

Sets the device whitelist used during topology discovery.

  • Typical use: restrict transfers to a subset of NICs or GPUs for performance isolation or testing.

TransferEngine::numContexts#

int numContexts() const;

Returns the number of active transport contexts.

  • Typical use: introspection and resource monitoring.

TransferEngine::getLocalTopology#

std::shared_ptr<Topology> getLocalTopology();

Returns the local topology model used for path selection.

  • Typical use: diagnose path selection or export topology to a scheduler.

Introspection and Safety#

TransferEngine::getLocalIpAndPort#

std::string getLocalIpAndPort();

Returns the resolved local RPC address.

  • Typical use: publish the resolved address to external components or logs.

TransferEngine::getRpcPort#

int getRpcPort();

Returns the active RPC port.

  • Typical use: verify port binding when dynamic ports are used.

TransferEngine::checkOverlap#

bool checkOverlap(void* addr, uint64_t length);

Checks whether a given address range overlaps with existing registered buffers.

  • Typical use: validate registration plans before calling registerLocalMemory in complex buffer managers.

Contents