Quick Start

Quick Start#

This document describes how to quickly start using Mooncake Transfer Engine and Mooncake Store.

Before using Mooncake#

Install the following prerequisites before running any Mooncake component:

  • Python 3.10 or later; a virtual environment is recommended.

  • RDMA driver and SDK (for example, Mellanox OFED), if you plan to use RDMA for data transfer.

  • CUDA 12.1 or later, if the package is built with -DUSE_CUDA (disabled by default). For most CUDA-enabled use cases, such as RDMA-based KV cache transfer between GPUs or between GPU and DRAM, NVIDIA GPUDirect support is also required. You may install them from here.

  • Cambricon Neuware, if the package is built with -DUSE_MLU. By default Mooncake looks for Neuware under NEUWARE_HOME or /usr/local/neuware.

  • Hygon DTK SDK, if the package is built with -DUSE_HYGON. By default Mooncake looks for DTK under DTK_HOME or /opt/dtk.

  • Iluvatar CoreX SDK, if the package is built with -DUSE_COREX. By default Mooncake looks for CoreX under COREX_HOME or /usr/local/corex.

Installation#

Install the Mooncake package from PyPI. The same package provides:

  • Mooncake Store Python bindings for vLLM and SGLang HiCache integrations.

  • Transfer Engine Python bindings and runtime components for direct mooncake.engine.TransferEngine usage.

Choose the package that matches your runtime environment. Install only one variant in an environment.

Runtime environment

PyPI package

Installation command

NVIDIA CUDA 12.1–12.9

mooncake-transfer-engine

pip install mooncake-transfer-engine

NVIDIA CUDA 13.0/13.1

mooncake-transfer-engine-cuda13

pip install mooncake-transfer-engine-cuda13

Non-CUDA

mooncake-transfer-engine-non-cuda

pip install mooncake-transfer-engine-non-cuda

Ascend NPU

mooncake-transfer-engine-npu

pip install mooncake-transfer-engine-npu

AMD ROCm

mooncake-transfer-engine-rocm

pip install mooncake-transfer-engine-rocm

Moore Threads MUSA

mooncake-transfer-engine-musa

pip install mooncake-transfer-engine-musa

AWS EFA with CUDA 12

mooncake-transfer-engine-efa

pip install mooncake-transfer-engine-efa

AWS EFA without CUDA

mooncake-transfer-engine-efa-non-cuda

pip install mooncake-transfer-engine-efa-non-cuda

Important:

  • The CUDA version (mooncake-transfer-engine) includes Mooncake-EP and GPU topology detection, requiring CUDA 12.1+.

  • The non-CUDA version (mooncake-transfer-engine-non-cuda) is for environments without CUDA dependencies, but it still needs system runtime libraries such as libcurl4, libibverbs1, rdma-core, librdmacm1, libnuma1, and liburing2 on Ubuntu. In a fresh environment, run sudo apt-get update before installing them:

    sudo apt-get update && sudo apt-get install -y libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2

  • The EFA variants require the AWS EFA driver and libfabric at runtime. See the EFA transport guide for prerequisites and configuration.

  • MLU support is currently available through source builds with -DUSE_MLU=ON; there is no dedicated prebuilt MLU wheel yet.

  • If users encounter problems such as missing lib*.so, first install the corresponding system runtime libraries. If the issue persists, uninstall the package and build the binaries manually.

Connect vLLM or SGLang#

Choose the integration path that matches your serving deployment.

PD Disaggregation#

PD disaggregation paths use Mooncake Transfer Engine for direct KV transfer between prefill and decode workers. Configure these paths through the serving framework guides, not by calling Transfer Engine APIs directly:

Mooncake Store#

Mooncake Store provides distributed KV cache storage for vLLM and SGLang HiCache:

Framework

Use case

Setup guide

SGLang

HiCache L3 storage backend with Mooncake Store

SGLang HiCache Quick Start

vLLM

KV cache storage and sharing with MooncakeStoreConnector

vLLM KV Cache Storage & Sharing

The serving framework guides include the required Mooncake Store service startup and connector configuration for each path.

Optional Python Smoke Test#

If you want to verify the Store Python API without a serving framework, run this single-node put/get example after starting mooncake_master. It uses P2PHANDSHAKE, so no separate Transfer Engine metadata service is required.

from mooncake.store import MooncakeDistributedStore

store = MooncakeDistributedStore()
store.setup(
    local_hostname="localhost",
    metadata_server="P2PHANDSHAKE",
    global_segment_size=512 * 1024 * 1024,
    local_buffer_size=128 * 1024 * 1024,
    protocol="tcp",
    rdma_devices="",
    master_server_addr="127.0.0.1:50051",
)

store.put("hello_key", b"Hello, Mooncake Store!")

data = store.get("hello_key")
print(data.decode())  # Output: Hello, Mooncake Store!

store.close()

AI Coding Assistant Skills#

If you use Claude Code or another coding assistant that supports reusable skills, Mooncake provides built-in playbooks for common development tasks:

Skill

Use it for

/mooncake-troubleshoot

Diagnose services, RDMA, environment variables, and runtime logs.

/mooncake-ci-local

Run pre-PR local validation with Mooncake’s CI script.

/mooncake-api

Work with Mooncake Store, Transfer Engine, and EP/Backend Python APIs.

Install them from the Claude Code plugin marketplace without cloning the full repository:

/plugin marketplace add kvcache-ai/Mooncake --sparse .claude-plugin
/plugin install mooncake-troubleshoot@mooncake
/plugin install mooncake-ci-local@mooncake
/plugin install mooncake-api@mooncake

Next Steps#

For production deployment, standalone store services, high availability, allocation strategies, SSD offload, and runtime tuning, continue to the Mooncake Store Deployment & Tuning Guide.

For API details, see the Mooncake Store Python API and Mooncake Store design.

Contents