Mooncake on Kubernetes

Mooncake on Kubernetes#

Deploy a Mooncake Store cluster — a mooncake-master plus replicated mooncake-store nodes — with plain Kubernetes objects (Deployment and Service).

Use it together with the Mooncake Store Deployment & Tuning Guide: that guide explains the components and tuning knobs; this page maps them to Kubernetes objects.

Deploy the Mooncake Store cluster#

A shareable Store cluster has one mooncake-master (the RPC coordinator) and a replicated set of stateless mooncake-store nodes that contribute DRAM to the pool. The nodes use Mooncake’s P2P handshake (P2PHANDSHAKE) for Transfer Engine peer discovery, so there is no separate metadata service to run — each node stores its metadata locally and exchanges it with peers during connection setup. It needs no GPUs, and multiple inference deployments can point at the same cluster. The store nodes reach the master through the mooncake-master Service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mooncake-master
  labels:
    app: mooncake-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mooncake-master
  template:
    metadata:
      labels:
        app: mooncake-master
    spec:
      containers:
        - name: mooncake-master
          image: lmsysorg/sglang:v0.5.5
          command: ["mooncake_master"]
          args:
            - --rpc_address
            - $(POD_IP)
            - --rpc_port
            - "50051"
            - --metrics_port
            - "9003"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: rpc
              containerPort: 50051
            - name: metrics
              containerPort: 9003
          readinessProbe:
            tcpSocket:
              port: 50051
            initialDelaySeconds: 10
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: mooncake-master
  labels:
    app: mooncake-master
spec:
  type: ClusterIP
  selector:
    app: mooncake-master
  ports:
    - name: rpc
      port: 50051
      targetPort: 50051
    - name: metrics
      port: 9003
      targetPort: 9003
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mooncake-store
  labels:
    app: mooncake-store
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mooncake-store
  template:
    metadata:
      labels:
        app: mooncake-store
    spec:
      containers:
        - name: mooncake-store
          image: lmsysorg/sglang:v0.5.5
          command: ["python3", "-m", "mooncake.mooncake_store_service"]
          args: ["--port", "8088"]
          env:
            - name: MOONCAKE_MASTER
              value: "mooncake-master:50051"
            - name: MOONCAKE_TE_META_DATA_SERVER
              value: "P2PHANDSHAKE"
            - name: MOONCAKE_GLOBAL_SEGMENT_SIZE
              value: "10gb"
            - name: MOONCAKE_LOCAL_BUFFER_SIZE
              value: "0"
            - name: MOONCAKE_PROTOCOL
              value: "rdma"
          resources:
            requests:
              memory: "16Gi"
            limits:
              memory: "16Gi"

See Notes for capacity, protocol (TCP/RDMA), and metadata guidance.

Verify:

kubectl get pods -l app=mooncake-master
kubectl get pods -l app=mooncake-store
# Master metrics summary:
kubectl port-forward svc/mooncake-master 9003:9003 &
curl -s http://localhost:9003/metrics/summary

Notes#

Metadata (P2P handshake). These manifests use Mooncake’s P2P handshake (MOONCAKE_TE_META_DATA_SERVER: P2PHANDSHAKE): each node stores Transfer Engine metadata locally and exchanges it with peers during connection setup, so there is nothing extra to run and the mooncake-master needs no --*http_metadata_server* flags. This is the recommended starting point. For large or long-lived clusters, switch the store nodes to the master’s embedded HTTP metadata server or an external etcd/Redis instead; see the store guide’s Deployment Scenarios.

High availability. A single mooncake-master is a single point of failure. For HA, see the store guide’s High Availability section for etcd/Redis backends.

TCP vs RDMA. MOONCAKE_PROTOCOL selects the fabric. These manifests use rdma. Granting pods RDMA access is cluster-specific and not fully wired into the YAML above; the production reference (see RBG Integration) does it with hostNetwork: true, a hostPath mount of /dev/infiniband, privileged + IPC_LOCK/SYS_RESOURCE, and an explicit NIC list via MOONCAKE_DEVICE=<rdma-devices>. (A device-plugin rdma/hca resource with MC_MS_AUTO_DISC / MC_MS_FILTERS auto-discovery is an alternative on clusters set up that way.) Switch MOONCAKE_PROTOCOL to tcp on clusters without an RDMA fabric.

Capacity. Keep MOONCAKE_GLOBAL_SEGMENT_SIZE within each pod’s memory limit. A pure store node issues no Get/Put itself, so its MOONCAKE_LOCAL_BUFFER_SIZE is small; the production RDMA reference sets a modest non-zero buffer (67108864 = 64 MiB) rather than 0.

Images. The example uses lmsysorg/sglang:v0.5.5. This tag is not a reproducible pin — for production, replace it with a verified tag or digest.

Contents