RBG Integration

This page covers the same Store + Transfer Engine P/D scenario as the main guide, deployed with the sgl-project/rbg operator instead of the vanilla Deployment / Service manifests on the Mooncake on Kubernetes page. Install the RBG operator before applying any RoleBasedGroup.

RBG example

The upstream RBG repository ships ready-to-use examples for running Mooncake on RBG:

• sgl-pd-disagg-with-mooncake-te.yaml — SGLang P/D disaggregation using Mooncake’s Transfer Engine for KV transfer.

• vllm-pd-disagg-with-mooncake-te.yaml — vLLM P/D disaggregation using Mooncake’s Transfer Engine for KV transfer.

• standalone-mooncake-store.yaml — the shareable standalone Mooncake Store cluster (mooncake-master + mooncake-store roles).

For background on the integration, see the RBG Mooncake integration KEP.

Production Mooncake Cluster Example

A production Mooncake cluster on RBG (workloads.x-k8s.io/v1alpha2): a Mooncake Store (one master plus NUMA-split store pods) and SGLang prefill/decode engines, all over RDMA. The two RBGs below are the Mooncake-side backend; a router/gateway (out of scope for this page) fronts the prefill/decode endpoints to make the P/D deployment servable — see the overview and the Prefill/Decode Disaggregation quick start.

Group	Roles	Purpose
Store (qwen3-0)	master, store-1000gb	Mooncake Store — the master coordinator plus NUMA-split store pods contributing the DRAM KV pool
Workers (sglang-workers-0)	prefill, decode	SGLang engines: prefill is a Store/HiCache client and P/D transfer; decode does P/D transfer only

Caution

These manifests are sanitized excerpts that show the structure and the Mooncake wiring — they cannot be applied directly. The decode role and the second NUMA store container are abbreviated to comments, and image / paths / devices are <…> placeholders. Fill them in against your own cluster before applying.

1. Store group — master + NUMA store

The Store RoleBasedGroup has a master (the Store coordinator) and a store-<size> role whose pods each run two NUMA-pinned store processes. The RBG operator creates a Service s-qwen3-0-master for the master role, so clients reach it at s-qwen3-0-master:50051 — no Service object of your own. Node scheduling uses custom kvcache.ai/master and kvcache.ai/store labels so a node belongs to exactly one role/size.

apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
  name: qwen3-0
  namespace: default
  labels: { app.kubernetes.io/part-of: mooncake }
spec:
  roles:
  - name: master
    replicas: 1
    standalonePattern:
      template:
        metadata:
          labels: { role: master, app.kubernetes.io/instance: qwen3-0 }
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - { key: kvcache.ai/master, operator: In, values: [qwen3_0_master] }
          containers:
          - name: master
            image: <image>
            command:
            - sh
            - -c
            - |
              ulimit -n 1048576
              ulimit -l unlimited
              mooncake_master \
                --rpc_address=$(POD_IP) \
                --rpc_port=50051 \
                --eviction_high_watermark_ratio=0.9 \
                --default_kv_lease_ttl=10000
            env:
            - { name: POD_IP, valueFrom: { fieldRef: { fieldPath: status.podIP } } }
            - { name: NVIDIA_VISIBLE_DEVICES, value: "void" }   # master needs no GPU
            securityContext:
              # master is an RPC coordinator with no RDMA data path — it does not need
              # `privileged`. IPC_LOCK/SYS_RESOURCE cover the `ulimit -l unlimited` / mlock above.
              privileged: false
              capabilities: { add: ["IPC_LOCK", "SYS_RESOURCE"] }
            livenessProbe:
              exec: { command: ["/bin/sh", "-c", "pgrep -x mooncake_master >/dev/null"] }
              initialDelaySeconds: 20
              periodSeconds: 15
            ports:
            - { containerPort: 50051, name: http }
            - { containerPort: 9003, name: metrics }

  - name: store-1000gb
    replicas: 3
    standalonePattern:
      template:
        metadata:
          labels:
            role: store-1000gb
            app.kubernetes.io/instance: qwen3-0
            app.kubernetes.io/part-of: mooncake
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - { key: kvcache.ai/store, operator: In, values: [qwen3_0_store-1000gb] }
            podAntiAffinity:                  # at most one store pod per node across sizes
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - { key: app.kubernetes.io/part-of, operator: In, values: [mooncake] }
                  - { key: app.kubernetes.io/instance, operator: In, values: [qwen3-0] }
                topologyKey: kubernetes.io/hostname
          hostNetwork: true                   # RDMA
          dnsPolicy: ClusterFirstWithHostNet
          containers:
          - name: store-numa0
            image: <image>
            command:
            - sh
            - -c
            - |
              ulimit -n 1048576
              ulimit -l unlimited
              exec numactl --cpunodebind=0 --membind=0 python3 -m mooncake.mooncake_store_service --port=8099
            env:
            - { name: MOONCAKE_LOCAL_HOSTNAME, valueFrom: { fieldRef: { fieldPath: status.podIP } } }
            - { name: MOONCAKE_MASTER, value: "s-qwen3-0-master:50051" }
            - { name: MOONCAKE_TE_META_DATA_SERVER, value: "P2PHANDSHAKE" }
            - { name: MOONCAKE_GLOBAL_SEGMENT_SIZE, value: "1000gb" }   # DRAM this store contributes
            - { name: MOONCAKE_LOCAL_BUFFER_SIZE, value: "67108864" }
            - { name: MOONCAKE_PROTOCOL, value: "rdma" }
            - { name: MOONCAKE_DEVICE, value: "<rdma-devices>" }
            - { name: MC_ENABLE_DEST_DEVICE_AFFINITY, value: "1" }
            # pin the client metrics HTTP server per container (numa0=9300 / numa1=9301);
            # under hostNetwork two processes cannot share 9300.
            - { name: MOONCAKE_ENABLE_CLIENT_HTTP_SERVER, value: "true" }
            - { name: MOONCAKE_CLIENT_HTTP_PORT, value: "9300" }
            ports: [{ containerPort: 8099 }]
            startupProbe:
              exec: { command: ["sh", "-c", "nc -z 127.0.0.1 8099"] }
              periodSeconds: 10
              failureThreshold: 90
            livenessProbe:
              exec: { command: ["sh", "-c", "nc -z 127.0.0.1 8099"] }
              initialDelaySeconds: 10
              periodSeconds: 10
            securityContext:
              # least-privilege RDMA: no `privileged` needed — IPC_LOCK/SYS_RESOURCE plus
              # the /dev/infiniband device mount below are enough for the Transfer Engine NICs.
              privileged: false
              capabilities: { add: ["IPC_LOCK", "SYS_RESOURCE"] }
            volumeMounts:
            - { mountPath: /dev/infiniband, name: ib }
          # store-numa1: identical, but `numactl --cpunodebind=1 --membind=1`, --port=8100,
          # containerPort 8100, and MOONCAKE_CLIENT_HTTP_PORT=9301.
          volumes:
          - { name: ib, hostPath: { path: /dev/infiniband, type: DirectoryOrCreate } }

2. Inference workers — prefill + decode

sglang-workers-0 runs the SGLang PD engines, and the two roles are wired differently:

• prefill is the Store/HiCache client. It sets MOONCAKE_MASTER=s-qwen3-0-master:50051 (the Store master), MOONCAKE_TE_META_DATA_SERVER=P2PHANDSHAKE (the Transfer Engine coordinates P/D directly, no metadata server), MOONCAKE_PROTOCOL=rdma + MOONCAKE_DEVICE, and MOONCAKE_GLOBAL_SEGMENT_SIZE=0 (a pure client — it contributes no DRAM; the store pods do). It launches with --enable-hierarchical-cache --hicache-storage-backend mooncake. The prefill manifest is shown below.

• decode only participates in the P/D Transfer Engine — --disaggregation-mode decode --disaggregation-ib-device. It is not a Store/HiCache client: it sets no MOONCAKE_MASTER and enables no hierarchical cache.

apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
  name: sglang-workers-0
  namespace: default
spec:
  roles:
  - name: prefill
    replicas: 4
    standalonePattern:
      template:
        metadata:
          labels:
            app: sglang-worker
            rbg.workloads.x-k8s.io/group-name: sglang-workers-0
            rbg.workloads.x-k8s.io/role-name: prefill
        spec:
          hostNetwork: true                   # RDMA
          dnsPolicy: ClusterFirstWithHostNet
          nodeSelector: { deployment: sglang_0_prefill }
          containers:
          - name: sglang-prefill
            image: <image>
            command:
            - bash
            - -c
            - |
              set -e
              ulimit -n 1048576; ulimit -l unlimited
              python -m sglang.launch_server \
                --model ${MODEL_PATH} --served-model-name Qwen3-0.6B \
                --host 0.0.0.0 --port 8000 \
                --disaggregation-mode prefill \
                --disaggregation-ib-device $IB_DEVICE_LIST \
                --enable-hierarchical-cache --hicache-storage-backend mooncake \
                --tp 8 --page-size 64 --trust-remote-code \
                --enable-metrics --enable-cache-report
                # … model/hardware tuning flags omitted (context length, mem fraction,
                #    NSA backends, EAGLE speculative decoding, KV-cache dtype, etc.)
            env:
            # --- pod identity ---
            - { name: POD_NAME, valueFrom: { fieldRef: { fieldPath: metadata.name } } }
            - { name: POD_IP, valueFrom: { fieldRef: { fieldPath: status.podIP } } }
            - { name: MOONCAKE_LOCAL_HOSTNAME, valueFrom: { fieldRef: { fieldPath: status.podIP } } }
            - { name: SGLANG_HOST_IP, valueFrom: { fieldRef: { fieldPath: status.podIP } } }
            # --- model + fabric ---
            - { name: MODEL_PATH, value: /models/Qwen3-0.6B }
            - { name: IB_DEVICE_LIST, value: "<rdma-devices>" }
            # --- Mooncake wiring ---
            - { name: MOONCAKE_TE_META_DATA_SERVER, value: P2PHANDSHAKE }
            - { name: MOONCAKE_MASTER, value: "s-qwen3-0-master:50051" }   # the Store group's master
            - { name: MOONCAKE_PROTOCOL, value: rdma }
            - { name: MOONCAKE_DEVICE, value: "<rdma-devices>" }
            - { name: MOONCAKE_GLOBAL_SEGMENT_SIZE, value: "0" }   # pure client, contributes no DRAM
            - { name: MC_TE_METRIC, value: "true" }
            # … SGLANG_* / MC_* performance tuning omitted (heartbeat, timeouts,
            #   spec-decoding v2, auto-empty-cache, NCCL, JIT, CPU affinity, PRC port range) …
            ports:
            - { containerPort: 8000, name: http }
            - { containerPort: 8998, name: bootstrap }
            readinessProbe:
              tcpSocket: { port: 8000 }
              initialDelaySeconds: 30
              periodSeconds: 10
            resources:
              limits: { nvidia.com/gpu: "8" }
              requests: { nvidia.com/gpu: "8" }
            securityContext:
              # least-privilege RDMA: no `privileged` needed — IPC_LOCK/SYS_RESOURCE plus
              # the /dev/infiniband device mount below are enough for the NICs.
              privileged: false
              capabilities: { add: ["IPC_LOCK", "SYS_RESOURCE"] }
            volumeMounts:
            - { mountPath: /models, name: model }
            - { mountPath: /dev/shm, name: dshm }
            - { mountPath: /dev/infiniband, name: ib }
          volumes:
          - { name: model, hostPath: { path: <model-host-path>, type: DirectoryOrCreate } }
          - { name: dshm, emptyDir: { medium: Memory, sizeLimit: 1300Gi } }
          - { name: ib, hostPath: { path: /dev/infiniband, type: DirectoryOrCreate } }

  - name: decode
    replicas: 1
    # Abbreviated excerpt — a real decode PodSpec mirrors the prefill container EXCEPT:
    #   launch:  --disaggregation-mode decode --disaggregation-ib-device $IB_DEVICE_LIST
    #            (NO --enable-hierarchical-cache / --hicache-storage-backend — decode is
    #             not a Store client), plus DP/EP attention, low-latency deepep, etc.
    #   env:     NO MOONCAKE_MASTER / MOONCAKE_GLOBAL_SEGMENT_SIZE / hierarchical cache;
    #            keeps POD_NAME / POD_IP / MODEL_PATH / IB_DEVICE_LIST (P/D transfer only)
    #   sched:   nodeSelector deployment: sglang_0_decode ; dshm sizeLimit 15Gi
    standalonePattern:
      template: { }   # fill in a real PodSpec to deploy

Mooncake integration points (recap)

• Store master — mooncake_master --rpc_address=$(POD_IP) --rpc_port=50051 …; the RBG operator exposes it as s-qwen3-0-master, and its clients (the store pods and the prefill engine) set MOONCAKE_MASTER=s-qwen3-0-master:50051.

• Only prefill is a Store client — prefill enables HiCache (--enable-hierarchical-cache --hicache-storage-backend mooncake) and connects to the master; decode participates only in the P/D Transfer Engine and connects to no Store.

• Transfer Engine uses P2P handshake — prefill sets MOONCAKE_TE_META_DATA_SERVER=P2PHANDSHAKE; the TE side-channel coordinates prefill↔decode directly, so there is no HTTP metadata server here (unlike the main guide).

• Store vs client segment size — store pods set MOONCAKE_GLOBAL_SEGMENT_SIZE=1000gb (they own the DRAM pool); the prefill client sets 0 (contributes no DRAM, only Get/Put).

• RDMA fabric — MOONCAKE_PROTOCOL=rdma + MOONCAKE_DEVICE=<rdma-devices>, hostNetwork: true, /dev/infiniband mounted, and the IPC_LOCK/SYS_RESOURCE capabilities (no privileged needed — a /dev/infiniband hostPath mount plus those two capabilities is the least-privilege way to reach the NICs; an RDMA device plugin is an alternative).

• NUMA split — each store pod runs two mooncake_store_service processes, each numactl-pinned to one NUMA node with its own port and client-metrics port.

Placeholders

The example values above (qwen3-0, default, store-1000gb, replica counts) are yours to change. The <…> placeholders are:

Placeholder	Replace with
<image>	the container image (registry/name:tag) for this role
<model-host-path>	host directory holding the model weights
<rdma-devices>	your RDMA NIC list (e.g. mlx5_0,mlx5_1,…)