Store-RS Deployment Guide

Store-RS Deployment Guide#

This document explains how to run mooncake-store-rs locally and how to map the runtime to common deployment roles.

Requirements#

  • Rust toolchain when building the optional Store-RS wheel component

  • cmake and a C++ toolchain

  • redis-server and redis-cli

  • Git submodule support for native dependencies

  • Python 3, if you want to run the Python compatibility layer

Prepare the Repository#

Set the Store-RS source, Mooncake source, and CMake build paths explicitly. Each path must be absolute. The top-level CMake project builds Transfer Engine, TENT, the Store-RS native shims, the Python extension, and the three command bins when WITH_STORE_RS=ON:

export MOONCAKE_STORE_RS_DIR=/path/to/Mooncake/mooncake-store-rs
export MOONCAKE_ROOT_DIR=/path/to/Mooncake
export MOONCAKE_BUILD_DIR=/path/to/Mooncake-build
cmake -S "${MOONCAKE_ROOT_DIR}" -B "${MOONCAKE_BUILD_DIR}" \
  -DWITH_TE=ON -DUSE_TENT=ON -DWITH_STORE=OFF -DWITH_STORE_RUST=OFF \
  -DWITH_STORE_RS=ON
cmake --build "${MOONCAKE_BUILD_DIR}" --target build_store_rs

The default build leaves WITH_STORE_RS=OFF and does not require Cargo. CMake owns the TE/TENT shim targets; mooncake-store-rs-transport-sys validates and links to those explicit build-tree artifacts.

Large-Memory Classic RDMA Bring-Up#

For classic_te deployments on RDMA hosts with very large local storage:

  • keep LocalMemoryConfig::numa_aware(true) unless you have a measured reason to collapse registration onto one CPU location

  • expect startup registration to fan out into multiple initial storage segments only on transports that opt into parallel startup registration; today that means classic_te + RDMA

  • the runtime now pre-touches startup storage automatically before RDMA MR registration once the total startup storage registration volume reaches 4 GiB

  • metadata publication happens after the local startup registration phase finishes, so operators should treat the segment set as appearing in one startup wave rather than one segment at a time

Deployment Roles#

The runtime is assembled from regular clients with different configuration.

Storage node#

A storage node owns local segments and can accept local or remote allocation requests.

Typical settings:

  • state(ClientLifecycleState::Active)

  • label("storage", "true")

  • non-zero LocalMemoryConfig

  • register_local_memory() after build(...)

  • optional hugepage-backed local memory through LocalMemoryConfig

Operational behavior:

  • storage=true opts the client into routed placement candidate sets

  • the same label also enables local storage-owner CLOCK eviction when reservation pressure appears

  • the same label enables background watermark eviction when local storage is configured

  • writers publish remote replica routes back to the storage owner automatically, so eviction does not need metadata scans on the hot path

  • storage=true requires storage_bytes > 0

Routed writer#

A routed writer accepts write requests and places replicas onto storage nodes selected by PlacementPlanner.

Typical settings:

  • label("storage", "false")

  • routed_writes(planner, replica_count)

  • local memory for scratch space and optional local placement

Operational behavior:

  • routed writers can run with storage_bytes=0 in inference/storage split deployments

  • when they do not own storage, they do not run local CLOCK eviction

  • remote placement still benefits from remote storage-owner eviction and route tracking

  • when storage_bytes=0 and no explicit storage label is provided, the runtime normalizes the label to storage=false

Reader or stateless client#

A reader can resolve routes and fetch objects without acting as a storage target.

Typical settings:

  • label("storage", "false")

  • no routed_writes(...) unless it also routes writes

  • transport enabled if it needs remote data transfer

Metadata and Transport#

mooncake-store-rs separates store metadata from transport metadata.

Store metadata#

Supported backends:

  • Redis

  • etcd

  • in-memory backend for tests

Store metadata is responsible for:

  • live client leases

  • segment announcements and lifecycle state

  • route policy and handoff plans

  • object routes in MetadataOnly deployments

  • handoff plans

Redis metadata supports two authentication forms:

  • set MC_REDIS_PASSWORD for password-only Redis deployments

  • set both MC_REDIS_USERNAME and MC_REDIS_PASSWORD for Redis ACL users

Credentials embedded in redis://username:password@host:port/db also work and take precedence over the environment variables. Prefer environment variables for cloud Redis passwords or any password containing URL-reserved characters.

Clean stale segment registrations#

Redis client resource hashes expire automatically. The Redis lease payload and owned segment records live in the same hash, so the lease TTL removes both. etcd client leases do not expire automatically; admin relies on the stored expires_at_ms field plus a backend work index, and etcd segment registration keys remain owner-scoped explicit keys.

That means a hard-killed etcd-backed storage client can leave stale segments/... metadata and owner-scoped segment bookkeeping behind even after its lease has disappeared. Redis can also need a repair sweep after abnormal metadata edits or legacy state. Strict tenant quota reservations can outlive a killed writer until an explicit reconcile repairs the state. Store-RS supports one-shot repair plus stateless background maintenance in the same admin binary:

  • a one-shot sweep through cleanup-stale-segments

  • a stateless background maintenance loop in mooncake-store-rs-admin server

  • optional tenant quota reservation reconcile for an explicit tenant list in that same admin process

Run the admin sweep when you want to remove dead-owner segment metadata. The same shape works with either redis://... or etcd://... metadata URLs:

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  cleanup-stale-segments

Run the stateless admin container shape when cleanup should happen continuously:

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  server \
  --bind-addr 0.0.0.0:8080 \
  --cleanup-interval-ms 5000 \
  --cleanup-batch-size 128 \
  --quota-reconcile-interval-ms 10000 \
  --quota-reconcile-tenant tenant-a \
  --quota-reconcile-tenant tenant-b

The background maintenance loop works like this:

  • lease create and heartbeat refresh both update a backend-native lease-expiry work index

  • Redis stores due work in one sorted set; etcd stores a by-runtime pointer plus a lexicographically ordered by-time queue

  • the admin worker polls due entries from that backend-native work index

  • each due owner is re-checked against the live lease key before cleanup

  • dead-owner segment deletion stays owner-scoped instead of falling back to a hidden full keyspace walk in the steady-state worker

  • if a lease key disappeared briefly, same-epoch reclaim is accepted as long as that epoch is still the stable-id HWM and no higher live epoch exists

  • tenant quota reconcile remains explicit: the worker only runs for tenants named on the command line, then internally reuses the same repair logic as mooncake-store-rs-admin quota reconcile

This keeps the admin pod stateless. If the pod restarts, the next reconcile loop resumes from Redis or etcd metadata instead of relying on in-memory work queues.

You can also manage tenant route policy and inspect strict-quota metadata through the same binary:

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  policy list

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  policy list \
  --tenant tenant-a

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  policy set \
  --tenant tenant-a \
  --route-topk 3 \
  --route-control embedded-wrh

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  quota state \
  --tenant tenant-a

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  quota reservations \
  --tenant tenant-a \
  --state pending

mooncake-store-rs-admin \
  --metadata-url redis://127.0.0.1:6380/0 \
  quota reconcile \
  --tenant tenant-a \
  --dry-run

Useful options:

  • --keyspace <prefix> to target a non-default metadata namespace for either policy management or stale cleanup

  • policy list --tenant <tenant> narrows Redis / etcd metadata reads to that tenant’s root policy and nested policy subtree

  • MC_REDIS_USERNAME / MC_REDIS_PASSWORD for Redis ACL authentication

  • terminal tenant-quota reservations (Finalized / Aborted) expire automatically after 24h by default; if a deployment needs a different retention window, configure RedisMetadataConfig::tenant_quota_terminal_ttl(...)

  • server --cleanup-interval-ms 0 to run the admin HTTP surface without the maintenance worker

  • server --quota-reconcile-tenant <tenant> repeated for each tenant whose strict quota reservations should be repaired automatically

Local e2e validation#

The repository’s main end-to-end validation binary is crates/mooncake-store-e2e/src/main.rs, and the standard local entrypoint is:

mooncake-store-rs/scripts/e2e/run-local-e2e.sh

For fast local verification while iterating on quota behavior, reduce benchmark noise and disable RDMA probing:

MC_STORE_RS_ENABLE_RDMA=0 \
MC_STORE_RS_BENCH_ITERS=1 \
mooncake-store-rs/scripts/e2e/run-local-e2e.sh

The local e2e now includes a focused strict tenant quota scenario that proves all of the following against Redis-backed metadata state:

  • a tenant quota policy is applied before the quota-scoped writer starts

  • an admitted write consumes quota and leaves pending_reserved_* == 0 after finalize

  • a second over-limit write is rejected without drifting quota state or reservation count

  • authoritative delete refunds quota immediately and makes the same capacity reusable

  • object accounting and finalized reservation records are visible in metadata for the test tenant

The command removes:

  • stale Redis client resource hashes whose lease field is gone but resource fields remain

  • stale etcd segment keys owned by clients with no live lease

  • stale backend index entries used by the maintenance scheduler

Transport metadata#

The transport layer is selected by the compatibility runtime and then configured through the matching backend config.

Current repository scripts and examples default to classic_te; set MC_STORE_RS_TRANSPORT_BACKEND=tent only when a deployment explicitly wants TENT.

Runtime selection:

  • MC_STORE_RS_TRANSPORT_BACKEND=tent|classic_te

  • mooncake-store-rs-client --transport-backend tent|classic-te

  • MooncakeDistributedStore.setup(..., transport_backend="tent"|"classic_te")

  • MC_STORE_RS_GID_INDEX=<n> when classic_te over RDMA must pick a non-default RoCE GID index; Store-RS forwards it to upstream MC_GID_INDEX

When metadata_url (arg7) is an etcd URL, the Transfer Engine still needs its own metadata at transport_metadata_url (arg2). For classic_te, P2PHANDSHAKE is the default; for tent, supply a redis://... value explicitly.

Port roles stay the same across backends:

  • transport_rpc_port is the real data-plane TCP port published to peer real clients

  • client_server_address is the dummy compatibility gRPC port

  • metrics_addr / MC_STORE_RS_METRICS_ADDR is the Prometheus /metrics listener

Backend reconnect behavior:

  • when Redis metadata connectivity returns, the heartbeat recovery path republishes both store metadata and transport metadata

  • classic_te repairs fresh Redis restarts by recreating its engine and republishing local buffers

  • tent repairs fresh Redis restarts by recreating its engine and re-registering the local buffers it still owns

  • if a live peer restart leaves callers with a stale cached remote segment handle, reads and routed writes refresh that handle by segment name before treating the peer as dead

  • runtime quarantine still applies to true reopen or transfer failures; stale cached handles alone do not demote an otherwise live peer

  • the process must remain alive across the outage for automatic recovery; a dead client still needs normal restart and lease takeover semantics

Routing Modes#

EmbeddedWrh#

This is the default mode.

  • route ownership is selected on the client with weighted rendezvous hashing

  • the top route_topk authorities are selected per key; the first authority is primary and the rest are mirrors

  • the client prewarms a live-client membership snapshot during build(...)

  • background membership sync refreshes that snapshot after startup

  • steady-state reads use that cached snapshot instead of refreshing membership inline

  • route reads and CAS stay off the metadata hot path in steady state

  • storage owners manage local eviction separately from route ownership

  • read paths keep Draining owners readable for handoff, fail fast on suspect or offline owners, keep suspect owners quarantined until membership shows a fresh lease/control-plane refresh, and best-effort prune unreadable replicas after fallback

Cluster policy is metadata-authoritative:

  • every client starts with a local route_control (cluster-level) and route_topk (fallback, overridable per-tenant)

  • the first client in a metadata keyspace persists the cluster route policy

  • later clients must match the stored policy or startup fails

  • route_control is uniform across all tenants in a metadata keyspace

MetadataOnly#

This mode is useful for simpler bring-up and debugging.

  • route reads and writes go directly to the metadata backend

  • fewer moving parts

  • higher dependence on metadata latency

Label Conventions#

The current implementation relies on a few label conventions.

Label

Meaning

storage=true

marks a client as a placement candidate for routed writes

pool=<name>

scopes placement planning; this is the default placement scope key

route_scope=<name>

optionally scopes route authority selection

The runtime also manages some labels internally, such as route capability and control-plane address publication.

Operational Conventions#

  • keep stable_id stable across restarts and upgrades

  • start successor processes with the same stable_id during hot-upgrade flows; the metadata backend allocates the next epoch and the runtime prints it as epoch=<n> on startup

  • mount local memory before serving data traffic

  • keep heartbeat_interval_ms comfortably below the default lease_ttl_ms=30000

  • use a dedicated metadata keyspace per environment or test run

  • if hugepage mode is enabled, preallocate matching hugepages on the host before starting clients

Next Reading#

Contents