Configuration Reference

This document collects the main runtime knobs exposed by the current implementation.

Tenant Policy Precedence

Tenant-scoped routing and resource policy should be authored through mooncake-store-rs-admin policy ... and stored in durable metadata. Runtime-local knobs remain available as compatibility fallbacks.

Precedence is:

1. admin-managed tenant policy in metadata

2. legacy compatibility metadata reads where still supported during migration

3. runtime-local builder / Python / standalone client fallback values when metadata does not provide the relevant section

Keep this distinction clear:

• tenant selects the default scope used for request builders and startup policy lookup

• request-scoped APIs and per-request ReplicationPolicy remain normal execution-time inputs

• local route/resource knobs are not the preferred long-term policy authoring surface

• strict quota usage/accounting inspection now comes from admin/metadata surfaces, not builder fields

Admin strict-quota inspection and repair

The admin surface now exposes metadata-backed strict quota inspection plus explicit repair commands:

• mooncake-store-rs-admin quota state --tenant <tenant> [--domain <domain>] [--object-set <set>]

• mooncake-store-rs-admin quota object --tenant <tenant> [--domain <domain>] [--object-set <set>] --key <logical-key>

• mooncake-store-rs-admin quota reservations --tenant <tenant> [--domain <domain>] [--object-set <set>] [--state pending|finalized|aborted]

• mooncake-store-rs-admin quota abort --tenant <tenant> [--domain <domain>] [--object-set <set>] --reservation-id <id> [--dry-run]

• mooncake-store-rs-admin quota reconcile --tenant <tenant> [--domain <domain>] [--object-set <set>] [--dry-run]

Operational notes:

• quota state and reservation queries resolve to the tenant-root scope because mutable quota state is tenant-root metadata in Phase 1

• object-accounting lookups still accept nested selectors so operators can specify the logical object namespace they care about

• reconcile currently aborts expired pending reservations and finalizes pending reservations whose authoritative route + accounting state is already visible

• finalized and aborted reservations are retained for a bounded terminal window, then expire from Redis automatically; the default terminal TTL is 24h

• Pending reservations still rely on reconcile / abort paths rather than Redis TTL so an in-flight write is never deleted out from under finalize

• crates/mooncake-store-rs-metadata exposes RedisMetadataConfig::tenant_quota_terminal_ttl(...) for deployments that want a different terminal retention window

• reservation listing prunes dangling reservation index entries opportunistically when the backing reservation record is already gone

• other mismatches remain operator-visible and are reported as skipped rather than repaired speculatively

• the admin HTTP surface rejects requests whose declared Content-Length exceeds 1 MiB with 413 Payload Too Large; because the server does not drain the remaining body, it responds with Connection: close

Admin maintenance

Redis-backed and etcd-backed deployments support explicit stale segment maintenance. The same admin server can also run tenant-quota reservation reconcile for an explicit tenant list.

Relevant surfaces:

• mooncake-store-rs-admin cleanup-stale-segments

• mooncake-store-rs-admin server --cleanup-interval-ms <ms> --cleanup-batch-size <n>

• mooncake-store-rs-admin server --quota-reconcile-interval-ms <ms> --quota-reconcile-tenant <tenant>

Operational model:

• the default metadata keyspace is mc/store-rs/v2; v2 is the Redis schema boundary for TTL-backed client resource hashes and intentionally does not read or migrate old v1 keys

• Redis client resource hashes use TTL as the liveness signal

• Redis stores the lease and owned segment records in that single client resource hash; when the lease TTL expires, Redis removes both the lease and the segment records

• etcd lease keys do not expire automatically; admin re-checks the stored expires_at_ms field when due work arrives

• etcd segment keys do not use TTL, but they are stored under the owning client runtime namespace

• lease publish and heartbeat refresh now also update a backend-native expiry work index under the same metadata keyspace: Redis uses one sorted set, etcd uses by-runtime + lexicographically ordered by-time keys

• the admin maintenance loop consumes due entries from that work index, re-checks lease liveness, and removes owner-scoped resource metadata only when the backend still has stale state to repair

• same-epoch lease reclaim after an expired lease key is allowed when that epoch is still the historical HWM and no higher live epoch exists, so heartbeat repair does not fail with StaleEpoch after a TTL gap

• tenant quota reconcile is intentionally opt-in per tenant because the current metadata model has no bounded global tenant-work index for the admin plane to consume safely

Admin tracing fanout

mooncake-store-rs-admin server is the cluster-level entry point for dynamic Jaeger tracing control. Operators should send tracing commands to the admin HTTP server instead of addressing every store node manually:

• mooncake-store-rs-admin tracing status --admin-url http://<admin-host>:<port>

• mooncake-store-rs-admin tracing on --admin-url http://<admin-host>:<port> [--endpoint <otlp-endpoint>] [--sample-ratio <0..1>]

• mooncake-store-rs-admin tracing off --admin-url http://<admin-host>:<port>

• mooncake-store-rs-admin tracing flush --admin-url http://<admin-host>:<port>

The admin server discovers store-node tracing endpoints from live client leases. Store runtimes publish only the system label metrics_port=<port> when the metrics HTTP endpoint is enabled. The host is derived from the existing lease endpoint data: control_addr first, then endpoints.rpc_address. This avoids duplicating another reachable-host field in metadata.

Compatibility behavior:

• metrics_port is stored inside ClientEndpointSet.labels, so the serialized lease schema does not gain a new required field

• newer admin servers use metrics_port when it exists and fall back to port 9300 for older leases that predate the label

• store startup removes any user-supplied metrics_port label and overwrites it with the actual bound metrics port when metrics are enabled

• store runtimes with metrics disabled do not publish metrics_port

Server maintenance defaults:

• --cleanup-interval-ms 5000

• --cleanup-batch-size 128

• --cleanup-interval-ms 0 disables the background maintenance worker

• --quota-reconcile-interval-ms 0 disables background tenant quota reconcile

• --quota-reconcile-tenant <tenant> can be repeated; when unset, no tenant quota reconcile worker is started

mooncake-store-rs-admin accepts every CLI parameter through an environment variable as a fallback. Explicit CLI flags override environment values.

Global admin knobs:

CLI flag	Environment variable	Default	Meaning
--metadata-url (alias --metadata_url)	MC_STORE_RS_METADATA_URL	required	Store-RS metadata URL (redis://... or etcd://...). Same naming as mooncake-store-rs-client / mooncake-store-rs-bench.
--admin-url	MC_STORE_ADMIN_URL	command-dependent	admin HTTP endpoint used by route-migration client commands
--keyspace	MC_STORE_RS_KEYSPACE	default keyspace	metadata keyspace
--trace-filter	MC_STORE_ADMIN_TRACE_FILTER	tracing default	tracing filter for the admin process

Server and migration knobs:

CLI flag	Environment variable	Default	Meaning
server --bind-addr	MC_STORE_ADMIN_BIND_ADDR	127.0.0.1:0	admin HTTP bind address
server --cleanup-interval-ms	MC_STORE_ADMIN_CLEANUP_INTERVAL_MS	5000	stale-segment maintenance interval; 0 disables it
server --cleanup-batch-size	MC_STORE_ADMIN_CLEANUP_BATCH_SIZE	128	stale-segment cleanup batch size
server --quota-reconcile-interval-ms	MC_STORE_ADMIN_QUOTA_RECONCILE_INTERVAL_MS	0	tenant-quota reconcile interval; 0 disables it
server --quota-reconcile-tenant	MC_STORE_ADMIN_QUOTA_RECONCILE_TENANTS	none	comma-separated tenants for background quota reconcile
migrate --authority	MC_STORE_ADMIN_AUTHORITY	required	route authority runtime id
migrate --tenant	MC_STORE_ADMIN_TENANT	required	tenant scope
migrate --domain	MC_STORE_ADMIN_DOMAIN	none	optional domain scope
migrate --object-set	MC_STORE_ADMIN_OBJECT_SET	none	optional object-set scope
migrate --key	MC_STORE_ADMIN_KEY	required	logical object key
migrate --source-segment	MC_STORE_ADMIN_SOURCE_SEGMENT	required	source segment for route migration
migrate copy --target-segment	MC_STORE_ADMIN_TARGET_SEGMENTS	required	comma-separated copy target segments
migrate move --target-segment	MC_STORE_ADMIN_TARGET_SEGMENT	required	move target segment
migrate --task-executor	MC_STORE_ADMIN_TASK_EXECUTOR	required	executor runtime id
migrate --max-retries	MC_STORE_ADMIN_MIGRATION_TASK_MAX_RETRIES	admin queue default	per-task retry override
migrate task get --task-id	MC_STORE_ADMIN_TASK_ID	required	route-migration task id

Policy and quota knobs:

CLI flag	Environment variable	Default	Meaning
policy --tenant / quota --tenant	MC_STORE_ADMIN_TENANT	command-dependent	tenant scope
policy --domain / quota --domain	MC_STORE_ADMIN_DOMAIN	none	optional domain scope
policy --object-set / quota --object-set	MC_STORE_ADMIN_OBJECT_SET	none	optional object-set scope
policy get --effective	MC_STORE_ADMIN_EFFECTIVE	false	resolve effective policy fallback chain
policy set/delete --expected-version	MC_STORE_ADMIN_EXPECTED_VERSION	none	optimistic policy version guard
policy set --updated-by	MC_STORE_ADMIN_UPDATED_BY	admin	policy audit author
policy list --tenant	MC_STORE_ADMIN_TENANT	none	optional tenant filter
policy set --route-topk	MC_STORE_ADMIN_ROUTE_TOPK	unchanged	WRH route-authority fanout
policy set --route-control	MC_STORE_ADMIN_ROUTE_CONTROL	unchanged	route control mode
policy set --max-bytes	MC_STORE_ADMIN_MAX_BYTES	unchanged	tenant quota byte limit
policy set --max-objects	MC_STORE_ADMIN_MAX_OBJECTS	unchanged	tenant quota object limit
policy set --max-remote-batch-items-per-tenant	MC_STORE_ADMIN_MAX_REMOTE_BATCH_ITEMS_PER_TENANT	unchanged	routed batch per-tenant item cap
policy set --max-remote-batch-bytes	MC_STORE_ADMIN_MAX_REMOTE_BATCH_BYTES	unchanged	routed batch byte cap
policy set --max-remote-batch-burst-items	MC_STORE_ADMIN_MAX_REMOTE_BATCH_BURST_ITEMS	unchanged	routed batch burst item cap
policy set --max-inflight-bytes-per-batch	MC_STORE_ADMIN_MAX_INFLIGHT_BYTES_PER_BATCH	unchanged	per-batch inflight byte cap
policy set --default-replica-count	MC_STORE_ADMIN_DEFAULT_REPLICA_COUNT	unchanged	default placement replica count
policy set --prefer-local	MC_STORE_ADMIN_PREFER_LOCAL	unchanged	default local placement preference
policy set --prefer-alloc-in-same-node	MC_STORE_ADMIN_PREFER_ALLOC_IN_SAME_NODE	unchanged	default same-node allocation preference
policy set --preferred-storage-owners	MC_STORE_ADMIN_PREFERRED_STORAGE_OWNERS	unchanged	comma-separated preferred storage owners
policy set --preferred-segments	MC_STORE_ADMIN_PREFERRED_SEGMENTS	unchanged	comma-separated preferred segments
quota object --key	MC_STORE_ADMIN_KEY	required	logical object key
quota reservations --state	MC_STORE_ADMIN_RESERVATION_STATE	none	reservation state filter
quota abort --reservation-id	MC_STORE_ADMIN_RESERVATION_ID	required	quota reservation id
quota abort/reconcile --dry-run	MC_STORE_ADMIN_DRY_RUN	false	inspect without changing metadata
tracing on --endpoint	MC_STORE_ADMIN_TRACING_ENDPOINT	runtime default	OTLP endpoint passed to store tracing enablement
tracing on --sample-ratio	MC_STORE_ADMIN_TRACING_SAMPLE_RATIO	runtime default	tracing sample ratio passed to store tracing enablement
tracing on/off/flush --timeout-ms	MC_STORE_ADMIN_TRACING_TIMEOUT_MS	2000	per-node admin fanout timeout
tracing on/off/flush --max-targets	MC_STORE_ADMIN_TRACING_MAX_TARGETS	1024	maximum store-node fanout count for one request

Boolean admin environment variables use falsey parsing: 0, false, no, and off disable the flag; other non-empty values enable it.

Admin route-migration queue

The standalone admin HTTP server keeps explicit route-migration tasks in process memory and retries transient executor failures automatically while the server remains alive.

Environment knobs:

Variable	Default	Meaning
MC_STORE_ADMIN_MIGRATION_MAX_RETRIES	5	default retry budget for tasks that do not override max_retries
MC_STORE_ADMIN_MIGRATION_RETRY_BASE_DELAY_MS	3000	base delay used by the admin retry backoff
MC_STORE_ADMIN_MIGRATION_RETRY_MAX_DELAY_MS	30000	maximum delay cap for the admin retry backoff
MC_STORE_ADMIN_MIGRATION_POLL_INTERVAL_MS	100	background admin polling interval for task dispatch and executor status refresh

Notes:

• these knobs are read by mooncake-store-rs-admin server through AdminService::from_config(...)

• the same defaults apply to tasks submitted through the long-lived admin HTTP server

• task state is not persisted in metadata, so these settings control a live in-memory queue rather than a durable scheduler

StoreClientBuilder

StoreClientBuilder is the main construction surface for Rust clients.

Method or Field	Default	Notes
new(metadata, stable_id)	required	stable_id is the long-lived node identity; the metadata backend assigns the epoch atomically on every build(...)
compatibility(...)	CompatibilityDescriptor::default()	controls compatibility matching
rpc_address(...)	empty	filled from transport when possible
segment_name(...)	none	filled from transport when possible
state(...)	Standby	use Active for serving clients
tenant(...)	"default"	default tenant for request builders and startup policy lookup
local_memory(...)	LocalMemoryConfig::default()	storage and scratch memory layout
transport(...) / with_tent(...)	none	required for remote transfer paths; low-level Rust transport is wired explicitly
transport_factory(...)	none	used to create transports for peers
routed_writes(...)	disabled	enables routed placement
route_control(...)	EmbeddedWrh	cluster-level route-control mode (EmbeddedWrh or MetadataOnly); fixed at startup, not overridable per-tenant
route_topk(...)	2	compatibility fallback WRH route-authority fanout; metadata tenant policy wins when present; must be >= 2
live_client_sync_interval(...)	1s	background refresh interval for the live-client membership snapshot; 0 disables the worker

System-managed behavior:

• the builder adds route=true when missing

• the control-plane address label is published automatically

• if transport is present, RPC address and segment name are inferred when possible

• build(...) prewarms the live-client membership snapshot before serving requests

• the background membership worker refreshes that snapshot after startup

LocalMemoryConfig

LocalMemoryConfig controls the memory that a client exposes to the store.

Field	Default	Meaning
storage_bytes	64 MiB	capacity for object storage
scratch_bytes	4 MiB	temporary space for transfer staging
location	"cpu:0"	placement string passed into transport registration
tags	["dram"]	tags published with the segment
alignment	64	allocation alignment
reclaim_grace_ms	1000	delayed reclaim window for scheduled cleanup; storage-owner eviction pressure may reclaim stale/orphan local bytes earlier after an authoritative active route no longer references them; route-missing ghosts are not released without a separate confirmed-absent proof
eviction_high_watermark_percent	90	start background eviction when local usage reaches this percentage
eviction_low_watermark_percent	80	stop background eviction after usage falls to this percentage
eviction_poll_interval	100ms	background storage-owner eviction polling interval; 0 disables the worker
hugepage_enabled	None	override hugepage enablement for local storage and scratch
hugepage_size_bytes	None	override hugepage size; 2 MiB and 1 GiB are supported

Validation rules:

• scratch_bytes must be greater than zero

• location must not be empty

• eviction_high_watermark_percent must be in 1..=100

• eviction_low_watermark_percent must be lower than eviction_high_watermark_percent

• hugepage size must be either 2 MiB or 1 GiB

storage_bytes may be 0. This is how scratch-only rw clients are configured when storage and inference are deployed as separate roles.

Storage-role normalization:

• storage=true requires storage_bytes > 0

• if storage_bytes=0 and the storage label is missing, the runtime normalizes it to storage=false

• a scratch-only client may still enable routed_writes; placement will choose live storage=true runtimes and will not use stale segment metadata as a candidate source

Hugepage behavior:

• if hugepage_enabled is Some(true), the client allocates native local memory with MAP_HUGETLB

• if hugepage_size_bytes is set, hugepage mode is implicitly enabled

• if both fields are None, the runtime falls back to MC_STORE_USE_HUGEPAGE and MC_STORE_HUGEPAGE_SIZE

Compatibility entrypoints expose the same background eviction watermarks:

Surface	High watermark	Low watermark
Standalone client	--eviction-high-watermark-percent / MC_STORE_RS_EVICTION_HIGH_WATERMARK_PERCENT	--eviction-low-watermark-percent / MC_STORE_RS_EVICTION_LOW_WATERMARK_PERCENT
Python setup(...)	eviction_high_watermark_percent= or MC_STORE_RS_EVICTION_HIGH_WATERMARK_PERCENT	eviction_low_watermark_percent= or MC_STORE_RS_EVICTION_LOW_WATERMARK_PERCENT
Python config dict	eviction_high_watermark_percent	eviction_low_watermark_percent
mooncake-store-rs-bench	--eviction-high-watermark-percent / MC_STORE_RS_EVICTION_HIGH_WATERMARK_PERCENT	--eviction-low-watermark-percent / MC_STORE_RS_EVICTION_LOW_WATERMARK_PERCENT

Startup registration behavior:

• classic_te + RDMA keeps startup storage-region planning NUMA-aware when location targets host memory and numa_aware remains enabled

• only transports that opt into parallel startup registration fan out initial storage into multiple segments; other transports keep the historical single-segment startup layout

• when startup planning produces multiple initial storage segments, the runtime allocates and registers those extra segments in parallel, then publishes the full local segment set after local registration completes

• under classic_te + RDMA, startup storage registration pre-touches pages before MR registration when the total startup storage registration volume reaches 4 GiB or more

• ordinary scratch-region registration still follows the transport registration limit split logic; the startup fast path is only for storage segments

Transport Backend Selection

The compatibility layer supports two real data-plane backends:

Backend	Compatibility Value	Notes
TENT	tent	opt-in compatibility backend
Classic TE	classic_te	runtime-selectable compatibility backend

Selection surfaces:

Surface	Knob
Environment	`MC_STORE_RS_TRANSPORT_BACKEND=tent
Standalone client	`–transport-backend tent
Python compatibility	transport_backend="tent" or transport_backend="classic_te"

Notes:

• explicit CLI or Python values override the environment variable

• when no explicit selection is present, the compatibility layer defaults to classic_te

• classic, classic-te, and te are accepted as compatibility aliases by the parser

• set MC_STORE_RS_GID_INDEX=<n> when classic_te over RDMA must use a non-default RoCE GID index; Store-RS forwards it to upstream MC_GID_INDEX

• for classic_te with Redis-backed transport metadata, Store-RS derives the upstream TE Redis key prefix from the default tenant and forwards tenants/<tenant> as the Mooncake metadata cluster id

• for classic_te with P2PHANDSHAKE, segment metadata keeps the configured logical segment_name and also publishes the TE transport_endpoint (ip:rpc_port) used when peers open the segment

• low-level Rust transport construction remains explicit; runtime backend selection is only a compatibility-layer feature

• current upstream SGLang Mooncake integration does not forward transport_backend from --hicache-storage-backend-extra-config; use MC_STORE_RS_TRANSPORT_BACKEND when SGLang real mode must select tent or classic_te

Reconnect behavior:

• both backends repair local transport metadata after Redis connectivity returns and the heartbeat repair path runs

• classic_te recreates its transport runtime before republishing local buffers

• tent also recreates its transport runtime before re-registering the local buffers it still owns

• peer restarts that invalidate a cached remote segment handle are repaired on demand: the read path drops the stale handle, reopens by the segment’s transport open name, and only quarantines that runtime if the fresh reopen still fails

• segment metadata lookup starts from live clients. In Redis, segment records share the owner client hash TTL with the lease; in every backend, a leftover segment record whose owner lease is gone is ignored for allocation, preferred-segment resolution, and HTTP remote-runtime resolution.

• routed writes apply the same stale-handle refresh once before escalating to outer soft-pin retry or failover, so a restarted live peer does not poison the cached remote-segment state

• if Redis restarts from an empty dataset, surviving storage clients republish both lease state and segment metadata into their client resource hash during recovery

• local memory registration refreshes the runtime lease for at least 30 seconds before publishing segment metadata, so slow startup registration does not publish into an expired Redis client resource hash

• requests that arrive while Redis is unavailable can still fail fast; recovery is designed for self-healing after metadata service returns, not for serving through a metadata blackout

Route Control

Mode	Default	Behavior
RouteControlMode::EmbeddedWrh	yes	client-side route authority selection with mirrored top-k publication, ranked authority reads, and a prewarmed background-refreshed membership snapshot
RouteControlMode::MetadataOnly	no	route reads and writes go directly to metadata

Use MetadataOnly for bring-up and debugging. Use EmbeddedWrh for normal deployments.

Embedded route authorities maintain in-memory indexes for scope and reuse-identity lookups. Hot-path existence and prefix-reuse queries resolve through exact route-authority reads instead of scanning the metadata backend or the full local route map. Batched existence checks stop at the mirrored route_topk authority set, so common misses do not fan out to every live authority.

Route authority policy

route_topk controls how many WRH-ranked route authorities each key uses.

• the highest-ranked authority is the CAS primary

• the remaining route_topk - 1 authorities are mirrors used for read repair and mirror publication

• route_topk is not the same as write-side replica_count

• the runtime rejects route_topk < 2

Startup bootstrap is metadata-authoritative:

• admin-managed tenant policy is the preferred source for tenant-scoped routing

• runtime-local route_topk is a bootstrap/compatibility fallback (tenant policy overrides it)

• route_control is always determined by the CLI setting (cluster-level, not per-tenant)

• if the metadata keyspace has no default route policy yet, the first successful client writes it with create-if-absent semantics

• startup then resolves the effective policy for the client’s default tenant: tenant override first, otherwise the default cluster policy

• later clients must match that effective policy or startup fails immediately

Use mooncake-store-rs-admin policy set --tenant <tenant> --route-topk <n> --route-control <mode> when one tenant in a shared metadata keyspace needs a different route-authority policy.

Routed Placement

Routed placement is enabled through:

let planner = PlacementPlanner::new(metadata.clone()).require_label("storage", "true");
let client = StoreClientBuilder::new(metadata, "router")
    .routed_writes(planner, 2);

PlacementPlanner

Method	Default	Meaning
scope_label("pool")	pool	label key used to scope candidate sets
require_label(key, value)	none	filters placement candidates

Planner behavior:

• only active leases are considered

• compatibility descriptors must match

• candidates are ranked with rendezvous hashing

ReplicationPolicy

ReplicationPolicy overrides placement behavior for a single request.

Field	Default	Meaning
replica_count	None	use the client default
with_soft_pin	false	request soft pin behavior
preferred_segments	empty	request-scoped segment preference; hard by default, best-effort when with_soft_pin=true
preferred_storage_owners	empty	prefer specific storage owners
prefer_alloc_in_same_node	false	bias allocation toward the same node
prefer_local	true	prefer the local node before remote spillover

If replica_count is not set:

• local-only clients default to one replica

• routed clients default to the replica count passed to routed_writes(...)

Keep the distinction clean:

• replica_count controls how many data replicas a write publishes

• route_topk controls how many route authorities keep mirrored route metadata

• request-level preferred_segments is the explicit pinning surface for a single write

• tenant policy placement.preferred_segments is only a default hint and falls back when the segment is missing or stale

• tenant-policy preferred_segments must use the exact active segment_name; a stable_id is not a valid segment identifier

Labels and Naming

The runtime and examples use these conventions:

Key	Purpose
storage	opt a client into routed placement candidate sets and enable local background/synchronous eviction when it owns storage memory
pool	default placement scope label
route_scope	optional route-authority scope

Recommended practice:

• treat stable_id as the persistent node identity

• treat segment_name as the current process-owned segment identity

• assign a distinct metadata keyspace per deployment

Metadata Backends

Rust

Rust code constructs metadata backends directly:

• RedisMetadataBackend

• EtcdMetadataBackend

• InMemoryMetadataBackend

Both Redis and etcd backends accept MetadataKeyspace.

Python compatibility layer

The Python compatibility layer accepts metadata URLs:

Scheme	Meaning
redis://host:port/db	Redis store metadata
etcd://host1:2379,host2:2379	etcd store metadata

Notes:

• HTTP metadata endpoints are not supported

• Redis store metadata accepts URL-embedded credentials or MC_REDIS_USERNAME / MC_REDIS_PASSWORD

• when metadata_url (arg7) is etcd, the Transfer Engine still needs its own metadata at transport_metadata_url (arg2): pass P2PHANDSHAKE for classic_te (default) or redis://... for tent

Redis authentication

Redis compatibility list operations are fail-closed and bounded. Indexed sets accept at most 4096 members per call, while legacy key migration scans accept at most 64 pages and 4096 matching keys. Larger namespaces must use an exact or paginated maintenance surface instead of relying on a collect-all list.

Use MC_REDIS_PASSWORD for password-only Redis deployments, including cloud Redis instances that authenticate the default user:

export MC_REDIS_PASSWORD='<redis-password>'
mooncake-store-rs-client \
  --local-hostname 127.0.0.1 \
  --metadata-url redis://redis.example.com:6379/0 \
  --stable-id store-a

Use MC_REDIS_USERNAME together with MC_REDIS_PASSWORD when Redis ACLs require a named user:

export MC_REDIS_USERNAME='<redis-username>'
export MC_REDIS_PASSWORD='<redis-password>'

URL-embedded credentials are also accepted:

redis://username:password@redis.example.com:6379/0

If both forms are present, credentials in the URL take precedence. Prefer environment variables when passwords contain reserved URL characters such as @, /, : or #; URL-embedded credentials must be percent-encoded. Route namespaces redact URL credentials before they are used for routing metadata identity.

For Redis 5 password-only deployments, the metadata backend also tolerates connections where a username was supplied by configuration but the server accepts only legacy AUTH <password>; the client retries with password-only auth after the server rejects the username form.

Standalone Client Configuration

mooncake-store-rs-client run accepts its runtime knobs either as CLI flags or as environment variables. Explicit CLI flags override environment values. For environment-only startup, keep the run subcommand because the root command with no arguments still renders help.

CLI flag	Environment variable	Default	Meaning
--local-hostname	MOONCAKE_LOCAL_HOSTNAME	required	hostname or IP published for this runtime
--metadata-url (alias --metadata_url)	MC_STORE_RS_METADATA_URL	required	Store-RS metadata URL (redis://... or etcd://...).
--transport-metadata-url (alias --transport_metadata_url)	MC_STORE_RS_TRANSPORT_METADATA_URL	P2PHANDSHAKE	Transfer Engine metadata input (redis://... or P2PHANDSHAKE). Defaults to P2PHANDSHAKE (classic_te peer handshake); tent requires redis://....
--storage-bytes	MC_STORE_RS_STORAGE_BYTES	67108864	local storage capacity published by this runtime
--scratch-bytes	MC_STORE_RS_SCRATCH_BYTES	4194304	local scratch capacity for transfer staging
--protocol	MOONCAKE_PROTOCOL	tcp	transport protocol such as tcp or rdma
--rdma-devices	MC_STORE_RS_RDMA_DEVICES	empty	RDMA device list passed through compatibility setup
--transport-rpc-port / --rpc-server-port	MC_STORE_RS_TRANSPORT_RPC_PORT	backend chooses	fixed real data-plane transport port
--transport-backend	MC_STORE_RS_TRANSPORT_BACKEND	classic_te	real transport backend; accepts tent, classic_te, classic-te, classic, or te
--stable-id	MC_STORE_RS_STABLE_ID	generated	persistent runtime identity
--initial-state	MC_STORE_RS_INITIAL_STATE	active	startup lifecycle state
--tenant	MC_STORE_RS_TENANT	default	default tenant scope
--domain	MC_STORE_RS_DOMAIN	default	default domain scope for scoped object identity
--object-set	MC_STORE_RS_OBJECT_SET	default	default object-set scope for scoped object identity
--label key=value	MC_STORE_RS_LABELS	none	comma-separated runtime labels such as pool=a,storage=true
--routed-writes	MC_STORE_RS_ROUTED_WRITES	false	enable routed writer mode; env accepts falsey values such as 0, false, no, or off, and treats other non-empty values as true
--replica-count	MC_STORE_RS_REPLICA_COUNT	1	default routed-writer replica count
--route-topk	MC_STORE_RS_ROUTE_TOPK	2	WRH route-authority fanout; must be >= 2
--keyspace	MC_STORE_RS_KEYSPACE	default keyspace	metadata keyspace
--local-segment-name	MC_STORE_RS_LOCAL_SEGMENT_NAME	generated from transport/runtime	explicit local segment name
--lease-ttl-ms	MC_STORE_RS_LEASE_TTL_MS	30000	lease TTL published by the standalone heartbeat loop
--heartbeat-interval-ms	MC_STORE_RS_HEARTBEAT_INTERVAL_MS	30000, normalized below TTL	heartbeat loop interval
--request-timeout-ms	MC_STORE_RS_REQUEST_TIMEOUT_MS	65000	dispatcher and routed operation timeout
--startup-timeout-ms	MC_STORE_RS_STARTUP_TIMEOUT_MS	adaptive	compatibility-managed registration timeout override
--heartbeat-timeout-ms	MC_STORE_RS_HEARTBEAT_TIMEOUT_MS	15000	heartbeat publish timeout
--transfer-stall-timeout-ms	MC_STORE_RS_TRANSFER_STALL_TIMEOUT_MS	10000	transfer no-progress timeout
--metrics-addr	MC_STORE_RS_METRICS_ADDR	disabled	in-process metrics bind address
--client-server-address	MC_STORE_RS_CLIENT_SERVER_ADDRESS	disabled	dummy compatibility gRPC server address
--use-hugepage	MC_STORE_USE_HUGEPAGE	disabled	enable hugepage-backed local store memory; standalone env accepts falsey values such as 0, false, no, or off
--hugepage-size	MC_STORE_HUGEPAGE_SIZE	backend default	hugepage size such as 2M or 1G
--trace-filter	MC_STORE_RS_TRACE_FILTER	tracing default	standalone tracing filter
--route-control	MC_STORE_RS_ROUTE_CONTROL	embedded_wrh	route-control fallback; accepts embedded_wrh / embedded-wrh or metadata_only / metadata-only
--drain-on-exit	MC_STORE_RS_DRAIN_ON_EXIT	false	drain owned routes during graceful shutdown; env accepts falsey values such as 0, false, no, or off, and treats other non-empty values as true

mooncake-store-rs-client stats also supports MC_STORE_RS_STATS_SERVER for --server and MC_STORE_RS_STATS_JSON for --json; the JSON switch uses the same falsey-value parsing as the standalone runtime boolean flags.

Python Compatibility Configuration

MooncakeDistributedStore.setup(...) accepts the core store knobs plus Python-specific convenience parameters.

Important Python-only compatibility knobs:

tenant, domain, and object_set form the default namespace scope used by Python compatibility read/write operations when a request does not provide a more specific scope. route_topk is kept for compatibility, but admin-managed tenant policy in metadata is the preferred place to author tenant-scoped routing policy. route_control is a cluster-level deployment setting fixed at startup.

Parameter	Meaning
stable_id	persistent client identity
tenant	default tenant scope
domain	default domain scope; falls back to default when omitted or empty
object_set	default object-set scope; falls back to default when omitted or empty
labels	lease labels such as pool and storage
routed_writes	enable routed placement from Python
replica_count	default replica count when routed writes are enabled
route_topk	WRH route-authority fanout; must match the policy already stored in the metadata keyspace
transport_backend	choose tent or classic_te for the real transport runtime
transport_rpc_port	fixed real data-plane TCP port for real-mode peers
use_hugepage	enable hugepage-backed local store memory
hugepage_size	hugepage size for local store memory; accepts 2MB or 1GB

The config-dict path accepts the same compatibility knobs. For port pinning, both transport_rpc_port and rpc_server_port map to the real-mode backend data-plane port.

local_hostname accepts either:

• a plain hostname such as 10.0.0.15

• host:port, which is normalized into local_hostname=host plus transport_rpc_port=port

If local_hostname already embeds a port and transport_rpc_port or rpc_server_port is also provided, the values must match. The compatibility layer rejects mismatches instead of silently publishing an invalid real-mode endpoint.

Port role reminder:

• transport_rpc_port / rpc_server_port is used by real clients and maps to the selected backend rpc_server_port

• client_server_address belongs to the dummy compatibility server and is not used by real-mode peers

• metrics_addr only exposes /metrics

For cross-host or cross-container real-mode deployments, set a reachable local_hostname together with a fixed transport_rpc_port.

Setup positional layout

setup(local_hostname, transport_metadata_url, global_segment_size, local_buffer_size, protocol, rdma_devices, metadata_url).

• transport_metadata_url — Transfer Engine metadata input. Accepts redis://... or P2PHANDSHAKE. Default is P2PHANDSHAKE wherever a default is expressible: the Python dict-form (the MC_STORE_RS_TRANSPORT_METADATA_URL env is honored as fallback before the default), the standalone CLI (no --transport-metadata-url flag and no env set), and the bench. The Python positional setup(...) requires it explicitly because Python disallows a defaulted positional before a required positional; pass the literal string "P2PHANDSHAKE" to use the default. The dict-form also accepts the upstream Mooncake key metadata_server as an alias.

• metadata_url — Store-RS metadata URL. Required. Accepts redis://... or etcd://.... The dict-form setup({...}) also accepts the upstream Mooncake keys master_server, master_server_addr, and master_server_address interchangeably as aliases; setup() raises a clear TypeError if none is provided.

transport_metadata_url is for the Transfer Engine only and never influences Store-RS routing or metadata decisions.

The compatibility client defaults lease_ttl_ms to 30000. Keep the heartbeat interval comfortably below that TTL so dead peers converge quickly without triggering avoidable churn.

Timeouts now converge on one shared compatibility helper so the Python wrapper, standalone client, dummy client, and transport builder all resolve the same knobs with the same precedence rules:

• CLI override, when a standalone binary exposes the flag

• explicit environment variable

• built-in default

The compatibility layer exposes five timeout scopes:

Knob	Default	Scope	Meaning
request_timeout_ms / --request-timeout-ms / MC_STORE_RS_REQUEST_TIMEOUT_MS	65000	dispatcher request budget, routed read/write request budget, dummy fallback	outer per-request deadline shared across replica failover
startup_timeout_ms / --startup-timeout-ms / MC_STORE_RS_STARTUP_TIMEOUT_MS	max(20000, ceil(registration_bytes / 1 GiB) * 1000)	startup register_local_memory, real-mode register_buffer, and compatibility-side buffer unregister	registration-specific timeout budget; explicit override wins over the adaptive default
heartbeat_timeout_ms / --heartbeat-timeout-ms / MC_STORE_RS_HEARTBEAT_TIMEOUT_MS	15000	standalone client heartbeat / state publish	dedicated health-channel publish budget
transfer_stall_timeout_ms / --transfer-stall-timeout-ms / MC_STORE_RS_TRANSFER_STALL_TIMEOUT_MS	10000	TENT / classic transfer engine	inner stall detector for one transfer slice or batch wait
MC_STORE_RS_DUMMY_RPC_TIMEOUT_MS	65000	dummy gRPC client	dummy RPC budget; falls back to request_timeout_ms when unset

Design intent:

• request timeout is the outer deadline for one logical store request

• startup timeout is the registration budget for compatibility-managed memory registration work; when unset, the runtime derives it from the current registration size with a conservative 20s floor so small scratch-only startup does not fail under host-side contention

• heartbeat timeout is independent, so a slow health publish does not block the shared data path

• transfer stall timeout is not a whole-request timeout; it only detects no-progress transport stalls

• dummy RPC timeout follows request timeout unless explicitly overridden, so compatibility scripts do not hang forever on one slow server

Standalone clients now retry failed heartbeat publishes instead of exiting on the first timeout. When cloud metadata links or long transfers need more head room, raise the matching timeout scope instead of stretching every timeout.

Control-plane RPC clients run on a dedicated shared Tokio runtime instead of a single global caller lock. Use MC_STORE_RS_CONTROL_PLANE_THREADS to tune the client worker count when a deployment needs more concurrent route / allocator RPCs or when an embedded client should keep its thread footprint smaller. The default is 2; values must be positive integers.

Each runtime also serves peer control-plane requests on a separate multi-thread Tokio runtime. Use MC_STORE_RS_CONTROL_PLANE_SERVER_THREADS to tune the server worker count when routed writers create high allocator / route-authority fanout. The default is 4; values must be positive integers.

Python Local Hot Cache

The local hot cache lives inside the Python compatibility runtime and the standalone dummy daemon. It stays disabled unless MC_STORE_LOCAL_HOT_CACHE_SIZE is set to a positive integer.

Variable	Default	Meaning
MC_STORE_LOCAL_HOT_CACHE_SIZE	disabled	total cache capacity in bytes; unset or invalid values disable the cache
MC_STORE_LOCAL_HOT_BLOCK_SIZE	16777216 (16 MiB)	block size in bytes and the maximum payload size that can be cached
MC_STORE_LOCAL_HOT_CACHE_USE_SHM	disabled	set to 1 to back cached payloads with shm so dummy clients on the same daemon can reuse them

Behavior notes:

• effective cache capacity is floor(total_size / block_size) blocks

• values larger than block_size bypass the cache instead of being partially cached

• successful read misses populate the cache from the fetched value

• successful local writes and deletes invalidate the matching local cache entry on that daemon

• Python compatibility runtimes partition local entries by effective namespace scope, including tenant, domain, and object_set, so different object sets do not reuse the same cached value even inside one process

• shm mode shares payload bytes with dummy clients connected to the same mooncake-store-rs-client, while LRU metadata, generations, and pins remain private to the daemon

• cache entries are daemon-local only and are never published to Redis or etcd

• metadata keyspace remains the authoritative read/write isolation boundary; cache partitioning does not make objects visible across keyspaces

Read-side Membership and Failure Semantics

The client keeps membership refresh out of the steady-state request path.

• build(...) prewarms a live-client lease snapshot before serving reads

• a background worker refreshes that snapshot after startup

• steady-state reads reuse the cached snapshot instead of issuing inline membership refreshes

• owners in Active and Draining state remain readable

• owners that become offline or hit transport/metadata failures are marked suspect and skipped on later reads

• suspect owners observe a minimum quarantine, then recover only after a fresh lease heartbeat or control-plane endpoint change appears in the membership snapshot

• when fallback to another replica succeeds, the client best-effort prunes unreadable owners from the route with CAS

MooncakeHostMemAllocator(...) exposes:

Parameter	Meaning
use_hugepage	request hugepage-backed shm regions
hugepage_size	hugepage size for shm regions; accepts 2MB or 1GB

When the native extension is unavailable, the pure-Python allocator falls back to mmap and does not support hugepages.

Environment Variables

The current repository uses these environment variables.

Variable	Used By	Meaning
MOONCAKE_LOCAL_HOSTNAME	standalone client and bench	hostname or IP published by the runtime
MOONCAKE_PROTOCOL	standalone client and bench	transport protocol such as tcp or rdma
MC_STORE_RS_TRANSPORT_BACKEND	compatibility layer, standalone client, Python wrapper	select tent or classic_te as the default real transport backend
MC_STORE_RS_METADATA_URL	standalone client, standalone admin, and bench	Store-RS metadata URL. Backs --metadata-url on mooncake-store-rs-client, mooncake-store-rs-admin, and mooncake-store-rs-bench. The Python wrapper setup({...}) dict-form also honors it as a fallback when neither metadata_url nor any of the upstream aliases (master_server / master_server_addr / master_server_address) is provided.
MC_STORE_RS_TRANSPORT_METADATA_URL	standalone client, bench, Python wrapper dict-form fallback	Transfer Engine metadata input. Backs --transport-metadata-url on mooncake-store-rs-client and mooncake-store-rs-bench. The Python wrapper setup({...}) dict-form honors it as a fallback when neither transport_metadata_url nor the upstream alias metadata_server is provided (default value P2PHANDSHAKE).
MC_STORE_RS_STORAGE_BYTES	standalone client	local storage bytes for mooncake-store-rs-client run
MC_STORE_RS_SCRATCH_BYTES	standalone client and bench	local scratch bytes for compatibility-managed clients
MC_STORE_RS_RDMA_DEVICES	standalone client and Rust e2e	RDMA device list
MC_STORE_RS_KEYSPACE	standalone client, standalone admin, and Python wrapper setup fallback	metadata keyspace used when SGLang cannot pass keyspace; this also defines Python compatibility read/write visibility and local hot-cache partitioning
MC_STORE_RS_STABLE_ID	standalone client and Python wrapper setup fallback	stable client id used when SGLang cannot pass stable_id
MC_STORE_RS_INITIAL_STATE	standalone client and Python wrapper setup fallback	initial lifecycle state, for example active, standby, draining, or offline
MC_STORE_RS_TENANT	standalone client, Python wrapper setup fallback, and bench	default tenant used when SGLang cannot pass tenant
MC_STORE_RS_DOMAIN	standalone client and Python wrapper setup fallback	default domain used when SGLang cannot pass domain
MC_STORE_RS_OBJECT_SET	standalone client and Python wrapper setup fallback	default object set used when SGLang cannot pass object_set; treated as an opaque namespace component
MC_STORE_RS_LABELS	standalone client and Python wrapper setup fallback	standalone labels as comma-separated key=value pairs; Python also accepts a JSON object
MC_STORE_RS_ROUTED_WRITES	standalone client and Python wrapper setup fallback	enable routed writer mode when set to 1, true, yes, or on; standalone also treats 0, false, no, or off as false
MC_STORE_RS_REPLICA_COUNT	standalone client, Python wrapper setup fallback, and bench	default routed-writer replica count
MC_STORE_RS_ROUTE_TOPK	standalone client, Python wrapper setup fallback, and bench	WRH route-authority fanout; must be >= 2
MC_STORE_RS_ROUTE_CONTROL	standalone client, Python wrapper setup fallback, and bench	route control mode, usually embedded_wrh
MC_STORE_RS_GID_INDEX	compatibility layer, standalone client, Python wrapper, and bench	classic_te RDMA GID index override; forwarded to upstream MC_GID_INDEX
MC_STORE_RS_TRANSPORT_RPC_PORT	standalone client and Python wrapper setup fallback	fixed real data-plane transport port
MC_STORE_RS_LOCAL_SEGMENT_NAME	standalone client and Python wrapper setup fallback	explicit local segment name
MC_STORE_RS_EXPIRES_AT_MS	Python wrapper setup fallback	absolute lease expiry timestamp in milliseconds
MC_STORE_RS_LEASE_TTL_MS	standalone client	lease TTL for the standalone heartbeat loop
MC_STORE_RS_HEARTBEAT_INTERVAL_MS	standalone client	heartbeat interval for the standalone heartbeat loop
MC_STORE_RS_REQUEST_TIMEOUT_MS	standalone client, Python compatibility runtime, applications	outer per-request deadline for dispatcher requests and routed client operations
MC_STORE_RS_STARTUP_TIMEOUT_MS	standalone client, Python compatibility runtime, applications	explicit override for compatibility-managed memory registration work; when unset the runtime uses max(20s, ceil(registration_bytes / 1 GiB))
MC_STORE_RS_HEARTBEAT_TIMEOUT_MS	standalone client, Python compatibility runtime, applications	dedicated dispatcher timeout for heartbeat publish
MC_STORE_RS_TRANSFER_STALL_TIMEOUT_MS	standalone client, Python compatibility runtime, applications	inner transfer stall detector for TENT / classic TE
MC_STORE_RS_TRANSFER_TIMEOUT_MS	legacy compatibility alias	deprecated alias of MC_STORE_RS_TRANSFER_STALL_TIMEOUT_MS
MC_STORE_RS_DUMMY_RPC_TIMEOUT_MS	dummy compatibility clients	dummy gRPC timeout; falls back to MC_STORE_RS_REQUEST_TIMEOUT_MS when unset
MC_STORE_RS_CLIENT_SERVER_ADDRESS	standalone client	dummy compatibility gRPC server address
MC_STORE_RS_DRAIN_ON_EXIT	standalone client	drain owned routes during graceful shutdown; accepts falsey values such as 0, false, no, or off
MC_STORE_RS_STATS_SERVER	standalone client stats command	server address used by mooncake-store-rs-client stats --server
MC_STORE_RS_STATS_JSON	standalone client stats command	emit compact JSON from the stats command; accepts falsey values such as 0, false, no, or off
MC_STORE_RS_CONTROL_PLANE_THREADS	standalone client, Python compatibility runtime, applications	worker thread count for the shared control-plane RPC runtime; default 2; must be > 0
MC_STORE_RS_CONTROL_PLANE_SERVER_THREADS	standalone client, Python compatibility runtime, applications	worker thread count for the embedded control-plane gRPC server; default 4; must be > 0
MC_STORE_LOCAL_HOT_CACHE_SIZE	Python compatibility runtime, standalone dummy daemon, and local e2e	total byte budget for the daemon-local hot read cache; unset disables it
MC_STORE_LOCAL_HOT_BLOCK_SIZE	Python compatibility runtime, standalone dummy daemon, and local e2e	cache block size and maximum cached object size; default 16777216 (16 MiB)
MC_STORE_LOCAL_HOT_CACHE_USE_SHM	standalone dummy daemon, dummy compatibility clients, and local e2e	set to 1 to back cached payloads with shm so dummy clients attached to the same daemon can reuse them
MC_STORE_RS_TRACE	Python wrapper setup fallback, e2e, and applications	enable tracing initialization from env
MC_STORE_RS_TRACE_FILTER	standalone client, Python wrapper setup fallback, e2e, and applications	tracing_subscriber filter string; --trace-filter overrides it for the standalone client
MC_STORE_RS_TRACE_FILE	Python real mode, standalone client, e2e, and applications	append Rust tracing logs to this file; also auto-enables Python real-client tracing
MC_STORE_RS_TRACE_SPAN_EVENTS	standalone client, Python real mode, e2e, and applications	tracing span lifecycle events; unset suppresses synthetic span close lines, close enables operation close timing logs
MC_STORE_RS_TRACE_JSONL_FILE	standalone client, Python real mode, bench, and applications	local profiling sink; appends one JSON object per completed operation span and can be changed through /tracing?file=... before enabling
MC_STORE_RS_TRACE_ITEM_METADATA	standalone client, Python real mode, bench, and applications	set to 1 to add local JSONL store.api_items.v1 records for batch_put_from and batch_get_into; records include per-item namespace, key hash/prefix, TP rank, KV kind, byte size, and status
MC_STORE_RS_TRACE_KEY_MODE	standalone client, Python real mode, bench, and applications	key representation for item metadata; default hash writes a truncated SHA-256 hash, full writes full keys for local debug only
MC_STORE_RS_OTLP_ENDPOINT	standalone client, Python real mode, bench, and applications	OTLP HTTP endpoint for Jaeger profiling; http://host:4318 is normalized to /v1/traces
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	standalone client, Python real mode, bench, and applications	standard OTLP traces endpoint; used as-is and takes precedence over MC_STORE_RS_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_ENDPOINT	standalone client, Python real mode, bench, and applications	standard base OTLP endpoint; normalized to /v1/traces
MC_STORE_RS_OTLP_TRACE	standalone client, Python real mode, bench, and applications	initial Jaeger profiling switch; unset keeps profiling off until /tracing enables it
MC_STORE_RS_OTLP_SERVICE_NAME / OTEL_SERVICE_NAME	standalone client, Python real mode, bench, and applications	Jaeger service name; default mooncake-store-rs
MC_STORE_RS_OTLP_SERVICE_INSTANCE_ID	standalone client, Python real mode, bench, and applications	Jaeger service instance id; defaults to HOSTNAME when present
MC_STORE_RS_OTLP_TIMEOUT_MS	standalone client, Python real mode, bench, and applications	OTLP export timeout; default 3000
MC_STORE_RS_OTLP_SAMPLE_RATIO	standalone client, Python real mode, bench, and applications	OTLP trace sampling ratio from 0.0 to 1.0; default 1.0, lower it for high-throughput profiling
MC_BENCH_TRACE_FILE	mooncake-store-rs-bench	append bench tracing logs to this file; bench otherwise logs to stderr and does not use MC_STORE_RS_TRACE_FILE for its own output
MC_BENCH_INTERFACES	mooncake-store-rs-bench	combined write/read interface selector; accepts <write>,<read>, <write>:<read>, or write=<...>,read=<...>; when set it overrides non-CLI interface defaults
MC_BENCH_WRITE_INTERFACE	mooncake-store-rs-bench	measured write-side bench API; put, batch_put, or batch_put_from; default batch_put_from
MC_BENCH_READ_INTERFACE	mooncake-store-rs-bench	measured read-side bench API; get, batch_get, or batch_get_into; default batch_get_into
MC_STORE_RS_METRICS_ADDR	Python wrapper setup fallback, e2e, and applications	bind address for the in-process metrics server
MC_STORE_RS_REDIS_URL	Rust e2e	metadata Redis URL
MC_STORE_RS_REDIS_PORT	local scripts and e2e	local Redis port
MC_REDIS_USERNAME	Redis metadata backends and transport Redis plugins	optional Redis ACL username
MC_REDIS_PASSWORD	Redis metadata backends and transport Redis plugins	optional Redis password; enables auth when set
MC_STORE_RS_REDIS_CONNECT_TIMEOUT_MS	Redis metadata backend	connection timeout for sync Redis metadata calls
MC_STORE_RS_REDIS_IO_TIMEOUT_MS	Redis metadata backend	read/write timeout for sync Redis metadata calls
MC_STORE_RS_REDIS_CONNECTION_POOL_SIZE	Redis metadata backend	bounded per-backend Redis connection pool size; default 16, capped at 256
MC_STORE_RS_REDIS_RETRY_ATTEMPTS	Redis metadata backend	retry count for transient Redis reconnect / IO failures on readonly and idempotent metadata operations
MC_STORE_RS_REDIS_RETRY_DELAY_MS	Redis metadata backend	delay between transient Redis retry attempts
MC_STORE_RS_VALUE_SIZE	Rust e2e	payload size for validation and benchmark loops
MC_STORE_RS_BENCH_ITERS	Rust e2e and local scripts	benchmark iteration count; default 64
MC_STORE_RS_PRINT_METRICS	Rust e2e	print the Prometheus text snapshot at the end of the run
MC_STORE_USE_HUGEPAGE	local memory, standalone client, and Python shm allocator	enable hugepage-backed allocation; standalone client treats 0, false, no, or off as an explicit disable
MC_STORE_HUGEPAGE_SIZE	local memory and Python shm allocator	hugepage size; 2MB or 1GB
MOONCAKE_STORE_RS_DIR	local scripts and Python source runtime	explicit Store-RS source tree
MOONCAKE_ROOT_DIR	local scripts and native shim builds	explicit Mooncake source tree
MOONCAKE_BUILD_DIR	local scripts and native runtime loading	explicit Mooncake CMake build output tree

mooncake-store-rs-bench reuses MC_STORE_RS_TRACE_FILTER for level control, but it has its own trace-file surface. Use MC_BENCH_TRACE_FILE for bench logs and keep MC_STORE_RS_TRACE_FILE for standalone-client or Python real-client logging.

Standalone-client heartbeat success refreshes are trace-level routine events. Heartbeat recovery and failure messages remain explicit stderr output. Standalone-client local segment publication is an info-level startup registration event. Background storage-owner eviction completion is an info-level aggregate state change. Standalone-client startup emits one debug-level state snapshot after ready. Debug output keeps bounded per-key summary samples. Trace output keeps full per-key route and eviction details. Control-plane stream open events are trace-level connection churn.

Observability

The Rust client exposes:

• init_tracing(...)

• init_tracing_from_env(...)

• render_prometheus_metrics()

• start_metrics_http_server(...)

• start_metrics_http_server_from_env(...)

• stop_metrics_http_server()

The metrics HTTP server exposes:

• GET /metrics

• GET /stats

• GET /breakdown

• GET /healthz

• GET /tracing

• GET /tracing/on

• GET /tracing/off

• GET /tracing?enabled=on&endpoint=http%3A%2F%2Fjaeger.observability.svc.cluster.local%3A4318&sample_ratio=0.05

• GET /tracing?enabled=on&file=%2Ftmp%2Fstore-rs.trace.jsonl

• GET /tracing/flush

/tracing returns JSON status for the profiler. Enabling requires at least one sink: an OTLP endpoint for Jaeger or a local JSONL file. The endpoint is locked after the exporter is initialized, because changing Jaeger backends while batch processors are running would make trace ownership ambiguous. The same lock applies to service_name, service_instance_id, and sample_ratio; set them before enabling export. The local JSONL file can be set with file=... and cleared with clear_file=true while profiling is disabled.

Set MC_STORE_RS_TRACE_ITEM_METADATA=1 with a local JSONL file to append store.api_items.v1 records next to the span records for successful batch_put_from / batch_get_into paths. These records are for offline per-key analysis and include trace_request_id, operation, tenant/domain/object set, runtime id, batch bytes, per-item key hash, readable key prefix, parsed SGLang TP rank and k/v suffix, size/read length, and status. The default MC_STORE_RS_TRACE_KEY_MODE=hash never writes full keys; use full only for local debug captures.

/breakdown returns JSON for SGLang/HiCache performance diagnosis. It includes API, phase, metadata-backend, transport, runtime, segment, and bottleneck-candidate sections. Use mooncake-store-rs-client stats --breakdown --server <host:port> for a compact human-readable summary, or add --json to print the raw JSON.

When OTLP profiling is enabled, Jaeger span names describe the performance surface instead of raw code hooks. API roots are named store.*; control-plane work is named control.*; data-plane work is named data.*; metadata backend calls are named metadata.*. Common span attributes include mooncake.phase, mooncake.flow, mooncake.request_id, mooncake.item_count, mooncake.bytes_in, mooncake.bytes_out, mooncake.replica_count, mooncake.local_target_count, and mooncake.remote_target_count. Use the Jaeger waterfall for one sampled request and the Prometheus operation histograms for the matching time-series view.

Exporter families:

• mooncake_store_operation_total, mooncake_store_operation_bytes_in_total, mooncake_store_operation_bytes_out_total, mooncake_store_operation_latency_microseconds_total, mooncake_store_operation_latency_microseconds_max

• mooncake_store_request_total, mooncake_store_request_inflight, mooncake_store_request_bytes_total

• mooncake_store_request_duration_seconds_bucket

• mooncake_store_metadata_operation_total, mooncake_store_metadata_operation_inflight, mooncake_store_metadata_operation_duration_seconds_bucket

• mooncake_store_segment_capacity_bytes, mooncake_store_segment_used_bytes

• mooncake_store_runtime_status, mooncake_store_runtime_lease_expires_at_ms

• mooncake_store_route_cas_total

• mooncake_store_segment_lifecycle_total, mooncake_store_eviction_total, mooncake_store_eviction_duration_seconds_bucket

• mooncake_store_transport_bytes_total, mooncake_store_rebalance_routes_total, mooncake_store_rebalance_bytes_total

• mooncake_store_tenant_quota_reservation_total, mooncake_store_tenant_quota_finalize_total, mooncake_store_tenant_quota_abort_total, mooncake_store_tenant_quota_reconcile_total

• mooncake_store_heartbeat_consecutive_failures, mooncake_store_heartbeat_last_success_ms

• mooncake_store_membership_refresh_total, mooncake_store_membership_refresh_duration_seconds

• mooncake_store_checksum_validation_total, mooncake_store_replication_publish_duration_seconds

• process_cpu_seconds_total, process_resident_memory_bytes, process_open_fds

All Store-RS and process metrics exported by this endpoint include tenant="<default tenant>", where the value is the process-bound tenant selected by StoreClientBuilder::tenant(...) or the corresponding compatibility-layer startup option. All Store-RS metric families use the mooncake_store_ prefix to reflect the store cluster perspective. Each runtime instance exports its own view through the /metrics endpoint regardless of its role (storage node or routed client).

Sparse operational counter families emit zero-valued baseline series for their known label set. This makes steady-state dashboards report an explicit zero for tenant quota, tenant-local eviction, preferred-segment skip, rebalance, and segment lifecycle activity until the corresponding real event occurs and increments the counter.

Metadata backend metrics are recorded around Store-RS calls into the configured MetadataBackend and are labeled by backend, operation, and result. They show whether Store-RS is waiting on or failing calls to Redis, etcd, or another metadata implementation; they are not a replacement for Redis or etcd server-internal exporters.

Storage and route-authority clients also record control-plane-derived traffic: route CAS handlers observe replication publish latency, replica-route tracking records storage-owner write bytes, and route-hit reports record storage-owner read bytes plus successful checksum-validated hits. Scratch-only benchmark workers therefore do not need to expose their own metrics endpoint for the long-running store clients to show benchmark traffic.

Recommended recording queries:

rate(mooncake_store_request_total{tenant="$tenant",operation="get",result="error"}[5m])
histogram_quantile(0.99, sum by (le, tenant, operation) (rate(mooncake_store_request_duration_seconds_bucket{tenant="$tenant"}[5m])))
rate(mooncake_store_metadata_operation_total{tenant="$tenant",backend=~"redis|etcd",result!="ok"}[5m])
histogram_quantile(0.99, sum by (le, tenant, backend, operation) (rate(mooncake_store_metadata_operation_duration_seconds_bucket{tenant="$tenant"}[5m])))
mooncake_store_segment_used_bytes{tenant="$tenant"} / mooncake_store_segment_capacity_bytes{tenant="$tenant"}

Infrastructure split:

• process-scoped facts come from this crate

• host CPU, disk, and network should come from node_exporter or cAdvisor

Practical Defaults

For local development:

• use Redis metadata

• keep route_control at EmbeddedWrh

• label storage targets with storage=true

• keep prefer_local=true unless you want remote-first behavior

For debugging:

• switch to MetadataOnly if you want all route state to flow through metadata

For upgrade and elasticity flows:

• preserve stable_id

• start the successor with the same stable_id; the metadata backend allocates the next epoch atomically

• use expand_local_memory, drain_segment, retire_segment, and evacuate_owned_replicas

---

Python Configuration

These sections supplement the client and operator settings above with Python-specific metadata, transport, and framework configuration.

Metadata URLs

The compatibility layer accepts these metadata URL forms:

Scheme	Meaning
redis://host:port/db	Redis metadata backend
etcd://host1:2379,host2:2379	etcd metadata backend

Redis authentication follows the same environment variables as the native store:

export MC_REDIS_PASSWORD='<redis-password>'
# Optional when Redis ACLs require a named user:
export MC_REDIS_USERNAME='<redis-username>'

Credentials embedded in redis://username:password@host:port/db are also accepted and take precedence over the environment variables. Prefer environment variables when passwords contain URL-reserved characters such as @.

Python setup positional layout

setup(local_hostname, transport_metadata_url, global_segment_size, local_buffer_size, protocol, rdma_devices, metadata_url).

• transport_metadata_url is forwarded to the Transfer Engine only. Accepts redis://... or P2PHANDSHAKE. Default value is P2PHANDSHAKE everywhere it can be defaulted: the Python dict-form (resolution order: dict key → MC_STORE_RS_TRANSPORT_METADATA_URL env → P2PHANDSHAKE), the standalone CLI bins (mooncake-store-rs-client / mooncake-store-rs-bench without --transport-metadata-url or env), and other defaultable surfaces. The Python positional setup(...) requires it explicitly because Python disallows defaults on a positional that precedes required positionals; pass "P2PHANDSHAKE" to opt into the default. The dict-form also accepts the upstream Mooncake key metadata_server as an alias.

• metadata_url is the Store-RS metadata URL. Accepts redis://... or etcd://... and is required. The dict-form setup({...}) also accepts the upstream Mooncake keys master_server, master_server_addr, and master_server_address interchangeably as aliases, with MC_STORE_RS_METADATA_URL env honored as a final fallback. The standalone CLI bins use the same env for their --metadata-url flag.

Important note for etcd

When metadata_url is an etcd URL, the Transfer Engine still needs its own metadata: pass either redis://... or P2PHANDSHAKE at transport_metadata_url. For classic_te, P2PHANDSHAKE (the transfer-engine peer-handshake mode) is the default; tent always needs an explicit redis://....

Python Transport Backend Selection

The compatibility layer can choose the transport backend at runtime.

Supported values:

• tent

• classic_te

Selection order:

1. explicit transport_backend=... argument to setup(...)

2. MC_STORE_RS_TRANSPORT_BACKEND

3. default classic_te

The standalone client follows the same rule, except the explicit override is --transport-backend.

SGLang Environment Fallbacks

Current upstream SGLang only forwards legacy Mooncake setup fields. When SGLang cannot pass Store-RS setup extensions, the Python wrapper reads these environment variables as fallbacks:

• MC_STORE_RS_METADATA_URL (dict-form metadata_url fallback) and MC_STORE_RS_TRANSPORT_METADATA_URL (dict-form transport_metadata_url fallback; defaults to P2PHANDSHAKE when unset)

• MC_STORE_RS_TRANSPORT_BACKEND=tent|classic_te

• MC_STORE_RS_KEYSPACE, MC_STORE_RS_STABLE_ID, MC_STORE_RS_TENANT, MC_STORE_RS_LABELS

• MC_STORE_RS_ROUTED_WRITES=1, MC_STORE_RS_REPLICA_COUNT=<n>, MC_STORE_RS_ROUTE_TOPK=<n>

• MC_STORE_RS_ROUTE_CONTROL=embedded_wrh|metadata_only

• MC_STORE_RS_TRANSPORT_RPC_PORT, MC_STORE_RS_LOCAL_SEGMENT_NAME

• MC_STORE_RS_INITIAL_STATE, MC_STORE_RS_EXPIRES_AT_MS

• MC_STORE_RS_METRICS_ADDR=host:port

• MC_STORE_RS_CONTROL_PLANE_THREADS=<n> to tune concurrent control-plane RPC client capacity; default 2

• MC_STORE_RS_CONTROL_PLANE_SERVER_THREADS=<n> to tune embedded control-plane gRPC server capacity; default 4

Explicit setup(...) arguments still take precedence. MC_STORE_RS_METRICS_ADDR starts the Python real-client /metrics endpoint after setup(...). MC_STORE_RS_LABELS accepts either JSON ({"storage":"false","pool":"rw"}) or comma-separated pairs (storage=false,pool=rw).