Features

This document explains the implemented capabilities by function rather than by crate.

Store API

Basic object operations

• put

• get

• remove

• is_exist

• get_size

• query_route

These APIs are the simplest entry points for applications that treat the store as an object service. Successful route queries and reads count as hot-object access. The client reports active route hits to the storage owners best-effort, so storage-owner CLOCK eviction sees prefix probes and completed reads before later pressure arrives. Existence checks stay boolean probes and do not report hits.

With tenant quota policy configured, single-object put uses metadata-backed reserve/finalize/abort semantics and remove applies the matching refund when the delete becomes authoritative. This gives create, overwrite, and delete a tenant-root quota state that survives concurrent writers better than the older best-effort namespace scan.

When an operator shrinks a tenant’s strict quota below its current usage, quota admission only rejects further positive growth on the constrained dimensions. Non-growing updates and same-tenant deletes stay admissible while the tenant is already over limit, so follow-up writes can still recover by deleting older objects within that same tenant and converge the namespace back under the new max_bytes / max_objects boundary instead of deadlocking on the already-over-limit state.

remove and batch_remove also retry when route delete CAS observes a fresher still-active current route that metadata can prove is the same hot-upgrade or standby-promotion successor lineage. Cleanup traffic that races with rollout-era route migration therefore converges onto the newer handoff route, while ordinary delete-versus-rewrite races, including same-payload rewrites, still fail with conflict instead of deleting the fresher rewrite.

The performance boundary stays the same: backend-owned metadata is not allowed to become a normal steady-state request-path dependency. When quota or policy needs backend truth, the runtime is expected to hide that behind exact local caches, authority indirection, or maintained indexes instead of paying a backend round trip per request.

Batch operations

• batch_put

• batch_get

• batch_remove

• batch existence checks

Batch APIs are first-class paths rather than thin wrappers around repeated single-item calls.

With tenant quota policy configured, routed batch_put now acquires metadata-backed quota reservations in deterministic scoped-key order, admits the batch all-or-nothing, publishes route CAS updates, and only finalizes quota for entries whose authoritative route publish succeeds. Reservation or publish failures abort unfinished quota state so batch retries do not inherit stale pending usage.

Buffer-Oriented I/O

Registered-buffer path

The runtime supports registering caller-owned buffers and then using them for put/get operations with less copy overhead.

Main capabilities:

• register and unregister buffers

• put_from

• batch_put_from

• batch_get_into

• registered subranges inside a larger caller-owned buffer

Remote read behavior:

• when a batch_get_into destination range is already registered, the runtime issues direct remote batch reads into that buffer instead of staging through local scratch first

• unregistered destination buffers still use the existing scratch-window planner and direct single-object fallback path when needed

• put_from and batch_put_from now preserve the same registered-buffer zero-copy contract for remote writes: the transport request sources point at the caller-registered buffer ranges instead of a scratch copy

• non-host-readable registered sources such as CUDA buffers are never exposed as CPU slices; routed writes use the registered transfer source directly, reject local/staged fallback placement, and publish replicas without a CPU payload checksum

• batch_put_from uses the same registered-buffer batch writer in both default local-write mode and explicit routed-write mode when the batch has a shared replication policy

• routed batch_put_from refreshes membership and re-ranks placement candidates on retry, so rollout-era write failures can fall forward onto newly visible live storage runtimes instead of reusing only the stale pre-retry ranking

• routed batch_put_from applies the same retry envelope to reserve-stage placement gaps, so brief hot-upgrade windows with no active storage=true owner wait for successor promotion instead of failing the batch immediately

• routed batch_put_from handles concurrent cache writers per key by treating route-CAS conflicts as successful cache insert races when an active route is already published; when the CAS response omits that active route, the client uses one exact bounded recheck for the conflicted key before surfacing a write failure

• Python compatibility batch_put_from reports best-effort per-key statuses; route-CAS conflicts are success, while true per-key write failures stay isolated to the affected item

• routed batch_put_from skips the pre-publish route preload on the accept-existing cache-write path when no tenant quota policy needs previous-route accounting; route CAS still resolves active conflicts and version-floor retries through the route authorities

• Python compatibility batch_put_from reports best-effort per-key statuses produced by the Rust batch writer; route-CAS conflicts with an active published route are success, while true per-key write failures stay isolated to the affected item without replaying the batch serially in Python

• Python compatibility registered-buffer batch restores also report best-effort per-key statuses; a route miss or dead storage owner for one entry does not poison other entries in the same restore batch

• compatibility get_into_ranges accepts a request-scoped route-query cache from higher-level planners, so tensor reconstruction can reuse one batch route lookup across prefix reads and final range reads instead of resolving the same shard keys repeatedly

• remote storage segment announcements publish explicit storage target chunks, and writers translate allocator segment offsets through those chunks instead of inferring storage from backend buffer ordering

• local reads and drain migration translate the published replica transport target offset through the storage target map before copying from local storage; allocator segment offsets remain reclaim bookkeeping only

• scratch buffers are registered for local transfer staging only and are not published as remote storage extents

• local replicas still copy into owned local segment memory; the zero-copy contract applies to the remote transport source buffer, not to local placement

• payload integrity uses a fast stable 64-bit checksum, and large batch validation fans out across multiple CPU workers so registered-buffer restore traffic does not serialize the verification tail on one core

Multi-buffer path

For fragmented payloads or caller-managed buffer layouts, the runtime supports multi-buffer request forms.

Main capabilities:

• batch_put_from_multi_buffers

• batch_get_into_multi_buffers

• HiCache-compatible dummy and real paths for the Python layer

Python Read Acceleration

Daemon-local hot cache

The Python compatibility runtime can keep recently fetched remote values in a daemon-local hot cache.

What it provides:

• populate on successful read paths such as get, get_into, batch_get, batch_get_into, and multi-buffer reads

• serve repeated reads from local memory without repeating the remote transfer

• invalidate local entries after successful writes or deletes for the same key on that daemon

• partition local entries by Python compatibility scope so different metadata keyspaces do not reuse the same cached value

• optional shm-backed payload storage so multiple dummy clients attached to the same standalone daemon can reuse the cached bytes

• bounded LRU eviction with pin protection while dummy readers hold an acquired handle

Design boundary:

• the cache is a local read accelerator only

• it does not publish route state, replica state, or metadata

• a miss falls back to the normal remote read path

• values larger than the configured cache block size bypass the cache

• metadata keyspace remains the authoritative read/write isolation boundary, and runtime coverage now includes a regression test that verifies the same object key can be written and read independently in separate keyspaces

Tensor parallel reconstruction

The Python tensor-parallel read path reconstructs missing full or shard targets from stored shard objects when no direct object already matches the requested layout.

What it provides:

• manifest-guided shard discovery for writer partitions and tensor parallelism layouts

• metadata-prefix reads for shard planning, so reconstruction can inspect dtype, shape, split dimension, and payload length without reading each full shard payload first

• request-scoped route-query reuse between shard discovery and get_into_ranges data transfer

• batch full _into reconstruction that combines multiple caller buffers into one ranged-read request where possible

• contiguous range planning for split dimension 0 and strided range planning for non-leading split dimensions

Routing

Embedded weighted rendezvous routing

Default route control uses embedded weighted rendezvous hashing.

What it provides:

• route owner selection without a dedicated master

• primary and secondary route authorities

• authority reads and compare-and-swap through the control plane

• replica removal through route-owner compare-and-swap

• mirrored-authority publication plus ranked authority reads for repair and convergence

• drain, migration, and shrink route updates through the object route CRUD/CAS facade once the affected key is known

Metadata-only routing

For simpler bring-up or debugging, the runtime can store and resolve routes directly through the metadata backend.

Explicit route migration tasks

The current implementation also exposes admin-driven explicit route migration tasks for key-level copy and move.

See Route Migration Guide for the current operator workflow, task semantics, and request examples.

What it provides:

• explicit source_segment -> target_segment(s) selection

• namespace-aware tenant/domain/object_set/key selection

• multi-target copy, including multiple explicit target segments on the same storage runtime

• single-target move

• admin HTTP queue plus mooncake-store-rs-admin migrate ... operator commands

• a task-executor model where admin submits control-plane RPC and the executor runtime performs the transfer

• in-memory admin task queue with pending, dispatching, running, retry_wait, succeeded, and failed states

• admin-side retry with configurable backoff and a default retry budget of 5

• route-based completion checks so admin can mark a task successful after executor loss when the final route is already authoritative

Design boundary:

• the admin queue is runtime memory only and does not recover across admin restart

• the CLI is only an HTTP client for this queue; migrate ... commands must talk to a long-lived mooncake-store-rs-admin server

• the authoritative durable state remains the object route, not the admin task record

• retry is for executor loss or transient RPC failure, not for preserving a persistent migration backlog

Placement and Replication

Local-first placement

The default request behavior prefers local storage when possible.

When local storage cannot satisfy the request, the client can spill over to remote storage nodes.

Routed remote placement

Router nodes can place data onto storage nodes selected by the placement planner.

The planner filters candidate nodes by labels and compatibility.

Request-level replication policy

A request can control placement behavior with:

• replica count

• preferred segment

• preferred segment list

• preferred storage owner

• preferred storage owner list

• prefer_local

• prefer_alloc_in_same_node

Admin Maintenance

Stateless stale-segment cleanup

The admin plane now supports Redis-backed and etcd-backed stale-segment cleanup without reintroducing a master on the request path.

What it provides:

• explicit one-shot stale cleanup through mooncake-store-rs-admin cleanup-stale-segments

• continuous stateless cleanup through mooncake-store-rs-admin server

• optional tenant-quota reservation reconcile in that same admin server for an explicit tenant list

• backend-native lease-expiry indexing for recoverable maintenance scheduling (ZSET in Redis, by-runtime + by-time keys in etcd)

• owner-scoped cleanup through backend-native owner metadata instead of a hidden global segment scan in the steady-state worker

• same-epoch lease reclaim after a lease TTL gap when no higher live epoch exists, so heartbeat repair and predecessor drain pinning do not fail with stale-epoch rejection

• with_soft_pin

Observability

The client Prometheus exporter reports both request-level and internal operation-level store health. Remote data-plane transfers now expose explicit transport operation outcomes through mooncake_store_transport_operation_total{direction,peer_kind,result} in addition to successful byte volume through mooncake_store_transport_bytes_total{direction,peer_kind}.

Route publication also exposes mooncake_store_replication_publish_total{result} alongside the existing publish latency histogram. Operators can therefore distinguish publish failures from slow successful publishes without inferring result counts from histogram internals.

Jaeger tracing can be controlled dynamically through the store metrics HTTP endpoint and through the admin server fanout layer. The per-node metrics endpoint still owns the actual tracing switch, while mooncake-store-rs-admin tracing ... provides the cluster-level operator entry point. Admin discovers live store nodes from metadata leases, derives the host from existing endpoint data, and uses the store-published metrics_port label for the metrics HTTP port.

The lease extension is intentionally label-based. Older metadata readers ignore the extra label, and newer admin servers fall back to the default metrics port for leases written before the label existed. When tracing is disabled, request-path tracing work remains guarded by the runtime tracing filter and dynamic exporter state rather than adding metadata calls to put/get hot paths.

Allocation and Segment Management

Local allocation

Local segment reservation and release are handled inside the client process.

The current implementation supports:

• transport-managed local memory

• hugepage-backed native memory for Rust store segments

• hugepage-backed shared memory for Python host buffers

• background async eviction on storage nodes

Startup behavior for large local storage is explicit:

• classic_te + RDMA keeps initial storage-region planning NUMA-aware for host memory, so RDMA-capable hosts can start with multiple local storage segments instead of collapsing everything into one CPU location

• startup registration of extra initial storage segments is transport-gated; non-RDMA and non-opted-in transports keep the historical single-segment startup layout

• when a transport enables startup fanout, the runtime registers extra initial storage segments in parallel and publishes them only after the local registration phase completes

• classic_te + RDMA pre-touches startup storage pages once total startup storage registration reaches 4 GiB, reducing cold-page MR registration stalls on very large hosts

Remote allocation

Remote storage reservation uses allocator RPC to the owning client.

Remote storage owners apply the same reserve loop as local storage owners:

• try allocator reservation

• if the failure is capacity-related, trigger storage-owner eviction and retry

• return an allocator error when capacity still cannot be recovered

Remote allocator semantics

Allocator ownership stays with the storage owner.

• reserve and release go through allocator RPC on the owning client

• capacity failures returned by a live storage owner remain capacity failures

• transport or protocol failures quarantine the remote storage owner and let routed placement skip to the next soft candidate

Storage-Owner Eviction

Local CLOCK state

Each storage owner maintains a local CLOCK over the replicas it currently stores.

The tracked identity is route-oriented rather than segment-only:

• route key

• segment name

• segment offset

This lets the storage owner evict a precise replica instead of reclaiming blind segment regions.

Background watermarks

Storage nodes reclaim proactively in the background.

The current behavior is:

• a background worker polls local usage

• when used bytes reach the high watermark, eviction starts

• eviction continues until used bytes fall to the low watermark or no more victims can be reclaimed

• synchronous reserve-fail eviction still remains as the last-resort fallback

Default local-memory settings are:

• high watermark: 90%

• low watermark: 80%

• poll interval: 100ms

Read-hit reporting

Successful reads update eviction heat without putting metadata back on the hot path.

The current implementation does this in batch:

• local hits update the local storage-owner state directly

• remote hits are grouped by storage owner

• grouped hits are sent through BatchReportRouteHits

Hit reporting is best-effort. A lost hit can reduce eviction quality, but it does not affect correctness.

Route tracking after publish

After a writer successfully publishes a route, it groups the remote replica owners and sends the published routes through BatchTrackReplicaRoutes.

This gives each storage owner a current route view for the replicas it stores without requiring a metadata scan on the write hot path. Freshly published replicas get one CLOCK grace pass on the storage owner. Completed reads and route-query access signals still set the stronger read-hot bit; boolean existence probes remain pure probes and do not report hits.

Route-owner CAS reclaim

When capacity pressure forces eviction:

• the storage owner first checks a bounded cursor slice of currently tracked local replicas for stale/orphan allocations whose authoritative route no longer references the local bytes

• stale/orphan local allocations found in that bounded slice are reclaimed before the storage owner considers a live CLOCK victim

• repeated reserve retries are rate-limited so pressure spikes do not repeatedly rescan the full tracked key set before every live eviction attempt

• pending local reservations remain protected until their publish/timeout window expires, so fresh writes are not reclaimed early

• reclaim_grace_ms applies to scheduled cleanup; eviction pressure may reclaim stale/orphan local bytes earlier after an authoritative active route has already moved away

• route-missing/current-absent lookups are treated as inconclusive and do not release allocator bytes in this path; confirmed-deleted ghost reclaim is handled by a separate proof path

• the storage owner picks a CLOCK victim

• the matching route owner removes that replica with CAS

• only after CAS succeeds does the storage owner release the local allocation

This preserves route correctness during eviction and keeps reclaim coupled to the authoritative route version.

Where local eviction runs

Local CLOCK eviction is enabled for clients labeled storage=true.

This keeps the roles clean:

• storage nodes reclaim their own local capacity

• routed rw nodes keep the existing spill-remote behavior

• scratch-only rw nodes can run with storage_bytes=0

Startup also enforces the role boundary:

• storage=true requires storage_bytes > 0

• when storage_bytes=0 and no explicit storage label is supplied, the runtime defaults the label to storage=false

Delete and Reclaim

Overwrite reclaim

When a key is overwritten, old replicas are scheduled for release.

Delete reclaim

When a key is removed, route state is removed and segment space is reclaimed.

For tenant-scoped quota policy, delete now also finalizes a negative quota delta when the route delete CAS succeeds. That means quota refund is tied to authoritative object deletion, not to later background reclaim of the freed segment space.

Graceful reclaim

A reclaim grace window can delay scheduled release to smooth transitions or handoff behavior. Under storage-owner eviction pressure, stale/orphan local allocations can be reclaimed earlier once an authoritative active route no longer references those bytes. Route-missing ghosts are retained until a confirmed cleanup or confirmed-absent reclaim path owns the release decision. Queued reclaim cleanup now treats duplicate local or remote release as idempotent cleanup noise, while direct non-cleanup release paths still return allocator errors.

Multi-Tenancy

The client supports tenant-scoped keys and tenant-aware request builders.

This keeps API usage explicit while still allowing a default tenant for simpler applications. Tenant, domain, object-set, and logical-key components are percent-encoded before route separators are added. Reserved separator characters therefore remain distinct across tenants and keys, including the default namespace path.

Route-authority policy has a default cluster policy per metadata keyspace. route_control is a cluster-level deployment decision fixed at startup (determines whether WRH authority mesh or Redis-only route storage is used). route_topk can be overridden per-tenant via admin policy. Runtime bootstrap resolves the effective route_topk from the tenant policy if present, otherwise uses the CLI default.

Dynamic Membership and Lifecycle

Client lifecycle

The runtime models explicit client states:

• standby

• active

• draining

Handoff and upgrade

The client can publish handoff plans and participate in successor upgrade flows.

Elastic capacity

The client can expand local storage, drain segments, retire empty segments, and evacuate all replicas owned by a draining client.

This allows segment-level shrink, full client shrink, and dynamic storage growth without changing the public API.

During full-client drain, replacement routes are published through the same object route CRUD/CAS facade used by normal writes. The client does not perform a separate route-authority fanout for the draining authority. Explicit drain migration reads the selected source replica through an exact readable lease lookup instead of the suspect-runtime placement cache, so allocator quarantine for new writes does not block copying the source bytes that are being evacuated. Migration placement excludes the draining source owner even when a separate writer performs the copy, and the source owner releases its old allocations after publication. Read selection also prefers a local readable replica before remote replicas with lower route priority, so replica-protected reads avoid a dead remote primary when the survivor already has the data locally.

Control Plane

The runtime includes a dedicated control plane for route and non-route control traffic:

• route operations

• allocator operations

• eviction-specific hit reporting and route tracking

• migration task submission and status queries

Important properties:

• protobuf schema

• tonic RPC implementation

• batched operations

• streaming session reuse

Route-control semantics live in mooncake-store-route; the client control plane remains responsible for transport concerns such as protobuf encoding, tonic channels, stream reuse, unary fallback, and request timeouts.

Observability

Tracing

Tracing can be enabled from code or environment variables. Jaeger profiling exports OTLP spans for the same operation and stage boundaries used by metrics, including request APIs, route lookup, allocation, transfer, route publication, metadata backend calls, heartbeat, membership refresh, and storage-owner eviction. The spans are named by profiling phase rather than raw implementation hook: API roots use store.*, control-plane stages use control.*, data-plane stages use data.*, and backend calls use metadata.*. A put/get request therefore appears in Jaeger as one in-process waterfall with mooncake.phase, mooncake.flow, request id, item count, byte count, replica count, and local/remote target-count attributes.

The in-process metrics HTTP server exposes /tracing to turn profiling on or off dynamically when a sink is configured. Use endpoint=... for OTLP/Jaeger or file=/tmp/store-rs.trace.jsonl for a local JSONL artifact. Set MC_STORE_RS_OTLP_SAMPLE_RATIO or the /tracing?sample_ratio=... query before enabling export when profiling a high-throughput workload against a memory-backed Jaeger collector.

When local JSONL profiling is enabled, MC_STORE_RS_TRACE_ITEM_METADATA=1 adds store.api_items.v1 records for batch_put_from and batch_get_into. These records keep per-item namespace, runtime id, key hash/prefix, parsed SGLang TP rank and k/v suffix, object size or read length, and item status out of Prometheus labels while still making offline per-key put/get counts possible. MC_STORE_RS_TRACE_KEY_MODE defaults to hash; full is only for local debug captures.

Metrics

The runtime records:

• operation counts

• input and output bytes

• total latency

• max latency

• histogram percentiles for request APIs and internal stages

• metadata backend operation counts and latency

• transport operation counts and bytes

The metrics HTTP server also exposes GET /breakdown, a structured diagnostic snapshot for SGLang/HiCache runs. It groups API calls, internal phases, metadata backend work, transport volume, runtime state, and segment usage, then ranks bottleneck candidates by observed total latency and p99 latency. The ranking is evidence for where time was spent, not proof of root cause.

• status labels such as ok and error

• strict tenant quota counters for reservation, finalize, abort, and reconcile outcomes

HTTP export

Metrics can be served through an in-process HTTP endpoint.

Python Compatibility

The Python layer mirrors the same runtime instead of wrapping a separate implementation.

What this means in practice:

• the same route, allocator, and reclaim logic is reused

• the same transport path is reused

• the same metrics and tracing capabilities are available

• the same replication controls are exposed through ReplicateConfig

• the same standalone compatibility server can serve dummy clients

• the root wheel exposes a Python Store facade with explicit C++ and Store-RS backends

Dummy and real compatibility paths

The compatibility layer exposes two integration modes:

• real mode uses the native StoreClient directly from Python

• dummy mode talks to a standalone mooncake-store-rs-client process and exchanges shm buffer registrations over a side channel

Both paths are covered by repository validation scripts.

The root validation entry point is scripts/ci/run_store_rs_smoke.sh. It covers the installed Python facade and the dummy and routed TCP/classic-TE read/write paths. See the validation guide for required build inputs and direct commands for additional scenarios.

Manual path-specific validators stay available as:

• python/tests/store/rs/clients/real_client_rw.py

• python/tests/store/rs/clients/dummy_client_rw.py