Rust Guide

Rust Guide#

This document shows how to assemble and use the Rust client from the current workspace.

Before You Start#

Bring MooncakeCompatibilityFacade into scope. Most store operations are exposed through that trait.

use mooncake_store_client::MooncakeCompatibilityFacade;

You also need:

  • a metadata backend

  • a transport engine

  • a transport factory

  • a StoreClientBuilder

Build a Local Client#

use std::sync::Arc;
use std::time::{SystemTime, UNIX_EPOCH};

use mooncake_store_rs_metadata::{MetadataKeyspace, RedisMetadataBackend, RedisMetadataConfig};
use mooncake_store_client::{
    LocalMemoryConfig, MooncakeCompatibilityFacade, StoreClientBuilder, TentTransportFactory,
};
use mooncake_store_core::{ClientLifecycleState, CompatibilityDescriptor, Result};
use mooncake_store_rs_transport::{TentEngine, TentEngineConfig};

fn now_ms() -> u64 {
    SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .expect("time should move forward")
        .as_millis() as u64
}

fn main() -> Result<()> {
    let metadata = Arc::new(RedisMetadataBackend::new(
        RedisMetadataConfig::new("redis://127.0.0.1:6380/0")
            .keyspace(MetadataKeyspace::new("mc/store-rs/demo")),
    )?);

    let tent_config = TentEngineConfig::new()
        .set("metadata_type", "redis")
        .set("metadata_servers", "127.0.0.1:6380")
        .set("redis_db_index", "0")
        .set("rpc_server_hostname", "127.0.0.1")
        .set("rpc_server_port", "0")
        .set("log_level", "warning")
        .set("transports/tcp/enable", "true")
        .set("transports/shm/enable", "false")
        .set("transports/rdma/enable", "false")
        .set("transports/io_uring/enable", "false");

    let engine = Arc::new(TentEngine::new(
        &tent_config.clone().set("local_segment_name", "demo-segment"),
    )?);
    let factory = Arc::new(TentTransportFactory::new(tent_config));

    let client = StoreClientBuilder::new(metadata, "demo-store")
        .state(ClientLifecycleState::Active)
        .compatibility(CompatibilityDescriptor::default())
        .tenant("default")
        .route_topk(2)
        .local_memory(
            LocalMemoryConfig::new()
                .storage_bytes(128 * 1024 * 1024)
                .scratch_bytes(16 * 1024 * 1024)
                .location("cpu:0"),
        )
        .with_tent(engine)
        .transport_factory(factory)
        .build(now_ms() + 600_000)?;

    client.register_local_memory()?;
    client.put("hello", b"world")?;
    assert_eq!(client.get("hello")?, b"world");
    Ok(())
}

rpc_server_port is the TENT TCP data-plane port. Leaving it at 0 lets TENT choose a random local port, which is fine for single-host demos. For cross-host or cross-container deployments, set a fixed port and make sure peers can reach rpc_server_hostname:rpc_server_port.

When a transport is supplied directly, its segment_name() must match any explicit builder segment_name(...). If the builder omits segment_name(...), the client adopts the transport segment name. This keeps the lease announcement, allocator metadata, and transport registration pointed at the same local segment.

tenant(...) remains the normal way to select the default scope used for startup policy lookup and request builders. Startup policy resolution reads exact tenant-policy scopes for tenant -> tenant/domain -> tenant/domain/object_set instead of listing all tenant policies. route_topk(...), namespace_quota(...), execution_fairness(...), and bandwidth_shaping(...) are compatibility fallbacks; admin-managed tenant policy in metadata is the preferred authoring surface when those settings are tenant-scoped. route_control(...) is a cluster-level deployment setting (not per-tenant) that determines whether EmbeddedWrh or MetadataOnly route storage is used.

When tenant quota policy is present, single-object put and remove now use metadata-backed reservation/finalize semantics instead of relying only on the older runtime-local preflight check. The write call returns success only after route publication and quota finalization both succeed. Overwrites are charged on committed byte delta, and deletes refund quota when the route delete CAS becomes authoritative rather than waiting for later segment reclaim.

Transport Backends#

The Rust transport layer exposes two explicit choices:

  • TentEngine + TentEngineConfig

  • ClassicTransferEngine + ClassicEngineConfig

This guide uses TENT for the concrete examples because it is the default repository path, but Rust applications can construct either backend directly.

Runtime backend selection is implemented by the compatibility layer, not by StoreClientBuilder itself:

  • MC_STORE_RS_TRANSPORT_BACKEND=tent|classic_te

  • mooncake-store-rs-client --transport-backend tent|classic-te

  • MooncakeDistributedStore.setup(..., transport_backend="tent"|"classic_te")

For classic_te with P2PHANDSHAKE metadata, Store-RS separates the logical segment identity from the transport identity. Segment announcements keep the configured logical segment_name, publish the peer RPC address in transport_endpoint, and publish the full TE RAM descriptor in transport_segment_descriptor. Remote clients preload that descriptor into classic TE before calling openSegment, matching the descriptor shape TE normally obtains from Redis metadata.

Redis authentication can come from URL-embedded credentials or from MC_REDIS_USERNAME / MC_REDIS_PASSWORD. Prefer environment variables when passwords contain URL-reserved characters such as @.

Check Without Native Build#

The top-level Mooncake CMake project owns the classic-TE and TENT shim targets and links them to its existing transfer_engine and tent_shared targets. mooncake-store-rs-transport-sys provides Rust FFI for those shims. Its build script validates MOONCAKE_ROOT_DIR, MOONCAKE_BUILD_DIR, and the explicit MOONCAKE_CLASSIC_SHIM_LIB_PATH / MOONCAKE_TENT_SHIM_LIB_PATH artifacts; it does not configure or compile CMake targets. The enclosing CMake build passes these paths, or direct Cargo users must provide absolute paths explicitly.

The enclosing CMake target builds the required native targets before Rust:

cmake -S /path/to/Mooncake -B /path/to/Mooncake-build \
  -DWITH_STORE_RS=ON -DWITH_TE=ON -DUSE_TENT=ON \
  -DWITH_STORE=OFF -DWITH_STORE_RUST=OFF
cmake --build /path/to/Mooncake-build --target build_store_rs

Developer machines that only need Rust type analysis can skip native artifact validation:

MOONCAKE_SKIP_NATIVE_BUILD=1 cargo check

This is intended for IDE analysis. Runtime builds require an existing CMake build tree with the TE, TENT, yalantinglibs, and shim artifacts; the enclosing CMake targets build and install those dependencies before Cargo compiles Rust.

Enable Routed Writes#

Use PlacementPlanner and routed_writes(...) when a client should route writes to storage nodes.

use mooncake_store_client::PlacementPlanner;

let planner = PlacementPlanner::new(metadata.clone()).require_label("storage", "true");
let router = StoreClientBuilder::new(metadata, "router-a")
    .state(ClientLifecycleState::Active)
    .label("storage", "false")
    .with_tent(engine)
    .transport_factory(factory)
    .local_memory(local_memory)
    .routed_writes(planner, 2)
    .build(now_ms() + 600_000)?;

In this mode:

  • placement candidates come from live compatible clients

  • candidates are filtered by planner labels

  • replica owners are ranked with rendezvous hashing

Use Request Policies#

ReplicationPolicy changes placement behavior for one request.

use mooncake_store_client::ReplicationPolicy;

let policy = ReplicationPolicy::new()
    .replica_count(2)
    .prefer_local(false)
    .preferred_storage_owner("store-b")
    .with_soft_pin(true);

client.put_with_policy("replicated", b"payload", &policy)?;

Useful knobs:

  • replica_count(...)

  • prefer_local(...)

  • prefer_alloc_in_same_node(...)

  • preferred_segment(...)

  • preferred_segments(...)

  • preferred_storage_owner(...)

  • preferred_storage_owners(...)

  • with_soft_pin(...)

Use Batch and Buffer-Oriented APIs#

The Rust client supports both object-oriented and buffer-oriented paths.

Basic and batch APIs#

  • put, get, remove

  • batch_put, batch_get, batch_remove

  • is_exist, batch_is_exist, get_size

query_route, get_size, and successful reads are treated as access signals when they find active routes. They report route hits to the storage owners best-effort through the same CLOCK hit-report path used by reads, so storage-owner eviction protects recently observed replicas. is_exist and batch_is_exist stay boolean probes and do not report hits.

Registered-buffer APIs#

  • register_buffer

  • unregister_buffer

  • put_from

  • batch_put_from

  • batch_get_into

Multi-buffer APIs#

  • batch_put_from_multi_buffers

  • batch_get_into_multi_buffers

Use the buffer-oriented forms when the application already owns stable memory and wants to avoid extra copies.

Behavior boundary:

  • put_from and batch_put_from send remote writes from the registered source buffer directly

  • batch_get_into reads remote payloads directly into registered destination buffers

  • registered source buffers that are not host-readable, such as CUDA buffers, are used only as direct transfer sources; they cannot satisfy local-storage writes or staged scratch writes, and the published replica route omits the CPU payload checksum for those writes

  • routed batch_put_from treats per-key route CAS conflicts as successful cache insert races when an active route is already published, releases its own temporary reservation, and returns the published route for that key; when the CAS response omits that active route, the client does one exact bounded recheck for the conflicted key before failing

  • ReplicaRoute::offset is the transport target address for the payload; segment_offset is allocator bookkeeping, so writers translate allocator reservations through the segment announcement’s published storage target chunks, and local reads plus drain migration translate offset back through the same storage target map before copying from local storage

  • unregistered get_into targets and batch_put_from_multi_buffers still fall back to the staged copy paths

Hugepage-Backed Local Memory#

LocalMemoryConfig can request hugepage-backed local storage and scratch memory.

let local_memory = LocalMemoryConfig::new()
    .storage_bytes(128 * 1024 * 1024)
    .scratch_bytes(16 * 1024 * 1024)
    .use_hugepage(true)
    .hugepage_size_bytes(2 * 1024 * 1024);

Supported sizes:

  • 2 MiB

  • 1 GiB

If these fields are not set explicitly, the runtime can also use MC_STORE_USE_HUGEPAGE and MC_STORE_HUGEPAGE_SIZE.

Multi-Tenant Access#

Use tenant-aware request builders when the default tenant is not enough.

Examples:

  • put_in_tenant(...)

  • get_in_tenant(...)

  • remove_in_tenant(...)

  • query_route_in_tenant(...)

Batch request builders also accept ObjectRef::tenant(...).

Lifecycle and Elasticity#

The compatibility facade also exposes lifecycle and memory-management operations.

Common calls:

  • heartbeat(expires_at_ms)

  • enter_standby()

  • activate()

  • enter_draining()

  • plan_handoff(...)

  • list_segments()

  • expand_local_memory(storage_bytes)

  • drain_segment(segment)

  • retire_segment(segment)

  • evacuate_owned_replicas()

  • evacuate_owned_replicas_via(writer)

evacuate_owned_replicas() drains the local client, rewrites every live route that still references it, immediately reclaims old allocations, and retires emptied local segments.

During evacuation, payload copy is pinned to the exact local replica being removed. The writer does not satisfy the migration read from another live mirror, because that would publish the wrong bytes when replicas temporarily diverge during a shrink. The migration read uses the replica’s transport target offset, not the allocator reservation offset, and maps that target coordinate through the storage target map before touching local memory.

Before returning, the draining client also mirrors any route-authority records that were only present locally to active route authorities. If its cached live-client snapshot cannot find a mirror, it refreshes membership and retries the mirror pass.

Use evacuate_owned_replicas_via(writer) when you want a separate routed client to publish replacement routes during shrink.

These operations are the current control surface for hot-upgrade, elastic growth, segment drain/retire, and full client shrink flows.

Route Inspection#

Use these methods when debugging placement or route state:

  • query_route(...)

  • cas_route(...)

  • list_segments()

  • runtime_id()

  • lease()

The default route control mode is EmbeddedWrh. Use route_control(RouteControlMode::MetadataOnly) if you want metadata-backed route control for bring-up or debugging.

Use route_topk(...) when you want a different WRH route-authority fanout. The runtime defaults to 2, rejects values below 2, and validates the setting against the effective route policy already stored in the active metadata keyspace: tenant override for the builder’s default tenant first, otherwise the default cluster policy.

Observability#

Tracing and metrics are part of the Rust client crate.

use mooncake_store_client::{
    init_tracing, render_prometheus_metrics, start_metrics_http_server,
};

init_tracing(Some("info"))?;
let addr = start_metrics_http_server("127.0.0.1:0")?;
println!("metrics: http://{addr}/metrics");
println!("{}", render_prometheus_metrics());

Key functions:

  • init_tracing(...)

  • init_tracing_from_env(...)

  • render_prometheus_metrics()

  • start_metrics_http_server(...)

  • stop_metrics_http_server()

The exported metrics now include strict tenant quota counters for reservation, finalize, abort, and reconcile outcomes, in addition to the existing request, route-CAS, replication, eviction, and process families. Sparse operational counters such as tenant quota, tenant-local eviction, preferred-segment skip, rebalance, and segment lifecycle are exported with zero-valued baseline series so dashboards show an explicit zero rate during steady state instead of No data.

Full Example#

The most complete runnable example is crates/mooncake-store-e2e/src/main.rs.

It covers:

  • local and routed writes

  • request-level replication

  • batch put/get

  • strict tenant quota on single-object and routed batch writes

  • strict tenant quota e2e validation for Redis-backed admit/finalize/reject/delete-refund state

  • registered-buffer and multi-buffer paths

  • delete reclaim and overwrite reclaim

  • dynamic expansion, true client shrink, and hot-upgrade

Benchmarking#

Use mooncake-store-rs-bench for throughput measurement, correctness verification, and long-duration stability testing. Its sources live under crates/mooncake-store-rs-cli/src/bin/mooncake-store-rs-bench/ in the Rust workspace.

Quick start:

# Correctness check
# Defaults use batch_put + batch_get; override interfaces when needed.
mooncake-store-rs-bench --metadata-url redis://127.0.0.1:6379/0 verify

# 30-second mixed benchmark, 8 concurrent workers
# Defaults measure batch_put + batch_get; override interfaces when needed.
# Start storage=true daemons first, and pass the same --keyspace they use
# when you are not using the default `mc/store-rs/v1` namespace.
mooncake-store-rs-bench --metadata-url redis://127.0.0.1:6379/0 bench \
  --keyspace mc/store-rs/bench-prod \
  --mode mixed --concurrency 8 --duration 30

# 1-hour soak test with Redis jitter fault injection
mooncake-store-rs-bench --metadata-url redis://127.0.0.1:6379/0 soak \
  --duration 3600 --fault redis-jitter:5:50 --verify-reads

See the Store-RS benchmark guide for the full CLI reference.

Next Reading#

Contents