mooncake-store-rs-bench

mooncake-store-rs-bench is a standalone benchmark and verification tool for the Mooncake store. It is built by mooncake-store-rs-cli alongside mooncake-store-rs-client; the separate admin command is built by mooncake-store-rs-admin.

Commands

mooncake-store-rs-bench [global options] <COMMAND>

Global options

Flag	Default	Description
--metadata-url <URL> (alias --metadata_url; env MC_STORE_RS_METADATA_URL)	required	Store-RS metadata URL. Accepts redis://... or etcd://....
--transport-metadata-url <URL> (alias --transport_metadata_url; env MC_STORE_RS_TRANSPORT_METADATA_URL)	P2PHANDSHAKE	Transfer Engine metadata input. Accepts redis://... or P2PHANDSHAKE. Defaults to P2PHANDSHAKE (classic_te peer handshake); tent requires redis://....
--keyspace <PREFIX>	mc/store-rs/v2 when storage_bytes=0; otherwise auto-generated	Metadata keyspace prefix
--transport-backend <classic-te|tent>	classic-te	Data-plane transport backend
--protocol <tcp|rdma>	tcp	Transport protocol
--local-hostname <HOST>	127.0.0.1	Transport bind hostname
--storage-bytes <BYTES>	0	Per-client storage allocation for bench RW clients
--scratch-bytes <BYTES>	16 MiB	Per-client scratch allocation
--tenant <NAME>	bench	Default tenant scope
--trace-filter <FILTER>	MC_STORE_RS_TRACE_FILTER, then RUST_LOG, otherwise info	tracing-subscriber filter
--metrics-addr <ADDR>	off	Prometheus metrics bind address
--seed <N>	42	Global RNG seed for deterministic data
--route-control <embedded-wrh|metadata-only>	embedded-wrh	Route control mode used by bench runtimes
--route-topk <N>	2	WRH route-authority fanout
--replica-count <N>	1	Replication factor for writes

The default bench topology is scratch-only for RW clients (MC_BENCH_STORAGE_BYTES=0). Start separate storage=true daemons with matching metadata, transport, and --keyspace settings before running the benchmark. This is the same storage/RW role split used by scripts/tests/client/test-client-rw-cli.sh. When --keyspace is omitted in scratch-only mode, bench joins the same default metadata namespace as mooncake-store-rs-client: mc/store-rs/v2. When --storage-bytes > 0 and --keyspace is omitted, bench generates an isolated mc/store-rs/bench/<unique> keyspace for that run.

Startup safety gate for classic_te compatibility deployments:

• bench refuses to start if active storage=true daemons in the same keyspace mix metadata-backed logical segment names with P2PHANDSHAKE rpc-address segment names

• bench also refuses to start if its effective transport metadata mode disagrees with the active storage daemons already visible in that keyspace

• fix the cluster by restarting every participating storage daemon and bench process with one transport metadata mode before benchmarking

Tracing behavior

mooncake-store-rs-bench initializes its own tracing subscriber.

• default sink: stderr

• effective filter: --trace-filter, then MC_STORE_RS_TRACE_FILTER, then RUST_LOG, then info

• dedicated trace file: MC_BENCH_TRACE_FILE=/path/to/bench.log

• MC_STORE_RS_TRACE_FILE remains the real-client / standalone-client trace-file surface and does not redirect bench output

• ANSI colors are enabled only when stderr is a TTY and no bench trace file is selected

• span close events are disabled, so per-request close time.busy=... subscriber noise does not flood benchmark output

bench — throughput and latency

mooncake-store-rs-bench --metadata-url redis://127.0.0.1:6379/0 bench \
  --mode mixed \
  --write-interface batch-put-from \
  --read-interface batch-get-into \
  --concurrency 8 \
  --value-size 65536 \
  --batch-size 16 \
  --duration 60 \
  --read-ratio 70 \
  --output-format text

Flag	Default	Description
--mode <put|get|mixed>	mixed	Operation mix
--write-interface <put|batch-put|batch-put-from>	batch-put-from	Write-side store API used for measured write traffic; also read from MC_BENCH_WRITE_INTERFACE
--read-interface <get|batch-get|batch-get-into>	batch-get-into	Read-side store API used for measured read traffic; also read from MC_BENCH_READ_INTERFACE
--concurrency <N>	4	Concurrent worker threads
--value-size <BYTES>	4096	Payload size per object
--batch-size <N>	8	Objects per batch_put / batch_put_from / batch_get / batch_get_into call; ignored by single-key put / get
--iterations <N>	1024	Total operations per worker
--duration <SECONDS>	off	Run for fixed wall time instead of iteration count
--warmup <N>	32	Warmup operations excluded from stats
--read-ratio <PERCENT>	70	Read fraction in mixed mode
--report-interval <SECONDS>	5	Periodic progress interval
--key-space-size <N>	10000	Distinct key count
--writers <N>	1	Writer StoreClient instances
--readers <N>	1	Reader StoreClient instances
--output-format <text|json|csv>	text	Report body format emitted through tracing

bench separates traffic shape from concrete client APIs:

• --mode controls whether the workload is write-only, read-only, or mixed

• --write-interface selects put, batch_put, or batch_put_from for measured writes

• --read-interface selects get, batch_get, or batch_get_into for measured reads

• defaults benchmark batch_put_from + batch_get_into

• MC_BENCH_INTERFACES can set both env defaults at once, for example MC_BENCH_INTERFACES=put,get

• batched operations advance their key cursor by the batch width, so adjacent batch calls do not repeatedly rewrite or reread the same object window

• mixed mode splits each worker’s key shard into a stable read half and a write half; this keeps the benchmark from turning ordinary read traffic into immediate read-after-overwrite route churn

• prefill, warmup, and measured phases heartbeat per-worker writer/reader clients so long scratch-only benchmark setup does not let route authorities expire before measured reads start

• writer/reader runtime stable IDs include a per-process run ID, so rerunning the same Kubernetes Job before old leases expire does not collide with the previous run’s still-active bench clients

• measured reads retry bounded route-readiness misses, such as a freshly written object route not yet being readable; the wait is included in read latency and persistent misses still count as read errors

Output at the end of a run:

=== Benchmark Results ===
Mode: mixed | Write: batch_put_from | Read: batch_get_into | Duration: 30.12s | Workers: 8 | Value size: 65536 B | Batch size: 16

  Metric           batch_put_from   batch_get_into
  ------           ---              ---
  Total ops        3840             8960
  QPS              127.5            297.4
  Throughput       7.7 GiB/s        18.0 GiB/s
  Latency p50      1.23ms           0.84ms
  Latency p90      2.45ms           1.56ms
  Latency p99      5.12ms           3.78ms
  Latency p999     12.4ms           8.92ms
  Latency max      18.7ms           11.3ms
  Errors           0                0

--output-format changes the report body only. The final text, JSON, or CSV payload is still emitted as INFO tracing events, so the normal tracing prefix remains around the report in stderr or MC_BENCH_TRACE_FILE.

Counters always count API calls, not individual objects:

• put and get count one object per call

• batch_put, batch_put_from, batch_get, and batch_get_into count one batch call per operation

• throughput still includes the bytes of every object returned by that call

verify — correctness checks

mooncake-store-rs-bench --metadata-url redis://127.0.0.1:6379/0 verify \
  --write-interface batch-put-from \
  --read-interface batch-get-into \
  --verify-overwrite \
  --verify-delete \
  --verify-multi-tenant

Flag	Default	Description
--write-interface <put|batch-put|batch-put-from>	batch-put-from	Write-side store API used for verify checks that issue writes; also read from MC_BENCH_WRITE_INTERFACE
--read-interface <get|batch-get|batch-get-into>	batch-get-into	Read-side store API used for verify checks that issue ordinary reads; also read from MC_BENCH_READ_INTERFACE
--value-size <BYTES>	4096	Payload size
--key-count <N>	256	Keys exercised in multi-key tests
--batch-size <N>	8	Objects per batch_put / batch_put_from / batch_get / batch_get_into; ignored by single-key put / get
--verify-overwrite	off	Enable overwrite correctness test (64 rounds)
--verify-delete	off	Enable delete + reclaim test
--verify-multi-tenant	off	Enable tenant isolation test

Each check prints PASS or FAIL with a diagnostic. The process exits non-zero if any check fails.

verify defaults to batch_put_from + batch_get_into. MC_BENCH_INTERFACES=put,get and the explicit --write-interface / --read-interface flags are both supported here too.

Default checks run without flags:

1. single-round-trip — configured write interface, then configured read interface, byte-compare

2. get-into-buffer — configured write interface, then get_into, size and byte-compare

3. batch-put-get — multi-key write/read using the configured interfaces

4. is-exist — configured write interface, then is_exist

5. multi-key-write / multi-key-read — N-key write/read using the configured interfaces with a heartbeat between phases

soak — long-duration stability

mooncake-store-rs-bench --metadata-url redis://127.0.0.1:6379/0 soak \
  --duration 3600 \
  --write-interface batch-put-from \
  --read-interface batch-get-into \
  --fault redis-jitter:5:50 \
  --fault metadata-drop:2 \
  --verify-reads

Flag	Default	Description
--duration <SECONDS>	3600	Total run time
--concurrency <N>	2	Worker count
--write-interface <put|batch-put|batch-put-from>	batch-put-from	Write-side store API used for soak writes; also read from MC_BENCH_WRITE_INTERFACE
--read-interface <get|batch-get|batch-get-into>	batch-get-into	Read-side store API used for soak reads; also read from MC_BENCH_READ_INTERFACE
--value-size <BYTES>	4096	Payload size
--batch-size <N>	8	Objects per batch_put / batch_put_from / batch_get / batch_get_into; ignored by single-key put / get
--report-interval <SECONDS>	30	Progress report cadence
--fault <SPEC>	none	Fault injection spec (repeatable)
--verify-reads	true	Verify every read against expected payload
--heartbeat-interval-ms <MS>	30000	Compatibility flag accepted by the CLI; the current soak loop still heartbeats every 64 iterations
--read-ratio <PERCENT>	70	Read fraction
--key-space-size <N>	10000	Distinct key count

soak defaults to batch_put_from + batch_get_into. Override either side with the same CLI flags or env vars used by bench when you want single-key traffic. MC_BENCH_INTERFACES=write=put,read=get is also accepted.

Fault spec format:

Spec	Effect
redis-jitter:<min_ms>:<max_ms>	Sleep a deterministic random interval before the next store operation
metadata-drop:<percent>	Inject a synthetic pre-op failure for N% of operations
transport-delay:<min_ms>:<max_ms>	Sleep a deterministic random interval before the next store operation
transport-error:<percent>	Inject a synthetic pre-op failure for N% of operations

Multiple --fault flags are applied in order before each operation.

Progress is reported on a single line per interval:

[60s] batch_put_from: 142 qps p50=1.1ms p99=4.8ms | batch_get_into: 331 qps p50=0.7ms p99=3.2ms | errors: 0

Architecture

Module layout

crates/mooncake-store-rs-cli/src/bin/mooncake-store-rs-bench/
  main.rs      entry point: parse args, init bench-local tracing, install ctrlc, dispatch
  cli.rs       clap structs: GlobalArgs, BenchArgs, VerifyArgs, SoakArgs, FaultSpec
  datagen.rs   deterministic LCG payload generation; key naming helpers
  latency.rs   LatencyRecorder: raw-sample p50/p90/p99/p999 percentile tracker
  reporter.rs  text/json/csv report-body formatting via tracing; interval progress printer
  setup.rs     BenchCluster: metadata backend, compat runtime construction, startup snapshots
  bench.rs     concurrent worker threads (std::thread::scope)
  verify.rs    sequential correctness checks
  soak.rs      long-running loop with per-iteration fault injection and read verification
  fault.rs     FaultInjector: deterministic jitter and drop decisions from LCG

Data generation

All payloads are deterministic. The seed string is {global_seed}-{key}-{generation} and the LCG is taken from mooncake-store-e2e. The same seed string always produces the same bytes regardless of when or where it is called, so read verification does not require storing the written values.

Latency tracking

LatencyRecorder stores raw per-operation microsecond samples in a Vec. Percentiles are computed by lazy sort on first access. Workers accumulate into a thread-local recorder and the main thread merges all recorders at the end. This approach gives exact percentiles without pre-defined histogram buckets.

Fault injection

FaultInjector uses the same LCG to produce deterministic random decisions per operation, seeded by a global operation counter. Each worker calls pre_op() before issuing a store request. Depending on the configured specs, pre_op() may sleep (jitter or transport-delay) or return an error string (metadata-drop or transport-error). Returned errors increment the error counter and skip the actual store call.

Client construction

BenchCluster wraps one or more StoreClient instances for writer and reader roles. It uses the same CompatRuntimeArgs / CompatSetupArgs path as mooncake-store-rs-client, so transport backend selection, RDMA env passthrough, and compatibility defaults stay aligned with the normal runtime.

Each client gets its own selected transport factory (classic-te by default, optional tent) with a unique segment name, registered against the same RedisMetadataBackend.

Keyspace behavior is conditional:

• scratch-only mode (storage_bytes=0) reuses the configured keyspace, or mc/store-rs/v2 when --keyspace is omitted

• storage-owning mode (storage_bytes>0) auto-generates mc/store-rs/bench/<unique> when --keyspace is omitted

Build

Configure the root CMake project as described in the Store-RS source-checkout quickstart, then build the build_store_rs_cli target. The unified root wheel installs the client, admin, and benchmark commands for direct use from its Python environment.

Next reading

• Rust API for store client usage

• Configuration for Redis and transport options

• mooncake-store-rs/crates/mooncake-store-e2e/src/main.rs for the upstream correctness test suite