Mooncake Transfer Engine Benchmark Tool (tebench) Guide

Mooncake Transfer Engine Benchmark Tool (tebench) Guide#

tebench is an end-to-end benchmarking tool for the Mooncake Transfer Engine. It evaluates bandwidth and latency across different (block_size, batch_size, concurrency) combinations, and supports both the classic TE backend and the new TENT backend.

1. Build and Deployment#

1.1 Prerequisites#

  • A C++ toolchain capable of building the project

  • Optional: GPU(s) and corresponding transport stacks (RDMA, shared memory, io_uring, etc.)

Exact dependencies (CUDA / OFED / io_uring / etc.) depend on the transport backend being used.

1.2 Build with TENT Enabled#

Typical out-of-tree build:

mkdir -p build
cd build
cmake .. -DUSE_TENT=ON
cmake --build . -j

2. Execution Model#

tebench runs in two roles:

  • Target (receiver): creates a memory segment and waits for connections

  • Initiator (sender): connects to a target and executes benchmark cases

Role Selection#

  • If --target_seg_name is empty → Target mode

  • If --target_seg_name is set → Initiator mode

Backend Selection#

  • --backend=classic → classic TE implementation

  • --backend=tent → TENT implementation

3. Quick Start#

3.1 Start Target (First)#

On the target machine:

./tebench \
  --seg_type=DRAM \
  --backend=tent

The program will print a command containing a generated segment name:

To start initiators, run
  ./tebench --target_seg_name=<SEG_NAME> --seg_type=DRAM --backend=tent
Press Ctrl-C to terminate

Copy the printed --target_seg_name to the initiator side.

3.2 Start Initiator#

On the initiator machine:

./tebench \
  --target_seg_name=<SEG_NAME_FROM_TARGET> \
  --seg_type=DRAM \
  --backend=tent \
  --op_type=read \
  --duration=5

3.3 Request Pacing#

Use --request_interval_us=<N> to add a per-thread delay before each transfer batch. The value is in microseconds; 0 disables pacing.

When pacing is enabled, Avg Lat (us) includes the pacing gap because it is computed from wall-clock runtime, while Avg Tx (us) and Tx percentiles only measure transfer execution time.

3.4 MLU Device Memory#

Build with USE_TENT=ON, USE_MLU=ON, USE_CUDA=OFF and BUILD_BENCHMARK=ON. Set NEUWARE_ROOT to the installed Neuware SDK. In this build, --backend=tent --seg_type=VRAM allocates MLU memory. --local_gpu_id=N selects a visible MLU ordinal; -1 allocates a buffer on each visible MLU. --seg_type_mix=dram,vram includes both host and MLU buffers.

Start the target:

./tebench --backend=tent --metadata_type=p2p --seg_type=VRAM \
  --local_gpu_id=0 --xport_type=tcp --total_buffer_size=33554432

Use the printed segment name on the initiator:

./tebench --backend=tent --metadata_type=p2p --seg_type=VRAM \
  --local_gpu_id=0 --xport_type=tcp --total_buffer_size=33554432 \
  --target_seg_name=<SEG_NAME_FROM_TARGET> \
  --start_block_size=65536 --max_block_size=65536 \
  --start_batch_size=1 --max_batch_size=1 \
  --start_num_threads=1 --max_num_threads=1 \
  --check_consistency=true --duration=5

For separate hosts, configure rpc_server_hostname in MC_TENT_CONF with an address reachable from the peer. MLU transfers over TCP or HP TCP use host staging, which requires both the network transport and the MLU local-copy transport. For VRAM or mixed DRAM/VRAM runs, an explicit --xport_type also enables the MLU local-copy transport in the benchmark configuration. --check_consistency performs WRITE/READ round trips and copies the result back to host memory for comparison. For throughput measurements without payload checks, use --check_consistency=false --op_type=read or write.

4. Output Metrics#

Each output row corresponds to one benchmark configuration.

Column

Description

BlkSize (B)

Block size per request (bytes)

Batch

Number of requests per submission

BW (GB/S)

Throughput (total bytes / total time)

Avg Inst GB/s

Average per-transfer instantaneous bandwidth

Avg Lat (us)

Average wall-clock per-operation latency, including pacing or scheduling gaps (scaled by thread count)

Avg Tx (us)

Average per-transfer execution time, excluding gaps

P99 Tx (us)

P99 transfer latency

P999 Tx (us)

P999 transfer latency

A short (~1 second) warmup phase is executed before measurements begin.

4.1 QoS Metrics Baseline#

Use --qos_classes to partition a fixed number of worker threads into QoS classes:

name:threads:slo_us:weight[:isolated_gbps],...

For readability, the same contract can be supplied as JSON with --qos_classes_json:

[
  {"name": "foreground", "threads": 4, "slo_us": 1000, "weight": 4, "isolated_gbps": 12.5},
  {"name": "checkpoint", "threads": 12, "slo_us": 0, "weight": 1, "isolated_gbps": 10.0}
]

Use only one of --qos_classes and --qos_classes_json.

For example, the following closed-loop mixed workload assigns four workers to an SLO-constrained foreground class and twelve workers to a best-effort checkpoint class:

./tebench \
  --target_seg_name=<SEG> \
  --backend=tent \
  --start_num_threads=16 \
  --max_num_threads=16 \
  --qos_classes=foreground:4:1000:4:12.5,checkpoint:12:0:1:10.0 \
  --qos_link_capacity_gbps=25 \
  --qos_output_jsonl=qos-results.jsonl

The class thread counts must add up to the fixed start_num_threads value. An slo_us of zero marks a best-effort class. The SLO is a reporting threshold: QoS baseline mode measures whether each completed transfer meets it, without changing request scheduling policy on either backend.

The human-readable summary and the optional versioned JSONL record report:

Metric

Definition

slo_attainment

Fraction of completed batches whose measured end-to-end transfer time is at most slo_us

p99_us

P99 end-to-end batch transfer latency for the class

goodput_gbps

Class throughput multiplied by SLO attainment; best-effort classes use attainment 1

weighted_goodput_gbps

Sum of weight × goodput_gbps

jain_fairness

Jain index over per-class throughput_gbps / weight

isolation_leakage

max(0, 1 - mixed_throughput / isolated_throughput)

total_utilization

Aggregate measured throughput divided by qos_link_capacity_gbps

Isolation leakage requires a matching class-only baseline, supplied as the optional fifth class field. Total utilization requires --qos_link_capacity_gbps. Missing baselines are emitted as N/A in text and null in JSON rather than being inferred from the mixed run. Run isolated and mixed cases with the same block size, batch size, transport, memory type, and host pair. JSONL records retain isolated_throughput_gbps and link_capacity_gbps alongside the derived values so every metric can be recomputed from one record.

4.2 Mixed Foreground/Background Traffic#

The QoS class flags above use one block size, batch size, and request intent for all workers. Use --workload_classes_json when classes must generate different transfer shapes concurrently, for example latency-sensitive reads competing with a bulk KV migration:

./tebench \
  --target_seg_name=<SEG> \
  --backend=tent \
  --start_num_threads=8 \
  --max_num_threads=8 \
  --workload_classes_json='[
    {"name":"foreground","threads":2,"block_size":4096,"batch_size":1,
     "intent_type":"foreground_get","deadline_us":250,"slo_us":300,"weight":4},
    {"name":"migration","threads":6,"block_size":4194304,"batch_size":2,
     "intent_type":"migration","slo_us":0,"weight":1}
  ]' \
  --qos_link_capacity_gbps=25 \
  --qos_output_jsonl=mixed-traffic.jsonl

This mode runs one fixed workload instead of sweeping the global block and batch size flags. Class thread counts must add up to the fixed benchmark thread count. deadline_us is optional and controls TENT deadline tagging; slo_us remains the independent reporting threshold. Per-class transferred_bytes is included in text and JSONL output so throughput is computed from each class’s actual transfer size.

--workload_classes_json is mutually exclusive with --qos_classes, --qos_classes_json, and the global --tent_intent_type. Non-default per-class intents and deadlines require the TENT backend.

4.3 Per-Target Metrics#

Multi-target runs print one [target-summary] line per target. Use --result_output_jsonl=<path> to also append a schema-versioned JSON record for each benchmark configuration. The record keeps the aggregate operation, byte, and throughput totals plus each target’s segment name, assigned thread count, completed operations, transferred bytes, throughput, and latency distribution. The aggregate throughput uses the pooled average worker duration, matching the existing BW (GB/s) table calculation.

Targets with no assigned worker are retained with zero-valued metrics. This makes an under-provisioned run (threads < targets) visible instead of silently dropping targets from the result.

4.4 Submit/Wait Split Log#

Avg Tx is submit plus completion wait in one timer. To separate those phases, pass --split_output_jsonl=<path> on the initiator. Default is empty: the transfer loop is unchanged, with no extra clocks or I/O.

When set, each completed transfer appends one JSON line:

{"batch_size":207360,"submit_us":66874,"wait_us":556363,"polls":201}

Field

Meaning

submit_us

Time in submitTransfer

wait_us

Time from submit return until the batch reports COMPLETED

polls

Number of status polls in that wait

The file is opened once as std::ofstream and written with stream buffering. Skip the first couple of lines when summarizing; they include warmup. This log is for control-path analysis. Do not treat wait_us or table BW as a substitute for an application poll interval.

./tebench \
  --target_seg_name=<SEG> \
  --backend=classic \
  --op_type=read \
  --split_output_jsonl=submit-wait.jsonl

5. Runtime Configuration#

This section summarizes the key runtime options that control workload behavior, resource usage, and backend selection.

5.1 Workload Mode (--op_type)#

Controls the transfer pattern executed by the initiator:

  • read — repeated READ transfers

  • write — repeated WRITE transfers

  • mix — alternating WRITE followed by READ (both recorded)

Example:

--op_type=write

5.2 Data Consistency Check (--check_consistency)#

When enabled, the benchmark runs and reports mix mode, including when --op_type=read or --op_type=write is requested. The op_type field in --result_output_jsonl records this effective mode. Both WRITE and READ batches contribute latency samples and transferred bytes:

  • Source buffers are filled with a known pattern before WRITE

  • Data is READ back and verified on the initiator

Example:

./tebench --target_seg_name=<SEG> --op_type=mix --check_consistency=true

Consistency checking introduces CPU-side overhead and should be disabled for pure performance measurements.

5.3 Notification Feature (--notifi)#

When enabled, the benchmark sends a notification message along with each transfer batch:

  • The notification contains the target_addr as the message payload

  • The peer can verify the notification was received correctly by checking the message

  • Useful for testing notification delivery and end-to-end communication

Example:

./tebench --target_seg_name=<SEG> --notifi=true

This feature is primarily for testing notification mechanisms. The notification message contains the target address for verification purposes.

5.4 Segment, Concurrency, and Memory Layout#

Segment and role

  • --seg_name : local segment name (typically left empty)

  • --seg_type : DRAM | VRAM (default: DRAM)

  • --seg_type_mix : comma-separated segment types for mixed DRAM+VRAM runs, e.g. dram,vram. When set, the target registers buffers of each listed type in one segment, and initiator threads round-robin across them so a single tebench process drives traffic over multiple memory types concurrently. Empty falls back to --seg_type. See Section 5.8 for transport configuration examples.

  • --target_seg_name : target segment name (empty → Target mode). A comma-separated list enables multi-target initiator mode, and worker threads are distributed across all listed target segments.

Scan ranges

  • --total_buffer_size : total buffer limit (bytes)

  • --start_block_size, --max_block_size : block size sweep (powers of two)

  • --start_batch_size, --max_batch_size : batch size sweep (powers of two)

  • --start_num_threads, --max_num_threads : thread sweep (powers of two)

  • --duration : measurement time per case (seconds)

A test case is skipped when:

block_size × batch_size × num_threads > total_buffer_size

Multi-target initiator

Use a comma-separated --target_seg_name value when one initiator process should send traffic to multiple target segments:

./tebench \
  --backend=tent \
  --metadata_type=p2p \
  --target_seg_name=<SEG_A>,<SEG_B>,<SEG_C> \
  --op_type=read \
  --start_num_threads=3 \
  --max_num_threads=3

Thread i selects target i % target_count. Within the selected target, the local target-thread index is i / target_count, so increasing the thread count spreads traffic across targets before advancing to the next buffer slot inside each target. --target_gpu_id shifts the per-target buffer slot and does not change the target selection order.

For multi-node benchmarks, tebench intentionally stays at the endpoint level: each process publishes its own segment, and an external launcher decides which target segment list each initiator receives. This keeps M-to-N, fan-out, incast, and all-to-all topologies as different launch configurations over the same comma-separated --target_seg_name primitive.

For multiple initiator processes sharing the same target segment, use --target_offset and --target_range_size to partition the remote address space. The range size is relative to each initiator; tebench also validates that target_offset + relative_offset + transfer_size stays inside the actual target buffer.

# Initiator 0 uses [0, 512MiB)
./tebench --target_seg_name=<SEG> --target_offset=0 --target_range_size=536870912

# Initiator 1 uses [512MiB, 1GiB)
./tebench --target_seg_name=<SEG> --target_offset=536870912 --target_range_size=536870912

For read-only verification with multiple readers, first write deterministic data using --op_type=write_seed, then run readers with --op_type=read_verify against the same target range. read_verify performs pure READs and validates the local buffer without modifying the remote data. Do not combine these modes with --check_consistency; --check_consistency remains the existing WRITE→READ self-check mode.

5.5 GPU Affinity#

  • --local_gpu_id : initiator base GPU ID

  • --target_gpu_id : target base GPU ID

Each worker thread uses:

gpu_id + thread_id

5.6 Backend, Transport, and Metadata#

Backend

  • --backend : classic | tent (default: tent)

Transport (TENT only)

  • --xport_type : selects the benchmark transport, such as tcp, hp_tcp, rdma, shm, or nvlink, and disables the other selectable transports. MLU VRAM runs also enable the local-copy transport needed for host staging. Empty means no transport is explicitly enabled or disabled by tebench — the engine reads the transport enable list from the MC_TENT_CONF config file (see Section 5.8 for multi-transport scenarios). Classic --backend=classic --xport_type=shm also uses this flag: DRAM buffers come from POSIX shm and are mbind’d onto the same NUMA node as numa_alloc_onnode before registering cpu:<node>.

  • --tent_intent_type : attach a standard transfer intent to every request, such as foreground_get, background_prefetch, or checkpoint. This is useful for validating intent-specific transport and QoS policy selection.

Metadata service

  • --metadata_type : p2p | etcd | redis | http (default: p2p)

  • --metadata_url_list : comma-separated URLs (ignored in p2p mode)

5.7 QoS Reporting#

  • --qos_classes : class/thread/SLO/weight contract described in Section 4.1

  • --workload_classes_json : class-specific transfer shape, intent, deadline, and QoS contract described in Section 4.2

  • --qos_link_capacity_gbps : measured usable link capacity in decimal GB/s

  • --qos_output_jsonl : append one schema-versioned JSON object per benchmark configuration

  • --result_output_jsonl : append aggregate and per-target metrics for each benchmark configuration

  • --split_output_jsonl : initiator-only per-transfer submit/wait JSONL described in Section 4.4; empty disables it

QoS mode intentionally requires a fixed thread count. Sweep offered load by running explicit cases with different class thread allocations so every output record has an unambiguous workload contract.

5.8 Mixed DRAM+VRAM and Multi-Transport#

By default, tebench allocates buffers of a single --seg_type. --seg_type_mix=dram,vram allocates both host and device buffers. In a USE_MLU build, combine it with --xport_type=tcp or hp_tcp to transfer both memory types using host staging for MLU buffers; see Section 3.4.

The following CUDA example uses SHM for DRAM→DRAM and NVLink for VRAM→VRAM within one process. It requires USE_CUDA=ON and enables multiple transports through MC_TENT_CONF:

  1. Enable multiple transports via MC_TENT_CONF — point it at a JSON config file (or inline JSON string) that enables the transports you want active:

    {
    "transports": {
        "shm":    {"enable": true},
        "nvlink": {"enable": true}
    }
}

    Leave --xport_type empty so tebench does not override the config’s enable list. MC_TENT_CONF is read by the engine’s ConfigHelper and applies to both target and initiator.

  2. Register a mixed DRAM+VRAM segment via --seg_type_mix:

    # Target: mixed DRAM+VRAM segment, all GPUs
MC_TENT_CONF=/path/to/config.json ./tebench --backend=tent \
  --metadata_type=p2p --seg_name=tgtmix --seg_type_mix=dram,vram \
  --local_gpu_id=-1 &

# Initiator: mixed DRAM+VRAM, 8 threads (4 DRAM→SHM, 4 VRAM→NVLink)
MC_TENT_CONF=/path/to/config.json ./tebench --backend=tent \
  --metadata_type=p2p --target_seg_name=<SEG> --seg_type_mix=dram,vram \
  --local_gpu_id=-1 --op_type=write --duration=30 \
  --start_num_threads=8 --max_num_threads=8

    Initiator threads round-robin across the listed seg_types by thread_id (even→DRAM, odd→VRAM). The engine selects transport based on the target buffer’s location — SHM for cpu:N buffers, NVLink for cuda:N buffers — so /metrics will show both transport="shm" and transport="nvlink" labels with non-zero counts from a single metrics endpoint.

  3. --local_gpu_id=-1 is recommended for VRAM so each worker thread gets its own GPU buffer. With the default --local_gpu_id=0, all VRAM threads share one GPU buffer and contend, which underrepresents NVLink throughput.

Use --start_num_threads ≥ 2 to exercise both memory types. When --seg_type_mix is empty, tebench uses --seg_type for all buffers.

Contents