# Mooncake Transfer Engine Benchmark Tool (`tebench`) Guide `tebench` is an end-to-end benchmarking tool for the Mooncake Transfer Engine. It evaluates **bandwidth and latency** across different `(block_size, batch_size, concurrency)` combinations, and supports both the **classic TE** backend and the **new TENT** backend. ## 1. Build and Deployment ### 1.1 Prerequisites * A C++ toolchain capable of building the project * Optional: GPU(s) and corresponding transport stacks (RDMA, shared memory, io_uring, etc.) > Exact dependencies (CUDA / OFED / io_uring / etc.) depend on the transport backend being used. ### 1.2 Build with TENT Enabled Typical out-of-tree build: ```bash mkdir -p build cd build cmake .. -DUSE_TENT=ON cmake --build . -j ``` ## 2. Execution Model `tebench` runs in **two roles**: * **Target (receiver)**: creates a memory segment and waits for connections * **Initiator (sender)**: connects to a target and executes benchmark cases ### Role Selection * If `--target_seg_name` is **empty** → Target mode * If `--target_seg_name` is **set** → Initiator mode ### Backend Selection * `--backend=classic` → classic TE implementation * `--backend=tent` → TENT implementation ## 3. Quick Start ### 3.1 Start Target (First) On the target machine: ```bash ./tebench \ --seg_type=DRAM \ --backend=tent ``` The program will print a command containing a generated segment name: ```text To start initiators, run ./tebench --target_seg_name= --seg_type=DRAM --backend=tent Press Ctrl-C to terminate ``` Copy the printed `--target_seg_name` to the initiator side. ### 3.2 Start Initiator On the initiator machine: ```bash ./tebench \ --target_seg_name= \ --seg_type=DRAM \ --backend=tent \ --op_type=read \ --duration=5 ``` ### 3.3 Request Pacing Use `--request_interval_us=` to add a per-thread delay before each transfer batch. The value is in microseconds; `0` disables pacing. When pacing is enabled, `Avg Lat (us)` includes the pacing gap because it is computed from wall-clock runtime, while `Avg Tx (us)` and Tx percentiles only measure transfer execution time. ## 4. Output Metrics Each output row corresponds to one benchmark configuration. | Column | Description | | -------------- | --------------------------------------------------- | | `BlkSize (B)` | Block size per request (bytes) | | `Batch` | Number of requests per submission | | `BW (GB/S)` | Throughput (total bytes / total time) | | `Avg Inst GB/s` | Average per-transfer instantaneous bandwidth | | `Avg Lat (us)` | Average wall-clock per-operation latency, including pacing or scheduling gaps (scaled by thread count) | | `Avg Tx (us)` | Average per-transfer execution time, excluding gaps | | `P99 Tx (us)` | P99 transfer latency | | `P999 Tx (us)` | P999 transfer latency | A short (~1 second) warmup phase is executed before measurements begin. ### 4.1 QoS Metrics Baseline Use `--qos_classes` to partition a fixed number of worker threads into QoS classes: ```text name:threads:slo_us:weight[:isolated_gbps],... ``` For readability, the same contract can be supplied as JSON with `--qos_classes_json`: ```json [ {"name": "foreground", "threads": 4, "slo_us": 1000, "weight": 4, "isolated_gbps": 12.5}, {"name": "checkpoint", "threads": 12, "slo_us": 0, "weight": 1, "isolated_gbps": 10.0} ] ``` Use only one of `--qos_classes` and `--qos_classes_json`. For example, the following closed-loop mixed workload assigns four workers to an SLO-constrained foreground class and twelve workers to a best-effort checkpoint class: ```bash ./tebench \ --target_seg_name= \ --backend=tent \ --start_num_threads=16 \ --max_num_threads=16 \ --qos_classes=foreground:4:1000:4:12.5,checkpoint:12:0:1:10.0 \ --qos_link_capacity_gbps=25 \ --qos_output_jsonl=qos-results.jsonl ``` The class thread counts must add up to the fixed `start_num_threads` value. An `slo_us` of zero marks a best-effort class. The SLO is a reporting threshold: QoS baseline mode measures whether each completed transfer meets it, without changing request scheduling policy on either backend. The human-readable summary and the optional versioned JSONL record report: | Metric | Definition | | ------ | ---------- | | `slo_attainment` | Fraction of completed batches whose measured end-to-end transfer time is at most `slo_us` | | `p99_us` | P99 end-to-end batch transfer latency for the class | | `goodput_gbps` | Class throughput multiplied by SLO attainment; best-effort classes use attainment 1 | | `weighted_goodput_gbps` | Sum of `weight × goodput_gbps` | | `jain_fairness` | Jain index over per-class `throughput_gbps / weight` | | `isolation_leakage` | `max(0, 1 - mixed_throughput / isolated_throughput)` | | `total_utilization` | Aggregate measured throughput divided by `qos_link_capacity_gbps` | Isolation leakage requires a matching class-only baseline, supplied as the optional fifth class field. Total utilization requires `--qos_link_capacity_gbps`. Missing baselines are emitted as `N/A` in text and `null` in JSON rather than being inferred from the mixed run. Run isolated and mixed cases with the same block size, batch size, transport, memory type, and host pair. JSONL records retain `isolated_throughput_gbps` and `link_capacity_gbps` alongside the derived values so every metric can be recomputed from one record. ### 4.2 Mixed Foreground/Background Traffic The QoS class flags above use one block size, batch size, and request intent for all workers. Use `--workload_classes_json` when classes must generate different transfer shapes concurrently, for example latency-sensitive reads competing with a bulk KV migration: ```bash ./tebench \ --target_seg_name= \ --backend=tent \ --start_num_threads=8 \ --max_num_threads=8 \ --workload_classes_json='[ {"name":"foreground","threads":2,"block_size":4096,"batch_size":1, "intent_type":"foreground_get","deadline_us":250,"slo_us":300,"weight":4}, {"name":"migration","threads":6,"block_size":4194304,"batch_size":2, "intent_type":"migration","slo_us":0,"weight":1} ]' \ --qos_link_capacity_gbps=25 \ --qos_output_jsonl=mixed-traffic.jsonl ``` This mode runs one fixed workload instead of sweeping the global block and batch size flags. Class thread counts must add up to the fixed benchmark thread count. `deadline_us` is optional and controls TENT deadline tagging; `slo_us` remains the independent reporting threshold. Per-class `transferred_bytes` is included in text and JSONL output so throughput is computed from each class's actual transfer size. `--workload_classes_json` is mutually exclusive with `--qos_classes`, `--qos_classes_json`, and the global `--tent_intent_type`. Non-default per-class intents and deadlines require the TENT backend. ### 4.3 Per-Target Metrics Multi-target runs print one `[target-summary]` line per target. Use `--result_output_jsonl=` to also append a schema-versioned JSON record for each benchmark configuration. The record keeps the aggregate operation, byte, and throughput totals plus each target's segment name, assigned thread count, completed operations, transferred bytes, throughput, and latency distribution. The aggregate throughput uses the pooled average worker duration, matching the existing `BW (GB/s)` table calculation. Targets with no assigned worker are retained with zero-valued metrics. This makes an under-provisioned run (`threads < targets`) visible instead of silently dropping targets from the result. ## 5. Runtime Configuration This section summarizes the key runtime options that control workload behavior, resource usage, and backend selection. ### 5.1 Workload Mode (`--op_type`) Controls the transfer pattern executed by the initiator: * `read` — repeated READ transfers * `write` — repeated WRITE transfers * `mix` — alternating WRITE followed by READ (both recorded) Example: ```bash --op_type=write ``` ### 5.2 Data Consistency Check (`--check_consistency`) When enabled, the benchmark validates correctness in `mix` mode: * Source buffers are filled with a known pattern before WRITE * Data is READ back and verified on the initiator Example: ```bash ./tebench --target_seg_name= --op_type=mix --check_consistency=true ``` > Consistency checking introduces CPU-side overhead and should be disabled for pure performance measurements. ### 5.3 Notification Feature (`--notifi`) When enabled, the benchmark sends a notification message along with each transfer batch: * The notification contains the `target_addr` as the message payload * The peer can verify the notification was received correctly by checking the message * Useful for testing notification delivery and end-to-end communication Example: ```bash ./tebench --target_seg_name= --notifi=true ``` > This feature is primarily for testing notification mechanisms. The notification message contains the target address for verification purposes. ### 5.4 Segment, Concurrency, and Memory Layout **Segment and role** * `--seg_name` : local segment name (typically left empty) * `--seg_type` : `DRAM | VRAM` (default: `DRAM`) * `--seg_type_mix` : comma-separated segment types for mixed DRAM+VRAM runs, e.g. `dram,vram`. When set, the target registers buffers of each listed type in one segment, and initiator threads round-robin across them so a single tebench process drives traffic over multiple memory types (and thus multiple transports — SHM for DRAM, NVLink for VRAM) concurrently. Empty falls back to `--seg_type` (single type, existing behavior). See Section 5.8 for usage and the multi-transport configuration it requires. * `--target_seg_name` : target segment name (empty → Target mode). A comma-separated list enables multi-target initiator mode, and worker threads are distributed across all listed target segments. **Scan ranges** * `--total_buffer_size` : total buffer limit (bytes) * `--start_block_size`, `--max_block_size` : block size sweep (powers of two) * `--start_batch_size`, `--max_batch_size` : batch size sweep (powers of two) * `--start_num_threads`, `--max_num_threads` : thread sweep (powers of two) * `--duration` : measurement time per case (seconds) A test case is skipped when: ``` block_size × batch_size × num_threads > total_buffer_size ``` **Multi-target initiator** Use a comma-separated `--target_seg_name` value when one initiator process should send traffic to multiple target segments: ```bash ./tebench \ --backend=tent \ --metadata_type=p2p \ --target_seg_name=,, \ --op_type=read \ --start_num_threads=3 \ --max_num_threads=3 ``` Thread `i` selects target `i % target_count`. Within the selected target, the local target-thread index is `i / target_count`, so increasing the thread count spreads traffic across targets before advancing to the next buffer slot inside each target. `--target_gpu_id` shifts the per-target buffer slot and does not change the target selection order. For multi-node benchmarks, tebench intentionally stays at the endpoint level: each process publishes its own segment, and an external launcher decides which target segment list each initiator receives. This keeps M-to-N, fan-out, incast, and all-to-all topologies as different launch configurations over the same comma-separated `--target_seg_name` primitive. For multiple initiator processes sharing the same target segment, use `--target_offset` and `--target_range_size` to partition the remote address space. The range size is relative to each initiator; tebench also validates that `target_offset + relative_offset + transfer_size` stays inside the actual target buffer. ```bash # Initiator 0 uses [0, 512MiB) ./tebench --target_seg_name= --target_offset=0 --target_range_size=536870912 # Initiator 1 uses [512MiB, 1GiB) ./tebench --target_seg_name= --target_offset=536870912 --target_range_size=536870912 ``` For read-only verification with multiple readers, first write deterministic data using `--op_type=write_seed`, then run readers with `--op_type=read_verify` against the same target range. `read_verify` performs pure READs and validates the local buffer without modifying the remote data. Do not combine these modes with `--check_consistency`; `--check_consistency` remains the existing WRITE→READ self-check mode. --- ### 5.5 GPU Affinity * `--local_gpu_id` : initiator base GPU ID * `--target_gpu_id` : target base GPU ID Each worker thread uses: ``` gpu_id + thread_id ``` --- ### 5.6 Backend, Transport, and Metadata **Backend** * `--backend` : `classic | tent` (default: `tent`) **Transport (TENT only)** * `--xport_type` : `rdma | shm | mnnvl | gds | iouring`. Selects a single transport to enable (all others are disabled). Empty means no transport is explicitly enabled or disabled by tebench — the engine reads the transport enable list from the `MC_TENT_CONF` config file (see Section 5.8 for multi-transport scenarios). * `--tent_intent_type` : attach a standard transfer intent to every request, such as `foreground_get`, `background_prefetch`, or `checkpoint`. This is useful for validating intent-specific transport and QoS policy selection. **Metadata service** * `--metadata_type` : `p2p | etcd | redis | http` (default: `p2p`) * `--metadata_url_list` : comma-separated URLs (ignored in `p2p` mode) ### 5.7 QoS Reporting * `--qos_classes` : class/thread/SLO/weight contract described in Section 4.1 * `--workload_classes_json` : class-specific transfer shape, intent, deadline, and QoS contract described in Section 4.2 * `--qos_link_capacity_gbps` : measured usable link capacity in decimal GB/s * `--qos_output_jsonl` : append one schema-versioned JSON object per benchmark configuration * `--result_output_jsonl` : append aggregate and per-target metrics for each benchmark configuration QoS mode intentionally requires a fixed thread count. Sweep offered load by running explicit cases with different class thread allocations so every output record has an unambiguous workload contract. ### 5.8 Mixed DRAM+VRAM and Multi-Transport By default, tebench allocates buffers of a single `--seg_type` and enables a single `--xport_type` per run. To exercise a mixed-transport workload where the engine's transport selector naturally picks SHM for DRAM→DRAM transfers and NVLink for VRAM→VRAM transfers within one process, combine `--seg_type_mix` with the `MC_TENT_CONF` environment variable: 1. **Enable multiple transports via `MC_TENT_CONF`** — point it at a JSON config file (or inline JSON string) that enables the transports you want active: ```json { "transports": { "shm": {"enable": true}, "nvlink": {"enable": true} } } ``` Leave `--xport_type` empty so tebench does not override the config's enable list. `MC_TENT_CONF` is read by the engine's `ConfigHelper` and applies to both target and initiator. 2. **Register a mixed DRAM+VRAM segment via `--seg_type_mix`**: ```bash # Target: mixed DRAM+VRAM segment, all GPUs MC_TENT_CONF=/path/to/config.json ./tebench --backend=tent \ --metadata_type=p2p --seg_name=tgtmix --seg_type_mix=dram,vram \ --local_gpu_id=-1 & # Initiator: mixed DRAM+VRAM, 8 threads (4 DRAM→SHM, 4 VRAM→NVLink) MC_TENT_CONF=/path/to/config.json ./tebench --backend=tent \ --metadata_type=p2p --target_seg_name= --seg_type_mix=dram,vram \ --local_gpu_id=-1 --op_type=write --duration=30 \ --start_num_threads=8 --max_num_threads=8 ``` Initiator threads round-robin across the listed seg_types by `thread_id` (even→DRAM, odd→VRAM). The engine selects transport based on the target buffer's location — SHM for `cpu:N` buffers, NVLink for `cuda:N` buffers — so `/metrics` will show both `transport="shm"` and `transport="nvlink"` labels with non-zero counts from a single metrics endpoint. 3. **`--local_gpu_id=-1`** is recommended for VRAM so each worker thread gets its own GPU buffer. With the default `--local_gpu_id=0`, all VRAM threads share one GPU buffer and contend, which underrepresents NVLink throughput. Mixed mode requires `--start_num_threads` ≥ 2 (at least one thread per seg_type) and a build with `USE_CUDA=ON` (for VRAM/NVLink support). When `--seg_type_mix` is empty, tebench falls back to `--seg_type` (single type) and `--xport_type` (single transport) — the existing behavior.