Benchmark performance on NVIDIA A10#
Here are some preview MooncakeStore benchmark results on A10 with “Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4”.
Varying PD ratio (input length = 1024, qps = 2, output length =6, num of requests = 200)#
Configuration |
Backend |
Duration (s) |
Output Token Throughput (tok/s) |
Total Token Throughput (tok/s) |
Mean TTFT (ms) |
Median TTFT (ms) |
P99 TTFT (ms) |
Mean TPOT (ms) |
Median TPOT (ms) |
P99 TPOT (ms) |
Mean ITL (ms) |
Median ITL (ms) |
P99 ITL (ms) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2P2D tp = 1 |
Redis |
99.47 |
12.06 |
2042.75 |
844.28 |
666.84 |
2270.91 |
16.88 |
11.57 |
104.83 |
16.84 |
11.56 |
239.67 |
MooncakeStore (TCP) |
99.44 |
12.07 |
2043.30 |
817.43 |
639.48 |
1969.89 |
12.49 |
11.55 |
45.52 |
12.46 |
11.55 |
15.31 |
|
MooncakeStore (RDMA) |
99.33 |
12.08 |
2045.57 |
763.58 |
604.22 |
2030.34 |
12.43 |
11.53 |
43.02 |
12.39 |
11.52 |
15.40 |
|
2P2D tp = 2 |
Redis |
98.92 |
12.13 |
2054.12 |
397.20 |
352.37 |
782.44 |
9.00 |
8.05 |
36.06 |
8.97 |
8.03 |
13.94 |
MooncakeStore (TCP) |
98.81 |
12.14 |
2056.43 |
327.91 |
309.38 |
573.36 |
8.33 |
8.04 |
17.62 |
8.30 |
8.03 |
11.23 |
|
MooncakeStore (RDMA) |
98.74 |
12.15 |
2057.79 |
271.25 |
250.11 |
532.00 |
8.34 |
8.12 |
14.70 |
8.31 |
8.10 |
11.12 |
|
3P3D (1 remote P, 1 remote D) tp = 2 qps = 2 |
Redis |
98.89 |
12.13 |
2054.80 |
382.73 |
358.18 |
659.31 |
8.07 |
8.02 |
8.86 |
8.04 |
7.99 |
10.14 |
MooncakeStore (TCP) |
98.71 |
12.16 |
2058.47 |
298.71 |
302.74 |
512.84 |
8.06 |
8.04 |
8.54 |
8.03 |
8.02 |
8.88 |
|
MooncakeStore (RDMA) |
98.69 |
12.16 |
2058.88 |
269.73 |
252.96 |
543.38 |
8.13 |
8.04 |
10.49 |
8.10 |
8.02 |
11.29 |
|
4P2D (2 remote P) tp = 2 |
Redis |
98.85 |
12.14 |
2055.66 |
350.39 |
339.15 |
506.78 |
8.54 |
8.01 |
26.42 |
8.51 |
7.99 |
11.43 |
MooncakeStore (TCP) |
98.76 |
12.15 |
2057.56 |
312.32 |
307.50 |
475.79 |
8.29 |
8.03 |
19.87 |
8.25 |
8.01 |
9.51 |
|
MooncakeStore (RDMA) |
98.71 |
12.16 |
2058.59 |
259.87 |
251.23 |
461.96 |
8.20 |
8.05 |
10.20 |
8.17 |
8.03 |
11.50 |
|
2P4D (2 remote D) tp = 2 |
Redis |
98.88 |
12.14 |
2054.90 |
381.91 |
338.25 |
722.00 |
8.07 |
8.05 |
8.55 |
8.04 |
8.02 |
9.15 |
MooncakeStore (TCP) |
98.78 |
12.15 |
2057.11 |
317.42 |
304.53 |
521.66 |
8.07 |
8.03 |
8.75 |
8.04 |
8.02 |
9.62 |
|
MooncakeStore (RDMA) |
98.73 |
12.15 |
2058.02 |
275.13 |
251.57 |
487.43 |
8.18 |
8.06 |
9.19 |
8.15 |
8.05 |
10.53 |