Multi-Tenant Deployment

Multi-Tenant Deployment#

Configure the Master#

File Connector#

Tenant quota admission is disabled by default. Enable strict multi-tenant mode on the master when you want memory writes admitted against connector-managed per-tenant quota:

mooncake_master \
  --enable_multi_tenants=true \
  --tenant_quota_connector_type=file \
  --tenant_quota_connector_uri=/etc/mooncake/tenant_quotas.yaml

etcd Connector#

You can also store the same YAML policy in etcd when Mooncake Store is built with STORE_USE_ETCD=ON:

mooncake_master \
  --enable_multi_tenants=true \
  --cluster_id=mooncake_cluster \
  --tenant_quota_connector_type=etcd \
  --tenant_quota_connector_uri=127.0.0.1:2379

The etcd connector stores the policy at mooncake-store/<cluster_id>/tenant_quota_policy. If the key does not exist, the master starts with an empty policy so the first tenant policy can be created through the admin API. It shares the process-wide store etcd client used by HA/oplog, so if HA or oplog also uses etcd, tenant_quota_connector_uri must match those etcd endpoints.

Define the Tenant Policy#

The policy must use schema version 1; tenant names must be non-empty, unique, must not start with _, and must not contain NUL or control characters; quotas must be positive integers with optional B, KB, MB, GB, or TB units:

version: 1

tenants:
  - name: tenant-a
    quota: 200GB

  - name: tenant-b
    quota: 500GB

When strict multi-tenant mode is enabled, write requests must include a registered tenant. The default tenant is not special unless it is explicitly registered in the connector policy.

Manage Tenant Quotas#

The same HTTP port used for metrics exposes the tenant quota admin API:

# List tenant quota snapshots
curl -s http://<master_host>:9003/api/v1/tenant_quotas

# Query one tenant
curl -s "http://<master_host>:9003/api/v1/tenant_quotas?tenant_id=tenant-a"

# Upsert an explicit policy. Explicit tenant policies must be between 1 byte
# and 2^63 - 1 bytes.
curl -s -X PUT "http://<master_host>:9003/api/v1/tenant_quotas?tenant_id=tenant-a" \
  -H 'Content-Type: application/json' \
  -d '{"requested_quota_bytes":2147483648}'

# Delete an explicit policy. The tenant must not own objects or quota usage.
curl -s -X DELETE "http://<master_host>:9003/api/v1/tenant_quotas?tenant_id=tenant-a"

Each tenant quota snapshot returns:

{
  "success": true,
  "data": {
    "tenant_id": "tenant-a",
    "requested_quota_bytes": 2147483648,
    "effective_quota_bytes": 2147483648,
    "charged_bytes": 0,
    "admission_closed": false,
    "over_quota": false,
    "has_explicit_policy": true
  }
}

charged_bytes includes completed MEMORY replicas and in-flight MEMORY allocations. Put, Copy, Move, and promotion charge quota when admission starts; failed, revoked, partially completed, or expired operations refund the unused charge. admission_closed is true when the account rejects new writes, including after its explicit policy is removed.

In HA mode, quota admin requests are served only by the active master service. Standby, candidate, or inactive services return HTTP 503. If strict multi-tenant mode is disabled, the quota admin API returns HTTP 409 with UNAVAILABLE_IN_CURRENT_MODE. Deleting a non-empty tenant returns HTTP 409 with TENANT_NOT_EMPTY.

SGLang#

When Mooncake is used as the HiCache storage backend, set tenant_id in the Mooncake backend configuration:

--hicache-storage-backend mooncake \
--hicache-storage-backend-extra-config \
  '{"master_server_address":"127.0.0.1:50051","tenant_id":"tenant-a"}'

Alternatively, add tenant_id to the JSON file selected by SGLANG_HICACHE_MOONCAKE_CONFIG_PATH, or use MOONCAKE_TENANT_ID when loading the Mooncake configuration from environment variables. SGLang forwards the resolved value to the Mooncake client.

All prefill, decode, and replica instances that should share KV cache entries must use the same tenant_id and compatible model and release namespaces.

vLLM#

Add tenant_id to the Mooncake client JSON configuration:

"tenant_id": "tenant-a"

Point MOONCAKE_CONFIG_PATH at that file and enable MooncakeStoreConnector through --kv-transfer-config:

MOONCAKE_CONFIG_PATH=/path/to/mooncake_config.json \
vllm serve <model> \
  --kv-transfer-config \
  '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}'

MooncakeStoreConnector reads the JSON during initialization and passes its tenant ID to the Mooncake client.

All prefill, decode, and replica instances that should share KV cache entries must use the same tenant_id and compatible model and release namespaces.

Contents