Multi-Tenant Admin Control-Plane Design

Contents

Multi-Tenant Admin Control-Plane Design#

Background#

Store-RS already enforces multi-tenant isolation inside the runtime and control plane:

  • request identity carries tenant / domain / object_set / qos_tier

  • route identity is namespace-aware

  • quota checks happen on write paths

  • route control RPCs already take a namespace parameter

  • metadata traits already support scope-aware route listing

What is missing is a management surface for operators to configure and inspect tenant-level policy without pushing that responsibility into the request path.

This document proposes extending mooncake-store-rs-admin into that management surface.

Goal#

Add a multi-tenant management plane to mooncake-store-rs-admin so operators can:

  • configure tenant-scoped control-plane policy

  • inspect tenant-scoped state

  • run explicit tenant-scoped maintenance and reconciliation commands

The admin CLI is not responsible for executing resource isolation in the request path. Runtime clients and storage/control-plane services remain the enforcement point.

Non-Goals#

This design does not move these responsibilities into admin:

  • request-path tenant identification

  • request-path quota enforcement

  • route selection during reads/writes

  • allocator reservation decisions during data writes

  • background eviction / reclaim execution

  • transport pacing / fairness enforcement

Admin only manages policy and triggers explicit maintenance actions.

Current State#

Existing admin surface#

mooncake-store-rs-admin currently provides a single explicit metadata maintenance command:

  • cleanup-stale-segments

Current entrypoint:

  • crates/mooncake-store-rs-admin/src/bin/mooncake-store-rs-admin.rs

Existing runtime/control-plane capabilities relevant to multi-tenancy#

Scope-aware object identity#

The runtime already models object identity as:

  • tenant

  • domain

  • object_set

  • qos_tier

  • key

Relevant code:

  • crates/mooncake-store-client/src/client/types.rs

  • crates/mooncake-store-core/src/identity.rs

  • crates/mooncake-store-core/src/identity_codec.rs

Tenant-aware route/control interfaces#

Control-plane route APIs already accept a namespace string.

Relevant code:

  • crates/mooncake-store-client/src/control_plane/mod.rs

  • crates/mooncake-store-client/src/control_plane/client.rs

  • crates/mooncake-store-client/src/control_plane/server.rs

Scope-aware metadata helpers#

Metadata traits already expose route listing by scope.

Relevant code:

  • crates/mooncake-store-core/src/traits.rs

Runtime-side quota enforcement#

Namespace quota exists today, but it is currently a runtime-side configuration object rather than a centralized policy store.

Relevant code:

  • crates/mooncake-store-client/src/client/types.rs

  • crates/mooncake-store-client/src/client/mod.rs

Status Update#

The core design in this document is now substantially implemented:

  • unified tenant policy is stored in metadata

  • mooncake-store-rs-admin is the preferred tenant policy entrypoint

  • policy list --tenant <tenant> is pushed down to metadata backends instead of scanning every tenant policy in process

  • Store-RS runtime consumes and enforces tenant-scoped routing/quota/fairness/shaping/placement defaults

  • non-default object scopes use deterministic full-scope route keys, while the default namespace keeps the legacy tenant::logical_key route key

The remaining follow-up work is compatibility cleanup: legacy runtime-local builder / Python / CLI / env knobs are still accepted as compatibility fallbacks, but should no longer be presented as the preferred policy authoring surface.

Problem Statement#

Today, multi-tenant control-plane behavior is partly implemented, but operators lack a first-class management surface for:

  • writing tenant policies into a durable store

  • reading back current tenant policy

  • inspecting scope-specific route/control-plane state

  • triggering explicit scope-specific reconciliation or cleanup

As a result, tenant configuration is fragmented across runtime builder options, deployment config, and ad-hoc inspection.

We need a dedicated management plane that:

  1. is explicit and operator-driven like existing admin behavior

  2. can persist tenant policy centrally

  3. can query both metadata and control-plane state

  4. does not become part of the request-path fast path

Design Summary#

Extend mooncake-store-rs-admin with two new capability groups:

  1. Policy management

    • get/set/delete tenant-scoped policy records in metadata

  2. Observation and explicit maintenance

    • inspect tenant-scoped routes and placement state

    • run explicit tenant-scoped reconciliation/cleanup actions

The design adds a new Tenant Policy Store in metadata and a thin Admin Service Layer inside the CLI that decides whether each command should talk to:

  • metadata directly

  • control-plane RPC

  • both

Architecture#

        flowchart TB
    operator[Operator]
    admin[mooncake-store-rs-admin]

    subgraph Admin Layer
        parser[CLI command parser]
        service[Admin service layer]
        policyapi[Tenant policy API]
        inspectapi[Inspection API]
        maintapi[Maintenance API]
    end

    subgraph Control Plane
        metadata[(Metadata backend\nRedis / etcd / memory)]
        authority[Route authority service]
        allocator[Allocator service]
        eviction[Eviction service]
    end

    subgraph Runtime/Data Plane
        clients[Store clients]
        storage[Storage runtimes]
    end

    operator --> admin
    admin --> parser
    parser --> service
    service --> policyapi
    service --> inspectapi
    service --> maintapi

    policyapi --> metadata
    inspectapi --> metadata
    inspectapi --> authority
    inspectapi --> allocator
    inspectapi --> eviction
    maintapi --> metadata
    maintapi --> authority
    maintapi --> allocator
    maintapi --> eviction

    metadata --> clients
    metadata --> storage
    authority --> clients
    allocator --> clients
    eviction --> storage

Architectural principle#

  • Admin writes policy and asks questions

  • Runtime enforces policy and performs isolation work

This keeps the fast path unchanged while still giving operators a real control surface.

Proposed Components#

1. Tenant Policy Store#

Add a durable metadata-backed store for tenant-scoped control-plane policy.

Stored entity#

A new record, conceptually:

TenantPolicy {
  scope: NamespaceScopeSelector,
  quota: {
    max_bytes,
    max_objects,
  },
  placement: {
    default_replica_count,
    prefer_local,
    preferred_storage_owners,
    preferred_segments,
  },
  qos: {
    default_qos_tier,
    execution_fairness,
    bandwidth_shaping,
  },
  routing: {
    route_scope_label,
    allowed_route_authorities,
  },
  metadata: {
    version,
    updated_at_ms,
    updated_by,
  }
}

Scope model#

Policy can target one of these levels:

  1. tenant

  2. tenant + domain

  3. tenant + domain + object_set

This matches the existing namespace identity model and avoids inventing a second scope system.

Storage backend abstraction#

Implemented metadata trait methods include:

  • get_tenant_policy(scope)

  • get_tenant_policies(scopes) for exact batched policy reads

  • list_tenant_policies(tenant) for optional tenant-filtered admin listing

  • put_tenant_policy(policy, expected_version)

  • delete_tenant_policy(scope, expected_version)

Tenant-filtered listing is a backend concern: Redis and etcd narrow the read to the tenant-root policy and that tenant’s nested subtree instead of scanning all tenant policies and filtering in the admin process.

2. Admin Service Layer#

Inside mooncake-store-rs-admin, add an internal service layer that maps CLI commands to backends.

Responsibilities#

  • parse scope flags into a namespace selector

  • route commands to metadata vs control-plane RPC

  • perform optimistic concurrency checks for policy updates

  • assemble user-facing reports

  • keep destructive operations explicit

Why a service layer#

The current admin binary directly constructs RedisMetadataBackend and calls one method. That is fine for one command, but it will not scale once some commands need metadata and others need authority/allocator inspection.

A service layer keeps the CLI parsing separate from operational logic.

3. Control-Plane Admin Adapters#

For commands that need live control-plane state, admin should talk to a thin adapter over existing control-plane primitives.

Candidate operations#

Read-only inspection#

  • list routes in scope

  • list routes by replica owner in scope

  • summarize authority placement for a tenant

  • summarize allocator-visible ownership or capacity by tenant

Explicit maintenance#

  • reconcile mirrored authorities for a scope

  • remove stale route mirrors for a scope

  • rebuild or repair scope summaries

  • optional future drain orchestration hooks

Important constraint#

These are operator-triggered commands, not background controllers.

That matches the same philosophy as cleanup-stale-segments: avoid automatic guesses when the blast radius is tenant data/control metadata.

CLI Design#

Top-level structure#

mooncake-store-rs-admin [global flags] <group> <subcommand> [flags]

Global flags#

  • --metadata-url (alias --metadata_url)

  • --keyspace

  • --trace-filter

  • optional future --control-address or --authority-runtime

Command groups#

1. Policy management#

mooncake-store-rs-admin policy get --tenant t1
mooncake-store-rs-admin policy set --tenant t1 --max-bytes 1TiB --max-objects 10000000
mooncake-store-rs-admin policy delete --tenant t1
mooncake-store-rs-admin policy list --tenant t1

2. Scope inspection#

Implemented Phase 2 read-only inspection commands currently focus on strict-quota metadata:

mooncake-store-rs-admin quota state --tenant t1
mooncake-store-rs-admin quota object --tenant t1 --key object-a
mooncake-store-rs-admin quota reservations --tenant t1 --state pending

The same data is also exposed over the admin HTTP surface:

GET /v1/tenant-quotas/<tenant>[/<domain>[/<object_set>]]
GET /v1/tenant-quotas/<tenant>[/<domain>[/<object_set>]]/reservations?state=pending|finalized|aborted
POST /v1/tenant-quotas/<tenant>[/<domain>[/<object_set>]]/reservations/<reservation_id>/abort
POST /v1/tenant-quotas/<tenant>[/<domain>[/<object_set>]]/reconcile
GET /v1/tenant-object-accounting/<tenant>/objects/<key>
GET /v1/tenant-object-accounting/<tenant>/<domain>/objects/<key>
GET /v1/tenant-object-accounting/<tenant>/<domain>/<object_set>/objects/<key>

Quota state and reservation listing intentionally collapse to the tenant-root scope because Phase 1 stores strict quota state authoritatively at the tenant root. Object-accounting lookup still accepts nested scope selectors so operators can ask for the logical object they care about while the service derives the canonical scoped key. The Phase 5 maintenance surface now adds explicit reservation abort and reconcile commands with --dry-run support so operators can inspect expired or visible-but-unfinalized reservations before applying repair.

3. Explicit maintenance#

mooncake-store-rs-admin maintain reconcile-routes --tenant t1
mooncake-store-rs-admin maintain cleanup-stale-routes --tenant t1
mooncake-store-rs-admin maintain cleanup-stale-segments --tenant t1

Note: existing global stale-segment cleanup remains, but tenant-scoped variants can be layered on top when enough route/segment metadata is scope-identifiable.

Proposed CLI Scope Flags#

All multi-tenant admin commands should share a common selector surface:

  • --tenant <tenant> (required for tenant-scoped commands)

  • --domain <domain> (optional)

  • --object-set <object-set> (optional)

  • --qos-tier <tier> (optional for some policy commands)

Rules:

  • domain requires tenant

  • object-set requires tenant and usually domain

  • commands must print the resolved canonical scope before executing

Policy Model#

v1 policy fields#

To keep the first version small, v1 should support only fields that already have strong runtime analogs:

Quota#

  • max_bytes

  • max_objects

Placement defaults#

  • default_replica_count

  • prefer_local

  • preferred_storage_owners

  • preferred_segments

QoS defaults#

  • default_qos_tier

  • execution_fairness.max_remote_batch_items_per_tenant

  • bandwidth_shaping.max_remote_batch_bytes

  • bandwidth_shaping.max_remote_batch_burst_items

  • bandwidth_shaping.max_inflight_bytes_per_batch

Explicitly excluded from v1#

  • per-request override policy templates

  • transport backend selection

  • lifecycle automation policies

  • dynamic background remediation policies

Runtime Consumption Model#

The admin tool writes tenant policy. Runtime clients consume it.

Proposed runtime behavior#

  1. client starts

  2. client loads default cluster policy as today

  3. client loads tenant policy snapshot for configured/default tenant scope

  4. request path resolves the most specific matching policy

  5. runtime applies policy during existing enforcement points

Matching order#

Most specific wins:

  1. tenant + domain + object_set

  2. tenant + domain

  3. tenant

  4. runtime default

This preserves backward compatibility and keeps config layering intuitive. Object route identity follows the same namespace boundary: default-scope routes keep the legacy tenant::logical_key key, while non-default scopes include tenant, domain, object_set, and logical_key in the deterministic route key.

Policy Update Flow#

        sequenceDiagram
    participant Op as Operator
    participant Admin as mooncake-store-rs-admin
    participant Meta as Metadata backend
    participant Client as Store client runtime

    Op->>Admin: policy set --tenant=t1 --max-bytes=1TiB
    Admin->>Meta: read current tenant policy
    Meta-->>Admin: current policy + version
    Admin->>Meta: CAS upsert tenant policy(version=n->n+1)
    Meta-->>Admin: success
    Admin-->>Op: updated policy report
    Client->>Meta: refresh tenant policy snapshot
    Meta-->>Client: latest tenant policy
    Client->>Client: apply on existing enforcement points

Why CAS#

We should use optimistic concurrency for policy writes so operators do not unknowingly overwrite each other.

Scope Inspection Flow#

        sequenceDiagram
    participant Op as Operator
    participant Admin as mooncake-store-rs-admin
    participant Meta as Metadata backend
    participant CP as Control plane authority

    Op->>Admin: inspect routes --tenant=t1
    Admin->>Meta: list routes in scope(t1)
    Meta-->>Admin: route snapshot
    Admin->>CP: optional authority validation for scope(t1)
    CP-->>Admin: live authority view
    Admin->>Admin: compare metadata vs authority view
    Admin-->>Op: route report + drift summary

This command should support both:

  • metadata-only inspection

  • metadata + live authority cross-check

The latter is slower but better for debugging divergence.

Explicit Maintenance Flow#

        sequenceDiagram
    participant Op as Operator
    participant Admin as mooncake-store-rs-admin
    participant Meta as Metadata backend
    participant CP as Control plane authority

    Op->>Admin: maintain reconcile-routes --tenant=t1
    Admin->>Meta: list routes in scope(t1)
    Meta-->>Admin: candidate routes
    Admin->>CP: inspect mirrored authority state
    CP-->>Admin: live route replicas / authority copies
    Admin->>Admin: build reconciliation plan
    Admin->>CP: apply explicit repair requests
    CP-->>Admin: per-route results
    Admin-->>Op: reconciliation report

Important invariant#

The admin tool computes and submits an explicit repair action. It does not become a continuously running reconciliation controller.

Metadata Model Proposal#

Key shape#

This document does not lock the exact Redis key strings, but the logical structure should be:

  • tenant_policies/<tenant>

  • tenant_policies/<tenant>/<domain>

  • tenant_policies/<tenant>/<domain>/<object_set>

  • optional index for listing by tenant prefix

Serialization#

Use the same serde-based pattern used elsewhere in store metadata records.

Versioning#

Each policy record should contain:

  • version

  • updated_at_ms

  • updated_by

updated_by should be an operator-supplied or host-derived identifier to simplify audits.

Security / Safety Model#

Admin is powerful and must stay explicit.

Safety principles#

  1. all write operations are explicit subcommands

  2. no background policy drift correction in v1

  3. maintenance commands print scope and counts before applying

  4. batch changes support --dry-run where practical

  5. policy writes use CAS/version checks

  6. scope parsing must reject ambiguous input

Access model#

This design does not introduce authz in the CLI itself. It assumes access control is enforced by:

  • deployment environment

  • metadata backend credentials

  • network reachability to control-plane endpoints

If stronger control is required later, the right place is likely backend-side or service-side auth, not local CLI argument checks.

Compatibility and Rollout#

Backward compatibility#

  • existing runtimes continue working with no tenant policy records present

  • absence of tenant policy means runtime defaults apply

  • existing cleanup-stale-segments command remains unchanged

Rollout phases#

Phase 1: metadata-backed tenant policy#

Add:

  • metadata trait + Redis/etcd implementations

  • admin policy get/set/delete/list

  • no runtime consumption yet beyond optional debug reads

Phase 2: runtime policy consumption#

Add:

  • runtime tenant policy snapshot loading

  • scope matching precedence

  • enforcement reuse through existing quota / placement / QoS hooks

Phase 3: admin inspection#

Add:

  • inspect routes

  • inspect owners

  • inspect quota-usage

  • metadata-only first, live authority cross-check second

Phase 4: explicit maintenance#

Add:

  • maintain reconcile-routes

  • scope-aware stale cleanup helpers

  • --dry-run

Open Questions#

  1. Should tenant policy live only in metadata, or also be cached in route authorities?

    • Recommendation: metadata is source of truth; runtime caches locally.

  2. Do we want scope selectors without domain but with object_set?

    • Recommendation: no in v1; keep hierarchy strict.

  3. Should qos_tier be part of policy scope or only a policy field?

    • Recommendation: policy field in v1, not part of scope identity.

  4. Should admin inspect allocator state through direct metadata reads or control-plane RPC?

    • Recommendation: metadata first where possible; RPC only for live/derived state.

  5. Do we need tenant policy inheritance visualization in CLI output?

    • Recommendation: yes, especially for policy get --effective.

File-Level Impact#

New / changed likely areas#

Admin CLI#

  • crates/mooncake-store-rs-admin/src/bin/mooncake-store-rs-admin.rs

    • add grouped subcommands

    • add shared scope flag parsing

    • call new admin service layer

Metadata traits and implementations#

  • crates/mooncake-store-core/src/traits.rs

    • add tenant policy trait methods

  • crates/mooncake-store-rs-metadata/src/redis_backend.rs

    • store/load/list/delete tenant policy

  • crates/mooncake-store-rs-metadata/src/etcd_backend.rs

    • matching implementation

  • crates/mooncake-store-rs-metadata/src/in_memory.rs

    • test implementation

Core policy types#

  • likely new policy structs in mooncake-store-core

    • keeps admin, metadata, and runtime sharing one schema

Runtime consumption#

  • crates/mooncake-store-client/src/client/*

    • load effective tenant policy

    • feed existing quota / placement / QoS hooks

Optional later control-plane admin helpers#

  • crates/mooncake-store-client/src/control_plane/*

    • only if inspection/maintenance needs new RPCs

Why This Design Fits Store-RS#

This design matches existing repository principles:

  • explicit operator action for cleanup and maintenance

  • metadata as source of truth for durable policy

  • runtime/control plane as the execution engine

  • no new always-on coordinator for management tasks

  • preserves current route-control and data-plane architecture

In short:

  • mooncake-store-rs-admin becomes the management plane entrypoint

  • metadata becomes the durable tenant policy plane

  • runtime/control plane remains the enforcement plane

That separation is the cleanest way to add tenant management without polluting the fast path.

Contents