Strict Tenant Quota E2E Test Guide

Strict Tenant Quota E2E Test Guide#

This document is for QA handoff of the new strict tenant quota end-to-end coverage in Store-RS.

Goal#

Validate that Store-RS exercises the new metadata-authoritative tenant quota logic in the real local e2e harness, not only in unit tests.

The e2e scenario proves all of the following in one run:

  • tenant quota policy is authored in metadata before the quota-scoped writer starts

  • a write within quota succeeds

  • quota usage becomes committed metadata state after finalize

  • a second over-limit write is rejected

  • the rejection does not drift quota state or reservation count

  • delete refunds quota at authoritative delete time

  • the refunded quota can be reused immediately

  • object accounting and reservation records are visible in metadata for the quota test tenant

Code Location#

The strict quota e2e scenario lives in:

  • crates/mooncake-store-e2e/src/main.rs

Relevant helpers:

  • put_tenant_quota_policy(...)

  • verify_strict_tenant_quota(...)

Preconditions#

Before running the e2e locally, ensure:

  • Redis is available on the configured port

  • the workspace builds successfully

  • the local environment can run the standard e2e harness

  • mooncake-store-rs-admin can reach the same metadata namespace if you want follow-up inspection after the run

The default local script uses the same conventions as the rest of the repository.

Expected Success Signal#

The run should finish with an e2e ok: line that includes:

strict tenant quota

Current success output includes:

e2e ok: single put/get, strict tenant quota, batch put/get, request-level replication policy, true delete reclaim, routed remote write, multi-replica publish, registered-buffer path, overwrite reclaim, multi-tenant, scale-out, elastic expand-shrink, true client shrink, hot-upgrade, rdma bandwidth isolation

What the Scenario Checks#

The strict quota scenario uses a dedicated tenant:

  • tenant-quota-e2e

It then performs these checks in order.

1. Seed tenant quota policy#

A tenant-root quota policy is written to metadata before constructing the quota-specific writer.

Test shape:

  • max_bytes = value_size

  • max_objects = 1

Expected result:

  • the subsequent writer for tenant-quota-e2e resolves and uses this strict quota policy

2. Successful admitted write#

The scenario writes one object that exactly fits the quota.

Expected result:

  • write succeeds

  • another client can read the object back from the same tenant

  • metadata quota state shows:

    • used_bytes == value_size

    • used_objects == 1

    • pending_reserved_bytes == 0

    • pending_reserved_objects == 0

  • metadata object accounting exists and shows:

    • active state

    • committed length equal to payload size

  • metadata reservations contain a finalized positive reservation for the object

3. Strict rejection without state drift#

The scenario attempts a second object of the same size in the same tenant.

Expected result:

  • write fails with a tenant quota conflict

  • quota state remains unchanged from the successful first write

  • reservation count remains unchanged

This proves the new metadata-authoritative admission path rejects without corrupting committed state.

4. Authoritative delete refund#

The scenario deletes the first object.

Expected result:

  • quota state returns to zero used bytes and zero used objects

  • pending counters remain zero

  • object accounting for the deleted object disappears

  • reservations contain a finalized negative reservation for the refund

This proves quota is refunded at authoritative delete time, not only after later reclaim.

5. Immediate quota reuse#

The scenario writes a new object of the same size after delete.

Expected result:

  • write succeeds immediately

  • another client can read it back

This is the final proof that delete refund made quota reusable right away.

Suggested QA Checklist#

When coordinating with QA, ask them to verify:

  • the script exits successfully

  • the final success line includes strict tenant quota

  • there is no error indicating quota state drift, missing accounting, or missing reservations

  • the scenario passes with the recommended fast command above

  • metadata inspection after the run matches the expected tenant-root strict quota behavior

Optional Follow-Up Checks#

If QA needs deeper inspection, they can also confirm the repository already has supporting lower-level tests for strict quota semantics in:

  • crates/mooncake-store-client/src/client/tests/mod.rs

Examples include coverage for:

  • over-limit write rejection

  • overwrite delta charging

  • delete refund behavior

  • routed batch all-or-nothing quota admission

  • tenant-local quota eviction observability via mooncake_store_tenant_local_eviction_total

  • successful eviction-triggered writes remaining readable while tenant usage stays within quota

  • cross-tenant isolation during tenant-local quota recovery

Focused commands for the tenant-local recovery path:

cargo test -p mooncake-store-client \
  namespace_quota_evicts_within_same_tenant_before_rejecting \
  -- --nocapture

cargo test -p mooncake-store-client \
  tenant_local_quota_eviction_does_not_touch_other_tenants \
  -- --nocapture

If you run the commands inside a container, enter the actual repository root first before invoking Cargo; running from the container’s default /workspace directory can target the wrong tree.

Those tests are complementary. The e2e described here is the process-level proof that the Redis-backed runtime path exercises the new logic in a real harness.

Notes#

  • The e2e is intentionally focused and does not try to cover every admin repair flow.

  • Its purpose is to prove the newly added strict quota runtime path is wired into the real end-to-end execution path.

  • Repair-specific behaviors such as admin reconcile/abort remain covered by targeted tests elsewhere in the repository.

  • The recommended mental model is now admin-authored policy plus runtime enforcement, not runtime-local quota configuration alone.

Contents