Mooncake Store HTTP Service

Mooncake Store HTTP Service#

The Mooncake Store HTTP Service provides RESTful endpoints for cluster management, monitoring, and data operations. This service is embedded within the mooncake_master process and can be enabled alongside the primary RPC services.

Overview#

The HTTP service serves multiple purposes:

  • Metrics & Monitoring: Prometheus-compatible metrics endpoints

  • Cluster Management: Query and manage distributed storage segments

  • Data Inspection: Examine stored objects and their replicas

  • Health Checks: Service availability and status verification

The Python mooncake.mooncake_store_service module also provides a lightweight Store REST API for data operations and standalone segment mount/unmount workflows. Unless configured otherwise, it listens on port 8080.

HTTP Endpoints#

Metrics Endpoints#

/metrics#

Prometheus-compatible metrics endpoint providing detailed system metrics in text format.

Method: GET Content-Type: text/plain; version=0.0.4 Response: Comprehensive metrics including request counts, error rates, latency statistics, and resource utilization

Example:

curl http://localhost:8080/metrics

/metrics/summary#

Human-readable metrics summary with key performance indicators.

Method: GET Content-Type: text/plain; version=0.0.4 Response: Condensed overview of system health and performance metrics

Example:

curl http://localhost:8080/metrics/summary

Data Management Endpoints#

/query_key#

Retrieve replica information for a specific key, including memory locations and transport endpoints.

Method: GET Parameters: key (query parameter) - The object key to query Content-Type: application/json; charset=utf-8 Response: JSON object with success status and replica data array

Example:

curl "http://localhost:8080/query_key?key=my_object"

Success Response (HTTP 200):

{
  "success": true,
  "data": [
    {
      "size_": 1073741824,
      "buffer_address_": 140732000000000,
      "protocol_": "rdma",
      "transport_endpoint_": "192.168.1.100:12345"
    }
  ]
}

Error Response (key not found, HTTP 404):

{
  "success": false,
  "error_code": -704,
  "error_message": "OBJECT_NOT_FOUND"
}

Error Response (service unavailable, HTTP 503):

{
  "success": false,
  "error_code": -1011,
  "error_message": "service plane is not active"
}

/batch_query_keys#

Retrieve replica information for multiple keys in a single request, including memory locations and transport endpoints for each key. The endpoint performs a read-only metadata lookup and does not grant leases, trigger promotion, or update cache-hit metrics.

Method: GET Parameters: keys (query parameter) - Comma-separated list of object keys to query (format: key1,key2,key3) Content-Type: application/json; charset=utf-8 Response: JSON-formatted mapping of keys to their respective replica descriptors

Example:

curl "http://localhost:8080/batch_query_keys?keys=key1,key2,key3"

Response Format:

{
  "success": true,
  "data": {
    "key1": {
      "ok": true,
      "values": [
        {
          "transport_endpoint_": "hostname:port",
          "buffer_descriptor": {...}
        }
      ],
      "disk_values": [
        {
          "file_path": "/path/to/object",
          "object_size": 4096
        }
      ],
      "local_disk_values": [
        {
          "client_id": "12345-67890",
          "object_size": 4096,
          "transport_endpoint": "hostname:port"
        }
      ],
      "nof_values": [
        {
          "transport_endpoint_": "hostname:port",
          "buffer_descriptor": {...}
        }
      ]
    },
    "key2": {
      "ok": false,
      "error": "error message"
    }
  }
}

The values field is always present (empty array when no memory replica exists). The disk_values, local_disk_values, and nof_values fields are optional and only appear when the corresponding replica type is present for the key.

/get_all_keys#

List all keys currently stored in the distributed system.

Method: GET Content-Type: text/plain; version=0.0.4 Response: Newline-separated list of all stored keys

Example:

curl http://localhost:8080/get_all_keys

Segment Management Endpoints#

/get_all_segments#

List all mounted segments in the cluster.

Method: GET Content-Type: text/plain; version=0.0.4 Response: Newline-separated list of segment names

Example:

curl http://localhost:8080/get_all_segments

/query_segment#

Query detailed information about a specific segment, including used and available capacity.

Method: GET Parameters: segment (query parameter) - Segment name to query Content-Type: text/plain; version=0.0.4 Response: Multi-line text with segment details

Example:

curl "http://localhost:8080/query_segment?segment=segment_name"

Response Format:

segment_name
Used(bytes): 1073741824
Capacity(bytes): 4294967296

/get_segments_detail#

Get detailed information of all segments in JSON format, including segment metadata, allocator usage, and status.

Method: GET Content-Type: application/json; charset=utf-8 Response: JSON object containing an array of segment details

Example:

curl http://localhost:8080/get_segments_detail

Response Format:

{
  "total_segments": 2,
  "segments": [
    {
      "segment_name": "segment_0",
      "segment_id": "00000000-0000-0000-0000-000000000001",
      "client_id": "00000000-0000-0000-0000-000000000002",
      "base_address": "0x300000000",
      "size_bytes": 17179869184,
      "size_human": "16 GiB",
      "te_endpoint": "192.168.1.1:12345",
      "protocol": "rdma",
      "status": "MOUNTED",
      "allocator_used_bytes": 1073741824,
      "allocator_capacity_bytes": 17179869184,
      "allocator_usage_percent": 6.25
    }
  ]
}

Fields:

  • total_segments (integer): Total number of segments in the cluster

  • segments (array): Array of segment detail objects

    • segment_name (string): Name of the segment

    • segment_id (string): UUID of the segment

    • client_id (string): UUID of the client that owns the segment

    • base_address (string): Base memory address in hex

    • size_bytes (integer): Segment size in bytes

    • size_human (string): Human-readable segment size

    • te_endpoint (string): Transport endpoint address

    • protocol (string): Transfer protocol (e.g., rdma, tcp)

    • status (string): Current segment status

    • allocator_used_bytes (integer): Bytes currently allocated

    • allocator_capacity_bytes (integer): Total allocator capacity in bytes

    • allocator_usage_percent (number): Percentage of allocator capacity used

Health Check Endpoints#

/health#

Basic health check endpoint for service availability verification.

Method: GET Content-Type: text/plain; version=0.0.4 Response: OK when service is healthy Status Codes:

  • 200 OK: Service is healthy

  • Other: Service may be experiencing issues

Example:

curl http://localhost:8080/health

/version#

Report the master version. Always available, including while the master is in standby.

Method: GET Content-Type: application/json; charset=utf-8 Response: JSON object with:

  • version (string): Store version used for RPC handshake compatibility

  • display_version (string): Human-readable release plus short git hash

Example:

curl http://localhost:8080/version

{"version":"2.0.0","display_version":"0.3.12.post1 (git: f9e8311f)"}

Real clients expose the same /version payload on their own client HTTP port when enable_client_http_server is on. See Client Metrics Endpoint.

Store REST API Endpoints#

The following endpoints are served by the Python store REST service, which wraps MooncakeDistributedStore with an aiohttp service. The HTTP handlers live in Python, while mount and unmount operations are delegated to the underlying store binding. Start the service with:

python -m mooncake.mooncake_store_service \
  --config /path/to/mooncake_config.json \
  --port 8080

If the wheel console scripts are installed, the equivalent command is:

mc_store_rest_server --config /path/to/mooncake_config.json --port 8080

/api/mount_shm#

Mount a named shared memory object as one or more Mooncake store segments. Protocols with a registration-size limit split oversized regions and return multiple segment ids. Protocols without such a Store-level limit, such as TCP and RDMA, use a single segment regardless of max_mr_size.

Method: POST Content-Type: application/json

Request Body:

{
  "name": "mooncake_segment",
  "size": 16777216,
  "offset": 0,
  "protocol": "tcp",
  "location": ""
}

Fields:

  • name (string, required): Named shared memory object name. A leading / is accepted, but path separators are not.

  • size (integer, required): Number of bytes to mount.

  • offset (integer, optional): File offset in bytes. Defaults to 0.

  • protocol (string, optional): Transfer protocol. Defaults to the service configuration protocol.

  • location (string, optional): Device or locality hint. Defaults to an empty string.

Success Response:

{
  "status": "success",
  "segment_ids": ["00000000-0000-0000-0000-000000000001"]
}

Example:

curl -X POST http://localhost:8080/api/mount_shm \
  -H "Content-Type: application/json" \
  -d '{
        "name": "mooncake_segment",
        "size": 16777216,
        "offset": 0,
        "protocol": "tcp",
        "location": ""
      }'

/api/unmount_shm#

Unmount one or more segment ids previously returned by /api/mount_shm.

Method: POST Content-Type: application/json

Request Body:

{
  "segment_ids": ["00000000-0000-0000-0000-000000000001"],
  "grace_period_seconds": 0
}

segment_ids may also be provided as a single string for one segment. grace_period_seconds is optional and defaults to 0, which keeps the existing immediate unmount behavior. When set to a positive value, the master keeps the segment readable for that grace period while preventing new allocations, then completes the unmount.

Success Response:

{
  "status": "success"
}

Example:

curl -X POST http://localhost:8080/api/unmount_shm \
  -H "Content-Type: application/json" \
  -d '{"segment_ids": ["00000000-0000-0000-0000-000000000001"],
       "grace_period_seconds": 30}'

/api/mount#

Allocate memory inside the store process and mount it as one or more Mooncake store segments. Protocols with a registration-size limit split oversized requests and return multiple segment ids. Protocols without such a Store-level limit, such as TCP and RDMA, use a single segment regardless of max_mr_size. The response includes the actual allocated size after alignment.

Method: POST Content-Type: application/json

Request Body:

{
  "size": 16777216,
  "protocol": "tcp",
  "location": ""
}

Fields:

  • size (integer, required): Number of bytes requested. Must be positive.

  • protocol (string, optional): Transfer protocol. Defaults to the service configuration protocol.

  • location (string, optional): Device or locality hint. Defaults to an empty string.

Success Response:

{
  "status": "success",
  "segment_ids": ["00000000-0000-0000-0000-000000000002"],
  "allocated_size": 16777216
}

Example:

curl -X POST http://localhost:8080/api/mount \
  -H "Content-Type: application/json" \
  -d '{"size": 16777216, "protocol": "tcp", "location": ""}'

/api/unmount#

Unmount one or more segment ids previously returned by /api/mount and free the memory allocated by the store process.

Method: POST Content-Type: application/json

Request Body:

{
  "segment_ids": ["00000000-0000-0000-0000-000000000002"],
  "grace_period_seconds": 0
}

segment_ids may also be provided as a single string for one segment. grace_period_seconds is optional and defaults to 0, which keeps the existing immediate unmount-and-free behavior. When set to a positive value, the master keeps the segment readable for that grace period while preventing new allocations, then the store releases the local allocated memory after cleanup.

Success Response:

{
  "status": "success"
}

Example:

curl -X POST http://localhost:8080/api/unmount \
  -H "Content-Type: application/json" \
  -d '{"segment_ids": ["00000000-0000-0000-0000-000000000002"],
       "grace_period_seconds": 30}'

/api/unmount_local_disk#

Deregister this store’s SSD offload tier from the master before the process goes away. Intended for a shutdown hook.

The master stops naming this store as the owner of the keys it offloaded, so a reader gets a clean miss instead of a peer that is about to disappear. Without this, a LOCAL_DISK segment leaves the master only when the client expires — one client_ttl after the store stops pinging — and reads that pick up the stale owner in that window block on the connect retries (see MC_RPC_CONNECT_TIMEOUT_MS) before missing.

The call then holds for grace_period_seconds before returning. Unlike a memory replica, which the NIC serves without help from the store process, a disk replica is read and pushed by that process, so it has to stay alive for the reads the master handed out before the deregistration. Offloading is stopped for good when this is called; the store is expected to exit afterwards.

Returns success and does nothing when SSD offload is not enabled on this store. Safe to call more than once.

Method: POST Content-Type: application/json

Request Body:

{
  "grace_period_seconds": 30
}

grace_period_seconds is optional and defaults to 0, which returns as soon as the master has dropped the segment. Must be a non-negative integer no greater than 3600 (1 hour); a malformed body or an out-of-range value gets a 400 without touching the store, so a mistake here (seconds where milliseconds were meant, say) cannot block a preStop hook for hours.

Success Response:

{
  "status": "success"
}

Example — as a Kubernetes preStop hook, with a terminationGracePeriodSeconds longer than the grace period:

curl -X POST http://localhost:8080/api/unmount_local_disk \
  -H "Content-Type: application/json" \
  -d '{"grace_period_seconds": 30}'
Contents