OSS Backend Design

Overview

The object-storage backend extends Mooncake Store’s client-owned offload path to key-based storage services, such as OSS and S3, through service-specific adapters. It uses the same LOCAL_DISK replica model as local-file and NVMe KV backends: the master records an owner and the requesting client reads through that owner. Object storage is not a separate first-class replica type here.

This document describes the OSS adapter implementation. Its signing protocol is OSS-specific. S3-compatible services use the separate S3 adapter (S3ObjectStorageAdapter, AWS Signature V4); see “S3-compatible configuration” in Object Storage Offload.

For prerequisites, configuration, and startup examples, see Object Storage Offload.

Design Goals

• Reuse FileStorage offload coordination and existing Store APIs.

• Issue concurrent object GETs and PUTs within a synchronous batch.

• Give each batch its own libcurl multi handle, without a shared batch lock.

• Avoid a full-object intermediate string on the direct GET path and avoid concatenating upload slices into a full-object buffer.

Architecture

        flowchart TB
    Master["Mooncake master<br/>LOCAL_DISK replicas and owners"]
    Requester["Requesting Mooncake client"]
    subgraph Owner["Offload owner process"]
        RealClient["RealClient"]
        FileStorage["FileStorage"]
        Backend["DistributedStorageBackend"]
        Adapter["OssObjectStorageAdapter<br/>libcurl and OSS signing"]
        RealClient --> FileStorage --> Backend --> Adapter
    end
    OSS["OSS bucket / owner prefix"]

    Requester <-->|"metadata query"| Master
    RealClient <-->|"offload heartbeat and completion"| Master
    Requester <-->|"offload RPC and Transfer Engine"| RealClient
    Adapter <-->|"HTTP"| OSS
    

Layer Responsibilities

Layer	Responsibilities
Master	Schedule offload tasks and track completed LOCAL_DISK replicas and their owners.
FileStorage	Obtain source slices, call the selected backend, report successful writes, and manage read staging buffers.
DistributedStorageBackend	Convert BatchOffload / BatchLoad into object batches and check per-object results and read lengths.
OssObjectStorageAdapter	Map keys, sign HTTP requests, run libcurl transfers, and implement GET/PUT/HEAD/LIST/DELETE.
OSS	Store object payloads under the configured bucket and prefix.

One FileStorage instance selects one backend. Selecting OSS does not also enable the local-file or NVMe KV backend, or create an SSD-to-OSS cache hierarchy.

Physical Keys and Object Layout

The adapter maps each storage key supplied by FileStorage to one OSS object:

physical_key = owner_prefix + "/" + URIEncode(storage_key)
object_body  = concatenated payload slices

The prefix separator is omitted when the prefix is empty. LIST decodes matching object keys back into logical storage keys; ScanMeta uses these keys and object sizes for metadata registration.

There is no shard/offset allocation, per-object UUID descriptor, root manifest, or adapter-level checksum envelope. A PUT targets the mapped key, not a store-if-not-exists operation. Separate owner prefixes prevent owners from overwriting each other’s objects.

Write Path

        sequenceDiagram
    participant M as Master
    participant F as Owner FileStorage
    participant B as DistributedStorageBackend
    participant A as OSS adapter
    participant O as OSS

    F->>M: OffloadObjectHeartbeat (default: every 10 s)
    M-->>F: Offload tasks for this owner
    F->>B: BatchOffload(keys and source slices)
    B->>A: PutBatch(key, iov, iovcnt)
    A->>O: Concurrent object PUTs
    O-->>A: Per-request completion
    A-->>B: Per-object results
    B-->>F: Completion callback for successful objects
    F->>M: NotifyOffloadSuccess
    Note over M: Register completed LOCAL_DISK replicas
    

The application’s memory Put is not a synchronous OSS write. Offload is scheduled separately; only successful uploads are reported as completed replicas. A batch is not a transaction: successful objects remain in OSS when another upload fails.

Read Path

        sequenceDiagram
    participant R as Requesting client
    participant M as Master
    participant F as Owner RealClient / FileStorage
    participant B as Backend / OSS adapter
    participant O as OSS

    R->>M: Query key
    M-->>R: LOCAL_DISK owner and object size
    R->>F: batch_get_offload_object
    F->>B: BatchLoad into staging buffers
    B->>O: Concurrent GETs (GetBatch)
    O-->>B: Object bytes
    B-->>F: Validate results and lengths
    F-->>R: Staging buffer descriptors
    R->>F: Fetch bytes through Transfer Engine
    R->>F: Release staging buffers after transfer
    

The RPC coroutine posts blocking work to the existing blocking pool. It can yield while a worker executes BatchLoad; the worker waits until the HTTP batch finishes. No dedicated OSS worker pool is introduced.

Get, GetRange, and GetBatch copy incoming libcurl chunks directly into the destination buffer. This removes the full-object response string and its final copy, not all copies. GetV still reads into a contiguous temporary buffer and then scatters. PutV / PutBatch consume iovec arrays through the upload callback; copies into libcurl’s upload buffer remain.

Batch Execution

Each GetBatch or PutBatch call creates a temporary CURLM and one easy handle per prepared request. It admits up to MOONCAKE_OSS_MAX_CONNECTIONS requests (default: 64), advances them together, and admits more as requests complete. Requests waiting for admission stay outside CURLM so that their transfer timeout is not consumed in libcurl’s connection queue.

Connections are reused within a batch, not across batches. Different caller threads can execute separate batches concurrently; the limit is per batch, not per adapter or process. Completion order may differ from input order, but the returned result vector preserves input order and cardinality.

Concurrency and Ownership

• Adapter APIs are synchronous: they return after batch processing and cleanup.

• Each batch owns its request contexts and libcurl handles; there is no shared CURLM or adapter-wide lock serializing complete batches.

• Download buffers, upload iovec arrays, and upload payloads must remain alive until the call returns. Upload descriptors and payloads must not be modified while in use.

• A failed read may have partially modified its destination; do not consume that buffer as valid data. A failed BatchLoad does not expose the batch as a successful Store read, even if some underlying GETs completed.

Failure Semantics

Condition	Behavior
Invalid buffer or iovec arguments detected before batch setup	Return INVALID_PARAMS for that entry; other valid entries can proceed.
GET reports a missing object	Return FILE_NOT_FOUND.
GET returns an unexpected status or length	Return a read failure; a nonempty range GET requires HTTP 206 and the requested length.
PUT returns a non-success HTTP status	Return a write failure; do not report that object as successfully offloaded.
Transport error or timeout	Return a request error; a timed-out PUT does not prove that OSS stored nothing.
Request setup or multi-handle failure	Return errors for requests that could not complete; do not roll back successful cloud writes.
One object fails in a batch	Preserve per-object results; successful uploads are not automatically deleted.

Bounded Object I/O Component

ObjectStorageBackend provides synchronous I/O for caller-selected object names, independently of the client-owned offload path above. Construct it with an initialized ObjectStorageAdapter and ObjectStorageBackendConfig:

• max_object_bytes defaults to 256 MiB and limits the declared object size.

• max_staging_bytes defaults to 1 GiB and bounds concurrent payload staging allocations within one backend instance. It does not include iovec metadata, caller buffers or the adapter’s own HTTP buffers.

Write validates and gathers all slices into a full-object host buffer, then invokes the adapter’s PutOnce. It requires enough staging budget for the entire object. The adapter must advertise SupportsSinglePutAttempt(); otherwise writes return NOT_SUPPORTED before staging or issuing PUT. Both OSS and S3 support this contract. The S3 single-attempt entry points disable retries for that call only; its legacy offload API is unchanged. The optional upload_started output becomes true immediately before the adapter call; an error with this flag set can have an ambiguous remote outcome. The backend does not retry PUT.

WriteV is the opt-in path for borrowing host buffers. The caller supplies an IsHostMemory predicate; the backend does not detect pointer address spaces. Nonempty slices classified as host are passed directly to the adapter’s PutVOnce and must remain valid and unchanged until the synchronous call returns. Other slices are gathered through to_host, reserving only their combined size from the staging budget. The predicate is evaluated once per nonempty slice. An all-host write needs neither payload staging nor a copy callback, even if the instance’s staging budget is occupied by another operation. For borrowed buffers this budget is not an upload concurrency limit; the caller remains responsible for request admission and its own buffer budget.

This path requires SupportsSinglePutVAttempt(), separately from the PutOnce capability. OSS and S3 support both single-attempt entry points. S3 uses a per-request retry policy: PutOnce and PutVOnce never retry, including empty uploads, while existing Put, PutV and batch calls retain their retry behavior. Concurrent calls do not change shared retry settings. All validation and staging complete before the single PutVOnce invocation; upload_started and failure/exception cleanup follow the same rules as Write. This removes the backend’s gathering copy for borrowed slices, not copies inside the adapter or network stack. Existing Write calls retain their full-object staging behavior.

For input known by the caller to be entirely host-accessible:

backend.WriteV(object_key, object_size, slices,
               [](const Slice&) { return true; }, {}, &upload_started);

Read validates the requested range against the caller-supplied object size, then reads consecutive chunks independently of destination slice boundaries and scatters each chunk into the slices. GETs remain serial. Each staging allocation is at most 4 MiB, or the configured staging limit if smaller. Reads fail on a short response. OSS and the REST adapter used by S3 also validate Content-Range against the requested offset and length, for both individual and batch reads. Adapters that do not implement the optional generic GetRange method return NOT_SUPPORTED.

The six-argument Read overload accepts an IsHostMemory predicate. With exactly one nonempty slice classified as host, GetRange writes directly to that destination, skipping from_host and payload staging. Empty slices are ignored. GETs retain the same serial chunking, including the configured limit when it is below 4 MiB. This path can run while other operations occupy the staging budget; the caller remains responsible for request admission. Multiple nonempty slices or a non-host destination use the staged path and require from_host. The original five-argument API always uses staging.

backend.Read(object_key, object_size, {{host_buffer, length}}, offset, {},
             [](const Slice&) { return true; });

A direct-read destination must remain valid, writable and exclusively owned until return. On failure, it may contain partial data or an HTTP error body, including when response range validation fails. Only use the result after a successful return. Direct reads remove the backend’s staging copy, not copies inside the adapter or network stack.

Copy callbacks allow callers to stage device buffers. They must finish copying before returning, and report failure with false. Invalid arguments return INVALID_PARAMS; staging exhaustion returns NO_AVAILABLE_HANDLE; copy failure returns TRANSFER_FAIL. Reservations are released on return or stack unwinding. A failed read can leave destination buffers partially written.

This component does not allocate immutable names, manage read leases, publish replicas, or decide when deletion is safe. Those responsibilities remain with the caller, which can use the adapter’s existing Delete operation. Sharing an adapter across concurrent operations requires a thread-safe adapter.

This component does not add a first-class replica type or enable owner-independent Store reads. Those guarantees require Master/Client integration with durable attempt identity, commit, recovery and lease-aware reclamation. See RFC #4296 for context.

Current Limitations

• Readers depend on a reachable owner and valid Master metadata, not just the existence of an OSS object. This is different from a first-class DFS replica.

• Master capacity accounting uses each owner’s configured capacity and live replica sizes. It neither queries OSS free space nor enforces a bucket quota.

• The backend does not implement capacity eviction or automatic cloud-object garbage collection. Removing Master metadata does not issue an OSS DELETE; metadata usage can therefore differ from actual bucket usage.

• Multipart uploads, automatic parallel range splitting, and automatic credential refresh are not implemented.

The object upload and cleanup recovery records describe the separate catalog review unit for first-class object replicas.