Object Storage Offload

Object Storage Offload#

Overview#

Object storage services, such as OSS and S3, provide key-based storage. Mooncake Store integrates object storage through ObjectStorageAdapter in the existing FileStorage offload path. As with local SSD and NVMe KV backends, the master records LOCAL_DISK replicas owned by a real client; readers still access the payload through that owner.

Two adapters are available:

Adapter	`MOONCAKE_DISTRIBUTED_FS_TYPE`	Services
OSS	`oss`	Alibaba Cloud OSS
S3-compatible	`s3`	AWS S3, SeaweedFS, MinIO, Ceph RGW and other services that accept AWS Signature V4

The two adapters are independent and are configured with separate variables, described in the “OSS configuration” and “S3-compatible configuration” sections below.

For implementation details, see OSS Backend Design.

Prerequisites#

An existing bucket and an endpoint reachable from each offload owner. No filesystem mount of the object store is required.
Credentials with permission to PUT, GET, HEAD, LIST, and DELETE within the chosen namespace. Temporary credentials (an OSS STS token or an S3 session token) are supported.
A dedicated object-key prefix for each offload owner.
An existing absolute, writable, non-symlink directory for MOONCAKE_OFFLOAD_FILE_STORAGE_PATH, required by common FileStorage initialization. This does not enable a local SSD cache for object storage.
libcurl and OpenSSL development libraries and headers.

Build Support#

The build enables both adapters when libcurl and OpenSSL are available. No service SDK or additional build flag is required. CMake reports each adapter on its own line (Alibaba Cloud OSS adapter: Enabled, S3-compatible adapter: Enabled). Follow the build guide to build and install Mooncake.

Batch I/O uses curl_multi_wait; libcurl 7.66.0 is not required. Upload-buffer tuning is optional: with headers older than 7.62.0, the library default is used.

Topology#

        flowchart TD
    App["Application or requesting Mooncake client"]
    Master["Mooncake master"]
    Owner["Offload-owning real client"]
    FileStorage["FileStorage"]
    Backend["DistributedStorageBackend"]
    Adapter["ObjectStorageAdapter (OSS or S3)"]
    Store["Bucket and owner prefix"]

    App <-->|"metadata query"| Master
    App <-->|"offload RPC and Transfer Engine"| Owner
    Owner <-->|"offload heartbeat and LOCAL_DISK updates"| Master
    Owner --> FileStorage --> Backend --> Adapter
    Adapter <-->|"HTTP requests"| Store

Only the offload owner needs object-storage credentials. The master and requesting clients do not directly read or write objects in the bucket.

Configuration#

Set the backend variables and the variables of one adapter in each offload owner’s environment. Each FileStorage instance selects one backend: object storage does not run alongside the local-file or NVMe KV backend within that instance.

Backend and namespace#

These settings are the same for both adapters.

export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=distributed_storage_backend
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/data/file_storage
export MOONCAKE_DISTRIBUTED_FS_TYPE=oss   # or s3
export MOONCAKE_DISTRIBUTED_ROOT_DIR=/mooncake/my-cluster/owner-1

Environment variable	Setting	Description
`MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR`	`distributed_storage_backend`	Select the backend that hosts the object-storage adapters.
`MOONCAKE_OFFLOAD_FILE_STORAGE_PATH`	`/data/file_storage` by default	Existing local directory required by common initialization; object payloads go to the bucket.
`MOONCAKE_DISTRIBUTED_FS_TYPE`	`oss` or `s3`	Select object-storage mode, and which adapter, rather than a filesystem adapter.
`MOONCAKE_DISTRIBUTED_ROOT_DIR`	An owner-specific prefix	Use an absolute-style path; the adapter strips leading and trailing slashes. This is not a mount point.
`MOONCAKE_DISTRIBUTED_HEALTH_CHECK`	`false`	Write and read back a probe object during initialization, then best-effort delete it.

Object-storage offload does not require Master DFS configuration and does not require disabling a separately configured DFS tier. In the offload owner’s environment, MOONCAKE_DFS_FS_ADAPTER and MOONCAKE_DFS_ROOT_DIR override the corresponding MOONCAKE_DISTRIBUTED_* values because they share a configuration parser. Leave these overrides unset when using the example above.

Configuration is read at initialization; changing environment variables does not reconfigure an active adapter or refresh its credentials.

OSS configuration#

export MOONCAKE_DISTRIBUTED_FS_TYPE=oss
export MOONCAKE_OSS_ENDPOINT=https://oss-cn-hangzhou.aliyuncs.com
export MOONCAKE_OSS_BUCKET=my-mooncake-bucket
export MOONCAKE_OSS_REGION=cn-hangzhou
# Supply MOONCAKE_OSS_ACCESS_KEY_ID and MOONCAKE_OSS_ACCESS_KEY_SECRET
# through your credential-management mechanism, not checked-in scripts.

Replace the example endpoint, bucket, region, and owner prefix for your deployment.

Environment variable	Default	Description
`MOONCAKE_OSS_ENDPOINT`	Required	Endpoint including `http://` or `https://`. Alias: `OSS_ENDPOINT`.
`MOONCAKE_OSS_BUCKET`	Required	Existing bucket. Alias: `OSS_BUCKET`.
`MOONCAKE_OSS_REGION`	Required	OSS signing region. Alias: `OSS_REGION`.
`MOONCAKE_OSS_ACCESS_KEY_ID`	Required unless anonymous	Access key ID. Alias: `OSS_ACCESS_KEY_ID`.
`MOONCAKE_OSS_ACCESS_KEY_SECRET`	Required unless anonymous	Access key secret. Alias: `OSS_ACCESS_KEY_SECRET`.
`MOONCAKE_OSS_SECURITY_TOKEN`	Empty	Optional STS token. Alias: `OSS_SESSION_TOKEN`.
`MOONCAKE_OSS_PATH_STYLE`	`false`	Use `endpoint/bucket/key` instead of virtual-hosted bucket addressing.
`MOONCAKE_OSS_ANONYMOUS`	`false`	Disable signing; only for test endpoints or suitably configured public access.

Primary names take precedence over aliases, including explicitly empty values.

S3-compatible configuration#

The S3 adapter signs requests with AWS Signature V4.

export MOONCAKE_DISTRIBUTED_FS_TYPE=s3
export MOONCAKE_S3_ENDPOINT=http://seaweedfs-s3:8333
export MOONCAKE_S3_BUCKET=my-mooncake-bucket
export MOONCAKE_S3_PATH_STYLE=true
# Supply MOONCAKE_S3_ACCESS_KEY_ID and MOONCAKE_S3_SECRET_ACCESS_KEY
# through your credential-management mechanism, not checked-in scripts.

Environment variable	Default	Description
`MOONCAKE_S3_ENDPOINT`	Required	Endpoint starting with `http://` or `https://`, without a path. Alias: `AWS_ENDPOINT_URL`.
`MOONCAKE_S3_BUCKET`	Required	Existing bucket.
`MOONCAKE_S3_REGION`	`us-east-1`	Signing region. Falls back to `AWS_REGION`, then `AWS_DEFAULT_REGION`; a warning is logged when none is set.
`MOONCAKE_S3_ACCESS_KEY_ID`	Required unless anonymous	Access key ID. Alias: `AWS_ACCESS_KEY_ID`.
`MOONCAKE_S3_SECRET_ACCESS_KEY`	Required unless anonymous	Secret access key. Alias: `AWS_SECRET_ACCESS_KEY`.
`MOONCAKE_S3_SESSION_TOKEN`	Empty	Optional session token. Alias: `AWS_SESSION_TOKEN`.
`MOONCAKE_S3_PATH_STYLE`	`false`	Use `endpoint/bucket/key`. Most self-hosted services (SeaweedFS, MinIO) need `true`; it is forced on for IP-address and `localhost` endpoints.
`MOONCAKE_S3_ANONYMOUS`	`false`	Disable signing; only for test endpoints or suitably configured public access.

Credentials are read as one set: if MOONCAKE_S3_ACCESS_KEY_ID or MOONCAKE_S3_SECRET_ACCESS_KEY is set, the key ID, secret and session token all come from MOONCAKE_S3_*; otherwise all three come from AWS_*. An AWS_SESSION_TOKEN left in the environment is therefore never sent with MOONCAKE_S3_* keys. Other primary names take precedence over their AWS_* aliases. The adapter does not create buckets. The payload is sent as UNSIGNED-PAYLOAD; use https:// endpoints outside trusted networks.

The S3 adapter retries transient failures (connection set-up errors, resets, HTTP 429 and 5xx) up to three attempts with backoff. A request that was sent and then timed out is not retried, and client errors such as 403 and 404 are not retried.

Concurrency and buffers#

Each adapter has its own tuning variables with the same meaning and bounds.

OSS variable	S3 variable	Default	Description
`MOONCAKE_OSS_MAX_CONNECTIONS`	`MOONCAKE_S3_MAX_CONNECTIONS`	`64`	Maximum admitted requests and total/per-host connections per batch; minimum `1`. Not a process-wide limit.
`MOONCAKE_OSS_RECEIVE_BUFFER_SIZE`	`MOONCAKE_S3_RECEIVE_BUFFER_SIZE`	`1048576` (1 MiB)	libcurl receive-buffer suggestion for batch requests, clamped to 16 KiB–10 MiB. Single-request GETs keep the library default.
`MOONCAKE_OSS_UPLOAD_BUFFER_SIZE`	`MOONCAKE_S3_UPLOAD_BUFFER_SIZE`	`1048576` (1 MiB)	Upload-buffer suggestion for `PutV` / `PutBatch`, clamped to 16 KiB–2 MiB. Applied only with libcurl headers 7.62.0 or newer; otherwise the library default is used.

Numeric tuning values are decimal integers. Invalid or out-of-range integers use the default; the bounds above then apply. Buffer sizes are libcurl suggestions, not TCP socket-buffer sizes or guaranteed throughput settings.

The common offload heartbeat defaults to 10 seconds and is configured through MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS. Other common client settings are described in SSD Offload.

Start Mooncake#

Start the master with offload enabled:

mooncake_master --rpc_port=50051 --enable_offload=true

After applying the backend settings above, create the required local directory and start a real client. This example uses a local master and TCP transfers:

mkdir -p /data/file_storage
export MOONCAKE_MASTER=127.0.0.1:50051
export MOONCAKE_LOCAL_HOSTNAME=127.0.0.1
export MOONCAKE_PROTOCOL=tcp
export MOONCAKE_TE_META_DATA_SERVER=P2PHANDSHAKE
export MOONCAKE_OFFLOAD_ENABLED=true

python -m mooncake.mooncake_store_service

Use routable addresses for a multi-node deployment. This launcher example assumes MOONCAKE_CONFIG_PATH is unset; a service configuration file otherwise takes precedence.

Embedded real-client mode uses the same backend and adapter variables. Pass enable_ssd_offload=True and ssd_offload_path to MooncakeDistributedStore.setup() alongside the normal connection and memory arguments. The SSD Offload guide describes embedded and standalone real-client deployment modes.

Troubleshooting#

The adapter cannot initialize#

Check build dependencies, required endpoint/region/credential settings, and the local directory. Check for stale MOONCAKE_DFS_* overrides. The optional health check exercises access to the bucket; it does not test the complete Store read path.

Requests fail with authentication or permission errors#

Verify the endpoint, signing region, bucket permissions, and the lifetime of any temporary credentials. The adapters do not refresh credentials automatically. For S3-compatible services, also check MOONCAKE_S3_PATH_STYLE: most self-hosted services need path-style addressing.

An object is in the bucket but cannot be read through Store#

A bucket object alone is not a readable Store replica. The master must have the key’s metadata and a reachable owner. Keep prefixes owner-specific; a shared bucket does not make owners interchangeable.

SSD capacity metrics do not match bucket usage#

MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES supplies a configured capacity value (default: 2 TiB), not a queried bucket capacity. Master usage tracks registered LOCAL_DISK replicas. There is no backend quota enforcement or automatic object GC here, so these metrics are not physical bucket usage or a cloud-cost limit.