Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration Reference

AgentENV reads configuration from a TOML file. The default path is config/default.toml. Override it with:

export AENV_CONFIG_PATH=/path/to/config.toml
# or
cargo run --bin server -- --config /path/to/config.toml

Global Settings

KeyTypeDefaultDescription
home_pathstring"/var/lib/aenv"Base directory for local AgentENV state. Overridden by AENV_HOME_PATH
runtime_pathstring"/run/aenv"Base directory for transient namespace and daemon-socket state. Overridden by AENV_RUNTIME_PATH
deps_pathstring"$AENV_HOME/deps"Root directory for auto-downloaded runtime assets. Overridden by AENV_DEPS_PATH

$AENV_HOME is a literal placeholder in state-path values, not a shell environment variable. AgentENV replaces it with the resolved home_path after applying AENV_HOME_PATH; ublk.daemon_socket_path additionally supports $AENV_RUNTIME, which resolves to runtime_path. Relative paths without these placeholders are resolved against the directory containing the configuration file.

Packaged runtime dependency versions and download URLs live in config/deps_manifest.toml. config.toml should contain runtime behavior and explicit local path overrides, not the default dependency catalog.

[firecracker]

Firecracker VM binary and boot configuration.

KeyTypeDefaultDescription
versionstringmanifest valueOptional Firecracker release override for auto-download
urlstringmanifest valueOptional download URL template override with {version} and {arch} placeholders
binary_pathstringderived from manifest/config versionExplicit path to a local firecracker binary. Setup skips the Firecracker download and requires this to be a readable, non-empty, executable regular file
boot_argsstring"console=ttyS0 reboot=k panic=1 pci=off init=/init …"Kernel command line arguments. The shipped default also includes DAMON memory-reclaim parameters; see config/default.toml for the full value.
allowed_extra_boot_args_prefixesarray of strings[]Allowed prefixes for extraBootArgs on cold-start sandboxes. If empty, no request-provided extra boot args are appended
socket_timeout_secsinteger3Max seconds to wait for the Firecracker API socket
socket_poll_msinteger1Poll interval (ms) for checking socket availability
work_dirstring"$AENV_HOME/firecracker-work"Parent directory for per-sandbox Firecracker work directories. These dirs contain runtime sockets, symlinks, local logs, and writable OverlayBD upper layer data such as overlaybd/upper.data and overlaybd/upper.index
serial_dirstring"$AENV_HOME/logs/serial"Directory for persistent Firecracker serial output (per-sandbox subdirectories)
log_levelstringunset (disabled)Optional Firecracker log level (Error, Warning, Info, Debug, Trace, case-insensitive). When set to a non-empty value, Firecracker’s own logging is enabled and written to a firecracker.log file in each sandbox’s log directory (alongside the serial output). Empty/unset disables it

[kernel]

Linux kernel image for microVMs.

KeyTypeDefaultDescription
versionstringmanifest valueOptional kernel version override for auto-download
urlstringmanifest valueOptional download URL template override with {version} placeholder
image_pathstringderived from manifest/config versionExplicit path to a local vmlinux.bin. Setup skips the kernel download and requires this to be a readable, non-empty regular file

[tools]

Tools drive image used to boot the AgentENV control plane inside each microVM.

KeyTypeDefaultDescription
versionstringmanifest valueImmutable SemVer release of the complete tools drive; custom distributions should use a unique prerelease such as 0.1.0-custom.1
urlstringmanifest valueOptional OCI image URL template override with a {version} placeholder; requires an explicit version when set
drive_pathstringunsetLocal tools ext4 source imported into the versioned dependency directory; requires an explicit version
control_plane_portinteger49983Port used by envd inside the guest

Snapshots and paused sandboxes keep using the tools drive version they were created with. Launch does not download missing releases: operators must install the recorded version under <deps_path>/tools/<version>/tools.ext4 before restore. Setup retains previously installed versions until they are removed manually.

Template Rootfs Images

User-visible rootfs images are selected at the template API layer. POST /v2/templates/{templateID}/builds/{buildID} accepts an optional fromImage field:

  • omitted: use [image.resolver].default_image
  • full OCI reference: use the supplied image
  • short name: normalize standard Docker Hub forms such as ubuntu:24.04 and node:20

[image.resolver]

KeyTypeDefaultDescription
default_imagestringubuntu:24.04Image used when template builds omit fromImage
search_registriesarray of strings["docker.io", "ghcr.io"]Registries tried when resolving short image references
allowed_registriesarray of stringsunset (no restriction)Whitelist of registry hosts (e.g. docker.io, registry.example.com:5000). Omitting the key imposes no restriction; an explicit empty list [] denies every registry. When set to a non-empty list, only references whose registry host is in the list resolve; any other host is rejected as a client (4xx, ImageReferenceError) error. See How the three registry settings interact below.
try_referrers_overlaybd_prefixesarray of strings[]Image reference prefixes for which AgentENV tries OCI Referrers API via regctl for an overlaybd-native artifact before converting a standard OCI image locally. Prefixes are matched with simple starts_with; include the trailing slash yourself, for example registry.example.com/ or registry.example.com/team/. Requires regctl on PATH; lookup failures fall back to the source image.

How the three registry settings interact

Image resolution runs in two phases, and the three keys act at different points:

  1. search_registries — completion. Only used for short / unqualified references (e.g. ubuntu). Each entry is prefixed to the name to build a list of fully-qualified candidates (docker.io/library/ubuntu:latest, ghcr.io/ubuntu:latest, …). Fully-qualified references skip this step.
  2. allowed_registries — gating. Applied right after candidates are built, to both fully-qualified references and the candidates expanded from search_registries. Candidates whose registry host is not whitelisted are dropped; if none remain, the reference is rejected with a 4xx error. In effect the resolvable set of short-name hosts is the intersection of search_registries and allowed_registries — e.g. searching docker.io and ghcr.io while only allowing ghcr.io resolves short names to ghcr.io only.
  3. try_referrers_overlaybd_prefixes — per-candidate optimization. Runs later, while resolving an already-permitted candidate: after its manifest is fetched, AgentENV may query the OCI Referrers API on the same registry/repository for an overlaybd-native artifact. Because referrer lookups never leave the source image’s own host, they are implicitly covered by allowed_registries — no separate whitelist entry is needed for referrers.

[image.cache]

Node-local cache root for resolved and converted user images.

KeyTypeDefaultDescription
root_dirstring"$AENV_HOME/image-cache"Root directory for AgentENV image-cache artifacts
capacity_gbinteger100Budget for capacity-driven eviction of local commit bytes. Enforced only when [image.cache.gc].enabled is true: the background GC evicts least-recently-used source configs once usage crosses the high watermark, down to the low watermark. Unset = no capacity cap.

[image.cache.gc]

Background hard-commit garbage collection for the image cache. When enabled, each pass reconciles metadata from the on-disk source configs and then deletes hard-commit objects that are no longer rooted by source configs, held by image-cache leases, or referenced by the in-process running set. Committed snapshots are durable SnapshotRepository state and do not pin ImageCache commits. With capacity_gb set, GC first evicts least-recently-used source configs over the high watermark so hard-commit GC can reclaim what they unrooted.

KeyTypeDefaultDescription
enabledbooltrueEnable the background image-cache GC task
interval_secsinteger1800Seconds between GC passes (a value <= 0 falls back to the default)
min_age_secsinteger600Minimum time since last use before a source config is eligible for capacity eviction (the LRU floor)
high_watermark_ratiofloat0.95Begin capacity eviction once local commit bytes exceed capacity_gb × this ratio. Clamped to (0, 1]
low_watermark_ratiofloat0.70Evict down to capacity_gb × this ratio once the high watermark trips. Clamped to (0, high_watermark_ratio]

Capacity-driven eviction runs only when [image.cache].capacity_gb is set; otherwise the GC still reclaims unreachable commits but performs no watermark eviction.

[image.cache.remote_blocks]

Overlaybd registryfs_v2 remote block cache settings. The directory is always <image.cache.root_dir>/remote-blocks.

KeyTypeDefaultDescription
max_size_gbinteger10Maximum size of the overlaybd remote block cache in GiB. This value is written to generated overlaybd cacheConfig.cacheSizeGB

Resolved image data is cached under:

<image.cache.root_dir>/
  commits/
    <sha256-commit-digest>/
      overlaybd.commit
                  # full overlaybd commit store shared by OCI conversion and download
  indexes/        # OCI layer + conversion context -> overlaybd commit descriptor
  remote-blocks/  # overlaybd-native remote block cache
  configs/         # resolved image configs
    <slug>-<hash>-image.json

[sandbox_proxy]

Optional host-based data-plane routing for sandbox services.

KeyTypeDefaultDescription
domainsarray of strings[]DNS domains accepted by the server for host-based proxy URLs shaped like {port}-{sandboxID}.{domain}. The first configured domain is returned in sandbox create/detail responses as domain.

When domains is empty, the server still supports /proxy and routing-header proxy requests, but does not classify requests by Host. Domains are normalized to lowercase, deduplicated, and must be valid DNS names. The configured order is preserved because domains[0] is the advertised sandbox domain.

Environment variable override:

  • AENV_SANDBOX_PROXY_DOMAINS

[network.egress]

Node-level sandbox egress guardrails. These rules are installed before per-sandbox allowOut / denyOut rules, so sandbox API requests cannot override them.

KeyTypeDefaultDescription
always_denied_cidrsarray of IPv4 CIDR strings["10.0.0.0/8", "100.64.0.0/10", "127.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "192.168.0.0/16"]Destination CIDRs that are always rejected from sandboxes before user egress policy is evaluated. Deployments can remove selected RFC1918 ranges when sandbox egress to those destinations is required.

[network.internal]

AgentENV-internal sandbox address plan. Change these only when the defaults overlap with host or deployment network ranges.

KeyTypeDefaultDescription
host_interaction_cidrIPv4 CIDR10.11.0.0/16Per-slot host interaction address pool. Must contain at least 32768 addresses.
veth_cidrIPv4 CIDR10.12.0.0/16Per-slot namespace veth pair pool. Must contain at least 65536 addresses.

The two configured CIDRs must not overlap each other or AgentENV’s fixed VM tap link 169.254.0.20/30. These networks are also treated as reserved sandbox egress destinations regardless of always_denied_cidrs.

[machine]

Default VM resources for sandboxes.

KeyTypeDefaultDescription
mem_size_mibinteger1024Guest RAM in MiB
vcpu_countinteger2Number of virtual CPUs

[envd]

In-guest envd daemon settings.

KeyTypeDefaultDescription
versionstring"0.5.15"Expected envd version baked into the tools drive image
init_timeout_secsinteger60Max seconds to wait for envd to become ready after VM start
poll_msinteger3Poll interval (ms) for envd health check retries

[orchestrator]

Sandbox lifecycle management.

KeyTypeDefaultDescription
auto_evict_interval_msinteger1000Poll interval (ms) for background timeout eviction
default_sandbox_timeout_secsinteger15Default keep-alive timeout for sandboxes
auto_resume_min_sandbox_timeout_secsinteger300When a data-plane request targets a non-running sandbox, automatically resume it (if auto-resume is enabled) and refresh its timeout for no-less than this duration
persisted_sandbox_store_pathstring"$AENV_HOME/persisted-sandboxes"Directory for persisted sandbox state

[pool]

Shared process-wide warm-pool defaults used by network slots, block devices, and pre-spawned Firecracker processes. Pools prewarm to the low watermark, then grow the refill target geometrically toward the high watermark when real acquisitions drain the pool.

KeyTypeDefaultDescription
low_watermarkinteger2Initial lower bound for all enabled warm-resource pools
high_watermarkinteger64Maximum idle target for all enabled warm-resource pools

Component sections:

SectionKeyTypeDefaultDescription
[pool.network]maintenance_enabledbooleantrueEnable the background network-slot maintenance worker
[pool.block]enabledbooleantrueEnable the ublk overlaybd warm-device pool
[pool.block]startup_prewarmbooleantruePrewarm block devices after the first reusable image shape is known
[pool.firecracker]enabledbooleantrueEnable pre-spawned Firecracker processes for snapshot resume
[pool.firecracker]maintenance_enabledbooleantrueEnable the background Firecracker process maintenance worker
[pool.firecracker]startup_prewarmbooleantrueSpawn warm Firecracker entries up to the low watermark during server startup
[pool.firecracker]fill_concurrencyinteger4Maximum number of warm Firecracker processes created concurrently by one maintenance refill batch

Validation rules:

  • low_watermark <= high_watermark
  • [pool.firecracker].fill_concurrency > 0

[node_identity]

Stable identity fields for this node. These values appear in node API responses and scheduler heartbeats.

KeyTypeDefaultDescription
node_idstringhostname-derivedStable node identifier returned by the admin/node APIs
cluster_idstring (UUID)nil UUIDLogical cluster identifier included in node snapshots
service_instance_idstringgenerated UUIDUnique process/service instance identifier for the current node runtime

[observability]

Node-level observability and host metrics collection.

KeyTypeDefaultDescription
enabledbooleantrueEnable the node/admin observability service. When disabled, /nodes returns an empty list and /nodes/{nodeID} returns 404

When observability is enabled, host CPU/memory/disk metrics are collected at request time. CPU percent is computed from two samples; the first node metrics request waits about 100ms to return a measured value.

[observability.scheduler_report]

Optional scheduler heartbeat reporting for multi-node control plane integration.

KeyTypeDefaultDescription
enabledbooleanfalseEnable periodic scheduler heartbeat reporting. Requires [cluster].scheduler_endpoint
interval_secsinteger5Heartbeat report interval in seconds

Environment variable overrides:

  • AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED
  • AENV_OBSERVABILITY_SCHEDULER_ENDPOINT
  • AENV_OBSERVABILITY_REPORT_INTERVAL_SECS

AENV_OBSERVABILITY_SCHEDULER_ENDPOINT overrides [cluster].scheduler_endpoint for the reporter process only.

[cluster]

Shared cluster-level service endpoints.

KeyTypeDefaultDescription
scheduler_endpointstringunsetgRPC endpoint for the scheduler, for example "http://127.0.0.1:9090". Used by scheduler heartbeat reporting and P2P peer discovery.

[p2p]

Project-wide artifact transport configuration. The transport is disabled by default. When enabled, it is used by the overlaybd P2P HTTP facade and by snapshot publication/runtime resolution as an optional artifact visibility and acceleration path.

KeyTypeDefaultDescription
enabledbooleanfalseEnable the P2P artifact transport. When false, AgentENV uses DisabledP2pTransport, so lookups miss and publishes are no-ops.
transportstring"iroh"Transport backend. Supported values are "disabled" and "iroh". Ignored while enabled = false.
store_dirstring"$AENV_HOME/p2p/store"Local store used by the transport backend. Relative explicit paths are resolved against the config file directory.
listen_addrstring"0.0.0.0:0"Optional local listen address for the embedded transport endpoint. Port 0 lets the OS choose a free port.
lookup_timeout_msinteger5000Timeout for one artifact catalog lookup against a peer.
fetch_timeout_msinteger30000Timeout for fetching one artifact from a peer.
peer_discovery_refresh_interval_secsinteger5Interval for refreshing peer endpoints from scheduler. Values below one second are clamped to one second.

[custom_extension]

Custom extension service configuration. When url is unset, the integration is fully disabled. See Custom Extension.

KeyTypeDefaultDescription
urlstringunsetHTTP base URL of the custom extension service. When set, AgentENV invokes sandbox lifecycle hooks under POST {url}/sandbox-hook/*.
timeout_msinteger5000Timeout for each custom extension HTTP call, in milliseconds.

[snapshot]

Snapshot storage/build configuration.

KeyTypeDefaultDescription
local_cache_pathstring"$AENV_HOME/snapshot-local-cache"Manager-owned node-local snapshot artifact/cache root. Relative explicit paths are resolved against the config file directory.
repository_backendstring"posix_fs"Snapshot repository backend. Supported values: "posix_fs" and "oss"
p2p_enabledbooleantrueWhen enabled, the snapshot manager publishes committed snapshots to the P2P transport and attempts to resolve from it before falling back to the repository backend.

Environment variable overrides:

  • AENV_SNAPSHOT_LOCAL_CACHE_PATH

[backend.posix_fs]

POSIX filesystem-backed snapshot repository configuration. This section is used when snapshot.repository_backend = "posix_fs".

KeyTypeDefaultDescription
snapshot_storestring"$AENV_HOME/snapshot-store"Root directory for durable committed snapshot repository state. Relative explicit paths are resolved against the config file directory.

Environment variable overrides:

  • AENV_SNAPSHOT_STORE

[backend.oss]

OSS-backed snapshot repository configuration. This section is required when snapshot.repository_backend = "oss".

KeyTypeDefaultDescription
endpointstringnoneOSS endpoint URL, for example "https://oss-cn-hangzhou.aliyuncs.com"
bucketstringnoneOSS bucket name used for committed snapshot state
prefixstringemptyOptional object key prefix under the bucket
credential_processstringunsetExternal command used to fetch OSS credentials. Use a plain executable-plus-args form without shell expansion, pipes, or command substitution so it behaves consistently across AgentENV and overlaybd credential consumers
access_key_idstringunsetStatic OSS access key ID. Required when credential_process is not set
access_key_secretstringunsetStatic OSS access key secret. Required when credential_process is not set
security_tokenstringunsetOptional session token paired with static access key credentials
regionstringnoneRegion passed to the S3-compatible object-store client; required for current OSS backend
cache_max_size_gbinteger10Maximum size of the node-local OSS artifact cache in GiB

Notes:

  • credential_process and static access key settings are mutually exclusive in practice; when credential_process is set, the backend ignores static credential fields.
  • credential_process should be written as a portable argv-style command line. Avoid $VAR, backticks, $(...), pipes, and shell builtins.
  • Although the config section is still named oss, the runtime path is implemented via a shared S3-compatible client, so region must be configured.

Other path override:

  • AENV_DEPS_PATH

Setup sysctl tuning is host-level setup. It is skipped before reading /proc/sys when the server detects that it is running inside a container. Set AENV_FORCE_SYSCTL_TUNING=1 only for a privileged container with writable host sysctls; otherwise configure these kernel parameters on the host.

[protoc]

Protobuf compiler metadata for code generation lives in config/deps_manifest.toml, not config.toml.

KeyTypeDefaultDescription
versionstring"33.4"protoc release version
urlstringGitHub release URLDownload URL template with {version} and {platform} placeholders

[ublk]

Optional userspace block device configuration. When enabled, rootfs is served through a ublk device instead of a plain file, managed by uvm-ublk-daemon.

KeyTypeDefaultDescription
enabledbooleantrueEnable ublk-backed rootfs
daemon_binary_pathstring"$AENV_HOME/ublk/uvm-ublk-daemon"Path to the uvm-ublk-daemon binary
daemon_socket_pathstring"$AENV_RUNTIME/ublk-daemon.sock"Unix socket path used by the daemon
daemon_log_pathstring"$AENV_HOME/logs/ublk-daemon.log"File path for daemon logs; deployments are responsible for rotation and retention
daemon_metrics_listen_addrstring"0.0.0.0:9103"HTTP listen address for daemon Prometheus metrics; empty string disables it
device_typestring"overlaybd""cow" (copy-on-write) or "overlaybd" (layered image)

Environment variable override:

  • AENV_UBLK_DAEMON_BINARY_PATH
  • AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR

[ublk.overlaybd]

Overlaybd-specific configuration used when ublk.device_type = "overlaybd".

KeyTypeDefaultDescription
global_config_pathstring"$AENV_HOME/overlaybd/overlaybd-global.json"Path to overlaybd global config JSON (see note below). Relative explicit paths are resolved against the config file directory.
read_onlybooleanfalseWhen set to true, materializes the rootfs without a writable upper
runtime_upper_modestring"hybridLogStructured"Runtime upper format for newly materialized writable rootfs OverlayBD images. Supported values are "logStructured", "hybridLogStructured", and "sparse". Existing source uppers keep their own mode
allow_shrinkbooleanfalseAllows an explicit cold-start diskSizeMB smaller than the source rootfs. Explicit sizes use MiB and must be divisible by 1024. Growth is always allowed; snapshot resume never resizes.
resize_timeout_secsinteger120Timeout in seconds for the cold-start OverlayBD resize tool. Must be greater than zero.
download_enablebooleanfalseEnables overlaybd layer-level background download for remote layers
p2p_lookup_timeout_msinteger300Timeout for one foreground Overlaybd descriptor lookup through the localhost P2P HTTP facade. Timeout is treated as a cacheable miss.
p2p_fetch_range_timeout_msinteger2000Timeout for one foreground Overlaybd range fetch through the localhost P2P HTTP facade before falling back to the origin registry.

global_config_path and auto-generated config

The file at the configured default path $AENV_HOME/overlaybd/overlaybd-global.json is auto-generated by the server at startup. The generated JSON incorporates several TOML settings — [image.cache].root_dir, [image.cache.remote_blocks].max_size_gb, download_enable, [backend.oss] credentials, and Docker registry credentials detected from ~/.docker/config.json — into a single overlaybd runtime config file.

The server regenerates the file at global_config_path on every startup, so these TOML settings always take effect automatically — any manual edits to the generated file are overwritten on the next startup. To keep customizations, make them through the TOML settings, not by editing the generated JSON.

[memory_snapshot]

Memory snapshot overlaybd configuration. The server auto-generates the file at the default path on every startup.

KeyTypeDefaultDescription
overlaybd_global_config_pathstring"$AENV_HOME/overlaybd/mem-overlaybd-global.json"Path to the overlaybd global config used for the memory-snapshot ublk backend. Regenerated at startup (manual edits are overwritten); change only to relocate the generated file.
direct_overlaybdbooltrueCreate memory overlaybd layers directly from Firecracker dirty memory ranges via process_vm_readv, skipping the intermediate mem.bin file. Set AGENTENV_MEMORY_SNAPSHOT_DIRECT_OVERLAYBD=false to force the legacy mem.bin conversion path.

[memory_snapshot.background_download]

Background download settings dedicated to remote memory-snapshot OverlayBD layers. They do not change the general rootfs or attached-drive defaults. All fields are serialized into the generated memory OverlayBD global config. Each remote layer is downloaded block by block: a sequential sparse-file scan collects the pending blocks, then at most concurrency block tasks fetch non-overlapping ranges in parallel on a dedicated multi-thread runtime. Layer files are still processed one at a time. Downloads of a sandbox-bound device start only after envd is ready (plus delay), with a 20s fallback if the ready signal is lost; while foreground remote reads are in flight, background block reads yield to a small guaranteed floor instead of competing at full speed. The generated memory config leaves throttling off (maxMBps = 0); image configs that carry a positive maxMBps keep their historical shared rate limit across the block tasks. Completed layers are switched to the local file only after a full-file digest check; a failed or canceled block never switches.

KeyTypeDefaultDescription
enablebooleantrueEnables background download for remote memory-snapshot layers.
delayinteger0Delay in seconds after envd is ready before background download begins (downloads never start before envd readiness; a 20s fallback applies if the ready signal is lost).
delay_extrainteger1Exclusive upper bound for random extra delay. The default 1 ensures delay = 0 adds no jitter.
try_cntinteger5Retry count, with the same semantics as OverlayBD DownloadConfig.tryCnt.
block_sizeinteger16777216Download block size in bytes (16 MiB). Peak scratch memory per active layer download is block_size * concurrency.
concurrencyinteger4Maximum number of in-flight block remote reads within a single remote layer. 1 keeps the historical serial behavior. Must be greater than zero.
max_inflight_blocksinteger16Process-wide cap on in-flight download blocks shared by every concurrent layer download; bounds total scratch memory to max_inflight_blocks * block_size. Must be greater than zero.