Getting Started

AgentENV (abbreviated as AENV) is a self-hosted sandbox runtime for AI agents. It runs isolated Firecracker microVMs and exposes an E2B-compatible HTTP API — so existing E2B SDK code works against it without modification. The repository is available at https://github.com/kvcache-ai/AgentENV.

Why AgentENV

• Scale across diverse environments: AENV runs massive numbers of Firecracker environments across machines and diverse OCI-compatible images, loaded on demand via overlaybd. Local disk acts as a bounded cache, retaining hot data and evicting cold, so images can exceed disk capacity while startup stays fast cluster-wide, without pre-warming every host.
• Make idle environments inexpensive: Snapshot-backed environments boot or resume in under 50 ms and pause in under 100 ms. Idle environments can quickly release CPU and memory, then return when new work arrives.
• Native snapshot and fork support: AENV snapshots memory and filesystem changes incrementally, completing in under 100 ms even under heavy disk modification. A running environment can fork into multiple independent sandboxes for parallel agent workflows. Snapshots persist to S3-compatible object storage or a shared distributed filesystem to prevent data loss.
• Preserve performance and density over time: AENV delivers high-performance I/O via ublk while sharing the host page cache across storage and memory-snapshot data. Memory ballooning returns reclaimable guest memory to the host, sustaining high overcommit as environments run longer and diverge.

Features

• Firecracker microVMs with full Linux kernel isolation per sandbox
• Pause and resume with memory + disk snapshots for instant cold start
• Layered block devices via overlaybd + ublk for copy-on-write image sharing
• Snapshot-backed template builder for publishing reusable, pre-configured sandbox runtimes
• E2B-compatible API so existing E2B SDKs and CLIs work out of the box
• Reverse proxy to reach services running inside sandboxes via HTTP and WebSocket
• Multi-node scaling with a gateway + scheduler control plane (prototype)

Who Is This For

AgentENV is built for teams running AI agents that need isolated execution environments: code interpreters, tool-use agents, autonomous coding agents, or any workload where you want a fresh (or cached) Linux environment per task.

Interacting with the Server

AgentENV exposes an HTTP API. There are four ways to use it:

Where to Go Next

• Quick Start — Install the server, run your first sandbox. Takes ~5 minutes on a supported Linux host.
• Deployment — Build from source, Docker Compose multi-node, or Kubernetes.

Quick Start

Prerequisites

• Linux kernel 6.8+; the install script additionally requires Ubuntu 24.04
• /dev/kvm access for Firecracker microVM execution

The install script attempts to install missing download and checksum commands, provisions KVM permissions, loads the ublk_drv kernel module, and downloads all runtime assets on first run.

Installation requires root, but the installed service does not run as root. It uses a dedicated aenv system account with CAP_NET_ADMIN and CAP_SYS_ADMIN, plus group access to /dev/kvm and the ublk devices.

Setup

1. Install and start the server

Option A — Install Script (Linux x86_64)

The script installs both the server and the aenv CLI. Set AENV_HOME_PATH to choose the data directory; if it is not set, AENV stores runtime dependencies and data in /var/lib/aenv. After installation, start the server as a systemd service:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh | sudo AENV_HOME_PATH=/path/to/aenv/data bash
sudo systemctl start aenv

To customize the server configuration, edit config.toml and restart the service:

sudo vim /var/lib/aenv/config/config.toml  # Or <AENV_HOME_PATH>/config/config.toml if AENV_HOME_PATH is set.
sudo systemctl restart aenv

To change the port, edit API_ADDR in /etc/default/aenv and restart the service.

The service’s persistent state is owned by aenv under /var/lib/aenv by default. Transient namespace and daemon-socket state lives under /run/aenv.

Option B — Docker

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/docker-setup.sh | sudo bash
docker pull ghcr.io/kvcache-ai/aenv-server:latest
docker run --rm -it \
  --device /dev/kvm --privileged -v /dev:/dev \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest

To customize the server configuration, download and edit the configuration file, then mount it at the path used by the container:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/config/default.toml -o config.toml
vim config.toml
docker run --rm -it \
  --device /dev/kvm --privileged -v /dev:/dev \
  -v "$PWD/config.toml:/workspace/config/default.toml:ro" \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest

To change the port, add the -e API_ADDR and -p flags.

The server is accessible at http://127.0.0.1:8000 by default. Verify it is running:

curl http://127.0.0.1:8000/health

2. Install the aenv CLI

Skip this step if you used Option A — the install script already includes aenv.

Supports Linux and macOS on x86_64 and arm64:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install-cli.sh | bash

3. Authenticate

aenv auth
# AENV server URL [http://localhost:8000]: http://127.0.0.1:8000
# API key: dummy

For local development, any non-empty string works as the API key.

4. Pull a template and run a sandbox

aenv pull ubuntu:22.04 --name ubuntu
aenv start ubuntu            # starts a sandbox and attaches an interactive shell

Next Steps

• Deployment — build from source, multi-node options
• Core Concepts — how sandboxes, templates, and snapshots work
• E2B — SDK and CLI compatibility
• API Reference — full HTTP API

On-Demand Loading from Shared Storage

AgentENV loads images on demand via overlaybd. Local disk acts as a bounded cache, retaining hot data and evicting cold data so nodes do not need to pre-warm every image or keep a complete copy of every snapshot.

Prerequisites

• Storage connectivity should be as fast as possible. Use at least a 1 Gbps network; 10 Gbps or faster is strongly recommended.

1. Open the configuration file

If AgentENV is running as a systemd service, open the installed configuration:

sudo vim /var/lib/aenv/config/config.toml  # Or use the path to your config file.

If AgentENV is running in Docker, download and edit the default configuration:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/config/default.toml -o config.toml
vim config.toml

2. Configure shared storage

AgentENV supports two shared storage backends: POSIXFS and OSS. Choose one of the following options.

Option A: POSIXFS

Enable the POSIXFS backend, set the shared snapshot location, and increase the remote-block cache:

[snapshot]
repository_backend = "posix_fs"

[backend.posix_fs]
snapshot_store = "/mnt/aenv-snapshots"

[image.cache.remote_blocks]
max_size_gb = 100

Option B: OSS

Enable the OSS backend, configure the OSS connection, and increase the remote-block cache:

[snapshot]
repository_backend = "oss"

[backend.oss]
endpoint = "YOUR_ENDPOINT"
bucket = "YOUR_BUCKET"
region = "YOUR_REGION"
prefix = "YOUR_PREFIX"
cache_max_size_gb = 100
access_key_id = "YOUR_ACCESS_KEY_ID"
access_key_secret = "YOUR_ACCESS_KEY_SECRET"

[image.cache.remote_blocks]
max_size_gb = 100

3. Apply the configuration

If AgentENV is running as a systemd service, restart it:

sudo systemctl restart aenv

If AgentENV is running in Docker, stop the current container and recreate it with the updated configuration and shared snapshot directory:

# The /mnt/aenv-snapshots mount is required only when using POSIXFS.
docker stop CONTAINER_ID_OR_NAME
docker run --rm -it --name aenv-server \
  --device /dev/kvm --privileged -v /dev:/dev \
  -v "$PWD/config.toml:/workspace/config/default.toml:ro" \
  -v /mnt/aenv-snapshots:/mnt/aenv-snapshots \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest

Once configured, AgentENV loads templates and snapshots from shared storage by default.

aenv CLI Reference

aenv is the native CLI for AgentENV. It wraps the HTTP API and envd gRPC endpoints into a developer-friendly interface for managing templates, sandboxes, and snapshots.

Installation

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install-cli.sh | bash

Or build from source (requires Rust):

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV
make install-aenv

Authentication

aenv auth

Save the server URL and API key. Credentials are stored at ~/.config/aenv/credentials (mode 0600).

aenv auth
# AENV server URL [http://localhost:8000]: The address of the AgentENV server
# API key: dummy (Any non-empty string works for local development.)

Templates

aenv pull <image>

Create a template from an OCI image. Waits for the build to complete by default.

aenv pull ubuntu:22.04
aenv pull ubuntu:22.04 --name my-ubuntu

aenv build <dockerfile>

Create a template from a local Dockerfile.

aenv build ./Dockerfile
aenv build ./Dockerfile -t my-app
aenv build ./Dockerfile --image ghcr.io/myorg/base:latest

aenv template list

List all templates. Alias: aenv template ls, aenv templates list.

aenv template list
aenv template list --output json

aenv template watch <template>

Watch a template build until it succeeds or fails. Accepts either a template name/alias or a template UUID.

aenv template watch my-ubuntu
aenv template watch <template-id>

aenv template delete <template>

Delete a template by name or ID. Alias: aenv template rm.

aenv template delete my-ubuntu
aenv template delete <template-id>

Sandboxes

aenv start <target>

Start a sandbox and attach an interactive shell. <target> is a template name or template UUID.

aenv start my-ubuntu
aenv start --cold ubuntu:24.04              # start directly from an OCI image

<target> accepts a template UUID, template name, or (with --cold) an OCI image reference.

aenv pause <sandbox-id>

Pause a running sandbox. The sandbox state is preserved and can be resumed later.

aenv pause <sandbox-id>

aenv resume <sandbox-id>

Resume a paused sandbox.

aenv resume <sandbox-id>

aenv timeout <sandbox-id> <seconds>

Set or extend the sandbox expiration to <seconds> from now.

aenv timeout <sandbox-id> 600

aenv connect <sandbox-id>

Attach an interactive shell to a running or paused sandbox. Alias: aenv cn.

aenv connect <sandbox-id>

Resumes the sandbox if paused before attaching.

aenv exec <sandbox-id> <command> [args...]

Run a one-shot command in a sandbox and stream its output.

aenv exec <sandbox-id> ls -la /

aenv list

List all sandboxes. Alias: aenv ls.

aenv list

Outputs a table on a TTY and JSON when piped. Override with --output table|json.

aenv delete <sandbox-id>

Kill and delete a sandbox. Alias: aenv rm.

aenv delete <sandbox-id>
aenv rm <sandbox-id>

Snapshots

aenv snapshot create <sandbox-id>

Capture a persistent snapshot from a running sandbox. The snapshot can be used as a template to start new sandboxes with aenv start.

aenv snapshot create <sandbox-id>
aenv snapshot create <sandbox-id> --name my-base

aenv snapshot list

List persistent snapshots. Alias: aenv snapshot ls, aenv snap ls.

aenv snapshot list
aenv snapshot list --sandbox-id <sandbox-id>

To delete a snapshot, use aenv template delete <snapshot-id> or aenv template delete <name> — snapshots share the same underlying store as templates and are deleted through the same command.

Docker (Single Node)

Run a single AgentENV node in a Docker container. This avoids installing the Rust toolchain on the host but still requires /dev/kvm.

Prerequisites

• Linux kernel 6.8+
• /dev/kvm access for Firecracker microVM execution
• Docker

Build

Option A — Pre-built Image

docker pull ghcr.io/kvcache-ai/aenv-server:latest
curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/docker-setup.sh | sudo bash

Option B — Build from Source

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV
sudo bash scripts/docker-setup.sh
docker build -f deploy/docker/Dockerfile.agentenv -t aenv:latest .

To use a regional apt mirror for both build and runtime stages, pass a base URL that contains debian, debian-security, and ubuntu mirror paths:

docker build \
  --build-arg APT_MIRROR_BASE=https://mirrors.example.com \
  -f deploy/docker/Dockerfile.agentenv \
  -t aenv:latest .

Run

docker run --rm -it \
  --device /dev/kvm --privileged -v /dev:/dev \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest   # or aenv:latest if built from source

The --privileged flag is required for Firecracker’s network namespace operations (veth pairs, iptables). The server auto-downloads runtime assets on first start and is accessible at http://127.0.0.1:8000 once ready.

Verify

curl http://127.0.0.1:8000/health

Docker Compose (Multi-Node Simulation)

Run a full multi-node stack on a single host using Docker Compose. This simulates a production-like topology with a gateway, scheduler, and multiple AgentENV backend nodes.

What Gets Started

Prerequisites

• Linux kernel 6.8+
• /dev/kvm access (passed into the runtime containers)
• Docker and Docker Compose
• build-essential (sudo apt install -y build-essential)

Clone the Repository

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV

Start the Cluster

sudo bash scripts/docker-setup.sh
make deploy-up

To enable host-based sandbox data-plane URLs, set the shared sandbox proxy domain variable when starting the stack:

SANDBOX_PROXY_DOMAINS=sandbox.example.com \
make deploy-up

Compose passes this value to both the gateway routing allowlist and runtime nodes’ sandbox response metadata. The domain must resolve to the gateway, usually through wildcard DNS for *.sandbox.example.com.

Verify

# Health check via gateway
curl http://127.0.0.1:8080/health

# Cluster node snapshots via gateway
curl http://127.0.0.1:8080/nodes

# Direct health check on a backend node
curl http://127.0.0.1:8001/health

Management Commands

make deploy-ps      # Show container status
make deploy-logs    # Stream logs from all services
make deploy-down    # Tear down the cluster

Configuration

Container deployments use deploy/docker/config/default.json. Scheduler and backend node endpoints are configured for the Docker network.

The runtime image includes uvm-ublk at /usr/local/bin/uvm-ublk. Compose uses that path instead of a host-built env/ublk/uvm-ublk binary.

The compose manifest also wires node heartbeat reporting from runtime nodes to scheduler:

• AENV_NODE_ID is set explicitly per node container (node-a, node-b).
• AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED=true enables scheduler heartbeat reporting.
• AENV_OBSERVABILITY_SCHEDULER_ENDPOINT is set to http://scheduler:9090.
• SANDBOX_PROXY_DOMAINS, when set, is passed through as both GATEWAY_SANDBOX_PROXY_DOMAINS and AENV_SANDBOX_PROXY_DOMAINS.

Kubernetes (Multi-Node)

Deploy AgentENV across a Kubernetes cluster with a gateway, scheduler, and runtime nodes on every worker.

Architecture

Why a DaemonSet for Runtime Nodes

• Each Pod needs host-local access to /dev/kvm
• Sandbox networking uses host iptables and network namespaces
• Runtime assets and committed snapshot state are cached per-host at /var/lib/aenv

Prerequisites

• Kubernetes worker nodes with Linux kernel 6.8+ and /dev/kvm access
• Runtime Pods run privileged
• Docker
• build-essential (sudo apt install -y build-essential)
• kubectl with Kustomize support

Clone the Repository

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV

Build Container Images

make k8s-build

This builds three images: agentenv-runtime:latest, agentenv-gateway:latest, and agentenv-scheduler:latest.

Deploy

# Run on each worker node before deploying
sudo bash scripts/docker-setup.sh

# Render manifests (preview)
make k8s-render

# Apply to cluster
make k8s-apply

To enable host-based sandbox data-plane URLs, set the shared sandbox proxy domain variable when rendering or applying manifests:

SANDBOX_PROXY_DOMAINS=sandbox.example.com make k8s-apply

The helper writes this value into the generated ConfigMap and applies it to both the gateway routing allowlist and runtime nodes’ sandbox response metadata. The domain must resolve to the gateway Ingress or LoadBalancer, usually through wildcard DNS for *.sandbox.example.com.

The default overlay is deploy/k8s/overlays/default, targeting the agentenv-system namespace. The gateway is exposed as ClusterIP by default. Add your own Ingress or LoadBalancer for external access.

The make targets build a temporary Kustomize context so runtime Pods mount the repository’s config/default.toml rather than a separate checked-in copy.

The runtime DaemonSet injects scheduler-report wiring for each node Pod:

• AENV_UBLK_DAEMON_BINARY_PATH=/usr/local/bin/uvm-ublk-daemon so the Pod uses the uvm-ublk-daemon binary included in the runtime image
• AENV_NODE_ID from Pod metadata name (metadata.name)
• AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED=true
• AENV_OBSERVABILITY_SCHEDULER_ENDPOINT=http://agentenv-scheduler:9090
• AENV_SANDBOX_PROXY_DOMAINS from the shared sandbox proxy ConfigMap

The P2P listen address must be reachable Pod-to-Pod; use a concrete container port or a Pod-reachable address if your cluster policy does not allow dialing ephemeral ports.

Operations

# Rollout restart all workloads
make k8s-redeploy

# Delete all resources
make k8s-delete

Local Development (k3s)

A dedicated local-dev overlay mounts the repository’s env/ directory directly into the DaemonSet at /workspace/env, avoiding runtime asset copies:

make k8s-build
make k8s-load-dev       # Import images into k3s/containerd
make k8s-render-dev     # Preview manifests
make k8s-apply-dev      # Apply to cluster
make k8s-refresh-dev    # Build + load + rollout restart (all-in-one)

Service Discovery

The scheduler watches EndpointSlices for the headless agentenv-nodes Service and watches Pods for optional label-based discovery policy. It schedules only serving, non-terminating DaemonSet Pods. Pods matching scheduler.discovery.kubernetes.no_schedule_pod_selector stay discoverable as lingering/no-schedule nodes, while Pods matching scheduler.discovery.kubernetes.ignore_pod_selector are excluded. Both IPv4 and IPv6 endpoint addresses are supported.

Sandbox bindings remain in-memory, so the scheduler should run as a single replica. Bindings are lost on restart.

Manual Compile (Single Node)

Run AgentENV directly from source on a single Linux host, useful for development and testing.

If you want to skip building from source, see Quick Start.

Prerequisites

• Ubuntu 24.04 with Linux kernel 6.8+
• /dev/kvm access for Firecracker microVM execution
• Rust toolchain (stable) — install via rustup
• sudo access

Clone the Repository

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV

Build

# Debug build
make

# Release build (recommended for production)
make release

Start the Server

# Debug build
API_ADDR=0.0.0.0:8000 make start-server

# Release build
API_ADDR=0.0.0.0:8000 make start-server-release

The server auto-downloads runtime assets (Firecracker binary, kernel, rootfs) on first start. Once ready, it listens at http://127.0.0.1:8000.

Verify

curl http://127.0.0.1:8000/health

Configuration

The server reads config/default.toml by default. Override with:

AENV_CONFIG_PATH=/path/to/config.toml make start-server

See Configuration Reference for all settings.

Configuration Reference

AgentENV reads configuration from a TOML file. The default path is config/default.toml. Override it with:

export AENV_CONFIG_PATH=/path/to/config.toml
# or
cargo run --bin server -- --config /path/to/config.toml

Global Settings

$AENV_HOME is a literal placeholder in state-path values, not a shell environment variable. AgentENV replaces it with the resolved home_path after applying AENV_HOME_PATH; ublk.daemon_socket_path additionally supports $AENV_RUNTIME, which resolves to runtime_path. Relative paths without these placeholders are resolved against the directory containing the configuration file.

Packaged runtime dependency versions and download URLs live in config/deps_manifest.toml. config.toml should contain runtime behavior and explicit local path overrides, not the default dependency catalog.

[firecracker]

Firecracker VM binary and boot configuration.

[kernel]

Linux kernel image for microVMs.

[tools]

Tools drive image used to boot the AgentENV control plane inside each microVM.

Snapshots and paused sandboxes keep using the tools drive version they were created with. Launch does not download missing releases: operators must install the recorded version under <deps_path>/tools/<version>/tools.ext4 before restore. Setup retains previously installed versions until they are removed manually.

Template Rootfs Images

User-visible rootfs images are selected at the template API layer. POST /v2/templates/{templateID}/builds/{buildID} accepts an optional fromImage field:

• omitted: use [image.resolver].default_image
• full OCI reference: use the supplied image
• short name: normalize standard Docker Hub forms such as ubuntu:24.04 and node:20

[image.resolver]

How the three registry settings interact

Image resolution runs in two phases, and the three keys act at different points:

1. search_registries — completion. Only used for short / unqualified references (e.g. ubuntu). Each entry is prefixed to the name to build a list of fully-qualified candidates (docker.io/library/ubuntu:latest, ghcr.io/ubuntu:latest, …). Fully-qualified references skip this step.
2. allowed_registries — gating. Applied right after candidates are built, to both fully-qualified references and the candidates expanded from search_registries. Candidates whose registry host is not whitelisted are dropped; if none remain, the reference is rejected with a 4xx error. In effect the resolvable set of short-name hosts is the intersection of search_registries and allowed_registries — e.g. searching docker.io and ghcr.io while only allowing ghcr.io resolves short names to ghcr.io only.
3. try_referrers_overlaybd_prefixes — per-candidate optimization. Runs later, while resolving an already-permitted candidate: after its manifest is fetched, AgentENV may query the OCI Referrers API on the same registry/repository for an overlaybd-native artifact. Because referrer lookups never leave the source image’s own host, they are implicitly covered by allowed_registries — no separate whitelist entry is needed for referrers.

[image.cache]

Node-local cache root for resolved and converted user images.

[image.cache.gc]

Background hard-commit garbage collection for the image cache. When enabled, each pass reconciles metadata from the on-disk source configs and then deletes hard-commit objects that are no longer rooted by source configs, held by image-cache leases, or referenced by the in-process running set. Committed snapshots are durable SnapshotRepository state and do not pin ImageCache commits. With capacity_gb set, GC first evicts least-recently-used source configs over the high watermark so hard-commit GC can reclaim what they unrooted.

Capacity-driven eviction runs only when [image.cache].capacity_gb is set; otherwise the GC still reclaims unreachable commits but performs no watermark eviction.

[image.cache.remote_blocks]

Overlaybd registryfs_v2 remote block cache settings. The directory is always <image.cache.root_dir>/remote-blocks.

Resolved image data is cached under:

<image.cache.root_dir>/
  commits/
    <sha256-commit-digest>/
      overlaybd.commit
                  # full overlaybd commit store shared by OCI conversion and download
  indexes/        # OCI layer + conversion context -> overlaybd commit descriptor
  remote-blocks/  # overlaybd-native remote block cache
  configs/         # resolved image configs
    <slug>-<hash>-image.json

[sandbox_proxy]

Optional host-based data-plane routing for sandbox services.

When domains is empty, the server still supports /proxy and routing-header proxy requests, but does not classify requests by Host. Domains are normalized to lowercase, deduplicated, and must be valid DNS names. The configured order is preserved because domains[0] is the advertised sandbox domain.

Environment variable override:

• AENV_SANDBOX_PROXY_DOMAINS

[network.egress]

Node-level sandbox egress guardrails. These rules are installed before per-sandbox allowOut / denyOut rules, so sandbox API requests cannot override them.

[network.internal]

AgentENV-internal sandbox address plan. Change these only when the defaults overlap with host or deployment network ranges.

The two configured CIDRs must not overlap each other or AgentENV’s fixed VM tap link 169.254.0.20/30. These networks are also treated as reserved sandbox egress destinations regardless of always_denied_cidrs.

[machine]

Default VM resources for sandboxes.

[envd]

In-guest envd daemon settings.

[orchestrator]

Sandbox lifecycle management.

[pool]

Shared process-wide warm-pool defaults used by network slots, block devices, and pre-spawned Firecracker processes. Pools prewarm to the low watermark, then grow the refill target geometrically toward the high watermark when real acquisitions drain the pool.

Component sections:

Validation rules:

• low_watermark <= high_watermark
• [pool.firecracker].fill_concurrency > 0

[node_identity]

Stable identity fields for this node. These values appear in node API responses and scheduler heartbeats.

[observability]

Node-level observability and host metrics collection.

When observability is enabled, host CPU/memory/disk metrics are collected at request time. CPU percent is computed from two samples; the first node metrics request waits about 100ms to return a measured value.

[observability.scheduler_report]

Optional scheduler heartbeat reporting for multi-node control plane integration.

Environment variable overrides:

• AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED
• AENV_OBSERVABILITY_SCHEDULER_ENDPOINT
• AENV_OBSERVABILITY_REPORT_INTERVAL_SECS

AENV_OBSERVABILITY_SCHEDULER_ENDPOINT overrides [cluster].scheduler_endpoint for the reporter process only.

[cluster]

Shared cluster-level service endpoints.

[p2p]

Project-wide artifact transport configuration. The transport is disabled by default. When enabled, it is used by the overlaybd P2P HTTP facade and by snapshot publication/runtime resolution as an optional artifact visibility and acceleration path.

[custom_extension]

Custom extension service configuration. When url is unset, the integration is fully disabled. See Custom Extension.

[snapshot]

Snapshot storage/build configuration.

Environment variable overrides:

• AENV_SNAPSHOT_LOCAL_CACHE_PATH

[backend.posix_fs]

POSIX filesystem-backed snapshot repository configuration. This section is used when snapshot.repository_backend = "posix_fs".

Environment variable overrides:

• AENV_SNAPSHOT_STORE

[backend.oss]

OSS-backed snapshot repository configuration. This section is required when snapshot.repository_backend = "oss".

Notes:

• credential_process and static access key settings are mutually exclusive in practice; when credential_process is set, the backend ignores static credential fields.
• credential_process should be written as a portable argv-style command line. Avoid $VAR, backticks, $(...), pipes, and shell builtins.
• Although the config section is still named oss, the runtime path is implemented via a shared S3-compatible client, so region must be configured.

Other path override:

• AENV_DEPS_PATH

Setup sysctl tuning is host-level setup. It is skipped before reading /proc/sys when the server detects that it is running inside a container. Set AENV_FORCE_SYSCTL_TUNING=1 only for a privileged container with writable host sysctls; otherwise configure these kernel parameters on the host.

[protoc]

Protobuf compiler metadata for code generation lives in config/deps_manifest.toml, not config.toml.

[ublk]

Optional userspace block device configuration. When enabled, rootfs is served through a ublk device instead of a plain file, managed by uvm-ublk-daemon.

Environment variable override:

• AENV_UBLK_DAEMON_BINARY_PATH
• AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR

[ublk.overlaybd]

Overlaybd-specific configuration used when ublk.device_type = "overlaybd".

global_config_path and auto-generated config

The file at the configured default path $AENV_HOME/overlaybd/overlaybd-global.json is auto-generated by the server at startup. The generated JSON incorporates several TOML settings — [image.cache].root_dir, [image.cache.remote_blocks].max_size_gb, download_enable, [backend.oss] credentials, and Docker registry credentials detected from ~/.docker/config.json — into a single overlaybd runtime config file.

The server regenerates the file at global_config_path on every startup, so these TOML settings always take effect automatically — any manual edits to the generated file are overwritten on the next startup. To keep customizations, make them through the TOML settings, not by editing the generated JSON.

[memory_snapshot]

Memory snapshot overlaybd configuration. The server auto-generates the file at the default path on every startup.

[memory_snapshot.background_download]

Background download settings dedicated to remote memory-snapshot OverlayBD layers. They do not change the general rootfs or attached-drive defaults. All fields are serialized into the generated memory OverlayBD global config. Each remote layer is downloaded block by block: a sequential sparse-file scan collects the pending blocks, then at most concurrency block tasks fetch non-overlapping ranges in parallel on a dedicated multi-thread runtime. Layer files are still processed one at a time. Downloads of a sandbox-bound device start only after envd is ready (plus delay), with a 20s fallback if the ready signal is lost; while foreground remote reads are in flight, background block reads yield to a small guaranteed floor instead of competing at full speed. The generated memory config leaves throttling off (maxMBps = 0); image configs that carry a positive maxMBps keep their historical shared rate limit across the block tasks. Completed layers are switched to the local file only after a full-file digest check; a failed or canceled block never switches.

Environment Variables

Deployment Helpers

These variables are consumed by the repository’s Docker Compose and Kubernetes helpers, then passed to the server or gateway-specific variables listed below.

Server

E2B SDK / CLI

These variables configure the E2B SDK and CLI to point at an AgentENV server. Values depend on your deployment mode.

Values by Deployment Mode

Manual compile (single node):

export E2B_API_URL=http://127.0.0.1:8000
export E2B_SANDBOX_URL=${E2B_API_URL}
export E2B_API_KEY=e2b_000000
export E2B_ACCESS_TOKEN=dummy

Docker Compose / Kubernetes (multi-node):

export E2B_API_URL=http://127.0.0.1:8080
export E2B_SANDBOX_URL=${E2B_API_URL}
export E2B_API_KEY=e2b_000000
export E2B_ACCESS_TOKEN=dummy

In both modes, sandbox data-plane requests can use routing headers with E2B_SANDBOX_URL=${E2B_API_URL}. The explicit /proxy prefix (${E2B_API_URL}/proxy) is still accepted for back-compat.

For local development, any non-empty value works for E2B_API_KEY and E2B_ACCESS_TOKEN because the server only checks that the auth header is present.

Gateway and Scheduler

These variables apply to both the gateway and scheduler processes.

Gateway

Scheduler

How AgentENV Works

AgentENV runs AI agents inside isolated Firecracker microVMs. Each sandbox is a lightweight Linux VM with its own kernel, filesystem, and network namespace.

System Overview

                    ┌──────────────────────────────────────────────────┐
                    │                  AgentENV Node                   │
                    │                                                  │
                    │  ┌──────────┐   ┌──────────────┐                 │
                    │  │ API      │──>│ Orchestrator │                 │
                    │  │ (Axum)   │   │ (lifecycle)  │                 │
                    │  └──────────┘   └──────┬───────┘                 │
                    │                        │                         │
                    │              ┌─────────▼───────────┐             │
                    │              │  Firecracker VM     │             │
                    │              │                     │             │
                    │              │  /dev/vda (rootfs)  │             │
                    │              │  /dev/vdb (extra)   │             │
                    │              │                     │             │
                    │              └─────────────────────┘             │
                    │                        │                         │
                    │              ┌─────────▼────────────┐            │
                    │              │  Block Device Layer  │            │
                    │              │  (overlaybd + ublk)  │            │
                    │              └──────────────────────┘            │
                    └──────────────────────────────────────────────────┘

Request Flow

1. A client sends an HTTP request to the AgentENV API (for example, POST /sandboxes).
2. The API layer validates the request, checks authentication, and forwards it to the orchestrator.
3. The orchestrator manages the sandbox lifecycle: it creates a Firecracker VM, sets up networking, and attaches block devices.
4. The VM boots with a layered block device (overlaybd) that stacks read-only base image layers with a writable upper layer. Multiple sandboxes share the same base layers.
5. Inside the VM, an envd daemon handles command execution, file operations, and health reporting.
6. Clients interact with running sandboxes via the reverse proxy (/proxy, routing headers, or configured sandbox proxy domains), which forwards HTTP and WebSocket traffic to services inside the VM.

Key Components

Multi-Node Architecture

For multi-node deployments, a gateway and scheduler sit in front of multiple AgentENV nodes:

    Client ──HTTP──> Gateway (:8080) ──gRPC──> Scheduler (:9090)
                        │                          │
                        │    ┌─────────────────────┘
                        │    │ node selection / lookup
                        ▼    ▼
                   Node A (:8000)    Node B (:8000)

The gateway routes requests by sandbox ID. For new sandboxes, the scheduler picks a node. For existing sandboxes, the scheduler looks up the node that owns it. See Deployment for setup instructions.

Templates

A template is the user-facing wrapper around a committed snapshot. Instead of booting a fresh VM and installing software every time, you build a template once and later create sandboxes from it in milliseconds.

How Templates Work

1. Define: specify an overlaybd-backed base rootfs and ordered steps such as run, env, and workdir.
2. Build: AgentENV boots a temporary sandbox and executes those steps inside it.
3. Finalize: optional startup commands can be started and checked for readiness before the snapshot is captured.
4. Publish: the result is committed as a snapshot in the snapshot repository.
5. Launch: creating a sandbox from a template resolves the committed snapshot and resumes from it.

Create Your Template

There are two ways to create a template: aenv pull imports an OCI image directly, and aenv build runs Dockerfile instructions inside a temporary build sandbox.

aenv pull

Pulling an existing OCI image as a template:

aenv pull ubuntu:24.04

Env, WorkingDir, and User are automatically inherited from the OCI image config. See Runtime Configuration for the full field list.

aenv build

⚠️ Experimental — Not recommended for production use.

Build a template by running Dockerfile instructions inside a temporary sandbox:

aenv build ./Dockerfile

Supported Dockerfile instructions:

Runtime Configuration

Both methods read the same set of OCI image config fields. For aenv pull, these come from the image config or flags; for aenv build, they are set by the corresponding Dockerfile instructions executed during the build. The following fields from the OCI image-spec config object are recognised:

Aliases

Each template is identified by a UUID. Pass --name or -t at creation time to assign a human-readable alias:

aenv pull ubuntu:24.04 --name my-base
aenv build ./Dockerfile -t my-service

The alias can be used wherever a template ID is accepted:

aenv start my-base
aenv template delete my-service

Manage Templates

List templates

aenv template list        # alias: aenv template ls

Displays all templates with their ID, name, build status, CPU, memory, disk size, and last-updated timestamp.

Delete a template

aenv template delete <template-id-or-name>   # alias: aenv template rm

Watch a build

aenv build always detaches immediately after submitting. Use watch to follow the build status until it succeeds or fails:

aenv template watch <template-id-or-name>

Start a sandbox from a template

aenv start <template-id-or-name>      # start and attach an interactive shell
aenv start -d <template-id-or-name>   # detach: print sandbox ID and exit

Relationship to Snapshots

Templates are the API and UX layer. Snapshots are the durable runtime layer.

• A template build publishes one committed snapshot.
• A template ID or alias resolves to one committed snapshot.
• A sandbox created from a template resumes from that snapshot.

If you want the storage and runtime model underneath templates, see Snapshots.

Sandboxes

A sandbox is an isolated Firecracker microVM with its own Linux kernel, filesystem, and network stack. Each sandbox runs independently and can be paused, resumed, or deleted.

Lifecycle

Creating ──> Running ──> Pausing ──> Paused
                │                      │
                ├──> Snapshotting      └──> Resuming ──> Running
                ├──> Forking
                └──> Killing

Starting a Sandbox

There are two ways to start a sandbox:

Warm Start (from a template)

Starting from a pre-built template restores a snapshot of a known filesystem state.

aenv start <template-id>

See Templates for how templates are created.

Cold Start (from an OCI image)

A cold start pulls an OCI image directly and converts it into a block device at runtime.

aenv start --cold ubuntu:24.04

The cold-start API accepts an optional diskSizeMB field to set the root filesystem’s virtual size in MiB. Explicit values must be at least 1024 MiB and divisible by 1024 because the current resize tool operates at 1 GiB granularity. Growth is allowed by default; shrinking below the source image size requires ublk.overlaybd.allow_shrink = true. If omitted, the image’s built-in virtual size is used. Resizing applies only when creating a fresh writable root filesystem, not to read-only images, images with an existing upper, or snapshot resume. Sandbox responses also report disk size as diskSizeMB.

Working with Sandboxes

Shell and command execution

# Attach to a sandbox
aenv connect <sandbox-id>

# Run a one-shot command and stream its output
aenv exec <sandbox-id> ls -la /

Pause and Resume

Pausing a sandbox captures:

• Memory snapshot of the running VM state
• Disk snapshot of the writable filesystem layer

Resuming restores the VM from these snapshots in milliseconds. The sandbox picks up exactly where it left off, including running processes and open network connections.

aenv pause <sandbox-id>
aenv resume <sandbox-id>

Persistent Snapshots

A snapshot captures the state of a running sandbox into a template that can be used to start new sandboxes.

aenv snapshot create <sandbox-id>
aenv snapshot create <sandbox-id> --name my-base

The resulting snapshot appears in aenv snapshot list and can be started with aenv start <name>. See Snapshots for details.

Fork

Forking clones a running sandbox into up to 16 child sandboxes on the same node. The source sandbox is briefly paused while the clone is captured, then resumes. All children inherit the source’s filesystem, memory, and resource configuration.

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"count": 3}' \
  http://127.0.0.1:8000/sandboxes/<sandbox-id>/fork

Managing sandboxes

# List all sandboxes
aenv list

# Delete a sandbox
aenv delete <sandbox-id>

Auto-Eviction

Every sandbox has a TTL (time-to-live). When it expires, one of two actions is taken:

• pause (default) — the sandbox is paused and its state is preserved
• kill — the sandbox is deleted permanently

Set TTL with --timeout <secs>:

# Start a sandbox with TTL of 600s
aenv start <template-id> -d --timeout 600

# Set the sandbox expiration for 600 seconds from now
aenv timeout <sandbox-id> 600

To delete instead of pause on expiry, use the API directly with autoPause: false:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"templateID": "<template-id>", "timeout": 600, "autoPause": false}' \
  http://127.0.0.1:8000/sandboxes

The default timeout is configured in config/default.toml under [orchestrator].default_sandbox_timeout_secs.

Networking

Each sandbox runs in its own network namespace. By default, outbound internet access is enabled. To disable it at creation time:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"templateID": "my-ubuntu", "allowInternetAccess": false}' \
  http://127.0.0.1:8000/sandboxes

For fine-grained egress control, pass a network object when creating the sandbox. Allowed entries take precedence over denied entries:

• allowOut — CIDR, IP, or domain patterns
• denyOut — CIDR or IP

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "templateID": "my-ubuntu",
    "network": {
      "allowOut": ["8.8.8.8/32", "example.com", "*.example.com"],
      "denyOut": ["0.0.0.0/0"]
    }
  }' \
  http://127.0.0.1:8000/sandboxes

Egress rules can also be updated on a running sandbox:

curl -X PUT \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"allowOut": ["8.8.8.8/32"], "denyOut": ["0.0.0.0/0"]}' \
  http://127.0.0.1:8000/sandboxes/<sandbox-id>/network

Omitting both fields clears all egress rules.

Data Storage

Snapshots

Snapshots are the durable runtime primitive in AgentENV. Everything else is built on top of them:

• Templates are stored as snapshots. A template build commits one snapshot; the template ID is an alias that resolves to it.
• Sandboxes launch by resuming from a snapshot.
• Running sandboxes can produce new snapshots, capturing their current state for later reuse or branching.

Create a Snapshot from a Running Sandbox

Capture the current state of a live sandbox:

aenv snapshot create <sandbox-id>
aenv snapshot create <sandbox-id> --name my-checkpoint

Pass --name to assign a human-readable alias. The alias can then be used with aenv start or any command that accepts a snapshot ID.

The sandbox transitions through Running → Snapshotting → Running during capture; the sandbox continues running after the snapshot is committed. Recoverable failures leave the sandbox running and surface the error to the caller. Terminal failures — where the runtime was mutated past safe resume — tear down the sandbox.

Manage Snapshots

List Snapshots

List all snapshots, optionally filtered by source sandbox:

aenv snapshot list                           # alias: aenv snapshot ls
aenv snapshot ls --sandbox-id <sandbox-id>

The --sandbox-id filter works even after the source sandbox has been deleted.

Start Sandboxes from Snapshots

Start a new sandbox from a snapshot:

aenv start <snapshot-id-or-name>

Delete Snapshots

Snapshots share the same catalog as templates. To delete a snapshot, use the template delete command with the snapshot ID or alias:

aenv template delete <snapshot-id-or-name>

Optional P2P Visibility

When [p2p].enabled = true, published snapshot artifacts are advertised through the node-to-node P2P transport after the repository commit succeeds. This makes new artifacts discoverable to peer nodes before slow backend reads become the bottleneck:

• vm_state.bin and firecracker-manifest.json are advertised under snapshot-scoped keys.
• Overlaybd commit layers are advertised under the shared overlaybd layer keys (overlaybd-layer/v1/sha256:<digest>), so overlaybd runtime reads can find them through the overlaybd P2P facade.

P2P does not change the committed snapshot model. The snapshot repository remains the source of truth, and failed P2P publication does not roll back a successful snapshot publish.

Storage

Three-layer model

It helps to treat these as three different layers rather than one combined store:

1. Builder staging

This is a manager-owned temporary workspace used while executing one build. It contains local snapshot artifacts captured from the build sandbox, for example:

<local_cache_path>/
  snapshots/<id>/
    vm_state.bin
    mem_image.json
    rootfs/
      image.json
    drives/<drive-id>/
      image.json

This directory is a TemplateBuilder implementation detail. It is not the durable committed record.

2. Committed snapshot repository

Once a build is published, the durable state lives in the snapshot repository:

<snapshot_store>/
  repository/
    catalog/
      aliases/
    snapshots/<id>/
      snapshot.json
      firecracker-manifest.json
      vm_state.bin
    managed-layers/
      <digest>.overlaybd.commit

This repository is the source of truth wrapped by the template API.

3. Node-local runtime cache

Before a sandbox is launched, AgentENV derives runnable overlaybd configs from the committed snapshot and materializes them in a node-local runtime cache:

<node_local_runtime_cache>/
  runtime/<id>/
    memory/
      image.json
    rootfs/
      image.json
    drives/<drive-id>/
      image.json

These are launch-time derived runtime inputs, not committed metadata.

What a committed snapshot contains

Each committed snapshot is split across two durable files:

snapshot.json contains:

• context — the runtime configuration baked in at build time: environment variables, working directory, user, exposed ports, volumes, and labels. Sandboxes launched from this snapshot inherit these values.
• rootfs_layers, attached_drives[].layers, memory_layers — overlaybd layer references, either as managed local layers (deduplicated by content digest) or as external OCI registry references.
• Startup configuration.

firecracker-manifest.json contains launch-time metadata not expressed as overlay layer lists, including memory.virtual_size, rootfs.virtual_size, and per-drive metadata.

The following files exist only as local build artifacts or node-local derived configs and are not committed:

• rootfs/image.json
• drives/<id>/image.json
• mem_image.json
• upper.data
• upper.index

Runtime resolution

Before launch, the committed state is resolved into node-local paths:

• memory_layers become runtime memory/image.json
• rootfs.layers become runtime rootfs/image.json
• attached_drives[].layers become runtime drives/<id>/image.json
• firecracker-manifest.json is loaded and hydrated with node-local absolute paths
• committed vm_state.bin becomes the runnable vm-state path

Custom Extension

The custom extension is an optional external HTTP service that AgentENV calls during the sandbox lifecycle. It lets you implement deployment-specific behavior — for example, connecting sandboxes into a VPN, custom firewall rules, or extra mounts — without changing AgentENV itself.

The extension implements a small set of HTTP endpoints (“hooks”); AgentENV is the client. The interface is defined in src/custom_extension_api/openapi.yml.

Configuration

Enable the integration by pointing AgentENV at your extension service:

# config/default.toml (or your AENV_CONFIG_PATH)
[custom_extension]
url = "http://127.0.0.1:9090"
# timeout_ms = 5000   # optional, per-call timeout in milliseconds

AENV_CUSTOM_EXTENSION_URL works as well. When url is unset, the integration is fully disabled: no hooks are called and customExtensionParams must be empty.

Lifecycle hooks

All hooks are POST {url}/sandbox-hook/* with a JSON body. Any connection error, timeout, or non-2xx response fails the corresponding sandbox operation (except stop, which is best-effort).

Notes:

• Instance identity. A sandboxId is reused across pause/resume cycles. Every start-fresh / start-resume carries a fresh sandboxInstanceId identifying that runtime instance, and the subsequent stop carries the same value. Because stop is best-effort and may be reordered (e.g. a pause’s stop arriving after the resume’s start-resume), treat (sandboxId, sandboxInstanceId) as the identity of a running instance and ignore stop notifications whose sandboxInstanceId is not the latest started instance for that sandbox.
• stop also fires on pause. Pausing persists the sandbox state and then stops the VM process and releases the network namespace; the subsequent resume creates a fresh runtime and fires start-resume. In-place pause+resume during snapshot capture does not fire any hook (and keeps the same sandboxInstanceId).
• stop is best-effort: delivery failures are only logged, and it is also fired fire-and-forget if a started sandbox is dropped without an explicit stop.
• networkNamespacePath is the host path of the sandbox’s netns file (e.g. /var/run/netns/agentenv-ns-*), so the extension can enter the namespace (e.g. nsenter --net=...) to set up firewall rules or VPN interfaces.
• Concurrent patch-params calls to the same sandbox are not serialized; if your patch semantics are not commutative, handle concurrency in the extension.

customExtensionParams

An opaque JSON object per sandbox, interpreted only by the extension. An absent value and an empty object are equivalent (empty params).

Set at creation — POST /sandboxes and POST /sandboxes-cold accept customExtensionParams:

{
  "templateID": "my-template",
  "customExtensionParams": { "vpn": { "network": "team-a" } }
}

Non-empty params are rejected with 400 when no extension is configured on the server.

Read — GET /sandboxes/{sandboxID}/custom-extension-params returns the current value ({} when empty).

Patch — PATCH /sandboxes/{sandboxID}/custom-extension-params (running sandboxes only). The request body is passed through verbatim to the extension’s patch-params hook; its semantics are defined entirely by the extension. The hook returns the updated full params, which AgentENV stores and returns:

curl -X PATCH .../sandboxes/{id}/custom-extension-params \
  -d '{"vpn": {"network": "team-a", "peers": ["10.8.0.2", "10.8.0.3"]}}'

Persistence — params survive pause/resume and are stored into snapshots created from the sandbox. When starting from a template, a customExtensionParams provided at creation overrides the one stored in the snapshot; otherwise the snapshot’s value is inherited.

Minimal extension example

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

# Latest started runtime instance per sandbox: (sandboxId, sandboxInstanceId)
# is the identity of a running instance; a stop for a superseded instance
# (e.g. arriving after a newer start) is ignored.
latest_instance: dict[str, str] = {}

@app.post("/sandbox-hook/start-fresh")
async def start_fresh(req: Request):
    body = await req.json()
    latest_instance[body["sandboxId"]] = body["sandboxInstanceId"]
    # e.g. nsenter --net={body["networkNamespacePath"]} wg-quick up ...
    return {"extraBootArgs": None}

@app.post("/sandbox-hook/start-resume")
async def start_resume(req: Request):
    body = await req.json()
    latest_instance[body["sandboxId"]] = body["sandboxInstanceId"]
    return {}

@app.post("/sandbox-hook/patch-params")
async def patch_params(req: Request):
    body = await req.json()
    # apply body["patch"] however you like, then return the full new params
    return {"customExtensionParams": body["patch"]}

@app.post("/sandbox-hook/stop")
async def stop(req: Request):
    body = await req.json()
    if latest_instance.get(body["sandboxId"]) == body["sandboxInstanceId"]:
        latest_instance.pop(body["sandboxId"], None)
        # tear down resources for this instance
    return {}

Any non-2xx response (or timeout) fails the corresponding sandbox operation, except for stop, which is always tolerated.

Proxy

The reverse proxy lets you reach services running inside a sandbox from outside. It supports HTTP requests, SSE streams, and WebSocket connections.

Endpoints

• ANY /proxy forwards to / inside the sandbox
• ANY /proxy/{path} forwards to /{path} inside the sandbox
• Header-routed requests on otherwise unmatched paths forward the original path unchanged. This lets clients use the same base URL for API and sandbox data traffic when they send routing headers.
• When sandbox proxy domains are configured, host-based URLs shaped like {port}-{sandboxID}.{domain} forward the original path without routing headers.

Query strings are forwarded unchanged.

Required Headers

Each proxied request must identify the target sandbox and port:

E2B-compatible aliases are also accepted:

These routing headers are stripped before the request is forwarded to the sandbox.

Host-based proxy requests derive both values from Host, for example http://8080-<sandbox-uuid>.sandbox.example.com/health targets port 8080. The configured domain must route to the AgentENV server in single-node mode or to the gateway in multi-node mode. Host-based proxy traffic is always treated as data-plane traffic; lifecycle and other control-plane APIs should use the base API host.

API Reference

E2B

E2B SDK

AgentENV exposes an E2B-compatible API, so the official E2B SDK works out of the box.

General Settings

Set environment variables to point at your AgentENV server. See Environment Variables for values per deployment mode.

# Single-node example
export E2B_API_URL=http://127.0.0.1:8000
export E2B_SANDBOX_URL=${E2B_API_URL}
export E2B_API_KEY=e2b_000000
export E2B_ACCESS_TOKEN=dummy

TypeScript SDK

Setup

Install the SDK:

npm install e2b

Usage

import { Sandbox } from "e2b";

// Create a sandbox from a template
const sandbox = await Sandbox.create("<template-id>", {
  apiKey: process.env.E2B_API_KEY,
});

// List running sandboxes
const running = Sandbox.list({
  apiKey: process.env.E2B_API_KEY,
  limit: 20,
  query: { state: ["running"] },
});
console.log(await running.nextItems());

// Run a command inside the sandbox
sandbox.commands.run("echo hello world");

// Pause the sandbox
await Sandbox.Pause(sandbox.sandboxId, {
  apiKey: process.env.E2B_API_KEY,
});

// Kill the sandbox
await sandbox.kill();

Replace <template-id> with a template that exists in your local template store. Use e2b template list or GET /v2/templates to see available templates.

Python SDK

Setup

Install the SDK:

pip install e2b

Usage

from e2b import Sandbox, SandboxQuery, SandboxState

# Reuse the environment variables set in your shell:
# E2B_API_URL / E2B_SANDBOX_URL / E2B_API_KEY

# Create a sandbox from a template
sandbox = Sandbox.create("<template-id>")

# List running sandboxes
running = Sandbox.list(
    limit=20,
    query=SandboxQuery(state=[SandboxState.RUNNING]),
)
print(running.next_items())

# Run a command inside the sandbox
result = sandbox.commands.run("echo hello world")
print(result.stdout, end="")

# Pause the sandbox
sandbox.beta_pause()

# Kill the sandbox
sandbox.kill()

E2B CLI

AgentENV is compatible with the E2B CLI, but we recommend using the aenv CLI for AgentENV workflows.

Common Issues

/dev/kvm not accessible

Symptom: Server fails to start with a KVM-related error.

Solution: Ensure your host has hardware virtualization enabled (Intel VT-x or AMD-V) and that /dev/kvm is readable by the current user. On most systems:

sudo usermod -aG kvm $USER
# Log out and back in for the group change to take effect

Permission denied for network operations

Symptom: Sandbox creation fails with network namespace or iptables errors.

Solution: The server requires both CAP_NET_ADMIN and CAP_SYS_ADMIN in its effective, permitted, and inheritable sets. The installed systemd unit configures these automatically. For a source checkout, use make start-server or scripts/run-with-capabilities.sh <server-binary>; do not run the whole server as root. Also verify that the runtime account belongs to the kvm group and can open /dev/ublk-control.

Sandbox namespaces are missing from ip netns list

Symptom: Sandboxes are running, but ip netns list does not show their network namespaces and ip netns exec <name> cannot find them.

Solution: AgentENV stores namespace mount points under $AENV_RUNTIME_PATH/netns instead of /var/run/netns; the installed service uses /run/aenv/netns. Inspect that directory directly and enter a namespace by path when needed:

sudo nsenter --net=/run/aenv/netns/agentenv-ns-<slot> ip addr

Config file not found

Symptom: Error: config file not found

Solution: The server looks for config/default.toml by default. Either run from the repository root or set AENV_CONFIG_PATH:

export AENV_CONFIG_PATH=/path/to/your/config.toml

Port already in use

Symptom: Address already in use when starting the server.

Solution: Another process is using port 8000. Either stop it or change the listen address:

API_ADDR=0.0.0.0:8001 make start-server

Sandbox creation timeout

Symptom: POST /sandboxes returns a timeout error.

Solution: Check that runtime assets (Firecracker binary, kernel, rootfs) have been downloaded. The server auto-provisions them on first start, but network issues can cause failures. Run cargo run --bin server -- --setup-only to provision manually and see detailed errors.

Also check [envd].init_timeout_secs in your config. The default is 60 seconds. If the rootfs image is large, the in-guest envd daemon may need more time to initialize.

TODO: Expand with more common issues as they are reported.

AgentENV Architecture

AgentENV runs AI agents inside isolated, snapshot-capable Firecracker microVMs. Its core is a storage subsystem that provides layered block devices mountable into VMs and ublk-backed memory snapshot restore. The system also includes a per-node orchestrator managing sandbox lifecycle, and a prototype distributed control plane (gateway + scheduler) for multi-node routing.

System Overview

                    ┌───────────────────────────────────────────────────────────┐
                    │                       AgentENV Node                       │
                    │                                                           │
                    │  ┌──────────┐   ┌──────────────┐                          │
                    │  │ API      │──>│ Orchestrator │                          │
                    │  │ (Axum)   │   │ (lifecycle)  │                          │
                    │  └──────────┘   └──────┬───────┘                          │
                    │                        │                                  │
                    │              ┌─────────▼───────────┐                      │
                    │              │  Firecracker VM     │                      │
                    │              │                     │                      │
                    │              │  /dev/vda (rootfs)  │                      │
                    │              │  /dev/vdb (extra)───┼───┐                  │
                    │              │  VM memory ─────────┼───┼──┐               │
                    │              │                     │   │  │               │
                    │              └─────────────────────┘   │  │               │
                    │                                        │  │               │
                    │    Block device path:                  │  │               │
                    │              ┌────────────────────────▼──┐│               │
                    │              │  ublk (/dev/ublkbN)       ││               │
                    │              │  userspace block device   ││               │
                    │              └────────────┬──────────────┘│               │
                    │                           │               │               │
                    │              ┌─────────────▼─────────────┐│               │
                    │              │  overlaybd                ││               │
                    │              │  ┌───────┐ ┌───────┐      ││               │
                    │              │  │ upper │ │layer 0│ ...  ││               │
                    │              │  │ (r/w) │ │(r/o)  │      ││               │
                    │              │  └───────┘ └───────┘      ││               │
                    │              └───────────────────────────┘│               │
                    │                                           │               │
                    │    Memory restore path:                   │               │
                    │              ┌────────────────────────────▼───┐           │
                    │              │  ublk (/dev/ublkbM)            │           │
                    │              │  read-only memory block device │           │
                    │              │  (shared across same-snapshot  │           │
                    │              │   sandboxes via refcounting)   │           │
                    │              └────────────┬───────────────────┘           │
                    │                           │                               │
                    │              ┌─────────────▼──────────────┐               │
                    │              │  overlaybd (mem layers)    │               │
                    │              │  ┌───────┐ ┌───────┐       │               │
                    │              │  │snap N │ │snap 0 │ ...   │               │
                    │              │  │(r/o)  │ │(r/o)  │       │               │
                    │              │  └───────┘ └───────┘       │               │
                    │              └────────────────────────────┘               │
                    └───────────────────────────────────────────────────────────┘

Storage

The storage subsystem turns layered image files into block devices mountable by VMs, and provides ublk-backed memory snapshot restore for snapshot resume. Four crates compose the active subsystem:

overlaybd (storage/overlaybd/)

LSMT (Log Structured Merge Tree) based layered image format.

Image structure: Each layer file has a HeaderTrailer (magic LSMT\0\1\2, UUID, flags, index/data offsets) and an array of DiskSegmentMapping entries (16 bytes each, bit-packed: 50-bit offset, 14-bit length, 55-bit physical offset, zeroed flag, layer tag). Layers are stacked: immutable compressed read-only layers at the bottom, a single writable upper layer on top.

Read path: ImageFile resolves a read request by searching layers top-down via the segment index. The first layer containing a mapping for the requested block range serves the data. Unmapped ranges in upper layers fall through to lower layers.

Write path: All writes append to the upper layer. The upper layer’s index is updated in memory and flushed on sync.

Backends (pluggable via VirtualFile trait):

• LocalFile: io_uring pread/pwrite with optional O_DIRECT
• registryfs_v2: OCI registry (remote layer download)
• tar: tar archive reading
• Optional cache layer for decompressed block caching

Compression: zstd (level 3) with random-access jump tables and CRC32C checksums.

Snapshot: snapshot_runtime() copies upper layer data/index to a snapshot directory using reflink (CoW) where the filesystem supports it.

Key files: image_file.rs (high-level image), index_file.rs (LSMT stacking: LSMTReadOnlyFile, LSMTFile), format.rs (binary format), index.rs (segment mapping), zfile.rs (compression), snapshot.rs.

ublk (storage/ublk/)

Async userspace block device server using Linux’s ublk kernel driver. Exposes overlaybd images (or raw CoW files) as /dev/ublkbN block devices.

Device lifecycle:

1. UVMUblkCtrlBuilder sends ADD to /dev/ublk-control via io_uring UringCmd
2. Kernel allocates device ID, creates /dev/ublkcN (control) and /dev/ublkbN (block)
3. Per-queue worker threads start, each with a thread-local AsyncIoRing and slab-allocated I/O slots
4. Kernel dispatches block I/O to mmap’d ublksrv_io_desc arrays; userspace processes them asynchronously
5. delete_dev() tears down the device

Target implementations (UVMUblkTarget trait):

• OverlaybdTarget: wraps ImageFile for full layered image I/O
• BasicCowTarget: chunk-based copy-on-write over a read-only origin file. Per-chunk AtomicU8 state tracking (Origin -> Copying -> Cow) prevents duplicate copies. Coalesces adjacent reads.

I/O buffers: AutoRegBuffer (zero-copy via sparse buffer table, kernel 6.8+) or UserBuffer (traditional allocation).

Key files: lib.rs (public API), ctrl.rs (device controller), dev.rs (device + queue management), queue.rs (I/O descriptor handling), io_buffer.rs, impls/cow.rs, impls/overlaybd_target.rs.

ublk-daemon (storage/ublk-daemon/)

Long-running daemon process (uvm-ublk-daemon) that manages all ublk devices in one process and communicates with the AgentENV node over a Unix domain socket.

• Supports RPCs for OverlayBD runtime creation for sandbox rootfs/extra drives, raw OverlayBD/COW device creation for non-runtime callers, warm-pool acquire/release, resize capability queries, restack snapshot, delete, and shutdown.
• UblkDaemonClient spawns and monitors the daemon process from the node runtime.
• UblkDeviceManager (src/sandbox/ublk/device.rs) is the node-facing singleton that delegates lifecycle operations to the daemon client; device IDs are allocated in the daemon.

This separation keeps ublk device ownership and io_uring control in a dedicated process while the node server orchestrates lifecycle state.

storage-util (storage/util/)

Shared io_uring abstractions used by both ublk and overlaybd.

• AsyncIoRing<S>: generic async io_uring wrapper with slab-based RingFuture for CQE delivery. Supports standard (64B) and extended (128B) SQE types.
• IoRingWorker: spawns dedicated worker threads with thread-local io_uring instances. MPSC channel submission eliminates cross-thread locking.
• ReloadableIDAllocator: O(1) bitmap-based ID allocation/recycling with free list. Supports reloading pre-occupied IDs on restart.

Sandbox integration (src/sandbox/ublk/ + src/sandbox/extra_drive.rs)

• device.rs: owns the process-wide UblkDeviceManager, which talks to uvm-ublk-daemon and creates / deletes / snapshots all runtime ublk devices.
• overlaybd.rs: materializes runtime configs (rewrites paths, creates symlinks to layer files) for rootfs and attached drives.
• extra_drive.rs: prepares user-specified extra block drives with rollback on failure. Read-only and writable drives now follow the same per-sandbox device lifecycle; the only semantic difference is whether overlaybd materializes a writable upper.

Memory Snapshot Restore

Memory snapshot restore uses ublk-backed overlaybd devices rather than userfaultfd. On resume, a read-only ublk device is created from the stacked memory overlaybd layers and passed to Firecracker as a BackendType::File memory backend. Firecracker mmaps the block device and COWs pages into anonymous memory on first write, so the underlying device is never modified.

Sharing: Multiple sandboxes booting from the same snapshot template share a single memory ublk device via reference counting. This allows the Linux page cache to be reused across all sandboxes using the same memory image, significantly reducing I/O for concurrent launches from the same template.

Memory snapshot creation: On pause, Firecracker’s native diff snapshot (SnapshotType::Diff) produces a sparse mem.bin (using mincore internally to find present pages). This sparse file is then packaged into an overlaybd layer via convert_sparse_mem_to_overlaybd. Parent layers from previous snapshots are stacked, forming the full layered memory image.

Note: storage/uffd-core/ contains an alternative userfaultfd-based memory restore implementation that is retained for reference but excluded from the workspace build.

Per-Node Subsystems

Each node is an AgentENV server binary (src/bin/server.rs) on a Linux host with /dev/kvm.

Sandbox Networking

Sandbox networking is managed by a process-wide NetworkManager (src/sandbox/network/manager.rs) plus per-slot Slot objects (src/sandbox/network/slot.rs).

• Each slot owns a stable index-derived address bundle from [network.internal] (defaulting to 10.11.0.0/16 and 10.12.0.0/16) plus the fixed VM tap link 169.254.0.20/30, together with the host veth name, namespace path, and iptables rules for one sandbox network namespace.
• Network policy supports base allow/deny plus explicit egress rules. The /sandboxes/{sandboxID}/network endpoint replaces per-sandbox allowOut (CIDR/IP/domain patterns) and denyOut (CIDR/IP only) rules at runtime; allow rules always take precedence.
• allocate_any() first tries a warm-slot pool and falls back to creating a new namespace/veth/tap/iptables setup on demand.
• Warm-pool maintenance uses a single Condvar-driven background worker with low/high watermarks.
• release() enqueues slots back to the warm pool; when maintenance is enabled, even releases above high watermark are first enqueued and then drained asynchronously by the worker.
• [pool] provides shared watermarks and [pool.network].maintenance_enabled controls network worker behavior.
• Because the manager is a process-wide singleton, orchestrator shutdown explicitly calls NetworkManager::shutdown() after deleting remaining sandboxes so cached slots are drained and no new allocations race with teardown.
• Although calling NetworkManager::shutdown() on exit is recommended for clean teardown, the manager also has a Drop and libc::atexit handler to best-effort cleanup of any remaining namespaces and veth interfaces on unexpected shutdown and during testing.

Snapshot resume can also use [pool.firecracker] to pre-spawn (network slot, Firecracker process) pairs. A warm entry transfers its network slot, process, and Firecracker CWD to the resumed sandbox, which avoids the spawn and API-socket wait in the resume critical path. [pool.block] controls the ublk daemon’s overlaybd warm-device pool; it shares the same top-level watermarks but performs async refill from request paths because reusable block devices are image/size-specific.

Observability Data Flow

The node observability path combines request-time host collection with request-time projection:

• src/orchestrator/metrics.rs maintains incremental runtime counters during lifecycle operations, including running sandbox count, starting sandbox count, allocated CPU/memory, and create success/failure totals.
• src/orchestrator/service.rs publishes those counters through a tokio::sync::watch channel whenever lifecycle state changes affect the node’s runtime accounting.
• src/observability/identity.rs resolves stable node identity fields such as node ID, cluster ID, service instance ID, package version, and build-time commit.
• src/observability/machine.rs captures static machine descriptors from /proc/cpuinfo.
• src/observability/host.rs collects host CPU, memory, and disk usage each time a node snapshot is requested. CPU percent is derived from two /proc/stat samples; on the first request it takes both samples with a 100ms window to avoid returning a synthetic zero.
• src/observability/service.rs merges the latest orchestrator counters, identity, machine info, request-time host metrics, and current sandbox ID roster into a NodeSnapshot returned by the admin endpoints and reused by heartbeat reporting.
• src/observability/reporter.rs optionally sends periodic heartbeat reports to scheduler over gRPC (Heartbeat) and performs best-effort UnregisterNode on shutdown.
• Scheduler report config can be provided from TOML ([observability.scheduler_report]) and uses [cluster].scheduler_endpoint as the shared scheduler address. The reporter enable flag, address, and interval can be overridden by env vars (AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED, AENV_OBSERVABILITY_SCHEDULER_ENDPOINT, AENV_OBSERVABILITY_REPORT_INTERVAL_SECS).
• If a P2P transport exposes a local endpoint, the reporter includes it in the scheduler heartbeat so other nodes can discover it.

This keeps node requests lightweight on orchestrator data: they avoid re-listing and sorting all sandboxes on every API call while still returning fresh host metrics.

The observability subsystem has two configuration-controlled scopes:

• observability.enabled: controls whether the node observability service is constructed at all. When disabled, node/admin observability endpoints degrade rather than trying to synthesize partial snapshots.
• observability.scheduler_report.enabled: controls optional scheduler heartbeat reporting. It can be overridden by AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED. When enabled, reporting requires [cluster].scheduler_endpoint or AENV_OBSERVABILITY_SCHEDULER_ENDPOINT.

P2P Artifact Transport

src/p2p/ provides a project-wide artifact transport abstraction for modules that need to exchange validated files between runtime nodes. Consumers depend on the P2pTransport trait, whose main operations are lookup, lookup_with_hints, fetch, publish, unpublish, local_endpoint, and shutdown.

The default DisabledP2pTransport keeps the feature inert: lookups return no descriptor, publish is a no-op, and fetch fails with TransportDisabled. The IrohBlobsP2pTransport backend starts an embedded iroh endpoint, serves artifact bytes through iroh-blobs, and serves a small AgentENV catalog protocol over the same endpoint to map stable artifact keys to transport-neutral descriptors.

One P2P artifact key represents one logical artifact. Lookup returns at most one descriptor, selected from the local catalog first and then from discovered peers in order. Artifact descriptors contain the stable key, provider node ID, optional provider endpoint, backend-specific locator string, and module-defined JSON metadata. Backend locators stay opaque to callers; for iroh the locator is the iroh-blobs hash used for content-addressed fetch.

A successful remote fetch also best-effort advertises the fetched blob from the local node. This makes the fetching node a provider for later peers and lets artifacts spread through the cluster.

Peer discovery is decoupled behind P2pPeerDiscovery. In normal multi-node deployments, SchedulerPeerDiscovery periodically calls scheduler ListP2pPeers, filters by backend and cluster, and excludes the local node. StaticP2pPeerDiscovery and NoopP2pPeerDiscovery cover tests and disabled/local-only operation.

Snapshot publishing also uses the P2P layer as a best-effort acceleration path. After a snapshot repository commit succeeds, SnapshotManager advertises the fixed Firecracker artifacts and overlaybd layers. OSS-backed snapshot resolution tries P2P before object storage for fixed artifacts; POSIX-backed resolution does not consume P2P because the POSIX repository path is already the committed artifact source. Overlaybd layer reads are accelerated by the overlaybd P2P HTTP facade rather than by snapshot resolvers.

See P2P Artifact Transport for the detailed design.

Node API endpoints (E2B-compatible):

• POST /sandboxes create a sandbox
• GET /sandboxes list sandboxes
• GET /sandboxes/{id} get sandbox metadata
• DELETE /sandboxes/{id} delete a sandbox
• POST /sandboxes/{id}/pause pause (snapshot) a sandbox
• POST /sandboxes/{id}/resume resume from snapshot
• GET /nodes return node-level observability snapshots
• GET /nodes/{id} return node details plus currently running sandboxes
• ANY /proxy, ANY /proxy/{path}, routing-header fallback, and configured sandbox proxy hosts reverse proxy to sandbox services

Distributed Control Plane (prototype)

The multi-node control plane in services/ is a prototype. It routes client traffic across multiple AgentENV backend nodes.

    Client ──HTTP──> Gateway (:8080) ──gRPC──> Scheduler (:9090)
                        │                          │
                        │    ┌─────────────────────┘
                        │    │ node selection / lookup
                        ▼    ▼
                   Node A (:8000)    Node B (:8000)

Gateway (services/gateway/): HTTP reverse proxy. Extracts sandbox data-plane routes from headers (x-agentenv-sandbox-id / e2b-sandbox-id) or configured host-based proxy domains ({port}-{sandboxID}.{domain}). Host-based routes are only enabled for explicit gateway.sandbox_proxy_domains entries, require RFC 952/1123 DNS-label-compatible sandbox IDs, and require the full {port}-{sandboxID} label to fit the 63-character DNS label limit. Runtime nodes have their own [sandbox_proxy].domains setting for the same host-based URL shape and return the first configured domain in sandbox metadata. In multi-node deployments, repository helpers can apply one SANDBOX_PROXY_DOMAINS value to both gateway and runtime node configuration. Sandbox control-plane routes such as /sandboxes/{id}/pause are routed by sandbox ID from the URL path; sandbox data-plane traffic is not inferred from URL path alone. For new sandboxes, calls Schedule() to pick a node. For existing sandboxes, calls LookupNode(). After sandbox creation, calls RecordAssignment() to seed a sandbox-to-node binding. Without explicit routing headers, it also handles cluster aggregation of GET /sandboxes, GET /v2/sandboxes, GET /nodes, and resolves GET /nodes/{id} via scheduler before proxying to the resolved node.

Scheduler (services/scheduler/): gRPC service with pluggable node discovery and in-memory sandbox-to-node bindings, plus observed-node snapshots reported by runtime nodes. RPCs include Schedule, LookupNode, RecordAssignment, Heartbeat, ListObservedNodes, ListP2pPeers, GetNode, and UnregisterNode. Strategies: round_robin (default), random. Proto contract: services/api/proto/scheduler.proto. For P2P, scheduler stores and returns opaque peer endpoints from heartbeat records; artifact catalog lookup and byte transfer stay node-to-node.

Binding lifecycle:

• RecordAssignment creates the initial binding immediately after sandbox creation succeeds.
• Runtime heartbeats include the node’s full sandbox ID roster. Scheduler treats that roster as the source of truth for that node and removes bindings missing from the latest heartbeat.
• binding_ttl is a freshness TTL for routing information, not a copy of sandbox timeout. If a binding stops being refreshed by gateway or heartbeats, scheduler drops it on the next lookup or roster reconcile.
• UnregisterNode removes the observed node record and proactively clears bindings owned by that node.

Discovery modes:

• static: explicit scheduler.nodes list from config
• kubernetes: EndpointSlice watch over the headless agentenv-nodes Service, using ready DaemonSet Pod IPs as backend endpoints

Limitations: All bindings are in-memory (lost on scheduler restart). After a scheduler restart, bindings are rebuilt from new sandbox creations plus the next heartbeat roster from each runtime node. Kubernetes discovery updates the schedulable node set dynamically, but binding persistence is still not replicated.

Deployment:

# local dev (single node)
make start-server && make -C services run-scheduler && make -C services run-gateway

# docker compose (multi-node)
make deploy-up     # gateway + scheduler + 2 backend nodes
make deploy-down   # teardown

# kubernetes (gateway + scheduler + daemonset runtime nodes)
make k8s-render
make k8s-apply

In Kubernetes deployments, AgentENV runtime nodes run as a privileged DaemonSet so each host gets exactly one runtime Pod with access to /dev/kvm, iptables/network-namespace operations, and a hostPath-backed workspace cache. The deployment helpers materialize the DaemonSet ConfigMap from config/default.toml at render/apply time so AgentENV runtime config remains single-sourced.

Directory Structure

storage/
├── overlaybd/src/              # layered image format (core)
│   ├── image_file.rs           # high-level image abstraction
│   ├── index_file.rs           # LSMT layer stacking
│   ├── format.rs               # binary format (HeaderTrailer, DiskSegmentMapping)
│   ├── index.rs                # segment mapping
│   ├── zfile.rs                # zstd compression + jump tables
│   ├── snapshot.rs             # snapshot capture
│   ├── local.rs                # LocalFile backend (io_uring)
│   └── registryfs_v2.rs        # OCI registry backend
├── ublk/src/                   # userspace block device server
│   ├── lib.rs                  # public API
│   ├── ctrl.rs                 # /dev/ublk-control interface
│   ├── dev.rs                  # device + queue management
│   ├── queue.rs                # I/O descriptor handling
│   ├── io_buffer.rs            # zero-copy + traditional buffers
│   └── impls/                  # target implementations
│       ├── cow.rs              # BasicCowTarget
│       └── overlaybd_target.rs # OverlaybdTarget
├── ublk-daemon/src/            # ublk daemon (unix socket RPC)
│   ├── client.rs               # daemon client used by node runtime
│   ├── server.rs               # daemon server + request loop
│   └── protocol.rs             # RPC message types
├── util/src/                   # shared io_uring abstractions
│   ├── io_ring/                # AsyncIoRing, IoRingWorker
│   └── id_allocator.rs         # bitmap-based ID allocation
└── uffd-core/src/              # userfaultfd memory restore (excluded from workspace, retained for reference)
    ├── handler.rs              # UffdHandle, page fault event loop
    ├── backend.rs              # MemoryImageBackend trait
    ├── overlaybd.rs            # OverlaybdMemoryImage backend
    ├── process_vm_reader.rs    # ProcessVmReader (process_vm_readv)
    └── scm.rs                  # SCM_RIGHTS fd passing

src/
├── bin/server.rs               # node binary entrypoint
├── api/                        # HTTP API layer
├── orchestrator/               # sandbox lifecycle
├── observability/              # node identity + host/runtime metrics projection
├── sandbox/                    # Firecracker VM management
│   ├── extra_drive.rs          # extra drive preparation
│   └── ublk/                   # storage integration
│       ├── device.rs           # daemon-backed ublk device lifecycle
│       └── overlaybd.rs        # runtime config materialization
├── snapshot/                   # committed snapshot model, repository backends, runtime resolution
├── template/                   # user-facing template builder over snapshots
└── cfg.rs                      # TOML config

services/                       # prototype distributed control plane (Go)
├── gateway/                    # HTTP reverse proxy
├── scheduler/                  # gRPC node selection + binding
├── api/proto/                  # protobuf contracts
└── shared/                     # config, logging

Sandbox and Test Flow Guide

This document describes the public sandbox API, the global ConfigManager under AgentENV/src/cfg.rs, and what AgentENV/tests/integration/fc.rs, AgentENV/tests/integration/process.rs, AgentENV/tests/integration/snapshot.rs, and AgentENV/tests/integration/orchestrator.rs validate.

1) Sandbox Public Interface

Types and responsibilities

• FirecrackerSandboxConfig (AgentENV/src/sandbox/firecracker/config.rs)

  • Purpose: describes a fresh VM boot (kernel + tools drive + user image).
  • Key fields
    • firecracker_binary: path to the firecracker executable.
    • kernel_image: path to vmlinux.bin.
    • tools_drive_version: immutable tools drive identity. The node resolves it to <deps_path>/tools/<version>/tools.ext4 before mounting it as /dev/vda.
    • user_image_config: overlaybd config for the writable user image (/dev/vdb).
    • boot_args: kernel boot args (e.g. console=ttyS0 ... init=/init).
    • vcpu_count, mem_size_mib: VM size.
    • runtime_policy: socket/envd timeouts and poll intervals.
    • firecracker_stdout_path, firecracker_stderr_path: capture FC logs. If unset, logs default to work_dir/logs/firecracker-stdout.log and work_dir/logs/firecracker-stderr.log.
    • envd_version: expected envd version for the guest image.
    • env_vars: optional default environment variables injected after envd init.
    • ublk_config: optional ublk-backed rootfs configuration.
  • Convenience
    • from_global_config() builds a config from ConfigManager using the internal default template rootfs image. The image must already be resolved under <image-cache-root>/configs.
    • from_global_config_with_user_image(...) builds a config from an already resolved overlaybd image config.
    • Runtime launches from committed snapshots are built through FirecrackerSandbox::from_snapshot(&RunnableSnapshot, &SandboxLaunchConfig).
    • uid/gid: File ownership (default 0).

• ProcessOpts (AgentENV/src/sandbox/process.rs)

  • Purpose: Options for starting a process inside the sandbox.
  • Key fields
    • envs: HashMap<String, String> for process environment variables.
    • cwd: Optional working directory.
    • timeout: Optional max time to wait for completion.
  • Builder methods: with_envs(), with_cwd(), with_timeout().

• ProcessOutput (AgentENV/src/sandbox/process.rs)

  • Purpose: Result of a completed process execution.
  • Fields: stdout, stderr, exit_code.

• ProcessHandle (AgentENV/src/sandbox/process.rs)

  • Purpose: Handle to a running process inside the sandbox.
  • Key methods
    • pid(): Returns the PID inside the guest VM.
    • wait().await: Waits for exit and collects all output.
    • send_stdin(data).await: Sends bytes to stdin.
    • send_signal(signal).await: Sends a signal (e.g. SIGTERM).
    • kill().await: Kills the process with SIGKILL.

• FirecrackerSnapshotConfig (AgentENV/src/sandbox/firecracker/config.rs)

  • Purpose: describes how to resume a VM from snapshot + memory + base disk.
  • Key fields
    • vm_state_path: Firecracker VM state file.
    • mem_overlaybd_config: runtime memory overlay image config.
    • base_rootfs_path: writable disk image or overlaybd rootfs image config paired with that snapshot.
    • firecracker_binary: FC binary to use for resuming.
    • runtime_policy: socket/envd timeouts and poll intervals.
    • envd_version: guest envd version metadata.
    • env_vars: optional default environment variables restored on start.
  • Notes
    • FirecrackerSnapshotConfig returned by pause() owns a tempdir (keeps files alive).
    • get_rootfs_path() resolves the snapshot rootfs path used by runtime tests.

• SandboxBackend and SandboxExecutor (AgentENV/src/sandbox/backend.rs)

  • Purpose: public traits for sandbox lifecycle and process execution.
  • Key methods
    • SandboxBackend: start, start_nowait, wait_for_ready, pause, resume, stop.
    • SandboxExecutor: run_command, run_command_with_opts, start_process.

• FirecrackerSandbox (AgentENV/src/sandbox/firecracker/sandbox.rs)

  • Purpose: lifecycle controller for a single Firecracker instance.
  • Main methods (detailed)
    • FirecrackerSandbox::new(FirecrackerSandboxConfig)
      • Creates a sandbox handle for a fresh boot.
      • Does not start Firecracker; it only prepares the object.
      • Each sandbox gets its own temporary work directory.
    • FirecrackerSandbox::resume_from_snapshot_config(&FirecrackerSnapshotConfig)
      • Creates a new sandbox handle and immediately starts it from a snapshot config.
      • Uses vm_state_path, the memory overlay image config, and base_rootfs_path from the config.
      • Returns a running sandbox after Firecracker and envd are ready.
    • FirecrackerSandbox::from_snapshot(&RunnableSnapshot, &SandboxLaunchConfig)
      • Creates a sandbox handle from committed snapshot state resolved for a node.
      • Uses snapshot vm_state.bin plus node-local materialized memory/image.json and rootfs/image.json.
      • Does not start Firecracker; call start().await afterward.
    • start().await
      • Spawns the Firecracker process, waits for the API socket, starts the VM, then waits for guest-level envd readiness.
    • pause().await
      • Sends PATCH /vm to pause the VM.
      • Creates snapshot artifacts including vm_state.bin, mem.bin, the packaged memory image config, and rootfs snapshot state.
      • Returns a FirecrackerSnapshotConfig that owns the tempdir holding these files.
    • resume().await
      • Resumes a paused VM in-place by sending PATCH /vm with Resumed.
      • Use this when you want to keep the same sandbox instance.
    • stop().await
      • Stops Firecracker, releases network resources, and cleans up optional ublk state.
    • run_command(cmd, args).await
      • Runs a command inside the sandbox via envd gRPC and waits for it to complete. Returns ProcessOutput with stdout, stderr, and exit code.
    • run_command_with_opts(cmd, args, opts).await
      • Same as run_command but accepts ProcessOpts for setting env vars, working directory, and execution timeout.
    • start_process(cmd, args, opts).await
      • Starts a long-running process inside the sandbox. Returns a ProcessHandle that can stream output, send stdin, or kill the process.
    • firecracker_stdout_path() / firecracker_stderr_path()
      • Resolved log paths used by this sandbox (including defaults).
    • work_rootfs_path()
      • Returns the path to the per-instance writable rootfs inside the work directory. Useful for inspecting or copying disk state after a run.

What happens under the hood (important for correct usage)

• Each FirecrackerSandbox uses its own temp work directory.
• Kernel is used via the configured host path.
• When resuming, the writable rootfs and memory image are materialized into the work dir so each instance can mutate independently. For overlaybd-backed snapshot-backed template launches, this means recreating runtime image configs and fresh writable uppers from the committed layer stack.
• Rootfs copies use copy-on-write helpers where available.
• Firecracker runs with current_dir = work_dir, so relative paths work.
• Firecracker logs default to work_dir/logs/* unless explicit log paths are configured.
• start() waits for both the API socket and envd readiness.
• Although the Firecracker process will be killed on drop, explicit stop() is recommended.

Minimal usage example

#![allow(unused)]
fn main() {
use agentenv::sandbox::{FirecrackerSandbox, FirecrackerSandboxConfig, SandboxExecutor};

let mut cfg = FirecrackerSandboxConfig::new(
    "/path/to/firecracker".into(),
    "/path/to/vmlinux.bin".into(),
    "/path/to/rootfs.ext4".into(),
);
cfg.boot_args = Some("console=ttyS0 reboot=k panic=1 pci=off init=/init".into());

let mut sandbox = FirecrackerSandbox::new(cfg)?;
sandbox.start().await?;

// Run a command inside the VM
let output = sandbox.run_command("echo", &["hello"]).await?;
assert_eq!(output.exit_code, 0);
println!("{}", output.stdout);

let snapshot = sandbox.pause().await?;
sandbox.stop().await?;

let mut resumed_sandbox = FirecrackerSandbox::resume_from_snapshot_config(&snapshot).await?;
resumed_sandbox.stop().await?;
}

Runtime prerequisites

• Linux host
• /dev/kvm accessible by current user
• debugfs (e2fsprogs) if you use init injection or disk-inspection helpers

2) Global Config Manager (src/cfg.rs)

The tests and benchmarks use a shared global config manager based on config/default.toml.

Config file format

AgentENV/config/default.toml (override via AENV_CONFIG_PATH or --config)

[firecracker]
boot_args = "your-kernel-boot-args"
binary_path = "/path/to/firecracker"
socket_timeout_secs = 3
socket_poll_ms = 20
# Optional override; defaults to $AENV_HOME/firecracker-work.
# work_dir = "/path/to/firecracker-work"
# Optional override; defaults to $AENV_HOME/logs/serial.
# serial_dir = "/path/to/serial"
# Optional; enables Firecracker's own logging when set to a non-empty level.
# log_level = "Info"

[kernel]
image_path = "/path/to/vmlinux.bin"

[tools]
version = "0.1.0-custom.1"
drive_path = "/path/to/tools.ext4"

[machine]
mem_size_mib = 128
vcpu_count = 1

[envd]
version = "0.5.15"
init_timeout_secs = 30
poll_ms = 10

[orchestrator]
auto_evict_interval_ms = 1000
default_sandbox_timeout_secs = 15

[snapshot]
repository_backend = "posix_fs"
local_cache_path = "$AENV_HOME/snapshot-local-cache"

[backend.posix_fs]
snapshot_store = "$AENV_HOME/snapshot-store"

[ublk]
enabled = false

TOML parameter reference

• [firecracker]

  • boot_args: Kernel command line. The runtime appends init=/init if not already present (tools drive provides the init script).
  • binary_path: Absolute or relative path to firecracker.
  • socket_timeout_secs: Max time to wait for the Firecracker API socket.
  • socket_poll_ms: Poll interval for checking socket existence.
  • work_dir: Optional override for the per-sandbox Firecracker working directories. These directories include runtime sockets, symlinks, local logs, and writable OverlayBD upper layer data (overlaybd/upper.data and overlaybd/upper.index). Defaults to $AENV_HOME/firecracker-work.
  • serial_dir: Optional override for the persistent output directory. Serial output is written under a per-sandbox subdirectory and not removed by the sandbox. Defaults to $AENV_HOME/logs/serial.
  • log_level: Optional Firecracker log level (Error, Warning, Info, Debug, Trace, case-insensitive). When set to a non-empty value, Firecracker’s own logging is enabled and written to a firecracker.log file in the sandbox’s log directory (the same directory used for serial output). If omitted or empty, Firecracker logging is disabled.

• [kernel]

  • image_path: Optional local kernel image path (vmlinux.bin). If omitted, AgentENV derives it from deps_path and the bundled dependency manifest.

• [tools]

  • drive_path: Optional local tools drive source. When set, an explicit version is required and setup imports it into the immutable version directory.

• Template rootfs image

  • User-visible rootfs images are selected when starting template builds through the optional fromImage field.
  • If fromImage is omitted, AgentENV uses [image.resolver].default_image.
  • Standard Docker Hub shortnames such as ubuntu:24.04 and node:20 are normalized before image resolution.
  • Resolved image configs are cached under <image-cache-root>/configs/<slug>-<hash>-image.json; local layer products are cached under <image-cache-root>/layers/; overlaybd-native remote block reads are cached under <image-cache-root>/remote-blocks/.
  • Registry authentication uses docker credentials from ~/.docker/config.json or $DOCKER_CONFIG/config.json.

• [machine]

  • mem_size_mib: Guest RAM size (default: 128).
  • vcpu_count: Number of vCPUs (default: 1).

• [envd]

  • version: Expected envd version baked into the runtime image.
  • init_timeout_secs: Max time to wait for the in-guest envd daemon to become ready after VM start (default: 30).
  • poll_ms: Poll interval for envd health-check retries (default: 10).

• [orchestrator]

  • auto_evict_interval_ms: Poll interval for background timeout eviction.
  • default_sandbox_timeout_secs: Default keep-alive timeout used by the orchestrator.
  • auto_resume_min_sandbox_timeout_secs: When a data-plane request targets a non-running sandbox, automatically resume it (if auto-resume is enabled) and refresh its timeout for no-less than this duration.

• [pool]

  • low_watermark: Shared lower bound for warm resources. Defaults to 2.
  • high_watermark: Shared upper bound target for warm resources. Defaults to 64.
  • [pool.network].maintenance_enabled: Enables the background network-slot maintenance worker. Defaults to true.
  • [pool.block].enabled: Enables the ublk overlaybd warm-device pool.
  • [pool.firecracker].enabled: Enables pre-spawned Firecracker processes for snapshot resume.

• [snapshot]

  • local_cache_path: Manager-owned node-local snapshot artifact/cache root. Defaults to $AENV_HOME/snapshot-local-cache.
  • repository_backend: Snapshot repository backend. Defaults to posix_fs.

• [backend.posix_fs]

  • snapshot_store: Committed snapshot store root. Defaults to $AENV_HOME/snapshot-store.

• [ublk]

  • enabled: Enables ublk-backed rootfs handling.
  • daemon_binary_path: Optional path to uvm-ublk-daemon.
  • daemon_socket_path: Optional unix socket path for daemon RPCs.
  • daemon_log_path: Optional log file path for the daemon process.
  • device_type: cow or overlaybd.

• [ublk.overlaybd]

  • global_config_path: path to the generated overlaybd runtime config. Required for overlaybd device_type. Per-image configs are derived from template build fromImage values and live under <image.cache.root_dir>/configs.

Defaults and optional fields

• [firecracker].binary_path: Optional. If omitted, AgentENV derives it from deps_path and the bundled dependency manifest.
• [firecracker].boot_args: Optional. Defaults to console=ttyS0 reboot=k panic=1 pci=off.
• [firecracker].socket_timeout_secs: Optional. Defaults to 3 seconds.
• [firecracker].socket_poll_ms: Optional. Defaults to 20 ms.
• [firecracker].work_dir: Optional override. Defaults to $AENV_HOME/firecracker-work.
• [firecracker].serial_dir: Optional override. Defaults to $AENV_HOME/logs/serial.
• [kernel].image_path: Optional. If omitted, AgentENV derives it from deps_path and the bundled dependency manifest.
• [tools].drive_path: Optional local source override; requires an explicit [tools].version when set.
• [tools].url: Optional registry override; requires an explicit [tools].version when set.
• [machine]: Optional. If omitted, defaults are 128 MiB RAM and 1 vCPU.
• [envd]: Optional, but template and sandbox flows expect a valid version.
• [envd].init_timeout_secs: Optional. Defaults to 30 seconds.
• [envd].poll_ms: Optional. Defaults to 10 ms.
• [orchestrator]: Optional. If omitted, orchestrator runtime defaults are used.
• [pool]: Optional. If omitted, network pool maintenance is enabled with low watermark 2 and high watermark 64; block and Firecracker process pools are disabled unless enabled in their component subsections.
• [snapshot]: Optional. If omitted, repository_backend defaults to posix_fs and the local cache root derives from AENV_HOME.
• [backend.posix_fs]: Optional. If omitted, the POSIX snapshot store defaults to $AENV_HOME/snapshot-store.
• [ublk]: Optional. If omitted, ublk is disabled.

Environment-based configuration

The config loader reads the full TOML file from the following sources:

• AENV_CONFIG_PATH
• --config <path>
• config/default.toml

Runtime image and binary paths are configured in the TOML file itself.

Required fields

• [kernel].image_path
• [tools].version when [tools].drive_path or [tools].url is overridden
• [envd].version

If the [orchestrator] section is present, it must include default_sandbox_timeout_secs.

ConfigManager (recommended entry point)

The helper is strict. If requirements are not met (missing files, invalid config), it returns an error and the test fails.

Path resolution order

1. AENV_CONFIG_PATH environment variable
2. --config CLI flag
3. config/default.toml (default)

Helper methods

• ConfigManager::global() -> Result<&'static ConfigManager>
  • Lazily initializes and returns a process-wide singleton.
• ConfigManager::new() -> Result<ConfigManager>
  • Loads config using the default resolution order.
• ConfigManager::new_from_path(path) -> Result<ConfigManager>
  • Loads a specific TOML file.
• ConfigManager::config() -> &AppConfig
  • Returns the parsed application config.
• ConfigManager::global_config() -> Result<&'static AppConfig>
  • Returns the global parsed config directly.
• ConfigManager::get_orchestrator_config() -> Option<OrchestratorConfig>
  • Returns orchestrator settings when configured.
• AppConfig::resolved_snapshot_store() -> PathBuf
  • Resolves the effective committed snapshot store root.
• AppConfig::resolved_snapshot_local_cache_path() -> PathBuf
  • Resolves the effective manager-owned node-local snapshot cache root.

Typical test setup flow

#![allow(unused)]
fn main() {
use agentenv::sandbox::{FirecrackerSandbox, FirecrackerSandboxConfig, SandboxExecutor};

let sandbox_config = FirecrackerSandboxConfig::from_global_config()?;

let mut sandbox = FirecrackerSandbox::new(sandbox_config)?;
sandbox.start().await?;

// Run commands via envd gRPC
let output = sandbox.run_command("echo", &["hello"]).await?;
assert_eq!(output.exit_code, 0);

let snapshot = sandbox.pause().await?;
sandbox.stop().await?;

let mut resumed_sandbox = FirecrackerSandbox::resume_from_snapshot_config(&snapshot).await?;
resumed_sandbox.stop().await?;
}

3) What the Integration Tests Validate

integration/fc.rs

• microvm_startup_moves_to_running Boots a fresh VM and verifies that:

  • Firecracker starts and accepts API calls.
  • Kernel + rootfs can be configured without error.
  • The VM can be cleanly shut down afterward.

• microvm_pause_transitions_to_paused Validates pause + snapshot creation by:

  • Pausing a running VM via FC API.
  • Creating vm_state.bin and mem.bin.
  • Ensuring those files exist on disk.

• microvm_resume_from_pause_transitions_to_running Validates in-place resume by:

  • Pausing a VM.
  • Resuming it in the same sandbox instance.
  • Confirming the control flow completes without error.

• microvm_resume_transitions_to_running Validates resume into a new sandbox by:

  • Pausing and snapshotting a VM.
  • Shutting down the original sandbox.
  • Resuming from the snapshot in a new sandbox instance.

• snapshot_preserves_disk_state_on_resume Confirms disk state is preserved by:

  • Writing a marker file via run_command.
  • Pausing and resuming from the snapshot.
  • Reading the marker file after resume and asserting it persists.

• snapshot_chain_survives_after_parent_snapshot_handle_is_dropped Validates multi-level snapshot chains by:

  • Creating a snapshot, resuming, creating a second snapshot.
  • Dropping the first snapshot handle.
  • Resuming from the second snapshot and verifying disk state.
  • Also tests pause_to_dir with snapshot-owned inherited layer adoption.

• microvm_can_access_internet Confirms guest networking works before and after snapshot resume.

• multiple_resumes_have_independent_disk_state Confirms multiple resumptions are isolated by:

  • Resuming multiple VMs from a single snapshot sequentially.
  • Each resumed VM verifies the shared marker and writes its own file.
  • Verifying each snapshot has independent state.

integration/process.rs

• run_command_captures_stdout Runs echo hello world and verifies stdout contains the expected string.

• run_command_captures_stderr Runs a shell command that writes to stderr and verifies stderr capture.

• run_command_reports_exit_code Runs exit 42 and verifies the exit code is reported correctly.

• run_command_with_opts_sets_env_and_cwd Verifies that environment variables and working directory options are honoured.

• run_command_handles_timeout Verifies long-running commands fail when a timeout is configured.

• run_command_with_unbounded_output Verifies excessive output is rejected instead of buffering forever.

• start_process_interactive_stdin Starts cat, sends data via stdin, kills the process, and verifies the echoed output.

• start_process_send_signal Starts sleep 300, sends SIGTERM, and verifies the process terminates with a non-zero exit code.

• run_command_after_snapshot_resume Pauses a sandbox, resumes from a snapshot config, and verifies run_command works on the resumed instance.

integration/snapshot.rs

• built_snapshot_can_be_loaded_by_alias_and_launched Builds a template-backed snapshot, loads it by alias, launches a sandbox from the resolved runnable snapshot, and verifies captured runtime state.

• listed_committed_snapshot_can_be_loaded_and_resolved Verifies SnapshotManager::list_committed() returns usable committed snapshots that can be loaded and resolved into runnable paths.

• loaded_runnable_snapshot_by_alias_can_be_resolved Verifies SnapshotManager::load_runnable() resolves alias-based lookups into runnable snapshot state in one step.

• delete_committed_snapshot_by_alias_removes_it_from_repository Verifies deleting a committed snapshot by alias removes the durable record.

• snapshot_rebuild_from_committed_snapshot_preserves_base_state Verifies rebuilding from a committed snapshot preserves the base filesystem state while adding new changes.

• rebuild_merges_env_metadata_and_list_filter_can_select_by_id Verifies rebuilds merge environment metadata correctly and that committed snapshot listing can filter by snapshot id.

integration/orchestrator.rs

• orchestrator_lifecycle Verifies the orchestrator can create, fetch, list, filter, keep alive, and delete template-backed sandboxes while keeping proxy lookup state in sync.

4) End-to-End Execution Checklist

1. Provide Firecracker binary, kernel, tools drive, overlaybd runtime, and ublk daemon. These are automatically downloaded when the server starts, or you can run cargo run --bin server -- --setup-only to provision runtime dependencies independently.
2. Provision host access once as root with server --setup-host --runtime-user <user> --runtime-group <group>. The group is the runtime service group: it owns AgentENV state and receives ublk device access. Normal server startup performs validation only and never invokes sudo.
3. Host setup installs a udev rule for /dev/ublk-control, /dev/ublkc*, and /dev/ublkb*, so the runtime group can access the control and dynamic device nodes.
4. Update config/default.toml paths, or point AENV_CONFIG_PATH to a custom config file.
5. Ensure /dev/kvm is accessible by the runtime user.
6. If you run template tests, ensure the host can run regctl (server setup installs it automatically from the [regclient] manifest entry) and access the registry for template fromImage resolution.
7. Run scripts/tests/e2e/run_e2e.sh for API-level E2E coverage. The runner exports E2E_TEMPLATE_USER_IMAGE. Suite 05_template_lifecycle.sh also creates a template build with E2E_SHORT_USER_IMAGE to verify short-name image resolution.
8. Run make test-agent-integration to run the agentenv integration test modules in tests/integration/ as a non-root user with the required capabilities, plus the Docker/MinIO-backed OSS snapshot repository test (tests/snapshot_oss_e2e_test.rs).
9. Run make test-ublk to run the uvm-ublk and overlaybd storage tests, including the Docker/MinIO-backed OSS backend test (storage/overlaybd/tests/oss_backend_minio.rs).

4.1) Firecracker Client and Runtime Upgrade Checklist

When bumping AgentENV to a patched Firecracker runtime, update both the generated client API and the binary that setup downloads:

1. Replace thirdparty/firecracker-client/firecracker.yaml with the target Firecracker OpenAPI spec.

2. Run make firecracker-client from the repository root and review generated changes under thirdparty/firecracker-client/.

3. Build the patched Firecracker release binary on a Linux host. Package it as a gzip tar archive named firecracker-{version}-{arch}.tgz containing a firecracker executable.

4. Update the [firecracker] entry in config/deps_manifest.toml with the new version and a download URL template that supports {version} and {arch}.

5. Before relying on the URL in tests, verify it is a direct download:

   curl -L "<url>" -o /tmp/firecracker.tgz
   tar -tzf /tmp/firecracker.tgz
   

   The archive listing should include firecracker.

6. Run setup against a clean or temporary Firecracker dependency directory so it really downloads the archive:

   cargo run --bin server -- --config /path/to/config.toml --setup-only
   

7. Validate the Rust side after code generation:

   cargo fmt --check -p firecracker_client -p agentenv
   cargo check -p firecracker_client
   cargo check -p agentenv
   cargo test -p agentenv --lib
   

5) Running Custom Scripts

Run commands via envd

Once the sandbox is started (with a rootfs that includes the envd daemon), you can execute commands inside the VM using the process API:

#![allow(unused)]
fn main() {
// Simple command
let output = sandbox.run_command("ls", &["-la", "/tmp"]).await?;
println!("exit={} stdout={}", output.exit_code, output.stdout);

// Command with options
use agentenv::sandbox::ProcessOpts;
let opts = ProcessOpts::new()
    .with_cwd("/tmp")
    .with_envs([("KEY".into(), "value".into())].into());
let output = sandbox.run_command_with_opts("env", &[], opts).await?;

// Interactive / long-running process
let mut handle = sandbox.start_process("cat", &[], ProcessOpts::default()).await?;
handle.send_stdin(b"hello\n").await?;
handle.kill().await?;
let output = handle.wait().await?;
}

The process API uses envd’s gRPC ProcessClient under the hood. It requires the envd daemon to be running inside the guest VM.

Run different scripts for multiple snapshot resumes

/init is not re-executed after resume, so per-instance behavior must come from per-instance disk contents.

A typical pattern:

1. Start a base VM and pause() to create a snapshot.
2. For each instance, copy the snapshot rootfs and write a per-instance file into it (for example using debugfs).
3. Create a FirecrackerSnapshotConfig that points to that per-instance rootfs and resume.

The guest must keep a long-running dispatcher (started during the initial boot) that watches for these per-instance files and executes them after resume.

Template Builder and Testing

This document explains how the current template-facing build API maps onto the snapshot-first internals, how committed snapshots move through the repository/runtime boundary, and what the relevant tests validate.

1) Public Surface vs Internal Model

User-facing builder API

• TemplateBuildSpec (src/template/build_spec.rs)

  • Describes a declarative build request.
  • Builder helpers include:
    • from_existing_rootfs(path)
    • from_overlaybd_configs(global_config_path, image_config_path)
    • run(cmd)
    • env(key, value)
    • workdir(path)
    • apt(packages)
    • alias(alias)
    • resources(cpu_count, memory_mib)
  • The current builder implementation accepts overlaybd-backed fresh builds and rebuilds from committed snapshots. Fresh ext4-rootfs builds are not wired through TemplateBuilder.

• TemplateBuilder (src/template/builder.rs)

  • Preserves the external template semantics for build / rebuild flows, while committed snapshot lifecycle operations live in SnapshotManager.
  • Main methods:
    • new()
    • with_local_store_root(path)
    • build_and_publish(snapshot_manager, config).await
    • rebuild_and_publish(snapshot_manager, config, snapshot_id, base_snapshot).await

Snapshot-first internals

• StoredSnapshot / RunnableSnapshot (src/snapshot/types/)

  • StoredSnapshot is the durable committed manifest state.
  • It stores snapshot metadata plus logical rootfs / attached-drive layer descriptions and memory layers.
  • Snapshot publication also persists firecracker-manifest.json for launch-time metadata such as rootfs/memory virtual size and attached-drive read_only flags.
  • It does not store fixed local artifact locations such as mem_image.json or runtime-derived rootfs/image.json.
  • RunnableSnapshot is a node-local resolved view with concrete artifact paths and runtime-ready overlaybd image configs.

• SnapshotRepository / SnapshotRuntimeResolver (src/snapshot/repository/interfaces.rs)

  • SnapshotRepository owns committed durable state.
  • SnapshotRuntimeResolver turns committed state into node-local runnable paths.

2) Build and Publish Flow

TemplateBuilder::build_and_publish(...) does the following:

1. Prepare the build base from either:
   • a committed snapshot (rebuild_and_publish)
   • overlaybd configs (from_overlaybd_configs)
2. Start a temporary Firecracker sandbox with the requested CPU/memory.
3. Execute template steps in order.
4. Probe envd, kernel, and firecracker versions from the running guest and executable.
5. Pause the sandbox and export a FirecrackerSnapshotManifest carrying metadata for repository publication
6. Publish those local artifacts into the configured snapshot repository.

The important boundary is:

• the builder API describes what to build
• local build artifacts are manager-owned temporary outputs
• committed snapshot records store logical durable state
• backend layout conventions determine where committed files and shared layers live
• runtime resolution turns committed snapshot state into node-local runnable paths

3) Repository Layout

The default backend is the POSIX filesystem backend. Given a backend root like /path/to/store/repository:

• committed snapshot manifest:
  • /path/to/store/repository/snapshots/<id>/snapshot.json
• committed fixed-layout files:
  • /path/to/store/repository/snapshots/<id>/vm_state.bin
  • /path/to/store/repository/snapshots/<id>/firecracker-manifest.json
• shared managed layers:
  • /path/to/store/repository/managed-layers/<digest>.overlaybd.commit
• alias bindings:
  • /path/to/store/repository/catalog/aliases/<alias>
  • /path/to/store/repository/catalog/aliases/<alias>.lock

Alias locking is per-alias, not a single global aliases.lock file.

snapshot.json intentionally does not repeat local build-artifact paths such as mem_image.json or rootfs/image.json. Runtime code derives node-local image configs from:

• repository root
• snapshot id
• attached drive id
• managed layer digest

firecracker-manifest.json complements this by persisting launch-time virtual size and attached-drive mode metadata without embedding node-local runtime paths.

4) Load and Launch Flow

Typical runtime usage is:

1. SnapshotManager::load_committed(id_or_alias).await
2. SnapshotManager::resolve_runnable(stored).await
3. FirecrackerSandbox::from_snapshot(&runnable, &SandboxLaunchConfig::default())
4. sandbox.start().await

If you want the two snapshot-manager calls combined, use SnapshotManager::load_runnable(id_or_alias).await.

resolve_runnable(...) is where backend-neutral committed state becomes node-local launch inputs:

• memory_layers -> runtime memory/image.json
• rootfs.layers -> runtime rootfs/image.json
• attached_drives[].layers -> runtime drives/<id>/image.json
• attached_drives[].read_only -> runtime mount mode for drives/<id>
• firecracker-manifest.json -> runtime manifest hydrated with node-local artifact paths
• committed vm_state.bin -> runnable vm-state path

5) Minimal Example

#![allow(unused)]
fn main() {
use agentenv::cfg::ConfigManager;
use agentenv::image::ImageResolver;
use agentenv::sandbox::{FirecrackerSandbox, SandboxExecutor, SandboxLaunchConfig};
use agentenv::snapshot::SnapshotManager;
use agentenv::template::{TemplateBuildSpec, TemplateBuilder};

let builder = TemplateBuilder::new();
let snapshot_manager = SnapshotManager::new()?;
let config = ConfigManager::global()?.config();
let image_resolver = ImageResolver::new(config);
let image_config = image_resolver
  .resolve(image_resolver.default_image())
  .await?
  .overlaybd_config_path;

let alias = "my-template-v1";

builder
    .build_and_publish(
        &snapshot_manager,
        TemplateBuildSpec::new()
            .from_overlaybd_config(image_config)
            .alias(alias)
            .resources(1, 128)
            .run("mkdir -p /workspace")
            .workdir("/workspace")
            .env("MARK", "ready")
            .run("printf '%s' \"$MARK\" > mark.txt"),
    )
    .await?;

let runnable = snapshot_manager
    .load_runnable(alias)
    .await?
    .expect("template should exist");

let mut sandbox =
    FirecrackerSandbox::from_snapshot(&runnable, &SandboxLaunchConfig::default())?;
sandbox.start().await?;

let out = sandbox.run_command("cat", &["/workspace/mark.txt"]).await?;
assert_eq!(out.exit_code, 0);
assert_eq!(out.stdout.trim(), "ready");

sandbox.stop().await?;
}

6) State Transition

The template-facing state transition is:

1. local build artifacts
2. committed snapshot.json + firecracker-manifest.json + fixed files + managed layers
3. node-local runtime-derived image configs
4. Firecracker resume inputs

7) What the Tests Validate

tests/integration/snapshot.rs covers:

• building and publishing from overlaybd configs
• loading committed snapshots by alias / id through the snapshot manager
• resolving committed snapshots into runnable snapshots
• launching Firecracker sandboxes from resolved snapshots
• rebuilding from committed snapshots while preserving base state
• listing and deleting committed snapshots through the template API surface

tests/integration/snapshot_attached_drive.rs covers overlaybd attached-drive handling across sandbox capture, publish, resolve, launch, and rebuild flows, including:

• committed metadata preserving readOnly
• runtime mount mode for readonly and writable drives
• rebuild keeping attached-drive metadata stable

tests/snapshot_oss_e2e_test.rs covers OSS-backed snapshot publication, resolution, alias cleanup, missing managed-layer failure modes, and deletion.

Repository/backend unit tests under src/snapshot/repository/backends/ cover:

• artifact import
• alias handling
• committed snapshot deletion
• runtime cache behavior

8) Common Failure Areas

• overlaybd base config paths are missing or invalid
• ublk is disabled while using overlaybd build bases
• Linux host prerequisites for Firecracker / KVM / namespaces are missing
• repository alias conflicts during publish
• runtime resolver cannot materialize local paths from committed artifacts

Persistence Artifact Inventory

This document lists AgentENV artifacts that can remain on disk or in object storage beyond a single function call. It is organized by the module that owns each artifact’s lifecycle.

Path Roots

Setup And Config

Owned by src/setup/* and src/cfg.rs.

Firecracker Sandbox

Runtime

Owned by src/sandbox/firecracker/*.

Firecracker Pool

Owned by src/sandbox/firecracker/pool.rs.

Extra Drives

Owned by src/sandbox/extra_drive.rs and Firecracker snapshot code.

Image Resolver

Owned by src/image/*.

Snapshot Repository

Owned by src/snapshot/repository/* and src/snapshot/types/*.

Repository records are the logical source of truth for committed snapshots. Build-time and runtime image.json files should not become committed truth unless they contain data that cannot be derived from committed metadata.

Snapshot Runtime Resolution

Owned by src/snapshot/artifact_cache.rs, src/snapshot/runtime_support.rs, and backend runtime resolvers.

LocalArtifactCache owns pinning, in-flight materialization deduplication, and LRU eviction for files it manages. It does not own the durable snapshot repository.

Orchestrator Persistence

Owned by src/orchestrator/persistence/*.

Paused sandbox artifacts are not committed snapshot repository artifacts. They are owned by the sandbox persister and should not be shared as snapshot truth.

P2P Artifact Transport

Owned by src/p2p/*.

Snapshot publication writes best-effort P2P catalog entries after repository commit:

• snapshot/v1/artifacts/{snapshot_id}/vm_state.bin
• snapshot/v1/artifacts/{snapshot_id}/firecracker-manifest.json
• overlaybd-layer/v1/sha256:<digest> for referenced rootfs, memory, and attached-drive overlaybd commit layers

Those P2P entries are optional copies, not committed snapshot truth. Clearing <p2p.store_dir> can reduce peer-to-peer availability and force peers back to OSS or origin-registry reads, but it must not make a committed snapshot invalid by itself.

Ublk Daemon Runtime

Owned by storage/ublk-daemon/* and src/sandbox/ublk/*.

Overlaybd Storage

Owned by storage/overlaybd/*.

Ownership Rules

• Snapshot repository artifacts are durable user-visible state. Do not delete them from node-local GC code.
• Paused sandbox artifacts are durable only for the paused sandbox that owns them. Do not treat them as committed snapshots.
• Runtime image.json files under snapshot local cache or ublk runtime dirs are derived artifacts. They should be rebuildable from committed metadata or source image configs.
• Node-local image-cache artifacts are disposable cache, but content-addressed commits may be expensive to regenerate. The metadata-backed GC only reclaims commits no longer rooted by on-disk source configs, held by image-cache leases, or referenced by the in-process running set. Committed snapshots are durable SnapshotRepository state and do not pin ImageCache commits. See [image.cache.gc] in the configuration reference for scheduling.
• P2P store contents are node-local and optional. Unpublish removes the local catalog entry and related blobs; clearing the store only affects peer-to-peer sharing and may require republishing or refetching artifacts.
• Logs and dependency downloads are operational artifacts. They are not part of sandbox or snapshot correctness, but may require separate retention policy outside image-cache GC.

AgentENV Proxy Design and Usage

This document describes the per-node reverse proxy that forwards requests into individual sandboxes. For the distributed routing layer (gateway to scheduler to node), see System Architecture.

Scope

Each AgentENV node runs a reverse proxy that accepts requests on its API server and forwards them to sandbox services over the sandbox interaction network. In a multi-node deployment, the gateway resolves sandbox ownership through the scheduler and forwards data-plane requests to the owning node’s proxy surface.

Current entrypoints:

• ANY /proxy
• ANY /proxy/{*proxy_path}
• Any otherwise unmatched path that carries sandbox routing headers
• Host-based sandbox proxy requests when [sandbox_proxy].domains is configured: {port}-{sandboxID}.{domain}

Implementation references:

• src/api/proxy.rs
• src/orchestrator/service.rs
• src/orchestrator/proxy.rs

Routing Contract

Each proxied request must identify:

• Sandbox ID
• Target port inside the sandbox service plane

Accepted headers:

• x-agentenv-sandbox-id
• x-agentenv-target-port
• E2B-compatible aliases:
  • e2b-sandbox-id
  • e2b-sandbox-port

Validation:

• Sandbox ID must be a valid UUID format.
• Target port must parse as u16 and be greater than 0.

Host-based routing derives both fields from Host. The configured domain must match exactly after lowercase normalization and optional trailing-dot removal. Sandbox IDs in host routes must be valid UUIDs and the target port must fit in u16.

Runtime Route Model

AgentENV keeps an in-memory runtime route table in orchestrator.

• Route key: SandboxId
• Route value: ProxyTarget (currently host interaction IP) plus route metadata (version, updated_at)

Design rule:

• Only Running sandboxes publish runtime routes.
• Non-running states rely on metadata fallback (not route-table state).

Lookup behavior (proxy_lookup_for):

1. If runtime route exists: Ready(target)
2. Else read metadata:
   • no metadata: NotFound
   • metadata state is Running: RouteMissing
   • metadata state is Paused: Paused { auto_resume }
   • other states: Unavailable(state)

This keeps hot-path reads lock-light and avoids reading sandbox instance internals in API request paths.

Paused Sandbox Auto-Resume

/proxy can auto-resume paused sandboxes when lifecycle policy enables it.

• Proxy never reads sandbox metadata directly.
• Orchestrator lookup returns Paused { auto_resume }, and proxy decides behavior from that signal.
• Auto-resume is attempted once per request.
• Resume timeout update uses EnsureMinimum(5 minutes):
  • effective sandbox timeout is max(existing_timeout, 5 minutes)
• Proxy waits up to:
  • test builds: short unit-test timeout
  • non-test runtime: 60s

Request outcomes for paused sandboxes:

• auto_resume = false: 410 Gone
• auto_resume = true and resume succeeds, then route becomes ready: request is forwarded normally
• auto_resume = true but resume/lookup fails: 502 Bad Gateway
• auto_resume = true but resume wait times out: 504 Gateway Timeout

Lifecycle Hooks and Race Hardening

Route publication/removal is tied to lifecycle transitions.

• Create/Resume success path:
  • Persist metadata to Running
  • Publish runtime route only if the launching sandbox handle is still the current handle
• Pause/Delete/Rollback paths:
  • Atomically detach sandbox handle and runtime route before stop/finalization

Race protections:

• Late route publication from stale handles is blocked via pointer identity check.
• Handle detachment and route removal happen in one critical section to reduce stale-route visibility windows.
• Launch rollback supports both transitional-state rollback and running-state rollback paths.

HTTP Forwarding Semantics

Path and Query

• /proxy forwards to upstream /
• /proxy/{*proxy_path} forwards raw URI path suffix after /proxy
• Header-routed fallback requests forward the original request path
• Host-based requests forward the original request path
• Query string is forwarded unchanged

Important details:

• Percent-encoded path segments are preserved.
• Repeated leading slashes are preserved.
  • Example: /proxy//api forwards as //api
• Host-based proxy routing runs before Axum route matching. When a configured sandbox proxy host is used, the request is data-plane traffic even if its path resembles a control-plane API.

Header Handling

Control-plane routing headers are stripped before forwarding upstream:

• x-agentenv-sandbox-id
• x-agentenv-target-port
• e2b-sandbox-id
• e2b-sandbox-port

Hop-by-hop headers are stripped on both request and response paths, including:

• Standard hop-by-hop headers (Connection, Upgrade, TE, Trailer, Transfer-Encoding, Proxy-Authenticate, Proxy-Authorization, Keep-Alive)
• Any extra headers nominated by Connection

Forwarded headers are injected:

• x-forwarded-host
• x-forwarded-proto
• x-forwarded-method
• x-forwarded-uri

Streaming

HTTP bodies are proxied as streams (request and response), including SSE and large uploads/downloads.

WebSocket Semantics

WebSocket upgrade is supported through the same /proxy endpoints.

• Client upgrade request is validated and forwarded upstream.
• Bidirectional frame bridging is established after successful upstream handshake.
• Selected subprotocol from upstream is propagated to the client.

Handshake failure behavior:

• If upstream rejects with an HTTP response (for example 401, 403, 404), status/body are forwarded as-is.
• Transport or connection failures return 502 Bad Gateway.
• Handshake timeout returns 504 Gateway Timeout.

Error Mapping

• 400 Bad Request
  • Missing/invalid sandbox routing header
  • Missing/invalid target port header
  • Invalid upstream URI construction
• 404 Not Found
  • Sandbox not found
• 410 Gone
  • Sandbox is paused and auto-resume is disabled
  • Sandbox exists but is not proxyable in current state
• 502 Bad Gateway
  • Upstream transport/connect failure
  • Sandbox is Running but runtime route is missing (RouteMissing)
  • Paused sandbox auto-resume failed
• 504 Gateway Timeout
  • Paused sandbox auto-resume timed out
  • Upstream response header timeout
  • Upstream websocket handshake timeout

Usage Examples

HTTP

curl -i \
  -H 'X-API-Key: test-key' \
  -H 'x-agentenv-sandbox-id: <sandbox-uuid>' \
  -H 'x-agentenv-target-port: 8080' \
  'http://127.0.0.1:8000/proxy/health?full=true'

E2B-compatible headers

curl -i \
  -H 'X-API-Key: test-key' \
  -H 'e2b-sandbox-id: <sandbox-uuid>' \
  -H 'e2b-sandbox-port: 8080' \
  'http://127.0.0.1:8000/proxy/status'

WebSocket

Example (generic):

• URL: ws://127.0.0.1:8000/proxy/ws/echo
• Headers:
  • x-agentenv-sandbox-id: <sandbox-uuid>
  • x-agentenv-target-port: <port>
  • API auth headers as required by your deployment

Host-based

curl -i \
  -H 'X-API-Key: test-key' \
  'http://8080-<sandbox-uuid>.sandbox.example.com/status'

Testing Notes

Relevant test coverage exists in:

• src/api/proxy.rs unit tests (HTTP, SSE, large body, websocket, headers, path preservation, error mapping)
• src/orchestrator/service.rs unit tests (route publication/removal behavior and stale-handle guard)
• Integration lifecycle tests in tests/integration/orchestrator.rs
• E2E proxy suite in scripts/tests/e2e/suites/06_proxy.sh (header compatibility and paused auto-resume behavior)

For environment-backed integration validation, use repository-prescribed integration targets.

Distributed Control Plane

The multi-node control plane lives in services/ as a separate Go module. It routes client traffic across multiple AgentENV backend nodes.

Components

• Gateway (services/gateway/): HTTP reverse proxy that routes by sandbox ID
• Scheduler (services/scheduler/): gRPC service for node selection, sandbox-to-node binding, observed node snapshots, and P2P peer endpoint discovery

Build and Test

Prerequisites: Go 1.21+

# From services/
make build      # builds both gateway and scheduler
make test       # tests both services
make tidy       # go mod tidy + formatting
make proto      # regenerate protobuf

Run Locally

# Start scheduler (default: 127.0.0.1:9090)
make -C services run-scheduler

# Start gateway
make -C services run-gateway

Discovery Modes

The scheduler supports two node discovery modes:

• static (default): explicit node list from config
• kubernetes: watches EndpointSlices for a headless Service, using ready Pod IPs as backends

Deployment

Docker Compose

make deploy-up      # gateway + scheduler + 2 backend nodes
make deploy-ps      # status
make deploy-logs    # logs
make deploy-down    # teardown

Kubernetes

make k8s-render     # render manifests
make k8s-apply      # apply to cluster

Deployment model:

• gateway: Deployment + ClusterIP Service
• scheduler: single-replica Deployment + ClusterIP Service
• agentenv-node: privileged DaemonSet with /dev/kvm and hostPath
• agentenv-nodes: headless Service for scheduler EndpointSlice discovery

gRPC API

Proto contract: services/api/proto/scheduler.proto

RPCs: Schedule, ListNodes, LookupNode, RecordAssignment, Heartbeat, ListObservedNodes, ListP2pPeers, GetNode, UnregisterNode

Runtime node heartbeats may include an opaque P2pEndpoint containing a backend name and backend-specific address. The scheduler stores that endpoint with the observed-node record and returns ready peers through ListP2pPeers(cluster_id, backend, exclude_node_id). The scheduler does not query artifact catalogs and never forwards artifact data.

For full configuration details (header compatibility, timeouts, logging), see the services README.

P2P Artifact Transport

AgentENV has a project-wide P2P artifact transport layer in src/p2p/. It lets runtime modules publish, discover, and fetch validated files from peer nodes without depending on a concrete transport implementation.

The first concrete backend is embedded in the AgentENV server process and uses iroh plus iroh-blobs. The public API stays behind traits and serializable types so future backends can be selected by configuration.

Goals

• Provide one reusable node-to-node artifact transport for multiple AgentENV modules.
• Keep module code independent from concrete backends.
• Use scheduler for endpoint discovery and a lightweight in-memory artifact-to-node index, not artifact metadata storage or data proxying.
• Let transport backends use their native content-addressing or integrity checks while keeping module code backend-neutral.
• Keep disabled mode cheap and safe for single-node deployments.

Module Layout

Public API

Consumers use Arc<dyn P2pTransport>:

#![allow(unused)]
fn main() {
#[async_trait]
pub trait P2pTransport: Send + Sync {
    async fn lookup(&self, key: &P2pArtifactKey) -> Result<Option<P2pArtifactDescriptor>>;
    async fn lookup_with_hints(
        &self,
        key: &P2pArtifactKey,
        hints: &[P2pArtifactProviderHint],
    ) -> Result<Option<P2pArtifactDescriptor>>;
    async fn fetch(&self, descriptor: &P2pArtifactDescriptor, destination: &Path) -> Result<u64>;
    async fn fetch_bytes(&self, descriptor: &P2pArtifactDescriptor) -> Result<Bytes>;
    async fn fetch_byte_range(
        &self,
        descriptor: &P2pArtifactDescriptor,
        offset: u64,
        len: usize,
    ) -> Result<Bytes>;
    async fn publish(&self, request: &P2pPublishRequest) -> Result<()>;
    async fn unpublish(&self, key: &P2pArtifactKey) -> Result<bool>;
    fn local_endpoint(&self) -> Option<P2pEndpoint>;
    async fn shutdown(&self) -> Result<()>;
}
}

P2pArtifactKey is a stable string chosen by the consuming module. One key represents one logical artifact. Callers should encode the complete cache context into that key and use descriptor metadata for module-specific validation.

P2pArtifactDescriptor contains:

• key: stable lookup key.
• providers: nodes that can serve the artifact, represented as P2pPeer values.
• backend_locator: optional transport-specific string used only by the matching backend. For iroh this is the iroh-blobs hash.
• metadata: module-defined JSON used by callers to interpret the file.

P2pPublishRequest wraps a stable key, a local source path or in-memory bytes, optional metadata, and a P2pPublishMode. P2pPublishRequest::file(key, source) defaults metadata to null and publish mode to Copy; P2pPublishRequest::bytes(key, bytes) publishes already-buffered bytes. Copy allows the transport to copy bytes into its own store; Reference lets backends retain or index the existing file when supported.

Disabled Transport

DisabledP2pTransport is selected when [p2p].enabled = false or transport = "disabled".

• lookup returns Ok(None).
• publish succeeds as a no-op.
• unpublish succeeds as a no-op and returns Ok(false).
• fetch returns TransportDisabled.
• local_endpoint returns None.

This lets consumers call the P2P layer unconditionally while keeping default deployments unchanged.

Iroh Backend

IrohBlobsP2pTransport is selected with [p2p].enabled = true and [p2p].transport = "iroh".

At startup it:

1. Creates the configured blob store directory.
2. Opens an iroh_blobs::store::fs::FsStore with gated background GC.
3. Opens a RocksDB-backed local catalog at <p2p.store_dir>/iroh/catalog.db.
4. Binds an iroh endpoint, optionally using [p2p].listen_addr.
5. Starts one router that serves both iroh-blobs data and AgentENV’s catalog protocol.
6. Publishes the local endpoint through local_endpoint() so observability heartbeat can advertise it.

The backend uses one local catalog:

• Published catalog: artifact key to the descriptor this node can serve. The in-memory map is loaded from RocksDB on startup; each RocksDB value is a compact JSON descriptor.

Lookup first checks the local published catalog, then asks scheduler for nodes indexed under the artifact key, and finally falls back to discovered peers over the AgentENV catalog ALPN in hint/discovery order. Scheduler-indexed and fallback candidates are still verified by querying each node’s catalog; scheduler never stores locators or metadata. The descriptor carries the iroh blob hash in its backend locator, so fetch does not depend on prior in-process lookup state.

Fetch parses the descriptor’s backend locator as an iroh blob hash, passes all descriptor providers to iroh-blobs, downloads that blob into the local FsStore, and exports it to the requested destination. Iroh’s content-addressed blob transfer validates bytes against that hash. If any descriptor provider is local node, fetch exports directly from the local store.

After a remote fetch downloads the blob into the local store, the iroh backend best-effort advertises the local copy. It applies the same deterministic retention tag used by publish, builds a descriptor with this node as the only local catalog provider, upserts that descriptor into the local catalog, and records the key in scheduler. This makes the fetching node a provider for later peers and lets artifacts spread through the cluster. Advertisement or scheduler-index failures are logged and do not fail fetch; only the download and destination export are part of the required fetch result.

Publish imports the local file into FsStore, applies a deterministic named tag derived from the artifact key (agentenv:p2p:v1:{sha256(key)}), builds a descriptor with this node as the only local catalog provider, local endpoint, iroh blob hash locator, and request metadata, upserts that descriptor into the local catalog, and best-effort records the key in scheduler.

Unpublish removes the key from the local catalog, deletes the deterministic named tag, forgets the key in scheduler, persists the catalog, and returns whether a local publication was removed. Deleting the tag stops future catalog lookup immediately, but it does not synchronously delete the blob bytes. The iroh store reclaims untagged blobs through its GC.

GC is gated to avoid periodic full-store scans when there is no known deletion work:

• The transport starts with one pending GC pass, so a restart can clean up blobs whose tags were removed before a previous process exited.
• unpublish marks GC as pending after deleting the retention tag.
• Each GC interval wakes up the iroh-blobs GC task, but the configured add_protected callback aborts the mark/sweep pass unless pending GC is set.
• When pending GC is set, one full mark/sweep pass runs. It preserves tagged and temp-tagged blobs and deletes blobs that are no longer protected.

Scheduler Discovery And Artifact Index

The scheduler is a directory for node endpoints and artifact-to-node hints only.

1. AgentENV starts the configured P2P transport in src/bin/server.rs.
2. If the transport exposes local_endpoint(), ObservabilityReporter includes it in heartbeat requests.
3. Scheduler stores the endpoint with the observed-node record.
4. SchedulerPeerDiscovery periodically calls ListP2pPeers(cluster_id, backend, exclude_node_id).
5. Scheduler returns ready nodes with non-empty endpoints for the requested backend.
6. The transport dials peers directly for catalog lookup and byte transfer.

For artifact lookup acceleration, the iroh backend also calls:

• RecordP2pArtifact(cluster_id, backend, key, node_id) after publish and after successful remote fetch advertisement.
• ForgetP2pArtifact(cluster_id, backend, key, node_id) after local unpublish.
• LookupP2pArtifact(cluster_id, backend, key, exclude_node_id) before broad peer polling.

Scheduler keeps this artifact index in memory and stores only key-to-node mappings. Lookup responses are filtered through the same ready-node and backend endpoint checks as ListP2pPeers. Node unregister removes all artifact mappings for that node. Scheduler does not persist artifact catalogs, inspect artifact metadata or locators, or proxy bytes.

Snapshot And Overlaybd Consumers

src/snapshot/p2p.rs is the snapshot-store integration layer. It is intentionally small: the snapshot manager publishes artifacts after the configured repository commit succeeds, and runtime resolvers treat P2P as an optional acceleration path rather than committed truth.

Snapshot publication advertises:

• Fixed Firecracker artifacts under snapshot/v1/artifacts/{snapshot_id}/{relative_path}:
  • vm_state.bin is published from its local file path.
  • firecracker-manifest.json is serialized from the committed manifest and published as bytes.
• Overlaybd layers referenced by the snapshot’s rootfs, memory, and attached-drive image configs. These are not published under a snapshot-specific key. They reuse the overlaybd layer artifact protocol owned by src/overlaybd/p2p/artifact.rs, with keys like overlaybd-layer/v1/sha256:<digest> and LayerMetadata understood by the overlaybd HTTP facade.

That split is important. Snapshot fixed artifacts are scoped to one snapshot ID and are only consumed by snapshot runtime resolvers. Overlaybd commit layers are content-addressed and may be consumed by any overlaybd runtime path, including foreground range reads through /p2p-http/{origin}. Do not add a second snapshot-specific key format for overlaybd layers; doing so publishes bytes that overlaybd cannot discover.

OSS snapshot resolution consumes fixed artifacts through P2P first:

• vm_state.bin is fetched into the node-local LocalArtifactCache from P2P, falling back to OSS on miss or fetch failure.
• firecracker-manifest.json is fetched with fetch_bytes from P2P, parsed in memory, and falls back to OSS on miss, fetch failure, or parse failure.

POSIX snapshot resolution does not consume P2P. A POSIX repository is already a directly accessible committed artifact store; if a required file is missing, the resolver returns ArtifactNotFound instead of repairing the repository from peers. Overlaybd layer acceleration for runtime block reads remains the responsibility of the overlaybd P2P facade, not the POSIX snapshot resolver.

Snapshot P2P publish is best-effort. A failed P2P publish logs a warning and does not roll back the committed snapshot record. This preserves the repository as the source of truth while making newly published artifacts visible to peers before slower backend downloads or uploads become the bottleneck.

Integrity And Ownership

The P2P transport does not define what an artifact means. Each consuming module owns:

• Constructing stable keys.
• Writing metadata into publish requests.
• Validating metadata before trusting a fetched artifact.
• Deciding whether and when to publish local artifacts.

This boundary keeps the transport reusable and avoids baking module-specific cache semantics into src/p2p.