Getting Started

AgentENV (abbreviated as AENV) is a self-hosted sandbox runtime for AI agents. It runs isolated Firecracker microVMs and exposes an E2B-compatible HTTP API — so existing E2B SDK code works against it without modification. The repository is available at https://github.com/kvcache-ai/AgentENV.

Why AgentENV

• Scale across diverse environments: AENV runs massive numbers of Firecracker environments across machines and diverse OCI-compatible images, loaded on demand via overlaybd. Local disk acts as a bounded cache, retaining hot data and evicting cold, so images can exceed disk capacity while startup stays fast cluster-wide, without pre-warming every host.
• Make idle environments inexpensive: Snapshot-backed environments boot or resume in under 50 ms and pause in under 100 ms. Idle environments can quickly release CPU and memory, then return when new work arrives.
• Native snapshot and fork support: AENV snapshots memory and filesystem changes incrementally, completing in under 100 ms even under heavy disk modification. A running environment can fork into multiple independent sandboxes for parallel agent workflows. Snapshots persist to S3-compatible object storage or a shared distributed filesystem to prevent data loss.
• Preserve performance and density over time: AENV delivers high-performance I/O via ublk while sharing the host page cache across storage and memory-snapshot data. Memory ballooning returns reclaimable guest memory to the host, sustaining high overcommit as environments run longer and diverge.

Features

• Firecracker microVMs with full Linux kernel isolation per sandbox
• Pause and resume with memory + disk snapshots for instant cold start
• Layered block devices via overlaybd + ublk for copy-on-write image sharing
• Snapshot-backed template builder for publishing reusable, pre-configured sandbox runtimes
• E2B-compatible API so existing E2B SDKs and CLIs work out of the box
• Reverse proxy to reach services running inside sandboxes via HTTP and WebSocket
• Multi-node scaling with a gateway + scheduler control plane

Who Is This For

AgentENV is built for teams running AI agents that need isolated execution environments: code interpreters, tool-use agents, autonomous coding agents, or any workload where you want a fresh (or cached) Linux environment per task.

Interacting with the Server

AgentENV exposes an HTTP API. There are four ways to use it:

Where to Go Next

• Quick Start — Install the server, run your first sandbox. Takes ~5 minutes on a supported Linux host.
• Deployment — Build from source, Docker Compose multi-node, or Kubernetes.
• PVM Deployment — Use AgentENV on a server where standard KVM is unavailable.

Quick Start

Prerequisites

• Linux kernel 6.8+
• /dev/kvm access for Firecracker microVM execution

If your server does not support standard KVM, use the dedicated PVM Deployment guide instead.

The install script attempts to install missing download and checksum commands, provisions /dev/kvm permissions, loads the ublk_drv kernel module, and downloads the required AgentENV runtime assets.

Installation requires root, but the installed service does not run as root. It uses a dedicated aenv system account with CAP_NET_ADMIN and CAP_SYS_ADMIN, plus group access to /dev/kvm and the ublk devices.

Setup

1. Install and start the server

Option A — Install Script

The script installs both the server and the aenv CLI. Set AENV_HOME_PATH to choose the data directory; if it is not set, AENV stores runtime dependencies and data in /var/lib/aenv. After installation, start the server as a systemd service:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh | sudo AENV_HOME_PATH=/path/to/aenv/data bash
sudo systemctl start aenv

To customize the server configuration, edit config.toml and restart the service:

sudo vim /var/lib/aenv/config/config.toml  # Or <AENV_HOME_PATH>/config/config.toml if AENV_HOME_PATH is set.
sudo systemctl restart aenv

To change the port, edit API_ADDR in /etc/default/aenv and restart the service.

The service’s persistent state is owned by aenv under /var/lib/aenv by default. Transient namespace and daemon-socket state lives under /run/aenv.

Option B — Docker

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/docker-setup.sh | sudo bash
docker pull ghcr.io/kvcache-ai/aenv-server:latest
docker run --rm -it \
  --name aenv-server \
  --device /dev/kvm --privileged -v /dev:/dev \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest

To customize the server configuration, download and edit the configuration file, then mount it at the path used by the container:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/config/default.toml -o config.toml
vim config.toml
docker run --rm -it \
  --name aenv-server \
  --device /dev/kvm --privileged -v /dev:/dev \
  -v "$PWD/config.toml:/workspace/config/default.toml:ro" \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest

To change the port, add the -e API_ADDR and -p flags.

The server is accessible at http://127.0.0.1:8000 by default. Verify it is running:

curl http://127.0.0.1:8000/health

2. Install the aenv CLI

Skip this step if you used Option A — the install script already includes aenv.

Supports Linux and macOS on x86_64 and arm64:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install-cli.sh | bash

3. Authenticate

The server generates the key on its first normal startup. Native installations reuse the managed key; a normal Docker container keeps it in its writable container layer:

# Native
sudo cat /var/lib/aenv/secrets/api-key
# Docker
docker exec aenv-server cat /workspace/env/secrets/api-key

aenv auth
# AENV server URL [http://localhost:8000]: http://127.0.0.1:8000
# API key: <paste the generated key>

4. Pull a template and run a sandbox

aenv pull ubuntu:22.04 --name ubuntu
aenv start ubuntu            # starts a sandbox and attaches an interactive shell

Next Steps

• Deployment — build from source, multi-node options
• PVM Deployment — deploy when standard KVM is unavailable
• Core Concepts — how sandboxes, templates, and snapshots work
• E2B — SDK and CLI compatibility
• API Reference — full HTTP API

On-Demand Loading from Shared Storage

AgentENV loads images on demand via overlaybd. Local disk acts as a bounded cache, retaining hot data and evicting cold data so nodes do not need to pre-warm every image or keep a complete copy of every snapshot.

Prerequisites

• Storage connectivity should be as fast as possible. Use at least a 1 Gbps network; 10 Gbps or faster is strongly recommended.

1. Open the configuration file

If AgentENV is running as a systemd service, open the installed configuration:

sudo vim /var/lib/aenv/config/config.toml  # Or use the path to your config file.

If AgentENV is running in Docker, download and edit the default configuration:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/config/default.toml -o config.toml
vim config.toml

2. Configure shared storage

AgentENV supports two shared storage backends: POSIXFS and OSS. Choose one of the following options.

Option A: POSIXFS

Enable the POSIXFS backend, set the shared snapshot location, and increase the remote-block cache:

[snapshot]
repository_backend = "posix_fs"

[backend.posix_fs]
snapshot_store = "/mnt/aenv-snapshots"

[image.cache.remote_blocks]
max_size_gb = 100

Option B: OSS

Enable the OSS backend, configure the OSS connection, and increase the remote-block cache:

[snapshot]
repository_backend = "oss"

[backend.oss]
endpoint = "YOUR_ENDPOINT"
bucket = "YOUR_BUCKET"
region = "YOUR_REGION"
prefix = "YOUR_PREFIX"
cache_max_size_gb = 100
access_key_id = "YOUR_ACCESS_KEY_ID"
access_key_secret = "YOUR_ACCESS_KEY_SECRET"

[image.cache.remote_blocks]
max_size_gb = 100

If your provider requires or prefers virtual-host bucket addressing — for example Tigris or a Cloudflare R2 deployment — also set addressing_style = "virtual"; see the configuration reference for details.

3. Apply the configuration

If AgentENV is running as a systemd service, restart it:

sudo systemctl restart aenv

If AgentENV is running in Docker, stop the current container and recreate it with the updated configuration and shared snapshot directory:

# The /mnt/aenv-snapshots mount is required only when using POSIXFS.
docker stop CONTAINER_ID_OR_NAME
docker run --rm -it --name aenv-server \
  --device /dev/kvm --privileged -v /dev:/dev \
  -v "$PWD/config.toml:/workspace/config/default.toml:ro" \
  -v /mnt/aenv-snapshots:/mnt/aenv-snapshots \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest

Once configured, AgentENV loads templates and snapshots from shared storage by default.

aenv CLI Reference

aenv is the native CLI for AgentENV. It wraps the HTTP API and envd gRPC endpoints into a developer-friendly interface for managing templates, sandboxes, and snapshots.

Installation

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install-cli.sh | bash

Each release publishes aenv-<os>-<arch>.tar.gz for Linux and macOS on x86_64 and aarch64 (arm64). The archive contains:

aenv
aenv-buildctl
manifest.json

The manifest records the AgentENV version, BuildKit version, and platform. SHA256SUMS accompanies the release archives. The release workflow verifies the pinned upstream BuildKit download before packaging its client with aenv.

Both install-cli.sh and the full install.sh download this single CLI archive and verify its GitHub release asset checksum before installing both executables. Installation does not contact the upstream BuildKit release. The private aenv-buildctl lives beside aenv, preserving an existing system buildctl. Set INSTALL_DIR for a user-local CLI installation. For manual installation, extract the archive and keep both executables in the same directory.

Or build from source (requires Rust):

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV
make install-aenv

Authentication

aenv auth

Save the server URL and API key. Credentials are stored at ~/.config/aenv/credentials (mode 0600).

aenv auth
# AENV server URL [http://localhost:8000]: The address of the AgentENV server
# API key: <the server's configured or generated API key>

Templates

aenv pull <image>

Create a template from an OCI image. Waits for the build to complete by default.

aenv pull ubuntu:22.04
aenv pull ubuntu:22.04 --name my-ubuntu

aenv build <context> --name <name>

Create a template from a local Dockerfile using BuildKit in an isolated microVM. The installers include the required buildctl executable. For source builds, install BuildKit’s buildctl v0.33.0 or select an existing executable with --buildctl.

aenv build . --name my-app
aenv build . -f deploy/docker/Dockerfile.agentenv --name aenv
aenv build ./my-app --name my-app-v2 --build-arg VERSION=2

The command supports multi-stage builds, file updates, and .dockerignore, shows BuildKit progress, and waits until the template is ready. The server manages the internal worker and releases it afterward. The first Dockerfile build prepares a reusable builder template. Each build clones a shared cache seed, allowing concurrent builds and cache reuse across template names and nodes. Builder image and sizing are configured in [template_build] on the server, with defaults of 16 vCPUs, 32 GiB memory, and 64 GiB cache disk. FROM, ENTRYPOINT, CMD, and HEALTHCHECK in the final image determine the template’s image, startup, and readiness. The final Dockerfile stage is always published. See templates for the complete workflow, cache management, and startup behavior.

Interactive builds show a three-stage bar for builder preparation, image build, and template publication, with elapsed time and Dockerfile logs above the bar. --progress plain keeps plain logs; --progress tty selects BuildKit’s native terminal display. Redirected output has no progress-bar control sequences.

aenv template list

List all templates. Alias: aenv template ls, aenv templates list.

aenv template list
aenv template list --output json

aenv template watch <template>

Watch a template build until it succeeds or fails. Accepts either a template name/alias or a template UUID.

aenv template watch my-ubuntu
aenv template watch <template-id>

aenv template delete <template>

Delete a template by name or ID. Alias: aenv template rm.

aenv template delete my-ubuntu
aenv template delete <template-id>

Sandboxes

aenv start <target>

Start a sandbox and attach an interactive shell. <target> accepts a template or snapshot ID or alias, or an OCI image reference with --cold.

aenv start my-ubuntu
aenv start --cold ubuntu:24.04              # start directly from an OCI image

Sandboxes started by aenv always require token-authenticated envd access. The CLI obtains and manages the access token automatically.

CPU, memory, and disk overrides are supported only for cold starts.

aenv pause <sandbox-id>

Pause a running sandbox. The sandbox state is preserved and can be resumed later.

aenv pause <sandbox-id>

aenv resume <sandbox-id>

Resume a paused sandbox.

aenv resume <sandbox-id>

aenv timeout <sandbox-id> <seconds>

Set or extend the sandbox expiration to <seconds> from now.

aenv timeout <sandbox-id> 600

aenv connect <sandbox-id>

Attach an interactive shell to a running or paused sandbox. Alias: aenv cn.

aenv connect <sandbox-id>

Resumes the sandbox if paused before attaching.

aenv exec <sandbox-id> <command> [args...]

Run a one-shot command in a sandbox and stream its output.

aenv exec <sandbox-id> ls -la /

aenv upload <sandbox-id> <local-path> <remote-path>

Upload a local file or directory to a sandbox through envd. Files are streamed individually, and missing remote directories are created automatically.

aenv upload <sandbox-id> ./config.json /workspace/config.json
aenv upload <sandbox-id> ./config.json /workspace/
aenv upload <sandbox-id> ./project /workspace/
aenv upload <sandbox-id> ./project /workspace/app
aenv upload --user app <sandbox-id> ./config.json config.json

For directory uploads, the remote path must be absolute and --user is not supported. If the remote destination ends in / or already exists as a directory, the local directory name is appended. Otherwise the destination is used as the new directory root. Hidden files and empty directories are copied; symbolic links and special files are rejected.

Upload copies file contents and directory structure only. It does not preserve host ownership or group, permissions (including executable bits), timestamps, ACLs, extended attributes, or hard-link relationships. Destination metadata is assigned by envd and the sandbox filesystem.

aenv download <sandbox-id> <remote-path> [local-path]

Download a file or directory from a sandbox through envd.

aenv download <sandbox-id> /workspace/result.txt ./result.txt
aenv download <sandbox-id> /workspace/result.txt
aenv download <sandbox-id> /workspace/result.txt ./output/
aenv download <sandbox-id> /workspace/project ./backup/
aenv download --user app --force <sandbox-id> result.txt ./result.txt

When the local path is omitted, the remote name is used in the current directory. When the local path names an existing directory or ends in /, the remote name is appended automatically. The resulting local parent directory must already exist. Directory downloads require an absolute remote path and do not support --user. Existing directories are merged; unrelated files remain, while conflicting files require --force. Each file is written through a temporary file and moved into place only after that file succeeds. Symbolic links and special files are rejected. Downloads do not preserve remote ownership or group, permissions (including executable bits), timestamps, ACLs, extended attributes, or hard-link relationships.

aenv list

List all sandboxes. Alias: aenv ls.

aenv list

aenv delete <sandbox-id>

Kill and delete a sandbox. Alias: aenv rm.

aenv delete <sandbox-id>
aenv rm <sandbox-id>

Volumes

aenv volume create <name>

Create an independent persistent volume. Volumes default to 65536 MiB (64 GiB) and exclusive mode.

aenv volume create workspace
aenv volume create models --mode ro --image ghcr.io/example/models:latest
aenv volume create job-workspace --from-volume workspace

We recommend creating an exclusive fork for each sandbox instead of mounting a shared writable volume directly:

aenv volume create job-data --mode exclusive --from-volume dataset-base
aenv start ubuntu --volume /workspace/data=job-data

See Volumes for access-mode semantics, lifecycle behavior, automatic sandbox fork and snapshot handling, and complete CLI and HTTP API examples.

Inspect and delete volumes

aenv volume list
aenv volume list --output json
aenv volume inspect job-data
aenv volume delete job-data

A mounted volume cannot be deleted. aenv volume ls is an alias for aenv volume list.

Snapshots

aenv snapshot create <sandbox-id>

Capture a persistent snapshot from a running sandbox. The snapshot can be used as a template to start new sandboxes with aenv start.

aenv snapshot create <sandbox-id>
aenv snapshot create <sandbox-id> --name my-base

When source-registry image publication is enabled on the server, the command also prints the published OverlayBD-native image reference on an Image: line; that tag can be used directly as a userImage.

aenv snapshot list

List persistent snapshots. Alias: aenv snapshot ls, aenv snap ls.

aenv snapshot list
aenv snapshot list --sandbox-id <sandbox-id>

The table output includes an IMAGE REF column (- when no image was published); JSON output includes the optional imageRef field.

To delete a snapshot, use aenv template delete <snapshot-id> or aenv template delete <name> — snapshots share the same underlying store as templates and are deleted through the same command.

Shell completion

aenv completion <shell> prints a shell-completion registration script for the aenv CLI to stdout. Supported shells: bash, zsh, and fish.

The script registers a completion function that calls back into the aenv binary at completion time (COMPLETE=<shell> aenv ...), so completion logic always matches the installed CLI version — upgrading aenv does not require regenerating the script. The script invokes aenv by name, so the binary must be on your PATH.

Generate and install a script

Bash

mkdir -p ~/.local/share/bash-completion/completions
aenv completion bash > ~/.local/share/bash-completion/completions/aenv

The Bash completion file is loaded on demand by bash-completion. This requires bash-completion to be installed and initialized in the current shell.

Zsh

mkdir -p ~/.local/share/zsh/site-functions
aenv completion zsh > ~/.local/share/zsh/site-functions/_aenv

Zsh loads completion functions from directories listed in fpath. ~/.local/share/zsh/site-functions is not included in fpath by default on all systems. Add the following lines to ~/.zshrc before any existing compinit invocation:

fpath=(~/.local/share/zsh/site-functions $fpath)
autoload -Uz compinit
compinit

Fish

mkdir -p ~/.config/fish/completions
aenv completion fish > ~/.config/fish/completions/aenv.fish

Activate without installing

To test completion for the current shell session without saving a generated file:

source <(aenv completion bash)         # Bash
eval "$(aenv completion zsh)"         # Zsh
aenv completion fish | source          # Fish

What completion covers

Static completion covers the full CLI surface:

aenv <TAB>                       # top-level commands
aenv snapshot <TAB>              # nested subcommands (create, list, ...)
aenv list --output <TAB>         # enum values: table, json
aenv build ./<TAB>               # local path arguments

Commands that take a sandbox ID also complete live sandbox IDs dynamically, filtered to the states each command accepts:

aenv resume <TAB>                # paused sandbox IDs
aenv exec <TAB>                  # running sandbox IDs

Where the shell supports it, candidates carry a description with the sandbox’s template and state.

Dynamic lookup is best-effort: it uses short timeouts (500 ms connect, 1 s request) and silently returns no candidates when credentials, the server, or the network are unavailable. Static command and flag completion keeps working in that case, and no diagnostic output is written to your command line.

Docker (Single Node)

Run a single AgentENV node in a Docker container. This avoids installing the Rust toolchain on the host but still requires /dev/kvm.

Prerequisites

• Linux kernel 6.8+
• /dev/kvm access for Firecracker microVM execution
• Docker

If the server does not support standard KVM, follow PVM Deployment for the required host setup and PVM image.

Build

Option A — Pre-built Image

docker pull ghcr.io/kvcache-ai/aenv-server:latest

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/docker-setup.sh | sudo bash

Option B — Build from Source

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV
sudo bash scripts/docker-setup.sh
docker build -f deploy/docker/Dockerfile.agentenv -t aenv:latest .

To use a regional apt mirror for both build and runtime stages, pass a base URL that contains debian, debian-security, and ubuntu mirror paths:

docker build \
  --build-arg APT_MIRROR_BASE=https://mirrors.example.com \
  -f deploy/docker/Dockerfile.agentenv \
  -t aenv:latest .

Run

docker run --rm -it \
  --name aenv-server \
  --device /dev/kvm --privileged -v /dev:/dev \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest   # or aenv:latest if built from source

The --privileged flag is required for Firecracker’s network namespace operations (veth pairs, iptables). The server auto-downloads runtime assets on first start and is accessible at http://127.0.0.1:8000 once ready.

On normal startup, the server generates the API key inside the container at /workspace/env/secrets/api-key. Read it while the container is running with:

docker exec aenv-server cat /workspace/env/secrets/api-key

Removing the container also removes this generated key. Supply an explicit AENV_API_KEY or a secret at /run/secrets/api-key when the key must remain stable across container replacements.

Verify

curl http://127.0.0.1:8000/health

Docker Compose (Multi-Node Simulation)

Run a full multi-node stack on a single host using Docker Compose. This simulates a production-like topology with a gateway, scheduler, and multiple AgentENV backend nodes.

For a real multi-machine deployment without Kubernetes, see Static Multi-Node.

Prerequisites

• Linux kernel 6.8+
• /dev/kvm access (passed into the runtime containers)
• Docker Engine with Docker Compose v2 (docker compose version)
• curl for the verification commands

The checked-in Compose setup uses standard KVM. If the host does not support it, read PVM Deployment before adapting the runtime image and host configuration.

Clone the Repository

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV

Start the Cluster

sudo bash scripts/docker-setup.sh
make deploy-up

make deploy-up builds the runtime, gateway, and scheduler images with Docker Compose before starting the stack. The Rust and Go toolchains are installed in the image build stages, so they are not required on the host.

Runtime nodes mount the host’s upstream DNS configuration read-only for guest networks, including template builders. The make deploy-* targets select /run/systemd/resolve/resolv.conf when available, otherwise /etc/resolv.conf. Override HOST_RESOLV_CONF with an absolute path to a resolver file containing DNS servers reachable from guests; loopback addresses such as 127.0.0.53 and Docker’s 127.0.0.11 cannot be used by guests. Direct docker compose commands default to /run/systemd/resolve/resolv.conf; set HOST_RESOLV_CONF explicitly on hosts without that file. Containers retain Docker DNS for service discovery.

To build without starting, run make deploy-build. To start images that are already built, run make deploy-up-no-build:

# Build only
make deploy-build

# Start previously built images
make deploy-up-no-build

On first startup, the runtime nodes atomically generate one API key and sandbox access-token seed in the shared agentenv-auth volume. The gateway mounts that volume read-only and reads the API key; sandbox tokens are validated by the runtime nodes. Normal make deploy-down calls preserve both values.

To enable host-based sandbox data-plane URLs, set the shared sandbox proxy domain variable when starting the stack:

SANDBOX_PROXY_DOMAINS=sandbox.example.com \
make deploy-up

Compose passes this value to both the gateway routing allowlist and runtime nodes’ sandbox response metadata. The domain must resolve to the gateway, usually through wildcard DNS for *.sandbox.example.com.

Verify

# Health check via gateway
curl http://127.0.0.1:8000/health

# Authenticated cluster node snapshots via gateway
export AENV_API_KEY="$(docker compose -f deploy/docker-compose.yml exec -T agentenv-a \
  cat /workspace/env/secrets/api-key)"
curl -H "X-API-Key: ${AENV_API_KEY}" http://127.0.0.1:8000/nodes

# Health check from inside a backend container
docker compose -f deploy/docker-compose.yml exec -T agentenv-a \
  curl -fsS http://127.0.0.1:8000/health

Management Commands

make deploy-ps      # Show container status
make deploy-logs    # Stream logs from all services
make deploy-down    # Tear down the cluster

Removing Compose volumes with docker compose down -v also removes both secrets. The next startup generates new values, so existing clients and sandbox access tokens are invalidated.

To use an existing API key instead of the generated value, export AENV_API_KEY before starting the stack. Compose passes it to the gateway and both runtime nodes:

export AENV_API_KEY="e2b_..."
make deploy-up

Configuration

Container deployments use deploy/docker/config/default.json. Scheduler and backend node endpoints are configured for the Docker network.

The compose manifest also wires node heartbeat reporting from runtime nodes to scheduler:

• AENV_NODE_ID is set explicitly per node container (node-a, node-b).
• AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED=true enables scheduler heartbeat reporting.
• AENV_OBSERVABILITY_SCHEDULER_ENDPOINT is set to http://scheduler:9090.
• SANDBOX_PROXY_DOMAINS, when set, is passed through as both GATEWAY_SANDBOX_PROXY_DOMAINS and AENV_SANDBOX_PROXY_DOMAINS.

Static Multi-Node (Without Kubernetes)

Run AgentENV across multiple physical or virtual machines without Kubernetes. This deployment uses the Go Gateway and Scheduler with a statically configured runtime-node list.

Static discovery is appropriate when node membership changes infrequently. The Scheduler does not automatically register an unknown node from its heartbeat: each runtime node must appear in scheduler.nodes, and changing that list requires a Scheduler restart.

Architecture

This example co-locates the Gateway and Scheduler on 10.0.0.10 and runs two AgentENV runtime nodes:

flowchart LR
    client["Client"] -->|"HTTP / WebSocket"| gateway["Gateway<br/>10.0.0.10:8080"]
    gateway -->|"gRPC"| scheduler["Scheduler<br/>10.0.0.10:9090"]
    gateway -->|"HTTP proxy"| nodeA["Runtime node A<br/>10.0.0.21:8000"]
    gateway -->|"HTTP proxy"| nodeB["Runtime node B<br/>10.0.0.22:8000"]
    nodeA -.->|"heartbeat"| scheduler
    nodeB -.->|"heartbeat"| scheduler
    scheduler -.->|"placement and lookup"| gateway

AgentENV authenticates HTTP requests but does not encrypt them. Use private addresses, a VPN, or TLS termination before traffic crosses an untrusted network.

Prerequisites

On every runtime node:

• Linux kernel 6.8+
• /dev/kvm access
• root access for the AgentENV installation
• network reachability to the Scheduler
• shared storage across all runtime nodes, using either POSIXFS or OSS

On the control-plane host:

• Go 1.21 or later
• network reachability to every runtime node
• a checkout of the AgentENV repository

Allow the following TCP flows:

The examples keep metrics listeners on loopback. Open their ports separately if an external metrics collector needs them.

1. Install the runtime nodes

Generate one API key and one sandbox access-token seed through your normal secret-management channel. Use the API key on the gateway and every runtime node; use the seed only on runtime nodes:

export AENV_API_KEY="e2b_$(openssl rand -hex 32)"
export AENV_SANDBOX_ACCESS_TOKEN_HASH_SEED="$(openssl rand -hex 32)"

Run the installation on each runtime node:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh | sudo bash

Edit /etc/default/aenv on each machine without removing the paths written by the installer, and add both shared values before starting the services. A multi-node deployment must not let each node generate independent managed secrets.

Node A uses:

API_ADDR="0.0.0.0:8000"
AENV_NODE_ID="node-a"
AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED="true"
AENV_OBSERVABILITY_SCHEDULER_ENDPOINT="http://10.0.0.10:9090"

Node B uses the same values except for its unique node ID:

API_ADDR="0.0.0.0:8000"
AENV_NODE_ID="node-b"
AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED="true"
AENV_OBSERVABILITY_SCHEDULER_ENDPOINT="http://10.0.0.10:9090"

The AENV_NODE_ID values must exactly match the corresponding IDs in the Scheduler configuration below. Restart and verify each runtime:

sudo systemctl restart aenv
sudo systemctl status aenv
curl http://127.0.0.1:8000/health

2. Build and install the control-plane binaries

On the control-plane host:

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV
make -C services build

sudo install -m 0755 services/bin/scheduler /usr/local/bin/agentenv-scheduler
sudo install -m 0755 services/bin/gateway /usr/local/bin/agentenv-gateway
sudo useradd --system --no-create-home --shell /usr/sbin/nologin agentenv-control
sudo install -d -o root -g agentenv-control -m 0750 /etc/agentenv

Create /etc/agentenv/auth.env with the API key used on the runtime nodes:

sudo install -o root -g agentenv-control -m 0640 /dev/null /etc/agentenv/auth.env
sudoedit /etc/agentenv/auth.env

AENV_API_KEY=<same shared key>

If the agentenv-control account already exists, the useradd command reports that fact and can be skipped.

3. Configure static discovery

Create /etc/agentenv/control-plane.json:

{
  "log_level": "info",
  "log_format": "json",
  "scheduler": {
    "grpc_listen_addr": "0.0.0.0:9090",
    "metrics_listen_addr": "127.0.0.1:9101",
    "strategy": "round_robin",
    "report_ttl": "30s",
    "binding_ttl": "30s",
    "discovery": {
      "mode": "static"
    },
    "nodes": [
      {
        "id": "node-a",
        "endpoint": "http://10.0.0.21:8000"
      },
      {
        "id": "node-b",
        "endpoint": "http://10.0.0.22:8000"
      }
    ]
  },
  "gateway": {
    "http_listen_addr": "0.0.0.0:8080",
    "metrics_listen_addr": "127.0.0.1:9102",
    "scheduler_addr": "10.0.0.10:9090",
    "request_timeout": "90s",
    "forward_response_size": 4194304,
    "sandbox_proxy_domains": []
  }
}

Protect the configuration after editing it:

sudo chown root:agentenv-control /etc/agentenv/control-plane.json
sudo chmod 0640 /etc/agentenv/control-plane.json

Each node endpoint must be reachable from the Gateway. The Scheduler returns that endpoint to the Gateway when it selects a node.

4. Run the Scheduler and Gateway with systemd

Create /etc/systemd/system/agentenv-scheduler.service:

[Unit]
Description=AgentENV Scheduler
Wants=network-online.target
After=network-online.target

[Service]
User=agentenv-control
Group=agentenv-control
ExecStart=/usr/local/bin/agentenv-scheduler -config /etc/agentenv/control-plane.json
Restart=on-failure
RestartSec=5
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Create /etc/systemd/system/agentenv-gateway.service:

[Unit]
Description=AgentENV Gateway
Wants=network-online.target
After=network-online.target agentenv-scheduler.service

[Service]
User=agentenv-control
Group=agentenv-control
EnvironmentFile=/etc/agentenv/auth.env
ExecStart=/usr/local/bin/agentenv-gateway -config /etc/agentenv/control-plane.json
Restart=on-failure
RestartSec=5
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Start both services:

sudo systemctl daemon-reload
sudo systemctl enable --now agentenv-scheduler agentenv-gateway

5. Verify the cluster

Check every network hop before creating a sandbox:

# On the control-plane host
curl http://10.0.0.21:8000/health
curl http://10.0.0.22:8000/health
curl http://127.0.0.1:8080/health

# Wait for node heartbeats, then inspect the cluster through the Gateway
export AENV_API_KEY="$(sudo sed -n 's/^AENV_API_KEY=//p' /etc/agentenv/auth.env)"
curl -H "X-API-Key: ${AENV_API_KEY}" http://127.0.0.1:8080/nodes

The node list should contain node-a and node-b. Point clients at the Gateway, not directly at a runtime node:

aenv auth
# AENV server URL: http://10.0.0.10:8080
# API key: <the same shared key>

Sandbox create, list, lifecycle, and data-plane requests can then be routed through the Gateway.

Operations

Follow service logs:

sudo journalctl -u agentenv-scheduler -f
sudo journalctl -u agentenv-gateway -f

# On a runtime node
sudo journalctl -u aenv -f

To add, remove, rename, or change the endpoint of a static node:

1. Update scheduler.nodes in /etc/agentenv/control-plane.json.
2. Ensure the runtime’s AENV_NODE_ID matches its configured ID.
3. Restart the Scheduler.

sudo systemctl restart agentenv-scheduler

Restarting the Scheduler temporarily interrupts routing that depends on its in-memory state. Runtime heartbeats repopulate observed sandbox assignments after the Scheduler comes back.

Troubleshooting

A runtime is healthy but absent from /nodes

• Confirm that AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED is true.
• Include the http:// scheme in AENV_OBSERVABILITY_SCHEDULER_ENDPOINT.
• Verify that AENV_NODE_ID exactly matches a configured scheduler.nodes[].id.
• Check that the runtime can reach Scheduler port 9090.
• Inspect journalctl -u aenv for heartbeat rejection or connection errors.

Heartbeats from IDs not present in the static node list are rejected; they do not register new nodes.

The Gateway returns no available nodes

• Verify both runtime health endpoints from the control-plane host.
• Verify the static endpoint addresses are reachable from the Gateway.
• Check Scheduler logs for expired heartbeat reports.
• Check that host firewalls allow runtime-to-Scheduler and Gateway-to-runtime traffic.

The Gateway cannot connect to the Scheduler

The Gateway’s scheduler_addr is a gRPC target and does not use an http:// prefix. Runtime heartbeat configuration uses a URL and does require that prefix:

gateway.scheduler_addr = 10.0.0.10:9090
AENV_OBSERVABILITY_SCHEDULER_ENDPOINT = http://10.0.0.10:9090

Kubernetes (Multi-Node)

Deploy AgentENV across a Kubernetes cluster with a gateway, scheduler, and runtime nodes on every worker.

Architecture

Why a DaemonSet for Runtime Nodes

• Each Pod needs host-local access to /dev/kvm
• Sandbox networking uses host iptables and network namespaces
• Runtime assets and committed snapshot state are cached per-host at /var/lib/aenv

Prerequisites

• Kubernetes worker nodes with Linux kernel 6.8+
• /dev/kvm access on every runtime worker
• Runtime Pods run privileged
• Docker
• build-essential (sudo apt install -y build-essential)
• kubectl with Kustomize support
• shared storage across all runtime nodes, using either POSIXFS or OSS

The provided manifests use standard KVM. To prepare a separate PVM node pool when standard KVM is unavailable, see PVM Deployment.

Clone the Repository

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV

Build Container Images

make k8s-build

This builds three images: agentenv-runtime:latest, agentenv-gateway:latest, and agentenv-scheduler:latest.

Configure the Access-Token Seed (Optional)

See Authentication for the shared sandbox access-token seed configuration.

Deploy

# Run on each worker node before deploying
sudo bash scripts/docker-setup.sh

# Render manifests (preview)
make k8s-render

# Apply to cluster
make k8s-apply

make k8s-apply generates a 256-bit API key on the first deployment and stores it in Secret/agentenv-auth. Later applies reuse it. Read the key locally when configuring clients:

kubectl -n agentenv-system get secret agentenv-auth \
  -o go-template='{{index .data "AENV_API_KEY" | base64decode}}{{"\n"}}'

Set AENV_API_KEY when applying to supply your own value. A standalone make k8s-render uses an invalid REDACTED placeholder so preview output never contains a deployable API key. The optional runtime seed keeps its existing agentenv-runtime-secrets contract described in Authentication.

To enable host-based sandbox data-plane URLs, set the shared sandbox proxy domain variable when rendering or applying manifests:

SANDBOX_PROXY_DOMAINS=sandbox.example.com make k8s-apply

The helper writes this value into the generated ConfigMap and applies it to both the gateway routing allowlist and runtime nodes’ sandbox response metadata. The domain must resolve to the gateway Ingress or LoadBalancer, usually through wildcard DNS for *.sandbox.example.com.

The default overlay is deploy/k8s/overlays/default, targeting the agentenv-system namespace. The gateway is exposed as ClusterIP by default. Add your own Ingress or LoadBalancer for external access.

The make targets build a temporary Kustomize context so runtime Pods mount the repository’s config/default.toml rather than a separate checked-in copy.

The runtime DaemonSet injects scheduler-report wiring for each node Pod:

• AENV_UBLK_DAEMON_BINARY_PATH=/usr/local/bin/uvm-ublk-daemon so the Pod uses the uvm-ublk-daemon binary included in the runtime image
• AENV_NODE_ID from Pod metadata name (metadata.name)
• AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED=true
• AENV_OBSERVABILITY_SCHEDULER_ENDPOINT=http://agentenv-scheduler:9090
• AENV_SANDBOX_PROXY_DOMAINS from the shared sandbox proxy ConfigMap

The P2P listen address must be reachable Pod-to-Pod; use a concrete container port or a Pod-reachable address if your cluster policy does not allow dialing ephemeral ports.

Operations

# Rollout restart all workloads
make k8s-redeploy

# Delete all resources
make k8s-delete

Local Development (k3s)

A dedicated local-dev overlay mounts the repository’s env/ directory directly into the DaemonSet at /workspace/env, avoiding runtime asset copies:

The apply helper provisions the same generated API key used by the default overlay. The local development overlay retains its fixed test-only runtime seed; do not reuse that seed outside local development.

make k8s-build
make k8s-load-dev       # Import images into k3s/containerd
make k8s-render-dev     # Preview manifests
make k8s-apply-dev      # Apply to cluster
make k8s-refresh-dev    # Build + load + rollout restart (all-in-one)

Service Discovery

The scheduler watches EndpointSlices for the headless agentenv-nodes Service and watches Pods for optional label-based discovery policy. It schedules only serving, non-terminating DaemonSet Pods. Pods matching scheduler.discovery.kubernetes.no_schedule_pod_selector stay discoverable as lingering/no-schedule nodes, while Pods matching scheduler.discovery.kubernetes.ignore_pod_selector are excluded. Both IPv4 and IPv6 endpoint addresses are supported.

Sandbox bindings remain in-memory, so the scheduler should run as a single replica. Bindings are lost on restart.

Manual Compile (Single Node)

Run AgentENV directly from source on a single Linux host, useful for development and testing.

If you want to skip building from source, see Quick Start.

Prerequisites

• Linux kernel 6.8+
• /dev/kvm access for Firecracker microVM execution
• Rust toolchain (stable) — install via rustup
• sudo access

If the server does not support standard KVM, follow PVM Deployment instead of this guide.

Clone the Repository

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV

Build

# Debug build
make

# Release build (recommended for production)
make release

Start the Server

Start the server. On first normal startup it generates an API key under $AENV_HOME/secrets/api-key and reuses it on later starts:

# Debug build
API_ADDR=0.0.0.0:8000 make start-server

# Release build
API_ADDR=0.0.0.0:8000 make start-server-release

The server auto-downloads runtime assets (Firecracker binary, kernel, rootfs) on first start. Once ready, it listens at http://127.0.0.1:8000. Read the generated key before making authenticated requests:

export AENV_API_KEY="$(cat "${AENV_HOME_PATH:-/var/lib/aenv}/secrets/api-key")"

Verify

curl http://127.0.0.1:8000/health
curl -H "X-API-Key: ${AENV_API_KEY}" http://127.0.0.1:8000/sandboxes

HTTP does not protect the key in transit. Use a trusted network, VPN, or TLS-terminating reverse proxy for remote clients.

Configuration

The server reads config/default.toml by default. Override with:

AENV_CONFIG_PATH=/path/to/config.toml make start-server

See Configuration Reference for all settings.

PVM Deployment

Note

Use this guide when standard KVM is unavailable, which commonly happens on cloud VMs where nested virtualization is not exposed. If standard KVM already works, use the Quick Start instead.

Warning

This feature is EXPERIMENTAL. The PVM feature has not yet been merged into the mainline Linux kernel, and the forked kernel may not receive the same level of testing and security updates as the mainline kernel.

PVM, originally proposed in the paper PVM: Efficient Shadow Paging for Deploying Secure Containers in Cloud-native Environment, is an alternative virtualization mode that can provide the KVM-compatible interface required by AgentENV without relying on conventional nested virtualization. After the PVM host environment is installed, AgentENV still uses /dev/kvm to create Firecracker microVMs.

Compared with a normal KVM deployment, a PVM deployment adds two host-level steps:

1. Install and boot a PVM-capable host kernel.
2. Load the PVM virtualization module before starting AgentENV.

AgentENV then uses its PVM-specific Firecracker and guest-kernel artifacts.

Before You Begin

PVM is not enabled by changing only an AgentENV configuration value. The host must first be prepared with a compatible PVM kernel.

You need:

• An x86_64 Linux server.
• Root access.
• Permission to install a host kernel and reboot the server.
• A DEB-based or RPM-based Linux distribution supported by the published PVM host-kernel packages, or the ability to build the kernel from source.
• Linux kernel 6.8 or newer for the remaining AgentENV requirements.

AgentENV does not replace the running host kernel automatically. Prebuilt PVM host-kernel packages are published separately in the kvcache-ai/linux releases. You must install the appropriate package, reboot into that kernel, and verify the PVM module before installing AgentENV.

Warning

Before changing kernels on a production server, confirm that you have console access or another recovery path in case the new kernel does not boot.

Host and Guest Kernel Compatibility

The PVM host kernel and the kernel running inside the AgentENV microVM must use compatible PVM ABIs. An incompatible pair may prevent the guest from booting or cause unexpected runtime failures.

For the most predictable setup, use host and guest kernels built from the same PVM kernel version. The PVM guest kernel packaged by AgentENV is based on the pvm-612 branch of virt-pvm/linux, at Linux version 6.12.33. The matching prebuilt host packages are published in kvcache-ai/linux release pvm-kernel-6.12.33.

The host and guest require different kernel configuration options:

• Host kernel: enable CONFIG_KVM_PVM=m.
• Guest kernel: enable CONFIG_PVM_GUEST.

How PVM Fits into AgentENV

An AgentENV node runs in exactly one virtualization mode:

The modes are intentionally isolated:

• Dependency provisioning installs only the selected Firecracker and guest kernel.
• Snapshots record the mode in which they were captured.
• Persisted paused sandboxes record their mode.
• A node refuses to restore state created in the other mode.

Do not point KVM and PVM nodes at the same persisted-sandbox directory. If nodes share a snapshot repository, ensure workloads resume only on nodes using the mode in which the snapshot was created.

Step 1: Install a PVM-Capable Host Kernel

Use the package format for your distribution. AgentENV only requires the kernel image and modules package. The separately published headers/development package is not required unless you need to build external kernel modules on the host.

Debian and Ubuntu

Download the kernel image:

curl -fLO \
  https://github.com/kvcache-ai/linux/releases/download/pvm-kernel-6.12.33/linux-image-6.12.33_6.12.33-7_amd64.deb

Install it and refresh the bootloader:

sudo dpkg -i linux-image-6.12.33_6.12.33-7_amd64.deb
sudo update-grub

If another installed kernel has a higher version, select Linux 6.12.33 from the bootloader’s advanced options or configure it as the default boot entry before rebooting.

RPM-Based Distributions (Fedora, RHEL, CentOS, TencentOS)

Download the kernel package:

curl -fLO \
  https://github.com/kvcache-ai/linux/releases/download/pvm-kernel-6.12.33/kernel-6.12.33_g91e9c9be4472-2.x86_64.rpm

Install it:

sudo rpm -ivh --oldpackage kernel-6.12.33_g91e9c9be4472-2.x86_64.rpm

On systems using grubby, select the installed PVM kernel:

sudo grubby --set-default /boot/vmlinuz-6.12.33-g91e9c9be4472
sudo grubby --default-kernel

Build from Source

If the published packages are not compatible with the distribution, build the host kernel from the pvm-612 branch. Enable CONFIG_KVM_PVM=m, install the kernel and modules, and configure the bootloader according to the distribution’s kernel-build documentation.

Reboot and Verify

Reboot the host:

sudo reboot

After reconnecting, confirm that the expected kernel is active:

uname -r

Expected output:

# DEB package
6.12.33

# RPM package
6.12.33-g91e9c9be4472

If uname -r reports the previous kernel, update the bootloader selection and reboot again before continuing.

Step 2: Load and Verify the PVM Module

Load the module:

sudo modprobe kvm_pvm

Verify that it is loaded:

lsmod | grep kvm_pvm
test -d /sys/module/kvm_pvm

Verify that the KVM-compatible device is now available:

ls -l /dev/kvm

The important result of the host setup is:

• /sys/module/kvm_pvm exists.
• /dev/kvm exists.
• The AgentENV runtime account can open /dev/kvm for reading and writing.

After confirming that the module loads successfully, configure the kernel to load it automatically at boot (different distributions may use different directories):

echo kvm_pvm | sudo tee /etc/modules-load.d/kvm-pvm.conf

Step 3: Install AgentENV in PVM Mode

Option A: Install Script

On Ubuntu 24.04:

curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh \
  | sudo AENV_VIRTUALIZATION_MODE=pvm bash

sudo systemctl start aenv

The installer:

• Downloads aenv-server-linux-x86_64-pvm.tar.gz.
• Installs the PVM Firecracker and guest-kernel artifacts.
• Writes AENV_VIRTUALIZATION_MODE="pvm" to /etc/default/aenv.
• Configures the service account’s access to /dev/kvm.

It does not install or load the PVM host kernel or kvm_pvm.

Option B: Docker

Use the dedicated PVM image:

docker pull ghcr.io/kvcache-ai/aenv-server:latest-pvm

docker run --rm -it --name aenv-server \
  --device /dev/kvm \
  --privileged \
  -v /dev:/dev \
  -p 8000:8000 \
  ghcr.io/kvcache-ai/aenv-server:latest-pvm

The image sets AENV_VIRTUALIZATION_MODE=pvm by default and contains only the PVM runtime artifacts.

Option C: Build from Source

Select PVM for both dependency provisioning and server startup:

export AENV_VIRTUALIZATION_MODE=pvm

cargo run --bin server -- --setup-only
make start-server

The server generates and persists the API key under $AENV_HOME/secrets/api-key on its first normal startup.

You can also set the mode in the TOML configuration:

virtualization_mode = "pvm"

The environment variable takes precedence over the TOML value.

Memory snapshot dirty-page tracking is enabled by default for KVM and automatically disabled in PVM mode because this combination has not been tested.

To build a PVM Docker image:

docker build \
  --build-arg AENV_VIRTUALIZATION_MODE=pvm \
  -f deploy/docker/Dockerfile.agentenv \
  -t aenv:pvm .

Step 4: Verify AgentENV

For an install-script deployment, verify the persisted mode:

grep AENV_VIRTUALIZATION_MODE /etc/default/aenv

Expected output:

AENV_VIRTUALIZATION_MODE="pvm"

Inspect startup status and logs:

sudo systemctl status aenv
sudo journalctl -u aenv -f

Verify the API:

curl http://127.0.0.1:8000/health

Once the server is healthy, template creation and sandbox operations are the same as in the standard Quick Start.

Multi-Node Deployment

Use a consistent virtualization mode within a runtime pool.

For Docker Compose:

• Use the PVM runtime image.
• Export AENV_VIRTUALIZATION_MODE=pvm.
• Prepare the host before starting the Compose stack.

For Kubernetes:

• Label x86_64 worker nodes that boot the PVM kernel.
• Load kvm_pvm on each selected node.
• Use the PVM AgentENV image.
• Set AENV_VIRTUALIZATION_MODE=pvm in the runtime DaemonSet.
• Add a node selector or affinity rule so PVM Pods cannot run on KVM nodes.

Avoid mixing KVM and PVM nodes in a pool that schedules from a shared set of snapshots unless the scheduler also enforces virtualization-mode affinity.

Troubleshooting

PVM virtualization mode is only supported on x86_64 hosts

The current machine architecture is unsupported. Deploy the PVM node on an x86_64 server or use standard KVM.

PVM mode requires the kvm_pvm host module to be loaded

The AgentENV mode is set to PVM, but the host module is not active.

Check the running kernel and try loading the module:

uname -r
sudo modprobe kvm_pvm
lsmod | grep kvm_pvm

If modprobe reports that the module cannot be found, the server is not running a compatible PVM host kernel.

/dev/kvm is missing after loading kvm_pvm

Confirm that kvm_pvm loaded successfully and review the kernel log:

lsmod | grep kvm_pvm
sudo dmesg | tail -n 100

If the module is loaded but /dev/kvm is still absent, verify the PVM kernel installation and boot parameters with the kernel provider.

/dev/kvm is not accessible

Check ownership and service-account groups:

ls -l /dev/kvm
id aenv

After changing group membership, restart the service or user session.

AgentENV downloads or uses standard KVM artifacts

Confirm the mode is present in the service environment:

grep AENV_VIRTUALIZATION_MODE /etc/default/aenv

Then rerun provisioning:

sudo AENV_VIRTUALIZATION_MODE=pvm \
  AENV_CONFIG_PATH=/var/lib/aenv/config/config.toml \
  AENV_HOME_PATH=/var/lib/aenv \
  /usr/local/bin/server --setup-only

Snapshot or paused-sandbox mode mismatch

The persisted state was created in the other virtualization mode. Restore it on a node using its original mode, or rebuild the workload and capture a new snapshot in the target mode.

Run Python Code

This quickstart imports the official Python image as an AgentENV template, starts a sandbox, and executes Python code inside it. The Python program is included directly in the command; no sample project or source file is needed.

Before starting, deploy AgentENV and configure the aenv CLI as described in Quick Start.

1. Create a Python Template

Import the official Python 3.12 image and wait for the template to become ready:

aenv pull python:3.12-slim --name python

2. Start a Sandbox

Start the template in detached mode and capture the generated sandbox ID:

SANDBOX_ID="$(aenv start python -d)"

3. Execute Python

Run this program inside the sandbox:

aenv exec "$SANDBOX_ID" python -c '
from statistics import mean

temperatures = [21.5, 23.0, 22.4, 24.1]
print(f"average temperature: {mean(temperatures):.2f} C")
'

Expected output:

average temperature: 22.75 C

4. Delete the Sandbox and the Template

aenv delete "$SANDBOX_ID"
aenv template delete python

Train Terminal-Bench-2 with Miles

AgentENV integrates with Miles as a self-hosted sandbox backend for agentic reinforcement learning. Miles is a high-performance reinforcement learning framework for large-scale model post-training.

This example follows Miles’ OpenEnv recipe to train GLM-4.7-Flash with GRPO on Terminal-Bench-2. OpenEnv provides the task interaction and evaluation interface used by the recipe, while AgentENV runs the isolated sandbox for each episode. Terminal-Bench-2 is a benchmark suite of terminal-based software tasks rather than a conventional prompt-only dataset: each task includes an environment, an instruction, and tests that determine whether the agent completed the task.

1. Deploy AgentENV

First deploy and authenticate with an AgentENV server by following the Quick Start.

When starting AgentENV with docker run, publish ports 8000 and 80 and set AENV_SANDBOX_PROXY_DOMAINS to an sslip.io domain for the AgentENV host:

docker run -d --name aenv --privileged -v /dev:/dev \
  -p 8000:8000 -p 80:8000 \
  -e AENV_SANDBOX_PROXY_DOMAINS=<ip-with-dashes>.sslip.io \
  ghcr.io/kvcache-ai/aenv-server:latest

Replace <ip-with-dashes> with the AgentENV host IP, replacing every dot with a dash.

2. Point Miles at AgentENV

pip install e2b

export E2B_API_URL=http://<server>:8000
export E2B_SANDBOX_URL=http://<server>:8000

# Export your api key fetched in the last step.
export E2B_API_KEY=<your-api-key>

# Per-sandbox URLs use HTTPS by default. This deployment uses plain HTTP.
export OPENENV_E2B_URL_SCHEME=http

The API key authenticates requests but does not encrypt them; keep this plain-HTTP setup on a trusted network.

3. Prepare Miles, OpenEnv, and Terminal-Bench-2

Run this tutorial inside an existing Miles training environment, as required by the upstream recipe. Cloning Miles below provides the recipe scripts; it does not install the CUDA, model-serving, or training stack.

git clone https://github.com/radixark/miles.git
git clone https://github.com/huggingface/OpenEnv.git
git clone --depth 1 https://github.com/laude-institute/terminal-bench-2.git

pip install -e ./OpenEnv/envs/tbench2_env

python ./miles/examples/experimental/openenv/make_tbench2_data.py \
  --tasks_dir ./terminal-bench-2 \
  --output /root/tbench2_train.jsonl

Add --n 8 to make_tbench2_data.py for a small smoke subset.

4. Train

export OPENENV_TB2_TASKS_DIR="$(realpath ./terminal-bench-2)"
OPENENV_SANDBOX_BACKEND=e2b \
  python ./miles/examples/experimental/openenv/run-openenv-tbench2.py

The launcher creates one AgentENV microVM per episode, and returns the task’s canonical test result as the GRPO reward. e2b uses the E2B-compatible backend pointed at the AgentENV endpoints.

For the complete training configuration, provider options, and operational notes, see the upstream Miles OpenEnv Terminal-Bench-2 recipe.

How AgentENV Works

AgentENV runs AI agents and their tools inside isolated, snapshot-capable Linux environments. Each environment, called a sandbox, is backed by a Firecracker microVM with its own kernel, filesystem, processes, and network stack.

User Workflow

flowchart LR
    image["OCI image"] -->|"aenv pull"| template["Template"]
    dockerfile["Dockerfile"] -->|"aenv build"| template
    image -->|"aenv start --cold"| sandbox
    template -->|"aenv start"| sandbox["Running<br/>sandbox"]
    sandbox -->|"aenv connect / aenv exec"| work["Run code, tools, and services"]
    work --> sandbox
    sandbox -->|"aenv snapshot create"| snapshot["Snapshot"]
    snapshot -->|"aenv start"| newSandbox["New sandbox"]

A typical workflow is:

1. Create a reusable template from an OCI image or Dockerfile.
2. Start an isolated sandbox from the template, or cold start one directly from an OCI image.
3. Run command in the sandbox.
4. Pause the sandbox when you want to preserve the same sandbox for later, or create a snapshot when you want a reusable checkpoint that can launch new sandboxes.
5. Delete sandboxes and snapshots when they are no longer needed.

Templates, Sandboxes, and Snapshots

These three concepts describe the reusable and running forms of an environment:

• Building a template produces a snapshot-backed starting point.
• Starting a template or snapshot creates a sandbox.
• Capturing a running sandbox creates a snapshot without replacing the source sandbox.

System Overview

flowchart TD
    subgraph node[AgentENV Node]
        api["API<br/>(Axum)"] --> orchestrator["Orchestrator<br/>(lifecycle)"]
        orchestrator --> vm["Firecracker VM<br/>/dev/vda (rootfs)<br/>/dev/vdb (extra)"]
        vm --> block["Block Device Layer<br/>(overlaybd + ublk)"]
    end
    style node fill:transparent,stroke:gray

Request Flow

1. A client sends an HTTP request to the AgentENV API (for example, POST /sandboxes).
2. The API layer validates the request, checks authentication, and forwards it to the orchestrator.
3. The orchestrator manages the sandbox lifecycle: it creates a Firecracker VM, sets up networking, and attaches block devices.
4. The VM boots with a layered block device (overlaybd) that stacks read-only base image layers with a writable upper layer. Multiple sandboxes share the same base layers.
5. Inside the VM, an envd daemon handles command execution, file operations, and health reporting.
6. Clients interact with running sandboxes via the reverse proxy (/proxy, routing headers, or configured sandbox proxy domains), which forwards HTTP and WebSocket traffic to services inside the VM.

Continue with Templates, Sandboxes, and Snapshots for the commands and options for each workflow.

Authentication

AgentENV uses three credentials for three separate kinds of access:

These credentials are not interchangeable. The API key belongs to the AgentENV deployment; the other two tokens belong to an individual sandbox.

API Authentication

The API key authenticates requests that create and manage AgentENV resources, including templates, sandboxes, and snapshots:

curl http://127.0.0.1:8000/sandboxes \
  -H "X-API-Key: <api-key>"

Get the API Key

A runtime node checks these API-key sources in order:

1. AENV_API_KEY
2. /run/secrets/api-key
3. $AENV_HOME/secrets/api-key

The gateway checks only AENV_API_KEY and /run/secrets/api-key. The gateway and every runtime node in one deployment must use the same key.

Provide an API Key

To provide your own key, generate a value containing 32 to 256 URL-safe characters and make it available through one of the locations above. For example, generate an E2B-compatible key and set it through the environment:

export AENV_API_KEY="e2b_$(openssl rand -hex 32)"

Use the same explicitly generated key for the gateway and every runtime node in a multi-node deployment. Docker Compose can provide it through its shared managed-secret volume, while Kubernetes stores it in Secret/agentenv-auth. See the corresponding deployment guide for setup instructions.

Use an Automatically Generated API Key

If a runtime node finds no key in any of the three locations, normal server startup generates an E2B-compatible key and atomically stores it at $AENV_HOME/secrets/api-key. Read that file after the server starts and use its value when configuring the CLI or another API client.

Automatic generation is convenient for a normal single-node deployment. The gateway never generates a key, so a multi-node deployment must make one shared key available to the gateway and all runtime nodes.

Configure the CLI

Run aenv auth, enter the AgentENV server URL, and paste the API key obtained above. Press Enter to accept the default local URL. The API key input is hidden:

$ aenv auth
AENV server URL [http://localhost:8000]: http://localhost:8000
API key: <paste-api-key-here>
Credentials saved.

E2B-compatible SDKs read the same AgentENV API key from E2B_API_KEY:

export E2B_API_KEY="<paste-api-key-here>"

Unauthenticated Endpoints

GET /health and node GET /metrics are outside API-key authentication. The gateway exposes Prometheus metrics on a separate metrics listener. Protect these endpoints with the network and authentication controls used by your monitoring deployment.

Rotate the API Key

Changing AENV_API_KEY invalidates existing API clients but does not change the credentials of existing sandboxes.

Application Ingress Authentication

Application ingress is traffic sent through the AgentENV proxy to a service running inside a sandbox. It is independent from API and envd authentication.

With allowPublicTraffic: true, which is the default, application ingress does not require an AgentENV credential.

To create a private sandbox, set allowPublicTraffic: false. The creation response includes that sandbox’s trafficAccessToken; capture it for later proxy requests:

SANDBOX_RESPONSE=$(curl -sS -X POST http://127.0.0.1:8000/sandboxes \
  -H "X-API-Key: <api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "templateID": "my-template",
    "network": {
      "allowPublicTraffic": false
    }
  }')

SANDBOX_ID=$(printf '%s' "$SANDBOX_RESPONSE" | jq -r '.sandboxID')
TRAFFIC_ACCESS_TOKEN=$(printf '%s' "$SANDBOX_RESPONSE" | jq -r '.trafficAccessToken')

Send that token in e2b-traffic-access-token when accessing an application in the sandbox:

curl http://127.0.0.1:8000/proxy/ \
  -H "x-agentenv-sandbox-id: $SANDBOX_ID" \
  -H "x-agentenv-target-port: 8080" \
  -H "e2b-traffic-access-token: $TRAFFIC_ACCESS_TOKEN"

Each sandbox has its own token, and forked sandboxes receive independent credentials. AgentENV removes the traffic token before forwarding the request to the sandbox application. See Proxy for the complete proxy routing and access-control behavior.

Secure Sandbox Authentication

A secure sandbox requires an envdAccessToken for envd control operations, including interactive connections, command execution, and file upload or download. This does not protect services exposed by the sandbox; application ingress uses trafficAccessToken as described above.

Enable Secure Mode

Set secure: true in the sandbox creation request. For example, create a secure sandbox and capture the token returned in the response:

SANDBOX_RESPONSE=$(curl -sS -X POST http://127.0.0.1:8000/sandboxes \
  -H "X-API-Key: <api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "templateID": "my-template",
    "secure": true
  }')

SANDBOX_ID=$(printf '%s' "$SANDBOX_RESPONSE" | jq -r '.sandboxID')
ENVD_ACCESS_TOKEN=$(printf '%s' "$SANDBOX_RESPONSE" | jq -r '.envdAccessToken')

Use the envd port 49983, and send the token in X-Access-Token:

curl http://127.0.0.1:8000/proxy/health \
  -H "x-agentenv-sandbox-id: $SANDBOX_ID" \
  -H "x-agentenv-target-port: 49983" \
  -H "X-Access-Token: $ENVD_ACCESS_TOKEN"

The aenv CLI always requests secure sandbox authentication and handles the envd access token automatically:

aenv start <template-or-snapshot>

For connect, exec, upload, and download, it obtains the token through the authenticated AgentENV API and adds X-Access-Token to the subsequent envd request.

Secure-Mode Lifecycle

Secure mode and its credentials remain valid across pause, server restart, and resume. Forked sandboxes receive independent credentials.

Sandbox Access-Token Seed

AgentENV derives both trafficAccessToken and envdAccessToken from the sandbox identity and a random access-token seed. This seed is independent of the AgentENV API key.

By default, no manual configuration is required. When a seed is not configured, AgentENV generates one on first startup and persists it at $AENV_HOME/secrets/sandbox-access-token-hash-seed. Later startups reuse the same value.

To provide your own seed, generate one value:

ACCESS_TOKEN_SEED="$(openssl rand -hex 32)"

Then configure it using one of the following methods.

Set the environment variable before starting AgentENV:

export AENV_SANDBOX_ACCESS_TOKEN_HASH_SEED="$ACCESS_TOKEN_SEED"

Or set it in config/default.toml, or in the configuration file selected by AENV_CONFIG_PATH:

[sandbox]
access_token_hash_seed = "<generated-seed>"

Kubernetes deployments store the shared value under the sandbox-access-token-hash-seed key in Secret/agentenv-runtime-secrets.

Preserve the seed across upgrades. Changing it rotates both sandbox token types and requires all runtime nodes to be updated together.

Transport Security

Authentication does not encrypt traffic. Use HTTPS termination, a VPN, loopback, or a trusted private network to protect API keys and sandbox tokens in transit.

Templates

A template is a reusable starting point for launching sandboxes. Build or import it once, then use it to create sandboxes whenever you need the same software and configuration.

Create Your Template

There are two ways to create a template: aenv pull imports an OCI image directly, and aenv build runs Dockerfile instructions inside a temporary build sandbox.

Some defaults below come from your AgentENV config file. This is config/default.toml by default, or the file specified by AENV_CONFIG_PATH.

aenv pull

Pull an existing OCI image as a template and optionally give it a memorable name:

Usage:

aenv pull <image> [options]

Example:

aenv pull ubuntu:24.04
aenv pull ubuntu:24.04 --name my-base

--name is optional. Without it, AgentENV uses the image repository name. A template name can be used anywhere a template ID is accepted.

Env, WorkingDir, and User are automatically inherited from the OCI image config. See Runtime Configuration for the full field list.

aenv build

Build a Dockerfile with BuildKit inside a temporary microVM, then convert the result to OverlayBD and capture a template. The CLI and full AgentENV installers include a private aenv-buildctl client. Source builds can use --buildctl to select a compatible client. Docker and a staging registry are not required on the CLI machine. On Linux and macOS, the local buildctl connection uses a Unix socket in a directory accessible only to the current user.

aenv build <context> --name <name> [options]
# From the repository root:
aenv build . -f deploy/docker/Dockerfile.agentenv --name aenv

COPY and ADD resolve from the context directory, independently of the Dockerfile’s location. Local directories are supported; URL and stdin contexts are not supported. The final Dockerfile stage is always published. Select base images with FROM (or ARG used by FROM) and startup with ENTRYPOINT/CMD. --start-cmd and --ready-cmd override startup and readiness independently. There are no image, stage-selection, SSH, or builder-resource overrides in the build CLI.

Managed builder settings belong to the server configuration:

[template_build]
max_concurrent_builds = 4
builder_image = "docker.io/moby/buildkit:v0.33.0"
builder_cpu_count = 16
builder_memory_mb = 32768
cache_size_mb = 65536

Each node admits at most max_concurrent_builds managed builds, including preparation, image publication, and cleanup. Excess builder PUT requests return HTTP 429 without changing the waiting build; retry once capacity is available. Each build permits up to 8 simultaneous WebSocket tunnels, including pending connections and upgrades. Excess tunnel connections also return HTTP 429.

The 64 GiB disk is the persistent /var/lib/buildkit data volume, where image layers, build contexts, and cache mounts live. Its capacity applies when creating the cache; changing the setting does not resize an existing volume. These resources are separate from the resulting template’s CPU and memory. Dockerfile build requests reject a cache capacity above volume.max_size_mb; a smaller volume limit does not prevent the server from starting or serving other APIs.

BuildKit handles multi-stage builds, COPY, ADD, .dockerignore, cache mounts, and Dockerfile syntax. The CLI remains connected during the build, streams build progress, and exits nonzero on failure. The server provisions and releases an internal worker for each template build. The first Dockerfile build prepares a reusable builder snapshot in a private namespace of the configured snapshot repository. Nodes sharing that repository restore the same builder and attach a new cache volume before starting BuildKit. Concurrent first requests on a node share initialization. Simultaneous first builds on different nodes may both prepare a builder; subsequent builds reuse the published snapshot. Builder image, CPU, memory, virtualization mode, and readiness setup identify the reusable snapshot; changing those inputs prepares a new builder. The CLI uses only template and build IDs; workers are absent from public sandbox listings and endpoints. Cancellation and deadlines release the worker and discard its incomplete cache child. A hard client failure is covered by the build deadline, and server restart recovers unfinished builds and cache reservations from a durable journal. Failed cleanup is retained and retried every 30 seconds while the server runs, and cancellation retries use the same cleanup path. Active builds reject template deletion until cancellation or completion. An unreadable entry does not block recovery of other builds or prevent server startup. Once BuildKit succeeds, publication proceeds even if the CLI is interrupted; its status remains available:

aenv template watch my-template

aenv template watch has no timeout option. Stop the local watch with Ctrl-C; the remote build continues.

The node reads the completed image directly from the builder by SHA-256 digest. It verifies the transferred bytes and converts only missing layers, reusing the same content-addressed OverlayBD cache as registry image imports. The completed image never travels through the CLI. Only the final image configuration becomes template configuration; intermediate stages and build-time arguments are not template environment variables.

Publishing also boots the image and runs its startup command. An image can compile and convert successfully but fail this step if its entrypoint needs devices absent from the guest. For example, deploy/docker/Dockerfile.agentenv starts the host AgentENV server, which requires /dev/kvm; it cannot run as a template in a guest without nested KVM support. Use --start-cmd to select another startup command, or --start-cmd "" --ready-cmd true to capture without starting the image’s application or running its health check.

The guest image needs /bin/sh for envd process execution, including exec-form Dockerfile commands. Numeric USER values, including UIDs without an account and explicit UID/GID pairs, are preserved during startup and restore.

Caches are shared across template names and nodes using the configured snapshot repository. Each build clones the latest immutable cache seed into its own writable volume. Sequential builds inherit the preceding build’s accumulated instruction cache and RUN --mount=type=cache data. Concurrent builds can fork the same seed without waiting for each other; the last successfully published cache becomes the next seed. Sibling cache additions are not merged.

After image import, the node stops BuildKit, checkpoints and publishes its cache volume through the normal volume freeze and capture path, and stops the VM without saving its memory, rootfs, or device state. A failed shutdown or cache capture keeps the previous shared seed. Volume ownership remains held until the VM stops.

Cache volumes use normal volume publication, uploading only missing layers. Cache publication adds cleanup time, but a failed cache upload does not fail an otherwise successful template build or replace the previous seed. Old cache volumes are removed after active children release their leases. The shared cache record commits the new seed and pending retirements together, and cleanup retries retirements independently of individual builds. BuildKit’s garbage collector manages cache contents. Cache sharing uses the repository’s existing API-key trust boundary. Registry credentials come from the local BuildKit session and normal Docker credential configuration.

The BuildKit API extends the existing template/build lifecycle:

1. POST /v3/templates allocates template/build IDs using the existing request.
2. PUT /templates/{templateID}/builds/{buildID}/builder prepares the worker. It accepts optional startCmd, readyCmd, and timeout, and returns imageName. A build that has already started returns 409.
3. Poll GET /templates/{templateID}/builds/{buildID}/status until building; waiting means the worker is still preparing.
4. GET /templates/{templateID}/builds/{buildID}/builder opens the authenticated binary WebSocket for BuildKit. Use the returned name in --output type=image,name=<imageName>,oci-mediatypes=true.
5. The server observes successful BuildKit completion, verifies the image digest, and automatically imports and publishes the template. No client submission request is needed. Continue polling status until ready or error.
6. DELETE /templates/{templateID}/builds/{buildID}/builder cancels the build before publication. Once publication starts, cancellation returns 409 and the server finishes publication and releases the worker.

These endpoints require the API key. The gateway schedules worker preparation and binds subsequent requests to that node. Status polling falls back to the shared repository after the binding expires. The unique image name identifies this build among cached BuildKit history; a dropped connection is never treated as successful completion. Failed solves become error. Existing statuses are unchanged; building includes image import and publication.

Image import has a one-hour deadline; metadata is limited to 4 MiB per blob and images to 1024 layers and 64 GiB of compressed data. Worker connections can drain for up to ten seconds after image import before the worker is stopped. Existing declarative template endpoints retain their behavior.

By default, image ENTRYPOINT and CMD are combined for startup through /bin/sh -c. Dockerfile HEALTHCHECK supplies the readiness command before snapshot capture; shell checks honor Dockerfile SHELL, and HEALTHCHECK NONE disables the check. It uses AgentENV’s readiness polling and deadline, not Docker’s health-monitoring intervals or restart behavior. Without a check, startup uses the normal template readiness delay. --start-cmd and --ready-cmd (API fields startCmd and readyCmd) override these commands independently without modifying the image configuration. Images must satisfy AgentENV’s normal guest runtime requirements; EXPOSE and VOLUME are metadata, not Docker runtime services.

API and repository compatibility

Existing template creation, declarative build, status, listing, and deletion endpoints remain available. BuildKit adds PUT, GET, and DELETE on the builder resource beneath an existing build. It adds no competing template creation endpoint or image submission endpoint. Upgrade the gateway and all nodes serving build requests before using BuildKit.

Existing remote snapshots and volumes need no migration. Their catalog keys, artifact paths, and layer formats are unchanged. Builder snapshots use the separate template-build/builder namespace; the new cache head and retirement record uses template-build/cache-head.json. Cache volumes retain the normal volume format.

Startup metadata adds an optional shell field. Older records omit it, retain /bin/bash -lc behavior, and are written back without adding the field. New BuildKit templates use /bin/sh -c. Older servers can deserialize these records, but ignore the shell field and drop it when rewriting the record. Building a derived template on an older server therefore uses Bash and can fail for images without Bash or change shell behavior. Keep builds derived from BuildKit templates on upgraded nodes; rolling back does not provide full BuildKit template support.

Runtime Configuration

Both methods read the same set of OCI image config fields. For aenv pull, these come from the image config or flags; for aenv build, they are set by the corresponding Dockerfile instructions executed during the build. The following fields from the OCI image-spec config object are recognised:

Manage Templates

List templates

aenv template list        # alias: aenv template ls
aenv template list --output json

Displays all templates with their ID, name, build status, CPU, memory, disk size, and last-updated timestamp. --output accepts table or json. It defaults to table in an interactive terminal and json when output is piped or redirected.

List template builds

List the complete build history for one template:

curl -H 'X-API-Key: test-key' \
  http://127.0.0.1:8000/templates/<template-id>

Check a template alias

Check whether an alias exists and resolve it to a template ID:

curl -H 'X-API-Key: test-key' \
  http://127.0.0.1:8000/templates/aliases/<alias>

Delete a template

aenv template delete <template-id-or-name>   # alias: aenv template rm

Relationship to Snapshots

Templates are the API and UX layer. Snapshots are the durable runtime layer.

• A template build publishes one committed snapshot.
• A template ID or alias resolves to one committed snapshot.
• A sandbox created from a template resumes from that snapshot.

If you want the storage and runtime model underneath templates, see Snapshots.

Sandboxes

A sandbox is an isolated Firecracker microVM with its own Linux kernel, filesystem, processes, and network stack. It is the environment where you run code, use tools, modify files, and start services.

Lifecycle

stateDiagram-v2
    [*] --> Creating
    Creating --> Running
    Running --> Pausing
    Pausing --> Paused
    Paused --> Resuming
    Resuming --> Running
    Running --> Snapshotting
    Snapshotting --> Running
    Running --> Forking
    Forking --> Running
    Running --> Killing
    Killing --> [*]

Starting a Sandbox

You can start a sandbox from a reusable template or snapshot, or cold start one directly from an OCI image.

From a Template or Snapshot

Pass either a template/snapshot alias or its ID:

Usage:

aenv start <template-or-snapshot> [options]

Example:

# Start by alias
aenv start my-python-template

# Start by ID
aenv start 018f0d93-aaaa-bbbb-cccc-0123456789ab

Warm-start options:

Without --detach, aenv start waits for the sandbox to become ready and then attaches an interactive shell. CPU, memory, and disk settings are inherited from the template or snapshot and cannot be overridden on a warm start. The CLI always enables secure sandbox authentication and manages the envd access token automatically; see Authentication.

To retrieve the current state and configuration of one sandbox, use the HTTP API:

curl -H 'X-API-Key: test-key' \
  http://127.0.0.1:8000/sandboxes/<sandbox-id>

Cold Start from an OCI Image

A cold start resolves an OCI image directly and prepares a fresh writable root filesystem at runtime:

Usage:

aenv start --cold <image> [options]

Example:

aenv start --cold ubuntu:24.04
aenv start --cold ubuntu:24.04 --cpu 4 --memory 4096 --disk-size-mb 65536

Cold-start options:

Cold-started sandboxes also use secure sandbox authentication by default.

The AgentENV config file is config/default.toml by default, or the file specified by AENV_CONFIG_PATH.

An OverlayBD-native image can start without downloading the complete image first; its filesystem data is loaded from the registry on demand. See On-Demand Loading.

Growth of the disk size is allowed by default. Shrinking below the source image size requires ublk.overlaybd.allow_shrink = true in your AgentENV config file. Resizing applies only when creating a fresh writable root filesystem, not to read-only images, images with an existing upper layer, or snapshot resume. Sandbox responses report the effective size as diskSizeMB.

Working with Sandboxes

Connect to a Sandbox

aenv connect opens an interactive shell inside the sandbox and attaches your terminal. aenv cn is its short alias:

aenv connect <sandbox-id>
aenv cn <sandbox-id>

If a sandbox is paused, aenv connect will automatically resume it.

Execute a Command

aenv exec runs one non-interactive command, streams its output to your local terminal, and exits with the remote command’s exit code. It does not attach an interactive shell. Flags intended for the remote command that collide with aenv’s own flags can be escaped with a leading --.

aenv exec <sandbox-id> ls -la /
aenv exec <sandbox-id> -- command-with-aenv-like-flags --timeout 10

Upload Files

aenv upload copies a local file or directory into a running sandbox:

Usage:

aenv upload <sandbox-id> <local-path> <remote-path> [options]

Example:

aenv upload 018f0d93-aaaa-bbbb-cccc-0123456789ab ./config.json /workspace/config.json

Download Files

aenv download copies a file or directory from a running sandbox to your local machine:

Usage:

aenv download <sandbox-id> <remote-path> [local-path] [options]

Example:

aenv download 018f0d93-aaaa-bbbb-cccc-0123456789ab /workspace/result.txt ./result.txt

Pause and Resume

Pausing saves the sandbox’s current runtime state and stops its microVM. While it is paused, programs inside it do not run, services do not handle requests, and the sandbox releases its CPU and memory resources. Its saved state remains in storage so the same sandbox can be resumed later.

After resume, the filesystem, running processes, environment variables, and in-memory data are restored to the state captured at pause time. Programs continue from that saved state instead of starting again from the beginning.

aenv pause <sandbox-id>
aenv resume <sandbox-id>
aenv resume <sandbox-id> --timeout 600

aenv resume accepts --timeout <seconds>, which defaults to 300 seconds and sets the new TTL from resume time.

By default, the sandbox automatically pauses when it reaches its TTL. See Auto-Eviction for how the deadline is set and how to delete instead of pause.

Persistent Snapshots

A snapshot is a durable, reusable checkpoint of a running sandbox. Creating one does not replace the sandbox: the source returns to Running after capture, and the snapshot can later launch one or more new sandboxes.

aenv snapshot create <sandbox-id>
aenv snapshot create <sandbox-id> --name my-base

The resulting snapshot appears in aenv snapshot list and can be started with aenv start <snapshot-id-or-name>. See Snapshots for its parameters and lifecycle.

Fork

Forking clones a running sandbox into independent child sandboxes on the same node. The source is briefly paused while its state is captured, then returns to Running. Children inherit the source filesystem, memory, network policy, security mode, and CPU/memory/disk configuration. All children use one captured state, but each child can succeed or fail independently.

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"count": 3, "timeout": 600}' \
  http://127.0.0.1:8000/sandboxes/<sandbox-id>/fork

A successful request returns an array with one result for each requested child. Each entry contains either a sandbox object—including its sandboxID—or an error explaining why that individual child failed. It is not a plain list of IDs. A non-201 response means the request failed before any child was attempted. See the API Reference for the complete fork request and response schemas.

Manage Sandboxes

List sandboxes:

aenv list
aenv list --output json

--output accepts table or json. It defaults to a table in an interactive terminal and JSON when output is piped or redirected.

Delete a sandbox:

aenv delete <sandbox-id>

Deletion is permanent, but snapshots previously created from the sandbox are unaffected.

Auto-Eviction

Every running sandbox has a time-to-live (TTL). The TTL establishes an expiration deadline so a sandbox cannot occupy CPU and memory indefinitely. When the TTL is reached, AgentENV automatically pauses or deletes the sandbox so those resources can be reclaimed.

Behavior at Expiration

When a sandbox reaches its TTL, AgentENV performs its configured timeout action:

• Pause (autoPause: true, the default): preserve the sandbox so it can be resumed later.
• Delete (autoPause: false): permanently remove the sandbox.

The timeout action is selected when the sandbox is created. The aenv start command uses the default action, autoPause: true. To delete on expiration instead, create the sandbox through the API with autoPause: false.

Warm start from a template or snapshot:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "templateID": "my-template",
    "timeout": 600,
    "autoPause": false
  }' \
  http://127.0.0.1:8000/sandboxes

Cold start from an OCI image:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "image": "ubuntu:24.04",
    "timeout": 600,
    "autoPause": false
  }' \
  http://127.0.0.1:8000/sandboxes-cold

Set or Extend the Deadline

aenv start --timeout <seconds> sets the initial TTL. If an automatically paused sandbox is needed again, aenv resume --timeout <seconds> resumes it and sets a new TTL from the resume time. Both commands default to 300 seconds.

For a running sandbox, replace its deadline with an exact number of seconds from now:

aenv timeout <sandbox-id> 600

This sets the deadline to 600 seconds from the time the command is sent. Calling it again replaces the previous deadline, so it can either extend or shorten the remaining time.

To keep a running sandbox alive without shortening a later existing deadline, use the refresh API:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"duration": 600}' \
  http://127.0.0.1:8000/sandboxes/<sandbox-id>/refreshes

Refresh does not shorten the remaining TTL if the current deadline is later. Refresh applies only to a running sandbox; resume a paused sandbox first. If duration is omitted, the server’s default sandbox timeout is used.

aenv connect resumes a paused sandbox when it connects and ensures that its TTL is at least the default 300 seconds.

Networking

Each sandbox has an isolated network stack. Networking controls two separate boundaries:

• Egress: which IP addresses, CIDRs, and domains the sandbox can connect to.
• Ingress: whether services exposed through the AgentENV proxy are public or require the sandbox traffic access token.

What You Can Configure

Node-wide egress denials are configured separately with [network.egress].always_denied_cidrs in your AgentENV config file. This lists IP ranges that every sandbox is prohibited from reaching, such as private or host-local networks. These rules are applied before per-sandbox rules and cannot be overridden by allowOut.

[network.egress]
always_denied_cidrs = [
  "10.0.0.0/8",
  "169.254.0.0/16",
]

Matching Rules

Rules are evaluated in this order:

flowchart LR
    A[Destination<br/>packet] --> N{Node-level<br/>deny?}
    N -->|Yes| E[Reject<br/>traffic]
    N -->|No| B{Matches<br/>allowOut?}
    B -->|Yes| C[Allow<br/>traffic]
    B -->|No| D{Matches<br/>denyOut?}
    D -->|Yes| E
    D -->|No| F{"allow_internet_access?"}
    F -->|Yes| C
    F -->|No| E

allowOut can override an overlapping user-configured denyOut, but it cannot override node-level internal/reserved-network deny rules. Setting allow_internet_access: false adds a deny-by-default base policy after the explicit rules.

Domain names can be used only in allowOut for HTTP/HTTPS connections. Exact names and wildcard forms such as *.example.com are supported. If allowOut contains a domain, also set denyOut to ["0.0.0.0/0"]; this blocks other destinations and leaves the listed domains as the allowed exceptions.

Configure at Creation

Both warm and cold sandbox creation support network policy.

Warm start from a template or snapshot:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "templateID": "my-ubuntu",
    "network": {
      "allowOut": ["*.example.com"],
      "denyOut": ["0.0.0.0/0"],
      "allowPublicTraffic": false
    }
  }' \
  http://127.0.0.1:8000/sandboxes

Cold start from an OCI image:

curl -X POST \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "image": "ubuntu:24.04",
    "allowInternetAccess": false,
    "network": {
      "allowOut": ["8.8.8.8/32"]
    }
  }' \
  http://127.0.0.1:8000/sandboxes-cold

Update a Running Sandbox

Replace the egress policy of a running sandbox:

curl -X PUT \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"allowOut": ["8.8.8.8/32"], "denyOut": ["0.0.0.0/0"]}' \
  http://127.0.0.1:8000/sandboxes/<sandbox-id>/network

The update replaces the current egress rules. Omitting both allowOut and denyOut clears the per-sandbox lists; omit allow_internet_access as well to restore the default base policy.

In the current implementation, updates primarily affect new connections and do not actively terminate existing ones. For domain-policy replacement, the old policy remains active until the new namespace rules are installed and the new proxy policy is activated.

Snapshots

A snapshot is a reusable checkpoint of a sandbox. It preserves the sandbox’s filesystem and runtime state so that you can later start a new sandbox from the same point instead of rebuilding the environment and rerunning setup work.

• Templates are stored as snapshots. A template build commits one snapshot; the template ID is an alias that resolves to it.
• Sandboxes launch by resuming from a snapshot.
• Running sandboxes can produce new snapshots, capturing their current state for later reuse or branching.

Create a Snapshot from a Running Sandbox

Capture the current state of a running sandbox:

aenv snapshot create <sandbox-id>
aenv snapshot create <sandbox-id> --name my-checkpoint

The source sandbox continues running after the snapshot is created. You can then pass either the snapshot ID or its alias to aenv start:

aenv start my-checkpoint
# Or:
aenv start <snapshot-id>

This creates a separate sandbox with a new sandbox ID. It inherits the captured filesystem, running processes, memory state, environment variables, runtime configuration, and resource settings.

To retrieve information about one snapshot by ID or alias, use the HTTP API:

curl -H 'X-API-Key: test-key' \
  http://127.0.0.1:8000/snapshots/<snapshot-id-or-alias>

Use a Snapshot Rootfs as an OCI Image

Starting with aenv start <snapshot> restores the complete snapshot state. If you only need the captured root filesystem, you can instead publish it as an OCI image and cold-start sandboxes from that image.

Publish an Image When Creating a Snapshot

AgentENV can automatically create and publish an OCI image of the snapshot rootfs whenever you create a snapshot. For snapshots backed by OSS and created from an OverlayBD-native OCI image, enable automatic publication in your AgentENV config file (config/default.toml, or the file selected by AENV_CONFIG_PATH):

[snapshot]
repository_backend = "oss"

[snapshot.image_publish]
enabled = true

Create the snapshot:

aenv snapshot create <sandbox-id> --name my-checkpoint

The command prints the published image reference when publication succeeds. Use that reference to cold-start a sandbox:

aenv start --cold registry.example.com/team/app:agentenv-snapshot-<snapshot-id>

Export an Existing Snapshot Rootfs

You can manually export the rootfs of an existing snapshot as an OCI image. This is done with aenv-snapshot-image, which is not included in the regular AgentENV installation packages and must first be built and installed from the repository:

git clone https://github.com/kvcache-ai/AgentENV.git
cd AgentENV
make build-snapshot-image
sudo install -m 0755 target/debug/aenv-snapshot-image /usr/local/bin/aenv-snapshot-image

Export the rootfs:

aenv-snapshot-image <snapshot-id-or-alias> \
  --target-repository registry.example.com/team/app \
  --tag release-1

The exported image can be stored in an OCI registry, shared independently of the AgentENV snapshot repository, and used to cold-start new sandboxes. The new sandbox inherits the exported root filesystem and OCI runtime configuration, but not the snapshot’s memory or running-process state.

You can capture it and start a sandbox from the resulting image:

aenv-snapshot-image <snapshot-id-or-alias> \
  --target-repository registry.example.com/team/app \
  --tag release-1

aenv start --cold <image_ref>

Manage Snapshots

List Snapshots

aenv snapshot list
aenv snapshot list --sandbox-id <sandbox-id>

aenv snapshot ls is an alias for aenv snapshot list.

Start a Sandbox from a Snapshot

aenv start <snapshot-id-or-name>

The CLI always enables secure sandbox authentication and manages the envd access token automatically.

Delete a Snapshot

Snapshots and templates share the same catalog. Delete a snapshot by passing its required ID or alias to the template delete command:

aenv template delete <snapshot-id-or-name>

Optional P2P Visibility

P2P visibility lets nodes discover and fetch committed snapshot artifacts from peers. The snapshot repository remains the durable source of truth, so P2P is an optional distribution path rather than a replacement for snapshot storage.

Enable both the node-wide P2P transport and snapshot publication in your AgentENV config file (config/default.toml, or the file selected by AENV_CONFIG_PATH):

[p2p]
enabled = true

[snapshot]
p2p_enabled = true

See the Configuration Reference for the optional transport, storage-directory, address, and timeout settings.

Volumes

Beta: The volume feature is still in beta. For workloads that can use a drive supplied only when cold-starting a sandbox, the attachedDrives feature on POST /sandboxes-cold is more extensively tested. See the API reference for the extra-drive request schema.

Volumes are persistent block filesystems managed independently from sandboxes. A volume has a stable ID, a unique name, a fixed size, and an access mode. You can mount it at an absolute guest path when creating a sandbox, then reuse its contents after that sandbox is deleted.

Deleting a running sandbox freezes its writable volume filesystems, seals and publishes their layers, and stops the VM without saving memory or rootfs state. This checkpoints the filesystem journal so the next mount avoids journal replay. Recoverable capture or publication failures thaw the filesystems and retain the running sandbox and its volume reservations. A terminal capture failure stops the VM and marks writable volumes failed, preventing mounts of incomplete data. Stop or catalog failures retain the deletion state for retry; reservations are released only after stop succeeds. Pausing still saves the full sandbox state for later resume.

The default volume size is 65536 MiB (64 GiB). By default, one sandbox may mount up to four volumes and each volume may be at most 262144 MiB (256 GiB). Administrators can change these limits in the [volume] configuration.

Access Modes

A volume’s mode is selected when the volume is created and cannot be changed.

An exclusive volume is reserved by its mounted sandbox. Another sandbox cannot mount or delete it until that reservation is released. A read-only volume can be mounted by multiple sandboxes at the same time, but guest writes fail.

Recommended Workflow: Fork Before Use

Treat a shared volume as an immutable base and create a copy-on-write fork for each sandbox that needs to modify it. This gives every sandbox an independent exclusive volume without copying all source data eagerly.

read-only base volume
        |
        +-- exclusive fork for sandbox A
        +-- exclusive fork for sandbox B

Forks capture the source volume state at creation time. Later changes to a fork do not change the source or another fork. The fork must have the same size as its source. An exclusive source must be unmounted before a public fork is created; a read-only source may remain shared.

Sandbox Forks and Snapshots

AgentENV automatically carries mounted volumes through sandbox fork and snapshot operations. You do not need to fork or snapshot each mounted volume separately.

Fork a sandbox

When a sandbox is forked, AgentENV processes every mounted volume for every child sandbox:

• Each mounted exclusive volume gets an independent copy-on-write volume fork. The child mounts the new volume at the same guest path and can modify it without changing the source sandbox’s volume.
• Each mounted ro volume remains mounted from the same read-only volume. It is safe to share because neither the source nor a child can modify it.

This behavior is separate from manually creating a reusable volume fork with aenv volume create --from-volume.

Snapshot a sandbox

When a sandbox is snapshotted, every mounted volume is included automatically. The volume snapshot records its layers, size, mount path, and access mode. Starting a sandbox from that snapshot creates new volumes with the captured contents and mounts them at the same paths. An exclusive volume remains exclusive; a read-only volume remains read-only.

Volume snapshots are independent from their source volumes. Deleting a source volume does not remove volume data already committed into a sandbox snapshot.

CLI Examples

Create and mount an empty volume

aenv volume create workspace --mode exclusive --size-mb 65536
aenv start ubuntu --volume /workspace=workspace

--volume accepts MOUNT_PATH=VOLUME_ID_OR_NAME and can be repeated. Mount paths must be absolute, cannot be /, and cannot overlap another mount.

aenv start ubuntu \
  --volume /workspace=workspace \
  --volume /models=models-base

Create a volume from an OCI image

Use --image to initialize a volume with the contents of an OCI image:

aenv volume create models-base \
  --mode ro \
  --image registry.example.com/team/models:latest

For a standard OCI image, AgentENV downloads and converts its layers when the volume is created. For an OverlayBD-native image, AgentENV keeps the remote layer references and does not download the layer contents during volume creation. Blocks are fetched from the OCI registry on demand when the mounted volume is read, and then retained in the local remote-block cache.

Create a reusable base and fork it

First populate an exclusive seed volume:

aenv volume create dataset-seed --mode exclusive
SANDBOX_ID=$(aenv start ubuntu -d --volume /data=dataset-seed)
aenv exec "$SANDBOX_ID" sh -lc 'printf "%s\n" training-data > /data/input.txt'
aenv delete "$SANDBOX_ID"

Create a read-only base from the seed, then create one exclusive fork per sandbox:

aenv volume create dataset-base --mode ro --from-volume dataset-seed
aenv volume create job-42-data --mode exclusive --from-volume dataset-base
aenv start ubuntu --volume /data=job-42-data

These commands use the default 64 GiB size for every volume. When the source has a custom size, pass the same --size-mb value while creating its fork.

Share a read-only volume

SANDBOX_A=$(aenv start ubuntu -d --volume /models=models-base)
SANDBOX_B=$(aenv start ubuntu -d --volume /models=models-base)

Both sandboxes can read /models. Neither sandbox can modify it.

Inspect lifecycle state

aenv volume list
aenv volume inspect job-42-data
aenv volume delete job-42-data

The volume status controls whether it can be mounted:

Deleting a mounted volume returns a conflict. Delete its sandbox first, then delete the volume.

HTTP API Examples

Set the server URL and API key before running these examples:

export AENV_URL=http://127.0.0.1:8000
export AENV_API_KEY=<api-key>

Create a read-only base volume:

curl -fsS -X POST "$AENV_URL/volumes" \
  -H "X-API-Key: $AENV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "dataset-base",
    "sizeMB": 65536,
    "mode": "ro",
    "image": "ghcr.io/example/dataset:latest"
  }'

Create an exclusive copy-on-write fork:

curl -fsS -X POST "$AENV_URL/volumes" \
  -H "X-API-Key: $AENV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "job-42-data",
    "sizeMB": 65536,
    "mode": "exclusive",
    "fromVolume": "dataset-base"
  }'

Mount the fork into a sandbox:

curl -fsS -X POST "$AENV_URL/sandboxes" \
  -H "X-API-Key: $AENV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "templateID": "ubuntu",
    "volumeMounts": [
      {
        "name": "job-42-data",
        "path": "/workspace/data"
      }
    ]
  }'

List and inspect volumes:

curl -fsS "$AENV_URL/volumes" -H "X-API-Key: $AENV_API_KEY"
curl -fsS "$AENV_URL/volumes/job-42-data" -H "X-API-Key: $AENV_API_KEY"

Custom Extension

The custom extension is an optional external HTTP service that AgentENV calls during the sandbox lifecycle. It lets you implement deployment-specific behavior — for example, connecting sandboxes into a VPN, custom firewall rules, or extra mounts — without changing AgentENV itself.

The extension implements a small set of HTTP endpoints (“hooks”); AgentENV is the client. The interface is defined in src/custom_extension_api/openapi.yml.

Lifecycle hooks

To support the complete sandbox lifecycle, your extension should implement the following four APIs. AgentENV sends a JSON request to POST {url}/sandbox-hook/<hook> when the corresponding event occurs. Any connection error, timeout, or non-2xx response fails the corresponding sandbox operation (except stop, which is best-effort).

See Minimal Extension Example for an example extension that implements all four hooks.

Notes:

• Instance identity. A sandboxId is reused across pause/resume cycles. Every start-fresh / start-resume carries a fresh sandboxInstanceId identifying that runtime instance, and the subsequent stop carries the same value. Because stop is best-effort and may be reordered (e.g. a pause’s stop arriving after the resume’s start-resume), treat (sandboxId, sandboxInstanceId) as the identity of a running instance and ignore stop notifications whose sandboxInstanceId is not the latest started instance for that sandbox.
• stop also fires on pause. Pausing persists the sandbox state and then stops the VM process and releases the network namespace; the subsequent resume creates a fresh runtime and fires start-resume. In-place pause+resume during snapshot capture does not fire any hook (and keeps the same sandboxInstanceId).
• stop is best-effort: delivery failures are only logged, and it is also fired fire-and-forget if a started sandbox is dropped without an explicit stop.
• networkNamespacePath is the host path of the sandbox’s netns file (e.g. /var/run/netns/agentenv-ns-*), so the extension can enter the namespace (e.g. nsenter --net=...) to set up firewall rules or VPN interfaces.
• hostInteractionIp is the per-runtime IPv4 address that AgentENV routes to this sandbox. It can change after pause/resume, so extensions must use the value from the current start hook rather than caching an older one.
• Concurrent patch-params calls to the same sandbox are not serialized; if your patch semantics are not commutative, handle concurrency in the extension.

Connect AgentENV to an Extension

Connect AgentENV to your extension service by configuring its URL:

# config/default.toml (or your AENV_CONFIG_PATH)
[custom_extension]
url = "http://127.0.0.1:9090"
# timeout_ms = 5000   # optional, per-call timeout in milliseconds

AENV_CUSTOM_EXTENSION_URL works as well. When url is unset, the integration is fully disabled: no hooks are called and customExtensionParams must be empty.

Use the Extension

Use customExtensionParams to pass extension-specific settings for a sandbox. It is an opaque JSON object interpreted only by your extension. An absent value and an empty object are equivalent.

Set at Creation

Both POST /sandboxes and POST /sandboxes-cold accept customExtensionParams. For example, create a sandbox from a template with VPN settings for the extension:

curl -X POST http://127.0.0.1:8000/sandboxes \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "templateID": "my-template",
    "customExtensionParams": {
      "vpn": { "network": "team-a" }
    }
  }'

For a cold-start sandbox, include the same field in the cold-start request:

curl -X POST http://127.0.0.1:8000/sandboxes-cold \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "image": "docker.io/library/ubuntu:24.04",
    "customExtensionParams": {
      "vpn": { "network": "team-a" }
    }
  }'

Read

Get the current params. AgentENV returns {} when they are empty:

curl http://127.0.0.1:8000/sandboxes/<sandbox-id>/custom-extension-params \
  -H 'X-API-Key: test-key'

Patch

The request body is passed through verbatim to the extension’s patch-params hook; its semantics are defined entirely by the extension. The hook returns the updated full params, which AgentENV stores and returns:

curl -X PATCH http://127.0.0.1:8000/sandboxes/<sandbox-id>/custom-extension-params \
  -H 'X-API-Key: test-key' \
  -H 'Content-Type: application/json' \
  -d '{"vpn": {"network": "team-a", "peers": ["10.8.0.2", "10.8.0.3"]}}'

Persistence

Params survive pause/resume and are stored into snapshots created from the sandbox. When starting from a template, a customExtensionParams provided at creation overrides the one stored in the snapshot; otherwise the snapshot’s value is inherited.

Minimal extension example

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

# Latest started runtime instance per sandbox: (sandboxId, sandboxInstanceId)
# is the identity of a running instance; a stop for a superseded instance
# (e.g. arriving after a newer start) is ignored.
latest_instance: dict[str, str] = {}

@app.post("/sandbox-hook/start-fresh")
async def start_fresh(req: Request):
    body = await req.json()
    latest_instance[body["sandboxId"]] = body["sandboxInstanceId"]
    # e.g. nsenter --net={body["networkNamespacePath"]} wg-quick up ...
    return {"extraBootArgs": None}

@app.post("/sandbox-hook/start-resume")
async def start_resume(req: Request):
    body = await req.json()
    latest_instance[body["sandboxId"]] = body["sandboxInstanceId"]
    return {}

@app.post("/sandbox-hook/patch-params")
async def patch_params(req: Request):
    body = await req.json()
    # apply body["patch"] however you like, then return the full new params
    return {"customExtensionParams": body["patch"]}

@app.post("/sandbox-hook/stop")
async def stop(req: Request):
    body = await req.json()
    if latest_instance.get(body["sandboxId"]) == body["sandboxInstanceId"]:
        latest_instance.pop(body["sandboxId"], None)
        # tear down resources for this instance
    return {}

Any non-2xx response (or timeout) fails the corresponding sandbox operation, except for stop, which is always tolerated.

Proxy

The reverse proxy lets you reach services running inside a sandbox from outside. It supports HTTP requests, SSE streams, and WebSocket connections.

Endpoints

• ANY /proxy forwards to / inside the sandbox
• ANY /proxy/{path} forwards to /{path} inside the sandbox
• Header-routed requests on otherwise unmatched paths forward the original path unchanged. This lets clients use the same base URL for API and sandbox data traffic when they send routing headers.
• When sandbox proxy domains are configured, host-based URLs shaped like {port}-{sandboxID}.{domain} forward the original path without routing headers.

Query strings are forwarded unchanged.

For example, start an HTTP server on port 8080 inside a running sandbox:

aenv exec <sandbox-id> sh -c 'echo "Hello from AgentENV" > /tmp/index.html'
aenv exec <sandbox-id> python3 -m http.server 8080 --directory /tmp

Access the service through the AgentENV proxy:

curl http://127.0.0.1:8000/proxy/index.html \
  -H "x-agentenv-sandbox-id: <sandbox-id>" \
  -H "x-agentenv-target-port: 8080"

Header-Based Routing

When using /proxy or routing an otherwise unmatched path with headers, identify the target sandbox and port with these headers:

E2B-compatible aliases are also accepted:

These routing headers are stripped before the request is forwarded to the sandbox.

Access Control

Proxy authentication is independent from AgentENV API authentication:

X-API-Key authenticates AgentENV control-plane APIs only. It does not grant access to private application ingress or secure envd. A matching platform key is stripped on proxy requests; other X-API-Key values remain available to sandbox applications. AgentENV also strips the traffic token, and forwards X-Access-Token only to the matching secure envd port.

Host-based proxy requests derive both values from Host, for example http://8080-<sandbox-uuid>.sandbox.example.com/health targets port 8080. The configured domain must route to the AgentENV server in single-node mode or to the gateway in multi-node mode. Host-based proxy traffic is always treated as data-plane traffic; lifecycle and other control-plane APIs should use the base API host.

API Reference

E2B

E2B SDK

AgentENV exposes an E2B-compatible API, so the official E2B SDK works out of the box.

General Settings

Set environment variables to point at your AgentENV server. See Environment Variables for values per deployment mode.

# Single-node example
export E2B_API_URL=http://127.0.0.1:8000
export E2B_SANDBOX_URL=${E2B_API_URL}
export E2B_API_KEY=${AENV_API_KEY}

AgentENV returns trafficAccessToken when network.allowPublicTraffic is false and (for secure sandboxes) envdAccessToken for envd control traffic. These credentials have different headers and trust boundaries: use e2b-traffic-access-token for private application routes and X-Access-Token only for envd. Public application routes require neither token.

TypeScript SDK

Setup

Install the SDK:

npm install e2b

Usage

import { Sandbox } from "e2b";

// Create a sandbox from a template
const sandbox = await Sandbox.create("<template-id>", {
  apiKey: process.env.E2B_API_KEY,
});

// List running sandboxes
const running = Sandbox.list({
  apiKey: process.env.E2B_API_KEY,
  limit: 20,
  query: { state: ["running"] },
});
console.log(await running.nextItems());

// Run a command inside the sandbox
sandbox.commands.run("echo hello world");

// Pause the sandbox
await Sandbox.Pause(sandbox.sandboxId, {
  apiKey: process.env.E2B_API_KEY,
});

// Kill the sandbox
await sandbox.kill();

Replace <template-id> with a template that exists in your local template store. Use e2b template list or GET /v2/templates to see available templates.

Volume mounts

The TypeScript SDK can create a volume and pass it directly when creating a sandbox:

import { Sandbox, Volume } from "e2b";

const volume = await Volume.create("workspace-volume", {
  apiKey: process.env.E2B_API_KEY,
});
const sandbox = await Sandbox.create("<template-id>", {
  apiKey: process.env.E2B_API_KEY,
  volumeMounts: {
    "/workspace": volume,
  },
});

For the Python SDK, create the volume with aenv volume create or POST /volumes, then pass its name when creating a sandbox:

sandbox = Sandbox.create(
    "<template-id>",
    volume_mounts={"/workspace": "workspace-volume"},
)

AgentENV supports the TypeScript SDK’s volume create, list, and delete operations, and accessing the mounted filesystem through the sandbox. The E2B SDK’s direct volume content API is not supported.

Python SDK

Setup

Install the SDK:

pip install e2b

Usage

from e2b import Sandbox, SandboxQuery, SandboxState

# Reuse the environment variables set in your shell:
# E2B_API_URL / E2B_SANDBOX_URL / E2B_API_KEY

# Create a sandbox from a template
sandbox = Sandbox.create("<template-id>")

# List running sandboxes
running = Sandbox.list(
    limit=20,
    query=SandboxQuery(state=[SandboxState.RUNNING]),
)
print(running.next_items())

# Run a command inside the sandbox
result = sandbox.commands.run("echo hello world")
print(result.stdout, end="")

# Pause the sandbox
sandbox.beta_pause()

# Kill the sandbox
sandbox.kill()

E2B CLI

AgentENV is compatible with the E2B CLI, but we recommend using the aenv CLI for AgentENV workflows.

Common Issues

/dev/kvm is missing or inaccessible

Symptom: The server cannot find or open /dev/kvm.

Solution: First check whether standard KVM is available:

ls -l /dev/kvm

If the device exists, ensure it is readable and writable by the runtime user. On most systems:

sudo usermod -aG kvm $USER
# Log out and back in for the group change to take effect

If the cloud server does not expose standard KVM, follow PVM Deployment. That guide covers the additional host setup needed before starting AgentENV.

The configured virtualization mode does not match the host

Symptom: Startup reports that KVM cannot run while the PVM module is loaded, or that PVM requires additional host setup.

Solution: Normal installations should use the default KVM mode. If the host was prepared for PVM, follow PVM Deployment and ensure the service environment contains:

AENV_VIRTUALIZATION_MODE=pvm

Permission denied for network operations

Symptom: Sandbox creation fails with network namespace or iptables errors.

Solution: The server requires both CAP_NET_ADMIN and CAP_SYS_ADMIN in its effective, permitted, and inheritable sets. The installed systemd unit configures these automatically. For a source checkout, use make start-server or scripts/run-with-capabilities.sh <server-binary>; do not run the whole server as root. Also verify that the runtime account belongs to the kvm group, can open /dev/kvm, and can open /dev/ublk-control.

Sandbox namespaces are missing from ip netns list

Symptom: Sandboxes are running, but ip netns list does not show their network namespaces and ip netns exec <name> cannot find them.

Solution: AgentENV stores namespace mount points under $AENV_RUNTIME_PATH/netns instead of /var/run/netns; the installed service uses /run/aenv/netns. Inspect that directory directly and enter a namespace by path when needed:

sudo nsenter --net=/run/aenv/netns/agentenv-ns-<slot> ip addr

Config file not found

Symptom: Error: config file not found

Solution: The server looks for config/default.toml by default. Either run from the repository root or set AENV_CONFIG_PATH:

export AENV_CONFIG_PATH=/path/to/your/config.toml

Port already in use

Symptom: Address already in use when starting the server.

Solution: Another process is using port 8000. Either stop it or change the listen address:

API_ADDR=0.0.0.0:8001 make start-server

Sandbox creation timeout

Symptom: POST /sandboxes returns a timeout error.

Solution: Check that runtime assets (Firecracker binary, kernel, rootfs) have been downloaded. The server auto-provisions them on first start, but network issues can cause failures. Run cargo run --bin server -- --setup-only to provision manually and see detailed errors.

Also check [envd].init_timeout_secs in your config. The default is 60 seconds. If the rootfs image is large, the in-guest envd daemon may need more time to initialize.

TODO: Expand with more common issues as they are reported.

Configuration Reference

AgentENV reads configuration from a TOML file. The default path is config/default.toml. Override it with:

export AENV_CONFIG_PATH=/path/to/config.toml
# or
cargo run --bin server -- --config /path/to/config.toml

Global Settings

Snapshots and paused sandboxes can only be restored in the mode in which they were created.

$AENV_HOME is a literal placeholder in state-path values, not a shell environment variable. AgentENV replaces it with the resolved home_path after applying AENV_HOME_PATH; ublk.daemon_socket_path additionally supports $AENV_RUNTIME, which resolves to runtime_path. Relative paths without these placeholders are resolved against the directory containing the configuration file.

Packaged runtime dependency versions and download URLs live in config/deps_manifest.toml. Only the dependencies for the selected mode are installed. User configuration should contain runtime behavior and explicit local path overrides, not the default dependency catalog.

[firecracker]

Firecracker VM binary and boot configuration.

[kernel]

Linux kernel image for microVMs.

[tools]

Tools drive image used to boot the AgentENV control plane inside each microVM.

Snapshots and paused sandboxes keep using the tools drive version they were created with. Launch does not download missing releases: operators must install the recorded version under <deps_path>/tools/<version>/tools.ext4 before restore. Setup retains previously installed versions until they are removed manually.

Template Rootfs Images

User-visible rootfs images are selected at the template API layer. POST /v2/templates/{templateID}/builds/{buildID} accepts an optional fromImage field:

• omitted: use [image.resolver].default_image
• full OCI reference: use the supplied image
• short name: normalize standard Docker Hub forms such as ubuntu:24.04 and node:20

[image.resolver]

How the three registry settings interact

Image resolution runs in two phases, and the three keys act at different points:

1. search_registries — completion. Only used for short / unqualified references (e.g. ubuntu). Each entry is prefixed to the name to build a list of fully-qualified candidates (docker.io/library/ubuntu:latest, ghcr.io/ubuntu:latest, …). Fully-qualified references skip this step.

2. allowed_registries — gating. Applied right after candidates are built, to both fully-qualified references and the candidates expanded from search_registries. Candidates whose registry host is not whitelisted are dropped; if none remain, the reference is rejected with a 4xx error. In effect the resolvable set of short-name hosts is the intersection of search_registries and allowed_registries — e.g. searching docker.io and ghcr.io while only allowing ghcr.io resolves short names to ghcr.io only.

3. try_referrers_overlaybd_prefixes — per-candidate optimization. Runs later, while resolving an already-permitted candidate: after its manifest is fetched, AgentENV may query the OCI Referrers API on the same registry/repository for an overlaybd-native artifact. Because referrer lookups never leave the source image’s own host, they are implicitly covered by allowed_registries — no separate whitelist entry is needed for referrers.

   Two referrer artifactTypes are recognized, in this order:

   artifactType	Produced by
   application/vnd.containerd.overlaybd.native.v1+json	accelerated-container-image (obdconv)
   application/vnd.azure.artifact.streaming.v1	Azure Container Registry artifact streaming (az acr artifact-streaming create)

   Both point at an overlaybd-native manifest; only the discovery label differs. The referrer manifest is re-validated after it is fetched, so a referrer that is not actually overlaybd-native is rejected rather than used. Turbo-OCI referrers (application/vnd.containerd.overlaybd.turbo.v1+json) are never selected — AgentENV’s overlaybd runtime does not implement the turbo read path.

   To stream from ACR, add your registry (with the trailing slash) to the list, e.g. try_referrers_overlaybd_prefixes = ["myregistry.azurecr.io/"].

[image.cache]

Node-local cache root for resolved and converted user images.

[image.cache.gc]

Background hard-commit garbage collection for the image cache. When enabled, each pass reconciles metadata from the on-disk source configs and then deletes hard-commit objects that are no longer rooted by source configs, held by image-cache leases, or referenced by the in-process running set. Committed snapshots are durable SnapshotRepository state and do not pin ImageCache commits. With capacity_gb set, GC first evicts least-recently-used source configs over the high watermark so hard-commit GC can reclaim what they unrooted.

Capacity-driven eviction runs only when [image.cache].capacity_gb is set; otherwise the GC still reclaims unreachable commits but performs no watermark eviction.

[image.cache.remote_blocks]

Overlaybd registryfs_v2 remote block cache settings. The directory is always <image.cache.root_dir>/remote-blocks.

Resolved image data is cached under:

<image.cache.root_dir>/
  commits/
    <sha256-commit-digest>/
      overlaybd.commit
                  # full overlaybd commit store shared by OCI conversion and download
  indexes/        # OCI layer + conversion context -> overlaybd commit descriptor
  remote-blocks/  # overlaybd-native remote block cache
  configs/         # resolved image configs
    <slug>-<hash>-image.json

[sandbox_proxy]

Optional host-based data-plane routing for sandbox services.

When domains is empty, the server still supports /proxy and routing-header proxy requests, but does not classify requests by Host. Domains are normalized to lowercase, deduplicated, and must be valid DNS names. The configured order is preserved because domains[0] is the advertised sandbox domain.

Environment variable override:

• AENV_SANDBOX_PROXY_DOMAINS

[network.egress]

Node-level sandbox egress guardrails. These rules are installed before per-sandbox allowOut / denyOut rules, so sandbox API requests cannot override them.

[network.internal]

AgentENV-internal sandbox address plan. Change these only when the defaults overlap with host or deployment network ranges.

The two configured CIDRs must not overlap each other or AgentENV’s fixed VM tap link 169.254.0.20/30. These networks are also treated as reserved sandbox egress destinations regardless of always_denied_cidrs.

[machine]

Default VM resources for sandboxes.

[envd]

In-guest envd daemon settings.

[sandbox]

Sandbox control communication settings.

The managed seed is node-local persistent state and must be included in backups of $AENV_HOME. AgentENV refuses to generate a replacement when persisted secure or private-ingress sandboxes exist. An explicit environment or TOML value takes precedence over the managed file; changing that effective value invalidates existing sandbox access tokens.

Configure AENV_SANDBOX_ACCESS_TOKEN_HASH_SEED with the same value on every runtime node in a clustered deployment. Standalone runtime nodes use their managed seed when it is unset.

[volume]

[orchestrator]

Sandbox lifecycle management.

[pool]

Shared process-wide warm-pool defaults used by network slots, block devices, and pre-spawned Firecracker processes. Pools prewarm to the low watermark, then grow the refill target geometrically toward the high watermark when real acquisitions drain the pool.

Component sections:

Validation rules:

• low_watermark <= high_watermark
• [pool.firecracker].fill_concurrency > 0

[node_identity]

Stable identity fields for this node. These values appear in node API responses and scheduler heartbeats.

[observability]

Node-level observability and host metrics collection.

When observability is enabled, host CPU/memory/disk metrics are collected at request time. CPU percent is computed from two samples; the first node metrics request waits about 100ms to return a measured value.

[observability.scheduler_report]

Optional scheduler heartbeat reporting for multi-node control plane integration.

Environment variable overrides:

• AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED
• AENV_OBSERVABILITY_SCHEDULER_ENDPOINT
• AENV_OBSERVABILITY_REPORT_INTERVAL_SECS

AENV_OBSERVABILITY_SCHEDULER_ENDPOINT overrides [cluster].scheduler_endpoint for the reporter process only.

[cluster]

Shared cluster-level service endpoints.

[p2p]

Experimental: P2P has not been tested in production. Keep it disabled in production unless the deployment accepts that operational risk.

Project-wide artifact transport configuration. The transport is disabled by default. When enabled, it is used by the overlaybd P2P HTTP facade and by snapshot publication/runtime resolution as an optional acceleration path.

[custom_extension]

Custom extension service configuration. When url is unset, the integration is fully disabled. See Custom Extension.

[snapshot]

Snapshot storage/build configuration.

Environment variable overrides:

• AENV_SNAPSHOT_LOCAL_CACHE_PATH

[snapshot.image_publish]

Source-registry image publication. Only takes effect when snapshot.repository_backend = "oss".

[snapshot.publish_compression]

Publish-time compression for snapshot layers uploaded to OSS/ACR. Local layers always stay raw, so local resume pays no decompression cost; enabled by default, memory layers and incremental read-write layers are compressed once as they are uploaded, cutting network bytes for cross-node resume. This is the only compression switch; the legacy capture-time knobs under [memory_snapshot] and [template_build] were removed from the configuration schema.

Known impact: compressed layers are recorded without a layer uuid (ZFile layers carry no LSMT uuid), so P2P uuid-keyed acceleration does not apply to them. Snapshot P2P publication also skips digest-keyed advertisements for local raw layers whose digest is absent from the committed record — the record names the compressed bytes, so the raw digest key would never be looked up by consumers.

[backend.posix_fs]

POSIX filesystem-backed snapshot repository configuration. This section is used when snapshot.repository_backend = "posix_fs".

Environment variable overrides:

• AENV_SNAPSHOT_STORE

[backend.oss]

OSS-backed snapshot repository configuration. This section is required when snapshot.repository_backend = "oss".

Notes:

• credential_process and static access key settings are mutually exclusive in practice; when credential_process is set, the backend ignores static credential fields.
• credential_process should be written as a portable argv-style command line. Avoid $VAR, backticks, $(...), pipes, and shell builtins.
• Although the config section is still named oss, the runtime path is implemented via a shared S3-compatible client, so region must be configured.
• Leave addressing_style unset when endpoint-based detection is correct. Set it to "virtual" or "path" when the provider’s required or preferred style differs from the detected default; for example, some Tigris or Cloudflare R2 deployments use virtual-host addressing.
• The setting covers both halves of the data path: the snapshot repository client (metadata and artifact upload/download) and the generated OverlayBD runtime config (ossConfig.defaultAddressingStyle), which the runtime uses when reading remote managed snapshot layers during sandbox restore.

For an S3-compatible provider where virtual-host addressing is required or preferred — for example Tigris — set addressing_style explicitly:

[backend.oss]
endpoint = "https://t3.storage.dev"
bucket = "agentenv-snapshots"
region = "auto"
addressing_style = "virtual"

Other path override:

• AENV_DEPS_PATH

Setup sysctl tuning is host-level setup. It is skipped before reading /proc/sys when the server detects that it is running inside a container. Set AENV_FORCE_SYSCTL_TUNING=1 only for a privileged container with writable host sysctls; otherwise configure these kernel parameters on the host.

[protoc]

Protobuf compiler metadata for code generation lives in config/deps_manifest.toml, not config.toml.

[ublk]

Userspace block device configuration. Rootfs is served through an OverlayBD-backed ublk device managed by uvm-ublk-daemon.

Environment variable override:

• AENV_UBLK_DAEMON_BINARY_PATH
• AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR

[ublk.overlaybd]

OverlayBD configuration for ublk. Legacy enabled and device_type keys are ignored.

global_config_path and auto-generated config

The file at the configured default path $AENV_HOME/overlaybd/overlaybd-global.json is auto-generated by the server at startup. The generated JSON incorporates several TOML settings — [image.cache].root_dir, [image.cache.remote_blocks].max_size_gb, download_enable, [backend.oss] credentials, and Docker registry credentials detected from ~/.docker/config.json — into a single overlaybd runtime config file.

The server regenerates the file at global_config_path on every startup, so these TOML settings always take effect automatically — any manual edits to the generated file are overwritten on the next startup. To keep customizations, make them through the TOML settings, not by editing the generated JSON.

[memory_snapshot]

Memory snapshot overlaybd configuration. The server auto-generates the file at the default path on every startup.

[memory_snapshot.background_download]

Background download settings dedicated to remote memory-snapshot OverlayBD layers. They do not change the general rootfs or attached-drive defaults. All fields are serialized into the generated memory OverlayBD global config. Each remote layer is filled block by block into the node-local remote file cache: the cache-owned background-download scheduler registers one task per remote layer (deduplicated by blob, shared across sandboxes) and downloads only chunks still missing from the entry bitmap — each source request fetches block_size bytes (aligned to whole cache blocks) and publishes the chunk’s cache blocks as soon as they land. Submission is never rejected under load — tasks run as scheduler capacity allows, with at most maxConcurrentFiles layer tasks concurrently per file-cache backend (from the generated overlaybd download config, default 8) and at most concurrency chunk reads in parallel per layer, subject to the scheduler’s max_inflight_blocks cap. Downloads of a sandbox-bound device start only after envd is ready (plus delay), with a 20s fallback if the ready signal is lost; while foreground remote reads are in flight, background block reads yield to a small guaranteed floor instead of competing at full speed. The generated memory config leaves throttling off (maxMBps = 0); image configs that carry a positive maxMBps keep their historical shared rate limit across the block tasks. A completed cache block becomes visible to foreground reads as soon as it is committed to the cache bitmap; there is no staging file, no full-file digest check, and no switch-to-local, so a failed or canceled block simply stays uncached and is fetched on demand by foreground reads or a later retry. The cache is a bounded working set: blocks may be evicted under capacity pressure and are then re-fetched on demand.

[template_build]

Managed Dockerfile builder resources. The first Dockerfile build on a node prepares a reusable internal builder template. Each build mounts a separate clone of the repository’s shared cache seed; concurrent builds do not queue for cache ownership. The last successfully published cache becomes the next seed, with best-effort reuse of concurrent branches. Builder resources do not change the resulting template’s CPU or memory.

Environment Variables

Deployment Helpers

These variables are consumed by the repository’s Docker Compose and Kubernetes helpers, then passed to the server or gateway-specific variables listed below.

Server

E2B SDK / CLI

These variables configure the E2B SDK and CLI to point at an AgentENV server. Values depend on your deployment mode.

Values by Deployment Mode

Manual compile (single node):

export E2B_API_URL=http://127.0.0.1:8000
export E2B_SANDBOX_URL=${E2B_API_URL}
export E2B_API_KEY=${AENV_API_KEY}

Docker Compose / Kubernetes (multi-node):

export E2B_API_URL=http://127.0.0.1:8080
export E2B_SANDBOX_URL=${E2B_API_URL}
export E2B_API_KEY=${AENV_API_KEY}

In both modes, sandbox data-plane requests can use routing headers with E2B_SANDBOX_URL=${E2B_API_URL}. The explicit /proxy prefix (${E2B_API_URL}/proxy) is still accepted for back-compat.

See Authentication for key generation and storage.

Gateway and Scheduler

These variables apply to both the gateway and scheduler processes.

Gateway

Scheduler

AgentENV Architecture

AgentENV runs AI agents inside isolated, snapshot-capable Firecracker microVMs. Its core is a storage subsystem that provides layered block devices mountable into VMs and ublk-backed memory snapshot restore. The system also includes a per-node orchestrator managing sandbox lifecycle, and a distributed control plane (gateway + scheduler) for multi-node routing.

System Overview

flowchart TD
    subgraph node[AgentENV Node]
        direction TB
        api["API<br/>(Axum)"] --> orchestrator["Orchestrator<br/>(lifecycle)"]
        orchestrator --> vm["Firecracker VM"]
        vm --> rootfs["/dev/vda (rootfs)"]
        rootfs --> ublkN["ublk (/dev/ublkbN)<br/>userspace block device"]
        ublkN --> overlay["overlaybd"]
        overlay --> upper["upper<br/>(r/w)"]
        overlay --> layer0["layer 0, 1, 2, ...<br/>(r/o)"]
        vm --> extra["/dev/vdb (extra)"]
        extra --> ublkExtra["ublk device"]
        ublkExtra --> overlayExtra["overlaybd<br/>(extra drive)"]
        vm --> memory["VM memory"]
        memory --> ublkM["ublk (/dev/ublkbM)<br/>read-only memory block device (shared across same-snapshot sandboxes via refcounting)"]
        ublkM --> memLayers["overlaybd<br/>(mem layers)"]
        memLayers --> snapN["snap N<br/>(r/o)"]
        memLayers --> snap0["snap 0, 1, 2, ...<br/>(r/o)"]
    end
    style node fill:transparent,stroke:gray

Key Components

Storage

The storage subsystem turns layered image files into block devices mountable by VMs, and provides ublk-backed memory snapshot restore for snapshot resume. Four crates compose the active subsystem:

overlaybd (storage/overlaybd/)

LSMT (Log Structured Merge Tree) based layered image format.

Image structure: Each layer file has a HeaderTrailer (magic LSMT\0\1\2, UUID, flags, index/data offsets) and an array of DiskSegmentMapping entries (16 bytes each, bit-packed: 50-bit offset, 14-bit length, 55-bit physical offset, zeroed flag, layer tag). Layers are stacked: immutable compressed read-only layers at the bottom, a single writable upper layer on top.

Read path: ImageFile resolves a read request by searching layers top-down via the segment index. The first layer containing a mapping for the requested block range serves the data. Unmapped ranges in upper layers fall through to lower layers.

Write path: All writes append to the upper layer. The upper layer’s index is updated in memory and flushed on sync.

Backends (pluggable via VirtualFile trait):

• LocalFile: io_uring pread/pwrite with optional O_DIRECT
• registryfs_v2: OCI registry (remote layer download)
• tar: tar archive reading
• Optional cache layer for decompressed block caching

Compression: zstd (level 3) with random-access jump tables and CRC32C checksums.

Snapshot: ImageFile::create_snapshot_and_restack() is the primary pause path. It seals the live upper layer via LSMTFile::close_seal_and_reopen() so the upper becomes the newest lower layer, then reopens a fresh writable upper in place. image/snapshot.rs::export_upper_as_snapshot_layer() retains the explicit upper-export path used by packaging and export flows.

Key files: image/image_file.rs (high-level image), lsmt/file/ (LSMT stacking: readonly.rs for LSMTReadOnlyFile, readwrite.rs for LSMTFile, stack.rs for open/merge/stack helpers), lsmt/format.rs (binary format), lsmt/index.rs (segment mapping), compression/zfile.rs (compression), image/snapshot.rs.

ublk (storage/ublk/)

Async userspace block device server using Linux’s ublk kernel driver. Exposes OverlayBD images as /dev/ublkbN block devices.

Device lifecycle:

1. UVMUblkCtrlBuilder sends ADD to /dev/ublk-control via io_uring UringCmd
2. Kernel allocates device ID, creates /dev/ublkcN (control) and /dev/ublkbN (block)
3. Per-queue worker threads start, each with a thread-local AsyncIoRing and slab-allocated I/O slots
4. Kernel dispatches block I/O to mmap’d ublksrv_io_desc arrays; userspace processes them asynchronously
5. delete_dev() tears down the device

Target implementation (UVMUblkTarget trait):

• OverlaybdTarget: wraps ImageFile for full layered image I/O

I/O buffers: AutoRegBuffer (zero-copy via sparse buffer table, kernel 6.8+) or UserBuffer (traditional allocation).

Key files: lib.rs (public API), ctrl.rs (device controller), dev.rs (device + queue management), queue.rs (I/O descriptor handling), io_buffer.rs, impls/overlaybd_target.rs.

ublk-daemon (storage/ublk-daemon/)

Long-running daemon process (uvm-ublk-daemon) that manages all ublk devices in one process and communicates with the AgentENV node over a Unix domain socket.

• Supports RPCs for OverlayBD runtime creation for sandbox rootfs/extra drives, raw OverlayBD device creation for non-runtime callers, warm-pool acquire/release, resize capability queries, restack snapshot, delete, and shutdown.
• UblkDaemonClient spawns and monitors the daemon process from the node runtime.
• UblkDeviceManager (src/sandbox/ublk/device.rs) is the node-facing singleton that delegates lifecycle operations to the daemon client; device IDs are allocated in the daemon.

This separation keeps ublk device ownership and io_uring control in a dedicated process while the node server orchestrates lifecycle state.

storage-util (storage/util/)

Shared io_uring abstractions used by both ublk and overlaybd.

• AsyncIoRing<S>: generic async io_uring wrapper with slab-based RingFuture for CQE delivery. Supports standard (64B) and extended (128B) SQE types.
• IoRingWorker: spawns dedicated worker threads with thread-local io_uring instances. MPSC channel submission eliminates cross-thread locking.
• ReloadableIDAllocator: O(1) bitmap-based ID allocation/recycling with free list. Supports reloading pre-occupied IDs on restart.

Sandbox integration (src/sandbox/ublk/ + src/sandbox/extra_drive.rs)

• device.rs: owns the process-wide UblkDeviceManager, which talks to uvm-ublk-daemon and creates / deletes / snapshots all runtime ublk devices.
• overlaybd.rs: materializes runtime configs (rewrites paths, creates symlinks to layer files) for rootfs and attached drives.
• extra_drive.rs: prepares user-specified extra block drives with rollback on failure. Read-only and writable drives now follow the same per-sandbox device lifecycle; the only semantic difference is whether overlaybd materializes a writable upper.

Memory Snapshot Restore

Memory snapshot restore uses ublk-backed overlaybd devices rather than userfaultfd. On resume, a read-only ublk device is created from the stacked memory overlaybd layers and passed to Firecracker as a BackendType::File memory backend. Firecracker mmaps the block device and COWs pages into anonymous memory on first write, so the underlying device is never modified.

Sharing: Multiple sandboxes booting from the same snapshot template share a single memory ublk device via reference counting. This allows the Linux page cache to be reused across all sandboxes using the same memory image, significantly reducing I/O for concurrent launches from the same template.

Memory snapshot creation: On pause, Firecracker creates a state-only diff snapshot. AgentENV queries Firecracker’s dirty/present memory ranges, reads the selected memory with process_vm_readv, and directly creates the OverlayBD memory layer. Parent layers from previous snapshots are stacked, forming the full layered memory image.

Note: storage/uffd-core/ contains an alternative userfaultfd-based memory restore implementation that is retained for reference but excluded from the workspace build.

Per-Node Subsystems

Each node is an AgentENV server binary (src/bin/server.rs) on a Linux host with /dev/kvm and one configured virtualization mode. KVM is the default; PVM currently requires x86_64 and the kvm_pvm host module.

Sandbox Networking

The network subsystem is managed by a process-wide NetworkManager and per-slot Slot objects. See Sandbox Network Architecture for the namespace topology, address plan, packet paths, firewall ordering, egress proxy, policy replacement, warm-pool behavior, and verification steps.

Snapshot resume can also use [pool.firecracker] to pre-spawn (network slot, Firecracker process) pairs. A warm entry transfers its network slot, process, and Firecracker CWD to the resumed sandbox, which avoids the spawn and API-socket wait in the resume critical path. [pool.block] controls the ublk daemon’s overlaybd warm-device pool; it shares the same top-level watermarks but performs async refill from request paths because reusable block devices are image/size-specific.

Observability Data Flow

The node observability path combines request-time host collection with request-time projection:

• src/orchestrator/metrics.rs maintains incremental runtime counters during lifecycle operations, including running sandbox count, starting sandbox count, allocated CPU/memory, and create success/failure totals.
• src/orchestrator/service.rs publishes those counters through a tokio::sync::watch channel whenever lifecycle state changes affect the node’s runtime accounting.
• src/observability/identity.rs resolves stable node identity fields such as node ID, cluster ID, service instance ID, package version, and build-time commit.
• src/observability/machine.rs captures static machine descriptors from /proc/cpuinfo.
• src/observability/host.rs collects host CPU, memory, and disk usage each time a node snapshot is requested. CPU percent is derived from two /proc/stat samples; on the first request it takes both samples with a 100ms window to avoid returning a synthetic zero.
• src/observability/service.rs merges the latest orchestrator counters, identity, machine info, request-time host metrics, and current sandbox ID roster into a NodeSnapshot returned by the admin endpoints and reused by heartbeat reporting.
• src/observability/reporter.rs optionally sends periodic heartbeat reports to scheduler over gRPC (Heartbeat) and performs best-effort UnregisterNode on shutdown.
• Scheduler report config can be provided from TOML ([observability.scheduler_report]) and uses [cluster].scheduler_endpoint as the shared scheduler address. The reporter enable flag, address, and interval can be overridden by env vars (AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED, AENV_OBSERVABILITY_SCHEDULER_ENDPOINT, AENV_OBSERVABILITY_REPORT_INTERVAL_SECS).
• If a P2P transport exposes a local endpoint, the reporter includes it in the scheduler heartbeat so other nodes can discover it.

This keeps node requests lightweight on orchestrator data: they avoid re-listing and sorting all sandboxes on every API call while still returning fresh host metrics.

The observability subsystem has two configuration-controlled scopes:

• observability.enabled: controls whether the node observability service is constructed at all. When disabled, node/admin observability endpoints degrade rather than trying to synthesize partial snapshots.
• observability.scheduler_report.enabled: controls optional scheduler heartbeat reporting. It can be overridden by AENV_OBSERVABILITY_SCHEDULER_REPORT_ENABLED. When enabled, reporting requires [cluster].scheduler_endpoint or AENV_OBSERVABILITY_SCHEDULER_ENDPOINT.

P2P Artifact Transport

src/p2p/ provides a project-wide artifact transport abstraction for modules that need to exchange validated files between runtime nodes. Consumers depend on the P2pTransport trait, whose main operations are lookup, lookup_with_hints, fetch, publish, unpublish, local_endpoint, and shutdown.

The default DisabledP2pTransport keeps the feature inert: lookups return no descriptor, publish is a no-op, and fetch fails with TransportDisabled. The IrohBlobsP2pTransport backend starts an embedded iroh endpoint, serves artifact bytes through iroh-blobs, and serves a small AgentENV catalog protocol over the same endpoint to map stable artifact keys to transport-neutral descriptors.

One P2P artifact key represents one logical artifact. Lookup returns at most one descriptor, selected from the local catalog first and then from discovered peers in order. Artifact descriptors contain the stable key, provider node ID, optional provider endpoint, backend-specific locator string, and module-defined JSON metadata. Backend locators stay opaque to callers; for iroh the locator is the iroh-blobs hash used for content-addressed fetch.

A successful remote fetch also best-effort advertises the fetched blob from the local node. This makes the fetching node a provider for later peers and lets artifacts spread through the cluster.

Peer discovery is decoupled behind P2pPeerDiscovery. In normal multi-node deployments, SchedulerPeerDiscovery periodically calls scheduler ListP2pPeers, filters by backend and cluster, and excludes the local node. StaticP2pPeerDiscovery and NoopP2pPeerDiscovery cover tests and disabled/local-only operation.

Snapshot publishing also uses the P2P layer as a best-effort acceleration path. After a snapshot repository commit succeeds, SnapshotManager advertises the fixed Firecracker artifacts and overlaybd layers. OSS-backed snapshot resolution tries P2P before object storage for fixed artifacts; POSIX-backed resolution does not consume P2P because the POSIX repository path is already the committed artifact source. Overlaybd layer reads are accelerated by the overlaybd P2P HTTP facade rather than by snapshot resolvers.

See P2P Artifact Transport for the detailed design.

Node API endpoints (E2B-compatible):

• POST /sandboxes create a sandbox
• GET /sandboxes list sandboxes
• GET /sandboxes/{id} get sandbox metadata
• DELETE /sandboxes/{id} delete a sandbox
• POST /sandboxes/{id}/pause pause (snapshot) a sandbox
• POST /sandboxes/{id}/resume resume from snapshot
• GET /nodes return node-level observability snapshots
• GET /nodes/{id} return node details plus currently running sandboxes
• ANY /proxy, ANY /proxy/{path}, routing-header fallback, and configured sandbox proxy hosts reverse proxy to sandbox services

Distributed Control Plane

The multi-node control plane in services/ routes client traffic across multiple AgentENV backend nodes.

flowchart LR
    client["Client"] -->|HTTP| gateway["Gateway<br/>(:8080)"]
    gateway -->|gRPC| scheduler["Scheduler<br/>(:9090)"]
    gateway -->|proxy HTTP| nodeA["Node A<br/>(:8000)"]
    gateway -->|proxy HTTP| nodeB["Node B<br/>(:8000)"]
    scheduler -.->|node selection /<br/> lookup result| gateway

Gateway (services/gateway/): HTTP reverse proxy. Extracts sandbox data-plane routes from headers (x-agentenv-sandbox-id / e2b-sandbox-id) or configured host-based proxy domains ({port}-{sandboxID}.{domain}). Host-based routes are only enabled for explicit gateway.sandbox_proxy_domains entries, require RFC 952/1123 DNS-label-compatible sandbox IDs, and require the full {port}-{sandboxID} label to fit the 63-character DNS label limit. Runtime nodes have their own [sandbox_proxy].domains setting for the same host-based URL shape and return the first configured domain in sandbox metadata. In multi-node deployments, repository helpers can apply one SANDBOX_PROXY_DOMAINS value to both gateway and runtime node configuration. Sandbox control-plane routes such as /sandboxes/{id}/pause are routed by sandbox ID from the URL path; sandbox data-plane traffic is not inferred from URL path alone. For new sandboxes, calls Schedule() to pick a node. For existing sandboxes, calls LookupNode(). After sandbox creation, calls RecordAssignment() to seed a sandbox-to-node binding. Without explicit routing headers, it also handles cluster aggregation of GET /sandboxes, GET /v2/sandboxes, GET /nodes, and resolves GET /nodes/{id} via scheduler before proxying to the resolved node.

Scheduler (services/scheduler/): gRPC service with pluggable node discovery and in-memory sandbox-to-node bindings, plus observed-node snapshots reported by runtime nodes. RPCs include Schedule, LookupNode, RecordAssignment, Heartbeat, ListObservedNodes, ListP2pPeers, GetNode, and UnregisterNode. Strategies: round_robin (default), random. Proto contract: services/api/proto/scheduler.proto. For P2P, scheduler stores and returns opaque peer endpoints from heartbeat records; artifact catalog lookup and byte transfer stay node-to-node.

Binding lifecycle:

• RecordAssignment creates the initial binding immediately after sandbox creation succeeds.
• Runtime heartbeats include the node’s full sandbox ID roster. Scheduler treats that roster as the source of truth for that node and removes bindings missing from the latest heartbeat.
• binding_ttl is a freshness TTL for routing information, not a copy of sandbox timeout. If a binding stops being refreshed by gateway or heartbeats, scheduler drops it on the next lookup or roster reconcile.
• UnregisterNode removes the observed node record and proactively clears bindings owned by that node.

Discovery modes:

• static: explicit scheduler.nodes list from config
• kubernetes: EndpointSlice watch over the headless agentenv-nodes Service, using ready DaemonSet Pod IPs as backend endpoints

Limitations: All bindings are in-memory (lost on scheduler restart). After a scheduler restart, bindings are rebuilt from new sandbox creations plus the next heartbeat roster from each runtime node. Kubernetes discovery updates the schedulable node set dynamically, but binding persistence is still not replicated.

Deployment:

export AENV_API_KEY="e2b_$(openssl rand -hex 32)" # shared by local runtime and gateway processes
# local dev (single node)
make start-server && make -C services run-scheduler && make -C services run-gateway

# docker compose (multi-node; shared auth volume is provisioned automatically)
make deploy-up     # gateway + scheduler + 2 backend nodes
make deploy-down   # teardown

# kubernetes (gateway + scheduler + daemonset runtime nodes)
make k8s-render
make k8s-apply

In Kubernetes deployments, AgentENV runtime nodes run as a privileged DaemonSet so each host gets exactly one runtime Pod with access to /dev/kvm, iptables/network-namespace operations, and a hostPath-backed workspace cache. Runtime Pods on a host must all use the host’s selected KVM/PVM mode. The deployment helpers materialize the DaemonSet ConfigMap from config/default.toml at render/apply time so AgentENV runtime config remains single-sourced.

Directory Structure

storage/
├── overlaybd/src/              # layered image format (core)
│   ├── image/                  # high-level image abstraction
│   │   ├── image_file.rs       # ImageFile: reads/writes across the layer stack
│   │   ├── image_service.rs    # shared io_uring and image services
│   │   ├── helper.rs           # runtime upper preparation, path rewriting
│   │   └── snapshot.rs         # explicit upper export
│   ├── lsmt/                   # LSMT layer stacking
│   │   ├── file/               # LSMTReadOnlyFile, LSMTFile, stack helpers
│   │   ├── format.rs           # binary format (HeaderTrailer, DiskSegmentMapping)
│   │   └── index.rs            # segment mapping
│   ├── compression/zfile.rs    # zstd compression + jump tables
│   └── backend/                # pluggable VirtualFile backends
│       ├── local.rs            # LocalFile backend (io_uring)
│       ├── registryfs_v2.rs    # OCI registry backend
│       └── tar.rs              # tar archive backend
├── ublk/src/                   # userspace block device server
│   ├── lib.rs                  # public API
│   ├── ctrl.rs                 # /dev/ublk-control interface
│   ├── dev.rs                  # device + queue management
│   ├── queue.rs                # I/O descriptor handling
│   ├── io_buffer.rs            # zero-copy + traditional buffers
│   └── impls/                  # target implementations
│       └── overlaybd_target.rs # OverlaybdTarget
├── ublk-daemon/src/            # ublk daemon (unix socket RPC)
│   ├── client.rs               # daemon client used by node runtime
│   ├── server.rs               # daemon server + request loop
│   └── protocol.rs             # RPC message types
├── util/src/                   # shared io_uring abstractions
│   ├── io_ring/                # AsyncIoRing, IoRingWorker
│   └── id_allocator.rs         # bitmap-based ID allocation
└── uffd-core/src/              # userfaultfd memory restore (excluded from workspace, retained for reference)
    ├── handler.rs              # UffdHandle, page fault event loop
    ├── backend.rs              # MemoryImageBackend trait
    ├── overlaybd.rs            # OverlaybdMemoryImage backend
    ├── process_vm_reader.rs    # ProcessVmReader (process_vm_readv)
    └── scm.rs                  # SCM_RIGHTS fd passing

src/
├── bin/server.rs               # node binary entrypoint
├── api/                        # HTTP API layer
├── orchestrator/               # sandbox lifecycle
├── observability/              # node identity + host/runtime metrics projection
├── sandbox/                    # Firecracker VM management
│   ├── extra_drive.rs          # extra drive preparation
│   └── ublk/                   # storage integration
│       ├── device.rs           # daemon-backed ublk device lifecycle
│       └── overlaybd.rs        # runtime config materialization
├── snapshot/                   # committed snapshot model, repository backends, runtime resolution
├── template/                   # user-facing template builder over snapshots
└── cfg.rs                      # TOML config

services/                       # distributed control plane (Go)
├── gateway/                    # HTTP reverse proxy
├── scheduler/                  # gRPC node selection + binding
├── api/proto/                  # protobuf contracts
└── shared/                     # config, logging

Sandbox Network Architecture

This document describes the network data plane for one AgentENV node. It covers the Linux network namespaces used by Firecracker sandboxes, host and namespace iptables, runtime egress policy updates, and the namespace-local egress proxy.

The user-facing policy contract is documented in the public Sandboxes documentation. This page explains how that contract is implemented.

Topology

Each running sandbox has a dedicated network namespace. The Firecracker VM is connected to the namespace through a TAP device. The namespace is connected to the host through a veth pair; the namespace side is named vpeer and the host side is named veth-{slot}.

flowchart LR
    subgraph sandbox["Sandbox network namespace"]
        direction LR
        subgraph vm["Firecracker VM"]
            direction LR
            eth0["eth0\n169.254.0.21"]
        end
        tap["tap0\n169.254.0.22"]
        vpeer["vpeer\nveth pool 10.12.0.0/16\n/31 per slot"]
        proxy["EgressProxy listener\n0.0.0.0:15000"]
        eth0 <--> tap
        tap <--> vpeer
        tap -. "TCP 80/443 REDIRECT" .-> proxy
        proxy --> vpeer
    end
    style sandbox fill:transparent,stroke:gray

    subgraph host["Host network namespace"]
        direction LR
        veth["veth-{slot}\n10.12.0.0/16\n/31 per slot"]
        host-interaction["Host interaction IP\n10.11.0.0/16\n/31 per slot"]
        internet["Internet"]
        veth <-- "Host routing" --> host-interaction
        veth -- "Host routing" --> internet
    end
    style host fill:transparent,stroke:gray

    vpeer <--> veth

The process contains one global NetworkManager and one global EgressProxy registry. The registry is shared for lifecycle and policy bookkeeping, but each namespace that uses proxy-backed policy has its own listener socket and listener thread. The fixed port 15000 is therefore reusable across namespaces without a host-port collision.

Ownership

Address Plan

NetworkAddressPlan derives addresses from [network.internal] and allocates them from the slot index:

• host_interaction_ip: one address per slot from host_interaction_cidr. Host routes use this address to reach the VM through the namespace.
• veth_host_ip and veth_vm_ip: the two endpoints of the slot’s /31 veth link. The host endpoint is assigned to veth-{slot} and the namespace endpoint is assigned to vpeer.
• vm_ip and tap_ip: fixed endpoints of the VM link on vm_link_cidr. The VM receives vm_ip; the namespace TAP interface receives tap_ip.

The namespace adds a default route through veth_host_ip. Firecracker receives an ip= boot argument containing the VM address, TAP link, netmask, and the guest DNS server selected from the host resolver configuration.

The complete internal pools are denied before user egress rules. This prevents a sandbox from reaching another slot’s host-interaction address, veth link, or VM link even when a user policy otherwise allows the destination.

Namespace Setup

Slot::create_network() performs namespace setup on a dedicated thread because network namespace membership is thread-local:

1. Create and bind-mount a persistent namespace file under the configured runtime netns directory.
2. Unshare CLONE_NEWNET and create the veth pair.
3. Move the host endpoint to the process’s baseline host namespace.
4. Configure lo, vpeer, tap0, addresses, link state, and the namespace default route.
5. Enable namespace IPv4 forwarding and configure namespace NAT/filter rules.
6. Configure the host veth address and a host route for the slot’s host_interaction_ip.

The baseline namespace rules are installed once during setup. Per-sandbox user rules and proxy redirects are replaced later by set_egress_policy().

Packet Paths

flowchart TB
    vm["Firecracker VM"] --> tap["tap0"]
    tap --> prerouting["namespace nat/PREROUTING"]
    prerouting -. "TCP 80/443 REDIRECT" .-> proxy["Namespace-local\nEgressProxy"]
    prerouting --> forward["namespace filter/FORWARD\nAGENTENV-EGRESS"]
    proxy --> upstream_proxy["Proxy upstream connection"]
    forward --> nsnat["namespace POSTROUTING\nSNAT to host_interaction_ip"]
    upstream_proxy --> nsnat
    nsnat --> vpeer["vpeer / veth"]
    vpeer --> host["Host FORWARD + POSTROUTING\nMASQUERADE"]
    host --> internet["External destination"]
    internet -. "return traffic" .-> host
    host -. "ESTABLISHED,RELATED" .-> vpeer
    vpeer -. "return traffic" .-> tap
    tap -. "return traffic" .-> vm

Sandbox-originated internet traffic

1. The VM sends a packet through eth0 to tap0.
2. The namespace NAT PREROUTING chain may redirect TCP ports 80 and 443 to the namespace-local proxy.
3. Packets that continue as routed traffic traverse filter/FORWARD from tap0 to vpeer, entering AGENTENV-EGRESS.
4. The namespace POSTROUTING chain SNATs VM-originated traffic to the slot’s host_interaction_ip. Proxy-originated upstream connections are SNATed from the namespace vpeer address to the same slot identity.
5. The host FORWARD and POSTROUTING rules accept the slot source range and MASQUERADE it for the external route.

Return traffic

Return packets are classified as ESTABLISHED,RELATED and are accepted before the user egress chain in the namespace. The host also accepts established traffic returning to veth-{slot}. This preserves host/envd/proxy responses and existing outbound flows while policy rules are replaced.

Host-to-VM traffic

Host-side proxy and envd traffic targets the slot’s host_interaction_ip. Namespace PREROUTING on vpeer DNATs that address to the VM’s fixed vm_ip. The corresponding return path is accepted by the established/related rules.

Host Firewall

NetworkManager installs one process-wide set of host rules for the configured host interaction CIDR. The rules are inserted/appended symmetrically and removed during manager shutdown:

INPUT      -i veth-+ -s <host_interaction_cidr> -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
INPUT      -i veth-+ -s <host_interaction_cidr> -j REJECT
FORWARD    -i veth-+ -s <host_interaction_cidr> -j ACCEPT
FORWARD    -o veth-+ -d <host_interaction_cidr> --state ESTABLISHED,RELATED -j ACCEPT
POSTROUTING -s <host_interaction_cidr> -j MASQUERADE

The INPUT pair prevents guest-originated new connections from reaching arbitrary host services while preserving established host-originated traffic. The FORWARD and MASQUERADE rules handle namespace-to-internet forwarding after namespace SNAT.

Namespace Firewall

Each namespace has three managed chains:

• AGENTENV-EGRESS is the static chain reached from FORWARD for guest traffic.
• AGENTENV-USER-EGRESS contains the replaceable user policy.
• AGENTENV-EGRESS-PROXY is the replaceable NAT chain reached from nat/PREROUTING for proxy-backed traffic.

The static filter chain is ordered as follows:

1. Accept ESTABLISHED,RELATED traffic from tap0 to vpeer.
2. Allow UDP/TCP DNS traffic to the configured guest DNS address on port 53.
3. Reject the complete internal address pools and configured always_denied_cidrs.
4. Jump to AGENTENV-USER-EGRESS.

The user chain is rendered in this order:

1. Explicit allowOut CIDR/IP rules (ACCEPT).
2. Explicit denyOut CIDR/IP rules (REJECT).
3. A 0.0.0.0/0 reject when the base policy is Deny.

This ordering gives user allow rules precedence over overlapping user denies, while static internal/platform denies remain non-overridable.

Egress Proxy

The egress proxy is namespace-local and transparent. It does not terminate TLS or rewrite HTTP. A policy that requires proxy mediation redirects only TCP ports 80 and 443 to 0.0.0.0:15000 inside that namespace.

Connection flow

1. EgressProxy::ensure_listener() enters the target namespace with setns, binds the listener, and starts a nonblocking accept loop.
2. The listener accepts a connection and copies the active policy for that connection.
3. The connection handler reads SO_ORIGINAL_DST to recover the destination altered by REDIRECT.
4. It buffers the initial request/handshake up to 64 KiB and extracts HTTP Host or TLS ClientHello SNI. TLS handshake bytes are accumulated across record boundaries.
5. A domain match is sent to the process-wide resolver worker, which remains in the host network namespace. Each candidate IPv4 address is checked by SandboxNetworkPolicy::is_domain_allowed, including absolute platform and internal-range denies, before it is selected.
6. A domain connection is dialed to the selected trusted address. An IP/CIDR connection uses the original destination after is_ip_allowed.
7. The buffered preface is written to the upstream and both directions are relayed with half-close handling.

An empty or unrecognized hostname cannot fall back to the base allow policy when a domain allowlist is present. This keeps domain policies fail-closed. Domain allow rules require the API request to include an explicit denyOut: ["0.0.0.0/0"].

Policy replacement

The running policy update is a replacement operation:

1. Stage the new policy in the proxy’s pending map.
2. Ensure the namespace listener exists when proxy mediation is required.
3. Apply the filter and NAT chain replacement as one iptables-restore batch.
4. When an active proxy policy already exists, activate the pending policy only after the namespace batch succeeds.
5. When proxy mediation is being enabled for a namespace with no active proxy policy, activate the prepared policy before installing the redirect. This preserves the namespace’s previous default-allow behavior until the atomic iptables batch has completed; no connection can reach the listener through REDIRECT before that batch succeeds.
6. If an update fails while an active proxy policy exists, discard the pending policy and retain the previous active policy.

Each accepted connection holds a copy of the active policy selected at accept time. Therefore updates primarily affect new connections; existing relays are not proactively terminated. Removing a policy or a slot closes tracked handler sockets and joins their threads before namespace resources are removed.

Namespace ownership

The process-global registry is keyed by host_interaction_ip, but listeners are not shared between namespaces. Every namespace has its own listener thread and socket. When a policy no longer requires proxy mediation, the listener and active/pending policy are removed. Full slot cleanup stops the listener before deleting the veth and unmounting the namespace.

Lifecycle and Warm Pool

NetworkManager::allocate_any() first acquires a warm Slot. If none is available it allocates a new slot index and creates the namespace/veth/TAP resources. The Firecracker backend then applies the launch policy before boot or resume continues.

On stop, a slot may return to the warm pool. The namespace and baseline networking remain available for reuse; user_egress_rules_present tracks whether the next tenant needs a user-chain replacement. The next policy application clears or replaces stale user rules and removes stale proxy policy as appropriate. Slots drained from the pool are fully cleaned up.

On shutdown, the manager stops pool maintenance, drains warm slots, stops all proxy listeners, removes global host rules, and removes namespace/veth state. An atexit hook and Drop path provide best-effort cleanup for abnormal exits.

Sandbox and Test Flow Guide

This document describes the public sandbox API, the global ConfigManager under AgentENV/src/cfg.rs, and what AgentENV/tests/integration/fc.rs, AgentENV/tests/integration/process.rs, AgentENV/tests/integration/snapshot.rs, and AgentENV/tests/integration/orchestrator.rs validate.

1) Sandbox Public Interface

Types and responsibilities

• FirecrackerSandboxConfig (AgentENV/src/sandbox/firecracker/config.rs)

  • Purpose: describes a fresh VM boot (kernel + tools drive + user image).
  • Key fields
    • firecracker_binary: path to the firecracker executable.
    • kernel_image: path to vmlinux.bin.
    • tools_drive_version: immutable tools drive identity. The node resolves it to <deps_path>/tools/<version>/tools.ext4 before mounting it as /dev/vda.
    • user_image_config: overlaybd config for the writable user image (/dev/vdb).
    • boot_args: kernel boot args (e.g. console=ttyS0 ... init=/init).
    • vcpu_count, mem_size_mib: VM size.
    • runtime_policy: socket/envd timeouts and poll intervals.
    • common.stdout_path, common.stderr_path: explicit capture directories. Otherwise stdout/stderr capture is enabled only when common.firecracker_log_level is non-empty, using the serial directory or work_dir/logs fallback.
    • envd_version: expected envd version for the guest image.
    • env_vars: optional default environment variables injected after envd init.
    • ublk_config: optional ublk-backed rootfs configuration.
  • Convenience
    • from_global_config() builds a config from ConfigManager using the internal default template rootfs image. The image must already be resolved under <image-cache-root>/configs.
    • from_global_config_with_user_image(...) builds a config from an already resolved overlaybd image config.
    • Runtime launches from committed snapshots are built through FirecrackerSandbox::from_snapshot(&RunnableSnapshot, &SandboxLaunchConfig).
    • uid/gid: File ownership (default 0).

• ProcessOpts (AgentENV/src/sandbox/process.rs)

  • Purpose: Options for starting a process inside the sandbox.
  • Key fields
    • envs: HashMap<String, String> for process environment variables.
    • cwd: Optional working directory.
    • timeout: Optional max time to wait for completion.
  • Builder methods: with_envs(), with_cwd(), with_timeout().

• ProcessOutput (AgentENV/src/sandbox/process.rs)

  • Purpose: Result of a completed process execution.
  • Fields: stdout, stderr, exit_code.

• ProcessHandle (AgentENV/src/sandbox/process.rs)

  • Purpose: Handle to a running process inside the sandbox.
  • Key methods
    • pid(): Returns the PID inside the guest VM.
    • wait().await: Waits for exit and collects all output.
    • send_stdin(data).await: Sends bytes to stdin.
    • send_signal(signal).await: Sends a signal (e.g. SIGTERM).
    • kill().await: Kills the process with SIGKILL.

• FirecrackerSnapshotConfig (AgentENV/src/sandbox/firecracker/config.rs)

  • Purpose: describes how to resume a VM from snapshot + memory + base disk.
  • Key fields
    • vm_state_path: Firecracker VM state file.
    • mem_overlaybd_config: runtime memory overlay image config.
    • base_rootfs_path: writable disk image or overlaybd rootfs image config paired with that snapshot.
    • firecracker_binary: FC binary to use for resuming.
    • runtime_policy: socket/envd timeouts and poll intervals.
    • envd_version: guest envd version metadata.
    • env_vars: optional default environment variables restored on start.
  • Notes
    • FirecrackerSnapshotConfig returned by pause() owns a tempdir (keeps files alive).
    • get_rootfs_path() resolves the snapshot rootfs path used by runtime tests.

• SandboxBackend and SandboxExecutor (AgentENV/src/sandbox/backend.rs)

  • Purpose: public traits for sandbox lifecycle and process execution.
  • Key methods
    • SandboxBackend: start, start_nowait, wait_for_ready, pause, resume, stop.
    • SandboxExecutor: run_command, run_command_with_opts, start_process.

• FirecrackerSandbox (AgentENV/src/sandbox/firecracker/sandbox.rs)

  • Purpose: lifecycle controller for a single Firecracker instance.
  • Main methods (detailed)
    • FirecrackerSandbox::new(FirecrackerSandboxConfig)
      • Creates a sandbox handle for a fresh boot.
      • Does not start Firecracker; it only prepares the object.
      • Each sandbox gets its own temporary work directory.
    • FirecrackerSandbox::resume_from_snapshot_config(&FirecrackerSnapshotConfig)
      • Creates a new sandbox handle and immediately starts it from a snapshot config.
      • Uses vm_state_path, the memory overlay image config, and base_rootfs_path from the config.
      • Returns a running sandbox after Firecracker and envd are ready.
    • FirecrackerSandbox::from_snapshot(&RunnableSnapshot, &SandboxLaunchConfig)
      • Creates a sandbox handle from committed snapshot state resolved for a node.
      • Uses snapshot vm_state.bin plus node-local materialized memory/image.json and rootfs/image.json.
      • Does not start Firecracker; call start().await afterward.
    • start().await
      • Spawns the Firecracker process, waits for the API socket, starts the VM, then waits for guest-level envd readiness.
    • pause().await
      • Sends PATCH /vm to pause the VM.
      • Creates snapshot artifacts including vm_state.bin, mem_image.json, mem_overlaybd/overlaybd.commit, and rootfs snapshot state.
      • Returns a FirecrackerSnapshotConfig that owns the tempdir holding these files.
    • resume().await
      • Resumes a paused VM in-place by sending PATCH /vm with Resumed.
      • Use this when you want to keep the same sandbox instance.
    • stop().await
      • Stops Firecracker, releases network resources, and cleans up optional ublk state.
    • run_command(cmd, args).await
      • Runs a command inside the sandbox via envd gRPC and waits for it to complete. Returns ProcessOutput with stdout, stderr, and exit code.
    • run_command_with_opts(cmd, args, opts).await
      • Same as run_command but accepts ProcessOpts for setting env vars, working directory, and execution timeout.
    • start_process(cmd, args, opts).await
      • Starts a long-running process inside the sandbox. Returns a ProcessHandle that can stream output, send stdin, or kill the process.
    • firecracker_stdout_path() / firecracker_stderr_path()
      • Resolved log paths used by this sandbox (including defaults).
    • work_rootfs_path()
      • Returns the path to the per-instance writable rootfs inside the work directory. Useful for inspecting or copying disk state after a run.

What happens under the hood (important for correct usage)

• Each FirecrackerSandbox uses its own temp work directory.
• Kernel is used via the configured host path.
• When resuming, the writable rootfs and memory image are materialized into the work dir so each instance can mutate independently. For overlaybd-backed snapshot-backed template launches, this means recreating runtime image configs and fresh writable uppers from the committed layer stack.
• Rootfs copies use copy-on-write helpers where available.
• Firecracker runs with current_dir = work_dir, so relative paths work.
• Firecracker logging is disabled by default. Set firecracker.log_level to enable stdout/stderr capture and firecracker.log under the configured serial directory. Direct Rust callers can also opt individual streams in using explicit destinations.
• start() waits for both the API socket and envd readiness.
• Although the Firecracker process will be killed on drop, explicit stop() is recommended.

Minimal usage example

#![allow(unused)]
fn main() {
use agentenv::sandbox::{FirecrackerSandbox, FirecrackerSandboxConfig, SandboxExecutor};

let mut cfg = FirecrackerSandboxConfig::new(
    "/path/to/firecracker".into(),
    "/path/to/vmlinux.bin".into(),
    "/path/to/rootfs.ext4".into(),
);
cfg.boot_args = Some("console=ttyS0 reboot=k panic=1 pci=off init=/init".into());

let mut sandbox = FirecrackerSandbox::new(cfg)?;
sandbox.start().await?;

// Run a command inside the VM
let output = sandbox.run_command("echo", &["hello"]).await?;
assert_eq!(output.exit_code, 0);
println!("{}", output.stdout);

let snapshot = sandbox.pause().await?;
sandbox.stop().await?;

let mut resumed_sandbox = FirecrackerSandbox::resume_from_snapshot_config(&snapshot).await?;
resumed_sandbox.stop().await?;
}

Runtime prerequisites

• Linux host
• /dev/kvm accessible by the current user, with the host modules matching virtualization_mode (KVM by default; PVM requires x86_64 and kvm_pvm)
• debugfs (e2fsprogs) if you use init injection or disk-inspection helpers

2) Global Config Manager (src/cfg.rs)

The tests and benchmarks use a shared global config manager based on config/default.toml.

Config file format

AgentENV/config/default.toml (override via AENV_CONFIG_PATH or --config)

[firecracker]
boot_args = "your-kernel-boot-args"
binary_path = "/path/to/firecracker"
socket_timeout_secs = 3
socket_poll_ms = 20
# Optional override; defaults to $AENV_HOME/firecracker-work.
# work_dir = "/path/to/firecracker-work"
# Optional override; defaults to $AENV_HOME/logs/serial.
# serial_dir = "/path/to/serial"
# Optional; enables stdout/stderr capture and firecracker.log when non-empty.
# log_level = "Info"

[kernel]
image_path = "/path/to/vmlinux.bin"

[tools]
version = "0.1.0-custom.1"
drive_path = "/path/to/tools.ext4"

[machine]
mem_size_mib = 128
vcpu_count = 1

[envd]
version = "0.5.15"
init_timeout_secs = 30
poll_ms = 10

[orchestrator]
auto_evict_interval_ms = 1000
default_sandbox_timeout_secs = 15

[snapshot]
repository_backend = "posix_fs"
local_cache_path = "$AENV_HOME/snapshot-local-cache"

[backend.posix_fs]
snapshot_store = "$AENV_HOME/snapshot-store"

[ublk]
enabled = false

TOML parameter reference

• [firecracker]

  • boot_args: Kernel command line. The runtime appends init=/init if not already present (tools drive provides the init script).
  • binary_path: Absolute or relative path to firecracker.
  • socket_timeout_secs: Max time to wait for the Firecracker API socket.
  • socket_poll_ms: Poll interval for checking socket existence.
  • work_dir: Optional override for the per-sandbox Firecracker working directories. These directories include runtime sockets, symlinks, local logs, and writable OverlayBD upper layer data (overlaybd/upper.data and overlaybd/upper.index). Defaults to $AENV_HOME/firecracker-work.
  • serial_dir: Optional override for the persistent output directory. When logging is enabled, serial output is written under a per-sandbox subdirectory and not removed by the sandbox. Defaults to $AENV_HOME/logs/serial. Setting this path alone does not enable logging.
  • log_level: Optional Firecracker log level (Error, Warning, Info, Debug, Trace, case-insensitive). When set to a non-empty value, stdout/stderr capture and firecracker.log are enabled in the sandbox’s log directory. If omitted or empty, stdout/stderr are discarded and no log directories are created, unless explicit Rust capture destinations are set.

• [kernel]

  • image_path: Optional local kernel image path (vmlinux.bin). If omitted, AgentENV derives it from deps_path and the bundled dependency manifest.

• [tools]

  • drive_path: Optional local tools drive source. When set, an explicit version is required and setup imports it into the immutable version directory.

• Template rootfs image

  • User-visible rootfs images are selected when starting template builds through the optional fromImage field.
  • If fromImage is omitted, AgentENV uses [image.resolver].default_image.
  • Standard Docker Hub shortnames such as ubuntu:24.04 and node:20 are normalized before image resolution.
  • Resolved image configs are cached under <image-cache-root>/configs/<slug>-<hash>-image.json; local layer products are cached under <image-cache-root>/layers/; overlaybd-native remote block reads are cached under <image-cache-root>/remote-blocks/.
  • Registry authentication uses docker credentials from ~/.docker/config.json or $DOCKER_CONFIG/config.json.

• [machine]

  • mem_size_mib: Guest RAM size (default: 128).
  • vcpu_count: Number of vCPUs (default: 1).

• [envd]

  • version: Expected envd version baked into the runtime image.
  • init_timeout_secs: Max time to wait for the in-guest envd daemon to become ready after VM start (default: 30).
  • poll_ms: Poll interval for envd health-check retries (default: 10).

• [orchestrator]

  • auto_evict_interval_ms: Poll interval for background timeout eviction.
  • default_sandbox_timeout_secs: Default keep-alive timeout used by the orchestrator.
  • auto_resume_min_sandbox_timeout_secs: When a data-plane request targets a non-running sandbox, automatically resume it (if auto-resume is enabled) and refresh its timeout for no-less than this duration.

• [pool]

  • low_watermark: Shared lower bound for warm resources. Defaults to 2.
  • high_watermark: Shared upper bound target for warm resources. Defaults to 64.
  • [pool.network].maintenance_enabled: Enables the background network-slot maintenance worker. Defaults to true.
  • [pool.block].enabled: Enables the ublk overlaybd warm-device pool.
  • [pool.firecracker].enabled: Enables pre-spawned Firecracker processes for snapshot resume.

• [snapshot]

  • local_cache_path: Manager-owned node-local snapshot artifact/cache root. Defaults to $AENV_HOME/snapshot-local-cache.
  • repository_backend: Snapshot repository backend. Defaults to posix_fs.

• [backend.posix_fs]

  • snapshot_store: Committed snapshot store root. Defaults to $AENV_HOME/snapshot-store.

• [ublk]

  • daemon_binary_path: Optional path to uvm-ublk-daemon.
  • daemon_socket_path: Optional unix socket path for daemon RPCs.
  • daemon_log_path: Optional log file path for the daemon process.

• [ublk.overlaybd]

  • global_config_path: path to the generated overlaybd runtime config. Required for ublk. Per-image configs are derived from template build fromImage values and live under <image.cache.root_dir>/configs.

Defaults and optional fields

• [firecracker].binary_path: Optional. If omitted, AgentENV derives it from deps_path and the bundled dependency manifest.
• [firecracker].boot_args: Optional. Defaults to console=ttyS0 reboot=k panic=1 pci=off.
• [firecracker].socket_timeout_secs: Optional. Defaults to 3 seconds.
• [firecracker].socket_poll_ms: Optional. Defaults to 20 ms.
• [firecracker].work_dir: Optional override. Defaults to $AENV_HOME/firecracker-work.
• [firecracker].serial_dir: Optional override. Defaults to $AENV_HOME/logs/serial.
• [kernel].image_path: Optional. If omitted, AgentENV derives it from deps_path and the bundled dependency manifest.
• [tools].drive_path: Optional local source override; requires an explicit [tools].version when set.
• [tools].url: Optional registry override; requires an explicit [tools].version when set.
• [machine]: Optional. If omitted, defaults are 128 MiB RAM and 1 vCPU.
• [envd]: Optional, but template and sandbox flows expect a valid version.
• [envd].init_timeout_secs: Optional. Defaults to 30 seconds.
• [envd].poll_ms: Optional. Defaults to 10 ms.
• [orchestrator]: Optional. If omitted, orchestrator runtime defaults are used.
• [pool]: Optional. If omitted, network pool maintenance is enabled with low watermark 2 and high watermark 64; block and Firecracker process pools are disabled unless enabled in their component subsections.
• [snapshot]: Optional. If omitted, repository_backend defaults to posix_fs and the local cache root derives from AENV_HOME.
• [backend.posix_fs]: Optional. If omitted, the POSIX snapshot store defaults to $AENV_HOME/snapshot-store.
• [ublk]: Optional. If omitted, ublk is disabled.

Environment-based configuration

The config loader reads the full TOML file from the following sources:

• AENV_CONFIG_PATH
• --config <path>
• config/default.toml

Runtime image and binary paths are configured in the TOML file itself.

Required fields

• [kernel].image_path
• [tools].version when [tools].drive_path or [tools].url is overridden
• [envd].version

If the [orchestrator] section is present, it must include default_sandbox_timeout_secs.

ConfigManager (recommended entry point)

The helper is strict. If requirements are not met (missing files, invalid config), it returns an error and the test fails.

Path resolution order

1. AENV_CONFIG_PATH environment variable
2. --config CLI flag
3. config/default.toml (default)

Helper methods

• ConfigManager::global() -> Result<&'static ConfigManager>
  • Lazily initializes and returns a process-wide singleton.
• ConfigManager::new() -> Result<ConfigManager>
  • Loads config using the default resolution order.
• ConfigManager::new_from_path(path) -> Result<ConfigManager>
  • Loads a specific TOML file.
• ConfigManager::config() -> &AppConfig
  • Returns the parsed application config.
• ConfigManager::global_config() -> Result<&'static AppConfig>
  • Returns the global parsed config directly.
• ConfigManager::get_orchestrator_config() -> Option<OrchestratorConfig>
  • Returns orchestrator settings when configured.
• AppConfig::resolved_snapshot_store() -> PathBuf
  • Resolves the effective committed snapshot store root.
• AppConfig::resolved_snapshot_local_cache_path() -> PathBuf
  • Resolves the effective manager-owned node-local snapshot cache root.

Typical test setup flow

#![allow(unused)]
fn main() {
use agentenv::sandbox::{FirecrackerSandbox, FirecrackerSandboxConfig, SandboxExecutor};

let sandbox_config = FirecrackerSandboxConfig::from_global_config()?;

let mut sandbox = FirecrackerSandbox::new(sandbox_config)?;
sandbox.start().await?;

// Run commands via envd gRPC
let output = sandbox.run_command("echo", &["hello"]).await?;
assert_eq!(output.exit_code, 0);

let snapshot = sandbox.pause().await?;
sandbox.stop().await?;

let mut resumed_sandbox = FirecrackerSandbox::resume_from_snapshot_config(&snapshot).await?;
resumed_sandbox.stop().await?;
}

3) What the Integration Tests Validate

integration/fc.rs

• microvm_startup_moves_to_running Boots a fresh VM and verifies that:

  • Firecracker starts and accepts API calls.
  • Kernel + rootfs can be configured without error.
  • The VM can be cleanly shut down afterward.

• microvm_pause_transitions_to_paused Validates pause + snapshot creation by:

  • Pausing a running VM via FC API.
  • Creating vm_state.bin, mem_image.json, and mem_overlaybd/overlaybd.commit.
  • Ensuring those files exist on disk.

• microvm_resume_from_pause_transitions_to_running Validates in-place resume by:

  • Pausing a VM.
  • Resuming it in the same sandbox instance.
  • Confirming the control flow completes without error.

• microvm_resume_transitions_to_running Validates resume into a new sandbox by:

  • Pausing and snapshotting a VM.
  • Shutting down the original sandbox.
  • Resuming from the snapshot in a new sandbox instance.

• snapshot_preserves_disk_state_on_resume Confirms disk state is preserved by:

  • Writing a marker file via run_command.
  • Pausing and resuming from the snapshot.
  • Reading the marker file after resume and asserting it persists.

• snapshot_chain_survives_after_parent_snapshot_handle_is_dropped Validates multi-level snapshot chains by:

  • Creating a snapshot, resuming, creating a second snapshot.
  • Dropping the first snapshot handle.
  • Resuming from the second snapshot and verifying disk state.
  • Also tests pause_to_dir with snapshot-owned inherited layer adoption.

• microvm_can_access_internet Confirms guest networking works before and after snapshot resume.

• multiple_resumes_have_independent_disk_state Confirms multiple resumptions are isolated by:

  • Resuming multiple VMs from a single snapshot sequentially.
  • Each resumed VM verifies the shared marker and writes its own file.
  • Verifying each snapshot has independent state.

integration/process.rs

• run_command_captures_stdout Runs echo hello world and verifies stdout contains the expected string.

• run_command_captures_stderr Runs a shell command that writes to stderr and verifies stderr capture.

• run_command_reports_exit_code Runs exit 42 and verifies the exit code is reported correctly.

• run_command_with_opts_sets_env_and_cwd Verifies that environment variables and working directory options are honoured.

• run_command_handles_timeout Verifies long-running commands fail when a timeout is configured.

• run_command_with_unbounded_output Verifies excessive output is rejected instead of buffering forever.

• start_process_interactive_stdin Starts cat, sends data via stdin, kills the process, and verifies the echoed output.

• start_process_send_signal Starts sleep 300, sends SIGTERM, and verifies the process terminates with a non-zero exit code.

• run_command_after_snapshot_resume Pauses a sandbox, resumes from a snapshot config, and verifies run_command works on the resumed instance.

integration/snapshot.rs

• built_snapshot_can_be_loaded_by_alias_and_launched Builds a template-backed snapshot, loads it by alias, launches a sandbox from the resolved runnable snapshot, and verifies captured runtime state.

• listed_committed_snapshot_can_be_loaded_and_resolved Verifies SnapshotManager::list_committed() returns usable committed snapshots that can be loaded and resolved into runnable paths.

• loaded_runnable_snapshot_by_alias_can_be_resolved Verifies SnapshotManager::load_runnable() resolves alias-based lookups into runnable snapshot state in one step.

• delete_committed_snapshot_by_alias_removes_it_from_repository Verifies deleting a committed snapshot by alias removes the durable record.

• snapshot_rebuild_from_committed_snapshot_preserves_base_state Verifies rebuilding from a committed snapshot preserves the base filesystem state while adding new changes.

• rebuild_merges_env_metadata_and_list_filter_can_select_by_id Verifies rebuilds merge environment metadata correctly and that committed snapshot listing can filter by snapshot id.

integration/orchestrator.rs

• orchestrator_lifecycle Verifies the orchestrator can create, fetch, list, filter, keep alive, and delete template-backed sandboxes while keeping proxy lookup state in sync.

4) End-to-End Execution Checklist

1. Provide Firecracker binary, kernel, tools drive, overlaybd runtime, and ublk daemon. These are automatically downloaded when the server starts, or you can run cargo run --bin server -- --setup-only to provision runtime dependencies independently.
2. Provision host access once as root with server --setup-host --runtime-user <user> --runtime-group <group>. The group is the runtime service group: it owns AgentENV state and receives ublk device access. Normal server startup performs validation only and never invokes sudo.
3. Host setup installs a udev rule for /dev/ublk-control, /dev/ublkc*, and /dev/ublkb*, so the runtime group can access the control and dynamic device nodes.
4. Update config/default.toml paths, or point AENV_CONFIG_PATH to a custom config file.
5. Ensure /dev/kvm is accessible by the runtime user and the configured virtualization mode matches the host modules.
6. If you run template tests, ensure the host can run regctl (server setup installs it automatically from the [regclient] manifest entry) and access the registry for template fromImage resolution.
7. Run scripts/tests/e2e/run_e2e.sh for API-level E2E coverage. The runner exports E2E_TEMPLATE_USER_IMAGE. Suite 05_template_lifecycle.sh also creates a template build with E2E_SHORT_USER_IMAGE to verify short-name image resolution.
8. Run make test-agent-integration to run the agentenv integration test modules in tests/integration/ as a non-root user with the required capabilities, plus the Docker/MinIO-backed OSS snapshot repository test (crates/e2e-tests/tests/snapshot_oss_e2e_test.rs).
9. Run make test-ublk to run the uvm-ublk and overlaybd storage tests, including the Docker/MinIO-backed OSS backend test (storage/overlaybd/tests/oss_backend_minio.rs).

4.1) Firecracker Client and Runtime Upgrade Checklist

When bumping AgentENV to a patched Firecracker runtime, update both the generated client API and the binary that setup downloads:

1. Replace thirdparty/firecracker-client/firecracker.yaml with the target Firecracker OpenAPI spec.

2. Run make firecracker-client from the repository root and review generated changes under thirdparty/firecracker-client/.

3. Build the patched Firecracker release binary on a Linux host. Package it as a gzip tar archive named firecracker-{version}-{arch}.tgz containing a firecracker executable.

4. Update the [firecracker] entry in config/deps_manifest.toml with the new version and a download URL template that supports {version} and {arch}.

5. Before relying on the URL in tests, verify it is a direct download:

   curl -L "<url>" -o /tmp/firecracker.tgz
   tar -tzf /tmp/firecracker.tgz
   

   The archive listing should include firecracker.

6. Run setup against a clean or temporary Firecracker dependency directory so it really downloads the archive:

   cargo run --bin server -- --config /path/to/config.toml --setup-only
   

7. Validate the Rust side after code generation:

   cargo fmt --check -p firecracker_client -p agentenv
   cargo check -p firecracker_client
   cargo check -p agentenv
   cargo test -p agentenv --lib
   

5) Running Custom Scripts

Run commands via envd

Once the sandbox is started (with a rootfs that includes the envd daemon), you can execute commands inside the VM using the process API:

#![allow(unused)]
fn main() {
// Simple command
let output = sandbox.run_command("ls", &["-la", "/tmp"]).await?;
println!("exit={} stdout={}", output.exit_code, output.stdout);

// Command with options
use agentenv::sandbox::ProcessOpts;
let opts = ProcessOpts::new()
    .with_cwd("/tmp")
    .with_envs([("KEY".into(), "value".into())].into());
let output = sandbox.run_command_with_opts("env", &[], opts).await?;

// Interactive / long-running process
let mut handle = sandbox.start_process("cat", &[], ProcessOpts::default()).await?;
handle.send_stdin(b"hello\n").await?;
handle.kill().await?;
let output = handle.wait().await?;
}

The process API uses envd’s gRPC ProcessClient under the hood. It requires the envd daemon to be running inside the guest VM.

Run different scripts for multiple snapshot resumes

/init is not re-executed after resume, so per-instance behavior must come from per-instance disk contents.

A typical pattern:

1. Start a base VM and pause() to create a snapshot.
2. For each instance, copy the snapshot rootfs and write a per-instance file into it (for example using debugfs).
3. Create a FirecrackerSnapshotConfig that points to that per-instance rootfs and resume.

The guest must keep a long-running dispatcher (started during the initial boot) that watches for these per-instance files and executes them after resume.

Template Builder and Testing

This document explains how the current template-facing build API maps onto the snapshot-first internals, how committed snapshots move through the repository/runtime boundary, and what the relevant tests validate.

1) Public Surface vs Internal Model

User-facing builder API

• TemplateBuildSpec (src/template/build_spec.rs)

  • Describes a declarative build request.
  • Builder helpers include:
    • from_existing_rootfs(path)
    • from_overlaybd_configs(global_config_path, image_config_path)
    • run(cmd)
    • env(key, value)
    • workdir(path)
    • apt(packages)
    • alias(alias)
    • resources(cpu_count, memory_mib)
  • The current builder implementation accepts overlaybd-backed fresh builds and rebuilds from committed snapshots. Fresh ext4-rootfs builds are not wired through TemplateBuilder.

• TemplateBuilder (src/template/builder.rs)

  • Preserves the external template semantics for build / rebuild flows, while committed snapshot lifecycle operations live in SnapshotManager.
  • Main methods:
    • new()
    • with_local_store_root(path)
    • build_and_publish(snapshot_manager, config).await
    • rebuild_and_publish(snapshot_manager, config, snapshot_id, base_snapshot).await

Snapshot-first internals

• StoredSnapshot / RunnableSnapshot (src/snapshot/types/)

  • StoredSnapshot is the durable committed manifest state.
  • It stores snapshot metadata plus logical rootfs / attached-drive layer descriptions and memory layers.
  • Snapshot publication also persists firecracker-manifest.json for launch-time metadata such as rootfs/memory virtual size and attached-drive read_only flags.
  • It does not store fixed local artifact locations such as mem_image.json or runtime-derived rootfs/image.json.
  • RunnableSnapshot is a node-local resolved view with concrete artifact paths and runtime-ready overlaybd image configs.

• SnapshotRepository / SnapshotRuntimeResolver (src/snapshot/repository/interfaces.rs)

  • SnapshotRepository owns committed durable state.
  • SnapshotRuntimeResolver turns committed state into node-local runnable paths.

2) Build and Publish Flow

TemplateBuilder::build_and_publish(...) does the following:

1. Prepare the build base from either:
   • a committed snapshot (rebuild_and_publish)
   • overlaybd configs (from_overlaybd_configs)
2. Start a temporary Firecracker sandbox with the requested CPU/memory.
3. Execute template steps in order.
4. Probe envd, kernel, and firecracker versions from the running guest and executable.
5. Pause the sandbox and export a FirecrackerSnapshotManifest carrying metadata for repository publication
6. Publish those local artifacts into the configured snapshot repository.

The important boundary is:

• the builder API describes what to build
• local build artifacts are manager-owned temporary outputs
• committed snapshot records store logical durable state
• backend layout conventions determine where committed files and shared layers live
• runtime resolution turns committed snapshot state into node-local runnable paths

3) Repository Layout

The default backend is the POSIX filesystem backend. Given a backend root like /path/to/store/repository:

• committed snapshot manifest:
  • /path/to/store/repository/snapshots/<id>/snapshot.json
• committed fixed-layout files:
  • /path/to/store/repository/snapshots/<id>/vm_state.bin
  • /path/to/store/repository/snapshots/<id>/firecracker-manifest.json
• shared managed layers:
  • /path/to/store/repository/managed-layers/<digest>.overlaybd.commit
• alias bindings:
  • /path/to/store/repository/catalog/aliases/<alias>
  • /path/to/store/repository/catalog/aliases/<alias>.lock

Alias locking is per-alias, not a single global aliases.lock file.

snapshot.json intentionally does not repeat local build-artifact paths such as mem_image.json or rootfs/image.json. Runtime code derives node-local image configs from:

• repository root
• snapshot id
• attached drive id
• managed layer digest

firecracker-manifest.json complements this by persisting launch-time virtual size and attached-drive mode metadata without embedding node-local runtime paths.

4) Load and Launch Flow

Typical runtime usage is:

1. SnapshotManager::load_committed(id_or_alias).await
2. SnapshotManager::resolve_runnable(stored).await
3. FirecrackerSandbox::from_snapshot(&runnable, &SandboxLaunchConfig::default())
4. sandbox.start().await

If you want the two snapshot-manager calls combined, use SnapshotManager::load_runnable(id_or_alias).await.

resolve_runnable(...) is where backend-neutral committed state becomes node-local launch inputs:

• memory_layers -> runtime memory/image.json
• rootfs.layers -> runtime rootfs/image.json
• attached_drives[].layers -> runtime drives/<id>/image.json
• attached_drives[].read_only -> runtime mount mode for drives/<id>
• firecracker-manifest.json -> runtime manifest hydrated with node-local artifact paths
• committed vm_state.bin -> runnable vm-state path

5) Minimal Example

#![allow(unused)]
fn main() {
use agentenv::cfg::ConfigManager;
use agentenv::image::ImageResolver;
use agentenv::sandbox::{FirecrackerSandbox, SandboxExecutor, SandboxLaunchConfig};
use agentenv::snapshot::SnapshotManager;
use agentenv::template::{TemplateBuildSpec, TemplateBuilder};

let builder = TemplateBuilder::new();
let snapshot_manager = SnapshotManager::new()?;
let config = ConfigManager::global()?.config();
let image_resolver = ImageResolver::new(config);
let image_config = image_resolver
  .resolve(image_resolver.default_image())
  .await?
  .overlaybd_config_path;

let alias = "my-template-v1";

builder
    .build_and_publish(
        &snapshot_manager,
        TemplateBuildSpec::new()
            .from_overlaybd_config(image_config)
            .alias(alias)
            .resources(1, 128)
            .run("mkdir -p /workspace")
            .workdir("/workspace")
            .env("MARK", "ready")
            .run("printf '%s' \"$MARK\" > mark.txt"),
    )
    .await?;

let runnable = snapshot_manager
    .load_runnable(alias)
    .await?
    .expect("template should exist");

let mut sandbox =
    FirecrackerSandbox::from_snapshot(&runnable, &SandboxLaunchConfig::default())?;
sandbox.start().await?;

let out = sandbox.run_command("cat", &["/workspace/mark.txt"]).await?;
assert_eq!(out.exit_code, 0);
assert_eq!(out.stdout.trim(), "ready");

sandbox.stop().await?;
}

6) State Transition

The template-facing state transition is:

1. local build artifacts
2. committed snapshot.json + firecracker-manifest.json + fixed files + managed layers
3. node-local runtime-derived image configs
4. Firecracker resume inputs

7) What the Tests Validate

tests/integration/snapshot.rs covers:

• building and publishing from overlaybd configs
• loading committed snapshots by alias / id through the snapshot manager
• resolving committed snapshots into runnable snapshots
• launching Firecracker sandboxes from resolved snapshots
• rebuilding from committed snapshots while preserving base state
• listing and deleting committed snapshots through the template API surface

tests/integration/snapshot_attached_drive.rs covers overlaybd attached-drive handling across sandbox capture, publish, resolve, launch, and rebuild flows, including:

• committed metadata preserving readOnly
• runtime mount mode for readonly and writable drives
• rebuild keeping attached-drive metadata stable

crates/e2e-tests/tests/snapshot_oss_e2e_test.rs covers OSS-backed snapshot publication, resolution, alias cleanup, missing managed-layer failure modes, and deletion.

Repository/backend unit tests under src/snapshot/repository/backends/ cover:

• artifact import
• alias handling
• committed snapshot deletion
• runtime cache behavior

8) Common Failure Areas

• overlaybd base config paths are missing or invalid
• ublk is disabled while using overlaybd build bases
• Linux host prerequisites for Firecracker, the selected KVM/PVM mode, or network namespaces are missing
• repository alias conflicts during publish
• runtime resolver cannot materialize local paths from committed artifacts

Persistence Artifact Inventory

This document lists AgentENV artifacts that can remain on disk or in object storage beyond a single function call. It is organized by the module that owns each artifact’s lifecycle.

Path Roots

Setup And Config

Owned by src/setup/* and src/cfg.rs.

Firecracker Sandbox

Runtime

Owned by src/sandbox/firecracker/*.

Firecracker Pool

Owned by src/sandbox/firecracker/pool.rs.

Extra Drives

Owned by src/sandbox/extra_drive.rs and Firecracker snapshot code.

Snapshot Storage Model

Snapshot data passes through three storage layers with different ownership and lifetime rules:

1. Builder staging is a manager-owned temporary workspace under <snapshot.local_cache_path>/snapshots/<id>/. It holds local rootfs, memory, VM-state, and attached-drive artifacts while a build or capture is in progress. It is not the durable snapshot record.
2. The committed snapshot repository stores the snapshot catalog, aliases, snapshot.json, firecracker-manifest.json, vm_state.bin, and referenced managed layers. This is the durable source of truth exposed by the template and snapshot APIs.
3. The node-local runtime cache materializes runnable rootfs, memory, and drive image.json files under <snapshot.local_cache_path>/runtime/<id>/ before launch. These files are derived runtime inputs and can be rebuilt from committed state.

The committed snapshot.json records the captured runtime context, startup configuration, and rootfs, drive, and memory layer references. The Firecracker manifest records launch metadata such as virtual sizes and attached-drive configuration. Temporary upper files and generated image.json files are not committed snapshot truth.

During runtime resolution, AgentENV converts the committed layer references into node-local rootfs, memory, and attached-drive configs, hydrates the Firecracker manifest with node-local paths, and resolves the committed vm_state.bin into a runnable path.

Image Resolver

Owned by src/image/*.

Snapshot Repository

Owned by src/snapshot/repository/* and src/snapshot/types/*.

Repository records are the logical source of truth for committed snapshots. Build-time and runtime image.json files should not become committed truth unless they contain data that cannot be derived from committed metadata.

Snapshot Runtime Resolution

Owned by src/snapshot/artifact_cache.rs, src/snapshot/runtime_support.rs, and backend runtime resolvers.

LocalArtifactCache owns pinning, in-flight materialization deduplication, and LRU eviction for files it manages. It does not own the durable snapshot repository.

Orchestrator Persistence

Owned by src/orchestrator/persistence/*.

Paused sandbox artifacts are not committed snapshot repository artifacts. They are owned by the sandbox persister and should not be shared as snapshot truth.

P2P Artifact Transport

Owned by src/p2p/*.

Snapshot publication writes best-effort P2P catalog entries after repository commit:

• snapshot/v1/artifacts/{snapshot_id}/vm_state.bin
• snapshot/v1/artifacts/{snapshot_id}/firecracker-manifest.json
• overlaybd-layer/v1/sha256:<digest> for referenced rootfs, memory, and attached-drive overlaybd commit layers

Those P2P entries are optional copies, not committed snapshot truth. Clearing <p2p.store_dir> can reduce peer-to-peer availability and force peers back to OSS or origin-registry reads, but it must not make a committed snapshot invalid by itself.

Ublk Daemon Runtime

Owned by storage/ublk-daemon/* and src/sandbox/ublk/*.

Overlaybd Storage

Owned by storage/overlaybd/*.

Ownership Rules

• Snapshot repository artifacts are durable user-visible state. Do not delete them from node-local GC code.
• Paused sandbox artifacts are durable only for the paused sandbox that owns them. Do not treat them as committed snapshots.
• Runtime image.json files under snapshot local cache or ublk runtime dirs are derived artifacts. They should be rebuildable from committed metadata or source image configs.
• Node-local image-cache artifacts are disposable cache, but content-addressed commits may be expensive to regenerate. The metadata-backed GC only reclaims commits no longer rooted by on-disk source configs, held by image-cache leases, or referenced by the in-process running set. Committed snapshots are durable SnapshotRepository state and do not pin ImageCache commits. See [image.cache.gc] in the configuration reference for scheduling.
• P2P store contents are node-local and optional. Unpublish removes the local catalog entry and related blobs; clearing the store only affects peer-to-peer sharing and may require republishing or refetching artifacts.
• Logs and dependency downloads are operational artifacts. They are not part of sandbox or snapshot correctness, but may require separate retention policy outside image-cache GC.

AgentENV Proxy Design and Usage

This document describes the per-node reverse proxy that forwards requests into individual sandboxes. For the distributed routing layer (gateway to scheduler to node), see System Architecture.

Scope

Each AgentENV node runs a reverse proxy that accepts requests on its API server and forwards them to sandbox services over the sandbox interaction network. In a multi-node deployment, the gateway resolves sandbox ownership through the scheduler and forwards data-plane requests to the owning node’s proxy surface.

Current entrypoints:

• ANY /proxy
• ANY /proxy/{*proxy_path}
• Any otherwise unmatched path that carries sandbox routing headers
• Host-based sandbox proxy requests when [sandbox_proxy].domains is configured: {port}-{sandboxID}.{domain}

Implementation references:

• src/api/proxy.rs
• src/orchestrator/service.rs
• src/orchestrator/proxy.rs

Routing Contract

Each proxied request must identify:

• Sandbox ID
• Target port inside the sandbox service plane

Accepted headers:

• x-agentenv-sandbox-id
• x-agentenv-target-port
• E2B-compatible aliases:
  • e2b-sandbox-id
  • e2b-sandbox-port

Validation:

• Sandbox ID must be a valid UUID format.
• Target port must parse as u16 and be greater than 0.

Authorization is evaluated by the owning runtime after route parsing:

• Control-plane routes require the deployment X-API-Key and do not accept sandbox credentials.
• Node Prometheus /metrics and health /health are public to application auth and should be protected separately at the deployment boundary.
• Non-envd application routes require e2b-traffic-access-token only when the sandbox has private ingress (allowPublicTraffic: false).
• The envd port requires X-Access-Token only for secure sandboxes.
• X-API-Key is never a data-plane credential.

The distributed gateway deliberately does not make sandbox authorization decisions. It routes data-plane requests, including public ingress and insecure envd requests with no credential, to the owning runtime. The runtime has the sandbox metadata needed to apply the policy and performs the authoritative token validation.

Host-based routing derives both fields from Host. The configured domain must match exactly after lowercase normalization and optional trailing-dot removal. Sandbox IDs in host routes must be valid UUIDs and the target port must fit in u16.

Runtime Route Model

AgentENV keeps an in-memory runtime route table in orchestrator.

• Route key: SandboxId
• Route value: ProxyTarget (currently host interaction IP) plus route metadata (version, updated_at)

Design rule:

• Only Running sandboxes publish runtime routes.
• Non-running states rely on metadata fallback (not route-table state).

Lookup behavior (proxy_lookup_for):

1. If runtime route exists: Ready(target)
2. Else read metadata:
   • no metadata: NotFound
   • metadata state is Running: RouteMissing
   • metadata state is Paused: Paused { auto_resume }
   • other states: Unavailable(state)