OpenHands agent-server · design · rev 3 · 5 Oct 2026
Reclaiming storage for local and remote sandboxes
Every conversation leaves something behind: caches and dependencies inside its sandbox, and for remote providers the sandbox itself, which keeps billing while paused. The design gives each piece one owner. The agent-server cleans the files in its own box. Whoever created the box owns the box. The provider pauses and destroys. The agent-server's HTTP API is the seam between them.
What changed since rev 1
Rev 1 (2 Oct) proposed one reclaimer inside the SDK that also drove remote providers through backend adapters. Review discussion and a real remote integration changed that.
- Every sandbox runs an agent-server. From its point of view everything is local, so the path-based reclaimer in #5263 → #5433 can run inside any remote box. Today it has two storage adapters: Docker runtimes (caches, disk budget, retention) and local git worktrees (disk budget, removal on delete). A box whose conversations use neither needs a third adapter: a new adapter, not a new architecture.
- Rev 1 said the agent-server "can't manage its own remote sandbox from the inside". That holds for the box (pause, delete, billing), but not for the files in it. Those are two problems with two owners.
- DigitalOcean Managed Agents in Canvas (OpenHands#17838) showed how it works in practice. The provider pauses idle boxes on its own; Canvas created the box, so Canvas is the only place a delete can come from; and nothing deletes paused sessions today.
The architecture
Five principles
- Everything is local to the agent-server. It cleans its own disk with plain paths, whether the box is a laptop, a Docker container or a DigitalOcean microVM. That is why the path-based code in the PR stack is right, not a limitation; new conversation layouts get a new storage adapter.
- Whoever created the box owns its lifecycle. Pausing, deleting and paying for a sandbox happen outside it: Canvas for DigitalOcean sessions, sandbox-server and runtime-api for Cloud and evals, the outer agent-server for its inner Docker containers.
- The remote extension point is HTTP, not a Python protocol. Orchestrators already reach the agent-server over HTTP. A few endpoints serve every future provider; a
RuntimeBackendhierarchy in the SDK would serve none of them better. - Native first. When the provider has idle and expiry settings, set them at creation and let the provider enforce them. Only build a loop for what it can't do.
- Timers need an always-on owner. A policy like "delete boxes paused for 30 days" can run in the provider or on a server, never in a desktop app that may be closed, and never in a box that is paused.
Who owns what
Agent-server (SDK)
- Caches, dependencies, archived runtimes, worktrees
- Activity: per conversation (from state-file times) and for the whole box (
idle_time) - History export
Doesn't own: the box, provider credentials, other boxes.
Orchestrator
- Provider credentials, client, tunnels
- Policies set when a box is created
- Export, then delete; server-side retention when always-on
Doesn't own: files inside a box (it asks over HTTP).
Provider
- Pause on idle, resume, destroy
- Snapshots and billing
- TTL for paused boxes (to ask for)
Doesn't own: which conversations matter to the user.
Frontend
- Status, paused age, sizes
- Connect, pause, delete
- The "history will be lost" confirmation
Doesn't own: tokens, provider calls, anything on a timer.
One word, two levels: "archived"
The user sees the same thing at both levels: history readable, no more prompting. The difference is only what gets freed.
| Level | Who does it | What is freed | What stays |
|---|---|---|---|
| Conversation | Agent-server, after N inactive days (#5429) | The conversation's whole runtime directory: sandbox home and workspace (Docker runtimes only; local mode has no retention) | History, which is stored separately; the API returns 410 "read-only" |
| Box | Orchestrator, with user confirmation | The whole sandbox and its billed snapshot | History exported first via /download-trajectory/{id} |
#5429 currently calls the conversation level "retired"; rename it to "archived" to match OpenHands Cloud.
The agent-server API that makes remote work
| Endpoint | Status | What it gives remote orchestrators |
|---|---|---|
| Storage status | new | Disk usage, plus per conversation: state, last_active, size, and what the next pass would free. Seven of fifteen providers expose no last-activity time, so the agent-server becomes the source of truth instead of every orchestrator tracking activity itself. |
| Reclaim now | new | Runs one pass immediately, with a dry-run option, still in-process under the idle locks. Shrinks a box before a manual pause or delete, serves as the "maintenance script" (a curl), and backs real-scenario tests. |
| Retention per conversation | new | Set when a conversation is created: default, N days, or on_stop. PR reviews and CI-style automations drop their runtime early; Slack conversations keep it longer. |
GET /server_info → idle_time | exists | Seconds since the agent-server last did real work for the whole box. runtime-api's reaper already pauses pods on it. This is the box-level activity signal; per-conversation last_active in storage status is the conversation-level one. |
GET /download-trajectory/{id} | exists | A zip of the conversation directory, with LLM keys and encrypted tokens in its JSON files redacted. Taken before deleting a box. GET /archive does the same for the workspace (git patch or tarball). |
Rule for the new endpoints: storage status and reclaim must not call update_last_execution_time(), or maintenance would keep boxes alive past the idle reaper. Today the idle clock moves only on bash, file and git operations, conversation events, streaming and ACP activity. Endpoint names for the new rows are not decided yet. There is no import endpoint, so reopening an exported history read-only elsewhere is still a follow-up.
Worked example: DigitalOcean sessions in Canvas
- Create. Canvas, through its Node layer, creates an agent config from an OpenHands manifest, then sessions from that config. That call takes no idle timeout (only DigitalOcean's legacy create does), so the 15-minute default applies. The SDK reclaim settings belong in the template.
- Work. The agent-server inside cleans caches as conversations stop and sheds dependencies above the disk budget. The box stays small.
- Idle. After 15 minutes without LLM or tool calls, DigitalOcean pauses the session. Compute billing stops; the snapshot keeps billing for storage, which the in-box reclaim has kept small.
- Return. Canvas resumes the session before opening the tunnel. DigitalOcean says processes, memory and files are preserved and resume takes 200–300 ms, yet #17838 still allows 20 s for the agent-server to answer after a wake. Treat resume as fast, not instant.
- Done. The user clicks Delete in Canvas. Canvas downloads the history, confirms, and calls
DELETE /v2/agents/sessions/{id}.
Gaps in #17838 today: no delete (paused sessions pile up, keep their slot in the team's session limit of up to 100 depending on tier, and bill for storage until deleted), and desktop only, because the DigitalOcean logic sits in Electron's main process. The fix is to move that logic into a Node module shared by Electron and the web ingress (/api/mars/*), and ask DigitalOcean for a TTL on paused sessions.
Fits every backend
The same three layers map onto everything we run or plan to run. OpenHands' own runtime-api, behind evals and staging (and very likely Cloud), already works this way; the design describes it rather than inventing something new.
| Local Docker / worktrees | runtime-api (evals; likely Cloud) | DigitalOcean | Modal | |
|---|---|---|---|---|
| Orchestrator | The outer agent-server (it creates the inner containers) | sandbox-server for users; runtime-api's reaper for boxes | Canvas mars module | Canvas, automations or eval harness (none yet) |
| Create / delete | Docker container / retire (#5429) | /start / /stop (stop deletes) | create / DELETE | create() / terminate() |
| Suspend | Stop the container | /pause: scale to 0, volume kept; optional snapshot after 1 h | Pause in place, memory kept | none: snapshot the filesystem, terminate, restore as a new sandbox |
| Activity signal | Per-conversation last_active | Agent-server idle_time (pause after 20 min in the Helm chart) | DO's own: no LLM or tool calls for 15 min (sessions also report last_event_at) | None native; an open tunnel connection counts |
| Native retention | — | Paused runtimes deleted after 1 day (fuse workspaces: 30 days); 12 h session cap | None known | Snapshots expire after 30 days (memory: 7) |
| In-box reclaim gives | The whole job | Smaller volume, snapshot, archive | Smaller billed checkpoint | Smaller snapshot |
runtime-api values from its code and Helm chart (private repo, HEAD d938163); production values for Cloud are not in these repos. Its workspace archive and volume snapshots are opt-in. Modal from its docs and modal-client @2bc73c2.
What the harder backends add
- A stable workspace id in the orchestrator. A box's identity can change: on Modal, resume is a new sandbox with a new id and URL; runtime-api rotates the session key on resume; DigitalOcean keeps the id. The orchestrator maps each workspace to its current box (
id, url, key). The SDK is unaffected. - Suspend comes in kinds. In place with memory (DigitalOcean, E2B), disk only (runtime-api, local Docker), snapshot-and-recreate (Modal), or none (Cloudflare). An orchestrator adapter exposes
suspend()/resume()with two flags: memory kept, same identity. This is the one piece of rev 1's role interfaces that returns, in the orchestrator rather than the SDK. - Reclaim, then archive, then snapshot. Every backend that snapshots (runtime-api's snapshotter when enabled, Modal before terminate, DigitalOcean checkpoints) bills the result as storage, and caches and
node_modulesare often a large share of it. Calling the reclaim endpoint first is its strongest use.
Two activity rules
- Maintenance is not activity. The reclaimer and its endpoints never reset the agent-server's idle clock.
- Polling from outside can be activity at the provider. On Modal any tunnel connection keeps the box awake, and #17838's notes say every DigitalOcean probe and tunnel dial counts (its code also notes that a quiet open tunnel does not). Only an in-cluster orchestrator like runtime-api can poll freely, so prefer native timers.
Share the contract, not code: there are already three orchestrators in different services (outer agent-server, sandbox-server with runtime-api, Canvas mars). What they share is the agent-server API and the adapter shape above. Extract a provider interface from the mars module when a second Canvas provider (Modal) arrives.
Provider capabilities (for orchestrator adapters)
Providers' official docs and client code, plus runtime-api's source, checked 2–5 Oct 2026. ✓ yes · ~ partly · ✗ no. This is orchestrator knowledge, not SDK knowledge.
| Provider | List + last activity | Suspend, files kept | Delete | Native lifecycle |
|---|---|---|---|---|
| DigitalOcean | ~ list; last_event_at (meaning unverified) | ✓ incl. memory | ✓ | ~ idle pause, no TTL known |
| Daytona | ✓ lastActivityAt | ✓ | ✓ | ✓ stop, pause, archive, delete |
| E2B | ~ list, no activity | ✓ incl. memory | ✓ | ~ auto-pause, never deletes |
| Modal | ~ list by tags, no activity | ~ no pause; snapshot + restore as new sandbox | ✓ | ~ idle_timeout terminates; snapshots expire in 30 days |
| Cloudflare Sandbox | ✗ no list | ✗ sleep wipes files | ✓ | ~ sleep after idle |
| OpenHands runtime-api | ✓ /list; activity from agent-server idle_time | ✓ volume kept (optional snapshot) | ✓ /stop | ✓ idle pause (20 min), paused deleted after 1 day |
Kept from rev 1, and dropped
Kept
- A reconcile pass that re-reads real state every time (
Reclaimer.run_pass()). - Acting only while a conversation is held idle; select outside the lock, move (and mark) inside it.
- Plan, then apply, now exposed as the dry run of the reclaim endpoint.
- Native first: provider settings at creation, our code only for the gaps.
- Suspend capability flags, moved from the SDK into orchestrator adapters.
Dropped
- Remote backends and provider adapters in the SDK.
- Role interfaces (
Suspendable,Deletable, …) and a path-free abstraction. - An activity tracker in every orchestrator: the agent-server reports
last_active. - Retention that deletes remote boxes automatically from a desktop app.
Pitfalls
- Retention in paused boxes (Docker runtimes, the only mode with retention). A paused box runs no passes. A pass a few minutes after resume can archive every conversation that crossed N days during the pause, right as the user returns. For provider-managed boxes, keep retention off and use cache cleanup and the disk budget only.
- Polling keeps boxes awake. On DigitalOcean, probes through the tunnel are treated as activity in #17838's notes. Orchestrators read storage status only while already connected, never on a timer.
- Maintenance that resets the idle clock keeps boxes alive and billing past runtime-api's reaper. The new endpoints must not touch it.
- Box identity changes on resume (Modal, runtime-api session key). An orchestrator that stores a box id instead of a workspace id loses track of it.
- Deleting a box deletes its history unless it was exported first. Always confirm with the user.
- A shared server holding provider tokens lets anyone who reaches it spend the account. Enable provider routes only on localhost or behind a login gate.
Next steps
- SDK stack: land #5263 → #5433, rename "retired" to "archived", document retention for provider-managed boxes.
- SDK follow-ups: storage status and reclaim-now endpoints, then per-conversation retention.
- Canvas (#17838): an Electron-free DigitalOcean module, a delete-session action with confirmation, paused age in the UI; web support through the ingress next.
- runtime-api: call reclaim before archiving and snapshotting (both opt-in there).
- Remote boxes: check whether conversations in runtime-api and DigitalOcean boxes use worktrees, inner Docker or a shared workspace; add a storage adapter if neither.
- Modal (when it becomes a Canvas backend): extract the provider interface from
mars; suspend = snapshot and terminate, resume = restore and remap; connect tokens for auth; snapshot expiry as retention. - DigitalOcean: ask for a TTL on paused sessions, and what counts as activity.
- Later: importing an exported history for read-only viewing; a provider-neutral orchestrator interface once a second provider arrives.