Building a Lightweight Code-Execution Sandbox

AI agents need somewhere to run code. I built an in-house sandbox for that on AWS EKS and took it to 1,000 concurrent sandboxes with 0 failures. This post covers the design axes that force the interesting choices, how the system fits together, and the numbers.

Why I built it#

I wanted to understand sandboxes by building one, not by reading product docs. The workloads I care about are short and mostly I/O-bound: an agent writing and running a scrap of Python, scoring a candidate answer, or checking a reward in an RLVR / RLHF loop. Those need isolation, a hard ceiling on time and resources, and a way to start a lot of them at once.

Hosted sandboxes (Modal, E2B, Daytona)¹¹Modal Sandboxes, E2B, Daytona. already do this: you get isolation, fast starts, and an SDK, and you pay per second with your code and data running on someone else's infrastructure. Building it yourself buys control over images, networking, credentials, and where the data lives. The cost is that you own the scheduler, the cluster, and every failure in between.

It was also practice for something I wanted to build at work. Shipping a toy version end-to-end forced the design questions I would otherwise skip past until production. I cut scope aggressively: no shared disks, no stronger-than-container isolation. The point was a working lightweight path.

What is a sandbox#

A sandbox is a short-lived, isolated place to run code you do not fully trust — usually code a model just wrote. You start it, give it a command or a file, get stdout / stderr / an exit code back, and throw it away. The host, other tenants, and any network you have not explicitly allowed should not be reachable from inside it.

You need one because the alternative is running model output on your laptop or in the training job itself. That fails in two ways: (i) a bad or adversarial program can hang, fill the disk, or touch secrets, and (ii) you cannot score thousands of candidates in parallel if each one shares the trainer's process. Isolation plus a hard resource and time ceiling turns "run this" into something you can call in a loop.

In training, sandboxes sit behind the reward. For coding RL or RLVR,²²Ai2, Tülu 3: The next era in open post-training — introduces RLVR: post-training against a deterministic verifier instead of a learned reward model. For a longer walkthrough, see What is RLVR?; for GRPO, the update rule most RLVR setups use, see Cameron R. Wolfe, Group Relative Policy Optimization (GRPO). each rollout's program gets executed against hidden tests or a verifier; the sandbox is what makes that score finite and safe when you are evaluating hundreds of candidates per step. I walked through building that loop — reset / step, a toy sandbox, and a verifiable reward — in Building an RL Environment for an LLM Coding Agent. The same need shows up in RLHF-style³³Hugging Face, Illustrating Reinforcement Learning from Human Feedback (RLHF). pipelines only where part of the reward comes from running the output, such as unit tests or execution checks mixed in alongside a learned reward model.

In inference, sandboxes are the tool the agent calls when it needs to execute. A coding agent writes a script, runs it, reads the error, edits, runs again. Search agents and data agents do the same with scrapers and notebooks. The difference from training is mostly lifetime and fan-out: inference wants a sandbox that stays up across a few turns for one user; training wants many cheap ones that die after a single score.

What a sandbox forces you to decide#

Before any code, eight decisions decide what you are actually building. Each one looks small. Each one changes the rest of the system.

Lifecycle. Is a sandbox a one-shot job that runs a command and exits, or a live environment you can drive for a while — start it, run something, run something else, tear it down? One-shot is simpler. Agents and verifiers usually want the live version: they need to iterate inside the same environment. That choice forces a control path into a running environment, not just a job submit API.

Compute substrate. Once you want live remote control at hundreds of concurrent sandboxes, you either build a scheduler yourself or borrow one. Rolling your own (EC2 + containers, a Firecracker fleet, ECS tasks with a custom control path) means you own placement, readiness, garbage collection, and fan-out networking. Kubernetes already has those. I put each sandbox in a pod and drive it through the Kubernetes exec API⁴⁴Kubernetes, Get a Shell to a Running Container — the exec API behind kubectl exec. — no per-pod server, no inbound ports to a thousand pods. The cost is real: the API server is a shared chokepoint, and you inherit Kubernetes operational weight. For this workload the borrowed primitives were worth it.

Isolation boundary. Hardened containers share the host kernel. gVisor⁵⁵gVisor documentation. puts a user-space kernel in between; microVMs (Kata, Firecracker)⁶⁶Kata Containers, Firecracker. give each sandbox its own kernel. Containers are denser and faster to start; the stronger boundaries are the answer when you assume a kernel escape is in scope. I stayed on hardened containers. That capped how strong the security story could be, and it is why density numbers later in this post are even possible.

Startup. Cold start means schedule a pod onto a node, pull an image, boot. Warm pools keep spare environments already up. Prebaked images skip installs at run time. If your workload is "score 500 candidates," startup dominates wall clock; if it is "one agent for five minutes," it mostly does not. The startup model is what decides whether warm pools and image baking are load-bearing or optional.

Density and sizing. How much CPU and memory does each sandbox reserve? Agent scripts are mostly waiting on I/O. Overcommit (small requests, higher ceilings)⁷⁷Kubernetes, Resource Management for Pods and Containers — requests drive scheduling; limits cap usage. packs more sandboxes onto the same quota. Pin every sandbox to a full core and you hit the account limit long before you hit 1,000. Density is the lever that makes the concurrency target reachable under a fixed vCPU budget.

Storage isolation. If sandboxes write outputs, each one needs credentials that cannot read its neighbor's files. One AWS IAM role (a named set of permissions)⁸⁸AWS, IAM roles and IAM and AWS STS quotas. per sandbox does not scale — role creation is slow and capped. One base role plus a per-sandbox session policy⁹⁹AWS, Session policies. does. That single choice sets the credential broker design.

Multi-tenancy. One operator talking straight to the cluster is fine for a demo. Many independent users each paying their own auth and connection setup is not — the per-user overhead dominates. A control-plane service in front of a pooled backend amortizes that cost and is where auth, scoping, and quotas actually live.

Backpressure. Creating a sandbox asynchronously is cheap: you get a handle before anything is ready. Without an admission check, a client can commit 1,500 sandboxes against capacity for 1,000 and leave hundreds stranded pending forever. Async create only works if something rejects or queues over-budget requests before the substrate is asked to schedule them.

Those eight are the spine. Here is the system they produce.

Design#

High-level components#

I built this on Kubernetes on EKS¹⁰¹⁰AWS, What is Amazon EKS? (Amazon's managed Kubernetes), not a custom sandbox runtime. The alternative is managing VMs or containers yourself — placement, readiness, cleanup, and how you reach a running sandbox to exec into it. Kubernetes already does the first three, and its exec API covers the fourth without putting a server inside every sandbox or opening inbound ports to a thousand of them. That is the trade: you accept API-server load and cluster ops in exchange for not writing a mini-orchestrator.

The whole thing sits on one EKS cluster, with AWS around it for container images (ECR¹¹¹¹AWS, What is Amazon Elastic Container Registry? ), credentials (IAM / STS), and object storage (S3¹²¹²AWS, What is Amazon S3? ). Users call a control-plane service (itself pods in the cluster). That service admits work, picks an image, mints credentials, and talks to the Kubernetes API. Each sandbox is an ordinary pod. Kubernetes is the source of truth for what is alive; the control plane keeps no durable sandbox state of its own.

Left to right: the client, the in-cluster control plane, the Kubernetes API and sandbox pods it manages, and AWS.

Storyline#

One sandbox, end to end:

1. Sign in. You prove who you are. From then on you only see your own sandboxes.
2. Start one. The system checks that the cluster still has room. If not, it says no immediately. If yes, it picks an environment image and either takes a pod that is already warm or starts a fresh one with storage credentials that only this sandbox can use. You get an id back right away; the pod may still be booting.
3. Wait until ready. The system blocks until that pod can accept work. Pre-built images and warm pods keep this short.
4. Run code. You send a command. Output streams back as it prints. Anything written to storage is invisible to other sandboxes.
5. Tear it down. You stop it yourself, or it dies on a time limit / idle timeout. Capacity goes back into the pool either way.

From the CLI, steps 2–5 look like this:

ID=$(sbx create --image base --size small)   # returns at once with an id
sbx wait "$ID"                                # blocks until the pod is ready
sbx run "$ID" python -c 'print("hello")'      # streams stdout / stderr back
sbx kill "$ID"

Those five steps are the only operations the rest of this section is about.

Components#

Manager#

Starting a sandbox is not one step. Someone has to: check the request is valid, reserve capacity, pick an image, then either claim a warm pod or mint credentials and start a fresh one — and undo the reservation if start fails halfway. Retries with the same name must not create duplicates.

That orchestration lives in the Manager. It is also the thing callers use to wait until ready, run a command, fetch logs, and stop a sandbox. It does not keep its own database of sandboxes. Every status check reads live state from Kubernetes. If the control-plane process restarts, sandbox status comes straight back from the cluster instead of from a private store that can drift. The one exception is the admission budget, which lives in memory and starts empty after a restart.

Admission#

Starting a sandbox returns an id before the pod is fully up. That is what you want when launching hundreds at once — and it is how you break the cluster. A client can ask for 1,500 sandboxes when there is only room for 1,000. Without a gate, the extras sit "accepted" but never become ready, still eating node pressure.

Admission is the gate. Before any pod is started, it checks CPU and memory against a capacity budget and either reserves a slice or rejects the request. A fast in-memory counter does the check; a Kubernetes quota on the namespace¹³¹³Kubernetes, Resource Quotas. is the hard ceiling if that counter is wrong — for example after a restart, when the counter starts from zero while sandboxes are still running. When a sandbox finally dies, the reservation is released. That release is easy to forget — which is why admission and the reaper have to work together.

Image catalog#

Early on, "start a sandbox" secretly meant "start a blank Python image and install packages at run time." Under concurrency that is slow and not repeatable: every sandbox resolves its own dependencies, and your startup time is mostly package install.

The catalog stops that. You ask for a named profile — base, data, web, ml, rl — and get a pre-built image from ECR (AWS's private container registry) with those libraries already installed. Which profile points at which image URI is configuration, so you can ship a new image without redeploying the control plane. Installing packages at run time stays possible for one-offs; it is not the path under load.

Credential broker#

Sandboxes that write files need cloud credentials. One IAM role (a named permission set in AWS) per sandbox sounds clean and does not work at scale: creating roles is slow, capped, and leaves junk when sandboxes die faster than you delete roles.

Instead there is one shared base role. When a sandbox starts, the broker calls STS (AWS Security Token Service)¹⁴¹⁴AWS, AssumeRole (STS API reference). to mint short-lived credentials narrowed to that sandbox's S3 folder only — a session policy on top of the base role, not a new role. Sandbox A can write its own folder and gets denied on B's — enforced by AWS, not by us checking a map in process.

Backend#

Everything above is policy. Something still has to start a pod in the cluster, notice when it is ready, run a command inside it, stream its output, and delete it. That is the Backend — the only piece that talks to Kubernetes directly.

It builds each sandbox as a locked-down pod (non-root, limited permissions,¹⁵¹⁵Kubernetes, Configure a Security Context for a Pod or Container. no talking to neighboring sandboxes¹⁶¹⁶Kubernetes, Network Policies. ). "Ready" is a watch on cluster events, not a poll loop per sandbox — otherwise a thousand sandboxes means a thousand status streams against the API server. Running a command goes through Kubernetes too, so the pods do not need a public port.

The Backend is swappable. Kubernetes is what I built. A different compute backend could plug in later without rewriting the Manager.

Warm pool#

A cold start pays for scheduling, pulling an image, and booting the container. Under a burst, that dominates how long users wait — and the slowest sandboxes are much slower than the median, because each node can only start so many containers per second.

The warm pool keeps a few pods already running for common shapes. Starting a sandbox tries to take one of those first; if it gets one, most of the boot is already done. The pool refills in the background. A taken pod is used once and destroyed — never handed to a second user with leftover files or processes still on disk.

Warm pods are generic. They boot before anyone asks for them, so they do not carry per-sandbox storage credentials. In this version a warm claim trades storage isolation for speed; sandboxes that need scoped S3 access take the cold path.

The naive pool is easy to get wrong. If every new sandbox grabs for the same head-of-line warm pod, they collide and the "fast path" becomes slower than a cold start. Claimers have to spread out, or the pool fights itself.

Reaper#

Users close their laptops. Load tests crash. If cleanup only happens while the caller is still connected, pods pile up — and the capacity budget slowly lies about how much room is left, because reservations were never released.

The reaper runs on its own schedule. It stops sandboxes that hit a hard time limit, that sat idle too long, or that finished and are past a grace period. On the way out it always returns capacity to admission. The time limit is also set on the pod itself, so even a broken reaper cannot leave a sandbox running forever.

Control-plane service#

One person talking straight to the cluster is fine. Many independent users is not. Each user would pay their own login cost into Kubernetes (often hundreds of milliseconds before any real work), and there is nowhere central to say "you only see your sandboxes" or to share one capacity budget.

The control-plane service is the front door. You send a token you already have. The service decides who you are, only shows you your sandboxes, and refuses to acknowledge anyone else's ids. Inside, it reuses one connection to the cluster and one Manager, so the expensive cluster login happens once. It runs as pods in the same EKS cluster. Your CLI or SDK talks to it over HTTP; command output streams back over that connection.

CLI#

You still need something on your laptop that can sign in, start a sandbox, wait, run a command, and stop it — without writing a new script for every experiment. That is the CLI.

For normal use it talks to the control-plane service. When you are debugging the cluster itself, it can talk to Kubernetes directly. Same verbs either way; you flip a config flag, not a second tool. There are also helpers for filling the warm pool and for running load tests.

Implementation#

I ran the experiments on a deliberately limited AWS account — small instance types, tight quotas — so every decision below was made under resource pressure, not on an unconstrained lab cluster. The same design (admission budget, warm pool, pooled control plane, overcommit) is what you would keep when the account is larger. Most of the ceilings I hit were account ceilings (instance size, vCPU quota), not design ceilings. The exception is the last one: a single control-plane replica, which is a deployment choice.

I did not build this top-down. Each experiment surfaced a constraint that forced the next change. What follows is that order. The Go interfaces are the seams I drew early so a rewrite stayed local when something was wrong.

Interfaces first#

Before a cluster existed, I wrote six boundaries: Manager (what callers use), Backend (compute), AdmissionController (capacity), CredentialBroker, ImageCatalog, and a Store for audit. The CLI and the later HTTP service both program against Manager. Kubernetes is one Backend. That kept every later swap — warm claim, STS creds, prebaked images — behind a typed seam.

// Manager is what the CLI / HTTP client call. Create is async: it returns a
// handle after admit → resolve image → Claim (warm), else mint creds →
// Provision (cold).
type Manager interface {
    Create(ctx context.Context, req CreateRequest) (*Sandbox, error)
    Wait(ctx context.Context, id string, o WaitOptions) (*Sandbox, error)
    Exec(ctx context.Context, id string, req ExecRequest) (ExecSession, error)
    Logs(ctx context.Context, id string, o LogOptions) (io.ReadCloser, error)
    Kill(ctx context.Context, id string) error
    Get(ctx context.Context, id string) (*Sandbox, error)
    List(ctx context.Context, f ListFilter) ([]*Sandbox, error)
}
 
// Backend is the compute substrate. Claim is the warm fast path; Provision is
// cold. AwaitReady watches cluster events — never a poll loop per sandbox.
type Backend interface {
    Claim(ctx context.Context, spec SandboxSpec) (h PodHandle, ok bool, err error)
    Provision(ctx context.Context, spec SandboxSpec) (PodHandle, error)
    AwaitReady(ctx context.Context, id string, o WaitOptions) error
    Exec(ctx context.Context, id string, req ExecRequest) (ExecSession, error)
    Delete(ctx context.Context, id string) error
    Describe(ctx context.Context, id string) (*Sandbox, error)
    // Touch, List, Logs, ...
}

Admission went in the core on purpose. Async Create is cheap; without a budget check a client can commit 1,500 sandboxes against room for 1,000 and leave 500 Pending forever.

type AdmissionController interface {
    Admit(ctx context.Context, spec SandboxSpec) (Decision, *Reservation, error)
    Release(ctx context.Context, id string) error
    Budget(ctx context.Context) (Budget, error)
}

Admit runs before any pod is created. Release has to run on every terminal path — kill, TTL, idle, crash — or the budget lies.

Lifecycle on EKS#

A one-shot batch Job is the wrong shape for agents. You need create → run → run again → kill against the same sandbox. I modeled each sandbox as a long-lived pod sitting on sleep infinity, with work started through the Kubernetes exec API. No server inside the pod, no inbound port to a thousand pods. Creating the pod object is instant; readiness is a watch.

Cluster bring-up hit the first hard constraint of that limited account. An SCP¹⁷¹⁷AWS Organizations, Service control policies (SCPs). (LimitDirectEC2InstanceTypes) denies anything larger than *.xlarge (4 vCPU), so the textbook "few big nodes" plan was rejected. I switched to many m7i.xlarge¹⁸¹⁸AWS, Amazon EC2 M7i instances. nodes. That later helped: more nodes means more kubelets starting containers in parallel — a property that still holds if you raise the account cap and add larger machines. Prefix delegation on the VPC CNI¹⁹¹⁹AWS, Assign more IP addresses to Amazon EKS nodes with prefixes. raised the pod cap per node from 58²⁰²⁰Amazon VPC CNI, eni-max-pods.txt — default max pods per instance type. to 110. At nano density (~30 per node) the default cap would not have bound, but with smaller requests or larger nodes, IP addresses run out before CPU does.

First live path: create → wait → run → kill. Ready in ~3.8s. Exit codes propagated. Hardening held — uid 1000, read-only rootfs blocked writes.

Storage isolation#

If sandbox A writes outputs, it must not read B's. One IAM role per sandbox does not scale — role creation is slow and capped. The broker assumes one base role and narrows it at AssumeRole time with a session policy scoped to s3://…/sandboxes/<id>/*.

// One base IAM role + N scoped sessions. Never one role per sandbox.
type CredentialBroker interface {
    MintScoped(ctx context.Context, id string) (ScopedCreds, error)
}

From inside A, boto3 wrote A's prefix and got AccessDenied on B's — from AWS, not from a check in our process.

Warm pool — and a self-inflicted bug#

Cold start was fine for demos and wrong for bursts. I added a warm pool of pre-booted pods. First measurement: warm was slower than cold — p50 6.8s against a cold p50 of 1.84s on 200 small sandboxes. Forty claimers all scanned the same list and fought over the head via optimistic concurrency.²¹²¹Kubernetes, Resource versions — an update carrying a stale resourceVersion is rejected with a conflict.

The warm path is Backend.Claim; cold is Provision. A claim that already returns READY should skip AwaitReady — that was part of the fix, along with shuffling each claimer's candidate order and capping retries. Warm p50 went 6.8s → 0.87s. The lesson was not "add a pool." It was measure the fast path; the first version was not fast.

Pushing toward 1,000#

An OOM test (800 MB in a 512 Mi sandbox) exited 137, but the platform still reported CREATING. Status only checked pod phase; a container can be dead while the phase still reads Running. Fix: treat terminated-nonzero containers as FAILED / OOMKilled.

Agent work is mostly I/O-bound, so I added a denser size class — nano, 100m CPU request, higher memory ceiling. On two nodes that held ~28 small (~14 per node), I packed 60 nano (~30 per node).

Scaling the node group by hand (no Karpenter²²²²Karpenter. ) and launching with the bench harness reached 1,000 concurrent sandboxes, 0 failures, create→ready p50 2.89s / p99 5.91s. p50 moved from 2.30s at 600 to 2.89s at 1,000, with no failure cliff.

With the nodegroup scaled back down to 2 nodes, a churn test (40 clients, each running create → ready → run → kill three times) showed the real tail: create→ready p99 47.6s. Identical test on 10 nodes: p99 4.9s. Create-accept (~60ms) and exec (~1s) barely moved. The tail is kubelet parallelism per node, not the API path. Warm pools skip that path; more nodes spread it.

Testing with real images#

Until here, profiles pointed at a public uv image and did runtime uv --with installs. That contradicted the design's own "prebake, don't install." I rebuilt each profile from locked deps (uv sync --frozen²³²³uv, Locking and syncing. ) into a baked ECR image: base, data, web, ml, later rl. Profile → URI is config, so a new image ships without redeploying the control plane.

type ImageCatalog interface {
    Resolve(profile string) (ImageRef, error)
    Profiles() []ImageProfile
}

Cold pull of the 222 MB ml image measured 4.75s on a node that had never seen it, then 0 for later pods on that node. Pull is per node, not per sandbox — the concrete case for baking into the node AMI or a pre-pull DaemonSet.²⁴²⁴Kubernetes, DaemonSet.

Load-test honesty#

The bench reused one Kubernetes clientset, so create-accept looked like ~60ms. That is the pooled number. Modeling unrelated users — fresh client per request, unique exec env so client-go cannot share the credential cache — a single independent user pays ~440ms (TLS + aws eks get-token on their machine). Piling a hundred "independent" clients on one laptop saturates the generator and invents a ~6s number that is not the platform.

Neither single-machine extreme is the real number. Faithful high concurrency needs distributed generators, and the auth tax argues for a control-plane service that amortizes login once.

Reaper#

While chasing that, the capacity budget had a leak: reservations released on explicit Kill, not when a sandbox died any other way. Clients that vanished left the budget lying about free room.

The reaper reconciles against Kubernetes on its own schedule. It deletes finished pods after a grace, enforces idle timeout via Touch, and always calls Admission.Release on terminal state. Hard TTL stays on the pod as activeDeadlineSeconds,²⁵²⁵Kubernetes, Pod API reference — activeDeadlineSeconds. so even a broken reaper cannot leave a sandbox running forever. Lifecycle no longer depends on the caller still being connected.

Control plane#

Independent users → one HTTP service → one pooled Manager into the cluster. Bearer token maps to owner; other people's sandbox ids return 404. The CLI gained a service mode whose HTTP client still implements Manager, so the same commands work either way — SBX_SERVER flips the transport, not the verbs.

Distributed in-cluster load (50 independent client pods) saw server-side create-accept p50 ≤50ms, 0 errors. The 440ms–6s per-user tax from the laptop tests disappeared.

A realistic workload#

Last scale test was not a sleeper. Each sandbox ran an RL-style verifier chain — search → pages → scrape → nltk claim extract → token-overlap score — on a prebaked rl image, through the control plane, with tools mocked in-cluster and a NetworkPolicy that lets sandboxes reach the tool gateway but not each other. Peak 496 concurrent, 999/1000 evaluations ok, ~1,000 lifecycles in ~60s.

The EKS substrate held all 496. The single 2-core control-plane replica did not: streaming wait/exec through one process saturated it and create-accept p50 rose to ≤2.5s (vs ≤50ms at 50 clients). The service holds no durable state, so the fix is more replicas. Two pieces of in-process state come along, though: each replica would run its own admission counter and its own reaper. The namespace ResourceQuota still stops over-commit across replicas, but the in-memory budget has to be split between them or moved to a shared store. That was the ceiling the load test was for.

Through-line#

Every step above was driven by the previous result:

Benchmarks and performance#

All numbers below are from the limited AWS account described earlier — many m7i.xlarge (4 vCPU) nodes, no Karpenter, manual nodegroup scaling. They are create→ready unless noted. The early scale runs (cold / warm / 1,000) used a public base image already cached on the nodes; prebaked ECR profiles landed after that and are what the image-pull and verifier numbers use.

Cold vs warm startup#

1,000 concurrent sandboxes launched in ~40s at ~25/s on 30 nodes (120 vCPU). On the nano ramp (200 → 600 → 1,000) there was no failure cliff — p50 moved from 2.30s at 600 to 2.89s at 1,000. Warm is ~2× faster than cold small at p50 because it skips pod boot; the first warm run before the claim fix was worse than cold (p50 6.8s), so the table only shows the fixed path.

Density#

nano is the I/O-bound agent default: small request (100m), 128Mi reserved, 512Mi ceiling, bursts freely. On the 250-vCPU namespace quota (the usable slice of the 300-vCPU account limit) that is about 2,500 nano of CPU request headroom (system pods take some) vs about 1,000 small.

Image pull#

First pod on a node that has never seen the 222 MB ml image:

Pull is per node, not per sandbox. Later pods on that node start warm. That is the case for baking images into the node AMI or a pre-pull DaemonSet.

Create-accept: what the number actually is#

Create-accept (return a handle, do not wait for ready) decomposes into three pieces that are easy to conflate:

A single independent user sees ~440ms. Piling 100 "independent" clients on one Mac invents a ~6s number that is generator meltdown, not the platform (0 errors, 0 throttling). The ~60ms figure is server-side create with a warm pooled client and no contention; under 100-way shared-client load on one machine, p50 rose to 539ms — still not the independent-user number. Neither single-machine extreme is what a real deployment sees.

Control-plane service (distributed)#

50 independent in-cluster client pods, each with its own connection and bearer token, driven through the control-plane service. Latency from the service's own /metrics:

The service's latency histogram²⁶²⁶Prometheus, Histograms and summaries. uses fixed buckets (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 30s), so these percentiles are bucket upper bounds: "50ms" means at or under 50ms.

0 errors. Create-accept stays ≤50ms under 50 independent clients — the 440ms–6s laptop tax disappears. The remaining tail is wait (kubelet / node readiness), not the request path.

Where create→ready tails come from#

Identical churn test (create→ready→run→kill), only node count changed:

Create-accept (~60ms) and exec (~1s) were unchanged. The tail is kubelet parallelism per node. Warm pools skip that path; more nodes spread it.

RL verifier workload#

Realistic I/O-bound chain through the control plane: search → scrape → nltk claims → token-overlap reward, prebaked rl image, tools mocked in-cluster. 16 × m7i.xlarge (64 vCPU).

Under that load, one 2-core control-plane replica was the ceiling:

The EKS substrate ran all 496. Funneling ~500 streaming watches/execs through one process saturated it. The fix is more replicas, with the admission budget split across them or moved to a shared store, and ResourceQuota as the hard ceiling.

What I'd do next#

The load test already named the first fix: run more than one control-plane replica. The service holds no durable state; the work is centralizing the in-memory admission budget, with ResourceQuota as the backstop. That unblocks the verifier-shaped workload past a few hundred concurrent streaming sessions.

After that, in order of leverage:

1. Karpenter (or equivalent) so the nodegroup grows and consolidates instead of eksctl scale²⁷²⁷eksctl. by hand. The 1,000-sandbox run needed 30 nodes; at idle the cluster sat on two.
2. Shared-informer warm claims²⁸²⁸client-go, tools/cache — shared informers keep a local, watch-updated cache of cluster objects. so claimers read pool state from memory and patch, instead of listing the pool on every attempt. That kills the residual warm p99 and makes a large pool cheap.
3. Bake images into the node AMI (or a pre-pull DaemonSet) so the first ~5s ECR pull per node is not on the hot path.
4. File transfer into sandboxes — today you only get exec plus whatever is baked in the image. Agents need put / get (tar-over-exec or a volume).
5. Stronger isolation if the threat model requires it. Hardened containers share the host kernel; gVisor / Kata / Firecracker is the next boundary, at the cost of density.

Smaller production gaps:

• Credentials on warm pods. Warm pods carry no per-sandbox credentials, so a warm claim currently gives up storage isolation. Inject scoped credentials at claim time instead.
• Rebuild the admission budget on startup by listing live sandboxes, so a restarted control plane does not start from an empty counter and lean on ResourceQuota alone.
• Credential delivery. Move sandbox credentials off env injection (visible in the pod spec) onto Pod Identity.²⁹²⁹AWS, EKS Pod Identity.
• Auth. Replace the toy bearer token with real OIDC/JWT.

Building it under a capped account was the useful part. The interesting failures — warm slower than cold, status lying about OOM, create-accept numbers that flattered the platform — only showed up because the budget was tight enough to force measurement.