Agent Sandbox

Performance Assessment

This guide covers the tools and techniques available for assessing the performance of Agent Sandbox — from tuning controller concurrency, to running load tests, to collecting and interpreting benchmark results.

Controller Performance Tuning

The agent-sandbox-controller exposes several flags that directly affect throughput and API server pressure. Raising these is the first step before running any load test. The table below is kept in sync with docs/configuration.md, which is the canonical flag reference — check there first if the two ever disagree.

Choosing worker counts

Each --*-concurrent-workers flag controls the number of independent goroutines the corresponding controller runs. For example, setting --sandbox-claim-concurrent-workers=10 allows the SandboxClaim controller to process up to 10 SandboxClaims in parallel.

A good starting point is to size each value to the expected steady-state object count for that type, or to a calculated fraction of your maximum burst load. A cluster with a single WarmPool only needs --sandbox-warm-pool-concurrent-workers=1.

Important caveat: increasing worker counts only improves throughput when the controller itself is the bottleneck. If the bottleneck is the kube-apiserver or the container runtime (e.g., slow pod startup), adding more workers can actually increase latency by creating additional API server contention. Use the metrics in the Metrics Collected section and APF dashboards to confirm where the bottleneck lies before tuning these values.

Applying Flags

Via manifest — edit the relevant manifest for your deployment. The snippets below show the in-repo source under k8s/; the corresponding release assets (sandbox.yaml, extensions.yaml) carry the same args, but with the ko:// placeholder replaced by a published registry.k8s.io/agent-sandbox/agent-sandbox-controller:<version> image.

Core install (k8s/controller.yaml):

containers:
- name: agent-sandbox-controller
  image: ko://sigs.k8s.io/agent-sandbox/cmd/agent-sandbox-controller
  args:
  - --leader-elect=true
  - --sandbox-concurrent-workers=10
  - --sandbox-claim-concurrent-workers=10
  - --sandbox-warm-pool-concurrent-workers=10
  - --kube-api-qps=50
  - --kube-api-burst=100

Extensions install (k8s/extensions.controller.yaml) — note the additional --extensions flag, which enables the SandboxTemplate controller and its associated RBAC. (k8s/extensions.yaml is RBAC only — the Deployment lives in k8s/extensions.controller.yaml, a second copy of the core Deployment applied after sandbox.yaml so the sequential sandbox.yaml → extensions.yaml install path lands a Deployment with --extensions set. Kustomize consumers get the same result via the patch in k8s/kustomization.yaml, which excludes this file.):

containers:
- name: agent-sandbox-controller
  image: ko://sigs.k8s.io/agent-sandbox/cmd/agent-sandbox-controller
  args:
  - --leader-elect=true
  - --extensions
  - --sandbox-concurrent-workers=10
  - --sandbox-claim-concurrent-workers=10
  - --sandbox-warm-pool-concurrent-workers=10
  - --sandbox-template-concurrent-workers=10
  - --kube-api-qps=50
  - --kube-api-burst=100

Via kubectl patch on a live cluster:

kubectl patch deployment agent-sandbox-controller \
  -n agent-sandbox-system \
  --type='json' \
  -p='[
    {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--sandbox-concurrent-workers=10"},
    {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--sandbox-claim-concurrent-workers=10"},
    {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--sandbox-warm-pool-concurrent-workers=10"},
    {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--sandbox-template-concurrent-workers=10"},
    {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kube-api-qps=50"},
    {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kube-api-burst=100"}
  ]'

Via Kustomize — use a JSON 6902 patch to append flags without replacing the existing args list. A strategic-merge patch on containers[].args replaces the entire list, which silently drops required flags such as --leader-elect and --extensions.

Note: If you consume k8s/kustomization.yaml (or its rendered sandbox-with-extensions.yaml release asset) rather than applying sandbox.yaml/extensions.yaml sequentially, start from that file — it already assembles core + extensions into a single Deployment. Add the JSON 6902 patch below as an additional entry under its patches: list.

Core install:

# patch-args.yaml
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-claim-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-warm-pool-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--kube-api-qps=50"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--kube-api-burst=100"

Extensions install — also tune --sandbox-template-concurrent-workers:

# patch-args.yaml
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-claim-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-warm-pool-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--sandbox-template-concurrent-workers=10"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--kube-api-qps=50"
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: "--kube-api-burst=100"

Reference the patch from kustomization.yaml using patches with an explicit target:

# kustomization.yaml
patches:
  - path: patch-args.yaml
    target:
      kind: Deployment
      name: agent-sandbox-controller
      namespace: agent-sandbox-system

E2E Benchmarks

The repository ships Go benchmarks in test/e2e/ that measure Sandbox and SandboxClaim startup latency against a live cluster.

Available benchmarks

Running benchmarks

Run all e2e benchmarks:

make test-e2e-benchmarks

Or target a specific benchmark directly with go test:

go test -bench=BenchmarkChromeSandboxClaimStartup -benchtime=10x ./test/e2e/...

Load Testing with ClusterLoader2

For scale testing, Agent Sandbox uses ClusterLoader2 (CL2), the same framework used for Kubernetes scalability testing.

Prerequisites

• A running Kubernetes cluster with the Agent Sandbox controller and CRDs installed.
• Go toolchain.
• The perf-tests repository cloned as a sibling to agent-sandbox:

workspace/
├── agent-sandbox/
│   └── dev/
│       └── load-test/
└── perf-tests/
    └── clusterloader2/

Basic startup latency test

This test creates a set of Sandboxes and measures their startup latency.

1. Build ClusterLoader2 (run from perf-tests/clusterloader2/):

go build -o clusterloader2 ./cmd/clusterloader.go

2. Run the test:

# Against a GKE cluster
./clusterloader2 \
  --testconfig=../../agent-sandbox/dev/load-test/agent-sandbox-load-test.yaml \
  --kubeconfig=$HOME/.kube/config \
  --provider=gke

# Against a local kind cluster
./clusterloader2 \
  --testconfig=../../agent-sandbox/dev/load-test/agent-sandbox-load-test.yaml \
  --kubeconfig=$HOME/.kube/config \
  --provider=kind

3. Verify results — results are saved to junit.xml in the clusterloader2/ directory:

<testsuite name="ClusterLoaderV2" tests="0" failures="0" errors="0" time="57.957">
  <testcase name="agent-sandbox-load-test: [step: 01] Start Startup Latency Measurement" .../>
  <testcase name="agent-sandbox-load-test: [step: 02] Create Sandboxes" .../>
  <testcase name="agent-sandbox-load-test: [step: 03] Wait for Sandboxes to be Ready" .../>
  <testcase name="agent-sandbox-load-test: [step: 04] Gather Results" .../>
  <testcase name="agent-sandbox-load-test: [step: 05] Delete Sandboxes" .../>
</testsuite>

Test Recipes

dev/load-test/test-recipes/ contains ready-made scenarios for more demanding performance testing. Some recipes query Prometheus directly for detailed latency metrics; others rely on ClusterLoader2’s built-in measurement methods. See Metrics Collected for which recipe reports what.

Available recipes

Running the rapid burst test

The rapid burst test is the primary scalability scenario. It creates SandboxClaims in repeated bursts and records per-burst latency distributions.

cd dev/load-test/test-recipes
chmod +x run_rapid_burst.sh
./run_rapid_burst.sh                           # script defaults
BURST_SIZE=400 QPS=400 ./run_rapid_burst.sh    # conservative starting point, recommended for a first run
./run_rapid_burst.sh test1                     # append a name to the output directory

Configuration parameters

Note: These are the script’s own defaults, sized for a well-tuned cluster. For a first run, or before raising controller concurrency, start with the more conservative BURST_SIZE=400 QPS=400 shown above.

Total claims created = BURST_SIZE × TOTAL_BURSTS.

For maximum throughput testing, consider raising controller concurrency alongside QPS:

args:
- --kube-api-qps=1000
- --kube-api-burst=2000
- --sandbox-concurrent-workers=400
- --sandbox-claim-concurrent-workers=400
- --sandbox-warm-pool-concurrent-workers=10
- --sandbox-template-concurrent-workers=2

Output

All artifacts (CL2 log, test overrides, Prometheus reports) are saved to a timestamped directory at ${TEST_DIR}/tmp/${RUN_ID}.

Metrics Collected

Recipes differ in which measurements they run and how they collect them. Some query Prometheus directly (CL2’s GenericPrometheusQuery method); others use CL2’s built-in PodStartupLatency or SchedulingThroughput methods, which don’t go through Prometheus at all.

SandboxClaim startup latency

Only rapid-burst-test.yaml collects this metric. It measures the end-to-end time from when the kube-apiserver receives the SandboxClaim create request to when the claim is marked as Ready (implying the claim, sandbox, and pod are all ready).

Note: This metric requires a mutating admission webhook to record the start timestamp. See the timestamp-injection webhook example for setup instructions. Because the kube-apiserver and controller may run on different nodes, this metric may include clock skew.

SandboxClaim controller startup latency

Only rapid-burst-test.yaml collects this metric. It measures the time from when the SandboxClaim controller’s informer first observes the SandboxClaim creation to when the controller marks it as Ready. Because the informer sees the resource only after it has been persisted by the API server, this is a subset of the agent_sandbox_claim_startup_latency_ms_bucket metric — the delta between the two represents time spent in the kube-apiserver plus network propagation delay (plus or minus any clock skew). This metric has no clock skew of its own, since both the start and end timestamps are recorded by the same controller. Unlike agent_sandbox_claim_startup_latency_ms_bucket, this metric requires no webhook and works out of the box.

Scheduling throughput and pod startup latency

These are two distinct measurements, backed by different CL2 methods:

• SchedulingThroughput (high-volume-test.yaml, throughput-test.yaml) — pod scheduling rate, via CL2’s built-in SchedulingThroughput method.
• SandboxStartupLatency / ColdAcquisitionLatency (all five recipes) — pod startup latency, via CL2’s built-in PodStartupLatency method. The label selector varies per recipe — see the table above. rapid-burst-test.yaml additionally collects the two Prometheus-backed SandboxClaim metrics documented below.

throughput-test.yaml also runs ReadyPerSecond, a GenericPrometheusQuery measurement; it isn’t documented elsewhere in this guide.

The controller exposes all metrics at its /metrics endpoint; a Prometheus ServiceMonitor is provided at dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml.

See Also

• Configuration reference — full flag reference for the controller
• Running tests — unit, integration and e2e test commands
• ClusterLoader2 getting started