Agent Sandbox

Controller Metrics Reference

Prometheus metric families exposed by the Agent Sandbox controller.

Scraping the endpoint

The controller serves Prometheus metrics from /metrics on the address given by --metrics-bind-address, which defaults to :8080. The bundled manifests expose that port as the metrics port of the agent-sandbox-controller Service in the agent-sandbox-system namespace.

For a one-off local check, port-forward the Service:

kubectl -n agent-sandbox-system port-forward svc/agent-sandbox-controller 8080:8080
curl http://127.0.0.1:8080/metrics

For continuous collection with the Prometheus Operator:

  • Helm chart — set metrics.serviceMonitor.enabled=true to create a ServiceMonitor, and metrics.prometheusRule.enabled=true to create a starter PrometheusRule. Both require the Prometheus Operator CRDs, and you will usually also need metrics.serviceMonitor.additionalLabels to match your Prometheus serviceMonitorSelector. See the chart values.
  • Plain manifests — apply the standalone ServiceMonitor at dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml, which scrapes the same port every 10 seconds.

Metric families declared by the agent-sandbox-controller for its metrics endpoint. The Go runtime, process, and controller-runtime metrics that every controller-runtime manager exposes are not listed here.

Metric Type Help Labels
agent_sandbox_build_info Gauge Agent sandbox controller build metadata exposed as labels with a constant value of 1. build_date, compiler, git_commit, git_version, go_version, platform
agent_sandbox_claim_controller_startup_latency_ms Histogram Latency from the controller first observing the SandboxClaim to the claim reaching Ready in milliseconds. launch_type, sandbox_template
agent_sandbox_claim_creation_total Counter Total number of Sandboxes created or adopted for a SandboxClaim. namespace, sandbox_template, launch_type, warmpool_name, pod_condition, created_by
agent_sandbox_claim_startup_latency_ms Histogram End-to-end latency from a mutating admission webhook first observing the SandboxClaim (the agents.x-k8s.io/webhook-first-observed-at annotation) to the claim reaching Ready in milliseconds. Note: Only recorded for claims carrying that annotation. launch_type, sandbox_template
agent_sandbox_client_claim_startup_latency_ms Histogram End-to-end latency from the client-recorded request time (the agents.x-k8s.io/client-first-requested-at annotation) to the claim reaching Ready in milliseconds. Note: Only recorded for claims carrying that annotation, and may be affected by clock skew between the client and controller. launch_type, sandbox_template
agent_sandbox_creation_latency_ms Histogram Latency from Sandbox creation to its first transition to Ready in milliseconds; source timestamps have second-level Kubernetes precision. Sandboxes in the controller’s initial inventory are excluded, so startups spanning restarts, upgrades, or leader failover may be omitted. Readiness flaps do not record another sample. For warm-pool sandboxes, observed at pool-member Ready (including unclaimed pool sandboxes), not at claim adoption. namespace, launch_type, sandbox_template
agent_sandbox_expired_total Counter Total number of Sandboxes that reached their spec.shutdownTime and expired. Note: SandboxClaim lifecycle expiry is not counted. namespace, sandbox_template, owned_by, shutdown_policy
agent_sandbox_finished_total Counter Total number of Sandboxes whose backing Pod reached a terminal phase (Finished=True). namespace, sandbox_template, owned_by, reason
agent_sandbox_resume_latency_ms Histogram Latency from the controller first observing operatingMode=Running (after a fully-suspended state) to Sandbox Ready, in milliseconds. Note: Not recorded across controller restarts or leader changes, or for transitions that complete in the pass that starts them. namespace, sandbox_template, owned_by
agent_sandbox_resume_total Counter Total number of Sandbox resume lifecycle events by result (requested: resume initiated from Suspended=True; ready: Sandbox reached Ready=True; error: resume hit an error reason, counted at most once per resume). Note: ready and error are not recorded across controller restarts or leader changes. namespace, sandbox_template, owned_by, result
agent_sandbox_suspend_latency_ms Histogram Latency from the controller first observing operatingMode=Suspended to the Sandbox reaching Suspended=True (PodTerminated), in milliseconds. Note: Not recorded across controller restarts or leader changes, or for transitions that complete in the pass that starts them. namespace, sandbox_template, owned_by
agent_sandbox_suspend_total Counter Total number of Sandbox suspend lifecycle events by result (requested: suspend initiated; pod_terminated: backing Pod terminated and Suspended=True reached; error: suspend hit an error reason, counted at most once per suspend). Note: pod_terminated and error are not recorded across controller restarts or leader changes. namespace, sandbox_template, owned_by, result
agent_sandbox_warmpool_size Gauge Number of sandboxes currently in each warm pool, labeled by sandbox status. namespace, warmpool_name, sandbox_template, sandbox_status
agent_sandboxes Gauge Monitor the point-in-time number of sandboxes in the cluster. namespace, ready_condition, expired, launch_type, sandbox_template, owned_by, created_by

Availability

Three of the families above are available in every install; the rest depend on how the controller is run.

  • Available in every install. agent_sandboxes, recomputed on each scrape by a collector that lists every Sandbox; agent_sandbox_build_info, a constant gauge present for the whole life of the process; and agent_sandbox_creation_latency_ms, emitted after an eligible Sandbox first reaches Ready.
  • Require --extensions. Every SandboxClaim family (agent_sandbox_claim_*) is recorded by the SandboxClaim controller, which the manager only starts when it is run with --extensions. Without that flag they are absent entirely, not zero.
  • Require a timestamp the controller does not write. The end-to-end latency families measure from an annotation stamped by something upstream, and are not recorded when it is missing. agent_sandbox_client_claim_startup_latency_ms needs agents.x-k8s.io/client-first-requested-at, which the Go and Python SDKs stamp and hand-written manifests do not; a missing annotation is skipped silently, and only an unparseable one is logged, at -v=1. agent_sandbox_claim_startup_latency_ms needs agents.x-k8s.io/webhook-first-observed-at from the optional timestamp-injection webhook, and does log the skip at -v=1 when it is absent.

Those last two measure across a process boundary and inherit any clock skew between the client or webhook and the controller; agent_sandbox_claim_controller_startup_latency_ms does not.

Label values

Label Values
namespace Namespace of the Sandbox or SandboxClaim.
launch_type warm when the Sandbox came from a SandboxWarmPool, cold otherwise. The claim latency histograms additionally report unknown when no Sandbox was resolved; agent_sandboxes never does, and reports a Sandbox carrying no launch-type label as cold.
sandbox_template Name of the SandboxTemplate resolved for the object. When none can be resolved the value is unknown on agent_sandboxes and __unknown__ on the claim families and agent_sandbox_creation_latency_ms.
created_by Component that created the object, normalized to go-client, python-client, controller, or unknown.
owned_by SandboxClaim or SandboxWarmPool when one of those controls the Sandbox; None for any other controller, or for none at all.
ready_condition true or false, from the Ready condition of the Sandbox.
expired true when the Ready condition reason is SandboxExpired, otherwise false.
pod_condition ready when the adopted Sandbox was already Ready at adoption, not_ready otherwise. Despite the name this reads the Sandbox’s Ready condition rather than the Pod’s, and cold launches are always recorded as not_ready.
warmpool_name The claim’s spec.warmPoolRef.name. On a warm launch the adopted Sandbox’s owning SandboxWarmPool is guaranteed to match it, since verifySandboxCandidate rejects a candidate owned by any other pool.
git_version, git_commit, build_date, go_version, compiler, platform Build metadata baked into the controller binary. Constant for the lifetime of a process.

agent_sandbox_claim_creation_total is incremented once per Sandbox the controller creates or adopts for a claim, not once per claim reaching Ready. Claim latency histograms use the agents.x-k8s.io/claim-first-ready-at annotation to record only the first time a claim goes Ready.

agent_sandbox_creation_latency_ms records the first Ready transition for Sandboxes newly observed after the controller’s initial inventory. The controller keeps eligible Sandbox UIDs in memory until their first Ready observation or deletion; it stores no metric bookkeeping in Sandbox status or annotations. Readiness flaps, pod restarts, and suspension/resumption do not add another sample. All Sandboxes in the initial inventory are excluded, regardless of readiness, so controller restarts, upgrades, or leader failover may omit samples for Sandboxes still starting. For warm-pool members, the sample is recorded when the member first becomes Ready, including members that are never claimed.

agent_sandboxes needs no guard because the collector recomputes it from a live Sandbox list on every scrape, so its series disappear along with the sandboxes they describe, whereas every label combination the counters and histograms have observed stays in the process until it restarts.

See Also

  • Performance Assessment — the load-test recipes that query agent_sandbox_claim_startup_latency_ms and agent_sandbox_claim_controller_startup_latency_ms, with the Prometheus queries and default thresholds they assert.
  • Sandbox Metrics — OpenTelemetry traces and metrics from the Python SDK, which are separate from the controller metrics on this page.
Last modified October 15, 2026: metric: add Prometheus lifecycle metrics for direct Sandbox creation (#1076) (a54fb0a)