Sandbox pods reference

Exhaustive reference for pod-per-run sandbox execution: configuration flags, pod identity and quota, run-scoped GitHub token injection, pod naming, and the security properties of the model. For the reasoning behind these mechanics, see the Sandbox pod execution deep dive; for the operator/user view, see the Sandbox pod execution experience.

This page documents the sandbox-pod execution surface (where the agent turn runs). The broader sandbox isolation model — filesystem containment, governance, executor selection, and claim lifecycle — is the Sandbox deep dive, and operator install/config is Sandbox setup.

Configuration flags

Flag	Values	Default	Effect
Sandbox:AgentExecutionMode	in-api, pod-per-run	in-api	in-api runs the agent turn in-process in the API/worker (today's behavior, the rollback path). pod-per-run relocates each run's agent turn into its own Kata-isolated sandbox pod via the A2A bridge.
Sandbox:ReleasePodOnSuspend	true, false	true	When pod-per-run is active and the workflow graph suspends on an external gate (a HITL/review RequestPort, or the coordinator idling while it awaits child runs), true checkpoints the run and releases the pod back to the warm pool. false keeps the pod warm across the suspension for low-latency resume or debugging, at the cost of held capacity.
Sandbox:Kubernetes:AgentHostClaimCreationGraceSeconds	Positive integer seconds	300	Minimum age before the orphan reaper may delete an AgentHost claim that is absent from the active-run map. The effective grace is the larger of this value and Sandbox:Kubernetes:AgentHostReadyTimeoutSeconds + 30 seconds.
AgentHost:ExecutionScratchRoot	Absolute path	/local-workspace	Root of the disk-backed emptyDir used for pod-local execution workspaces and package caches.
AgentHost:ExecutionScratchMinimumFreeBytes	Non-negative integer bytes	8589934592 (8 GiB)	Minimum available scratch space required before AgentHost prepares a local workspace. Failure returns typed reason insufficient_ephemeral_storage.
Coordinator:AssemblyBuildTestTimeoutMinutes	Positive number	20	Total assembly Build/Test wall-clock limit. Expiry cancels the gate and releases its retained AgentHost claim.
Coordinator:AssemblyBuildTestStallTimeoutMinutes	Positive number	12	Maximum interval without a forwarded Build/Test run event before the stall watchdog fails the gate.

Flag semantics

• pod-per-run is the only value that activates the bridge. Any other value (the in-api default) keeps execution in-process. There is no separate "pod-per-turn" mode — granularity within pod-per-run is the hybrid model (warm across consecutive turns, release on suspend), governed by Sandbox:ReleasePodOnSuspend, not by a distinct execution-mode value.
• ReleasePodOnSuspend only matters under pod-per-run. It is a tuning sub-flag; it never changes the execution-mode value. The release is internal behavior of pod-per-run.
• Rollback is a flag flip, not a redeploy. Setting Sandbox:AgentExecutionMode=in-api restores in-process execution immediately. This is the documented mitigation for any instability in the -preview A2A transport — there is no alternate wire transport to deploy. See the A2A reference for the transport's preview status and pinning.

AgentHost receives only a live, immutable capability credential redeemed for the configured run and purpose through /configure. It has no Key Vault, CSI, shared-filesystem, or ambient-token fallback.

Pod identity and quota

A pod-per-run sandbox is the same Kata-isolated pod shape the sandbox subsystem already uses, claimed from a warm pool, but now hosting the full agent (worker agents and the coordinator's own agent turns) rather than only ad-hoc shell commands.

Property	Value / behavior
Runtime class	kata-vm-isolation — a VM boundary around the container, so each run's secret and execution live inside a per-run microVM and are destroyed with it.
Identity	Dedicated sandbox service account federated to agentweaver-agenthost-identity, a managed identity with no Key Vault role assignments (issue #471). Workload identity (federated OIDC) projects only the narrowly-scoped workload-identity token volume — not the full Kubernetes API service-account token — but it grants no vault access, so the sandbox cannot read any user's secrets.
Cluster API access	None. The pod does not automatically receive Kubernetes API credentials; the sandbox stays tokenless for the cluster API even when workload identity is enabled for the model endpoint.
Provisioning	Claimed from a warm pool via a SandboxClaim; the executor waits until the claim is bound to a concrete pod. AgentHost uses the shared agentweaver-agent-host pool (replicas: 2), then receives per-run context through POST /configure before /healthz is expected to become ready. No separate per-run template or per-run warm pool is created for AgentHost. A claim that stays unbound (pod Pending) while Kubernetes schedules is a legitimate wait — there is no app-side capacity pre-check — surfaced on the child run's stream via sandbox.provisioning_pending heartbeats (issue #217).
AgentHost readiness gate	Warm AgentHost pods start in standby. After binding, the executor calls POST /configure with run/user/token/KV secret context plus the workspace descriptor, then polls GET {scheme}://{podIP}:8088/healthz (bounded Sandbox:Kubernetes:AgentHostReadyTimeoutSeconds, default 90s; …ReadyPollIntervalMs, default 1000) before the first A2A turn. /configure is excluded from readiness and returns 409 if called again. The a2a-sandbox-pod HttpClient additionally retries connection-refused only.
Transient API resilience	The idempotent claim create and the bind/IP polls (WaitForBoundAsync, GetPodIpAsync) retry transient Kubernetes API faults up to MaxK8sAttempts (3 total) with exponential backoff + jitter (ExecuteK8sWithRetryAsync): connection resets (SocketException 104/IOException/HttpRequestException), 429/5xx, and HttpClient timeouts. 409 Conflict is not treated as transient — it is attempt-aware to preserve idempotency (a retry-409 = our own create that committed before a reset, so the claim is configured, not reused). Caller cancellation is never retried. The non-idempotent POST /configure is intentionally excluded (issue #230).
A2A turn authentication	Run launch generates a 256-bit random turn bearer token, sends it to the claimed warm pod in POST /configure, and registers it in IAgentHostTurnTokenRegistry. RemoteAgentProxy sends Authorization: Bearer {token} on message:stream; each pod accepts only its configured run token.
Tool-approval return path	When the API-side durable approval gate reports Unknown, pod-per-run mode forwards the grant/deny to the owning AgentHost pod's authenticated root endpoint so its in-memory gate can resolve.
Per-pod resources	AgentHost requests 500m CPU, 1Gi memory, and 1Gi ephemeral storage; limits are 2000m, 4Gi, and 8Gi. The lower storage request avoids reserving the full workspace budget for each warm standby replica.
Quota	Namespace ResourceQuota (k8s/base/quota.yaml) bounds only object counts — pod count, sandbox-claim count, PVCs, and storage. It no longer caps CPU/memory: Kubernetes schedules on pod requests and the cluster autoscaler owns headroom, so a Pending pod waits for the pool to scale rather than being rejected on admission (issue #217). The object-count caps are raised deliberately via a reviewed manifest change, never a live patch.
Lifetime	Bounded by the run and the claim TTL. Under the hybrid model, a pod is released on suspend and a fresh pod is re-claimed on resume; pods never persist past the run.
Egress	Default-deny NetworkPolicy with a narrow allowlist (see Security properties).
Storage	Mounts the shared workspace volume plus a dedicated disk-backed execution-scratch emptyDir at /local-workspace (sizeLimit: 8Gi) for pod-local execution. Assembly Build/Test and preview use LocalReadOnly; implementation turns use LocalWritable and publish through the verified Git write-back flow. Existing disk-backed tmp and home emptyDirs remain separate.

Orphan reaper creation grace

An AgentHost claim missing from the active-run map is not reaped while its Kubernetes creationTimestamp is inside the effective creation-grace window. This keeps a newly bound claim alive through the readiness wait (AgentHostReadyTimeoutSeconds, default 90 seconds); a missing or unparseable timestamp receives no grace and remains eligible for cleanup.

Run-scoped GitHub capability delivery

A pod-per-run sandbox receives only capability credentials tied to the run's immutable snapshots. RunGitHubCapabilitySnapshotLifecycle captures snapshots before launch and gives retries/resumes fresh references to the inherited capability. The API's GitHubCapabilityBroker fences the selected UnattendedCopilot or UnattendedRepository snapshot before and after redemption, then bounds the credential expiry.

In GitHub Copilot mode, KubernetesSandboxExecutor requires a live Copilot credential for the exact run. It transfers that credential in-memory through the one-time /configure call. In BYOK mode, the sandbox resolves its configured provider separately and does not require or use copilotCredential. AgentHostGitHubCapabilityCredentialProvider rejects credentials for a different run or past expiry. AgentHost does not read Key Vault, CSI mounts, shared filesystem tokens, user token stores, or configuration credentials.

/configure field	Required	Meaning
runId	Yes	Configured run identity.
copilotCredential	GitHub Copilot mode only	Opaque snapshot reference, credential, and bounded expiry for that run's unattended Copilot capability. BYOK mode does not use this field.
repositoryAccessToken	No	Separately redeemed repository capability for narrowly-scoped Git/GitHub operations.
turnBearerToken	No	A2A turn authorization token, distinct from the GitHub capability.

Credentials are never logged or persisted. Missing, revoked, expired, or purpose-mismatched snapshots fail closed before the pod becomes ready.

A2A turn bearer token

The A2A turn endpoint has a separate per-run bearer token from the GitHub user token above:

1. KubernetesSandboxExecutor creates 32 random bytes (256 bits) at AgentHost run launch.
2. The token is sent to the claimed warm pod in POST /configure and stored in AgentHostRuntimeState.
3. The same token is stored in IAgentHostTurnTokenRegistry for the owning run.
4. RemoteAgentProxy reads the registry and sends Authorization: Bearer {token} on all calls to POST /a2a/agent/v1/message:stream.
5. AgentHost rejects turn requests whose header does not exactly match its own AgentHostOptions.TurnBearerToken.

This is application-layer auth on top of the A2A NetworkPolicy/mTLS boundary. The important blast-radius property is that a stolen token from one run cannot be reused against another run's pod.

Tool-approval forwarding endpoints

These are internal API-to-AgentHost routes, not public client endpoints. The public caller continues to use /api/runs/{id}/tool-approvals and /api/runs/{id}/tool-denials.

Method	AgentHost path	Body	Purpose
POST	/tool-approvals	runId, requestId, scope	Grant the pod-local pending request. Unknown scope values use once; always is pod/run-scoped and does not survive restart.
POST	/tool-denials	runId, requestId	Deny the pod-local pending request.

Both routes accept the same pod-root bearer authorization used by PreviewRunner controls: either the configured turn bearer or the per-run previewRunnerCredential. A mismatched runId returns 409 state: "run_mismatch".

AgentHost response	Meaning
200 with resolved: true	State is approved, denied, or expired
404 with state: "unknown"	The pod-local gate does not know the request
409 with state: "pending"	The request remains pending
401	The bearer did not match the configured pod credentials

The API locates the pod with IAgentHostOriginResolver, calls it through the a2a-sandbox-pod client, and caps the decision call at 10 seconds. Missing origins, timeouts, transport failures, 5xx responses, and invalid responses surface publicly as 503 state: "agenthost_unreachable". Terminal forwards cause the API to emit tool.approval_resolved for the owning run.

The credential's secret-store key is derived by PreviewRunnerCredential.SecretKey(runId) with the prefix preview-runner-cred--; KubernetesSandboxExecutor mints it, persists it, and delivers its value in-memory through /configure. Key Vault cleanup uses soft delete rather than purge. If the same run launches again while that deterministic key is deleted but recoverable, the API recovers the key, waits up to 30 seconds for it to become writable, and replaces the recovered value with a fresh credential. Concurrent recovery attempts converge on the same active key. Secret values are never included in recovery errors or logs, and terminal cleanup continues to bound the credential lifetime to the backing pod without weakening Key Vault purge protection.

Sources: apps/Agentweaver.AgentHost/Program.cs:287-288,486-588, apps/Agentweaver.Api/Sandbox/AgentHostApprovalHttpClient.cs:28-112, apps/Agentweaver.Api/Endpoints/RunEndpoints.cs:2590-2718, apps/Agentweaver.Api/Sandbox/Preview/PreviewRunnerCredential.cs:22-35, and apps/Agentweaver.Api/Sandbox/KubernetesSandboxExecutor.cs:706-759, apps/Agentweaver.Api/Auth/KeyVaultSecretStore.cs, and apps/Agentweaver.Api/Auth/KeyVaultRecoverableSecretWriter.cs.

Pod naming and the executing-pod surface

A run's executing pod name is tracked so the UI can show where a run is running.

• PodNameRegistry is an in-memory map from run id → bound pod name. It is populated by the Kubernetes sandbox executor once a SandboxClaim reports its Ready condition True, and the entry is removed when the claim is deleted (e.g. on run cleanup or release).
• The registry is consumed in two places:
  • the system runtime endpoint (GET /api/system/runtime) returns { kubernetes, podName }, where podName is the API/host pod name when running inside Kubernetes — the global fallback; and
  • the run graph endpoint (GET /api/runs/{id}/graph) populates an executionPodName field on each node from the registry, so a per-run/per-node pod name overrides the global fallback as the pod-per-run rollout begins carrying the correct per-pod value automatically.
• The frontend resolves node.executionPodName ?? globalPodName and renders it as a small pod pill (the "executing pod name" surfaced on agent boxes). The pill renders only on Kubernetes — when not running in-cluster (kubernetes: false) or when the pod name is null, nothing is shown, so local/dev runs stay clean. See the experience doc for the rendered behavior.

Field	Source	Meaning
kubernetes	GET /api/system/runtime	Whether the backend is running inside Kubernetes; gates whether any pod pill is shown.
podName (global)	GET /api/system/runtime	The host/API pod name — the fallback pill when no per-node value exists.
executionPodName (per node)	GET /api/runs/{id}/graph, topology deltas, subtask.* events	The bound sandbox pod name for that run/node, from PodNameRegistry; overrides the global fallback.

The same PodNameRegistry also lets preview/port-forward tooling locate a run's pod. That preview path is documented in the Sandbox deep dive and, for its API surface, in Sandbox preview port-forward below.

Sandbox preview port-forward (Feature 017)

Dedicated pages: this feature now has its own Reference, User Guide, and Deep Dive. The summary below stays here for context within the sandbox-pods surface.

A preview port-forward exposes a port of a run's sandbox pod back through the API, so an operator can reach a server the agent started inside the pod (a dev server, a built app, a debug endpoint) as a live preview scoped to that one run's pod. PortForwardService shells out to kubectl port-forward --address 127.0.0.1 pod/{podName} :{targetPort} -n {namespace} (it does not use the Kubernetes API), parses the Forwarding from 127.0.0.1:<port> -> line to learn the local port, and probes loopback TCP until ready. The pod is the same one KubernetesSandboxExecutor provisions through the agent-sandbox controller — the preview tunnels into that pod, not an MXC local sandbox.

This surface is Kubernetes-only: it tunnels through the Kubernetes claim backend's pod, located by run id via the PodNameRegistry. On local/dev backends (no claim pod) there is nothing to forward, and the start call fails with a conflict — "the run must be in_progress with an active Kubernetes sandbox". Every call also verifies the run exists and the caller owns it (403/404 otherwise).

Endpoints

Method & path	Body	Returns	Effect
POST /api/runs/{runId}/sandbox/port-forward	{ "targetPort": <1..65535> }	PortForwardSessionDto	Starts a kubectl port-forward from the run's target port to a loopback port on the API, and returns the new session. 429 when a session cap is hit; 409 when the run has no active sandbox pod.
GET /api/runs/{runId}/sandbox/port-forward	—	PortForwardSessionDto[]	Lists the active preview sessions for the run.
DELETE /api/runs/{runId}/sandbox/port-forward/{sessionId}	—	{ session_id, stopped: true }	Stops the identified session and tears down its tunnel.

PortForwardSessionDto

Field	Meaning
session_id	Identifier for this preview session; used as {sessionId} to stop it via DELETE.
local_port	The loopback port on the API host that kubectl bound; what the API forwards from. The backend returns this port, not a public URL.
target_port	The port inside the sandbox pod that is being forwarded.
pod_name	The bound sandbox pod the tunnel targets (from PodNameRegistry).
started_at	When the session started.
preview_url / previewUrl	Web-only, optional. The frontend reads these to render an embedded iframe, but the backend does not currently populate them; the UI explicitly says so when no proxied URL is returned.

Behavior

• Per-port, explicit. A session forwards one target port; opening another preview is a second POST. Sessions are listed and stopped individually.
• Scoped to the run's pod. A session can only reach that run's sandbox pod — the run id resolves to a single bound pod, so a preview never crosses into another run's pod.
• Inbound only, no egress widening. The tunnel is an inbound path the operator opens to the pod; it does not alter the pod's default-deny egress allowlist (see Security properties).
• Capped per run and globally. Default 3 concurrent sessions per run (Sandbox:PortForward:MaxConcurrentSessionsPerRun, fallback :MaxPerRun) and 20 globally (Sandbox:PortForward:MaxConcurrentSessionsGlobal, fallback :MaxGlobal); exceeding either raises PortForwardLimitExceededException → 429.
• In-memory, no TTL. Sessions live only in PortForwardService's in-process maps (_sessions / _sessionsByRun); there is no expiry timer. They end only on explicit DELETE, run end (via RunWatchLoopService, which also unregisters the pod), the kubectl process exiting on its own, or Dispose() at shutdown.
• Bounded by the pod. A session is only valid while the run's pod is bound; releasing or replacing the pod (suspend/resume, run end) ends forwarding, and a new preview must be started against the re-claimed pod.

Security properties

Property	Pod-per-run guarantee
Execution isolation	Each run's agent turn, tools, shell, and file ops run in the run's own Kata-isolated pod (kata-vm-isolation), not a shared process.
Control-plane isolation	The orchestration graph, HITL decisions, and run record stay in the worker; a compromised pod cannot alter what happens next.
Credential blast radius	The pod holds only a short-lived, run-scoped credential — never a broker key, never refresh material, never another run's or user's scope. There is no CapabilityTokenService and no central token broker.
A2A turn auth	message:stream requires Authorization: Bearer {per-run token}. The token is delivered only to the claimed AgentHost pod via /configure and removed from the registry when the pod is released.
GitHub token exposure	Brokered by the API for the configured run owner only and delivered in the one-time /configure call, then cached in memory for the pod lifetime; the sandbox identity has no Key Vault access (issue #471), and no CSI user-token file or shared workspace copy exists.
Egress	Default-deny with a narrow allowlist: model endpoint, the API/worker bridge endpoint, and the run's legitimate git remote(s). The database is not reachable from sandbox pods — all run-state I/O flows through the worker.
At rest / past run	Token material does not persist past the pod lifetime; no per-run Secret/SPC is created, and the bearer token is no longer written to SandboxClaim.spec.env in etcd.
Reversibility	The whole mode is gated by Sandbox:AgentExecutionMode; flipping to in-api restores in-process execution with no redeploy.

Related reference

• Sandbox setup — operator install/config of the sandbox backends.
• API reference — the endpoints surfaced above.
• A2A reference — the -preview transport (experimental) that carries agent turns.
• Sandbox pod execution deep dive — the reasoning.
• Sandbox pod execution experience — the user/operator view.
• Sandbox browser preview — preview routes (start/keepalive/stop) that expose a pod-internal server over a public HTTPS reverse proxy.
• Tool Approval SSE Contract — public approval outcomes and coordinator-to-child routing.