Environment Sandbox leases

An Environment outlives a particular Sandbox placement. Its lease records which provider resource belongs to the current execution attempt; the workspace volume retains files independently. Retiring a Sandbox releases placement, not workspace data or the Environment itself.

Environment reserves a durable lease before calling the provider and checks the exact workspace attachment and network policy before reporting dispatch readiness. Generation fences reject stale attempts instead of letting them change newer owner state. This page explains those source contracts, not a live Kubernetes or AgentHost test. The runtime deep dive connects the lifecycle, placement and recovery views.

Environment discovery, fenced recovery, root and child startup, AgentHost activation, and terminal Sandbox cleanup. The Gateway forwards to the fixed Environment owner. A ready lease requires exact Workspace and network observations. Retirement releases Sandbox placement, not the Environment or its Workspace data. Cleanup uses the current owner fence for authorization and retains the saved provider fence for resource identity.

Open full-size PNG · Open editable draw.io source

Opt-in Host recovery at a pending gate or saved child join. Environment retires the exact old placement and creates each replacement with the retained workspace. Orchestrator registers each Host at the next fence. The Host must prove recovery from authenticated cache or journal evidence. For a prepared child dispatch, Orchestrator appends an immutable rebind receipt before it releases the hold. The original dispatch row remains unchanged, and ordinary dispatch sends the saved message after release. Transport and timeout retries reuse the same operation. A spent-bootstrap activation failure needs a new operation and one more fence advance. The gate remains pending.

Open full-size PNG · Open editable draw.io source

Reconcile a committed pull request

Committed OpenPullRequest recovery is limited to one exact operation at a saved child join. The owner returns the stored pull request under current authority; Orchestrator records the original receipt and keeps the dispatch hold until Host readiness.

Open full-size PNG · Open editable draw.io source

Coordinator first-start

After Orchestrator accepts the root and admits its first message, it stores a fenced startup intent and runs a bounded continuation in a fresh service scope. The same continuation starts direct roots and invocation-bound roots after their admission checks pass. The start response and session status expose the saved startup state; pending does not mean that AgentHost is ready.

Environment first registers the run owner, verifies the exact current run selection, applies the selected network policy, and creates a private workspace volume. The first-start volume is 1 GiB, ReadWriteMany, Strict, and uses Retain for both reclaim and owner deletion. It is bound to the Environment, has no authorized consumer list, and mounts at /workspace. Environment then creates the first Sandbox placement and returns its current UID, lease, and fences with the selected default source profile.

Orchestrator creates the matching runtime registration, requests Broker bootstrap, and calls the authenticated POST /internal/runtime/v1/activate route. Environment validates the Host receipt and current placement before Orchestrator commits ready. Startup does not send a model turn. A retry requires fresh authenticated run authority; the durable intent stores operation IDs and non-credential receipts, not bearer credentials.

If the network-policy step fails because Kubernetes or Projects & Config is unavailable, a retry of the same start runs that step again. A conflicting policy object, a changed owner fence, or a changed run authority still stops the start. Environment reports a transient failure as unavailable (HTTP 503), not as a denial.

Dispatch uses the same placement and activation owners for a committed child, with a separate child-specific authority check and a durable startup intent. Missing or stale readiness prevents its task message from being sent.

Terminal cleanup uses the saved startup intent for the root and each child. Environment rechecks the current terminal projection and releases only the exact resource and provider fence. A lost placement reply is recovered through the saved operation key; a conflicting registration or resource remains pending. Cleanup does not create a placement, activate a Host, or delete workspace data. The separate Orchestrator runtime-start flow is described in the runtime deep dive.

Checkpoint recovery uses a separate run operation. For the supported pending-gate boundary, Orchestrator advances the execution fence once, Environment checks the same pending recovery operation and confirms the current owner at the new fence. It then releases the exact old placement and creates a new placement over the retained workspace. Orchestrator registers and activates the Host at the new fence. It reports ready only after authenticated native-cache or journal recovery and current placement readiness. It keeps the same gate request and choices. Other checkpoint boundaries remain blocked in this slice.

BuildTest repository preparation and output admission. Version 2 retains full native output until the journal acknowledges it. Command logs are not file evidence. Core owns workflow completion.

Open full-size PNG · Open editable draw.io source

Ownership and admission

EnvironmentSandboxManager owns the Sandbox lease for the complete tenant, project, run, and Environment tuple. EnvironmentGenerationFence protects the Environment lifecycle; each Sandbox resource also has a resource generation, a provider fencing generation, and a current fencing generation. The PostgreSQL lease store uses the Environment owner's advisory lock and compare-and-swap checks, and stores the immutable run selection, provider/options snapshot, idempotency key, and provider recovery request before provider dispatch.

Each external provider effect also has its own durable attempt, tied to the exact owner, lifecycle/resource/provider fences, and request fingerprint. The attempt is written before the call. Its exact result is recorded before the lease or cleanup result is committed. An unresolved, unknown, or legacy attempt blocks release and lifecycle generation advance; timeout, cancellation, client close, an empty provider list, or a terminal lease is not proof that an external call has stopped or had no effect.

The store admits one current Sandbox lease per Environment. A lease in Provisioning, Active, Releasing, or ReconciliationRequired consumes that slot. The lease prevents duplicate provisioning for one owner; it does not enforce a cluster-wide CPU or memory quota. An unresolved or ambiguous provider effect remains current and blocks Environment lifecycle advancement. Old resource generations remain stored after release.

Reservation and provider-operation leases, including retirement, have a fixed 60-second expiry. A valid transition into Active, including reconciliation recovery, assigns a fixed 600-second lifetime from that transition. Replaying the same active resource, reading the lease, or restarting the store preserves the persisted expiry and does not renew it. Released and Failed leases have no expiry. Runtime registrations copy the exact persisted active expiry; this does not extend credential grants, which remain bounded by their configured lifetime and existing registration and bearer checks.

A different valid provider resource returned after a lease is terminal is stored in a separate cleanup record. The original terminal state, resource, and partial-release receipt remain unchanged. Reconciliation can discover pending cleanup under the active owner fence across lifecycle generations. The current fence authorizes the cleanup attempt; the saved source fence and binding remain the provider request and receipt identity. A durable, expiring claim prevents concurrent reconcilers from racing the same delete; the validated release receipt is saved with a current-fence compare-and-swap. Reconciliation detaches the matching Workspace attachment only after that receipt is durable when the source and current lifecycle fences match. For an older source generation, it does not read or change the current Workspace. If there is no current lease, the reconcile route returns the terminal lease result for the late resource it processed.

The Sandbox selection must contain exactly one exclusive candidate. The configured provider ID, adapter version, options schema, options revision, hosting pattern, and advertised capabilities must match that candidate. The manager does not substitute a default provider when the selected one is missing or incompatible. The provider itself advertises VM isolation and PVC attachment; Environment separately verifies the selected Cilium policy generation.

The Sandbox and Azure Files adapters use the same cluster pin: a lower-case SHA-256 digest of the trusted API CA certificate set. Each certificate is normalized to DER, duplicate certificates are removed, and the remaining bytes are sorted and length-prefixed before hashing with the agentweaver.cluster-identity.v1\0cluster-ca\0 domain prefix. This makes the pin independent of PEM whitespace, certificate order, and API address. The Environment adapters read the projected service-account CA file; the native process adapter reads the selected kubeconfig's CA data and applies the same normalization. This avoids requiring permission to read the kube-system namespace. Both paths still require HTTPS and normal TLS validation. Cluster pin validation does not replace the exact PVC/PV UID, claim reference, or owner-annotation checks. Pins created from API-address hashes are not accepted as aliases.

Authorization and API

Invocation-bound creation

The internal invocation-bound source path consumes Core's current invocation binding before Environment reserves creation work. A Projects setup receipt alone is not admission to Environment, and a Core binding alone is not proof of a provider resource. Environment checks the authenticated actor and exact tenant, project, run, root session, invocation IDs, receipt and accepted-selection hashes, and execution fence against the current Core read. Missing, stale, revoked, foreign, or unsupported bindings are rejected rather than replaced with ordinary creation.

These authenticated owner routes use the existing exact run-bound ReadRunSelection authority plus the current Core binding. They are not mapped through Gateway or MCP. Ordinary public creation still requires WriteProjects and rejects invocation-bound creation input; Projects still withholds WriteProjects from run-bound tokens. The internal path does not change sign-in, claims, or role grants.

Method and internal route	Creation fact
POST /internal/projects/{projectId}/runs/{runId}/environments	Register the owner and retain its actual lifecycle operation ID and generation.
POST /internal/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}	Create volume metadata with its own operation/revision link, under an already-bound Environment.
POST /internal/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}/provision	Reserve provider work only for already-bound volume metadata; retain the actual Storage result separately.
POST /internal/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/provision	Reserve the Sandbox under the retained Environment and volume metadata bindings; retain the actual provider result and lease fences.

Each body requires p1Binding with coreInvocationId, setupInvocationId, setupReceiptHash, rootSessionId, acceptedSelectionHash, and executionFence, alongside the existing operation fields. These expected values never substitute for the authenticated Core response.

The creation record is committed in the same Environment owner transaction as the operation it describes. Registration, a reserved volume or Sandbox operation, and a completed provider result are distinct facts. Only a committed provider result can retain the real provider reference; a planned reference or successful reservation must not be presented as that result. Existing resources without a retained creation binding cannot be enrolled retroactively.

Schema version 12 stores these facts in environment.p1_creation_links. WorkspaceVolumeMetadata is a completed metadata-only fact, distinct from a WorkspaceVolume provider reservation. Pending provider links retain the operation ID and reserved generations but no invented provider identity. Completed links retain the real provider seam, ID, resource reference and binding, plus the volume transition revision or Sandbox provider fence and lease revision. The operation ID joins the owner record for the exact data generation and request/result snapshot. Sandbox resource identity is the owner-generated operation ID, not a relabeled Pod UID.

Binding and reservation fields are immutable. A pending provider link can be finalized once as Completed, Failed, or Uncertain; failed or uncertain links are never upgraded into invocation-bound creation evidence by a later recovery or retry. Ordinary resource recovery remains separate. Only retained completed creation facts can qualify for a later observation join.

sequenceDiagram
    participant Caller
    participant Core
    participant Environment
    participant Provider
    Caller->>Environment: Request invocation-bound creation with expected binding
    Environment->>Core: Authenticated current-binding read
    Core-->>Environment: Current exact run and root binding, no-store
    Note over Core,Environment: Remote reads do not hold a distributed lock
    Environment->>Environment: Owner lock; validate binding; commit operation link
    Environment->>Core: Recheck before provider effect
    Core-->>Environment: Current binding or denial
    Environment->>Provider: Exact reserved operation and generations
    Provider-->>Environment: Actual result and provider identity
    Environment->>Environment: Owner transaction; retain result and binding link
    Note over Environment,Provider: Pod configuration is not native mount success

The bounded verification callback runs under the existing Environment owner lock. It must not acquire that lock again through an Environment placement, registration, or lease route, and must not perform provider effects. A callback failure rolls back its local owner transaction. Current checks are repeated after bounded callbacks and before effects where required by the creation path. They are remote reads, not a distributed lock: Core or Projects authority can change after a response and before Environment commits or dispatches. This slice does not claim current-at-commit revocation across services.

The creation evidence reads expose the retained identities through the current owner-locked runtime read path and a separate root-only Host consumer. Native CLI consumption remains separate. They do not implement that join, prove a native mount, run the native execution acceptance fixture, or close the parent integration issue. PodObservation and downward-API fields remain configuration/control-plane observations, distinct from Host's actual Linux mount observation.

Environment lifecycle registration and release use separate owner routes. Registration accepts only idempotencyKey; the tenant and actor come from the fresh Projects context, and the exact immutable run selection is resolved and rechecked at the owner-store commit for both a first write and replay. The Environment ID is a stable hash of the canonical tenant, project, run, and idempotency tuple. The first durable lifecycle generation is registered from expected generation zero. Release accepts only a positive expectedLifecycleGeneration and idempotencyKey, and always targets Released; callers cannot supply provider resources, receipts, or a different target state. Its final commit rechecks authority and selection and is blocked by unresolved provider attempts or pending late cleanup.

GET .../lifecycle requires fresh ReadProjects authority and rechecks it after the durable read, and responds with Cache-Control: no-store. The provider-attempt and late-cleanup reads require a current authenticated caller with run-selection authority; each also rechecks authority after its store read. These read routes do not invoke a provider effect.

All Sandbox routes require an authenticated caller and fresh Projects authority. Provision, inspect, abandon, and reconcile require target-project WriteProjects plus the separate ReadRunSelection permission before they read the immutable run selection or contact Kubernetes. They compare the fresh authorization and selection again after asynchronous work and before accepting provider callbacks. The public placement projection requires fresh WriteProjects authority; it does not read run selection or contact Kubernetes. The internal run-bound placement read requires fresh ReadRunSelection authority and exact non-null project/run token bindings. The Environment JWT middleware continues to enforce the configured Environment audience for both routes.

Environment discovery uses owner-scoped reads. Listing Environments or workspace volumes requires fresh WriteProjects authority and the separate ReadRunSelection permission. Environment reads the exact immutable run selection and rechecks both authorities after the store read. Volume detail uses fresh ReadProjects authority and rechecks it after the read. These routes return bounded pages. Environment-list cursors are tied to the tenant, project, and run. Workspace-volume cursors are also tied to the Environment and current lifecycle generation. Returned volume statuses keep the generation of their latest completed operation, including retained or released records from earlier generations. A cursor for another scope is rejected.

The discovery and status routes return lifecycle and resource references, provider identity pins, and pending-operation retry fields. They do not return the raw volume specification, provider options, or provider release descriptor. For a pending operation, retry the same operation with its original idempotency key and expected generations. Do not start a replacement operation because a response was lost or the status is pending. These Environment routes do not browse mounted files, and releasing access does not restore data removed by a provider operation.

The Gateway exposes the recovery routes listed in the HTTP API reference under /api/v1; it does not expose every native route below. It forwards the validated bearer and optional tenant selector to the fixed Environment owner. Environment remains responsible for current authority and fence checks. These Gateway routes are not MCP tools.

Route	Operation
GET /api/projects/{projectId}/runs/{runId}/environments	List owner-scoped Environment summaries with bounded, selection-authorized pagination.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes	List safe workspace-volume status across the Environment owner history.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}	Read safe volume status and its exact pending-operation retry descriptor.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}/bind	Bind an existing volume generation under its exact resource and data fences.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}/unbind	Unbind an eligible volume generation under its exact resource and data fences.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}/release	Release an eligible volume generation using the saved provider policy.
POST /api/projects/{projectId}/runs/{runId}/environments	Register the exact Environment owner from trusted Projects context; body contains only idempotencyKey.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/release	Release the exact owner using expectedLifecycleGeneration and idempotencyKey; provider evidence is read from durable owner state.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/lifecycle	Read the durable lifecycle snapshot under fresh ReadProjects authority.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/provision	Validate the exact Workspace generation and Cilium policy generation, reserve an owner lease, then provision the selected adapter.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox?networkPolicyGeneration={generation}	Inspect the current lease. It reports ReadyForDispatch only after fresh Cilium verification and exact provider observation.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/v1/placement	Return the versioned owner-local projection of the exact current, active, unexpired lease and its placement references.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/v1/internal/placement	Return the same projection to a run-bound caller with exact project/run bindings and ReadRunSelection; no run selection is fetched and no provider effect occurs.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/v1/operations/{operationId}/provider-attempts	Read the exact operation's durable provider attempts, including unresolved and unknown outcomes.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/v1/late-resource-cleanups	Read retained late-resource cleanup records and their exact attempt, claim, and receipt state.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/abandon	Explicitly abandon one current resource generation and provider fence. The request is not itself proof of ownership; fresh Projects authorization and the durable owner CAS are required.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/reconcile	Recover an interrupted operation by its recorded owner, operation ID, resource generation, and provider fence. It also claims and releases at most one pending late resource under the active owner fence, using the saved source fence for provider identity. It detaches the matching Workspace only when the source and current lifecycle fences match, and returns its terminal lease result when no current lease remains. A network-policy generation can be supplied to verify readiness.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/build-test/binding-preparation?sessionId={sessionId}&executionProfileReference={profile}	Validate the current Sandbox profile binding for a session without starting a command.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/build-test/commands	Resolve the accepted Core checkpoint, reserve its immutable command intent, and execute the stable BuildTest operation.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/build-test/commands/{operationId}	Read the exact persisted operation under the current owner and Sandbox binding.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/build-test/commands/{operationId}/reconcile	Reconcile the same operation and immutable intent; do not create a replacement operation.
GET /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/build-test/commands/{operationId}/workspace-capture	Return the exact sealed version 2 native document as bounded binary bytes with no-store, after current owner checks. The digest and length must match the stored capture reference. After acknowledgment, read retained bytes through Source Control instead.
POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/sandbox/build-test/commands/{operationId}/workspace-capture/acknowledge	Accept the original API request and exact Events journal acknowledgment. Verify it through Core, save the acknowledgment, then request exact-UID native Pod cleanup. Retry preserves the same capture and acknowledgment.
POST /internal/provider-lifecycle/reports	Provider report route. It fails closed until provider identity authentication and Core interruption acknowledgement are admitted.

Provider lifecycle reports have a durable idempotent contract and lease/fence reconciliation, but the mapped report route currently returns unavailable until provider identity authentication and the Core interruption command are admitted. This source does not claim end-to-end suspend, relocation, or resume.

Both placement routes use the owner-scoped ISandboxLeaseStore.GetCurrentAsync callback overload under the exact active Environment owner fence. The public callback rechecks fresh WriteProjects; the internal callback rechecks fresh ReadRunSelection and the same exact project/run binding. The store retains the same transaction and owner advisory lock while the callback runs. It then rereads the current lease and compares its owner fence, resource generation, operation, lease revision, provider/current fences, state, current flag, expiry, and update stamp before commit and return. A mismatch fails closed. A callback exception rolls back and returns no projection. The manager also compares the retained snapshot with its first read before making the final authorization request. The projection returns the tenant/project/run/Environment tuple, lifecycle and provider fences, lease revision/expiry/current state, and the exact resource, endpoint, and opaque placement references recorded by the lease. It does not derive lease data from EnvironmentSandboxResult, caller arguments, run selection, or runtime registration. The provider ResourceId is preserved as stored; it is not relabeled as a Kubernetes UID because a planned aw-claim-* identity and a returned Kubernetes UID are distinct values. Missing placement returns no content; expired, stale, non-current, foreign, or non-active snapshots are rejected. Provider options, release descriptors, and credentials are not included. Profile mapping, receiver registration, and delivery receipts remain separate from placement projection; a missing profile mapping remains unavailable until the authenticated resolver verifies and persists an endpoint-bound replacement.

The additive providerPin contains the successful consumer's provider, adapter version, options schema and revision, resource identity and generation, and negotiated capabilities. It contains no option values, recovery metadata, credentials, or unresolved candidates. The current placement read still denies an expired lease. Historical material and usage receipts retain their original runtime and selection facts under separate current read authorization.

The callback is for a bounded owner-scoped read, such as the fixed Core authorization-context request. It must not recurse through a registration route that reads the same Environment lease or perform provider effects. A successful placement read is not authorization for a later configure or provider effect; those operations recheck current authority and lease state independently.

Pod configuration observations

SandboxObservation.PodObservation is an additive, optional SandboxPodObservation. The AgentSandbox provider populates it only from validated current Kubernetes Sandbox, Pod, and PVC reads, never ConfigMap identity. It carries contractVersion, observedAt, environmentFence, resource, fencingGeneration, provisionOperationId, clusterIdentity, namespace, claimUid, sandboxUid, podName, podUid, podResourceVersion, podGeneration, and podPhase. resource.resourceId and claimUid identify the SandboxClaim; sandboxUid identifies the Sandbox custom resource; podUid identifies the distinct actual Host Pod. clusterIdentity uses the CA-set digest described above.

The provider rereads the exact Pod by namespace/name after its asynchronous PVC, runtime-class, event, and optional endpoint reads. A changed Pod UID or resource version, disappearance, foreign ownership, or a deleting Pod fails closed. RuntimeEndpoint.PodUid, when present, must equal the observed Pod UID. Absent Pods or missing Pod generation, resource version, or phase produce no Pod observation. Older Kubernetes versions can omit Pod generation; it is never inferred from the Sandbox, claim, or lease. Legacy startup reads remain available, but that missing observation is incomplete evidence.

workspaceAttachment contains only the observed configuration's PVC namespace, claimName, claimUid, mountPath, and readOnly after the exact owner/fence/PVC checks. isolationConfigurationVerified describes the Pod security and runtime-class configuration. Neither proves successful filesystem mounting, storage reads/writes, byte isolation, or actual VM execution inside the Host.

The Kubernetes API server omits readOnly (on volume mounts and PVC volume sources), hostNetwork, hostPID, and hostIPC from a Pod when their value is false. The provider reads an absent value as false for sandbox Pods and for BuildTest command, preparation, and collector Pods, so a writable workspace or /state mount passes. A present value must be a JSON boolean. Mounts that must be read-only, such as the BuildTest collector workspace mount and its PVC source, still fail when readOnly is absent. Fields that the API server keeps when false, such as automountServiceAccountToken, enableServiceLinks, and shareProcessNamespace, must be present.

New AgentHost Pod templates inject AGENTWEAVER_HOST_POD_UID exclusively through Kubernetes Downward API fieldRef (apiVersion: v1, fieldPath: metadata.uid). A present identity variable must use that exact source, not a literal or another field. Legacy Pods without this additive variable remain inspectable, but cannot supply Host self-identity evidence; the Host's separate evidence path must reject missing or mismatched values against the observed podUid.

For a strict read, use authenticated GET /internal/projects/{projectId}/runs/{runId}/environments/{environmentId}/coordination/sessions/{sessionId}/runtime-bootstrap/profiles/{profileId}/pod-observation. It retains readiness's exact run-bound Projects authorization, current Core session context, profile resolution, and owner-locked current lease checks. It returns only the typed observation with Cache-Control: no-store; no provider options, recovery descriptors, or credential values are included. Missing Pod/attachment/isolation configuration, expired or changed owner fences, inconsistent placement, or unavailable current owner state fail closed. Incomplete configuration observations return 409 with runtime_pod_observation_unavailable; provider and authorization failures retain their existing error codes.

This source API is not live evidence and is not an invocation-bound ownership receipt. It accepts no invocation ID. Native execution acceptance still needs a separately authoritative invocation-to-creation binding for the exact actor, owner, session, execution fence, and resource, plus actual Host/native mounted-storage and isolation observations. Joining those receipts is a separate owner integration; a caller-supplied invocation ID cannot establish historical resource ownership.

The source EnvironmentRuntimePlacementReader.GetPodObservationAsync<TResult> overload accepts a bounded read-only verification/projection callback with the current readiness context and typed Pod observation. It runs under the same retained Environment owner lock. Core owner context and current Projects authorization are rechecked after the callback, then the store checks the exact lease again before returning. The callback must use owner-retrieved creation bindings, never caller-asserted provenance. It must not perform effects, reenter placement APIs, or reacquire the same owner advisory lock. This hook does not itself implement invocation issuance, consumption, or durable receipts.

Before creating or replacing a Runtime registration, Orchestrator calls the authenticated POST .../runtime-bootstrap/profiles/{profileId}/resolve owner route with the source profile ID. Environment reads the current run-authorized Sandbox placement, exact owner and fences, and actual receiver HTTPS endpoint. It checks the receiver configure URL and audience from the immutable AgentHost configuration, then verifies /health/live over HTTPS by connecting to the observed Pod IP while retaining the receiver DNS name for certificate validation. The probe accepts only a no-store JSON live response, sends no credentials, disables redirects and proxy use, and has a ten-second deadline and a 4 KiB response-body limit. It does not reuse a pooled connection across endpoint checks, so an old Pod connection cannot verify a replacement Pod. Environment stores an immutable profile mapping bound to that placement and endpoint. The derived profile ID is stable for the same placement and changes when the observed Pod or Service changes. Orchestrator uses only that returned ID for the registration and re-reads the current placement before committing its binding. A replacement Host therefore resolves a new profile after Pod or Service replacement; later bootstrap, readiness, and Workspace reads reject an old profile when the current endpoint no longer matches the stored one. The mapping carries no credential values and does not dispatch a provider effect. Both read and resolve routes return no-store responses. These routes do not change Workspace attachment, retirement, or public write permission. The replacement pins the authenticated actor, accepted selection hash, source profile configuration hash, and exact copied replacement profile. Its ID includes the requested source profile ID, actor, and accepted selection hash. Commit replay and later bootstrap, readiness, and Workspace reads recheck the actor, selection, and source configuration; a changed source profile after application reload or a changed profile copy fails closed. Only the replacement profile ID and configure endpoint differ from the source profile. Base profiles and the accepted model, tool, and provider selections remain unchanged.

The optional global Environment:RuntimeBootstrap:DefaultSourceProfileId removes the need for an exact per-run static source entry. Static sources take precedence; otherwise only that default ID derives a source from the authorized placement and receiver configure URL. Environment derives the observation URL from its Orchestrator owner root and sets no MCP server. Startup requires both bootstrap owners and the AgentHost launch profile, and rejects invalid or colliding default IDs. An unset default preserves static-only behavior. Replacement-ID lookup and registration, readiness, Workspace, and resume reads reconstruct the same source and verify its hashes. A new Pod IP or UID does not change the source hash when the logical receiver is unchanged. Changed receiver or observation configuration still fails closed. Profiles contain no credentials. This does not add a scheduler or a new registration caller.

sequenceDiagram
    participant O as Orchestrator
    participant E as Environment owner
    participant S as Sandbox provider
    participant H as AgentHost Pod
    O->>E: POST profile resolve (source profile ID)
    E->>S: Read current owner-fenced placement and HTTPS endpoint
    S-->>E: Exact Service, Pod, receiver URL, and fences
    E->>H: HTTPS GET /health/live (Pod IP; receiver DNS for TLS)
    H-->>E: Live health response
    E->>E: Resolve static source, else opted-in lease-derived source
    E->>E: Persist endpoint-bound profile mapping
    E-->>O: Stable resolved profile ID
    O->>E: Re-read placement and bind Runtime registration

The Environment owner must already exist and be active. These routes do not register or release an Environment lifecycle, create a Core run provider pin, or start a model run. The admitted Core source has no run-terminal or durable run-owner evidence API for this consumer. Therefore, no caller-supplied abandoned flag, lease age, stale credential, timeout, Pod failure, or failed lookup is treated as proof that a run ended.

Trusted suspend Workspace flush

Before a sandbox suspends, Environment calls the trusted AgentHost flush operation. The flush makes AgentHost persist its Workspace data and native-turn cache, so a later resume starts from a verified data generation instead of losing work. It does not claim a complete deployed suspend/resume flow.

Core calls authenticated POST /internal/runtime/suspend/resolve with its current suspend operation and Host session proof. This request cannot contain Workspace evidence. Environment reads the current registration, attached Workspace, and volume snapshot through their owners. It returns the exact Storage resource, provider binding, transition revision, and next data generation. Resolve is a read-only step while Core's operation is Reserved. Core stores the resolved request and its target phase version when it advances Reserved to Fenced. Host delivery accepts only that saved Fenced request. Retries must use that saved request, not new generations. The shared request and client live in Agentweaver.Identity; Core does not need an Environment assembly reference.

POST /api/projects/{projectId}/runs/{runId}/environments/{environmentId}/workspace-volumes/{volumeId}/flush reserves the exact volume transition. Environment checks fresh Projects authority, Core's exact current-operation response, registration, placement, and attached Workspace before and after Host delivery. The trusted Host response must contain the matching native-turn receipt, durable cache acknowledgment, and Workspace flush receipt. The Azure Files adapter verifies its pinned PVC, PV, full claim specification, and approved mount options around that delivery. Both routes require an expiring authenticated bearer and return no-store. The original actor is retained; no service credential replaces it.

Only a verified receipt advances data generation by one. Migration 20261011133000_WorkspaceFlushEvidence adds nullable owner_effects.flush_evidence_json. The receipt and Host evidence are saved with the transition in the same owner transaction. Authority is checked again after database waits and before commit. Uncertain effects keep their evidence for reconciliation without advancing the generation. Failure before provider dispatch records no effect. Completed replay returns the saved evidence without repeating Host flush. Conflicting evidence cannot replace a saved receipt. A resource and a caller-supplied durability flag alone cannot complete a flush.

Placement retirement accepts changed Workspace counters only with the exact saved suspend flush for that retiring lease. Core calls authenticated POST /internal/runtime/suspend/release-placement only after it commits the complete suspend manifest. The shared release request contains the saved Host receipt and exact flush transition revision. Environment asks Core's POST /internal/runtime/suspend/require-release-current to confirm that saved evidence before provider release and owner completion. Detach and lease completion repeat the Projects and Core checks after owner database waits and before transaction commit. Ordinary abandonment and retirement recovery apply the same check when their Workspace has trusted suspend flush evidence. The Sandbox provider must confirm release before Environment detaches the exact Workspace and records the lease as Released. Completed detach replay keeps the same resource, data generation, and transition. Core's client rejects pending release, changed placement, and different detach counters. Credential revocation alone is not placement release. It does not rewrite the original attachment or immutable provider selection. Workspace retention and reclaim rules remain unchanged. Pure checks cover these source contracts; PostgreSQL and mounted CSI execution require their separate acceptance runs.

Retained Workspace resume

The Environment source first prepares and verifies the retained Workspace. Those two authenticated, no-store steps do not require a new Runtime registration or a configured model session. Both retain the original expiring actor and ask Core to confirm the exact saved operation.

POST /internal/runtime/resume/prepare accepts RuntimeWorkspaceResumeRequest. It contains the complete source manifest, saved Host receipt, and exact flush transition revision. The optional RestoredWorkspace must be absent. Resume requires the exact private configuration acknowledgment before any owner call or Environment effect. Ordinary suspend receipts retain their existing validation; a receipt without saved configuration cannot authorize resume. Environment checks the recorded flush, released source Sandbox, retained volume, and unchanged accepted provider selection. Historical flush reads require the current active owner lock and an exact older lifecycle from that same owner. An old lifecycle does not gain permission to change current state.

Prepare advances the Environment lifecycle once with a stable operation key. Core and Projects authority are checked after database waits and before that transition commits, including replay. Environment reapplies the same accepted Network rules with the source policy generation plus one. The recorded previous policy generation must equal the manifest's source generation. It provisions a new Sandbox with the retained Storage resource and data generation, then verifies the actual mount. The response contains the exact new lease, Workspace attachment, and transition revision. Placement evidence does not verify Workspace content or authorize dispatch. Core saves that complete response before requesting the second step.

POST /internal/runtime/resume/verify requires that saved RestoredWorkspace. Environment compares it with a fresh owner-scoped mount and Sandbox observation. It calls the new Host's read-only /runtime/v1/workspace/resume operation. The actual tree hash must equal the manifest hash. Environment then repeats the placement, Network object, and current-operation checks. Only the final receipt contains both the native content proof and verified new Network generation. Core must obtain it before advancing its execution fence, registering the new Runtime, restoring the existing cache, or dispatching. Provider-checkpoint and guest-snapshot restoration remain explicitly unsupported on this path.

The new Host address comes from the owned Kubernetes Sandbox Service and its exact singleton Pod. Environment checks the Service owner, DNS name, selector, Pod UID, and Pod address. It connects to that observed Pod address while retaining normal TLS verification for the Service DNS name. This permits the pre-configuration read without requiring a readiness-gated DNS address. It does not bypass certificate validation or create another Service. The original configured owner/profile remains the source of the observation address and MCP setting. The saved provider request retains the resume source, so later bootstrap reads reconstruct the new address after restart. No caller or new placement-specific configuration supplies that address.

After Core advances its execution fence, POST /internal/runtime/resume/bootstrap starts the new Runtime. It requires Core's phase-4 request, current registration, exact saved mount, and server-issued resume attempt. Environment reads the acknowledged server configuration from the protected Events material route. The saved configuration contains no credential values; no caller supplies replacement configuration. Events returns its exact bytes only after current Core and registration checks. Public event streams expose neither those bytes nor their private object reference.

Environment uses the existing Broker delivery and Host activation paths. It stores each attempt's immutable request and complete receipt in the existing owner_effects table. The RuntimeResumeBootstrap migration changes only that table's check constraints. It adds no table or grant, and service startup never applies it. Attempts for one resume operation run in order. Owner and lease locks cover database changes, not calls back into Environment. Receipt replay still requires fresh Core, registration, placement, Host state, and recovery-evidence checks. A saved receipt alone cannot authorize dispatch.

Recovery distinguishes native-cache from journal-rebuild. The native path validates the exact compatible saved cache. If the cache is missing or incompatible, the Host can rebuild history from the authenticated saved session journal. It requests pages for the manifest's saved source session, keeps the exact checkpoint and flushed position, and checks every page. Events uses saved fork lineage and immutable source-position bounds to include only that session's visible ancestors. Run-wide positions can have valid gaps in a session-visible page. For each TURN, the Host reads material by the page event's exact session and event ID. The event payload, its single stored object reference, the read acknowledgment, and the event position must agree. The Host also checks the current tenant, byte bound, and content digest. Each TURN keeps its own recorded runtime, revision, fence, selection, SDK, model, and capacity facts. These historical facts need not match the latest cache or current Runtime. A direct material read does not establish session visibility, and unrelated session history is not a fallback. Authorization, transport, configuration, hash, and fence failures do not permit recovery from another source.

JournalRebuild keeps the original source cache acknowledgment separate from the current Host facts. It records the saved checkpoint, upper position, ordered journal digest, and current resume attempt. The native cache state is PendingUntilAuthorizedTurn. The PendingUntilAuthorizedTurn state means that resume did not capture or prove a new native cache. The pinned SDK does not establish a supported client-filesystem checkpoint before a real turn. The next real authorized turn uses the existing native cache capture and Events write path. Suspend before genuine native files exist fails explicitly as runtime_sdk_cache_unavailable. The Host must not create placeholder files, send a synthetic turn, or report an empty cache as successful persistence. Replay and Host replacement use the same saved journal boundary, not a later journal. The shared digest validates returned references; authenticated producer reads must verify bytes and page completion.

An acknowledged delivery followed by a fresh Host Missing state produces an explicit state-loss outcome. A timeout or generic refusal does not authorize replacement. Core must revoke the exact old registration under current authority before issuing a replacement attempt. Each attempt has distinct bootstrap, consume, and exchange IDs under the same durable resume operation. Ordinary Runtime bindings omit the resume fields and keep their existing hashes. The Core replacement and Host journal-recovery paths require separate runtime acceptance. Missing or incomplete saved journal evidence cannot establish readiness. Lost Host state still requires the exact replacement proof.

The pure checks cover request, receipt, actor, authority, attempt identity, explicit loss proof, schema metadata, and controlled Kubernetes routing behavior. They do not prove PostgreSQL execution, native TLS, actual remount contents, Cilium traffic enforcement, or a deployed resume.

Ordinary root Host replacement

Ordinary Host replacement is separate from suspend/resume. The caller uses the run-bound replaceRuntimeHost and readRuntimeReplacement operations for the accepted root. The request names the expected runtime instance, registration revision, execution fence, accepted-selection hash, and one stable operation ID.

The Orchestrator saves the operation and cleanup obligations before provider effects. It rechecks current authority and blocks competing work only while that replacement is active. Environment retires the exact prior placement and provisions a new Pod and lease on the same workspace volume. Broker bootstraps the new runtime; Environment activates that exact registration. The operation keeps the accepted selection, execution fence, workspace volume, and workspace generations unchanged. Uncertain provider outcomes block a competing replacement until the owner reconciles the recorded attempt.

The new Host must consume the exact acknowledged SDK cache. The replacement receipt includes the full reconstruction interval, including SDK resume; the interval must show no new journal event, native model send, or tool effect. Orchestrator promotes readiness only after the current Host, registration, placement, and recovery evidence match. It accepts an exact prior root request on the new registration only when the durable completion and completed replacement lineage both validate. The replay does not send another model request.

The former native CLI native-material-recovery journey used these operations, but its acceptance runner has been removed. The current read-only API case does not exercise Host replacement or session-material recovery. See the testing guide for the current API case boundary.

Workspace, network, and readiness

Provisioning accepts a Workspace volume ID and exact resource/data generations, mount path, read-only choice, and Cilium policy generation. The volume must already be bound to the Environment. Environment checks the owner-scoped Workspace record, its current Storage selection and pinned provider/options, and the PVC namespace, UID, and owner-generation annotations. It reserves the exact volume generation as Attached through the Workspace owner CAS before provider dispatch. Replace and release remain blocked until the exact Sandbox placement is retired and Environment detaches that generation. The Sandbox/Workspace binding does not change Workspace retention or reclaim state.

The manager verifies the requested Cilium policy generation and object before Sandbox dispatch, then rechecks it after provider observation. The provider reports actual Sandbox/Pod, RuntimeClass handler, PVC attachment, and startup observations. Kubernetes Sandbox and container Ready conditions do not imply Environment or AgentHost readiness. ReadyForDispatch requires an explicit configured phase and is an Environment observation only; it is not Core admission or permission to execute a run.

The current provider emits scheduled, image ready, and started phases from Kubernetes observations. It does not emit configured or dispatch ready on its own. The separate AgentHost handshake adds authenticated configuration and current readiness evidence. Its internal readiness owner read retains the exact lease across fresh egress and Sandbox observations. It uses the retained selection snapshot, not a recursive accepted-selection lookup. It validates the stored recovery record and reads its nested provision request. That request supplies the exact network generation and Workspace mount path. Object readback does not prove Cilium datapath enforcement.

The internal bootstrap profile route also exposes a read-only /workspace suffix. It returns EnvironmentRuntimeWorkspaceContext under current run-read authority and the exact registered runtime profile. One owner transaction retains both the current Sandbox lease and the Workspace record during the Core context read. The record must remain Attached at the exact saved transition revision, volume generation, data generation, Storage resource, and provider attachment. The response includes the validated mount negotiation and transition revision, not provider options, credentials, or the PVC release descriptor. Data generation zero is valid metadata.

This read does not flush Storage or produce a content checkpoint. It supplies no tree hash and does not prove that a session suspended. Core still requires genuine native completion, cache, checkpoint, and Workspace content evidence before a successful suspend manifest.

For the proposed credential-less mode, admission must also prove that credential-consuming services are outside the untrusted guest, with no guest-accessible private memory, volumes, sockets or cloud assertions. The existing ISandboxProvider negotiated capability and lease/profile surfaces carry placement evidence; the selected L3/L4 and L7 providers must supply current forced-mediation evidence. Two containers in one Kata guest, a VM-isolation descriptor, or policy readback alone do not prove this boundary. These readiness additions are proposed, not current Sandbox capability or deployment evidence; legacy placement must not be relabeled credential-less.

Agent Sandbox resources and recovery

The adapter uses the extensions.agents.x-k8s.io/v1beta1 API. For each owner operation it creates or validates an owner-specific SandboxTemplate, a zero-replica SandboxWarmPool, and a SandboxClaim with its required warmPoolRef and DeleteForeground shutdown policy. The template disables service-token automount and environment injection, selects the configured VM RuntimeClass, mounts the exact Workspace PVC, and carries the verified Cilium selector labels. Ownership labels bind the tenant/project/run/Environment hash, Environment lifecycle generation, Sandbox resource generation, provider fence, and operation ID.

The optional typed AgentHost launch profile pins only a trusted ConfigMap name and TLS Secret name. The adapter adds fixed private state/temporary volumes, read-only configuration/TLS mounts, UID/GID/fsGroup 1000, and HTTPS health probes. Only this profile asks the Sandbox controller to create its private ClusterIP Service. The provider and runtime endpoint require HTTPS on TCP 8443 and verify that the Service selector resolves to exactly the actual AgentHost Pod; no shared AgentHost Deployment or Service is created. The deployment helper keeps the AgentHost platform digest, linux/amd64 value, and compressed byte count identical in the AgentHost ConfigMap and Environment Sandbox options. It checks the operator-owned TLS Secret type and the Environment ServiceAccount's effective permissions before writing; it does not create the Secret or apply RBAC. The proposed permissions are kept outside the active Kustomize bundle. See the Azure deployment guide for installer checks and recovery behavior. The Pod also mounts service-backend-ca read-only at /var/run/service-backend-ca for its internal owner HTTP clients. Environment's own bootstrap and sandbox HTTPS clients validate internal servers against the same CA. Their direct-Pod transport still verifies the TLS hostname while connecting to the validated Pod IP. That Pod has stable agentweaver.io/workload: agenthost and agentweaver.io/identity-client: "true" labels. The latter selects the existing Identity Broker network rule and grants no application authority. It uses the exact negotiated workspace as the SDK working directory. It validates the profile on both template and actual Pod readback. Absent profiles preserve the legacy template without native-host configuration. See the AgentHost image and launch boundary.

Before returning a resource, the adapter validates Kubernetes object names, UIDs, owner labels, the pinned template and pool references, the RuntimeClass handler, and the bound PVC's UID and Environment/volume-generation annotations. Describe validates the claim UID, Sandbox controller owner reference, Pod controller owner reference, PVC mount, Pod security settings, and the actual RuntimeClass. ListOwned reads the namespace from the lease's persisted provider-options snapshot rather than current defaults, then recovers only claims with the exact owner and fence; Environment further matches the recorded operation and resource generation.

The provision request is stored before dispatch. A crash after dispatch can be recovered through exact ListOwned results. A lease abandoned before any provider request was persisted can be completed as a known no-effect operation. An uncertain or failed lookup is never interpreted as absence.

If a different valid provider resource arrives after release, Environment records that exact result in the late-resource cleanup table instead of overwriting the terminal lease. Reconciliation discovers it under the current owner authorization, claims one item at a time, and records a durable release attempt before calling the provider with the original binding and fence. Claims expire so another reconciler can recover the work, but expiry does not prove the original provider call stopped. The original attempt's exact receipt may be accepted after expiry only while its claim token remains current; a newer claim supersedes it. Receipt persistence and attempt resolution happen under the owner lock. Matching duplicates remain idempotent after completion. Provider failures stay visible and leave cleanup available for retry.

BuildTest command and output collection

BuildTest is a separate command operation on the current Sandbox lease. The caller supplies a checkpoint reference and expected binding, not a command, image, argument list, or authority grant. Environment resolves the complete checkpoint with Core under current authorization. Core returns the immutable accepted command, execution options, selection hash, and stable operation ID. The manager records the intent and request fingerprint in the existing owner_effects row before provider effects. Attempts, Pod references, and results use the same operation. The store writes UTC timestamps at PostgreSQL microsecond precision so JSON and row timestamps agree after restart. A terminal snapshot is immutable except for its update timestamp and one verified version 2 journal acknowledgment. Identical saves and additional durable attempt markers cannot replace its status, Pod references, or evidence. An uncertain create remains ReconciliationRequired; it does not authorize another Pod. Command create, read, and reconcile responses use SandboxBuildTestOperationResult, which contains the operation and replay flag. Reconcile requests retain the same complete checkpoint and expected binding as the original command request. Core validates a read result before it reconciles; invalid owner response shapes cannot start another command.

Configure Environment:Sandbox:AgentSandbox:AcceptedBuildTestProfile to enable the BuildTest capability. The profile pins the executable allow list, image digests, resource limits, offline egress, and trusted collector command. An absent profile does not advertise the capability. An invalid profile fails configuration validation; the provider does not use a default profile. The active lease retains its accepted profile even if current server defaults change. BuildTest reads the same string-enum JSON format that Sandbox provisioning persists. It obtains the original mount request through the validated Sandbox recovery record, not a separate caller-supplied request.

Before each effect, Environment rechecks the owner, accepted command, Sandbox lease, Workspace generation, and Cilium binding. The command Pod uses the exact writable Workspace PVC and an operation/role-scoped offline deny-all policy. RuntimeClass admission can add its configured scheduling selectors to the Pod. The provider accepts only those exact entries plus kubernetes.io/os=linux and kubernetes.io/arch=amd64. Missing, malformed, duplicate, conflicting, or extra entries fail validation. Only verified terminal success with exit code zero can start the separate required-output collector. The collector uses the pinned image and executable with a read-only Workspace mount. AgentHost routes --build-test-output-collector-v1 before web configuration or native SDK startup. This Linux-only mode accepts one bounded encoded request. It opens accepted paths without following symlinks, checks regular files and mount identity, and returns a bounded receipt. Native directory and no-follow flags match the executing Linux architecture; unsupported architectures fail before collection. The stream matches the synchronous native file descriptor and keeps bounded cancellable reads. The receipt binds the collector Pod UID, container, operation, checkpoint, and request fingerprint. Missing required files or an invalid collector receipt are Failed. Bound collector timeout, cancellation, or output-limit termination is Interrupted, even after the command exits successfully. Logs are bounded command output, not file evidence.

BuildTest does not advance MAF checkpoints, turns, or run status. Core retains that state and its interruption acknowledgement. Disposable PostgreSQL tests verify owner-store reservation, restart replay, and terminal-evidence preservation. They do not prove command-Pod execution, Kubernetes isolation, Cilium datapath enforcement, or complete native accounting.

Version 2 repository preparation and full output

Version 2 binds a credential-free Git bundle and the full baseline inventory to the accepted command. The source binding includes the current actor, selection, repository pin, workspace incarnation, branch, and content digests. Authenticated MAF dispatch can provide buildTestWorkspaces, keyed by the confirmed BuildTest step ID. Each value contains only workspaceId, workspaceIncarnationId, baseSha, and branchName. These values locate an existing ready workspace; they do not grant access or select a repository. The existing Source Control owner resolves the current accepted run and repository pin. It opens that owned workspace and creates the bounded seed without cloning or redeeming repository credentials. The owner derives source metadata from the verified seed, current pin, actor, and accepted selection.

The confirmed workflow command remains unchanged. The MAF intent stores server-produced source metadata in a separate optional workspaceInput field. The effective version 2 execution command binds this input in its immutable hash. The exact confirmed-command comparison still applies. An explicit source in an already-confirmed command must match the verified input exactly. Version 1 requests and intents without input retain their existing JSON and hash behavior. The selected execution profile must already support the version 2 collector mode; dispatch does not upgrade it.

The original execution checkpoint stores the bounded seed in a separate PostgreSQL bytea field. Dispatch holds the existing execution lock and checks the current Source Control owner within the checkpoint transaction. It stores the command, input metadata, origin reference, and seed before returning or reserving an Environment operation. The owner rechecks the accepted selection, decision, and fence after preparation awaits. Retry reads the saved intent before seed creation and does not recapture a changed checkout. A conflicting workspace locator or changed owner is an error; it cannot replace saved input. Completed input can remain in a later accepted decision without changing its saved origin. Dispatch still checks the current source owner, selection, and fence. Running input cannot continue under a different decision version. Workspace preparation errors use the existing Source Control response and stop checkpoint creation. Later checkpoints retain only the typed origin reference. The JSON checkpoint limit remains 4 MiB. The seed limit remains 67,188,876 bytes.

Explicit seed reads require the current owner and fence. They validate the original intent, declared length, digests, and package before they return bytes. Ordinary checkpoint reads do not load the binary field. An identical append must match both the JSON metadata and the stored bytes. The parent foreign key prevents deletion of an origin that has later checkpoints. The seed is source input, not SDK cache data or a produced-output record.

Core's accepted-command route returns metadata only, including the bound seed digest and length. The separate /workspace-seed route reads the original binary field through the existing checkpoint store. It uses the same full checkpoint reference and current actor authorization. Core rechecks the accepted selection, owner decision, execution fence, and command after the read. Both routes also check the independently pinned Source Control owner for version 2 input. Both routes return Cache-Control: no-store. Version 1 commands have no seed route result.

Environment fetches the raw seed only from its fixed Core owner, not a caller-supplied address. The response must be application/octet-stream, use no-store, and match the accepted length and SHA-256 digest. Redirects, short or extra bytes, denied reads, and changed command metadata are errors. The client deadline covers both response headers and the complete bounded body. Environment rechecks Core metadata after the transfer and before reserving the command. Later current-command checks do not fetch the seed again. Stored operation updates, request fingerprints, and Core terminal receipts validate the bound metadata without seed bytes. Reconciliation restores the freshly verified seed only after it matches the stored command metadata. Version 2 still requires the actual validated bytes before execution; metadata alone is not executable input.

Source Control includes a full output-document foundation for version 2. Its AWSCAPT2 format retains the manifest, Git patch, and binary file package. The 24-byte header declares each section length; existing section limits bound the complete document. Parsing rejects invalid lengths, extra bytes, invalid file digests, and malformed packages. Serialization also checks the saved manifest and section digests.

The source can construct an output document from the accepted seed and a complete file inventory. It rejects missing baseline files, changed undeclared files, and outputs that differ from their declared evidence. It uses a private Git index and disables hooks, filters, external diff commands, and network protocols. The shared prepared-repository path retains the existing owner-scoped hash. The same Source Control package constructs the native output document. The capture does not create another output store or a second model runtime.

The Kubernetes client adds a bounded native WebSocket exchange for fixed preparation and collector containers. It retains the existing service-account handler and trusted server certificate. The helper must identify the exact Pod UID and request fingerprint before the client sends seed bytes. Local TLS tests cover binary transfers at the exact admitted limit and reject invalid identity, protocol, output, or terminal evidence. The provider uses this exchange to prepare the repository before command execution. The pinned AgentHost image contains Git and the shared Source Control package. Its trusted init container receives the exact accepted seed and calls the existing repository-preparation helper. The command starts only after the preparation receipt matches its operation, checkpoint, source digest, native path, and Pod UID. Its working directory is the prepared repository, not another local checkout. The command container cannot mount the helper's private state volume.

After verified command success, a separate read-only collector reads the entire prepared repository through native no-follow file descriptors. It excludes only the root Git metadata directory. Symlinks, special files, nested mounts, changed file identities, and incomplete trees prevent capture. The full package retains every baseline file and each declared output. Missing baseline files, changed undeclared files, and undeclared additional files are errors. Version 2 output obligations cannot exceed the existing 16 MiB package file limit. The complete-tree limits remain 10,000 files and 64 MiB total file content. A version 2 command requires this collector even when it declares no output files. The collector request records the actual accepted command version. Only explicit version 2 permits an empty output list; absent version fields retain strict version 1 behavior. The provider checks this version against the accepted command before any transport request. Workspace preparation and capture require valid version 2 source input. This field supplies no authority and does not change command hashes or receipt versions.

The collector seals the complete document and its exact reference in private Pod state. Both remain bound to the operation, source, request fingerprint, native repository path, and collector Pod UID. A restart or retry reuses those sealed bytes instead of recapturing the tree. A complete pending write is verified and promoted unchanged. An incomplete unsealed write can be retried only under the exact private-state binding, with an explicit diagnostic. Foreign or corrupt complete state remains an error. The native command is not restarted during capture recovery.

Environment persists preparation and capture references, never seed or file-package bytes in operation JSON. Core reads the exact binary document from Environment's fixed owner route only when Source Control has not retained it. The transfer requires no-store, the exact document length and digest, and current owner checks before and after the read. Core validates the accepted seed, full baseline, declared outputs, document sections, and native capture reference. It then uses the existing Source Control pending-package rows and Events produced-output admission path. Retry reconstructs the same document from retained pending or admitted bytes without a native export.

Environment receives the exact Events journal acknowledgment through its acknowledgment route. It asks Core to verify the canonical admitted record, object reference, positive journal position, and all capture bindings. Current command checks surround that verification. Environment saves the verified acknowledgment before requesting deletion of the exact collector and command Pod UIDs. Cleanup retries preserve the stored acknowledgment and do not execute or export the command again. Core's final owner transaction rereads the admitted capture and persists the preparation, full capture reference, and acknowledgment before completing the workflow step. Owner locks do not remain held across HTTP waits.

Controlled source tests cover full binary documents, baseline preservation, exact references, acknowledgment checks, and response-loss replay. They do not prove native Kubernetes execution, deployed PVC isolation, Cilium enforcement, or complete BuildTest acceptance.

Retirement and storage retention

An exact Agent Sandbox Finished=True condition with a supported PodSucceeded or PodFailed reason and a matching observed resource generation can retire Sandbox placement. Explicit abandonment requires current Projects authority and a durable CAS that increments the current fencing generation before release. The retired Sandbox cannot be revived or rebound.

Release rechecks the exact current retiring lease and fresh Projects authority, then uses the recorded provider binding and claim UID for foreground deletion. The adapter waits for the claim and its Sandbox/Pod children to be absent before deleting the owner-specific warm pool and template. The durable receipt must identify the exact provider resource and release idempotency key. KnownOwnedAbsent is returned only after an exact successful lookup confirms the claim and children are absent. Permission, transport, or ownership errors remain errors.

Retiring Sandbox placement does not transition the Environment to Released and does not release, delete, or erase its Workspace volume. Storage retention and cleanup continue through the independent Workspace lifecycle. There is no background abandoned-run reaper; Core run-terminal and supersede evidence must be added by its owning contract before automatic run-status reclamation is enabled.

Configuration

For Sandbox provider selection and Environment settings, see the provider configuration guide and runtime configuration reference.

The Environment service requires the Environment:Sandbox:AgentSandbox configuration section. ASP.NET environment variables use the corresponding double-underscore names, for example Environment__Sandbox__AgentSandbox__OptionsRevision.

Key	Purpose
OptionsSchemaVersion	Must be 2.
OptionsRevision	Bounded immutable revision selected for this adapter.
Namespace	Namespace for Sandbox objects; it must match the Cilium and Workspace PVC namespace.
WorkspaceStorageProviderId	Supported Storage adapter ID; currently azure-files-csi.
ContainerImage	Exact digest-pinned AgentHost image reference used by the Sandbox template.
ContainerImagePlatform	Must be linux/amd64 for the measured AgentHost image.
ContainerImageCompressedPullBytes	Positive compressed-pull-byte measurement for that exact image.
AgentHost	Optional typed launch profile naming the existing configuration ConfigMap and TLS Secret; configures the controller Service and native HTTPS listener.
RuntimeClassName	Kubernetes RuntimeClass selected for the Pod.
ExpectedRuntimeHandler	Handler value that must be read back from that RuntimeClass.
CpuRequest, MemoryRequest	Bounded Kubernetes resource quantities used as both requests and limits.
ReconciliationTimeoutSeconds, PollIntervalMilliseconds	Bounds for provider deletion reconciliation.

When Kubernetes:ApiServer is not set, Environment uses the kubelet-injected KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT address. This needs no DNS lookup, so the Cilium DNS rules do not have to allow the API server name. Outside a Pod, the default is https://kubernetes.default.svc/. Kubernetes:ServiceAccountTokenFile and Kubernetes:CertificateAuthorityFile default to the in-cluster service-account token and CA paths. The adapter requires trusted HTTPS and uses that service account for Kubernetes requests. Kubernetes RBAC and cluster configuration are separate operator prerequisites.

Test and evidence boundary

The focused commands and limits are listed in the Sandbox testing guide. The provider test and production-authority API integration use the production AgentSandboxProvider and KubernetesAgentSandboxClient against a fake Kubernetes HTTP API. The integration also uses real Broker, Projects, and Environment HTTP paths plus disposable PostgreSQL; only the Workspace provider and Cilium policy resource store are controlled. The fake Kubernetes boundary records object ownership and UID-preconditioned deletes. These tests do not prove RuntimeClass or nested-virtualization availability on AKS, datapath enforcement, AgentHost configuration, run pin acceptance by Core, or deployed service configuration.