Cluster page
The Cluster page gives operators a real-time view of the Kubernetes cluster backing the Agentweaver AKS deployment: sandbox capacity, Kubernetes health checks, and any legacy subtasks recorded as waiting for capacity.
Kubernetes owns scheduling (issue #217)
The platform no longer pre-gates on quota. It submits the SandboxClaim and waits for Kubernetes to schedule and bind the pod, so a Pending pod is an expected transient state, not a failure. The namespace ResourceQuota no longer caps CPU/memory (only object counts). The Pending capacity KPI, the Waiting for capacity badge, and the pending-capacity table are back-compat surfaces that only reflect historical runs.
It is available under the Cluster nav item in the SYSTEM section of the project left rail. Route: /projects/:projectId/cluster. The page auto-refreshes every 30 seconds by default; you can toggle Auto-refresh off when you want to inspect a static snapshot.
📸 Screenshot —
cluster-page.pngShows: the Cluster page with Orphaned, Pending capacity, Checks OK, and Warm pool KPI cards plus Health checks, Sandbox claims, orphaned pods, pending capacity, and warm pools. Path: open a project → click Cluster in the SYSTEM section of the left rail →/projects/:projectId/cluster.
When to use the Cluster page
Open the Cluster page when:
- a coordinator run shows subtasks in ⏳ Waiting for capacity (amber badge in the topology graph);
- runs are dispatching slowly and you suspect pod scheduling or node-pool autoscaling delays;
- you want to confirm that all Kubernetes API components are reachable;
- orphaned pods are accumulating (the reaper has not swept them yet);
- after a deployment or scaling event, to confirm the cluster is healthy.
KPI cards
The KPI cards at the top of the page summarize the cluster signals the current UI exposes:
| Card | What it shows |
|---|---|
| Orphaned | Agent pods that no longer match an active run. |
| Pending capacity | Legacy. Subtasks recorded in the historical PendingCapacity status; empty for new runs (Kubernetes now owns scheduling). |
| Checks OK | Healthy checks divided by all reported cluster checks. |
| Warm pool | Ready vs. desired warm sandbox replicas when warm-pool data is available. |
Below the KPIs, the page shows Health checks, Sandbox claims, Orphaned agent pods when present, Pending capacity, and Warm pools.
Component health table
Cluster checks run concurrently each time the page loads:
| Check | What it tests | Typical failure cause |
|---|---|---|
| Postgres | Connectivity to the Postgres database | Network policy, password rotation |
| Azure Key Vault | CSI delivery of the required mcp-api-key | Managed identity misconfiguration, network policy, or a missing API authentication secret |
| Agent pod quota | Effective admission headroom from the enforced pods and SandboxClaim object quotas. Healthy means plenty of room remains, warning means only a handful of starts remain, and critical means no new AgentHost can be admitted. | Namespace object-quota exhaustion |
| Warm pool | Warm-pool agent-sandbox availability for generic sandboxes (replicas: 3) and AgentHost (replicas: 2) | Warm-pool replica count below target, SandboxTemplate CRD issue |
| Kubernetes API | Kubernetes API server reachability | In-cluster network policy, apiserver overload |
Each check shows:
- A status badge such as
healthy,warning,degraded, orcritical. - A detail message (visible on warn/fail) explaining the specific failure.
- The duration the check took in milliseconds.
All five checks have a 5-second individual timeout. A timed-out check appears as fail with the detail "timed out".
If the Key Vault row shows critical: secret 'mcp-api-key' not found, restore the required API authentication secret with npm run azure:provision-infra before redeploying.
Active agent pods table
Lists pods currently running that have a matching active run record:
| Column | Meaning |
|---|---|
| Pod name | Kubernetes pod name |
| Run ID | The run the pod is serving (links to an orchestration detail when available) |
| Node | Kubernetes node the pod is scheduled on |
| Started at | When the pod was created |
A healthy system should show only pods with active runs here.
Orphaned agent pods table
Lists pods that are running but have no matching active run. These will be terminated on the next reaper sweep (default: every ~2 minutes).
If orphaned pods are not being cleaned up, check:
- That the heartbeat is enabled and ticking (see the Heartbeat page).
- That
Coordinator:ReaperIntervalTicksis not set to an unusually large value.
Pending-capacity runs table
Subtasks that could not get a sandbox immediately because the warm pool had no free capacity appear here until a slot frees up. Zero is healthy: it means every run got a sandbox right away.
Legacy / back-compat. Kubernetes now owns pod admission and scheduling (issue #217), so new runs rarely enter
PendingCapacity. This table stays in the UI mainly to render historical records; for a live run whose pod is still being scheduled, look forsandbox.provisioning_pendingheartbeats on the child run rather than an entry here.
Lists coordinator subtasks recorded in the historical PendingCapacity status:
| Column | Meaning |
|---|---|
| Coordinator run ID | The parent coordinator run |
| Subtask | The subtask that was waiting |
| Pending since | When the subtask entered PendingCapacity |
| Retry count | How many dispatch attempts were made under the removed park/retry loop |
Warm pools table
Lists every SandboxWarmPool CRD object in the namespace. Each row represents one pool:
| Column | Meaning |
|---|---|
| Name | Kubernetes name of the SandboxWarmPool object |
| Desired | Target number of pre-warmed sandboxes declared in the pool spec |
| Ready | Sandboxes currently ready to accept a claim |
| Available | Sandboxes that are ready and not yet claimed by a run |
| Status | healthy when ready equals desired; warning when below desired; critical when none are ready |
A pool in warning or critical means new run dispatches fall back to creating an ad-hoc sandbox, which adds latency to run startup.
Resource topology
The Resource topology graph now expands each warm pool to the individual warm-pool sandbox instances the API can currently see:
| State | Meaning |
|---|---|
| Available | An idle warm sandbox is ready to be claimed by the next run. |
| Claimed | A run currently owns the instance. When the backend can resolve both run and project, the node exposes a direct link to the orchestration detail page. |
| Warming | The sandbox pod exists but is not ready yet. |
This is the quickest way to answer “which warm spares are still idle?” and “which run is holding this exact warm-pool pod?”
Sandbox claims table
Lists all SandboxClaim CRD objects in the namespace:
| Column | Meaning |
|---|---|
| Name | Kubernetes name of the SandboxClaim object |
| Phase | bound (assigned to a sandbox), pending (waiting for one), or unknown |
| Ready | Whether the claimed sandbox is ready |
| Run | The run ID that created this claim, linking to an orchestration detail when present |
| Bound sandbox | The Sandbox object this claim is bound to; blank when still pending |
| Warm pool | The pool the bound sandbox came from; blank for ad-hoc claims |
| Age | How long the claim has existed |
404 fallback
When the API is not deployed on AKS, or the cluster diagnostics endpoint is unavailable, the page displays a message indicating that cluster diagnostics are not available for this deployment. No other page functionality is affected.
Related reading
- Operations — all operations surfaces at a glance.
- Cluster diagnostics reference — full API response schema.
- Sandbox pod execution — reaper design and Kubernetes-owned pod admission (
sandbox.provisioning_pending). - Heartbeat — the heartbeat that drives the reaper.