PRD — Agent Orchestration Harness

1 Executive summary

2 Scope — v1 vs v2 vs v3 Ship small, design for growth

v1 scope is everything needed to run Sentinel on a real client by Day 22. Everything else is explicit v2/v3.

3 Architecture overview

Design principles

1. Single process, synchronous for v1. No workers, no queues. One HTTP server handles trigger → session → delivery in one flow. Resilient enough for <10 runs per day. Simpler = fewer failure modes.
2. Agent-agnostic dispatcher. The dispatcher doesn't know what "Sentinel" does. It reads the agent's config file to determine which Managed Agent ID to invoke, which ClickUp fields to read from the triggering task, and which Slack/Drive channels to post to. Adding Catalyst later = new config file, zero code change.
3. Local state in SQLite. No external database for v1. SQLite file on VPS tracks run history + idempotency keys + cost. Upgrade to Postgres when we need multi-process concurrency.
4. Fail loud in Slack, not silent. Any unexpected error posts a formatted alert to #seo-automation-alerts with session ID, agent name, error, and last event. Never swallow exceptions.
5. Event log for post-mortems. Every SSE event persisted to ./logs/sessions/{session_id}.jsonl. Size is manageable (~MB per session). Kept 90 days.

4 Tech stack decisions Opinionated defaults — swap if Trung prefers

5 Agent config schema The core abstraction that makes this reusable

One JSON file per agent in ./agents/{agent_name}.json. The dispatcher reads this to know how to handle the agent's session.

6 Trigger sources How a run starts

Manual CLI v1

ClickUp webhook v1

ClickUp fires a webhook on task status change. Our endpoint receives it, validates, and triggers the run.

Handler responsibilities

1. Verify HMAC signature using shared secret. Return 401 on mismatch.
2. Idempotency check — has this task_id + status_change_id already been processed? If yes, return 200 immediately (don't re-run).
3. Resolve agent config — URL path tells us which agent (/webhook/sentinel → sentinel.json).
4. Fetch task details from ClickUp API. Extract required_fields from task custom fields. If any missing, post error to Slack, update task status to "Blocked", return 200.
5. Build kickoff payload, dispatch to session creation logic.
6. Return 200 to ClickUp within 3s (webhook timeout). The actual session runs asynchronously — we acknowledge fast, work slow.

Future triggers (v2)

Both follow the same dispatcher pattern — just different payload sources.

• Slack slash command /sentinel audit <client-name> — POST to /slack/command, Slack HMAC verification, lookup client by name, trigger run.
• Cron — systemd timer fires a binary that reads a schedule config (which agent, which clients, which frequency), dispatches runs. Used for monthly reporting.

7 Session lifecycle state machine The spine of the dispatcher

8 SSE event handling The event loop

Critical rules from the Managed Agents docs

1. Send the user event BEFORE opening the stream. The API buffers events until a stream attaches. If you open the stream first, you may miss the initial burst.
2. Always check session.status_idle events for stop_reason. end_turn = done. requires_action = session paused waiting on you.
3. Reconnect on stream drop. Network blip shouldn't kill a 30-minute session. On connection error, wait 2s and reopen the stream — Anthropic buffers events so you don't lose them.
4. Log every event. Post-mortem debugging is impossible without the event log.

Event types we care about in v1

Minimal handler implementation (TypeScript)

9 Output routing Where deliverables go

The flow

1. List session files: GET /v1/files?scope_id={session_id} with files-api-2025-04-14 beta.
2. Download each file via GET /v1/files/{id}/content. Save to ./tmp/{session_id}/.
3. For each enabled delivery channel in config.delivery, execute in parallel. Collect results.
4. Post a final summary to the main Slack channel with links to all deliverables.
5. Clean up tmp files.

Channel handlers (abbreviated)

Slack message format (design)

10 Error handling & retries

11 Idempotency

Idempotency key derivation

For webhooks: SHA256(agent_name + task_id + status_change_timestamp). Stored in idempotency_keys SQLite table with 24h TTL.

Flow

1. On webhook arrival, compute key.
2. INSERT OR IGNORE into table with key + session_id (or null if still creating).
3. If INSERT succeeded → new run, proceed. If IGNORED → duplicate, return 200 with {already_processed: true, original_session: ...}.
4. After session creation succeeds, UPDATE row with session_id.

12 Logging & monitoring

Three log streams

1. Application log (pino) → stdout → journalctl. Structured JSON. Level: info in prod, debug if DEBUG=1.
2. Session event log → ./logs/sessions/{session_id}.jsonl. Raw SSE events. 90-day retention, cron cleans up older.
3. Run history → SQLite runs table: session_id, agent, trigger_source, payload_hash, started_at, ended_at, status, total_cost, error.

Monitoring — day-1 minimum

• Process alive: systemd auto-restart on crash. Uptime Kuma or equivalent hitting a /health endpoint every 5min.
• Disk space: cron checks session log dir, Slack alert if >80% full.
• Failed runs: cron queries runs table hourly, Slack summary of any ERROR status.
• Cost anomalies: daily cron sums runs from SQLite, Slack alert if daily total >1.5× baseline.

Monitoring — later (v2+)

Grafana + Prometheus feels excessive for a 1-service VPS. Revisit when we have >3 agents + high volume.

13 HITL gate handler Stub for v2

Design for when we add it

When the agent fires agent.tool_use with a tool that requires confirmation (configured in the Anthropic agent definition), the session emits session.status_idle with stop_reason: requires_action and requires_action.event_ids[] listing the blocking events.

Our dispatcher's REQUIRES_ACTION state handler will:

1. Post a Slack interactive message with Approve / Deny buttons and the tool call context.
2. Store pending session state in SQLite.
3. Return control (release the stream connection).
4. When human responds via Slack interaction, POST user.tool_confirmation event with result: allow or deny + optional deny_message.
5. Reopen stream, resume STREAMING state.

For v1: stub that logs unexpected requires_action + Slack alerts. Full implementation in Sprint 2.

14 VPS setup & deployment

15 Project structure

16 Environment variables

17 Test plan

Unit tests (Vitest)

• Config loader: valid config parses, missing required field throws, unknown agent returns null.
• Idempotency: first call inserts, duplicate call returns existing.
• Payload resolver: builds payload from mock ClickUp task, handles missing fields.
• Template rendering: {payload_json} and {client_id} substitute correctly.
• Cost sampler: computes correct USD from token counts.

Integration tests (run against Anthropic)

• OT1: Manual CLI trigger with synthetic payload → session creates, event stream flows, at least one agent.message received, session reaches end_turn. Uses a minimal throwaway test agent to avoid Sentinel cost.
• OT2: Same but with bad SENTINEL_AGENT_ID → errors gracefully, Slack alert fires, no session record leaked.
• OT3: Simulate stream drop (kill connection mid-session) → reconnect happens, events don't repeat in log, session completes.
• OT4: Webhook endpoint with valid HMAC → returns 200 within 3s, session starts in background.
• OT5: Webhook endpoint with invalid HMAC → returns 401, no session created.
• OT6: Idempotency: fire same webhook 2x within 1s → second returns already_processed: true, no duplicate session.
• OT7: Delivery routing: mock session complete with fake output files → Slack post, ClickUp comment, Drive upload all succeed. Verify content manually on first run.

End-to-end (alongside Sentinel T1)

• Full Sentinel run via orchestrator CLI with synthetic client.
• Full Sentinel run via ClickUp webhook on a test task.

18 Definition of Done

19 Open questions

Things I don't know for certain — verify before committing.

1. ClickUp webhook HMAC algorithm and header name. I've assumed X-Signature with HMAC-SHA256 but Trung should verify from ClickUp API docs. Blocker for: webhook security. Low risk — just look it up.
2. Session reconnect behavior after long disconnect. If the VPS reboots mid-session, can we reconnect to the stream on recovery? Or is the session lost? Docs suggest events persist server-side. Worth verifying with a deliberate test. Not blocking v1 — rare edge case.
3. Concurrent session limit per org. Rate limits state 60/min for create endpoints, but is there a cap on concurrent running sessions? We'd only hit it with high volume, but worth knowing. Ask Anthropic sales.
4. ClickUp API rate limits when reading task details. Free tier is stricter. Verify our tier allows enough reads. Non-blocker — upgrade plan if needed.
5. How Google Drive service account permissions propagate to shared drives. If client folders live in a shared drive, the SA needs explicit membership. Verify first client's folder setup before assuming this works. Non-blocker — standard Drive setup question.
6. Slack bot token scopes needed. chat:write and files:write at minimum. Interactive message support needs interactivity in app config (v2 only).

20 Risks & mitigations

21 Appendix — reference code

A1. package.json (minimal)

A2. SQLite schema

A3. Health endpoint