Context-Lifecycle-Management-f-r-Agents/docs/architecture.md

CLM Architecture

Komponenten und Datenfluss des Context Lifecycle Management Systems.
Design-Grundlage: docs/superpowers/specs/2026-08-04-clm-design.md.

Neu hier? Praktischer Einstieg (offline, ~15–25 Min, ausführlich): getting-started.md.

---

Überblick

CLM ist ein framework-agnostischer Outer Control Loop über einem ausführenden Agenten. Es misst Context/Tokens, entscheidet zwischen CONTINUE, COMPACT und HANDOFF, schreibt Memory und erzeugt bei Handoff ein strukturiertes Manifest.

Drei Schichten: Outer Loop (Token/Entropy-Monitor, Compact-vs-Handoff-Entscheidung, Manifest-Generator), Orchestration Layer (Phasen/Artefakte, z. B. Traycer oder custom), Context Management Core (Long-Term Memory + Native/Verified Compaction) um die Working Agent Session.

Runtime-Datenfluss (Implementierung)

┌─────────────────────────────────────────────────────────────────┐
│                     OuterControlLoop                            │
│  run() / on_step_end()                                          │
└────────────┬────────────────────────────────────────────────────┘
             │
             ▼
      Agent.step(messages)
             │
             ▼
      MetricsMonitor.update(state, step)  →  TokenSnapshot
             │                              →  metrics.jsonl
             ▼
      DecisionEngine.decide(state, metrics)  →  Decision
             │                               →  decisions.jsonl
             │
     ┌───────┼───────────────────┐
     │       │                   │
     ▼       ▼                   ▼
 CONTINUE  COMPACT             HANDOFF
     │       │                   │
     │       ▼                   ▼
     │  CompactionPipeline    HandoffPipeline
     │       │                   │
     │       ├─ MemoryStore      ├─ HandoffManifest
     │       ├─ Compact-Hint     ├─ MemoryStore
     │       └─ shrink messages  ├─ SeedContext
     │                           └─ Session reset / stop
     ▼
  next step …

---

Vier Schichten

Schicht	Module (Ziel)	Verantwortung
Outer Control Loop	clm.loop.control, clm.decision.*, clm.metrics.*, clm.compaction.*, clm.handoff.*	Orchestrierung, Metriken, Entscheidung, Pipelines
Working Session	clm.session (core) + clm.agents.* (adapters)	Protocol/session state vs. vendor adapters; inject/reset
Composition	clm.runtime	Paths, config build, script load (shared by CLI + experiments)
Long-Term Memory	clm.memory.base, clm.memory.file_store, namespaced, backup	Persistente Keys/Inhalte; Multi-Agent-Namespace; Backup
Phase-Orchestration (P3)	clm signal, HITL confirm	PHASE_BOUNDARY, Confirm-Policy, Export/Calibrate
Production & Experiments (P4)	logging.structured, metrics.cost, experiments, decision.policy	Structured logs, Cost model, A/B Memory, Policy-Plugins

Runtime-Artefakte liegen unter .clm/ (sessions, memory, manifests, backups, exports).

---

Kernkomponenten

Agent (Protocol)

step(messages) → AgentStepResult
inject_context(system_addon)
reset_with_seed(seed_messages)

Protocol und optionale Capabilities (ExternalSessionCapable, LiveEventCapable) leben in clm.session (vendor-free). Konkrete Adapter implementieren das Protocol unter clm.agents (siehe unten). Factory-Registry (build_agent / register_agent) ist rein und importiert keine Vendor-Module; Builtins registrieren sich via import clm.agents → register_builtin_agents().

WorkingSession

Hält SessionState (goal, messages, tokens, step_index, blockers, …) und delegiert Steps an den Agent.

MetricsMonitor

• Akkumuliert Input-/Output-/Total-Tokens pro Step.
• Fallback-Schätzung: chars / 4 wenn Provider keine Counts liefert.
• Append-only Log: .clm/sessions/<id>/metrics.jsonl.

DecisionEngine

Regelbasiert (P1); optional LLM hinter Flag (nicht MVP-Must).

Priorität:

1. Phasengrenze + prefer_handoff_on_phase → HANDOFF
2. tokens >= hard_handoff → HANDOFF
3. tokens >= soft_compact und genug Steps seit Compact → COMPACT
4. Degradation-Signal (P1 stub) → COMPACT
5. sonst CONTINUE

Ausgabe: Decision(action, rationale, trigger).

CompactionPipeline (P2: verified)

1. Strukturierte Summary aus SessionState (kein LLM nötig).
2. Snapshot in Memory: sessions/{id}/compact-{step}.
3. Compact-Hint (Markdown): Goal, Constraints, Next action, Memory keys, Verified/Not verified.
4. VerificationSuite (deterministisch): Fragen aus goal / verified / constraints / blockers;
   RecallScore = passed / total gegen Summary+Hint; Default-Schwellwert 0.8.
5. Verify vor Message-Mutation:
   • Score ≥ threshold → Messages ersetzen (system-hint + retain_recent_messages), last_compact_step.
   • Score < threshold → Messages unangetastet; fell_back_to_handoff=True → Loop erzwingt HANDOFF (trigger=verify_fail).
6. Config: verify_compaction (default True), recall_score_threshold (default 0.8).

HandoffPipeline

1. HandoffManifest aus State + Decision bauen und strikt validieren.
2. Schreiben: .clm/manifests/{timestamp}_{session_id}.json (+ Kopie unter session handoffs).
3. Memory-Keys aktualisieren / relevante Einträge schreiben.
4. SeedContext: System-Message mit Manifest-Summary + Goal + next_safe_action + Memory-Hits (keine volle Transcript-Historie).
5. agent.reset_with_seed bzw. Loop-Stop (continue_after_handoff=False im MVP).

MemoryStore (Protocol)

write(key, content, *, tags=None)
read(key) → str | None
search(query, *, limit=10) → list[MemoryEntry]
list_keys() → list[str]

P2 Memory backends (ClmConfig.memory_backend / factory):

Backend	Klasse	Hinweis
file (default)	FileMemoryStore	.clm/memory/
mem0	Mem0Store	optional extra clm[mem0]; API-Key nötig
hybrid	HybridMemoryStore	write file+mem0; read prefer file

P4 Multi-Agent: ClmConfig.memory_namespace wrappt den Store als NamespacedMemoryStore (Key-Prefix). agent_id landet im Manifest (schema_version=1.1). Shared namespace = gemeinsame Keys über Agents ohne Framework-Annahme.

OuterControlLoop

run(max_steps=None) → LoopResult
on_step_end(step) → Decision

Bindet Monitor, Engine, Compaction, Handoff und Session zusammen.

P2 Loop-Verhalten:

• Nach Step: Decision auditiert mit step_index, total_tokens, ts (+ optional context_tokens).
• Terminal-Status (completed/failed) nach Step → keine Compact/Handoff-Pipeline.
• COMPACT mit Verify-Fail → Override-Decision trigger=verify_fail → HandoffPipeline.
• Token-Basis: default session-kumulativ; optional use_context_tokens_for_thresholds (Window-Größe).
• Live Progress: optionaler progress_callback am Loop (CLI default an, --quiet aus) — Events: run_start, step_start (vor Agent, inkl. max_turns), status (Heartbeat alle 20 s während Agent/Compact/Handoff; einzeilig + event-driven ticks), step (nach Decision: phase=done, turns/tools/notes/summary), compact_start/compact, handoff_start/handoff, confirm, run_end. Mit --max-steps: step=i/max und open=N (clm.loop.progress). Run-State: tokens.total= + soft=/hard= + compacts=/handoffs=; activity= (waiting_for_agent | agent_streaming | running_compaction | running_handoff); optional last_decision= / external_session=. Während Agent-Phase: doing= bevorzugt aus Live-Events (LiveEventCapable + optional injected describe_live_event; CLI wires clm.agents.grok.stream.describe_stream_event for grok-build — loop does not import grok); Fallback Workspace-mtime. goal=/cwd=/detail= nur auf step_start, nicht auf jedem Heartbeat.

P3 Loop- / CLI-Erweiterungen:

• HITL: require_confirm_on: ["compact"\|"handoff"] → vor Pipeline confirm_callback; Denied → CONTINUE, human_interventions++, Audit event=confirm / trigger=confirm_denied.
• Phase signal: clm signal --phase-done setzt SessionState.status=phase_boundary (ohne Pipeline); DecisionEngine → HANDOFF wenn prefer_handoff_on_phase.
• Calibrate: clm calibrate liest metrics.jsonl/decisions.jsonl, empfiehlt soft/hard (Perzentile + optional Decision-Blend).
• Export: clm export metrics --format csv|json|html → .clm/exports/<id>/.

P4 Loop- / CLI-Erweiterungen:

• Structured logging: clm.logging.structured (JSON lines; optional in Ops-Wiring).
• Cost model: provider + price_table / globale price_per_1k_*; MetricsMonitor loggt step_cost_usd / session_cost_usd.
• Memory backup: clm backup memory → .clm/backups/memory_*.tar.gz; clm backup restore --archive ….
• A/B Memory: clm experiment ab-memory --script … --backends file,mem0,hybrid → Report (completion, tokens, recall, interventions).
• Policy plugins: DecisionEngine(..., policies=[…]) — z. B. BlockersForceHandoffPolicy.

Working Session / Agent-Adapter

Session core (clm.session) is vendor-free: Agent protocol, capabilities, WorkingSession / TrackingWorkingSession, pure factory registry.

Adapters live in clm.agents (not under session):

Adapter	Package path	Zweck
SimulatedAgent	clm.agents.simulated	Deterministische Scripts; script_offset für Resume
GrokBuildAgent	clm.agents.grok	Live headless Grok Build; default streaming-json
OpenAICompatibleAgent	clm.agents.openai_compatible	PoC: /v1/chat/completions (stdlib HTTP, inject http_post für Tests)

Composition root (clm.cli) registers builtins with import clm.agents and, for grok-build, injects describe_stream_event as OuterControlLoop.describe_live_event. Loop/decision/compaction/handoff must not isinstance concrete adapters.

See docs/coding-agents.md and docs/modules/agents.md.

---

Datenfluss (Schritt)

1. Agent.step(messages)
       → AgentStepResult (messages_added, tokens, tool_calls, task_status, …)

2. MetricsMonitor.update(state, step)
       → TokenSnapshot
       → append metrics.jsonl

3. DecisionEngine.decide(state, metrics)
       → Decision { CONTINUE | COMPACT | HANDOFF }
       → append decisions.jsonl

4a. CONTINUE
       → nächster Step mit unverändertem Window (ggf. neue Messages)

4b. COMPACT
       → CompactionPipeline.run(state)  # summary → memory → verify → maybe mutate
       → if verify fail: Decision(verify_fail) → HandoffPipeline (wie 4c)
       → else: state.messages ← hint + recent tail; Agent.inject_context

4c. HANDOFF
       → HandoffPipeline.run(state, decision)
       → Manifest write + Memory updates
       → SeedContext bauen
       → Agent.reset_with_seed(seed) und/oder LoopResult(stopped_reason="handoff")

---

Artefakt-Layout (Runtime)

.clm/
├── config.yaml                 # optional local override
├── calibrations/<profile>.json # P3 calibrate --write
├── exports/<id>/               # P3 export metrics; P4 A/B reports
├── backups/                    # P4 memory_*.tar.gz
├── sessions/<id>/
│   ├── metrics.jsonl
│   ├── decisions.jsonl
│   ├── state.json
│   ├── compact-hints/
│   └── handoffs/
├── memory/
│   ├── MEMORY.md
│   ├── topics/
│   └── logs/
└── manifests/
    └── {timestamp}_{session_id}.json   # schema_version 1.1 + optional agent_id

---

Decision-Mapping (Code ↔ Spec)

Spec-Option	DecisionAction	Pipeline
A – Compact + Hint	compact	CompactionPipeline
B – Handoff + neue Session	handoff	HandoffPipeline
C – Weitermachen	continue	—

Schwellen und Task-Profile: siehe Design-Spec §6 und ClmConfig (Task 0.4).
P3: Adaptive Vorschläge via clm calibrate — manuelle Overrides bleiben führend.

---

Phasen-Grenzen (Architektur)

In P4	Bewusst out of scope / später
Structured logging, Manifest schema_version, Memory backup/restore	Live Mem0/OpenAI (weiter Mock/Unit)
Cost model pro Provider (price_table)	LLM-Summarizer als Default
A/B Memory experiment runner + Report	Dashboard-Framework
Multi-Agent: agent_id, memory_namespace, shared keys	Spezielles Multi-Agent-Framework
DecisionPolicy plugins (Beispiel-Policies)	Traycer vendor lock-in; ARC/ACM research hardcode

---

Verwandte Dokumente

• Design: docs/superpowers/specs/2026-08-04-clm-design.md
• Metriken: docs/metrics.md
• Manifest: docs/manifest-schema.md
• Plan: docs/superpowers/plans/2026-08-04-clm-mvp.md