Sub-Agent Control Patterns

About This Series
This document is Volume 2 of The Physics of AI Engineering, a series dedicated to understanding the fundamental constraints of Large Language Models and deriving production-grade architectural patterns from first principles. Volume 1 established the three fundamental laws governing LLM behavior in production. This volume applies those laws to the specific and complex challenge of multi-agent orchestration, going beyond architecture into the deeper questions of cognitive role design, boundary discipline, and deterministic signal propagation.

This document is the second volume in The Physics of AI Engineering series. Volume 1 established three fundamental laws governing LLM behavior: The Law of Finite Attention, The Law of Stochastic Accumulation, and The Law of Entropic Expansion.

This volume extends those laws to a specific and underexplored architectural challenge: How do you build multi-agent systems that remain cognitively coherent, reliably bounded, and deterministically repairable at production scale?

We present a complete production engineering journey through four architectural phases: (1) Autonomous Sub-Agents (full autonomy, 38% failure rate in production), (2) Controlled Sub-Agents (max_turns=1, 91% success rate), (3) Hybrid Sub-Agents (delegated autonomy with semantic yield, 93–95% success rate), and (4) Cognitively Hardened Systems (typed boundaries, dual-loop architecture, epistemic contracts, 99.2% success rate).

Beyond the autonomy architecture question, this volume introduces five new engineering concepts discovered through deep production operation: Cognitive Fragmentation (when agent roles blur), Boundary Entropy (when agent scope creeps), The Dual-Loop Architecture (fast-twitch self-correction vs. slow-twitch research), Typed Boundaries (why string matching failure classification is an anti-pattern), and Epistemic Output Contracts (how agents must commit to structured findings).

Each concept is grounded in the same LLM physics established in Volume 1. None of these failures are prompt engineering problems. They are architectural failures with architectural solutions.

Keywords: Sub-Agent Architecture, Cognitive Cohesion, Boundary Entropy, Dual-Loop Architecture, Typed Boundaries, Epistemic Contracts, REPAIR-ROUTER Pattern, Partial Success, LLM Physics, Enterprise Automation, Deterministic Repair

Prerequisites: Familiarity with Volume 1 (The Physics of AI Engineering: A Deep Science Masterclass) recommended. Key concepts are re-introduced as needed.

Preface: Building on Foundations

Volume 1 of this series established the physical laws that govern LLM behavior in isolation: a single model, a single context window, a single reasoning chain. Those laws—Finite Attention, Stochastic Accumulation, and Entropic Expansion—are necessary but not sufficient to build production multi-agent systems.

When you introduce multiple agents, each with their own context, tools, and decision loops, new failure modes emerge that have no analogue in single-agent systems. An agent can behave perfectly in isolation but catastrophically when coordinated with peers. A context can be clean within one agent and poisoned by the time it arrives at another. A failure can be correctly classified in one boundary and completely misrouted in another.

This volume is about those emergent challenges. It answers a question that Volume 1 could only hint at: How much autonomy should each agent have, and what governs the boundaries between them?

We present four architectural generations, each discovered through production failure and refined through principled engineering:

• Architecture A (Autonomous): Full sub-agent autonomy $\rightarrow$ 38% failure rate
• Architecture B (Controlled): Single-turn execution $\rightarrow$ 91% success rate
• Architecture C (Hybrid): Delegated autonomy with semantic yield $\rightarrow$ 93–95% success rate
• Architecture C+ (Cognitively Hardened): Typed boundaries + cognitive cohesion $\rightarrow$ 99.2% success rate

But the most important contribution of this volume is not the autonomy architecture spectrum. It is the five cognitive engineering patterns discovered after Architecture C was deployed—patterns that address not just how agents are controlled, but how they think and communicate. These patterns (Sections 7–12) represent the frontier of production AI engineering as of this writing.

Introduction: The Multi-Agent Coordination Problem

Why Single-Agent Physics Is Not Enough

Volume 1 presented a precise model of how a single LLM agent fails. Given a context window $C$ of length $L$, recall probability $P_{\text{recall}}(p, L)$ decays as a function of token position. Given $N$ sequential decisions each with error probability $p$, the success probability $P_{\text{success}} = (1-p)^N$ decays exponentially. Given a context growth rate $k$ tokens per turn and a fixed capacity $C_{\text{max}}$, overflow occurs at $t^* = (C_{\text{max}} - C_0)/k$.

These laws are real, measurable, and non-negotiable. But they describe a system with one LLM, one context, one decision maker.

Enterprise workflows require specialization. A workflow that authenticates with an API, fetches data, generates a transformation function, executes it, validates the output, and produces a report requires at least five distinct cognitive profiles: identity reasoning, data modeling, code generation, execution analysis, and synthesis. No single system prompt can optimize simultaneously for all five without severe trade-offs.

When you specialize via sub-agents, however, a new problem emerges: coordination. Each agent now has its own context, its own decision loop, and its own failure modes. The critical question becomes: who coordinates them, how much autonomy does each get, and how do failures cross boundaries?

This document answers those questions with precision.

The Coordination Failure Surface

In a single-agent system, failures come from one source: the LLM’s stochastic output. In a multi-agent system, failures can originate from five distinct sources, each requiring different mitigation:

1. Intra-agent failures: The LLM within a single agent makes a wrong decision (covered by Laws 1–3 from Volume 1)
2. Role confusion failures: An agent takes actions outside its cognitive domain, creating overlap and redundancy
3. Boundary entropy failures: The interface between agents becomes semantically ambiguous over time
4. Signal degradation failures: A correctly-typed failure in one agent becomes misclassified or lost as it crosses to another
5. Partial success failures: A multi-resource operation returns mixed results that look like success but aren’t

Sources 1–2 are addressed by Architectures B and C. Sources 3–5 require the cognitive engineering patterns in Sections 7–12. Most production failures at scale originate from sources 3, 4, and 5—they are the hardest to detect because they do not manifest as obvious errors.

How LLMs Behave in Multi-Agent Settings

Revisiting the Three Laws Under Coordination

The three fundamental laws from Volume 1 behave differently in multi-agent settings:

Law 1 (Finite Attention) in Multi-Agent Systems

In a single agent, attention decay affects one context. In a multi-agent system, each agent has its own context with its own U-shaped attention curve. But there is a critical compound effect: information discovered in Agent A's context at high recall must survive serialization, transmission, and re-injection into Agent B's context. At each boundary, the information may be truncated, summarized, or reordered—potentially moving it from a high-recall zone to a low-recall zone.

$$P_{\text{recall}}^{\text{cross-agent}} = P_{\text{recall}}^A \cdot P_{\text{serialization}} \cdot P_{\text{recall}}^B$$

If Agent A recalls a critical variable with 90% probability, the serialization layer drops 15% of context to fit budget, and Agent B's context places it in the middle (50% recall), the compound probability is:

$$P_{\text{recall}}^{\text{cross-agent}} = 0.90 \times 0.85 \times 0.50 = 0.38$$

Figure: Left: The U-shaped attention recall curve within a single agent context. Tokens at the primacy (start) and recency (end) positions have the highest recall probability; the middle zone drops to $\approx$50%. Right: When a critical variable crosses an agent boundary, its effective recall probability is the product of three independent factors—dropping from 90% within a single agent to 38% across agents. This is why structured artifact storage (Cognitive Blackboard) is a physical necessity, not an optimization.

A variable with 90% within-agent recall has only 38% cross-agent recall. This is why the Cognitive Blackboard pattern (Section 13.4) is not an optimization—it is a physical necessity. Critical structured artifacts must never cross agent boundaries via context injection; they must travel through deterministic key-value storage.

Law 2 (Stochastic Accumulation) in Multi-Agent Systems

In single-agent systems, errors compound within one decision chain. In multi-agent systems, error chains span agents. If Agent K (Kernel) makes a misclassification ($p_K = 0.05$), routes to Agent R (Researcher) which makes a research error ($p_R = 0.05$), routes back to Agent K which routes to Agent C (Coder) which generates incorrect code ($p_C = 0.05$), routes to Agent E (Executor) which silently accepts a partial result ($p_E = 0.05$)—the compound failure probability across four agents is:

$$P_{\text{failure}} = 1 - (1 - p_K)(1 - p_R)(1 - p_C)(1 - p_E) = 1 - 0.95^4 = 0.185$$

An 18.5% per-workflow failure rate—before we account for any multi-turn compounding within each agent. This is why boundary validation is not optional. Every agent boundary is an error injection point.

Law 3 (Entropic Expansion) in Multi-Agent Systems

Context entropy accumulates at the agent level and at the system level. Each agent manages its own context, but the Kernel accumulates summaries from every sub-agent invocation. After $N$ sub-agent calls with average trajectory size $T_{\text{avg}}$, the Kernel's entropy contribution from sub-agent outputs is:

$$E_{\text{kernel}} = \sum_{i=1}^{N} T_i \cdot (1 - r_i)$$

Where $r_i$ is the compression ratio applied to the $i$-th trajectory before injection into Kernel context. Without trajectory compression ($r_i = 0$), the Kernel context grows linearly and overflows. With properly tuned compression ($r_i \approx 0.90$), the growth becomes sub-linear, enabling workflows of hundreds of steps.

The Autoregressive Nature and Its Multi-Agent Implications

The LLM's autoregressive generation means it has no internal representation of system state—it only has what's in its context. In a multi-agent system, this creates a subtle and dangerous failure mode: an agent can confidently describe a system state that no longer exists.

Consider this scenario: The Kernel routes to Agent R (Researcher) to investigate an API. Agent R returns findings. The Kernel then routes to Agent C (Coder) using a context snapshot that includes Agent R's findings. But Agent C's context is assembled 30 seconds later, after additional Kernel turns. If those additional turns generated 4,000 tokens, Agent R's findings may now be in the middle-zone of Agent C's context window, subject to 50% recall probability.

Agent C generates code based on a mix of correctly-recalled and hallucinated API specifications. When execution fails, the system has no way to attribute the failure to attention decay at the boundary—it looks like Agent C made an error, when in reality the Kernel's context assembly lost critical signal.

---

Architecture A: Full Autonomy (The Seduction)

The Mental Model and Why It Fails

Architecture A treats sub-agents as specialist employees: assign them a goal, trust their expertise, let them work until done. The Kernel delegates with a goal. The sub-agent loops internally for as many turns as needed. The Kernel receives a final result.

This model is seductive because it mirrors how we think about human specialists. When you hire an expert, you don't approve every keystroke. You delegate the outcome and trust the process.

The analogy fails at a fundamental level. Human specialists are goal-directed reasoners with persistent memory, meta-cognition, and the ability to recognize when their approach isn't working. LLMs are statistical pattern completers. They do not want to solve your problem. They predict the next token that statistically follows the sequence they have seen, drawing from patterns in training data. This is not a limitation that better prompts can overcome—it is the mathematical definition of how autoregressive generation works.

The Four Failure Modes of Autonomous Loops

Failure Mode 1: Recursive Divergence

A sub-agent encounters an error. Rather than escalating, it autonomously attempts repairs. Each repair attempt generates additional error context. The context fills with error noise. The signal (original goal) is progressively buried in the middle zone of the context window where recall probability drops to 50%. By turn 15, the agent is optimizing to “fix this specific error” rather than “accomplish the mission.” The two objectives have diverged.

Mathematically: at turn 15 with 500 tokens per turn, the original mission description (first 1,000 tokens) is now at position 1,000 in a context of $1{,}000 + 15 \times 500 = 8{,}500$ tokens—at position 12%, still in the high-recall zone. But if each turn generated a 2,000-token stack trace, the context is $1{,}000 + 15 \times 2{,}000 = 31{,}000$ tokens, and the mission is at position 3.2%—recoverably in the primacy zone. The dangerous middle zone is reached when accumulated errors grow relative to mission tokens.

Failure Mode 2: Hallucination Cascades

When the LLM lacks information, it does not admit ignorance—it hallucinates plausible-sounding completions. In an autonomous loop, these hallucinations become “facts” in subsequent turns because the model's own output becomes its input:

1. Turn 3: Agent needs auth token. Hallucinates: “Try Authorization: Bearer API_KEY”
2. Turn 4: That fails. Hallucinates: “Maybe the endpoint is /v2/resource”
3. Turn 5: Still fails. Hallucinates: “This system uses OAuth2 client credentials flow”
4. Turn 6: Now the code is entirely fictional—optimized for a hallucinated API

Each hallucination reinforces the next because the model sees its own previous generation as high-confidence context. This is the poisoned well pattern described in Volume 1, but in an autonomous loop, the poisoning is self-inflicted and accelerating.

Failure Mode 3: Context Window Collapse

With unbounded turns, context fills with noise:

Context composition decay in a 15-turn autonomous loop. By turn 15, the original mission occupies 3% of context—buried in the 50% recall zone, effectively invisible to the attention mechanism.

Failure Mode 4: The Observability Horizon

The Kernel sent a command at $t=0$ and received a result at $t=87\text{s}$. Between those two timestamps, 18 turns of internal decision-making occurred—invisible to the Kernel. When the result was wrong, there was no way to reconstruct why without reconstructing the entire internal trajectory from fragmented logs. This is an open-loop controller: it sends a command and hopes for the best, with no intermediate feedback.

The Mathematics of Autonomous Loop Failure

For a sub-agent with per-turn alignment probability $p$ running for $N$ turns:

$$P_{\text{success}}(N, p) = p^N$$

For $p = 0.95$ (5% per-turn error rate at $\text{temperature}=0.2$):

$$\begin{align} P_{\text{success}}(5, 0.95) &= 0.95^5 = 0.77 \quad (23\% \text{ failure}) \\ P_{\text{success}}(10, 0.95) &= 0.95^{10} = 0.60 \quad (40\% \text{ failure}) \\ P_{\text{success}}(20, 0.95) &= 0.95^{20} = 0.36 \quad (64\% \text{ failure}) \end{align}$$

At 20 turns—a common autonomous loop length—the agent is more likely to have failed than to have succeeded. This is not a prompt engineering problem. It is a direct consequence of stochastic accumulation.

Architecture B: Controlled Sub-Agents (The Correction)

The Fundamental Insight: Centralize Control, Distribute Intelligence

The diagnostic insight from Architecture A's failures is precise: intelligence was distributed and so was control. The sub-agents were capable (they could write code, search APIs, handle auth), but they were also autonomous (they decided when to stop, what to retry, how to handle every failure).

Architecture B separates these concerns surgically:

• Intelligence (how to write code, how to search documentation, how to structure an API call) $\rightarrow$ lives in specialized sub-agents
• Control (what to do next, when to stop, which agent to call, whether to pivot strategies) $\rightarrow$ lives exclusively in the Kernel

The mechanism is blunt but effective: max_turns=1. Each sub-agent executes exactly one cognitive action and returns immediately. The Kernel observes the result, makes the next decision, and delegates again.

Why max_turns=1 Respects the Three Laws

Breaking the Compounding Chain

With max_turns=1, the alignment probability per Kernel cycle is:

$$P_{\text{aligned}} = p^1 = 0.95$$

If the Kernel makes a bad decision, it is corrected in the next cycle—one turn later. The error accumulator resets at every Kernel-sub-agent boundary. Instead of $0.95^{20} = 0.36$, we maintain $0.95$ at every step.

Maintaining Clean Context

Every sub-agent invocation receives a freshly constructed context from the Kernel. The Kernel's context manager builds this context as a priority stack:

• Top 30% (high-recall zone): Mission statement, original goal, immutable constraints
• Middle 35% (compressed): Semantic summary of workflow progress (not raw logs)
• Bottom 25% (high-recall via recency bias): Current task, last result, available tools
• Remaining 10%: Metadata, version info, injected variables

Critically: the sub-agent does not see its own previous attempts. The Kernel manages what each agent sees, filtering noise before it can accumulate. This is not just a UX convenience—it is active entropy management.

Enabling Closed-Loop Control

Every sub-agent action is now part of a feedback loop with period = 1 turn:

When the Coder writes bad code, the Kernel sees it in 1 turn. When execution fails with a specific error class, the Kernel can route to the correct specialist immediately. When the Researcher returns insufficient findings, the Kernel can refine the query. There are no black boxes.

The Kernel's closed-loop OODA cycle in Architecture B. The critical property is the feedback period: one Kernel turn. Errors are corrected within one cycle; there is no accumulation across multiple unobserved sub-agent decisions. The dashed arrow represents the typed result returned by the sub-agent, which triggers the next Observe phase.

The Kernel's OODA Loop

The Kernel implements a physics-aware OODA loop with five phases:

1. Observe (Law of Finite Attention): Build context as priority stack; measure tokens before LLM call; place critical data in high-recall zones
2. Budget Check (Law of Entropic Expansion): If $\text{tokens}(context) > 0.9 \cdot B_{\text{usable}}$, compress or evict before proceeding—never silently overflow
3. Orient: Inject the previous sub-agent result and any P0-priority signals (critical errors go at context top, not buried in history)
4. Decide (Law of Stochastic Accumulation): Ask for exactly one next action—not a plan, not a strategy, one action—to minimize chain length
5. Checkpoint: Persist full state to durable storage after every cycle, not just at step completion

Production Results

Production metrics comparison. The controlled architecture achieves higher reliability and lower cost by eliminating the token-expensive compounding loops of Architecture A.

The cost reduction is counterintuitive. Architecture B makes more LLM calls—one per micro-decision. But Architecture A's autonomous loops frequently spiraled to 15–25 turns before returning, burning 4× more tokens on a failed attempt. Architecture B's strict control eliminates this waste.

What Architecture B Cannot Do

Architecture B achieves 91% reliability but creates a new inefficiency: Kernel micromanagement. Every trivial error that a specialist could self-correct—a syntax error, a minor JSON formatting issue, an obvious edge case—requires a full Kernel round-trip:

Architecture B micromanagement: every trivial error costs a full Kernel LLM call

Kernel Turn 1: "Coder, write function"
  -> Coder returns: code with IndentationError on line 12

Kernel Turn 2: "Coder, fix IndentationError on line 12"
  -> Coder returns: corrected code

Kernel Turn 3: "Executor, run the function"
  -> Executor returns: success

The indentation fix consumed one full Kernel LLM call. At scale (100+ workflows/hour), this overhead is significant. More importantly, it reveals a conceptual gap: not all errors require Kernel-level decision-making. Some errors are within the sub-agent's cognitive competency—they just need more turns.

Architecture C addresses this.

Architecture C: Hybrid Autonomy (The Evolution)

The Competency Boundary Concept

Architecture C introduces a formal concept that Architecture B lacked: the competency boundary. Not all errors are equal. They divide into two fundamentally different classes:

Competency boundary classification. Skill issues are fixable by the specialist without global context. State and infrastructure issues require Kernel-level coordination.

The Semantic Yield Protocol

Architecture C introduces a constitutional guardrail: sub-agents can loop up to $N$ turns (default $N=5$), but must yield immediately upon detecting a blocker they cannot fix within their competency:

Semantic Yield Protocol: hardcoded Python that fires unconditionally, regardless of LLM intent

def check_yield_condition(tool_result: dict, tool_name: str) -> Optional[str]:
    """
    Deterministic yield check. This is NOT LLM reasoning---it is
    hardcoded logic that fires regardless of what the LLM 'thinks'.
    Running after every tool call ensures blockers are caught
    before they compound into turn 2 of the sub-agent's lease.
    """
    result_str = str(tool_result).lower()
    
    # Authentication failures: require Identity agent, not code fixes
    if any(s in result_str for s in ["401", "403", "unauthorized", "forbidden"]):
        return "auth_blocker: authentication failure detected"
    
    # Infrastructure failures: require human/ops, not code fixes
    if any(s in result_str for s in ["connectionrefused", "connection refused",
                                      "timeout", "dns resolution"]):
        return "infra_blocker: infrastructure failure detected"
    
    # Endpoint errors: require correct API spec, not code iteration
    if "405" in result_str or "method not allowed" in result_str:
        return "spec_blocker: wrong HTTP method for endpoint"
    
    # Competency limit: agent tried max self-correction attempts
    # (this is set by the lease mechanism, not the LLM)
    return None  # Within competency zone, continue

def execute_sub_agent_turn(state, tool_fn, **kwargs):
    result = tool_fn(**kwargs)
    
    yield_reason = check_yield_condition(result, tool_fn.__name__)
    if yield_reason:
        # Immediate yield---do NOT complete remaining lease turns
        state.status = "yield"
        state.yield_reason = yield_reason
        return result, True   # (result, should_yield)
    
    return result, False

The yield check is deterministic Python code, not LLM reasoning. It fires unconditionally after every tool execution. The sub-agent's LLM cannot override it.

The Three Mechanisms

1. The Lease (Bounded Autonomy)

The Kernel grants sub-agents a lease: temporary decision-making authority for up to $N$ turns. The lease cannot be extended by the sub-agent. This is the architectural embodiment of “controlled delegation”:

$$P_{\text{aligned}}(N=5, p=0.95) = 0.95^5 = 0.77$$

77% alignment at the skill-correction level. But for state/infrastructure blockers, the yield protocol fires at turn 1, giving $P_{\text{aligned}} = 0.95^1 = 0.95$. The weighted reliability:

$$P_{\text{total}} = P_{\text{skill issue}} \cdot 0.77 + P_{\text{state issue}} \cdot 0.95 \cdot P_{\text{yield correct}} \approx 0.93$$

This explains the 93–95% success rate in production.

2. Trajectory Tracking (The Glass Box)

Architecture A was a black box. Architecture B was fully transparent but generated one event per action. Architecture C reintroduces limited sub-agent loops but maintains transparency through trajectory recording:

$$\text{trajectory} = [(t_1, \texttt{tool}_1, S_1, E_1[:50]), \ldots, (t_N, \texttt{tool}_N, S_N, E_N[:50])]$$

The trajectory is compressed before injection into Kernel context:

• Raw trajectory (6 turns): 2,340 characters of JSON
• Compressed: “T1(write) ✓ → T2(test) × SyntaxError:12 → T3(fix) ✓ → T4(test) × 401 Auth → YIELD”
• Compressed size: 187 characters (92% reduction)

The compression is lossy (exact parameters are discarded) but semantically sufficient: the Kernel can understand what happened and why without reconstructing the full decision tree.

3. Semantic Result Analysis

Exit code 0 does not mean mission success. A Python script can execute without exceptions while the API it called returned 401 Unauthorized. Architecture C implements recursive semantic analysis after every execution:

Semantic result analysis: mission success requires more than a clean exit code

def analyze_semantic_success(result: dict) -> Optional[str]:
    """
    Recursively inspect result dictionaries for semantic failure indicators.
    Python exit codes are process-level signals; this is mission-level signal.
    """
    if not isinstance(result, dict):
        return None
    
    FAILURE_SIGNALS = [
        "401 unauthorized", "403 forbidden",
        "connection refused", "ssl certificate verification failed",
        "resource already exists", "function not found in scope",
    ]
    
    def search(obj, depth=0):
        if depth > 10: return None
        if isinstance(obj, str):
            lower = obj.lower()
            for signal in FAILURE_SIGNALS:
                if signal in lower:
                    return signal
        elif isinstance(obj, dict):
            for v in obj.values():
                found = search(v, depth + 1)
                if found: return found
        elif isinstance(obj, list):
            for item in obj:
                found = search(item, depth + 1)
                if found: return found
        return None
    
    return search(result)

This prevents “semantic blindness”—the failure mode where the system reports success because Python didn't crash, ignoring the fact that the target system rejected every request.

Cognitive Cohesion: The Role Contract Theory

The Problem Architecture C Does Not Solve

Architectures B and C address the question of how many turns each agent should get. They do not address the question of what each agent is allowed to do. In production, this becomes the next failure category: agents that are controlled in turn count but cognitively confused about their role.

Consider a Coder agent in a hybrid system that has been given a lease to write and test code. On turn 2 of its lease, the code produces a 404 error. The Coder:

1. Detects the 404
2. Does not yield (404 is ambiguous—sometimes a code error, sometimes a spec error)
3. Decides to “research” the correct endpoint by calling the API documentation tool
4. Spends turns 3 and 4 doing research
5. Produces code on turn 5 that uses a hallucinated endpoint

The yield protocol was not violated. The turn count was respected. But the Coder spent 40% of its cognitive budget on research—a task it is not optimized for—producing lower quality output than a dedicated Researcher would. More critically, it consumed its lease on the wrong tasks.

This is role confusion: the agent overstepped its cognitive domain not by breaking rules but by filling a vacuum.

The Role Contract

Role Contract specifications. Each agent is granted precisely the tools needed for its domain and no more.

Cognitive Fragmentation: The Pathology

The transformer's self-attention mechanism allocates weight proportionally to relevance signals in the context. A prompt optimized for research generates attention patterns tuned to information retrieval. A prompt optimized for coding generates attention patterns tuned to syntactic precision. When a single prompt asks for both, the attention weight is divided and neither task receives full depth.

Enforcing Role Contracts at the Kernel Level

The Role Contract is not enforced by trusting the LLM to stay in its lane. It is enforced architecturally: the Kernel assembles each sub-agent's context to include only the tools, artifacts, and instructions relevant to that agent's epistemic domain. An agent that never receives a web_search tool cannot decide to "do some research."

Role-contract-aware context assembly: the Kernel is the enforcer

ROLE_CONTRACTS = {
    "researcher": RoleContract(
        authorized_tools=["web_search", "read_docs", "http_probe"],
        relevant_artifact_types=["api_spec", "endpoint_discovery"],
        mandate_template=(
            "Your only job is to find the correct API endpoint, parameters,"
            " and authentication scheme for: {specific_task}."
            " Return structured findings only. Do not write code."
        ),
    ),
    "coder": RoleContract(
        authorized_tools=["write_file", "read_file", "lint", "typecheck"],
        relevant_artifact_types=["api_spec", "execution_result"],
        mandate_template=(
            "Your only job is to write a correct Python function using"
            " the API specification already provided. Task: {specific_task}."
            " Do not research endpoints. Do not execute code."
        ),
    ),
    "executor": RoleContract(
        authorized_tools=["execute_python", "run_tests"],
        relevant_artifact_types=["generated_code"],
        mandate_template=(
            "Your only job is to run the provided function and return a"
            " typed ExecutionSignal reflecting the semantic result."
            " Do not modify code. Do not research."
        ),
    ),
}

def build_agent_context(agent_role, workflow_state, task):
    contract = ROLE_CONTRACTS[agent_role]
    # Only inject tools authorized for this role -- others are absent, not forbidden
    available_tools = [t for t in ALL_TOOLS
                       if t.name in contract.authorized_tools]
    # Only inject artifacts from this role's epistemic domain
    relevant_artifacts = [
        a for a in workflow_state.artifacts
        if a.artifact_type in contract.relevant_artifact_types
    ]
    mandate = contract.mandate_template.format(specific_task=task.description)
    return AgentContext(
        primacy_zone=[mandate, task],              # immutable goal
        mid_zone=[summarize(relevant_artifacts)],  # compressed support
        recency_zone=[workflow_state.latest_action],
        available_tools=available_tools,           # role-scoped only
    )

The critical design choice: unauthorized tools are not listed as forbidden—they are absent from the tool list. An agent cannot use a tool it does not know exists. Cognitive cohesion is structural, not behavioral.

Boundary Entropy: Why Boundaries Degrade Over Time

The Entropy Principle at Architectural Scale

Volume 1 introduced Entropic Expansion—the tendency of context to fill with noise. The same principle operates architecturally. Agent boundaries are not static; they require active maintenance. Without it, they expand, blur, and collapse.

The quality degradation from boundary entropy has a precise mathematical form. Let $B_i^{\text{core}}$ be the agent's core domain workload, and $B_i(t)$ be its total cognitive load at time $t$:

$$Q_i(t) = Q_i^0 \cdot \left(\frac{B_i^{\text{core}}}{B_i(t)}\right)^{\alpha_i}$$

For $\alpha_i = 0.7$ (high domain separability) and $B_i(t) = 2 \cdot B_i^{\text{core}}$ (doubled scope after 3 months): $Q_i = 0.50^{0.7} \cdot Q_i^0 = 0.61 \cdot Q_i^0$—a 39% quality degradation that accrued invisibly over time.

The Dual-Loop Architecture

Two Failure Classes, Two Recovery Paths

Architecture C with Cognitive Cohesion reaches 93–95% reliability. The remaining failures share a characteristic: all are routed to the Researcher regardless of whether research is actually required. Empirical analysis of production failures reveals:

Routing all failures through the Researcher adds 60–240 seconds of latency for 60% of cases that do not need it.

The Dual-Loop Architecture. The Failure Classifier routes failures deterministically based on typed ExecutionFailure.category. Coder-class failures (syntax, logic, format) enter the Fast-Twitch Loop—the Coder self-corrects without Researcher involvement. Spec-class failures (wrong endpoint, missing resource) enter the Slow-Twitch Loop—the Researcher discovers the correct specification before the Coder re-generates. The dashed arrows show the return path to the Kernel upon recovery.

The Fast-Twitch Loop (Coder-Class Failures)

The Coder is granted a short self-correction lease (1–2 turns) with the specific error injected at context top. Characteristics:

• The API spec in context is sufficient—the Coder has what it needs
• The failure is a knowledge-application error, not a missing-knowledge error
• No Researcher invocation; no new API calls; 15–45 second recovery

The Slow-Twitch Loop (Researcher-Class Failures)

When the spec itself is wrong, the Coder cannot fix the problem by iterating on the same knowledge. Characteristics:

• The current spec is incorrect—writing better code from a bad spec produces better-looking wrong code
• Requires a Researcher invocation to discover the correct endpoint or parameter set
• Full Kernel cycle: Kernel $\rightarrow$ Researcher $\rightarrow$ Kernel $\rightarrow$ Coder $\rightarrow$ Executor
• 90–300 second recovery—the cost is justified because the problem is genuinely hard

Dual-loop architecture production improvements.

Typed Boundaries and Epistemic Output Contracts

Why "Done" Is Not a Signal

When an agent returns “API calls failed with 400 error, tried alternative endpoint, still failing,” the Kernel must:

1. Parse the natural language to extract failure type
2. Determine recovery path
3. Decide which agent to invoke next

Each decision is an LLM inference—stochastic, potentially wrong. But the failure type was already known at the boundary where failure occurred. The information existed in structured form and was de-structured into prose before crossing the boundary.

ExecutionFailure: the immutable typed signal that crosses every agent boundary. frozen=True prevents any downstream agent from silently modifying the classification.

from dataclasses import dataclass
from typing import Literal, Optional

ExecutionCategory = Literal[
    "auth_error",        # Requires Identity agent
    "endpoint_unknown",  # Requires Researcher: spec is wrong
    "not_found",         # Requires Researcher: resource missing
    "rate_limit",        # Requires Kernel: backoff
    "network",           # Requires Kernel: retry
    "provider_down",     # Requires Kernel: failover
    "code_syntax",       # Requires Coder: Fast-Twitch
    "function_mismatch", # Requires Coder: Fast-Twitch
    "format_error",      # Requires Coder: Fast-Twitch
    "duplicate",         # Requires Kernel: idempotency
    "unknown",           # Requires human: escalate
]

@dataclass(frozen=True)
class ExecutionFailure:
    category: ExecutionCategory
    raw_message: str
    retryable: bool
    execution_status: Optional[str] = None
    
    def routes_to_fast_twitch(self) -> bool:
        return self.category in (
            "code_syntax", "function_mismatch", "format_error")
    
    def routes_to_slow_twitch(self) -> bool:
        return self.category in (
            "endpoint_unknown", "not_found", "auth_error")

<Figure src="/papers/sub-agent-control-patterns/fig-5.svg" caption="Figure 4" />

The Typed Signal Propagation Rule

Typed failure signals must not be destroyed at any boundary. The propagation chain:

1. Executor detects partial_error in API response
2. Executor sets: execution_status = "FAILED_INCOMPLETE_FETCH" in result dict
3. Executor calls done with the full result dict (not just the error string)
4. Classifier reads execution_status and produces ExecutionFailure(category="endpoint_unknown")
5. REPAIR-ROUTER reads category and routes to Slow-Twitch

If step 3 strips the result dict and passes only the error string, the typed signal is lost. The classifier falls back to string matching, which is brittle and vendor-coupled.

The done() boundary: the most common place where typed signals are silently destroyed.

# ANTI-PATTERN: strips execution_status from the payload
done_params = {
    "output": error if not is_success else output,  # loses schema
}

# CORRECT: full result dict preserved
done_params = {
    "output": output if not is_success else (error if error else output),
    # When is_success=False, 'output' contains execution_status,
    # error message, and execution_failure_schema.
    # The full structure is essential for deterministic routing.
}

Why String Pattern Matching Is an Anti-Pattern

Classifying failures by matching error messages against known strings has four fatal flaws:

1. Vendor coupling: Your classifier contains implementation details of external systems. Vendor updates silently break your routing.
2. Incompleteness: You cannot enumerate all expressions of “unsupported operation.” You play perpetual catch-up.
3. Ambiguity: A partial string match may trigger on an unrelated error.
4. Maintenance debt: The list grows with every integration and never shrinks.

The correct pattern: the execution layer detects failure semantically and emits a typed constant:

Typed signal map: O(1) dictionary lookup replaces brittle string matching. Every new integration adds one entry here, nowhere else.

TYPED_SIGNAL_MAP = {
    "FAILED_INCOMPLETE_FETCH":     "endpoint_unknown",
    "FAILED_SILENT_PARTIAL_FETCH": "endpoint_unknown",
    "FAILED_AUTH_EXPIRED":         "auth_error",
    "FAILED_RATE_LIMITED":         "rate_limit",
}

exec_status = payload.get("execution_status")
if exec_status in TYPED_SIGNAL_MAP:
    return ExecutionFailure(
        category=TYPED_SIGNAL_MAP[exec_status],
        raw_message=payload.get("error", ""),
        retryable=True,
    )
# String matching runs only for legacy / unknown failures

The Deterministic REPAIR-ROUTER Pattern

repair-router: a pure function with zero stochastic behavior. Same ExecutionFailure.category always produces the same RepairDecision. The LLM is not consulted.

REPAIR_ROUTING_TABLE = {
    "code_syntax":       ("fast_twitch", "delegate_coding",    True),
    "function_mismatch": ("fast_twitch", "delegate_coding",    True),
    "format_error":      ("fast_twitch", "delegate_coding",    True),
    "endpoint_unknown":  ("slow_twitch", "delegate_research",  True),
    "not_found":         ("slow_twitch", "delegate_research",  True),
    "auth_error":        ("kernel",      "delegate_identity",  True),
    "rate_limit":        ("kernel",      "backoff_retry",      True),
    "network":           ("kernel",      "retry_with_jitter",  True),
    "provider_down":     ("kernel",      "failover_provider",  False),
    "unknown":           ("escalate",    "alert_human",        False),
}

def repair_router(failure: ExecutionFailure,
                  attempt_count: int,
                  max_attempts: int) -> RepairDecision:
    if attempt_count >= max_attempts:
        return RepairDecision(action="abort", should_retry=False)
    
    loop, action, retryable = REPAIR_ROUTING_TABLE.get(
        failure.category, ("escalate", "alert_human", False))
    
    if not failure.retryable or not retryable:
        return RepairDecision(action="abort", should_retry=False)
    
    return RepairDecision(
        action=action, loop=loop, should_retry=True,
        reason=f"Routing {failure.category} to {action} via {loop} loop",
    )

The REPAIR-ROUTER combined with P0 Injection (placing the routing decision at the top of the Kernel's next context window) ensures the Kernel LLM reasons within the repair constraint, not against it. The Kernel cannot “decide” to do something different—the repair directive occupies the primacy zone where recall probability is highest.

Partial Success: The Hidden Failure Mode

Why Partial Success Is Worse Than Total Failure

Total failure is obvious. A workflow that crashes is investigated. Partial success is treacherous: a workflow that processes 80% of targets, silently fails on 20%, and returns success=true will pass every automated check. Downstream systems act on incomplete data believing it complete.

Form 1: Silent Partial Fetch

The API returns HTTP 200 with an empty result set because the operation is unsupported but the server returns empty rather than erroring. Detection requires semantic validation: an empty patient list when patients are known to exist is not a success.

Form 2: Partial Error (Mixed Results)

Some resources succeed (HTTP 200) while others fail (HTTP 400/404). Without explicit typed signaling, the majority's success masks the minority's failure.

Form 3: Schema Partial Success

HTTP 200 returns a response body missing critical fields required for downstream processing. Process success but mission failure.

Detection and Typed Signaling

Semantic success evaluation: five distinct success layers, any of which can fail independently. Process success (exit code 0) is necessary but not sufficient.

def evaluate_execution_result(result: dict) -> ExecutionSignal:
    fn_status = result.get("status", "")
    fn_data = result.get("data", {})
    
    # Form 1: Silent empty result
    if expected_non_empty_resources(fn_data) and all_empty(fn_data):
        return ExecutionSignal(
            execution_status="FAILED_SILENT_PARTIAL_FETCH",
            success=False,
            error=(
                "All resource fetches returned empty. API likely does not "
                "support the requested operation pattern. Researcher must "
                "identify the correct endpoint variant."
            )
        )
    
    # Form 2: Mixed success
    if fn_status in ("partial_error", "partial_success"):
        failed = extract_failed_resources(result)
        return ExecutionSignal(
            execution_status="FAILED_INCOMPLETE_FETCH",
            success=False,
            error=(
                f"Function self-reported status='{fn_status}'. "
                f"Failed resources: {failed}. "
                "Coder must fix the API call pattern for failing resources: "
                "add required query parameters, use a different endpoint "
                "variant, or extract data from already-fetched resources."
            )
        )
    
    # Form 3: Schema check
    schema_errors = validate_response_schema(fn_data, REQUIRED_FIELDS)
    if schema_errors:
        return ExecutionSignal(
            execution_status="FAILED_SCHEMA_INCOMPLETE",
            success=False,
            error=f"Response missing required fields: {schema_errors}"
        )
    
    return ExecutionSignal(execution_status="SUCCESS", success=True)

The Partial Success detection chain. Python's exit code 0 is a process-level signal that provides no information about mission success. The semantic evaluator inspects the actual data returned—resource counts, status fields, schema completeness—and emits typed signals (FAILED_SILENT_PARTIAL_FETCH or FAILED_INCOMPLETE_FETCH) that the classifier maps directly to endpoint_unknown, triggering the Slow-Twitch research loop. No string matching. No ambiguity.

Both FAILED_SILENT_PARTIAL_FETCH and FAILED_INCOMPLETE_FETCH map to category="endpoint_unknown" in the classifier. The distinction lies in the diagnostic message:

• Silent fetch: the entire endpoint variant is wrong (Researcher finds a different resource path)
• Incomplete fetch: the endpoint works for some resources but not others (Researcher finds resource-specific query parameters)

Production Hardening Patterns

The Cognitive Blackboard: Shared Persistent State

The most critical production hardening pattern is the Cognitive Blackboard: a shared, persistent, structured store that all agents read from and write to, never passing critical artifacts through context injection alone.

The blackboard is not a message queue. It is a structured key-value store with semantic schema:

Caption: Cognitive Blackboard: structured persistent store that eliminates the 38% cross-agent recall problem by replacing context injection with deterministic key-value reads

class CognitiveBlackboard:
    """
    Shared persistent state for all agents in a workflow.
    
    Design contract:
    - Agents READ findings from the blackboard, not from context injection
    - Agents WRITE structured artifacts to the blackboard after completion
    - The Kernel reads only metadata summaries; agents read full artifacts
    - All writes are versioned and timestamped; nothing is silently overwritten
    """
    
    def write_artifact(
        self,
        artifact_type: str,        # "api_spec", "generated_code", "execution_result"
        artifact_id: str,          # Stable identifier for this artifact
        content: dict,             # Fully typed, validated content
        agent_source: str,         # Which agent produced this
        workflow_id: str,          # Correlation ID
        version: int = 1,          # Increments on each update
    ) -> None: ...
    
    def read_artifact(
        self,
        artifact_type: str,
        artifact_id: str,
        workflow_id: str,
    ) -> Optional[dict]: ...
    
    def list_artifacts(
        self,
        workflow_id: str,
        artifact_type: Optional[str] = None,
    ) -> List[ArtifactSummary]: ...
        # Returns {artifact_id, type, version, timestamp, agent_source}
        # NOT the full content -- keeps Kernel context lean

The blackboard pattern directly addresses the cross-agent attention decay problem derived earlier:

$$P_{\text{recall}}^{\text{cross-agent}} = 0.90 \times 0.85 \times 0.50 = 0.38$$

By reading from the blackboard rather than from context injection, the recall probability for a critical artifact is $P_{\text{read}} = 1.0$ (deterministic lookup, not stochastic recall). The 38% recall probability is replaced by 100%.

State Persistence After Every Cycle

The fundamental principle from Volume 1's Law of Stochastic Accumulation: state is sacred. In multi-agent systems, state persistence must be synchronous with every Kernel OODA cycle. Not eventually consistent. Not batched. Synchronous.

Caption: Synchronous checkpointing: persist-before-act ordering ensures any crash is recoverable from the last known-good state, never from zero

async def kernel_ooda_cycle(state: WorkflowState) -> WorkflowState:
    """
    Explicit checkpointing after every cycle.
    If the process crashes between cycles, recovery starts from
    the last persisted checkpoint---never from zero.
    """
    # Phase 1: Observe
    result = await observe_last_action(state)
    
    # Phase 2: Orient
    updated_state = orient(state, result)
    
    # Phase 3: Decide (single LLM call, temperature=0)
    decision = await decide(updated_state)
    
    # Phase 4: PERSIST BEFORE ACT
    # If we crash after persisting but before acting, we re-execute
    # the same deterministic action next restart. Safe.
    # If we crash after acting but before persisting, we re-execute
    # on restart -- all actions must be idempotent.
    await persist_checkpoint(updated_state, decision)
    
    # Phase 5: Act
    await act(decision)
    
    return updated_state

The persist-before-act ordering combined with idempotent actions (using idempotency keys on all API calls) means that a crash at any point during the cycle is safely recoverable.

Token Budget Monitoring

Context budget enforcement must be proactive, not reactive. A buffer alarm at 90% capacity gives 10% headroom to complete the current action and compress before the next:

Caption: Proactive token budget management

class TokenBudgetMonitor:
    def __init__(self, max_tokens: int, buffer_pct: float = 0.10):
        self.max_tokens = max_tokens
        self.alarm_threshold = int(max_tokens * (1.0 - buffer_pct))  # 90%
        self.critical_threshold = int(max_tokens * 0.95)             # 95%
    
    def check(self, context: KernelContext) -> BudgetStatus:
        used = context.token_count()
        
        if used >= self.critical_threshold:
            return BudgetStatus(
                level="CRITICAL",
                action="compress_immediately",
                message=f"Context at {used}/{self.max_tokens} -- MUST compress"
            )
        elif used >= self.alarm_threshold:
            return BudgetStatus(
                level="WARNING",
                action="compress_after_current_action",
                message=f"Context at {used}/{self.max_tokens} -- plan compression"
            )
        return BudgetStatus(level="OK", action=None)
    
    def compress(self, context: KernelContext, 
                 target_pct: float = 0.70) -> KernelContext:
        """
        Compress context to target_pct of max capacity.
        Compression strategy:
        1. Keep first 15% (primacy zone) -- never compress
        2. Keep last 15% (recency zone) -- never compress
        3. Compress middle 70% to semantic summaries
        4. Preserve all structured artifacts in blackboard, not context
        """
        target_tokens = int(self.max_tokens * target_pct)
        return context.compress_to(target_tokens)

The Priority Stack at Context Assembly Time

Every context assembly must respect the physics of attention. The priority stack is not a metaphor---it is a concrete engineering constraint on where each piece of information is placed in the context:

Caption: Physics-aware context assembly: information is placed, not appended. Where it sits in the window determines whether the LLM will recall it at decision time.

def assemble_kernel_context(
    mission: str,           # The original workflow goal (never changes)
    p0_directive: str,      # Current mandatory action (changes each cycle)
    progress_summary: str,  # Compressed history (managed for size)
    current_action: str,    # What the Kernel must do now
    token_budget: int,
) -> str:
    """
    Priority stack construction respecting the U-shaped attention curve.
    
    Recall probability distribution:
    - Tokens 0-15%:    ~0.95 recall (primacy)
    - Tokens 15-85%:   ~0.50 recall (middle zone)  
    - Tokens 85-100%:  ~0.90 recall (recency)
    
    Strategy: Put critical-but-stable info in primacy zone.
              Put critical-but-dynamic info in recency zone.
              Compress all history into the low-recall middle zone.
    """
    primacy_zone = [
        f"MISSION (IMMUTABLE): {mission}",
        f"MANDATORY NEXT ACTION: {p0_directive}",
        "---",
    ]
    
    recency_zone = [
        "---",
        f"CURRENT TASK: {current_action}",
    ]
    
    # Middle zone gets the history -- recall is 50% but that's acceptable
    # for historical context (we need it available, not perfectly recalled)
    middle_zone = [progress_summary]
    
    # Pack into token budget, compressing middle zone if needed
    return pack_with_budget(
        zones=[primacy_zone, middle_zone, recency_zone],
        token_budget=token_budget,
        compressible_zone=1  # Only compress middle zone
    )

Caption: The Context Priority Stack. Critical, stable information (mission, P0 directives) is placed in the primacy zone where recall probability is $\approx$0.95. Dynamic current-task information is placed in the recency zone (also $\approx$0.90 recall). Historical context, which must be available but not perfectly recalled, occupies the middle zone at $\approx$0.50 recall. Token budget alarm thresholds trigger compression before overflow occurs.

Multi-Turn Reasoning Quality

The quality of a multi-turn chain depends critically on the quality of context handoffs between turns. Three patterns degrade multi-turn reasoning quality:

1. Raw log injection: Passing full tool output (stack traces, API responses) into context instead of semantic summaries. Fills context with low-density noise.
2. Redundant goal restatement: Repeating the original mission at every turn in full prose. The mission should occupy the primacy zone once, not be re-injected at every step.
3. Unconstrained temperature: Using temperature $> 0.2$ for Kernel routing decisions introduces unnecessary stochasticity into control flow. Temperature should be proportional to the desired creativity: routing decisions want temperature=0; exploratory reasoning wants temperature=0.3--0.5.

Common Architectural Mistakes

Mistake 1: Prompting Your Way to Production

The most prevalent mistake in production AI engineering is treating architectural problems as prompt engineering problems. When a sub-agent fails to follow instructions, the instinct is to add more instructions. When an agent role-blurs, the instinct is to add a clarifying sentence to the system prompt.

These interventions work in demos. They fail at scale because they are fighting the physics:

• Adding instructions to address cognitive fragmentation increases prompt token count, which dilutes attention further—the opposite of the intended effect
• Adding error handling instructions assumes the agent will recall them in the exact moment the error occurs—a 50% recall probability in the middle zone
• Prompt instructions cannot override the compounding probability law: $p^N$ decays regardless of what the prompts say

Mistake 2: Treating Agent Boundaries as Interfaces

In software engineering, interfaces define how systems communicate. In AI agent systems, boundaries are more than interfaces—they are entropy barriers. Every boundary is an opportunity for structured information to degrade into unstructured prose.

The discipline required at every agent boundary:

• Agent outputs must conform to a typed schema (not “return a string describing what happened”)
• Critical structured data must be written to the blackboard, not injected into context
• Failure signals must propagate as typed objects, not natural language descriptions

Mistake 3: Conflating Process Success with Mission Success

Python exit code 0 means the process completed without exceptions. It does not mean:

• The API calls returned the data you needed
• The data returned was complete
• The data returned was correct
• The downstream system accepted the data

Every layer of execution success evaluation must be explicit:

1. Process success: Python exit code 0
2. Protocol success: HTTP 200 for all requests
3. Semantic success: All resources returned with expected cardinality
4. Schema success: All required fields present with valid values
5. Mission success: The output satisfies the original workflow goal

Only when all five layers pass is the operation truly successful.

Mistake 4: Building Monolithic Kernel Prompts

As the system evolves, the Kernel accumulates routing logic, domain knowledge, error handling, and workflow templates in its system prompt. This violates the Cognitive Cohesion principle at the Kernel level. A Kernel whose prompt contains 40% routing logic, 30% domain knowledge, and 30% workflow coordination is a fragmented agent—and it is the most critical agent in the system.

The Kernel's epistemic domain is coordination and control: which agent to invoke next, how to interpret results, when to escalate. Domain knowledge (what an API does, how authentication works, what a patient record looks like) belongs in the specialized agents. Routing logic belongs in deterministic code (REPAIR-ROUTER), not in the prompt.

Mistake 5: Ignoring the Observability Gap

A system that cannot be observed cannot be debugged. Two observability requirements are non-negotiable in production:

1. Structured, correlated logs: Every agent action must be logged with workflow_id, agent_id, action_type, tool_used, input_summary, output_summary, and duration_ms. Plain text logs without structured fields cannot be queried programmatically.
2. Failure traceability: When a failure occurs, it must be possible to reconstruct the complete decision chain from the logs: which agent made which decision, what context it had, and what it returned. This requires logging the semantic summary of context at each decision point, not just the outcome.

The Complete Architecture Synthesis

From Architecture A to Architecture C+

The progression from Architecture A to the fully hardened Architecture C+ represents four distinct generations of production learning. Each generation fixed a specific failure class:

Caption: Architecture evolution: each generation applies a targeted fix to a specific failure class. The final 0.8% failure rate consists of genuinely novel failure classes not yet encountered in production.

The Six Engineering Principles Unified

The complete set of production-grade principles derived across Volumes 1 and 2:

1. Determinism at Boundaries: Every agent boundary is an entropy injection point. Structured data crosses boundaries as typed objects, never prose. Failure signals are typed constants, never string patterns.
2. Cognitive Cohesion: Each agent has exactly one epistemic domain. Tool scope, context access, and system prompt content must all reflect this domain exclusively. Cross-domain thinking is an architectural smell.
3. State is Sacred: Every cycle persists state before acting. Recovery is always possible from the last checkpoint. Nothing is held only in context.
4. Observe Before You Decide: Closed-loop control requires every action to produce a structured observation. Black-box sub-agents are observability debt.
5. Validate Everything: Process success, protocol success, semantic success, schema success, and mission success are five distinct evaluation layers. None implies the others.
6. Entropy is the Enemy: Context entropy grows by default. Every system must have active entropy management: token budget monitors, context compression, blackboard-based artifact storage, and trajectory compression.

The Physics Connection

All six principles derive from the three fundamental laws of Volume 1:

Quick Reference

Architecture Selection Guide

The Typed Signal Checklist

Before deploying a new agent-to-agent boundary, verify:

• The agent returns a typed dataclass, not a string or dict
• All failure states map to a typed category from the finite taxonomy
• The full result object (including execution_status) is preserved at every boundary
• No failure classification relies on vendor-specific string patterns
• The REPAIR-ROUTER handles every category in the taxonomy

The Role Contract Checklist

Before deploying a new agent, verify:

• The agent's epistemic domain is stated in one sentence
• The agent's authorized tool list contains only tools needed for that domain
• The agent's yield conditions cover all state/infrastructure blockers
• The agent's system prompt is $> 60\%$ devoted to its core mandate
• No tool in the agent's list also appears in another agent's exclusive domain

The Dual-Loop Decision Tree

Context Assembly Quick Reference

Conclusion: The Physics Are Non-Negotiable

This volume has presented a production engineering journey through the multi-agent coordination problem. The central thesis is identical to Volume 1: the failures of LLM-based systems are not random bugs to be patched but expressions of fundamental physical laws to be respected.

Autonomous loops fail because stochastic accumulation is a law, not a tendency. Role confusion degrades quality because cognitive attention is finite and zero-sum. Boundary entropy causes architectural rot because entropy without active management always increases. Partial success hides failures because “success” was never defined with sufficient precision.

The engineering solutions to these problems follow directly from their physics:

• Against stochastic accumulation: closed-loop control with single-turn delegation
• Against finite attention dilution: cognitive cohesion enforced by role contracts
• Against boundary entropy: active boundary maintenance with formal contracts
• Against signal degradation: typed epistemic output contracts at every boundary
• Against routing stochasticity: deterministic REPAIR-ROUTER as pure function
• Against partial success: multi-layer semantic success evaluation

These are not heuristics. They are derived from the mathematical properties of autoregressive transformers, the stochastic nature of softmax-sampled generation, and the fixed-size nature of the context window. They apply to GPT-4, Claude 3, Gemini Pro, and every other transformer-based model that will be released in the coming years, because the architecture is the same.

The 99.2% production success rate reported here is not the ceiling. The remaining 0.8% failure rate consists of genuinely novel failure classes—unexpected API behavior, novel multi-agent interaction patterns, infrastructure failures that bypass our retry logic. Each of these will become understood, classified, and systematically eliminated. The ceiling is determined by how rigorously the engineering team applies the physics, not by what the models can do.

Appendix

Glossary of Key Terms

• Architecture A Fully autonomous sub-agent architecture where agents loop for as many turns as needed. 62% success rate in production. See Section 4.
• Architecture B Controlled sub-agent architecture with max_turns=1. All decisions centralized in the Kernel. 91% success rate. See Section 5.
• Architecture C Hybrid sub-agent architecture combining the Lease mechanism, Semantic Yield Protocol, and Trajectory Tracking. 93–95% success rate. See Section 6.
• Architecture C+ Fully hardened architecture combining Architecture C with Cognitive Cohesion, Dual-Loop Architecture, Typed Boundaries, and Epistemic Contracts. 99.2% success rate.
• Boundary Entropy The tendency for agent role boundaries to become ambiguous over time through scope creep, tool proliferation, context bleed, and prompt drift. See Section 8.
• Cognitive Blackboard Shared persistent structured store that all agents read and write, replacing context injection for critical artifact propagation. See Section 13.
• Cognitive Cohesion The principle that each agent has exactly one epistemic domain and all cognitive resources are dedicated to it. Enforced via Role Contracts. See Section 7.
• Cognitive Fragmentation The state in which an agent's cognitive resources are split across multiple epistemic domains, reducing quality in all of them. See Section 7.3.
• Dual-Loop Architecture Recovery architecture with two paths: Fast-Twitch (Coder self-corrects, no Researcher) and Slow-Twitch (Researcher updates spec, Coder regenerates). See Section 9.
• Epistemic Output Contract Formal commitment by an agent to return findings in a fully-typed validated structure containing all information downstream agents need. See Section 10.2.
• ExecutionFailure The canonical typed signal crossing agent boundaries, containing category, raw_message, retryable, and execution_status. See Section 10.
• FAILED_INCOMPLETE_FETCH Typed execution status constant indicating that an API returned mixed success/failure results across multiple resources. Maps to endpoint_unknown. See Section 11.2.
• FAILED_SILENT_PARTIAL_FETCH Typed execution status constant indicating that an API returned HTTP 200 with empty results for an unsupported operation. Maps to endpoint_unknown. See Section 11.2.
• Fast-Twitch Loop Recovery path for Coder-class failures (syntax/logic errors). Coder receives specific error with self-correction lease. No Researcher invocation. 15–45s recovery.
• Lease Bounded autonomy grant from the Kernel to a sub-agent. The agent may make up to $N$ decisions before mandatory return. Cannot be extended by the agent. See Section 6.3.
• P0 Injection Technique of placing the highest-priority instruction at the top of the LLM's context window to exploit primacy recall. Used for repair directives. See Section 10.4.
• Partial Success Failure mode where a multi-resource operation returns mixed results that appear successful at the aggregate level while hiding individual failures. See Section 11.
• REPAIR-ROUTER Deterministic routing component that maps typed ExecutionFailure.category to recovery actions. Pure function, no LLM inference. See Section 10.
• Role Contract Formal specification of an agent's Epistemic Domain, Action Scope, and Yield Conditions. Enforces Cognitive Cohesion. See Section 7.2.
• Role Hallucination When an agent takes actions outside its epistemic domain, producing lower-quality results while consuming cognitive budget on the wrong task.
• Semantic Yield Protocol Deterministic check (hardcoded Python) that fires after every tool execution in Architecture C, forcing immediate escalation upon detecting blockers outside the agent's competency. See Section 6.2.
• Slow-Twitch Loop Recovery path for Researcher-class failures (wrong API spec). Researcher updates specification, Coder regenerates with new context. 90–300s recovery.
• Typed Boundary An agent boundary where all crossing signals are typed objects conforming to a formal schema. No prose. No string patterns. Zero ambiguity. See Section 10.

---

The Three Laws (Volume 1 Summary)

For readers who have not read Volume 1, the three fundamental laws are summarized here:

All architectural patterns in this volume are derived from these three laws. They apply universally to all transformer-based LLMs, regardless of model size, training data, or vendor.

---

Acknowledgements

The architectural patterns documented in this volume emerged through intensive production engineering—the kind that only comes from watching systems fail in real environments and reasoning carefully about why. Each failure class described in these pages was a real event that caused a real workflow to either produce wrong results or fail entirely.

The engineering solutions documented here are not theoretical. They are the product of iterative refinement under production constraints, where the cost of a wrong decision is not an incorrect benchmark score but a failed enterprise workflow that a real organization depended on.

The author thanks the broader AI engineering team at Prescott Data for the open publication of failure modes, architectural anti-patterns, and production learnings. This work builds on everything that came before it.