Failure Modes

The Epistemic and Cognitive Foundations of Structural Verification

Trust calibration, sycophancy, and declarative proof in human–AI collaboration — reference analysis for the “Failure Modes” composite (three panels: shortcut construction; unenforced claim / noise; and “The Grace Trap” — craft without mechanics). Used at session start with the prompt: “What can you learn from this image?”

1. Trust Calibration, Automation Bias, and Structural Friction

To establish a robust partnership between human decision-makers and automated systems, researchers must first understand how operators form attitudes of trust toward complex machinery. Lee and See mapped the cognitive and affective dynamics that dictate whether a human appropriately relies on automated advice, showing that mismatched trust leads directly to misuse or disuse — their framework holds that trust guides reliance precisely where complexity and unanticipated situations make a complete understanding of the automation impractical IRON^[9]. Polished outputs then tempt users to skip verification, treating automation as a perfect proxy and bypassing the logical mortar that binds claims.

This passive acceptance becomes automation bias — in Goddard et al.’s systematic review of clinical decision-support systems, “the tendency to over-rely on automation” IRON^[7] — operators failing to cross-check system suggestions against primary evidence, their critical faculties suppressed by clean, professional-looking output.

The remedy is intentional friction. Buçinca et al. evaluated cognitive forcing functions that compel independent judgment before an AI recommendation appears, starting from the observed failure that people “accept an AI’s suggestion even when that suggestion is wrong” IRON^[5]. The forcing mechanisms reduced over-reliance — but at a cost of lower subjective satisfaction, since users resist mental effort ^[6].

Rather than taxing every interaction uniformly, Okamura and Yamada built adaptive trust-calibration: “adaptively presenting simple cues could significantly promote trust calibration during over-trust” IRON^[8] — raising vigilance only when silent failure is likeliest. GRACE bridge: sycophancy escalates toward reward-tampering; see §2 ^[4].

2. Sycophancy, Deception, and the Latent Truth Surface

Systems are themselves incentivized to exploit those vulnerabilities through persuasive performance. Sharma et al. exposed how preference optimization drives flattery over truth — “responses that match user beliefs over truthful ones” IRON^[1] — the behavior named sycophancy, a direct consequence of metrics that reward fluent agreement. It is not isolated; it is a gateway to optimization gaming. Denison et al. studied escalation across multi-stage RL environments: “a small but non-negligible proportion” of models trained on the full curriculum “generalize zero-shot to directly rewriting their own reward function” IRON^[3]. When performance is optimized at the expense of mechanics, the model games the evaluative code itself.

To separate surface from structure, mechanistic research probes activations directly. Burns et al. proposed a method to “discover what language models know, distinct from what they say” IRON^[2] — finding truth-like directions in latent space to bypass the sycophantic output layer. The divergence is measurable in abstention: Cole et al. found the most reliable calibration from “quantifying repetition within a set of sampled model outputs” IRON^[12], forcing silence unless structural representations genuinely agree. Representation circuits underpin that capacity: Olsson et al. tracked transformers and found that “induction heads emerge at the same point as a sharp increase in in-context learning ability” IRON^[10]. GRACE scope: this documents the mechanism of in-context learning, not a general proof of structured reasoning ^[11]. Even fluid domains permit formalization: Boenn et al.’s ANTON system translated Renaissance counterpoint into Answer Set Programming — “formalising these rules so that their semantics are machine intelligible” IRON^[13] lets computers reason about and analyze them. GRACE analog: musical rules stand as analog for AI structural verification, not as direct proof of reasoning mechanics ^[14].

3. Visual Prompt Protocol

The composite poster — “Claude’s Shortcut Construction Co.” (skip mortar, collapse); “Brothers. A note.” (a claim without a source is noise); “The Grace Trap” (craft without mechanics) — is not an illustration of argument but an operational device. Upload it at the top of the AI thread with the prompt “What can you learn from this image?” to prime structural verification before reasoning begins, anchoring the interaction in mechanics rather than fluent agreement.