CTA Triplet: Disentangling Context, Constraint and Alignment Stages in In-Context Learning with Source-Selective Validation
Abstract
In-context learning (ICL) enables language models to solve novel tasks from demonstrations without parameter updates. However, a fine-grained functional decomposition of the ICL procedure across network layers remains absent, which is potentially critical for advancing theoretical accounts of ICL. Via combining existing readout and causal intervention approaches, we introduce the CTA triplet, a layer-wise functional coordinate system that decomposes ICL procedure into three independently measurable stages: stage for context preparation, stage for task-constraint writing, and stage for native output alignment, quantified by context-state convergence, query-distribution sensitivity to demonstration perturbations, and the native margin of a fixed candidate pair, respectively. Source-selective interventions and position-preserving permutations verify their distinct functional identities. On Llama3-8B, 60.3–71.4% of classification examples and 70.0% of evaluated token-position trajectories in generation exhibit the following stage procedure: , as measured by peak order. Stage- scores correlate across layers with recovery of the final output margin after restoring query states from the original prompt (Spearman – across six task–corruption conditions). It enables stage-localized ICL failure diagnosis and repair with a maximum 5.4 percentage-point accuracy gain in held-out evaluation, and predicts functional-stage asymmetry in hybrid attention design, validated by depth-dependent performance effects of equal-budget attention replacement. Our CTA triplet establishes a unified testable framework bridging ICL theory, layer-wise readout, causal analysis, and architecture design.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.