acceptodds
Under review as a conference paper at ICLR 2027

Task Geometry: Response Channels Predict Explicit Recovery and Attention Localization

Abstract

In-context learning on fresh low-dimensional tasks requires attention to infer which input directions define task geometry. We study an explicit response-derived route to that geometry. A clipped second-moment response channel yields cross-fitted quadratic-softmax attention with log-free, dimension-explicit subspace recovery and the intrinsic point-query rate when the channel is open; when it is closed, that statistic is non-identifying even though its finite-sample geometry need not be Haar. In a controlled replication with frozen hypotheses on ten synthetic tasks at and , the mean Haar-minus-explicit geometry advantage is larger on six open tasks than on three closed tasks, with one preregistered numerically weak task outside the contrast. On the three closed odd-link tasks, a small task-specific Transformer attains lower prediction risk than both the Haar control and the explicit route. This trained result is predictive rather than geometric: it uses task information beyond the selected second-moment statistic, while an open first-moment channel provides a classical alternative explanation. Together, response channels predict explicit geometry recovery and separate it from broader task-specific prediction.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.