acceptodds
Under review as a conference paper at ICLR 2027

Borrowed Futures: Recipe-Dependent Attribution in Two Matched Revision Gates

Abstract

Attention gates such as CASTLE and Selective Attention let later tokens change how a language model reads earlier positions, a lookahead credited with their lower loss. The credit treats the share of the gain that needs the sequence's own future as fixed by the architecture. We build a pooled-lookahead gate and an accumulated mask from the published equations, each matched to a causal baseline in small models. Retraining a gate with its later-token input taken from an unrelated sequence, at unchanged parameters and FLOPs, isolates the future-free part of its gain. In the pooled gate this part is 91% of the gain at the shortest training length and indistinguishable from zero by a third of the Chinchilla ratio. In the accumulated mask it persists at the full Chinchilla ratio, and a causal model with only a data-dependent decay gains about as much as this part. Limiting the trained pooled gate's lookahead to a short window costs more than its whole gain, while retraining with that window costs nothing measurable. In both gates, the part that survives losing the future is not specific to revision. Attributing these gates' gains therefore needs a retrained substitution at the training length in question.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.