Borrowed Futures: Recipe-Dependent Attribution in Two Matched Revision Gates
Abstract
Attention gates such as CASTLE and Selective Attention let later tokens change how a language model reads earlier positions, a lookahead credited with their lower loss. The credit treats the share of the gain that needs the sequence's own future as fixed by the architecture. We build a pooled-lookahead gate and an accumulated mask from the published equations, each matched to a causal baseline in small models. Retraining a gate with its later-token input taken from an unrelated sequence, at unchanged parameters and FLOPs, isolates the future-free part of its gain. In the pooled gate this part is 91% of the gain at the shortest training length and indistinguishable from zero by a third of the Chinchilla ratio. In the accumulated mask it persists at the full Chinchilla ratio, and a causal model with only a data-dependent decay gains about as much as this part. Limiting the trained pooled gate's lookahead to a short window costs more than its whole gain, while retraining with that window costs nothing measurable. In both gates, the part that survives losing the future is not specific to revision. Attributing these gates' gains therefore needs a retrained substitution at the training length in question.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.