Same Code, Different Reach: Learning Positional Comparisons for Length Extrapolation

Length extrapolation in copying requires reliable source retrieval beyond the training range. We study how positional structure and training shape this learned computation through controlled experiments in small transformers. Across six harmonic frequency grids and twelve paired seeds, 25% frequency-plane dropout improves correct-history token accuracy at four times the training length by 34–41 percentage points on four grids; one shows no resolved gain and one worsens. All bands are restored at evaluation, so these gains arise with the positional code held fixed. To investigate the learned addressing computation, we distinguish context matching from selection among matching occurrences. Positional features help construct retrieval contexts as well as distinguish addresses. A task-derived contrast predicts mean spectral-growth peaks at four training lengths, but different learned band mixtures implement similar local comparisons with divergent distant responses. Removing distant attention competitors improves context selection and copying. Frozen short-range spectral forecasts locate consequential distant attention, although equal-mass controls show no resolved accuracy advantage for the exact selected addresses. Matched continuations and signed-coefficient transfers establish a contributing role for learned positional comparisons. Band interventions reveal useful alternatives, and a conditional subset-margin theorem gives sufficient conditions for distant separation in a reduced model after all bands are restored. Context discrimination improves, while ambiguity among matching occurrences remains. Together, these findings distinguish a positional code’s available comparisons from those learned through training, and identify support-dependent positional regularization as a promising approach to length extrapolation.