acceptodds
Under review as a conference paper at ICLR 2027

Supervision Density Shapes Transformer Solutions and Long-Context Generalization

Abstract

Which tokens receive supervision is typically treated as an efficiency choice: supervising fewer tokens can reduce training cost while preserving performance. Here, we show that supervision density also determines how a Transformer learns to solve a reasoning task, with direct consequences for long-context generalization. We train 12-layer Transformers on a controlled multi-hop variable-binding task while varying the fraction of suffix tokens contributing to the loss, from standard all-token prediction to last-token supervision, and evaluate generalization to both longer contexts and deeper reasoning chains. We uncover a strikingly non-monotonic relationship between supervision, learning speed, and generalization. Intermediate supervision densities converge substantially faster than either all-token or answer-only training, yet the fastest-converging model is not the best generalizer. At convergence, sparse 25% supervision yields the strongest long-context generalization, whereas prolonged training eventually makes the last-token model the strongest generalizer despite validation accuracy having saturated much earlier. Mechanistic analyses reveal that these learning regimes steer models toward qualitatively different solutions: suffix-supervised models predominantly write the answer through the MLP pathway, whereas last-token supervision produces an attention-dominated solution. Crucially, attention remains causally necessary across all regimes, and the strongest generalizer at convergence is the model that most strongly requires both attention and MLP pathways. Together, our results show that supervision density does more than control training efficiency: it selects among memory- and structure-based solutions, thereby shaping when and how Transformers generalize beyond their training distribution.

Primary area:
interpretability and explainable AI
Keywords:
Token-level supervision, variable binding, multi-hop reasoning, length generalization, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aEmcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a study on how supervision density (the number of tokens supervised in the loss function) shapes the type of solution learned by a transformer and how well that solution generalizes. They train 12-layer transformer to solve a multi-hop variable binding task, and supervise a range of tokens (p=0 refers to the standard last-token supervision, while p=1 means all the tokens are supervised). They analyze how this affects convergence dynamics, attention and MLP solutions, and generalization capabilities.

Critical strengths
  1. The experimental setup is clear and the analysis makes sense. Both the experimental setup and analysis are clearly related to the problem at hand. Additionally, the plots are well-made and understandable, and I can (for the most part) follow the train of thought in the results section.
Critical weaknesses

The primary weakness I identified is that it is unclear how generalizable these results are. Per my understanding (please correct me if I missed something, especially in the appendix): the study is done with i) 12-layer transformers and ii) one toy task. Thus, it is very unclear how these results hold to language / reasoning models more broadly, especially as you scale up parameters, layers, etc. Additionally, I couldn't find the number of seeds used, which further calls into question the generalizability (see Questions for Authors 1.). Some concrete suggestions that would improve the paper:

1. Run multiple seeds (again, refer to questions for authors and please correct me if I'm mistaken).

2. Evaluate on longer contexts. Currently, the paper argues for "long-context generalization" (line 152), but the ctx length x goes only up to 26. Can the authors clarify how long-context this setup actually is? Transformers today are scaled up to much bigger size and hundreds of thousands (even 1-M token context length), so the claim about long-context generalization is in doubt for me, especially with the title being about "transformer solutions", which is quite a broad claim.

3. Add an additional toy task. The task doesn't have to be huge, but this would certainly help with my worry about how generalizable these results are.

Questions for authors
  1. Can the authors clarify the number of seeds they used? Lines 132 and 133 say "Splits are deterministic given a seed (450,000 training, 1,000 validation, and 49,000 test programs) and are the same in every run". Does this mean 450,000 different models / seeds? I do not see error bars on the plots in the main text, which is making me question the statistical significance of the results (e.g. it's hard to place in context what the difference between p=0 and p=0.25 is in Figures 2 and 3 means in Figures 2 and 3). If the authors can provide information on the number of seeds used / how many different models were trained, that would be helpful.

  2. How strongly do these experimental results transfer to transformers more generally? The claim in the title is about "transformer solutions and long-context generalization", which is broad. However, the actual study is conducted on 12-layer transformers and one multi-hop task, which seems narrow.

3 comments

We are grateful to the reviewer for their positive assessment of our work and for the time and effort devoted to its evaluation. We particularly appreciate the constructive observations and valuable suggestions provided, which have increased the quality of our work. In the following, we respond to each comment in detail and clarify the points raised.

Critical weakness 1: We apologize for this confusion. In fact, we had run more runs which all showed similar results, and omitted to add the aggregate results in the appendix. The revised version will also contain the averaged results from 3 seeds. Across the three seeds, the results hold: intermediate densities converge fastest, p = 0 generalizes best at step 300k in every seed, and at step 300k the solution is attention-dominated for p = 0 and MLP-dominated for p ≥ 0.25 (see Table 1 below).

Table 1. Mean accuracy ± sd over three seeds. Steps: first maximal accuracy model. LC: long-context accuracy (%), 18–26 lines × 5–8 hops. d_attn − d_mlp: direct contributions to the answer margin (positive: attention dominates).

pSteps (k)LC, first max.LC, step 300kd_attn − d_mlp, first max.d_attn − d_mlp, step 300k
1102 ± 2675.9 ± 14.279.5 ± 12.4−5.2 ± 2.6−11.5 ± 1.3
0.7529 ± 663.6 ± 14.171.8 ± 4.2−3.9 ± 1.1−11.9 ± 0.2
0.535 ± 272.0 ± 5.985.2 ± 1.5−3.6 ± 0.3−11.3 ± 1.1
0.2535 ± 971.8 ± 8.479.6 ± 5.4−2.1 ± 5.6−14.1 ± 1.2
0137 ± 282.3 ± 8.695.9 ± 4.29.5 ± 1.011.1 ± 2.1

Critical weakness 2: Evaluating on longer contexts, similar to those of frontier models (1M tokens), is prohibitively costly in our case. However, we do want to clarify that the condition “lines 26” implies a context length of 26*4+4 = 108 tokens, which is an increase of ~60% over the training input length.

Critical weakness 3: We thank the reviewer for this suggestion. To demonstrate the generalizability of our results, we added a second task, nested boxes: we extended the entity-tracking task [1], in templated natural language, to multi-hop containment (e.g., "the computer is in Box W, Box W is in Box N, …, Box N contains the" → "computer"), and reduced the objects from 85 to 10 to match the 10 digits of x=y. We replicated our findings with this natural language task: answer-only supervision generalizes best after continued training, and denser supervision yields more MLP-dominated solutions (see Table 2 below). We will add the results of this novel task in the revised version.

Table 2. Nested boxes results. Steps: first maximal accuracy model. LC: long-context accuracy (%), 18–26 clauses × 5–8 hops. d_attn − d_mlp: direct contributions to the answer margin (positive: attention dominates).

pSteps (k)LC, first max.LC, step 300kd_attn − d_mlp (first max.)d_attn − d_mlp (step 300k)
14635.541.0−0.8−6.6
0.756036.936.20.1−6.8
0.523056.456.22.51.4
0.257237.637.49.114.0
013069.988.77.85.2

We will add the results of this novel natural language task to the revised version of the paper and thank again the reviewer for suggesting this addition which broadens the scope of our paper.

[1] Prakash, Nikhil, et al. "Fine-tuning enhances existing mechanisms: A case study on entity tracking." International Conference on Learning Representations. Vol. 2024. 2024.

Questions:

(i) We ran 3 seeds leading to 15 models trained. Our results remain unchanged and our revised version will contain error bars in the plots.

(ii) Evaluating this on broader models is, unfortunately, prohibitively costly for us, and we do not have access to the infrastructure to evaluate the generalizability of our results to bigger, state-of-the-art models. Note that our work requires training models from scratch. However, as suggested by the reviewer we did expand the scope of our results to another, natural language, tasks.

I thank the authors for the updates. I'm still concerned about the lack of scale, using only a 12-layer transformer is prohibitive to the study. Even going to 24 and 36 layers would be beneficial. Also this may be pedantic, but regarding weakness 2) I would refer to that as length generalization instead of long context. Long context typically means a huge # of tokens, but an increase over training tokens is more length generalization. I have updated to a borderline 3 with these concerns.

We would like to thank the reviewer for their time to evaluate our rebuttal, raising their score, and seeing value in our work. We are currently evaluating the possibility of scaling these results to bigger models in a collaboration with other institutions. Again, thank you for your time and suggestions that have improved the quality of our work.

Reviewer_PfyLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the trade-off between supervision density and long context generalization using controlled multi-hop binding task. The authors train a GPT-j style transformer with RoPE attention and vary the supervision density (p=1 all tokens ~ 0 as final token).

They find that two extreme conditions (p=1,0) learn most slowly, whereas intermediate densities converge faster for reaching first validation 100% accuracy, p=0 as the worst generalizer. However, after continued training to 300k steps, p=0 model achieves the strongest generalization. To further investigate which mechanism allows this, the authors decompose the direct contribution of attention and MLP. They argue that dense supervision tends to produce MLP dominated (memory-based) computation and sparse supervision favors more algorithmic computation. Also, they suggest that generalization is not explained solely by attention but with a combination of attention and MLP contribution.

Lastly, they conduct causal mean ablation on each component. Removing attention causes much larger degradation in accuracy, indicating that attention is necessary for generalization. Throughout attention flow and t-SNE representation analyses, they argue that sparse supervision develops more structured attention representations.

Critical strengths

The paper investigates the respective contributions and training dynamics of the MLP and attention in length generalization, using a controlled setup that enables causal assessment of each component’s contribution. It does, in my view, provide interesting insights into how the training setup affects the algorithmic solutions learned by the network, also with respect to the course of learning. I therefore do overall find the paper valuable, even if some of the claims are not strictly proven but rather demonstrated in terms of supporting evidence.

Critical weaknesses

The paper’s claim that sparse supervision promoted attention based algorithm generalization might be entangled with the use of RoPE. RoPE by its nature provides structured attention especially when the context length gets longer. Thus, sparse supervision may preferentially select a RoPE supported routing strategy rather than reveal an architecture-independent relationship between sparse supervision and algorithmic generalization. This needs to be discussed prominently, or the paper needs to be extended with further non-RoPE experiments.

Another concern is with Section 3.3 and its causal interpretation. Earlier sections suggest that strong generalization depends on the joint contribution of attention and MLP (i.e., attention is substantial, but the MLP contribution remains non-negligible after 300k steps). However, the causal analysis primarily ablates attention and MLP separately. This does not establish how the two components interact causally to support generalization.

Moreover, the attention flow analysis does not clearly explain why attention is crucial. Qualitatively showing that attention flows with p=0 follows the structured flow is consistent with the conclusion that is drawn, but does not strictly guarantee that these flows cause the generalization benefit. More targeted causal analyses, such as interventions on specific attention edges could be performed to test whether those routing patterns are necessary.

In line with this, t-SNE results provide only indirect evidence for the mechanism. The core challenge of this task is to bind variables to values (or variables to variables) and compose those bindings across a chain. Therefore demonstrating them separately does not show that the model represents their relational bindings. Again, this does not prove that the model is using these structured representations.

Questions for authors

Clarifications

The definition of the ‘direct contribution’ in section 3.2 should be stated more clearly in the main text. Although the formal metric appears in the appendix, the main text should explicitly indicate where it is defined and briefly explain the metric. This is essential for interpreting Figures 4 and 5. Moreover, it is not clear which checkpoint has been used for Figure 4, which is important to mention because generalization of p=0 changes substantially with training time. The interpretation of Figure 5 also needs clarification. The trajectories appear highly asymmetric: reaching the first max accuracy seems to require relatively little attention but a much larger MLP contribution (dattnd_{attn} 13; dmlpd_{mlp} 59). For Table 1, it would be helpful to explain why the ablation effects appear to differ more strongly across input length conditions than across hop conditions, particularly for attention ablation. In addition, Section 3.2 reports a large direct MLP contribution for (p=1), whereas Table 1 shows that MLP ablation causes a substantially smaller accuracy drop than attention ablation. This discrepancy should be discussed explicitly where direct logit contribution and causal contribution seem to capture different aspects of the computation.

Suggestions

The paper would benefit from causal analyses that test the interaction between attention and MLP, rather than ablating each component separately. Attention flow visualization does not establish that the identified structural flows are causally responsible for generalization. The authors could show that targeted intervention of the attention paths degrades task performance. The authors should test whether their results depend on RoPE. Repeating the supervision density comparison with alternative positional encoding methods would help determine whether sparse supervision generally promotes algorithmic generalization or instead selects a routing strategy that is particularly compatible with the chosen architecture.

5 comments

We thank the reviewer for recognizing the contributions of our work and for taking the time to provide thoughtful and constructive feedback that has improved the quality of our work. We appreciate the insightful comments and suggestions, which offer valuable opportunities to strengthen our paper. Below, we address each of the reviewer's points individually.

Critical weakness 1: We thank the reviewer for this very interesting point. Indeed, RoPE generates biases in terms of attention [Barbero et al., 2024], and may therefore be a confounding effect. To address this concern, we replicated our results training models with another positional encoding (Alibi); models with NoPE never learn the task p=0 density (we had tried this in earlier experiments within our work with different seeds and model architecture). All our results remain unchanged, we still observe the effect of supervision density on the type of solutions developed, and that attention-dominated solutions produce stronger generalization results versus MLP-dominated ones:

Table 1. ALiBi Generalization.

DensityLen 18Len 20Len 22Len 24Len 26Hop 5Hop 6Hop 7Hop 8
alltoken70.169.767.759.243.769.763.659.355.8
0.7570.468.467.765.258.982.264.560.557.3
0.5064.366.162.352.139.662.858.554.651.6
0.2580.680.978.971.161.381.276.371.669.3
lasttoken81.378.076.174.166.578.277.774.470.6

Table 2. ALiBi attention-MLP DLA trade off

DensityAttention (RoPE)MLP (RoPE)Attention (alibi-PE)MLP (alibi-PE)
alltoken1.148.093.375.03
0.752.495.985.453.02
0.502.356.311.387.39
0.251.119.611.088.63
lasttoken10.08-0.1411.39-0.39

As suggested by the reviewer, we will acknowledge the potential confound of RoPE and improve our revised version with evidence of our results under another positional encoding scheme.

Critical weakness 2: We thank the reviewer for raising this interesting comment. To investigate how attention and MLP interact to support generalization we parametrically and independently modulate the ablation ratio for the attention and MLP pathways:

Hℓ+1=hℓ+αAttnℓ(hℓ)+β MLPℓ(hℓ)H_{\ell+1} = h_\ell + \alpha Attn_\ell(h_\ell) + \beta\ MLP_\ell(h_\ell)

where α,β∈[0.25,1]\alpha, \beta \in [0.25,1], and used values of α,β∈{0.25,0.5,0.75,1}\alpha, \beta \in \{0.25, 0.5, 0.75, 1\}.

We performed these additional analyses for all densities, but for the sake of space we report the resulting grid-table below for density p = 1 and p = 0 (i.e., the model with strongest generalization; the revised version will contain tables for all densities in the appendix). Analyzing these tables allows us to reveal that indeed the MLP and attention interact to support generalization. In fact, increasing MLP contribution improves generalization much more when attention is relatively intact than when attention is strongly ablated.

Table 1. Interaction between MLP and attention (values are task accuracy)

Model p = 0

β (MLP) ↓ / α (Attention) →0.250.50.751
0.25100.094.740.821.3
0.589.8100.0100.094.7
0.7556.899.3100.0100.0
140.289.899.9100.0

Model p = 1

β (MLP) ↓ / α (Attention) →0.250.50.751
0.25100.099.137.64.6
0.597.3100.0100.098.6
0.7561.5100.0100.0100.0
133.497.8100.0100.0

These tables reveal a nonlinear and asymmetric interaction between attention and MLP contributions. For the MLP-dominated model (p=1), if the attention contribution strongly overweighs the MLP one, generalization performance sharply drops. This drop is much less pronounced for the attention-dominated. In contrast, for the attention-dominated model (p=0), if the MLP contribution strongly overweighs the attention one, generalization performance sharply drops. And similarly, this drop is less pronounced for MLP models. Interestingly, reducing both components simultaneously can partially restore generalization. These results suggest that generalization depends not simply on the individual strength of each pathway, but on their joint configuration, with certain combinations producing particularly detrimental interactions. We will add this result (in the form of a heatmap for better visualization) in the revised version (also including the table for models p=0.25-0.75, which show a nice transition between the results of p=0 and p=1).

Responses to other weaknesses continue in the following comment

Critical weakness 3: We agree with the reviewer. To demonstrate more convincingly the proposed distinction between the two mechanisms, we target the specific attention weights that follow the hop-chain. For instance, in a 4-hop sample, we specifically target either hop 1, 2, 3 or 4. Following our hypothesis, these independent ablations should impair the performance of the attention-dominated (p=0) solution in a uniform manner; that is, irrespective of which hope is targeted the model should drop its performance. In contrast,MLP-dominated (i.e., p=0.25-1) models should not display this uniform performance drop. Indeed, the latter should be impacted by the distinct hop ablations in a much more random fashion. Crucially, our novel targeted-ablation experiment reveals the hypothesized effect:

Table 2. Targeted hop ablation (values are generalization performance)

Maskp = 0p = 0.25p = 0.5p = 0.75p = 1
No ablation1.0001.0001.0001.0001.000
Ablation hop 40.5720.9880.9880.8440.220
Ablation hop 30.4760.7841.0000.9960.792
Ablation hop 20.3960.8720.9921.0000.984
Ablation hop 10.5280.8480.6400.7440.856

Note that we observe these ablation effects both for the first maximal accuracy (reported in the table above) and 300k models. These results will be added to the revised version.

Questions: We thank the reviewer for the suggestions to improve the clarity of our work and will incorporate all of them in the revised version:

(i) we will include a formal brief definition of “direct contribution” in the main text.

(ii) In figure 4, the checkpoint is informed in the figure caption. We will include it in the text as well to optimize clarity (the checkpoint is the first maximal accuracy model).

(iii) the change in trajectory directly reflects the tradeoff we reveal in this work, and we will clarify that the plane allows us to reflect the differences in trajectory, which in turn reflects the solution types and generalization performance.

(iv) we will also explain Table 1 with more details; what matters here is the relative change of accuracy drop when ablating the attention or MLP in either attention-dominated versus MLP-dominated solutions. We observe that ablating the attention pathway in attention-dominated solutions penalizes performance much more than ablating the MLP pathway, the opposite is true for MLP-dominated solutions. Furthermore, we will explain more clearly the interaction between attention and MLP as suggested above by the reviewer.

Dear reviewer, given the approaching deadline for the author-reviewer discussion period, we hope you find the time to evaluate our response, and thank you again for your time and interest in our work.

Reviewer_waZsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how token-level supervision, ranging from full next-token to answer-only prediction, affects Transformer generalization and learned mechanisms on a synthetic variable-binding task.

The authors observe a non-monotonic relationship: intermediate supervision converges fastest, 25% supervision generalizes best at initial validation convergence, and answer-only supervision excels after prolonged training. Mechanistic analyses such as logit attribution, ablations, and representation/attention visualizations reveal that changing the objective fundamentally alters attention patterns and MLP utilization. The authors convincingly frame this as a shift between memory-based and structure-based learned solutions.

Critical strengths
  1. The paper asks a meaningful question about training objectives. Whether the placement and density of supervision influence the computation a model learns is scientifically interesting. Building on the variable-binding setting of previous works, the paper investigates how changing the objective affects both learning dynamics and extrapolation. This provides a useful direction beyond evaluating token selection solely through final task accuracy.
  2. The experimental setting supports informative comparisons. Architecture, training programs, tokenization, and data splits are held fixed across supervision regimes. The evaluation also varies context length and reasoning depth. Importantly, Figures 9–10 include conditions that change only one dimension, alongside the jointly longer and deeper programs emphasized in the main text. These results help distinguish distractor sensitivity from difficulty with deeper composition. 3.The distinction between convergence speed and eventual generalization is valuable. The fastest model is not the strongest generalizer, and the ranking changes substantially after validation accuracy saturates. For example, answer-only supervision improves from approximately 72% to 91% average accuracy on the joint generalization grid. This makes training duration a substantive part of the result and motivates caution when selecting checkpoints solely through in-distribution accuracy. 4.The analysis goes beyond descriptive attention plots. Combining direct logit attribution with mean ablations is a useful methodological choice. The paper explicitly distinguishes writing to the final answer from being necessary for producing it. The finding that attention remains important even when its direct answer contribution is small is particularly informative.
Critical weaknesses
  1. Supervision density is confounded with other changes to the objective. Reducing density changes both which positions receive supervision and the answer’s weight in the loss. The results therefore do not isolate the effect of density itself. Comparisons with different masks at matched density and controlled answer weighting are needed.

  2. The main findings lack evidence of robustness across training runs. The paper appears to report one run per density. Statistical tests across evaluation examples do not capture variation across training seeds. The non-monotonic ranking also needs to be checked under alternatives to selecting the first checkpoint with 100% validation accuracy.

  3. The mechanistic interpretation exceeds the evidence. A larger MLP contribution does not establish memorization, and attention that skips a hop may be using an already-composed representation. The broad pathway ablations establish dependence on attention and MLPs, but do not demonstrate the proposed distinction between memory-based and structure-based computation.

  4. The readout definitions are inconsistent. The appendices describe intermediate readouts using different normalization conventions, while sometimes treating their outputs as equivalent. These choices can affect predicted tokens, confidence, and the inferred layer of answer commitment. The authors should clarify which convention each figure uses and verify that the conclusions remain valid.

Questions for authors
  1. Can you separate mask density, mask location, and answer weighting? Do the main results persist with randomly distributed supervised positions at matched density, and with a fixed relative weight between answer loss and the auxiliary prediction objective?

  2. What evidence distinguishes memorization from reusable computation? Could targeted activation interventions test whether apparent hop skipping reflects an incorrect shortcut or valid intermediate composition? How do pathway ablations affect in-distribution accuracy, and how should Table 3’s stronger ablation sensitivity for p=0p=0 be reconciled with the proposed explanation?

  3. Why does answer-only supervision lose its advantage on the longest, deepest programs? Does its failure reflect distractor interference, positional extrapolation, insufficient computational depth, or another mechanism? Explaining this boundary would materially strengthen the connection between the mechanistic analysis and generalization.

4 comments

We thank the reviewer for seeing value in our work, and appreciate their time reviewing it and the interesting and useful comments raised that have allowed us to improve the quality of our work. Below we provide a point-by-point reply that addresses the issues raised by the reviewer.

Critical weakness 1 (and answer to Q1): We fully agree with the reviewer. To isolate the effect of density and decouple it from both potential positional and an answer weighting effects we performed two additional experiments.

First, we decouple density with location and, given a fixed density (p=0.25 to p=0.75), supervise the location of tokens randomly. Note that this can only be done for models with densities between p = 0.25 and p = 0.75. These extra simulations maintain the patterns observed in our results unchanged, i.e., lower density (irrespective of location) drives stronger generalization in particular at longer training steps (300k). Moreover, the directionality of the tradeoff is also maintained at the 300k checkpoint. The tables below provide the quantitative evidence:

Table 1. Long-context generalization (%) and the difference between d_attn - d_MLP both at first maximal accuracy and 300k checkpoint for the suffix and random loss masks.

MaskpLearnedLC, first max.LC, step 300kd_attn − d_mlp, first max.d_attn − d_mlp, step 300k
suffix0.753/363.6 ± 14.171.8 ± 4.2-3.9 ± 1.1-11.9 ± 0.2
suffix0.53/372.0 ± 5.985.2 ± 1.5-3.6 ± 0.3-11.3 ± 1.1
suffix0.253/371.8 ± 8.479.6 ± 5.4-2.1 ± 5.6-14.1 ± 1.2
random0.753/371.3 ± 9.467.8 ± 13.6+0.7 ± 5.5-5.3 ± 7.7
random0.52/372.0 ± 13.976.1 ± 12.4-5.4 ± 3.6-11.8 ± 2.9
random0.252/370.9 ± 1.381.3 ± 7.9+6.3 ± 3.7-2.2 ± 12.6

Second, we decoupled the Loss such that:

Lp=Lansnmax+1N∑i∈SpauxLiL_p = \frac{L_{ans}}{n_{max}} + \frac{1}{N} \sum_{i \in S_p^{aux}} L_i

with NN equals to the number of supervised auxiliary predictions and nmaxn_{max} to the longest input length (i.e., sequence length with p = 1. Note that this number is fixed for all densities, and thus the contribution of the answer is similar in all models. Crucially, we find similar patterns as using a standard cross entropy loss, again both for the generalization performance and the attention-MLP tradeoff:

Table 1. Long-context generalization accuracy (%)

Weighted-loss, first-maximal-accuracy

DensityLen 18Len 20Len 22Len 24Len 26Hop 5Hop 6Hop 7Hop 8
alltoken84.478.573.268.464.283.375.370.466.0
0.7586.584.081.679.276.397.589.573.665.4
0.5076.266.458.553.148.673.162.355.651.2
0.2580.671.563.457.752.775.167.261.856.5
lasttoken87.580.772.464.457.292.978.764.154.1

Weighted-loss, checkpoint-300k

DensityLen 18Len 20Len 22Len 24Len 26Hop 5Hop 6Hop 7Hop 8
alltoken93.990.886.179.972.089.885.882.680.0
0.7587.681.575.068.361.591.480.767.359.7
0.5085.277.269.260.352.576.771.866.061.2
0.2584.977.670.664.959.680.673.468.264.0
lasttoken99.598.995.385.876.4100.098.988.976.9

Table 2. Attention-MLP DLA trade off

First-maximal-accuracy

DensityAttention (Classic)MLP (Classic)Attention (Weighted-loss)MLP (Weighted-loss)
alltoken1.148.091.148.09
0.752.495.981.187.05
0.502.356.310.598.09
0.251.119.610.779.16
lasttoken10.08-0.1410.08-0.14

Checkpoint-300k

DensityAttention (Classic)MLP (Classic)Attention (Weighted-loss)MLP (Weighted-loss)
alltoken0.4912.920.4912.92
0.750.7512.450.5612.94
0.501.1113.481.2312.88
0.250.3515.650.6911.90
lasttoken11.900.9911.900.99

We thank the reviewer for raising these insightful points. While our results suggest that density is an important driving factor, we acknowledge that it may interact with location supervision, particularly in structured reasoning tasks, to jointly determine the attention–MLP trade-off. We believe that investigating this interaction is a promising direction for future research and we will explicitly acknowledge this possibility and highlight it in the Limitations and Future Work section.

Responses to other weakness continue in the following comments

Critical weakness 2: We apologize for this confusion. In fact, we had run more runs which all showed similar results, and omitted to add the aggregated results in the appendix. The revised version will also contain the averaged results from 3 seeds. Across the three seeds, the results hold: intermediate densities converge fastest, p = 0 generalizes best at step 300k in every seed, and at step 300k the solution is attention-dominated for p = 0 and MLP-dominated for p ≥ 0.25 (see Table 1 below).

Table 1. Mean accuracy ± sd over three seeds. Steps: first maximal accuracy model. LC: long-context accuracy (%), 18–26 lines × 5–8 hops. d_attn − d_mlp: direct contributions to the answer margin (positive: attention dominates).

pSteps (k)LC, first max.LC, step 300kd_attn − d_mlp, first max.d_attn − d_mlp, step 300k
1102 ± 2675.9 ± 14.279.5 ± 12.4−5.2 ± 2.6−11.5 ± 1.3
0.7529 ± 663.6 ± 14.171.8 ± 4.2−3.9 ± 1.1−11.9 ± 0.2
0.535 ± 272.0 ± 5.985.2 ± 1.5−3.6 ± 0.3−11.3 ± 1.1
0.2535 ± 971.8 ± 8.479.6 ± 5.4−2.1 ± 5.6−14.1 ± 1.2
0137 ± 282.3 ± 8.695.9 ± 4.29.5 ± 1.011.1 ± 2.1

Regarding checkpoint selection, Table 2 compares several selection rules on the run reported in the paper, which isolates the effect of the rule from seed variability. Long-context accuracy is non-monotonic in p for every rule. Regardless of the stopping rule, we can see that the best models are significantly dominated by models with low supervision density.

Table 2. Long-context accuracy (%) of the checkpoint chosen by each rule, mean ± sd over 3 seeds, and the best density of each run. Lowest validation loss: the saved checkpoint with the lowest validation loss on the answer (200k–300k).

Selection rulep = 1p = 0.75p = 0.5p = 0.25p = 0Best p, run 1 (paper)Best p, run 2Best p, run 3
First 100% validation accuracy (paper)75.9 ± 14.263.6 ± 14.172.0 ± 5.971.8 ± 8.482.3 ± 8.60.2501
Lowest validation loss78.9 ± 13.472.0 ± 4.484.9 ± 1.683.0 ± 0.795.9 ± 4.2000
40k steps34.7 ± 0.773.2 ± 14.480.2 ± 6.265.5 ± 26.837.2 ± 1.10.750.250.5
100k steps77.3 ± 11.872.9 ± 8.987.8 ± 5.487.1 ± 1.841.6 ± 1.90.250.250.5
140k steps81.0 ± 13.275.5 ± 6.587.2 ± 1.488.0 ± 2.877.6 ± 14.50.250.250
200k steps81.1 ± 13.572.8 ± 6.286.7 ± 0.583.5 ± 0.691.3 ± 7.60.500
300k steps79.5 ± 12.471.8 ± 4.285.2 ± 1.579.6 ± 5.495.9 ± 4.2000

Critical weakness 3 (and answer to Q2): To demonstrate more convincingly the proposed distinction between the two mechanisms, we target the specific attention weights that follow the hop-chain. For instance, in a 4-hop sample, we specifically target either hop 1, 2, 3 or 4. Following our hypothesis, these independent ablations should impair the performance of the attention-dominated (p=0) solution in a uniform manner; that is, irrespective of which hope is targeted the model should drop its performance. In contrast,MLP-dominated (i.e., p=0.25-1) models should not display this uniform performance drop. Indeed, the latter should be impacted by the distinct hop ablations in a much more random fashion. Crucially, our novel targeted-ablation experiment reveals the hypothesized effect:

Maskp = 0p = 0.25p = 0.5p = 0.75p = 1
No ablation1.0001.0001.0001.0001.000
Ablation hop 40.5720.9880.9880.8440.220
Ablation hop 30.4760.7841.0000.9960.792
Ablation hop 20.3960.8720.9921.0000.984
Ablation hop 10.5280.8480.6400.7440.856

We thank the reviewer for pointing out this weakness and allowing us to provide this convincing analysis, which we are happy to incorporate to our revised version.

Critical weakness 4: We apologize for this issue; the text was confusing and we will fix it. The appendix uses two distinct readouts that answer different questions. The logit lens, and the per-layer margin M(ℓ)M^{(\ell)} derived from it, normalizes each partial residual with its own LayerNorm statistics, as is standard. Direct logit attribution instead uses the final-layer scale for all components, which is necessary to allow for the additive decomposition of the final margin MM into attention and MLP contributions. The two coincide only at the last layer (TL=MT_L = M up to the bias constant). In the revision we will make this distinction explicit, correct the notation where TℓT_\ell is related to M(ℓ)M^{(\ell)}, and state in each figure caption which convention is used.

Responses to questions continue in the following comments

Answer to Q3: We thank the reviewer for this question, and investigated what caused this drop in performance for the answer-only supervision model (p=0). We found that the issue is not in information transmission through the hops. In fact, the correct answer reaches the final token location. The issue is located at the readout of the answer, where more distractors induce a drop in the performance. In particular, if we focus on programs with 26 lines and 8 hops, in 63% of the errors, the correct digit is present in an intermediate layer but is overwritten in the last layers, and 98% of the errors are another line's digit. We will add this explanation in the revised version.

Dear reviewer, given the approaching deadline for the author-reviewer discussion period, we hope you find the time to evaluate our response, and thank you again for your time and interest in our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.