acceptodds
Under review as a conference paper at ICLR 2027

Effects of Token-level Loss Reweighting on Long-Range Recall in Chunked Training of Mamba-2

Abstract

Truncated backpropagation through time trains a stateful sequence model by carrying the state across chunks while stopping gradients at chunk boundaries, which biases learning toward short-range dependencies. A standard approach is to use a longer gradient window. In this study, we investigate whether extending the gradient path is necessary for improving long-range recall. Our hypothesis is that, with the state carried forward, which tokens the objective emphasizes can matter more for long-range recall than how far the gradient reaches. We test this hypothesis in continued training of Mamba-2 (1.3B and 370M) on PG-19, varying two choices independently: (1) whether the loss is the standard cross-entropy or is reweighted toward rare tokens absent from the recent context, and (2) whether gradients stop at chunk boundaries or flow through the whole sequence. We deliberately use a simple reweighting scheme as a probe of the hypothesis: a deterministic mask on token ids with a single weight set by a static calibration rule. With the gradient window fixed at one chunk, reweighting raises variable-tracking accuracy at the 8K training length from 18.3 to 33.1 and needle retrieval at four times the training length from 67.7 to 94.0. At 1.3B and the 8K training length, full backpropagation gives no statistically significant improvement under the standard objective. The gains are task-dependent and trade off against perplexity and accuracy on an aggregation task (frequent-word extraction). In this setting, long-range recall improved by changing what the objective emphasizes, without extending the gradient path.

Primary area:
foundation or frontier models, including LLMs
Keywords:
state-space models, Mamba-2, long-context, chunked training, truncated backpropagation through time, loss reweighting, long-range recall, RULER

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2siNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether changing the training loss can improve long-range recall in Mamba-2 without extending the gradient window. It gives extra weight to rare tokens that are absent from the recent context. The authors compare standard and reweighted losses under chunked and full backpropagation, using two model sizes trained on PG-19 and evaluated on RULER across three seeds. At 1.3B, reweighting improves variable tracking from 18.3% to 33.1% and 32K needle retrieval from 67.7% to 94.0%. Retrieval also improves at 370M. These gains come with higher perplexity and lower accuracy on aggregation tasks.

Critical strengths
  1. The paper shows clear recall gains without extending the gradient window. At 1.3B, reweighting improves variable tracking from 18.3% to 33.1% and 32K needle retrieval from 67.7% to 94.0% across three training seeds. Needle retrieval also improves at 370M, strengthening the evidence that the finding holds beyond one model size.

  2. The weighting rule is simple and supported by useful ablations. It uses token frequency and recent context, without a reference model or an extra model pass. Comparisons with random weighting and rarity-only weighting support the finding that which tokens receive extra weight matters.

Critical weaknesses
  1. The evaluation leaves a risk of selection bias. Page 5 says the mask variant and primary metric were chosen using Set A, which also supplies the main results. An independent Set B is described on page 16, but its results are not reported. The reproducibility statement also claims that seed-level values are provided for every table, although Appendix A.2 omits FWE and ablation results. These gaps limit independent confirmation and checking of the reported findings.

  2. The comparisons do not fully isolate gradient length. Chunked and full-backpropagation runs use different kernels and checkpointing settings. The reweighted 370M runs also use different loss normalization (page 5). Furthermore, Table 15 labels a reweighted comparison “path only,” although its older full-backpropagation checkpoint also differs in normalization (pages 15–17). The authors disclose these issues, but matched training controls and analyses using the corrected checkpoints are still needed to support the interpretation.

Questions for authors
  1. How were the mask variant and primary metric selected using Set A? Did this involve comparing results from trained models?

  2. Were the final models evaluated on the independent Set B? If so, please provide the results for the four main training conditions.

  3. Can you provide a training comparison that changes only gradient truncation while keeping the kernels, checkpointing, and loss normalization fixed?

  4. Do the findings in Tables 14–15 hold for the corrected checkpoints used in Table 1? Does the reweighted comparison labelled “path only” remain similar after matching normalization?

  5. Can you provide the missing seed-level FWE and ablation results so that the reported paired confidence intervals can be checked?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2siN

We thank the reviewer for the careful reading, and in particular for catching the "path only" label in Table 15 of the submission.

Q1. How were the mask variant and primary metric selected on Set A? Yes, both choices used trained models, at the level of a pilot. The mask family and the constants were fixed before any training run of this study. We then trained two mask variants for 0.1 epoch on one seed and evaluated them on Set A. Based on this pilot, we made two choices before the one-epoch runs of the reweighted objective. First, the primary metric became VT@8K. We had planned to use NIAH@8K, but it was 100 for every arm and for the untrained model. Second, we chose rare ∧ non-recent over a variant that adds a third condition (no occurrence in a previous chunk). The two were indistinguishable on the pilot (VT@8K 22.2 vs. 22.6; both runs are in results/ruler.csv of the supplementary material), and we took the one with fewer conditions. After the first one-epoch results, the ablation arms were evaluated on Set A as well.

Set B is a second instance set that we had planned from the start for re-measuring the headline results. It was generated after the one-epoch Set A results were known, and no decision was made on it. We report it for all four conditions. The Evaluation paragraph (page 6) and Appendix A.5 of the revision now give this account.

Q2. Set B for the four main conditions. We evaluated all four conditions on Set B. The results are in item A of the general response, with seed-level values in new Appendix Table 9 (VT, FWE and CWE at 8K, NIAH at 16K/32K). The remaining tasks and lengths are in the supplementary material (results/ruler.csv). The directions match Set A with one exception reported in item A: under the reweighted objective, full backpropagation minus chunked is +3.1 points of VT@8K on Set B and −0.1 on Set A.

Q3. A training comparison that changes only gradient truncation. We ran this for both objectives (item B of the general response, new Appendix Table 21). The two runs of a pair share the cache path, the kernels, the per-chunk normalization and the data order. The carried state is connected instead of detached, which is one flag in the launch command. At seed 42 on Set A, NIAH@32K goes from 65 (detached) to 53 (connected) under the standard objective and from 95 to 65 under the reweighted one. The fused full-backpropagation arm gives 54 and 65, and the other metrics in the table of item B differ from it by at most 2.2 points. The gradient window therefore accounts for the 12-point and 30-point losses at 32K. There is one seed per pair.

Q4. Tables 14–15 of the submission on the corrected checkpoint and the "path only" label. The reviewer is right that the starred rows used the whole-sequence-normalized checkpoint, so "path only" was not an accurate label for them. We recomputed the three tables (14–16 in the revision, 13–15 in the submission) with the per-chunk-normalized checkpoint used in Table 1. The gradient-norm statistics change only in the third decimal place. For the reweighted full-backpropagation arm the mean goes from 0.162 to 0.164, and no statistic changes by more than 0.003. The decay-parameter and weight-movement row is unchanged at the precision shown. The update-direction cosines are unchanged as well: 0.981 for the reweighted chunked-vs-full pair and 0.837 for the standard-vs-reweighted pair under full backpropagation. The revised tables use the corrected checkpoint, so the "path only" label is now accurate.

Q5. Seed-level FWE and ablation values. Appendix Table 8 now has FWE@8K per seed for the four main conditions, and VT@8K, NIAH@32K, FWE@8K and perplexity per seed for the rare-only and density-matched random arms. The ablation intervals can be recomputed from the table. The reproducibility statement is corrected. It now lists what Appendix A.2 contains and points to the supplementary ledger for the remaining per-seed values (370M FWE and the secondary tasks).

On the 370M normalization confound we have no new runs. The revision keeps the explicit caveat and does not draw a gradient-path conclusion at 370M under the reweighted objective.

The Set B results address my concern about relying on Set A alone and confirm that reweighting improves recall on RULER. I am keeping my score at 2 because the experiment that isolates gradient length uses one training seed on Set A (Table 21). In the three-seed comparison on Set B, full backpropagation under reweighting improves variable tracking by 3.1 points and reduces 32K needle retrieval by about 36 points (Table 9). My remaining concern is whether these task-specific effects hold across seeds when the kernel and loss normalization are kept fixed.

Thank you for checking the Set B results and for stating the remaining concern precisely. Both numbers you cite are correct.

Loss normalization is already matched. At 1.3B all four arms use per-chunk normalization (Section 4), so in Tables 1 and 9 the chunked and full-backpropagation arms differ only in the gradient window and the kernel. The kernel has little effect on NIAH@32K: in the seed-42 runs of Table 21 the connected run and the fused arm give 65 vs. 65 (reweighted) and 53 vs. 54 (standard). The 36-point loss is therefore unlikely to be a kernel effect, and the fused arm varies little across seeds (60.0/60.5/60.0 on Set B).

The 3.1-point VT difference is another matter. At seed 42 the connected run and the fused arm differ by 2.2 points of VT@8K, which is of the same size. On Set A the full-backpropagation arm minus the chunked arm is −0.1 over three seeds. We do not claim this difference.

This does not replace the experiment you ask for. We will add the connected runs for seeds 43 and 44 under both objectives, on Set A and Set B, to the next version. Thank you again.

Reviewer_5j39OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the long-range recall capabilities of stateful sequence models under a "chunked training" regime—specifically, Truncated Backpropagation Through Time (TBPTT), where the model state is propagated across chunks while gradients are truncated at chunk boundaries. The authors hypothesize that, given the propagation of the model state, the specific tokens emphasized by the objective function may have a greater impact on long-range recall than the distance over which gradients can be backpropagated.

Critical strengths

The experimental design is clean, and the statistical reporting is honest. The two factors are orthogonal by design (the mask depends solely on the token ID and is independent of the gradient path). For any given seed, all experimental arms utilize identical token streams, enabling paired comparisons. Constants are fixed prior to training, and the experiments are documented via launch banners and a ledger to ensure traceability. The authors proactively report negative results, instances of overly wide confidence intervals, and a bug related to the normalization scope—along with the subsequent re-runs—which is rare in similar work and highly commendable.

The research question itself is meaningful. The decomposition of "gradient window" versus "target emphasis" offers valuable insights for long-context training in SSMs and linear attention models. Furthermore, the gradient decomposition presented in Appendix A.5—which separates the kernel, detach operations, and checkpointing—is executed with great precision.

Critical weaknesses

There is an inconsistency between the training and evaluation kernels for the "full BPTT" variant. While the full BPTT models are trained using a fused kernel, evaluation is performed exclusively via the chunked path. The appendix indicates that, under the bf16 precision setting, the discrepancy in logits between these two paths can reach as high as 8.6. This could directly account for the performance degradation observed in the full BPTT variant on the NIAH-s2 (Needle In A Haystack - sequence length 2) and extrapolation tasks. I recommend retraining the full BPTT variant using the "cache path" approach without detaching the computation graph—ensuring that the only difference between the two settings is the detachment step. Consequently, the claim that "full BPTT impairs extrapolation" should not be presented as a definitive conclusion at this stage.

The conclusion that "gradient paths are unnecessary" is limited in scope and lacks sufficient statistical power. Each sequence contains only three chunk boundaries; the training process consists of continued training for a single epoch with a small learning rate (resulting in a cosine similarity of 0.98 between the full-path and path-only updates); and the analysis relies on only three random seeds (with a 95% confidence interval of [−5.0, +10.6] for the VT metric). Furthermore, both the primary metrics and the masking variants were selected based on "Set A"—the same dataset used for reporting the final results—without providing any evaluation on an independent "Set B."

Questions for authors

Do the reweighted models "hallucinate" seven-digit numbers more frequently on instances where the needle is absent?

Does the collapse in recall during extrapolation persist if the "full" arm is retrained with only the detach operation applied?

What are the results for the main table on Set B? Are they consistent with the conclusions drawn from Set A?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5j39

We thank the reviewer for the precise reading of Appendix A.5 and for the three questions.

Kernel inconsistency (Weakness 1). The reviewer is right that the full arms are trained on the fused path and evaluated on the chunked path. We tested this in two ways. First, we re-evaluated the same checkpoints with the fused prefill, which is the training-time path of the full arms (new Appendix Table 20). It reproduces the chunked-prefill numbers within one point on every seed and length. NIAH@32K is 54/57/57 under both paths for the standard full arm and 65/65/64 for the reweighted one. Second, we ran the retraining that the reviewer recommends, with the cache path and the state not detached (Q2 below). The lower extrapolation of the full-backpropagation arms is therefore attributable to the gradient window, with the qualification that the detach-only pairs are one seed each. For clarity, NIAH-s2 in our tables is RULER's second single-needle variant, with one needle in the Paul Graham essay haystack. The "s2" does not refer to a sequence length.

Scope and power (Weakness 2). We agree with every element, and the revision states the null result on the gradient path with its interval where the result is given. The paper's title claim does not rest on this null result. It rests on the comparison of reweighted-chunked with standard-full, which is +11.9 [+6.8, +17.0] on Set A and +8.4 [+4.1, +12.8] on Set B for VT@8K, and +38.0 [+30.5, +45.5] and +52.0 [+48.7, +55.3] for NIAH@32K.

Thank you for also raising the issue of selection. The mask family and the constants were fixed before any training run of this study. The primary metric and the choice between two mask variants were fixed on a 0.1-epoch pilot evaluated on Set A, before the one-epoch runs. Set B is a second instance set that we had planned from the start for re-measuring the headline results. It was generated after the one-epoch Set A results were known, and no decision was made on it. Details are in our answer to Reviewer 2siN (Q1).

Q1 (hallucinated seven-digit numbers without a needle). We ran this control (new Appendix Table 24). We removed only the needle sentence from the Set A instances and decoded the base model and all four 1.3B arms (three seeds, n = 100 at 8K and at 32K). The base model and the two standard-objective arms never produce a seven-digit integer. The two reweighted arms do, in 8–31% of the no-needle instances, and in every case (246 of 1,200) it is the literal "1234567", which is the answer of no instance. With the needle present, a seven-digit integer other than the answer occurs in at most 2% of the Set B predictions at 32K and never at 16K and below. So the RULER score is not inflated by guessed numbers. The reweighted objective does make the model more willing to put a placeholder number in the answer slot when there is nothing to recall, and we report this as a side effect in the appendix.

Q2 (does the collapse persist with detach-only retraining?). Yes, under both objectives (item B of the general response, new Appendix Table 21). With the detach as the only change, NIAH@32K goes from 65 to 53 under the standard objective and from 95 to 65 under the reweighted one (seed 42). In both cases the connected run matches the fused full-backpropagation arm of the same seed (54 and 65). It is also close to that arm on VT@8K, FWE@8K and held-out perplexity, with differences of at most 2.2 points. There is one seed per pair, so these are paired point comparisons.

Q3 (Set B main table). The results are in item A of the general response. The directions are preserved with one exception. Under the reweighted objective the gradient path adds +3.1 [+0.9, +5.4] points of VT@8K on Set B (−0.1 on Set A), while NIAH@32K falls from 96.3 to 60.2.

Thank you for the detailed response and the additional analyses. I have one further question.

Stuffed Mamba (Chen et al., 2025), which the paper cites in §2, shows that a recurrent model trained on sequences that are short relative to its state size does not learn to forget, so its state collapses on contexts longer than the training length. The pretrained Mamba-2 checkpoints used here appear to start in exactly this regime: NIAH@16K and NIAH@32K are 0 before continued training (Table 26), the pretrained model is worse with the full context than with a short one on part of the targets (3.48 vs. 2.51 nats), and in every arm about 98% of heads shorten their nominal memory length (Table 15). The paper, however, cites Stuffed Mamba only as background and does not discuss what this starting point means for its conclusions.

Could the authors discuss whether the main findings, namely that the objective matters more than the gradient window, that full backpropagation hurts extrapolation, and that LongCE underperforms, might be specific to repairing state collapse in a checkpoint heavily trained at short context, and to what extent they would expect them to hold for a model that does not start in this regime?

Thank you for this question. We agree that the pretrained checkpoint starts in the regime that Chen et al. describe, and that the paper should have discussed it. Our answer differs for the three findings.

LongCE. This finding is specific to the starting point. LongCE gives a weight below one to the tokens on which the long context hurts, and such tokens are common only in the collapsed model (33.1% of tokens have a weight below 0.5 before training, 0.6% after standard fine-tuning). We do not expect it to hold for a model that uses long context well before fine-tuning.

Full backpropagation and extrapolation. This finding is about where collapse sets in beyond the training length. Every arm has collapsed by 64K (Table 11), and NIAH@32K measures how far the onset is delayed. We do not expect its sign to be general. Under the standard objective, full backpropagation lowers NIAH@32K at 1.3B (67.7 → 56.0) and raises it at 370M (44.3 → 52.0).

The objective at the training length. Here we have indirect evidence that the effect is not only a repair of the collapse. (i) Inside the 8K window the degradation is gone after 0.2 epoch under both objectives (loss with full vs. short context 2.46 vs. 2.48 and 2.63 vs. 2.66, Table 26). At that point the objectives differ by 2.0 points of VT@8K, and after one epoch of the same seed by 14.6. (ii) The effect is present at lengths where the pretrained model shows no collapse (+10.4 points of VT at 1K, +8.9 at 2K; Table 12). (iii) The decay biases move by the same amount in every arm (Table 15). Point (i) rests on one seed, and Table 15 covers only the input-independent part of the decay. For the null effect of the gradient window we have no such evidence, and we do not claim that it holds outside this regime.

We have no model trained at long context from the start, so we cannot test this directly yet. We will add the direct test to the next version: the two-by-two design repeated from a checkpoint without the collapse. The Limitations paragraph of the revision now states this scope. Thank you for pointing us to this.

Reviewer_HiBzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  • This paper studies long-range recall of Mamba2 models on synthetic long-range recall tasks

  • They fine-tuned public pretrained Mamba2 models on PG-19 datasets at sequence length 8192, and probe the resulting long-range recall

  • They propose a token reweighting scheme based on upweighting uncommon tokens, and show that it improves downstream accuracy on RULER tasks

  • Surprisingly, they show that even when they use chunk-truncated backpropagation through time (by modifying the backwards pass of the Mamba2 kernel), the effect still persists, suggesting that full-backpropagation through time may not be necessary for good long-range recall

Critical strengths
  • I think the finding the long range recall can improve with truncated backpropagation through time is novel, but requires substantially more develop for me to consider it solid

  • The empirical result that simple rare-token upweighting improves performance is good, but again, is not robust and requires more comparison. The benefits of this method is that it is relatively cheap to implement and doesn't require multiple model forward passes, just offline cheap calculations on the real dataset.

Critical weaknesses
  • Very little analysis of the result, or why it holds is done. As it stands, the paper present a slightly unexpected empirical finding, but there is little to no analysis (empirical or theoretical) backing it up. In particular, I think a further experiment breaking down the cause of the long-range recall improvement would be useful. For example, not needing full BPTT suggests that recall ability can be modified by behaviour near the readout/query: is the better recall because the model learns to avoid putting obstructing key/value items, or because the model learns to modify it's query to do better reading from the Mamba2 hidden state? I think this experiment could be be done by giving the query path gradients from upweighted rare-token loss, and giving the other paths gradients from the non-upweighted loss, and vice-versa.

  • There is very little analysis of how the behaviour of the model changes beyond the synthetic RULER tasks. For example, how do these changes reflect in actual task behaviour on natural language?

  • The analysis the strictly on Mamba2 models. There are a very large family on linear recurrences, and generally speaking, they all suffer from poor long-range recall. However, some methods, e.g. DeltaNet, are known to be better recallers than Mamba2. Does this effect generalize to all other linear recurrences?

  • If the main contribution of this paper is the proposed reweighting scheme for long-range recall, it needs to be compared to prior reweighting schemes in the literature, such as [1,2,3]. If the main contribution of the paper is studying the recall behaviour of Mamba2 in light of the TBPTT finding, then the methods in [1,2,3] should be evaluated with TBPTT, to see if they also have the same effect.

[1] Fang, L., et al. (2025). What is Wrong with Perplexity for Long-context Language Modeling? ICLR 2025. Introduces LongCE. https://arxiv.org/abs/2410.23771

[2] Helm, F., Daheim, N., and Gurevych, I. (2025). Token Weighting for Long-Range Language Modeling. Findings of NAACL 2025. https://aclanthology.org/2025.findings-naacl.79/

[3] Lin, Z., et al. (2024). Rho-1: Not All Tokens Are What You Need. arXiv preprint. https://arxiv.org/abs/2404.07965

Questions for authors

See the weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer HiBz

We thank the reviewer for the detailed criticism.

One clarification on the summary: we did not modify the backward pass of the Mamba-2 kernel. Both arms use the stock implementation, and the chunked arms detach the carried state between chunks.

Analysis of why the result holds. We agree that the paper shows the effect but explains it only in part, and we ran two experiments (new Appendix Table 25). The first is a state exchange. One checkpoint (the writer) reads the prompt except its last 64 tokens, which always contain the whole question, and produces the carried state. The other (the reader) receives that state, reads those 64 tokens and generates the answer. Writer and reader are the standard and the reweighted chunked models of the same seed (three seeds, n = 100 per task).

writer → readerNIAH-s1@32KVT@8KFWE@8K
standard → standard67.3 ± 7.820.0 ± 0.950.2 ± 0.4
reweighted → reweighted93.7 ± 1.533.3 ± 0.640.4 ± 1.1
reweighted → standard100.0 ± 0.027.7 ± 2.043.6 ± 1.0
standard → reweighted65.7 ± 8.625.3 ± 1.538.6 ± 0.5

The needle gain is entirely in what is written to the state. The standard model retrieves every needle from a state written by the reweighted model, and the reweighted model does no better than the standard model on a state written by the standard model (65.7 vs. 67.3). The VT gain is shared between the two sides, and the FWE loss mostly follows the reader. The second experiment is the path-wise gradient split that the reviewer proposes. We implemented it, and with different losses on the two sides training diverged within 0.2 epoch under both assignments. We report it as a negative result and draw no conclusion from it.

Behaviour on natural language. We binned every target of the held-out PG-19 stream by the mask's own coordinates (rare or common, and recent, far or never seen) and compared the per-token loss of all arms (new Appendix Tables 22 and 23). The reweighted objective lowers the loss of every rare bucket and raises that of every common bucket. The gain on rare targets seen earlier (−0.52 to −0.66 nats) is close to the gain on rare targets never seen (−0.62), so on natural text the objective is mostly a shift of probability toward rare types. A recall gain exists but is small. Within capitalised names, accuracy for names last seen at least 2048 tokens earlier rises from 43.7% to 47.5% and does not change for never-seen names. The Limitations section now states that the synthetic gains have no natural-text counterpart of that size. The gradient path moves no bucket by more than 0.04 nats under either objective.

Other linear recurrences. We agree that the result is shown for one architecture. We chose Mamba-2 because its fixed-size state is the model's only memory of the prefix, which is the premise of the hypothesis. Whether the effect transfers to DeltaNet-style models is open, and we state this as a limitation.

Prior reweighting schemes. Of the two readings the reviewer offers, ours is the second, the hypothesis test. We therefore ran LongCE [1] and the dense token weighting of Helm et al. [2] under chunked training, re-implemented from the released code (new Appendix Table 26). Each row is one run of 0.2 epoch with seed 42.

objective (1.3B, chunked, 0.2 epoch)VT@8KFWE@8KNIAH@16KNIAH@32Kheld-out loss
pretrained13.41.0003.172
standard17.050.7100802.354
rare ∧ non-recent (ours)19.039.099952.518
LongCE5.851.739292.557
LongCE with weight ≥ 1 (our diagnostic variant)6.252.3100882.377
dense weighting of [2]9.248.0100932.366

LongCE stays far below the standard objective on retrieval beyond the training length. The pretrained Mamba-2 is worse with the full context than with a short one (3.48 vs. 2.51 nats), and LongCE gives exactly those tokens a weight below one, so the error that fine-tuning must remove is the one it discounts. Bounding the weight below by one restores retrieval. The dense weighting of [2] has a lower bound by construction and does not fail in this way. On NIAH@32K it is within two points of our mask (93 vs. 95), which one run cannot resolve, so we do not claim that our mask is better than [2] on retrieval. The mask needs no additional forward pass, because its weights come from token ids alone. On VT@8K all three of these runs are below the pretrained model, and we do not have an explanation for this. These are single runs of 0.2 epoch because the discussion period did not allow more. We will add full-budget runs (one epoch, three seeds) to the next version.

Rho-1 [3] selects tokens by the excess loss over a reference model trained on a curated corpus. Neither such a corpus nor released training code exists for our setting, so a comparison under the conditions of [3] is not possible.

I appreciate the new results from the authors and while I think they contribute to making the paper more complete, I still do not think they constitute a substantial and complete finding sufficient for ICLR. As mentioned in the original review, I feel that the following questions are still left open:

  1. A proper working explanation for why this is the case, and more thorough science behind the finding
  2. "The Limitations section now states that the synthetic gains have no natural-text counterpart of that size." I think this is a significant limitation. I encourage the authors to look at more natural language evaluation metrics rather than just looking at perplexity on different token groups on pretraining data. For example, how does this translate to natural language recall tasks (e.g. SWDE, FDA, DROP, TQA)
  3. Broaden this to other linear recurrences. If the authors are able to explain and expand this finding to the more general class of linear recurrences, then the findings of this paper could have more broader implications for the design of these recurrences at large

Thank you for reading the new results and for stating clearly what remains open. The three points give us a clear direction for the next version.

We will evaluate all arms on the natural-language recall tasks you name (SWDE, FDA, DROP and TQA) and add the results. We do not report them here only because we could not validate the protocol within the discussion period. We would also like to extend the study to other linear recurrences, starting with a DeltaNet-style model, and to decide between the two candidate explanations given in the Discussion.

Thank you again.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful readings. The revised PDF adds Appendix Tables 9 and 20–26 and revises Tables 8 and 14–16. Reviewers 5j39 and 2siN both asked for results on the independent instance set (Set B) and for a training comparison that isolates gradient truncation from kernel, checkpointing and normalization. We give these two results here. Everything else is in the individual responses.

A. Set B, all four 1.3B conditions, three seeds (new Appendix Table 9). Set B was generated with a different generator seed, with n = 200 per task and length, and was frozen before use. No metric, mask or training decision was made on it. Means ± seed s.d. (Set A in parentheses):

armVT@8KNIAH@32KFWE@8K
standard + chunked20.5 ± 2.8 (18.3 ± 1.2)60.3 ± 9.6 (67.7 ± 7.4)41.9 ± 0.2 (50.1 ± 1.0)
standard + full backprop.20.7 ± 1.9 (21.1 ± 3.1)44.3 ± 0.3 (56.0 ± 1.7)40.1 ± 0.4 (44.6 ± 1.2)
rare ∧ non-recent + chunked29.2 ± 0.1 (33.1 ± 1.0)96.3 ± 1.6 (94.0 ± 1.7)37.3 ± 0.5 (40.0 ± 0.9)
rare ∧ non-recent + full backprop.32.3 ± 0.8 (33.0 ± 1.7)60.2 ± 0.3 (64.7 ± 0.6)33.5 ± 0.3 (34.9 ± 0.5)

The comparisons of Table 1 keep their direction on Set B, with one exception that we state here and in the caption of Table 9. Under the reweighted objective, full backpropagation minus chunked on VT@8K is +3.1 [+0.9, +5.4] on Set B, against −0.1 [−6.3, +6.1] on Set A.

The intervals below are paired by seed (95% t-intervals, df = 2) and are computed from the per-seed values of Appendix Tables 8 and 9.

comparisonSet BSet A
objective effect under chunked training, VT@8K+8.7 [+2.0, +15.4]+14.7 [+11.2, +18.2]
gradient-path effect under the standard objective, VT@8K+0.3 [−2.1, +2.6]+2.8 [−5.0, +10.6]
reweighted-chunked minus standard-full, VT@8K+8.4 [+4.1, +12.8]+11.9 [+6.8, +17.0]
reweighted-chunked minus standard-full, NIAH@32K+52.0 [+48.7, +55.3]+38.0 [+30.5, +45.5]

So on Set B the gradient path adds about three points of VT@8K on top of the reweighted objective, a third of the objective's own effect, and costs 36 points of NIAH@32K. Table 9 gives VT, FWE and CWE at 8K and NIAH at 16K/32K per seed. The remaining Set B tasks and lengths are in the supplementary material (results/ruler.csv).

B. A training-level comparison that changes only the detach (new Appendix Table 21). We added a mode that runs the full-backpropagation arm through the same cache path as the chunked arms. The carried state is left connected instead of detached, with per-chunk activation checkpointing (gradient effect 4e-8, Table 19) and the same per-chunk normalization. Data order, seed, learning rate, batch, number of steps and kernel are identical to the chunked arm of the same seed, and the launch command differs from that arm's by one flag. Both pairs are done (seed 42, Set A, n = 100).

run (seed 42)VT@8KFWE@8KNIAH@16KNIAH@32Kheld-out ppl
standard, detached (chunked arm)19.451.01006510.34
standard, connected (new)23.844.31005310.32
standard, fused full backprop.23.843.31005410.32
reweighted, detached (chunked arm)34.040.71009512.26
reweighted, connected (new)35.034.7986512.25
reweighted, fused full backprop.32.834.31006512.24

Changing only the detach reproduces the full-backpropagation arm on every metric under both objectives, including the 12-point loss at 32K under the standard objective and the 30-point loss under the reweighted one. The gradient window is therefore what separates the two paths, not the kernel, the checkpointing or the normalization. On the natural-text probe (new Appendix Table 22) the connected run agrees with the fused arm within 0.002 nats in every bucket under the standard objective and within 0.004 nats under the reweighted one. Under the reweighted objective the detached arm is 0.019–0.026 nats higher on the three rare-target buckets with an earlier occurrence.

The limits are these. There is one seed per pair, so these are paired point comparisons and not intervals (seed s.d. of the original arms: VT@8K 1.2–3.1, NIAH@32K 1.7–7.4 points). The differences of at most 2.2 points between the connected and fused runs are within seed noise. The 12-point and 30-point differences between connected and detached at 32K are not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.