- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether changing the training loss can improve long-range recall in Mamba-2 without extending the gradient window. It gives extra weight to rare tokens that are absent from the recent context. The authors compare standard and reweighted losses under chunked and full backpropagation, using two model sizes trained on PG-19 and evaluated on RULER across three seeds. At 1.3B, reweighting improves variable tracking from 18.3% to 33.1% and 32K needle retrieval from 67.7% to 94.0%. Retrieval also improves at 370M. These gains come with higher perplexity and lower accuracy on aggregation tasks.
Critical strengths
-
The paper shows clear recall gains without extending the gradient window. At 1.3B, reweighting improves variable tracking from 18.3% to 33.1% and 32K needle retrieval from 67.7% to 94.0% across three training seeds. Needle retrieval also improves at 370M, strengthening the evidence that the finding holds beyond one model size.
-
The weighting rule is simple and supported by useful ablations. It uses token frequency and recent context, without a reference model or an extra model pass. Comparisons with random weighting and rarity-only weighting support the finding that which tokens receive extra weight matters.
Critical weaknesses
-
The evaluation leaves a risk of selection bias. Page 5 says the mask variant and primary metric were chosen using Set A, which also supplies the main results. An independent Set B is described on page 16, but its results are not reported. The reproducibility statement also claims that seed-level values are provided for every table, although Appendix A.2 omits FWE and ablation results. These gaps limit independent confirmation and checking of the reported findings.
-
The comparisons do not fully isolate gradient length. Chunked and full-backpropagation runs use different kernels and checkpointing settings. The reweighted 370M runs also use different loss normalization (page 5). Furthermore, Table 15 labels a reweighted comparison “path only,” although its older full-backpropagation checkpoint also differs in normalization (pages 15–17). The authors disclose these issues, but matched training controls and analyses using the corrected checkpoints are still needed to support the interpretation.
Questions for authors
-
How were the mask variant and primary metric selected using Set A? Did this involve comparing results from trained models?
-
Were the final models evaluated on the independent Set B? If so, please provide the results for the four main training conditions.
-
Can you provide a training comparison that changes only gradient truncation while keeping the kernels, checkpointing, and loss normalization fixed?
-
Do the findings in Tables 14–15 hold for the corrected checkpoints used in Table 1? Does the reweighted comparison labelled “path only” remain similar after matching normalization?
-
Can you provide the missing seed-level FWE and ablation results so that the reported paired confidence intervals can be checked?
3 comments
The Set B results address my concern about relying on Set A alone and confirm that reweighting improves recall on RULER. I am keeping my score at 2 because the experiment that isolates gradient length uses one training seed on Set A (Table 21). In the three-seed comparison on Set B, full backpropagation under reweighting improves variable tracking by 3.1 points and reduces 32K needle retrieval by about 36 points (Table 9). My remaining concern is whether these task-specific effects hold across seeds when the kernel and loss normalization are kept fixed.
Thank you for checking the Set B results and for stating the remaining concern precisely. Both numbers you cite are correct.
Loss normalization is already matched. At 1.3B all four arms use per-chunk normalization (Section 4), so in Tables 1 and 9 the chunked and full-backpropagation arms differ only in the gradient window and the kernel. The kernel has little effect on NIAH@32K: in the seed-42 runs of Table 21 the connected run and the fused arm give 65 vs. 65 (reweighted) and 53 vs. 54 (standard). The 36-point loss is therefore unlikely to be a kernel effect, and the fused arm varies little across seeds (60.0/60.5/60.0 on Set B).
The 3.1-point VT difference is another matter. At seed 42 the connected run and the fused arm differ by 2.2 points of VT@8K, which is of the same size. On Set A the full-backpropagation arm minus the chunked arm is −0.1 over three seeds. We do not claim this difference.
This does not replace the experiment you ask for. We will add the connected runs for seeds 43 and 44 under both objectives, on Set A and Set B, to the next version. Thank you again.
Response to Reviewer 2siN
We thank the reviewer for the careful reading, and in particular for catching the "path only" label in Table 15 of the submission.
Q1. How were the mask variant and primary metric selected on Set A? Yes, both choices used trained models, at the level of a pilot. The mask family and the constants were fixed before any training run of this study. We then trained two mask variants for 0.1 epoch on one seed and evaluated them on Set A. Based on this pilot, we made two choices before the one-epoch runs of the reweighted objective. First, the primary metric became VT@8K. We had planned to use NIAH@8K, but it was 100 for every arm and for the untrained model. Second, we chose rare ∧ non-recent over a variant that adds a third condition (no occurrence in a previous chunk). The two were indistinguishable on the pilot (VT@8K 22.2 vs. 22.6; both runs are in results/ruler.csv of the supplementary material), and we took the one with fewer conditions. After the first one-epoch results, the ablation arms were evaluated on Set A as well.
Set B is a second instance set that we had planned from the start for re-measuring the headline results. It was generated after the one-epoch Set A results were known, and no decision was made on it. We report it for all four conditions. The Evaluation paragraph (page 6) and Appendix A.5 of the revision now give this account.
Q2. Set B for the four main conditions. We evaluated all four conditions on Set B. The results are in item A of the general response, with seed-level values in new Appendix Table 9 (VT, FWE and CWE at 8K, NIAH at 16K/32K). The remaining tasks and lengths are in the supplementary material (results/ruler.csv). The directions match Set A with one exception reported in item A: under the reweighted objective, full backpropagation minus chunked is +3.1 points of VT@8K on Set B and −0.1 on Set A.
Q3. A training comparison that changes only gradient truncation. We ran this for both objectives (item B of the general response, new Appendix Table 21). The two runs of a pair share the cache path, the kernels, the per-chunk normalization and the data order. The carried state is connected instead of detached, which is one flag in the launch command. At seed 42 on Set A, NIAH@32K goes from 65 (detached) to 53 (connected) under the standard objective and from 95 to 65 under the reweighted one. The fused full-backpropagation arm gives 54 and 65, and the other metrics in the table of item B differ from it by at most 2.2 points. The gradient window therefore accounts for the 12-point and 30-point losses at 32K. There is one seed per pair.
Q4. Tables 14–15 of the submission on the corrected checkpoint and the "path only" label. The reviewer is right that the starred rows used the whole-sequence-normalized checkpoint, so "path only" was not an accurate label for them. We recomputed the three tables (14–16 in the revision, 13–15 in the submission) with the per-chunk-normalized checkpoint used in Table 1. The gradient-norm statistics change only in the third decimal place. For the reweighted full-backpropagation arm the mean goes from 0.162 to 0.164, and no statistic changes by more than 0.003. The decay-parameter and weight-movement row is unchanged at the precision shown. The update-direction cosines are unchanged as well: 0.981 for the reweighted chunked-vs-full pair and 0.837 for the standard-vs-reweighted pair under full backpropagation. The revised tables use the corrected checkpoint, so the "path only" label is now accurate.
Q5. Seed-level FWE and ablation values. Appendix Table 8 now has FWE@8K per seed for the four main conditions, and VT@8K, NIAH@32K, FWE@8K and perplexity per seed for the rare-only and density-matched random arms. The ablation intervals can be recomputed from the table. The reproducibility statement is corrected. It now lists what Appendix A.2 contains and points to the supplementary ledger for the remaining per-seed values (370M FWE and the secondary tasks).
On the 370M normalization confound we have no new runs. The revision keeps the explicit caveat and does not draw a gradient-path conclusion at 370M under the reweighted objective.