- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ReMem, a training-free inference-time mechanism designed to mitigate performance degradation during recurrent-depth extrapolation in Looped Language Models (LoopLMs). Through a controlled state re-entry diagnostic, the authors demonstrate that intermediate recurrent states preserve reusable latent computation. To leverage this, ReMem maintains a rolling buffer of recent recurrent states and fuses them into the candidate state using feature-wise weights determined by contrastive evidence from the frozen language modeling head (SCFA). Across 11 benchmarks on Ouro-1.4B and Ouro-2.6B, ReMem improves capability retention at extended recurrent depths () without adding trainable parameters or extra Transformer evaluations.
Critical strengths
- The motivation of viewing historical recurrent states as reusable latent computation is conceptually clean, and the one-shot re-entry diagnostic in Section 2 provides convincing empirical motivation.
- The control interventions in Figure 3 (Gaussian random, anti-restoration, shuffled history, and feature permutation) effectively verify that the observed re-entry gains originate from structured historical information.
- The method is training-free, operates strictly within compute-matched Transformer evaluation budgets, and includes thorough execution-level efficiency and memory benchmarking in Appendix F.
- Empirical gains across language understanding, math, and coding benchmarks are consistent across two model sizes and become pronounced at deep recurrence regimes ().
Critical weaknesses
- The formulation of State-Contrastive Feature Allocation (SCFA) lacks theoretical grounding: the element-wise product in Eq. 8 is introduced heuristically, without validation demonstrating that this specific metric correlates with true historical feature utility.
- The sequence-level fusion relies on a restrictive spatial assumption: Eq. 8 derives allocation weights solely from the single boundary token and broadcasts them uniformly across all sequence positions (line 289), which leaves unverified whether output-layer contrast at the final token suitably guides historical feature mixing for preceding tokens.
- Statistical confidence is missing for marginal gains: while the re-entry diagnostics report bootstrap confidence intervals, the main benchmark results (Tables 1–3) report single point estimates without sample-level confidence bounds or significance testing, which makes small differences (such as MMLU at or high-variance AMC23 scores) difficult to confirm as real improvements.
Questions for authors
- Can you provide empirical validation for the choice of in Eq. 8, such as comparing it against simpler feature selectors (e.g., token-wise absolute activation magnitude) or analyzing its alignment with oracle feature importance?
- Could you report bootstrap confidence intervals or significance tests across test instances for the main results in Table 1 to confirm whether improvements at shallow extrapolation depths () are statistically significant?
- What drives the performance decline observed when expanding the memory window to in Table 9: does blending older latent states exacerbate representation drift?
3 comments
Author Response (Part 2/3)
[W2] The sequence-level fusion relies on a restrictive spatial assumption: Eq. 8 derives allocation weights solely from the single boundary token and broadcasts them uniformly across all sequence positions (line 289), which leaves unverified whether output-layer contrast at the final token suitably guides historical feature mixing for preceding tokens.
[R2] We thank the reviewer for this important question. We agree that the submission does not verify the boundary broadcast in isolation. We clarify its scope and test it against a position-specific alternative below.
The broadcast applies only to positions processed in one block, i.e., the prompt and multiple-choice scoring. During generation, the evidence is recomputed whenever the prediction boundary advances and fusion acts only on the current active block, typically one token, so each generated position already uses an allocation derived from its own state.
Comparison with position-specific allocation. To test the assumption, we let every prompt position derive its allocation from its own state via Eqs. 7–9 instead of receiving the boundary allocation, keeping all other settings unchanged.
Table D. Boundary broadcast vs. position-specific allocation over the prompt.
| Depth | Spatial allocation | MC Avg. | Gen. Avg. | All Avg. |
|---|---|---|---|---|
| R=6 | Position-specific | 69.29 | 58.04 | 64.18 |
| Boundary broadcast (Ours) | 69.99 | 61.70 | 66.22 | |
| R=8 | Position-specific | 68.88 | 57.90 | 63.89 |
| Boundary broadcast (Ours) | 69.24 | 59.44 | 64.79 |
As shown in Table D, the boundary broadcast achieves higher MC, generation, and overall averages than position-specific allocation at both depths. MC tasks test the assumption most directly, since the whole prompt is processed in one block and the broadcast applies to every position; there, the broadcast is higher on almost every task at both depths. This is consistent with how the allocation evidence is formed. The boundary state carries the pending prediction, so its top-1/top-2 contrast is aligned with the token the model outputs next, whereas the contrast at an interior position reflects the prediction of a prompt token that is already given. The broadcast is also more efficient: it applies one allocation to the whole block and requires a single vocabulary readout per retained state rather than one per position, so the cost of computing the allocation does not grow with the prompt length.
Revision: We now state the assumption in Sec. 3.2 and have added Table D with its analysis to Appendix E (Table 14).
Author Response (Part 3/3)
[W3] Statistical confidence is missing for marginal gains: while the re-entry diagnostics report bootstrap confidence intervals, the main benchmark results (Tables 1–3) report single point estimates without sample-level confidence bounds or significance testing, which makes small differences (such as MMLU at or high-variance AMC23 scores) difficult to confirm as real improvements.
[Q2] Could you report bootstrap confidence intervals or significance tests across test instances for the main results in Table 1 to confirm whether improvements at shallow extrapolation depths () are statistically significant?
[R3] We appreciate this suggestion. We agree that the main results should report statistical confidence and that small per-task differences should not be read as improvements on their own. Table E reports the ReMem − Original differences in Table 1 with 95% bootstrap confidence intervals over test instances, for the two tasks named by the reviewer and the three averages; the full per-task table is provided in Appendix D.8 (Table 13) of the revised paper.
Table E. ReMem − Original on Ouro-1.4B, with 95% bootstrap confidence intervals in brackets.
| Benchmark | R=5 | R=6 | R=7 | R=8 |
|---|---|---|---|---|
| MMLU | +0.07 [−0.41, +0.55] | +0.74 [+0.28, +1.18] | +0.07 [−0.38, +0.50] | +0.39 [−0.06, +0.83] |
| AMC23 | +7.50 [−7.50, +22.50] | +7.50 [−7.50, +22.50] | +5.00 [−5.00, +15.00] | +5.00 [−5.00, +15.00] |
| MC Avg. | +0.30 [−0.11, +0.71] | +0.84 [+0.45, +1.23] | +0.84 [+0.46, +1.22] | +0.78 [+0.39, +1.17] |
| Gen. Avg. | +3.44 [+0.22, +6.75] | +4.61 [+1.40, +7.88] | +5.33 [+2.91, +7.80] | +8.18 [+5.69, +10.83] |
| All Avg. | +1.73 [+0.24, +3.25] | +2.55 [+1.08, +4.06] | +2.88 [+1.76, +4.02] | +4.15 [+2.99, +5.35] |
Revision: We have added the full table to Appendix D.8 (Table 13).
[Q3] What drives the performance decline observed when expanding the memory window to in Table 9: does blending older latent states exacerbate representation drift?
[R4] We thank the reviewer for this question. We attribute the decline to the older state that brings into the fusion. Because the budget is fixed, does not add history; it moves part of the budget from the two most recent states to an older one. SCFA normalizes evidence across the retained states, so it can rank them but cannot reject one, and the older state receives a share whenever its own readout evidence is positive. A wider window therefore keeps very early states in the fusion for longer, and our diagnostic shows that these are generally less useful, with the useful region moving toward later states as recurrence proceeds. Consistent with this, the gap to is largest at and narrows at , once the window has moved past the early states (Table 15 in the revised paper).
Revision: We have added this explanation to Appendix E.
We thank the reviewer again for taking the time to review our paper. We have incorporated these discussions into the revised version, with changes marked in blue, and look forward to any further feedback.
Author Response (Part 1/3)
We appreciate the reviewer's thorough assessment and valuable suggestions. Our responses [R] to the raised weaknesses [W] and questions [Q] are provided below.
[W1] The formulation of State-Contrastive Feature Allocation (SCFA) lacks theoretical grounding: the element-wise product vs⊙Δws in Eq. 8 is introduced heuristically, without validation demonstrating that this specific metric correlates with true historical feature utility.
[Q1] Can you provide empirical validation for the choice of vs⊙Δws in Eq. 8, such as comparing it against simpler feature selectors (e.g., token-wise absolute activation magnitude) or analyzing its alignment with oracle feature importance?
[R1] We thank the reviewer for this careful and constructive comment. We agree that Eq. 8 was introduced without sufficient grounding, and that a comparison with simpler feature selectors was missing. We provide both below.
ReMem aims to preserve prediction-relevant information from earlier recurrent states. In a LoopLM, all recurrent states share one representation space and are read out by the same frozen LM head. Because its output projection is linear, the logit gap between a state's predicted token and its strongest competitor (the top-2 token) decomposes into per-feature contributions, which is the element-wise product in Eq. 8. A positive contribution means the feature supports that state's decision, and since the final fused state is read out by the same head, a feature's contribution to the logits is carried into the output in proportion to its allocation. SCFA therefore keeps positive contributions, applies a square root to soften the split between states, and normalizes across retained states for each feature, so that each feature draws more from the state in which it was more decisive. Accordingly, SCFA adopts the LM-head contrast as a label-free, readout-aligned criterion for deciding which historical features to preserve.
Comparison with simpler signals. Following the reviewers' suggestions, we evaluated hidden-state similarity, prediction confidence, and absolute activation magnitude on Ouro-1.4B, keeping all other settings identical. Table A compares SCFA with these alternatives.
Table A. Comparison of allocation signals.
As shown in Table A, SCFA achieves the best MC, generation, and overall averages at both depths, with prediction confidence, the other LM-head-based signal, coming closest. This suggests that grounding the allocation in the frozen readout is important. SCFA weights each feature by its contribution to a state's top-1/top-2 logit gap, which is well aligned with the goal of preserving prediction-relevant information and allows different features to be taken from different states. By contrast, prediction confidence yields a single scalar per state and cannot distinguish among features. Absolute activation magnitude is feature-wise but ignores the readout, so a large activation does not imply relevance to the prediction. Hidden-state similarity measures agreement with the candidate and thus favors information the candidate already contains. Together, these results support the LM-head contrast as an effective allocation signal in the evaluated settings.
Revision: We have added Table A with its analysis to Sec. 4.3 (Table 4) and this rationale to Appendix D.6.