acceptodds
Under review as a conference paper at ICLR 2027

ReMem: Reusable Latent Computation for Depth Extrapolation in Looped Language Models

Abstract

Looped Language Models (LoopLMs) provide a promising approach to test-time scaling by repeatedly applying parameter-shared Transformer blocks to increase latent computational depth without increasing model size. However, performance often deteriorates when recurrent depth extends beyond the regime encountered during training, limiting the effectiveness of recurrent-depth extrapolation. Several existing approaches seek to improve recurrent scaling through additional training or post-training, but require extra optimization. In this work, we revisit how information is propagated across recurrent iterations and find that previously computed recurrent states retain reusable computation that can support deeper recurrent processing. Based on this insight, we propose ReMem, a training-free recurrent memory mechanism that explicitly preserves and reuses historical computation during inference. ReMem maintains a compact, evolving recurrent history, allowing newly formed recurrent states to remain directly accessible to later iterations, and uses evidence from the frozen LM head to adaptively allocate historical contributions at the feature level. Importantly, this design requires no parameter updates or additional Transformer evaluations at matched recurrent depth. Experiments across language understanding, mathematical reasoning, and code generation benchmarks show that ReMem generally improves performance retention under recurrent-depth extrapolation, with gains becoming more pronounced as recurrent depth increases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped language models, latent reasoning, recurrent transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ivX5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReMem, a training-free inference-time mechanism designed to mitigate performance degradation during recurrent-depth extrapolation in Looped Language Models (LoopLMs). Through a controlled state re-entry diagnostic, the authors demonstrate that intermediate recurrent states preserve reusable latent computation. To leverage this, ReMem maintains a rolling buffer of recent recurrent states and fuses them into the candidate state using feature-wise weights determined by contrastive evidence from the frozen language modeling head (SCFA). Across 11 benchmarks on Ouro-1.4B and Ouro-2.6B, ReMem improves capability retention at extended recurrent depths (R=5–16R=5\text{--}16) without adding trainable parameters or extra Transformer evaluations.

Critical strengths
  1. The motivation of viewing historical recurrent states as reusable latent computation is conceptually clean, and the one-shot re-entry diagnostic in Section 2 provides convincing empirical motivation.
  2. The control interventions in Figure 3 (Gaussian random, anti-restoration, shuffled history, and feature permutation) effectively verify that the observed re-entry gains originate from structured historical information.
  3. The method is training-free, operates strictly within compute-matched Transformer evaluation budgets, and includes thorough execution-level efficiency and memory benchmarking in Appendix F.
  4. Empirical gains across language understanding, math, and coding benchmarks are consistent across two model sizes and become pronounced at deep recurrence regimes (R≥10R \ge 10).
Critical weaknesses
  1. The formulation of State-Contrastive Feature Allocation (SCFA) lacks theoretical grounding: the element-wise product vs⊙Δwsv_s \odot \Delta w_s in Eq. 8 is introduced heuristically, without validation demonstrating that this specific metric correlates with true historical feature utility.
  2. The sequence-level fusion relies on a restrictive spatial assumption: Eq. 8 derives allocation weights solely from the single boundary token vs=Sel(hs)v_s = \text{Sel}(h_s) and broadcasts them uniformly across all sequence positions (line 289), which leaves unverified whether output-layer contrast at the final token suitably guides historical feature mixing for preceding tokens.
  3. Statistical confidence is missing for marginal gains: while the re-entry diagnostics report bootstrap confidence intervals, the main benchmark results (Tables 1–3) report single point estimates without sample-level confidence bounds or significance testing, which makes small differences (such as MMLU at R=5R=5 or high-variance AMC23 scores) difficult to confirm as real improvements.
Questions for authors
  1. Can you provide empirical validation for the choice of vs⊙Δwsv_s \odot \Delta w_s in Eq. 8, such as comparing it against simpler feature selectors (e.g., token-wise absolute activation magnitude) or analyzing its alignment with oracle feature importance?
  2. Could you report bootstrap confidence intervals or significance tests across test instances for the main results in Table 1 to confirm whether improvements at shallow extrapolation depths (R=5,6R=5, 6) are statistically significant?
  3. What drives the performance decline observed when expanding the memory window to K=3K=3 in Table 9: does blending older latent states exacerbate representation drift?
3 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)

We appreciate the reviewer's thorough assessment and valuable suggestions. Our responses [R] to the raised weaknesses [W] and questions [Q] are provided below.

[W1] The formulation of State-Contrastive Feature Allocation (SCFA) lacks theoretical grounding: the element-wise product vs⊙Δws\mathbf{v}_s \odot \Delta\mathbf{w}_s in Eq. 8 is introduced heuristically, without validation demonstrating that this specific metric correlates with true historical feature utility.

[Q1] Can you provide empirical validation for the choice of vs⊙Δws\mathbf{v}_s \odot \Delta\mathbf{w}_s in Eq. 8, such as comparing it against simpler feature selectors (e.g., token-wise absolute activation magnitude) or analyzing its alignment with oracle feature importance?

[R1] We thank the reviewer for this careful and constructive comment. We agree that Eq. 8 was introduced without sufficient grounding, and that a comparison with simpler feature selectors was missing. We provide both below.

ReMem aims to preserve prediction-relevant information from earlier recurrent states. In a LoopLM, all recurrent states share one representation space and are read out by the same frozen LM head. Because its output projection is linear, the logit gap between a state's predicted token and its strongest competitor (the top-2 token) decomposes into per-feature contributions, which is the element-wise product in Eq. 8. A positive contribution means the feature supports that state's decision, and since the final fused state is read out by the same head, a feature's contribution to the logits is carried into the output in proportion to its allocation. SCFA therefore keeps positive contributions, applies a square root to soften the split between states, and normalizes across retained states for each feature, so that each feature draws more from the state in which it was more decisive. Accordingly, SCFA adopts the LM-head contrast as a label-free, readout-aligned criterion for deciding which historical features to preserve.

Comparison with simpler signals. Following the reviewers' suggestions, we evaluated hidden-state similarity, prediction confidence, and absolute activation magnitude on Ouro-1.4B, keeping all other settings identical. Table A compares SCFA with these alternatives.

Table A. Comparison of allocation signals.

DepthAllocation SignalMC Avg.Gen. Avg.All Avg.
R=6Hidden-state similarity69.5060.2265.28
Prediction confidence69.5661.4265.86
Absolute activation magnitude69.5161.3965.82
Top-1/top-2 LM-head contrast (SCFA)69.9961.7066.22
R=8Hidden-state similarity69.0957.2163.69
Prediction confidence69.1759.3864.72
Absolute activation magnitude68.9058.8864.35
Top-1/top-2 LM-head contrast (SCFA)69.2459.4464.79

As shown in Table A, SCFA achieves the best MC, generation, and overall averages at both depths, with prediction confidence, the other LM-head-based signal, coming closest. This suggests that grounding the allocation in the frozen readout is important. SCFA weights each feature by its contribution to a state's top-1/top-2 logit gap, which is well aligned with the goal of preserving prediction-relevant information and allows different features to be taken from different states. By contrast, prediction confidence yields a single scalar per state and cannot distinguish among features. Absolute activation magnitude is feature-wise but ignores the readout, so a large activation does not imply relevance to the prediction. Hidden-state similarity measures agreement with the candidate and thus favors information the candidate already contains. Together, these results support the LM-head contrast as an effective allocation signal in the evaluated settings.

Revision: We have added Table A with its analysis to Sec. 4.3 (Table 4) and this rationale to Appendix D.6.

AuthorsOpenReview ↗

Author Response (Part 2/3)

[W2] The sequence-level fusion relies on a restrictive spatial assumption: Eq. 8 derives allocation weights solely from the single boundary token vs=Sel(hs)\mathbf{v}_s=\mathrm{Sel}(\mathbf{h}_s) and broadcasts them uniformly across all sequence positions (line 289), which leaves unverified whether output-layer contrast at the final token suitably guides historical feature mixing for preceding tokens.

[R2] We thank the reviewer for this important question. We agree that the submission does not verify the boundary broadcast in isolation. We clarify its scope and test it against a position-specific alternative below.

The broadcast applies only to positions processed in one block, i.e., the prompt and multiple-choice scoring. During generation, the evidence is recomputed whenever the prediction boundary advances and fusion acts only on the current active block, typically one token, so each generated position already uses an allocation derived from its own state.

Comparison with position-specific allocation. To test the assumption, we let every prompt position derive its allocation from its own state via Eqs. 7–9 instead of receiving the boundary allocation, keeping all other settings unchanged.

Table D. Boundary broadcast vs. position-specific allocation over the prompt.

DepthSpatial allocationMC Avg.Gen. Avg.All Avg.
R=6Position-specific69.2958.0464.18
Boundary broadcast (Ours)69.9961.7066.22
R=8Position-specific68.8857.9063.89
Boundary broadcast (Ours)69.2459.4464.79

As shown in Table D, the boundary broadcast achieves higher MC, generation, and overall averages than position-specific allocation at both depths. MC tasks test the assumption most directly, since the whole prompt is processed in one block and the broadcast applies to every position; there, the broadcast is higher on almost every task at both depths. This is consistent with how the allocation evidence is formed. The boundary state carries the pending prediction, so its top-1/top-2 contrast is aligned with the token the model outputs next, whereas the contrast at an interior position reflects the prediction of a prompt token that is already given. The broadcast is also more efficient: it applies one allocation to the whole block and requires a single vocabulary readout per retained state rather than one per position, so the cost of computing the allocation does not grow with the prompt length.

Revision: We now state the assumption in Sec. 3.2 and have added Table D with its analysis to Appendix E (Table 14).

AuthorsOpenReview ↗

Author Response (Part 3/3)

[W3] Statistical confidence is missing for marginal gains: while the re-entry diagnostics report bootstrap confidence intervals, the main benchmark results (Tables 1–3) report single point estimates without sample-level confidence bounds or significance testing, which makes small differences (such as MMLU at R=5R=5 or high-variance AMC23 scores) difficult to confirm as real improvements.

[Q2] Could you report bootstrap confidence intervals or significance tests across test instances for the main results in Table 1 to confirm whether improvements at shallow extrapolation depths (R=5,6R=5,6) are statistically significant?

[R3] We appreciate this suggestion. We agree that the main results should report statistical confidence and that small per-task differences should not be read as improvements on their own. Table E reports the ReMem − Original differences in Table 1 with 95% bootstrap confidence intervals over test instances, for the two tasks named by the reviewer and the three averages; the full per-task table is provided in Appendix D.8 (Table 13) of the revised paper.

Table E. ReMem − Original on Ouro-1.4B, with 95% bootstrap confidence intervals in brackets.

BenchmarkR=5R=6R=7R=8
MMLU+0.07 [−0.41, +0.55]+0.74 [+0.28, +1.18]+0.07 [−0.38, +0.50]+0.39 [−0.06, +0.83]
AMC23+7.50 [−7.50, +22.50]+7.50 [−7.50, +22.50]+5.00 [−5.00, +15.00]+5.00 [−5.00, +15.00]
MC Avg.+0.30 [−0.11, +0.71]+0.84 [+0.45, +1.23]+0.84 [+0.46, +1.22]+0.78 [+0.39, +1.17]
Gen. Avg.+3.44 [+0.22, +6.75]+4.61 [+1.40, +7.88]+5.33 [+2.91, +7.80]+8.18 [+5.69, +10.83]
All Avg.+1.73 [+0.24, +3.25]+2.55 [+1.08, +4.06]+2.88 [+1.76, +4.02]+4.15 [+2.99, +5.35]

Revision: We have added the full table to Appendix D.8 (Table 13).

[Q3] What drives the performance decline observed when expanding the memory window to K=3K=3 in Table 9: does blending older latent states exacerbate representation drift?

[R4] We thank the reviewer for this question. We attribute the decline to the older state that K=3K=3 brings into the fusion. Because the budget BB is fixed, K=3K=3 does not add history; it moves part of the budget from the two most recent states to an older one. SCFA normalizes evidence across the retained states, so it can rank them but cannot reject one, and the older state receives a share whenever its own readout evidence is positive. A wider window therefore keeps very early states in the fusion for longer, and our diagnostic shows that these are generally less useful, with the useful region moving toward later states as recurrence proceeds. Consistent with this, the gap to K=2K=2 is largest at R=5,6R=5,6 and narrows at R=7,8R=7,8, once the window has moved past the early states (Table 15 in the revised paper).

Revision: We have added this explanation to Appendix E.

We thank the reviewer again for taking the time to review our paper. We have incorporated these discussions into the revised version, with changes marked in blue, and look forward to any further feedback.

Reviewer_hustOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReMem, a training-free recurrent memory mechanism that adaptively reuses historical hidden states to mitigate performance degradation during recurrent-depth extrapolation in Looped Language Models.

Critical strengths
  1. The paper addresses an important challenge in recurrent-depth extrapolation and proposes a simple training-free solution without additional Transformer evaluations.
  2. The historical state re-entry experiments provide empirical evidence that intermediate recurrent states retain useful information.
  3. Extensive experiments across multiple benchmarks, model scales, and recurrent depths demonstrate consistent improvements, particularly at larger depths.
Critical weaknesses
  1. The paper does not sufficiently distinguish the benefits of reusable latent computation from those of recurrent dynamics stabilization or smoothing.
  2. The motivation for using top-1/top-2 logit contrast as a proxy for historical feature utility is insufficiently justified.
  3. The evaluation is primarily limited to the Ouro model family, leaving the generalizability to other recurrent architectures unclear.
Questions for authors
  1. How can the authors distinguish the benefits of historical computation reuse from those of recurrent dynamics stabilization?
  2. Why is top-1/top-2 logit contrast preferred over simpler alternatives such as hidden-state similarity or prediction confidence for feature-wise allocation?
  3. Can the authors evaluate ReMem on other recurrent language model architectures to demonstrate its generalizability?
3 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)

We appreciate the reviewer’s thoughtful and valuable feedback. Our responses [R] to the raised weaknesses [W] and questions [Q] are provided below.

[W1] The paper does not sufficiently distinguish the benefits of reusable latent computation from those of recurrent dynamics stabilization or smoothing.

[Q1] How can the authors distinguish the benefits of historical computation reuse from those of recurrent dynamics stabilization?

[R1] We appreciate this question, as it concerns the central claim of the paper. We apologize that the main text of the submission does not discuss this point explicitly. To distinguish the benefits of historical computation reuse from those of recurrent dynamics stabilization, we compare ReMem with two simplified variants under the same history budget: Single-State Damping, which only damps the update with the immediately preceding state, and Uniform Historical Fusion, which shares ReMem's memory policy but splits the budget equally between the retained states. The submission reports this comparison at R=10, 12, 16 in Appendix D.2. To make it complete, we additionally evaluated both variants at R=6; at R=8, they correspond to the w/o Multi-state and w/o Adaptive ablations in Table 3. Table B reports all depths together.

Table B. Comparison with simplified historical reuse on Ouro-1.4B across recurrent depths (%).

DepthMethodMC Avg.Gen. Avg.All Avg.
R=6Original69.1557.0963.67
Single-State Damping69.6160.1265.30
Uniform Historical Fusion69.3957.5864.02
ReMem69.9961.7066.22
R=8Original68.4651.2660.64
Single-State Damping69.0758.5564.29
Uniform Historical Fusion69.0457.5463.81
ReMem69.2459.4464.79
R=10Original66.4934.9452.15
Single-State Damping67.9245.4457.70
Uniform Historical Fusion68.2749.4859.73
ReMem68.5254.5462.16
R=12Original65.1722.1245.60
Single-State Damping66.9031.3950.76
Uniform Historical Fusion67.2237.0453.50
ReMem67.8648.6759.14
R=16Original60.233.4134.40
Single-State Damping64.017.0938.14
Uniform Historical Fusion64.7615.0742.17
ReMem66.3324.2147.18

As shown in Table B, at every depth both variants improve over original inference, confirming that stabilization contributes to the gain, yet ReMem is better than both on all three averages. Single-State Damping isolates stabilization, while Uniform Historical Fusion differs from ReMem only in which state each feature draws from. The additional gain is therefore attributable to selecting which historical features to reuse, which smoothing alone does not provide. The higher-depth comparisons reinforce the contribution of adaptive reuse, with the clearest gains on math and coding tasks.

Revision: We now discuss this point explicitly in Sec. 4.3 (Table 5).

AuthorsOpenReview ↗

Author Response (Part 2/3)

[W2] The motivation for using top-1/top-2 logit contrast as a proxy for historical feature utility is insufficiently justified.

[Q2] Why is top-1/top-2 logit contrast preferred over simpler alternatives such as hidden-state similarity or prediction confidence for feature-wise allocation?

[R2] We thank the reviewer for this insightful question. We agree that the motivation for the top-1/top-2 logit contrast was insufficiently justified in the submission, and that a comparison with simpler alternatives was missing. We provide both below.

ReMem aims to preserve prediction-relevant information from earlier recurrent states. In a LoopLM, all recurrent states share one representation space and are read out by the same frozen LM head. Because its output projection is linear, the logit gap between a state's predicted token and its strongest competitor (the top-2 token) decomposes into per-feature contributions, which is the element-wise product in Eq. 8. A positive contribution means the feature supports that state's decision, and since the final fused state is read out by the same head, a feature's contribution to the logits is carried into the output in proportion to its allocation. SCFA therefore keeps positive contributions, applies a square root to soften the split between states, and normalizes across retained states for each feature, so that each feature draws more from the state in which it was more decisive. Accordingly, SCFA adopts the LM-head contrast as a label-free, readout-aligned criterion for deciding which historical features to preserve.

Comparison with simpler signals. Following the reviewers' suggestions, we evaluated hidden-state similarity, prediction confidence, and absolute activation magnitude on Ouro-1.4B, keeping all other settings identical. Table A compares SCFA with these alternatives.

Table A. Comparison of allocation signals.

DepthAllocation SignalMC Avg.Gen. Avg.All Avg.
R=6Hidden-state similarity69.5060.2265.28
Prediction confidence69.5661.4265.86
Absolute activation magnitude69.5161.3965.82
Top-1/top-2 LM-head contrast (SCFA)69.9961.7066.22
R=8Hidden-state similarity69.0957.2163.69
Prediction confidence69.1759.3864.72
Absolute activation magnitude68.9058.8864.35
Top-1/top-2 LM-head contrast (SCFA)69.2459.4464.79

As shown in Table A, SCFA achieves the best MC, generation, and overall averages at both depths, with prediction confidence, the other LM-head-based signal, coming closest. This suggests that grounding the allocation in the frozen readout is important. SCFA weights each feature by its contribution to a state's top-1/top-2 logit gap, which is well aligned with the goal of preserving prediction-relevant information and allows different features to be taken from different states. By contrast, prediction confidence yields a single scalar per state and cannot distinguish among features. Absolute activation magnitude is feature-wise but ignores the readout, so a large activation does not imply relevance to the prediction. Hidden-state similarity measures agreement with the candidate and thus favors information the candidate already contains. Together, these results support the LM-head contrast as an effective allocation signal in the evaluated settings.

Revision: We have added Table A with its analysis to Sec. 4.3 (Table 4) and this rationale to Appendix D.6.

AuthorsOpenReview ↗

Author Response (Part 3/3)

[W3] The evaluation is primarily limited to the Ouro model family, leaving the generalizability to other recurrent architectures unclear.

[Q3] Can the authors evaluate ReMem on other recurrent language model architectures to demonstrate its generalizability?

[R3] We thank the reviewer for raising this concern. We agree that evaluating only the Ouro family leaves the generalizability unclear. Following the reviewer's suggestion, we applied ReMem to the recurrent block of two other recurrent architectures, Recurrent-Llama-3.2 and Recurrent-OLMo-2-0425, without any retraining and keeping all other settings identical.

Table C. ReMem on other recurrent architectures.

ModelDepthMethodMC Avg.Gen. Avg.All Avg.
Rec.-Llama-3.2R=8Original51.4922.7038.40
ReMem51.6922.8038.56
R=12Original51.4922.3938.26
ReMem51.5522.7538.46
Rec.-OLMo-2-0425R=8Original53.4319.3837.95
ReMem53.5020.9038.68
R=12Original53.4819.0437.83
ReMem53.4919.1937.90

ReMem transfers directly to both architectures, without retraining and with the same hyperparameters, and yields consistent gains: all three averages are above the original at both depths on both models. This indicates that reusing recurrent history carries over beyond the Ouro family. The gains are smaller than on Ouro, which follows from how ReMem works: it recovers useful computation from earlier states, so its benefit grows with the degradation that extrapolation causes. These models start from a lower accuracy and remain almost unchanged from R=8 to R=12, leaving less to recover in this range, whereas on Ouro, where degradation is pronounced, the gains are substantial. ReMem thus improves performance where depth extrapolation hurts and remains safe to apply where it does not.

Revision: We have added the new results to Appendix D.7 (Tables 11 and 12).

We thank the reviewer again for taking the time to review our paper. We have incorporated these discussions into the revised version, with changes marked in blue, and look forward to any further feedback.

Reviewer_7kkROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMem, a training-free method for improving recurrent-depth extrapolation in Looped Language Models. The key idea is to retain previous hidden states and reuse them through feature-wise weighted fusion, with weights derived from the frozen LM head. Experiments on Ouro-1.4B and Ouro-2.6B show better accuracy retention at larger recurrent depths without additional Transformer evaluations.

Critical strengths
  1. Improving recurrent-depth extrapolation is an important problem for test-time scaling. The proposed method is simple and requires no additional training.

  2. The evaluation is fairly comprehensive, covering multiple benchmarks and recurrent depths. The ablations also demonstrate meaningful gains over simple damping and uniform historical fusion, especially at larger depths.

Critical weaknesses
  1. Heuristic method design. The historical fusion relies on several heuristic choices, such as the fixed history budget B=0.2 and using the top-1/top-2 LM-head contrast to determine feature-wise weights. While the paper provides sensitivity studies, it's unclear why the LM-head contrast is a good indicator of which historical features should be preserved. More justification or comparison with simpler weighting strategies would be beneficial.

  2. Limited practical benefits of depth extrapolation. Although ReMem reduces accuracy degradation at larger recurrent depths, performance still drops as the number of iterations increases. For instance, Ouro-1.4B with ReMem drops from 66.82% at R=5 to 47.18% at R=16. It's unclear when users would benefit from running additional iterations rather than using a shallower recurrent depth.

Questions for authors
  1. Can the authors provide more justification for using the top-1/top-2 LM-head contrast for feature-wise weighting? Have the authors compared against simpler signals, such as hidden-state similarity or magnitude?

  2. Can the authors discuss the practical use cases of recurrent-depth extrapolation when additional iterations still reduce accuracy, even with ReMem?

4 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We sincerely thank the reviewer for the constructive comments. Our responses [R] to the raised weaknesses [W] and questions [Q] are provided below.

[W1] Heuristic method design. The historical fusion relies on several heuristic choices, such as the fixed history budget B=0.2 and using the top-1/top-2 LM-head contrast to determine feature-wise weights. While the paper provides sensitivity studies, it's unclear why the LM-head contrast is a good indicator of which historical features should be preserved. More justification or comparison with simpler weighting strategies would be beneficial.

[Q1] Can the authors provide more justification for using the top-1/top-2 LM-head contrast for feature-wise weighting? Have the authors compared against simpler signals, such as hidden-state similarity or magnitude?

[R1] We thank the reviewer for this valuable comment. We agree that the submission does not sufficiently explain why the LM-head contrast is a suitable indicator of which historical features to preserve, and that a comparison with simpler signals was missing. We provide both below.

ReMem aims to preserve prediction-relevant information from earlier recurrent states. In a LoopLM, all recurrent states share one representation space and are read out by the same frozen LM head. Because its output projection is linear, the logit gap between a state's predicted token and its strongest competitor (the top-2 token) decomposes into per-feature contributions, which is the element-wise product in Eq. 8. A positive contribution means the feature supports that state's decision, and since the final fused state is read out by the same head, a feature's contribution to the logits is carried into the output in proportion to its allocation. SCFA therefore keeps positive contributions, applies a square root to soften the split between states, and normalizes across retained states for each feature, so that each feature draws more from the state in which it was more decisive. Accordingly, SCFA adopts the LM-head contrast as a label-free, readout-aligned criterion for deciding which historical features to preserve.

Comparison with simpler signals. Following the reviewers' suggestions, we evaluated hidden-state similarity, prediction confidence, and absolute activation magnitude on Ouro-1.4B, keeping all other settings identical. Table A compares SCFA with these alternatives.

Table A. Comparison of allocation signals.

DepthAllocation SignalMC Avg.Gen. Avg.All Avg.
R=6Hidden-state similarity69.5060.2265.28
Prediction confidence69.5661.4265.86
Absolute activation magnitude69.5161.3965.82
Top-1/top-2 LM-head contrast (SCFA)69.9961.7066.22
R=8Hidden-state similarity69.0957.2163.69
Prediction confidence69.1759.3864.72
Absolute activation magnitude68.9058.8864.35
Top-1/top-2 LM-head contrast (SCFA)69.2459.4464.79

As shown in Table A, SCFA achieves the best MC, generation, and overall averages at both depths, with prediction confidence, the other LM-head-based signal, coming closest. This suggests that grounding the allocation in the frozen readout is important. SCFA weights each feature by its contribution to a state's top-1/top-2 logit gap, which is well aligned with the goal of preserving prediction-relevant information and allows different features to be taken from different states. By contrast, prediction confidence yields a single scalar per state and cannot distinguish among features. Absolute activation magnitude is feature-wise but ignores the readout, so a large activation does not imply relevance to the prediction. Hidden-state similarity measures agreement with the candidate and thus favors information the candidate already contains. Together, these results support the LM-head contrast as an effective allocation signal in the evaluated settings.

Regarding the fixed budget, BB is a single global hyperparameter, fixed across all tasks, depths and model sizes without tuning; the default B=0.2B=0.2 stays at or near the best value at every depth in the sensitivity study.

Revision: We have added Table A with its analysis to Sec. 4.3 (Table 4) and this rationale to Appendix D.6.

AuthorsOpenReview ↗

Author Response (Part 2/2)

[W2] Limited practical benefits of depth extrapolation. Although ReMem reduces accuracy degradation at larger recurrent depths, performance still drops as the number of iterations increases. For instance, Ouro-1.4B with ReMem drops from 66.82% at R=5 to 47.18% at R=16. It's unclear when users would benefit from running additional iterations rather than using a shallower recurrent depth.

[Q2] Can the authors discuss the practical use cases of recurrent-depth extrapolation when additional iterations still reduce accuracy, even with ReMem?

[R2] We thank the reviewer for raising this important point. We agree that, for current large LoopLMs, fully achieving the ideal scaling behavior, in which more recurrent computation always improves reasoning, remains an open problem (Sec. 6). Our goal is to take a concrete empirical step toward it. ReMem does not eliminate degradation under deep extrapolation; a likely reason is that it is applied only at inference to a frozen model that was pretrained at a fixed depth with single-state recurrence, and was never optimized to use historical access. That said, ReMem already substantially improves robustness to increased depth without any training.

This robustness is what makes input-adaptive compute allocation practical. Loop depth is a scaling axis distinct from model size [1]: it adds computation at fixed parameters and lets harder inputs receive more iterations [2]. Across tasks, the loop depth that best serves an input is not a fixed shallow value [3], so the range available to such allocation is bounded by the depth at which recurrent states remain reliable. ReMem widens this range.

Revision: We have revised Sec. 6 to state this more clearly.

We thank the reviewer again for taking the time to review our paper. We have incorporated these discussions into the revised version, with changes marked in blue, and look forward to any further feedback.

[1] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, NeurIPS 2025.

[2] Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation, NeurIPS 2025.

[3] Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks, NeurIPS 2021.

Thanks for the Author Responses. I'd keep my score.

We sincerely thank you for reading our responses and for maintaining a positive assessment of our paper. Your comments have helped us improve the paper, and the corresponding revisions are included in the updated version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.