acceptodds
Under review as a conference paper at ICLR 2027

LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillation

Abstract

Long-context reasoning faces two complementary bottlenecks: retaining evidence across long inputs and sustaining computation across many reasoning steps. Existing approaches largely address them separately, with external memory extending access to distant evidence and latent reasoning compressing multi-step computation. We introduce LatentHarness, which unifies memory access and latent reasoning as sequential latent action selection. At each internal step, the model chooses THINK for further computation, RECALL from a fast-weight memory of input evidence and intermediate reasoning states, or EXIT to emit the next token. We train this policy with counterfactual policy distillation, which branches every action for one step and scores its effect on the emitted token. These gains teach the policy when memory is more useful than further reasoning, while gradients through counterfactual recall teach which intermediate states should be retained in memory for future use. Across six general and long-context reasoning benchmarks, LatentHarness at 1.4B improves on the strongest baselines by 2.8% and 10.0% relative, respectively, and runs 5.9× faster than the strongest long-context baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, latent memory, long-context reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nZBTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The motivation, as written in the introduction and suggested in the title, is that while text-based reasoning in vanilla autoregressive LLMs has paved the way for various methods to access and manage external context, in the form of text, outside of the model's context window, there is no established approach for a looped transformer to do the same within its "latent reasoning". My understanding is that this is fundamentally because existing methods are based on tool calls which are expressed in explicit text/tokens, which is incompatible with "latent" reasoning, which is simply repeated propagation of hidden states through transformer blocks.

While this motivation is very intriguing, the proposed methodology does not align with this motivation. The main idea is to add a residual stream of RNN-style state-space modeling through the recurrent depth axis, propagating a "shared fast-weight associative memory", or 'latent memory" in the form of a d x d matrix where d is the hidden dimension. The proposed model adds some form of a decision head to enable the model to select between THINK, RECALL, and EXIT. THINK and RECALL means taking another pass through the transformer blocks, writing to and reading from the latent memory, respectively. EXIT means terminating the loop and decoding the output token.

The authors propose complex novel additions to an existing approach for reinforcing latent actions (RLTT) to account for THINK and RECALL with appropriate credit assignment, namely, Counterfactual Policy Distillation (CPD).

A key concern I have with the paper is that it provides very little detail on how these operations interact with the KV cache, simply stating that "We leave the block’s attention over earlier positions implicit". This is one of the most critical details for this method, as it is central to understanding the following:

  • (1) How does the added residual stream via latent memory through the recursive depth axis interact with the sequence axis. Modeling information across a long sequence is typically what we refer to as "memory" and long-context modeling. The authors associate gains in long-context evaluations such as HotpotQA with the "memory" mechanism of the proposed method.
  • (2) The claimed theoretical efficiency gains appear to stem from early-exiting but there is no mention of how early-exiting is handled in practice with respect to KV cache. Variable-depth recursion in looping transformers is tricky because early-exiting leaves gaps in the KV cache for subsequent iterations which are required for attention when a later token does not exit early. There are complex ways to account for this as shown in Mixture-of-Recursions [1] but I believe it is considered impractical:
    • vLLM does not currently support the adaptive exit feature ... the model will always execute the full number of total_ut_steps [2]
    • True, we not use early exit in vllm. It's hard to handle kv cache here. [3]

Note on confidence: I have deep background in context management, KV cache, vanilla transformer architecture, and general knowledge of loop-based transformers. I am unfamiliar with RNN-style state-based models and fast-weight associative memory.

[1] https://arxiv.org/abs/2507.10524 [2] https://huggingface.co/ByteDance/Ouro-1.4B-Thinking [3] https://github.com/vllm-project/vllm/pull/27794#discussion_r2477168197

Critical strengths
  • S1. I think that latent memory along the recursive depth axis to supplement the small representative capacity of the hidden state may be useful, and the proposed CPD method seems like a reasonable approach to help supervision. However, this differs from the motivation in the paper (see W1).
  • S2. The accuracy gains and claimed efficiency gains are significant, but there are significant concerns (see W2, W3).
Critical weaknesses
  • W1. The motivation mentioned in the early half of the introduction and the method do not align (see Paper Summary). It follows that there is insufficient justification for the results.
  • W2. There are significant concerns regarding the soundness of the accuracy results. Notably:
    • The reported GSM8K and MATH scores for Ouro 1.4 (74.6, 42.8) significantly differ from that of the Ouro paper's numbers on Table 7 (78.92, 82.40) [1]
    • The comparison with text-based agentic memory methods (Search-R1, MemAgent, MEM1, Memory-R2) seem out of place because (1) the evaluation benchmarks do not involve handling large external context, (2) the Ouro base models are not trained with agentic capabilities, and (3) their relevance comes from the written motivation, which does not align with the actual method (W1).
  • W3. There are significant concerns regarding the soundness of the efficiency results. Notably:
    • It is unclear if the gains in FLOPS are attainable in real inference scenarios, as these seem to stem from early-exiting, for which adequate explanation of the exact mechanism is not provided (see Paper Summary). Details regarding early-exiting and handling of KV cache, detailed accounting cost components, e.g., prefill vs decode cost, would help support the soundness of the results.
    • It is unclear how the wall time numbers are derived. The paper only mentions We report FLOPs relative to the full-depth backbone and per-answer latency following Wang et al. (2025b), but the cited paper also does not provide clear explanation. Details regarding the measurement such as measurement protocol, hardware, software libraries (exact versions, options, kernels, etc) would clarify things and support the soundness of the results.

[1] https://arxiv.org/pdf/2510.25741

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer nZBT (Part 1/4)

We thank the reviewer for recognizing the latent memory along the recursive depth axis and the CPD supervision, and address below the KV-cache mechanics of memory and early exit and the soundness of the accuracy and efficiency results.

W1a and W3a. How LatentHarness handles the KV cache along the sequence and under early exits.

LatentHarness keeps ordinary attention over the KV cache and copies an early-exited token's last computed keys and values into the skipped recurrent steps, adapting the widely used early-exit idea of copying hidden states forward, as in Depth-Adaptive Transformer [1] and CALM [2]. A separate fast-weight memory carries evidence across positions and reads a value for a query key from one d×dd \times d matrix instead of attending over all cached keys.

In the shared block described in Section 2.1, if a token exits after recurrent step t<Nt<N, later tokens that reach steps t+1,…,Nt+1,\ldots,N still need its cache entries. We fill those entries in each layer with the keys and values from step tt, a rule we call copy-forward. Later tokens can then attend to earlier positions without rerunning the block for an exited token. Mixture-of-Recursions [3] discusses sharing first-step keys and values or masking missing entries, while copy-forward keeps the last computed ones, and we apply it in prefill, decoding and training rollouts.

THINK writes a key-value pair to the matrix with a gated delta rule, and RECALL multiplies the matrix by a query key (Eqs. 2 and 3). A read adds no tokens to the cache and costs about 1% of one block application. The memory recovers evidence that attention misses at long range. At 64k, 63.9% of evidence recalls hit a supporting passage (Table 4), while the passage receiving the most attention was a supporting one at only 31.5% of answer positions, down from 58.7% at 4k.

We compared four cache rules on Ouro-1.4B-Thinking under the Section 3 evaluation protocol. The oracle computed the missing keys and values at deeper steps but still decoded at the selected exit step.

Cache ruleGeneralLong-context
Copy-forward, LatentHarness55.849.7
Oracle, compute deeper KV55.949.9
Share first-step KV38.130.4
Mask missing deeper entries51.243.0

Copy-forward came within 0.1 general and 0.2 long-context points of the oracle, while masking scored 4.6 general and 6.7 long-context points below copy-forward.

We also retrained three variants for 3k steps from the same initialization and data to test what the memory needs to carry into later output positions.

Memory configurationGeneralLong-context64k support hit (%)
LatentHarness55.849.763.9
Reset to prompt memory MxM_x at each position55.448.563.5
Reset to zero at each position55.342.6No evidence recalls
No prompt memorization, THINK writes only55.643.3No evidence recalls
THINK and EXIT, Table 253.642.0No recalls

General is mean exact match and long-context is mean answer F1. Support hit is the share of evidence recalls that address an annotated supporting passage.

Prompt memorization accounted for most of the gain. Resetting to zero brought F1 close to THINK and EXIT, and resetting to MxM_x cost 1.2 long-context points, showing an additional benefit from carrying memory across positions. We will replace the implicit-attention sentence of Section 2.1 with a description of the KV cache, the memory and copy-forward, and add the cache-rule comparison, the phase accounting and the timing protocol to Appendix B.3.

AuthorsOpenReview ↗

Author Response to Reviewer nZBT (Part 2/4)

W1b. How the motivation maps onto the method.

We respectfully disagree that the motivation and the method diverge. The abstract identifies two bottlenecks in text-based reasoning, reading long inputs and generating intermediate steps, and the latent actions address both.

On the input side, LatentHarness writes the prompt into fast weights once, and RECALL then reads evidence from memory instead of rereading the context. The read takes one matrix-vector product inside the loop and generates no text. On the output side, text-based reasoning spends tokens on intermediate steps. THINK instead runs another recurrent step in latent space, and EXIT ends the loop and emits the next token, so each output token gets only the computation it needs. In Table 1, LatentHarness improves on the strongest baselines in long-context F1 and general accuracy while using 48% and 62% of full-depth FLOPs, respectively. At 64k, 63.9% of evidence recalls hit supporting passages, and the share of recalls that read THINK-written states grows with input length (Table 4).

We tested retrieval beyond the attention window by extending the suites to 96k and 128k, past Ouro's 65,536-token positional range [4]. RLTT and LatentHarness attended only to the latest 64k prompt tokens, but LatentHarness also wrote every prompt token into memory in one streaming pass. Search-R1 retrieved from all passages, and Memory-R2 read the prompt in chunks, so both retained access to the full input.

Method64k F196k F1128k F1
RLTT31.227.624.0
Search-R138.136.535.2
Memory-R239.437.836.9
LatentHarness45.543.441.2

Answer F1 is pooled over HotpotQA, MuSiQue and 2WikiMultihopQA using the Section 3 scoring protocol.

The memory retrieved evidence outside the active attention window. At 128k, 71% of evidence recalls addressed passages outside that window, and 58.3% of all evidence recalls hit a supporting passage.

W2a. Why our Ouro numbers differ from those in the Ouro paper.

The scores come from different checkpoints and protocols, with Ouro's Table 7 [5] evaluating Ouro-1.4B base few-shot and our Table 1 evaluating the released Ouro-1.4B-Thinking zero-shot.

Ouro's Table 16 [5] specifies 3-shot CoT for GSM8K and 5-shot CoT for MATH500, with an in-house pipeline for MATH500. The Ouro paper reports neither score for the Thinking checkpoint. Our Table 1 labels MATH500 as MATH and uses the zero-shot Thinking chat template, with five sampled answers per question at temperature 0.6 and top-p 0.95 and a cap of 2,048 new tokens. It reports avg@5, or mean exact match across the five samples with one answer extractor for all methods, not greedy accuracy. The submission should have stated this protocol.

To further demonstrate that our evaluation is not the source of the gap, following the reviewer's concern, we reproduced the published base protocol and tested Ouro-1.4B-Thinking, RLTT and LatentHarness using Ouro's Thinking decoding settings (temperature 1.0, top-p 0.7), an 8,192-token budget and the mean over four samples per question.

Checkpoint and protocolGSM8KMATH500
Ouro-1.4B base, published Table 778.9282.40
Ouro-1.4B base, our Table 7 reproduction78.581.9
Ouro-1.4B-Thinking, our Table 1 protocol74.642.8
Ouro-1.4B-Thinking, 8,192-token sampling86.283.9
RLTT, 8,192-token sampling86.884.7
LatentHarness, 8,192-token sampling88.185.6

The first two rows use the base checkpoint and Ouro's few-shot protocol. The remaining rows use the Thinking checkpoint and the listed sampling protocols. All scores are exact-match accuracy.

Our evaluation recovered the published base scores within 0.5 points, so the gap reflects checkpoint and protocol differences. LatentHarness still led both baselines under the longer protocol, by smaller margins than in Table 1. The shorter cap truncated 34% of Ouro-1.4B-Thinking's MATH500 answers and 11% of LatentHarness's, making MATH500 sensitive to the cap, although the longer protocol also changed temperature and top-p.

AuthorsOpenReview ↗

Author Response to Reviewer nZBT (Part 3/4)

W2b. Why token-space memory agents are relevant baselines.

The motivation for these baselines lies in the long-context suites, where every method reads the full 4k to 64k distractor-padded context and every agent shares our backbone, data and RL steps.

Following the long-context setting of MemAgent [6] and Search-R1 [7], each method must find supporting passages among distractors. The baselines span the token-space ways of managing such input. Search-R1 retrieves passages, MemAgent and Memory-R2 read and write textual memory, and MEM1 keeps a compact state for memory and reasoning. LatentHarness reads memory inside the latent loop. None of the agents needs a separately pretrained controller, and all four gained 4.4 to 6.0 long-context points over memory-free GRPO on Ouro (Table 1). The W1b response above also tests them beyond the attention window, the setting the Introduction motivates.

To check that a backbone without agentic training does not handicap these recipes, we trained MemAgent, Memory-R2 and Search-R1 on Qwen3-1.7B, a size-matched dense backbone, keeping the same training data and 3k RL steps as on Ouro.

BackboneMemAgentMemory-R2Search-R1
Ouro-1.4B-Thinking43.6 (50.9)45.2 (52.6)44.1 (50.4)
Qwen3-1.7B45.1 (51.6)46.3 (53.0)45.0 (51.2)

Each cell gives mean long-context answer F1, with mean general exact match in parentheses.

Switching to Qwen3-1.7B improved long-context F1 by only 0.9 to 1.5 points, suggesting that Ouro's recurrence did not limit the agents. LatentHarness on Ouro stayed 3.4 long-context and 2.8 general points ahead of the best Qwen3 agent.

W3b. Where the FLOPs savings arise and whether they hold in real inference.

The FLOPs savings come from running fewer recurrent steps per token, mostly in prefill on long inputs and in decoding on general reasoning, and they carry into wall-clock time, with LatentHarness using 48% of Ouro's FLOPs and 58% of its time on long-context reasoning.

Copy-forward fills the cache for tokens that exit early, so later tokens can attend to them without recomputing skipped steps (W1a and W3a). We split the FLOPs in Table 1 into prompt prefill and answer decoding. Each phase includes memory writes, read probes and policy operations, and FLOPs are relative to full-depth execution over the same tokens.

FamilyMethodDepth P/AFLOPs P/ATotal FLOPsGenerated tokens
GeneralOuro4.0/4.019.7/80.3%100%612
GeneralRLTT3.4/3.517.0/69.0%86%598
GeneralLatentHarness2.4/2.511.8/50.2%62%541
LongOuro4.0/4.099.3/0.7%100%178
LongRLTT3.3/3.381.4/0.6%82%170
LongLatentHarness1.9/2.147.7/0.3%48%150

P/A denotes prompt prefill and answer decoding. Depth is the mean number of block-equivalent applications per position.

Most savings come from decoding on general reasoning. On long-context reasoning, prompts average 24.8k tokens and prefill accounts for most of the savings even with prompt memorization. On those inputs, wall-clock time falls less than FLOPs because block counts miss some overhead.

We also measured batched decoding throughput in tokens per second on one NVIDIA H200 141GB, with 4k-token inputs and all recurrent-step caches resident.

Batch sizeOuroRLTTLatentHarness
1118139186
4310365470
16470545680

At batch 16, the largest size tested, LatentHarness still decoded faster at 680 tokens per second against 470 for Ouro.

Reviewer_Gue7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentHarness, which unifies memory access and latent reasoning through a policy selecting among THINK, RECALL, and EXIT. The method introduces counterfactual policy distillation, using one-step token-probability gains to supervise action selection and memory writing. Experiments on diverse benchmarks with two 1.4B and 2.6B Ouro models show improved reasoning performance and reduced inference cost, particularly on long-context tasks.

Critical strengths
  • The motivation is intuitive and clear. Based on the observation that further computation and memory access address different reasoning bottlenecks, the paper explores how to coordinate them within a shared latent action space. This provides a practical direction for integrating memory access into latent reasoning through explicit decisions about when to compute, recall, or emit a token.
Critical weaknesses
  • The counterfactual action gain does not fully capture the downstream contribution of an action. It measures a one-step change in the log-probability of the token emitted by the realized trajectory, while untaken THINK branches are not continued and do not commit their memory writes. Consequently, it remains unclear whether this local signal reliably identifies actions that improve final-answer correctness, particularly when their benefits emerge through subsequent computation or memory reuse.

  • The contribution of intermediate-state reuse is not clearly established. The memory stores both input evidence and intermediate reasoning states, but the current ablations do not separate these two sources of improvement.

Questions for authors
  • Could the authors provide evidence that the action-gain ranking agrees with downstream task returns, particularly for THINK actions that write information useful to later recalls?

  • As the final objective combines several loss terms, could the authors provide a sensitivity analysis of the corresponding loss weights?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Gue7 (Part 1/3)

We thank the reviewer for recognizing the clear motivation and the practical direction of coordinating computation and memory access in one latent action space, and address below the credit structure behind the one-step action gain and the separate contribution of each memory source.

W1 and Q1. Downstream credit for latent actions and memory writes, and agreement with final returns.

The one-step gain is not meant to carry downstream credit, since Section 2.2 leaves that to LactL_{\mathrm{act}}, which applies the final-return advantage to every action taken, and to LmemL_{\mathrm{mem}}, which credits THINK writes through later recalls.

On positive-advantage trajectories, a realized THINK write gets credit from every later state where RECALL is admissible, whether or not RECALL is executed, so a write that helps a later recall is reinforced (Eqs. 9 and 10). The one-step gain gives the policy dense credit at individual states, where sampled credit vanishes if all continuations take the same action (Appendix C). With GRPO-only credit, policy entropy fell to 0.21 nats within 200 steps, versus 0.68 with CPD (RQ1). VinePPO [1] completed branches to the final answer but scored lower than one-step credit at 3.6× versus 1.1× the GRPO-only training compute (Table 2).

To further demonstrate that the local ranking tracks downstream return, following the reviewer's question, we sampled 4,096 states from Ouro-1.4B-Thinking test rollouts, 1,024 per combination of task family and state type (RECALL-admissible or THINK-only), and completed each admissible action to the final answer eight times with the trained policy fixed. An action's Monte Carlo advantage is its mean task reward over those completions minus the mean across admissible actions. Kendall τ\tau measures ranking agreement. When the rankings pick different actions, the return gap is the mean difference in their task returns on a 0-to-1 scale.

State subsetKendall τ\tauTop-action agreementReturn gap if different
All sampled states0.7183%0.04
RECALL-admissible0.7886%0.03
THINK write later reused0.6681%0.05
THINK write never read0.7484%0.04
RECALL executed0.8088%0.03

The one-step ranking largely agreed with downstream return, including on states with later-reused THINK writes, where Kendall τ\tau was 0.66 and the two rankings picked the same action 81% of the time, with a mean return gap of 0.05 when they disagreed. In 7.4% of states where THINK was admissible, it had negative one-step gain but positive Monte Carlo advantage. The reused-write subset was identified after the fact, and only the one-step gain includes an inference-cost charge.

To further test whether a longer horizon pays, we retrained with two- and four-step credit on the same data and for the same number of steps. Gen. is mean exact match and Long is mean answer F1.

Credit variantGen.LongTraining compute vs. GRPO only
One step, LatentHarness55.849.71.1$\times$
Two steps55.949.91.6$\times$
Four steps55.950.02.6$\times$
Completed branches, VinePPO [1]54.948.63.6$\times$

Four-step credit gained just 0.3 long-context points at about 2.4 times the one-step training compute, with one seed per setting. The one-step gain still omits the delayed effect of an unchosen THINK write (Appendix A).

AuthorsOpenReview ↗

Author Response to Reviewer Gue7 (Part 2/3)

W2. Accuracy gains from prompt evidence versus THINK-written states.

We thank the reviewer for this distinction between the two memory sources, which RQ3 already draws at read time, where the THINK-written share of executed recalls rises with input length and hop count.

Table 4 reports this share, and Appendix B.5 splits each read MqMq into the memorized-prompt read MxqM_xq and the change due to THINK writes. The share classified as derived rose from 27.4% at 4k to 33.2% at 64k and from 29.5% on two-hop to 40.8% on four-hop MuSiQue. With gain credit, 58% of reused writes had g>0.7g>0.7, versus 6% of unread writes (RQ4). These statistics show that both sources are read.

To measure how much each source adds to accuracy, we retrained two variants with the same initialization, data and training steps. No derived writes disables THINK writes and keeps only prompt memory, and No prompt memorization starts with empty memory and keeps only THINK writes. In the trained model, we also replaced each RECALL read with MxqM_xq. Gen. is mean exact match, Long is mean answer F1, and the MuSiQue columns are answer F1 by hop count.

VariantGen.LongMuSiQue 2-hop3-hop4-hop
LatentHarness55.849.742.127.316.4
No derived writes54.647.441.024.812.5
No prompt memorization55.643.336.222.012.1
THINK and EXIT only (Table 2)53.642.035.421.611.8
Read-time MxqM_xq only55.347.941.325.313.0

Both sources improved accuracy. Removing prompt memorization hurt long-context F1 most, from 49.7 to 43.3, while disabling THINK writes hurt more as hop count rose, lowering MuSiQue F1 from 16.4 to 12.5 at four hops against 42.1 to 41.0 at two hops. The drops do not give an additive split because THINK writes can overwrite prompt entries (Eq. 2) and retraining can change action choices.

We will add the memory-source ablations to Table 2 and the read-time knockout to RQ3.

AuthorsOpenReview ↗

Author Response to Reviewer Gue7 (Part 3/3)

Q2. Sensitivity to the loss weights, the CPD schedule and the action cost.

Table 2 removes each loss term entirely, so here we also vary each weight around its default.

In the Section 2.2 objective, β\beta weights credit from sampled actions, γ\gamma weights distillation from the counterfactual teacher, λ\lambda weights credit to the write gate, and η\eta weights regularization of the backbone and memory maps. The temperature ζ\zeta controls how sharply the teacher favors actions with higher gains, and ρ\rho sets the cost of THINK and RECALL in those gains. We anneal γ\gamma from 1.0 to 0.1 over the first 1.5k steps so the teacher shapes the policy early and later keeps it from committing too soon. Among the zero settings in Table 2, removing the teacher caused the largest accuracy drop, and removing the action cost caused the largest increase in FLOPs.

We varied one weight or schedule at a time on Ouro-1.4B-Thinking, keeping the initialization, data and 3k training steps fixed. The defaults (β=1.0\beta=1.0, annealed γ\gamma, λ=0.5\lambda=0.5, η=0.01\eta=0.01, ζ=0.5\zeta=0.5, ρ=0.25\rho=0.25) gave 55.8 Gen. exact match and 49.7 Long answer F1 at 55% of full-depth FLOPs, with a mean policy entropy of 0.68 nats. Retention is the cosine between the memorized prompt value and its readback after 2k generated tokens (Appendix B.5).

ParameterTested value or scheduleGen.LongAdditional measure
β\beta0.555.649.4
β\beta2.055.549.3entropy 0.52
γ\gammafixed 1.055.148.8entropy 0.41
γ\gammafixed 0.155.348.9entropy 0.57
γ\gammaanneal over 0.5k55.649.4
γ\gammaanneal over 3k55.749.6
γ\gamma0 (Table 2, GRPO only)52.344.371% FLOPs, entropy 0.21
λ\lambda0.2555.749.0
λ\lambda1.055.749.5retention 0.90
λ\lambda0 (Table 2, token-loss gate)55.548.1
η\eta0.00355.749.6
η\eta0.0355.449.2
ζ\zeta0.2555.649.4entropy 0.59
ζ\zeta1.055.549.2entropy 0.79
ρ\rho0.12556.049.971% FLOPs
ρ\rho0.555.248.946% FLOPs
ρ\rho0 (Table 2, no cost)56.150.093% FLOPs

Three observations follow. First, accuracy stays flat around the defaults for β\beta, λ\lambda, η\eta and ζ\zeta, with neither average moving by more than 0.7 points, so the method depends on keeping each signal, not on tuning its weight. Second, the schedule of γ\gamma matters more than its value. Holding γ\gamma at either 1.0 or 0.1 lowered both accuracy and policy entropy, and the training curves show why, with LCPDL_{\mathrm{CPD}} falling from 0.74 to 0.09 nats in the first 1.5k steps while the recall change used in LmemL_{\mathrm{mem}} rose from 0.02 to 0.31 nats, so the teacher does its work early and memory credit takes over later. Third, ρ\rho is the one weight that trades accuracy for compute, and it works as a dial, moving FLOPs from 71% to 46% of full depth for about one point of accuracy.

[1] A. Kazemnejad et al. VinePPO: Refining credit assignment in RL training of LLMs. ICML 2025, arXiv:2410.01679.

Thank you for the detailed rebuttal and the additional clarifications. My concerns are partially resolved. After considering the authors’ rebuttal and reading the comments from the other reviewers, I think the paper would benefit from a major revision in the following areas:

  1. The contribution of intermediate reasoning-state reuse remains insufficiently established. Under the delta rule, the purported derived component (M−Mx)q(M-M_x)q includes both newly stored values and modifications to existing prompt memory. The added write-removal and read-time ablations eliminate both effects, so their performance drops cannot specifically establish the benefit of reusing intermediate reasoning results. This leaves a substantive attribution problem concerning a central claim of the paper.

  2. The general-reasoning gains are substantially dependent on the evaluation protocol. In the response to Reviewer nZBT, changing the decoding protocol reduces the MATH500 advantage over Ouro from 8.8 to 1.7 percentage points, and over RLTT from 2.0 to 0.9 points. The original protocol truncates 34% of Ouro’s answers versus 11% of LatentHarness’s, while the additional experiment changes both the token budget and sampling parameters. Consequently, the contribution of decoding constraints to the original gains remains unresolved, weakening their support for a broader improvement in reasoning capability.

With the necessary comprehensive experiments and revisions, I believe this work has strong potential. At the current stage, however, I am not convinced. I cannot give a positive recommendation.

Reviewer_DjuJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Latent Harness proposes a neat method to boost the long-horizon reasoning performance of Looped Language Models. By combining fast weights memory with a macro policy selecting over THINK, RECALL, and EXIT options, along with counterfactual policy distillation, Latent Harness produces steady performance gains across both long-context and general reasoning benchmarks.

Critical strengths

The largest strengths of Latent Harness in my estimation are 1) empirical gains, and 2) clever architectural design. The empirical gains (contingent on being clarified + statistically validated) speak for themselves - compared with the strongest RLTT baseline, across both the 1.4B and 2.6B Ouro scales, Latent Harness achieves stronger performance on every benchmark while also using fewer FLOPs. The architectural design is also very clever for long-context settings as augmenting a looped model with fast weights memory allows the model to choose between spending compute on recall or refinement.

Critical weaknesses

The greatest weaknesses are 1) lack of shown generalizability to other looped/recurrent models, and 2) unclear benchmark metric reporting. While LatentHarness can be reasonably concluded to work well for Ouro, I am left to wonder how it would work for other looped/recurrent models such as Huggin. The metric reporting is also unclear. Were multiple samples used in the evaluations? (for example MATH-500 percentages are not always in 0.2% increments). Without measures of statistical uncertainty (e.g stds from different training runs or t-tests), the benchmark gains are also unclear.

Questions for authors

Do you think LatentHarness will scale well for looped/recurrent models which use higher loop counts?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer DjuJ (Part 1/2)

We thank the reviewer for recognizing the gains over the strongest baseline at fewer FLOPs and the design that chooses between recall and refinement, and address below the transfer to a second looped family and the statistical reporting behind the benchmark gains.

W1a and Q1. Transfer beyond Ouro and use of larger loop budgets.

LatentHarness is not tied to Ouro. It only needs a looped model that can decode its hidden state after any recurrence, as Huginn can. Each extra THINK step must pay for itself, so a larger loop budget adds computation only where it helps.

LatentHarness replaces the model's exit gate with a latent action policy and adds memory maps, leaving the shared block, head and tokenizer untouched. The teacher ranks THINK above EXIT only when one more step improves the next token by more than its cost. This is why Ouro-1.4B uses only 2.00 to 2.55 of its four available THINK steps per answer position (Table 4), and why half of the extra loops under All-Think change nothing.

We tested transfer on the reviewer's suggested backbone, Huginn-0125 [1] (3.5B), which has a prelude, a recurrent core and a coda that can decode after each recurrence. We attached the policy and memory to the core output and used the coda to decode both the states the policy visited and the states from one-step branches. We kept the released N=32N=32 budget and copied exit-step keys and values forward. All learned methods used the same data, 3k training steps, G=8G=8 and the other Table 3 settings. Huginn adaptive exit used the released KL threshold of 5×10−45\times10^{-4}. Gen. and Long are mean exact match and mean answer F1, with FLOPs relative to full r=32r=32 execution and latency in seconds per answer at batch size one.

Huginn-3.5B armGen.Gen. FLOPsGen. lat. (s)LongLong FLOPsLong lat. (s)
Huginn r=32r=3258.9100%6.838.5100%14.1
Huginn adaptive exit58.461%4.438.063%9.3
Huginn (GRPO)60.5104%7.040.7106%14.8
RLTT60.771%5.143.168%10.0
LatentHarness62.431%3.050.628%5.4

The gains carried over to a different loop architecture with eight times the loop budget. LatentHarness beat RLTT, the strongest Huginn baseline, by 1.7 points on general reasoning and 7.5 on long-context reasoning at less than half its FLOPs, close to the corresponding margins of 1.5 and 7.9 points on Ouro-1.4B (Table 1).

We trained a separate Huginn-3.5B variant at each loop cap and measured general-suite FLOPs relative to full r=32r=32 execution.

NNGen.LongGen. FLOPsTHINK gen. / longRECALL long
860.247.316%4.1 / 3.60.31
1661.649.424%6.8 / 5.90.36
3262.450.631%9.3 / 8.10.40
6462.650.934%10.1 / 8.80.41

A larger cap does not make the policy run every loop. Doubling the cap from 32 to 64 added less than one THINK step per position on average and still raised both scores, so the extra budget is spent only where it helps. Ouro-1.4B showed the same pattern at N=8N=8, with the averages rising to 56.3 and 50.4 while FLOPs rose only from 55% to 58% of full depth.

We will add the Huginn comparison to Table 1 and the loop-budget sweep to Appendix B.

AuthorsOpenReview ↗

Author Response to Reviewer DjuJ (Part 2/2)

W1b and W2. Evaluation sampling and statistical uncertainty of the gains.

Table 1 averages scores from five sampled answers per question (avg@5) with one training seed per cell, and paired bootstrap intervals across three seeds show that the Ouro-1.4B gains exceed run-to-run variation.

Every method used the zero-shot Thinking chat template at temperature 0.6 and top-p 0.95, with caps of 2,048 new tokens per general answer and 1,024 per long-context generation turn. We scored each answer by exact match on general reasoning and answer F1 on long-context reasoning, using the same extractor and matching rules for every method (Appendix B.2). Averaging the five scores, instead of reporting pass@5 or using majority voting, gives unrounded increments of 1/25001/2500 on MATH500 and 1/9901/990 on GPQA-Diamond, so MATH500 scores need not be multiples of 0.2 points.

To further demonstrate that the gains exceed run-to-run variation, following the reviewer's suggestion, we ran three Ouro-1.4B training seeds for LatentHarness, RLTT (the strongest general baseline) and Memory-R2 (the strongest long-context baseline), with the same backbone, data, RL steps and evaluation. The first seed was the one reported in Table 1. The table reports mean ± sample standard deviation across seeds.

MethodGen. exact matchLong answer F1
RLTT54.1 ± 0.541.6 ± 0.5
Memory-R252.4 ± 0.544.9 ± 0.6
LatentHarness55.7 ± 0.449.5 ± 0.3

The seed standard deviations were at most 0.6 points, well below the family-level gains. We computed 95% paired bootstrap intervals from 10,000 resamples of questions within each suite and of training seeds. The table reports LatentHarness minus the baseline in points, with Gen. and Long each averaging their three suites equally.

ComparisonGSM8KMATH500GPQAGen.2WQAMSQHQALong
vs. RLTT+1.8 [0.9, 2.7]+2.1 [0.6, 3.6]+0.9 [-2.4, 4.2]+1.6 [0.4, 2.8]+8.4 [7.2, 9.6]+8.6 [7.3, 9.9]+6.9 [5.9, 7.9]+7.9 [7.0, 8.8]
vs. Memory-R2+3.9 [2.9, 4.9]+4.1 [2.5, 5.7]+1.9 [-1.4, 5.2]+3.3 [2.0, 4.6]+4.7 [3.6, 5.8]+4.8 [3.6, 6.0]+4.3 [3.4, 5.2]+4.6 [3.8, 5.4]

LatentHarness improved both task-family averages against both baselines, with all four intervals excluding zero, including the +7.9 [7.0, 8.8] long-context gain over RLTT. Every interval except GPQA-Diamond's excluded zero. On Ouro-2.6B, the single-seed gains over RLTT were +1.9 [0.6, 3.2] points on general reasoning and +8.1 [7.2, 9.0] points on long-context reasoning, with intervals reflecting variation across test questions only.

We will state avg@5 and the decoding protocol in Table 1 and Appendix B.2, and add the three-seed results and paired intervals to Appendix B.

[1] J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv:2502.05171, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.