- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The motivation, as written in the introduction and suggested in the title, is that while text-based reasoning in vanilla autoregressive LLMs has paved the way for various methods to access and manage external context, in the form of text, outside of the model's context window, there is no established approach for a looped transformer to do the same within its "latent reasoning". My understanding is that this is fundamentally because existing methods are based on tool calls which are expressed in explicit text/tokens, which is incompatible with "latent" reasoning, which is simply repeated propagation of hidden states through transformer blocks.
While this motivation is very intriguing, the proposed methodology does not align with this motivation. The main idea is to add a residual stream of RNN-style state-space modeling through the recurrent depth axis, propagating a "shared fast-weight associative memory", or 'latent memory" in the form of a d x d matrix where d is the hidden dimension. The proposed model adds some form of a decision head to enable the model to select between THINK, RECALL, and EXIT. THINK and RECALL means taking another pass through the transformer blocks, writing to and reading from the latent memory, respectively. EXIT means terminating the loop and decoding the output token.
The authors propose complex novel additions to an existing approach for reinforcing latent actions (RLTT) to account for THINK and RECALL with appropriate credit assignment, namely, Counterfactual Policy Distillation (CPD).
A key concern I have with the paper is that it provides very little detail on how these operations interact with the KV cache, simply stating that "We leave the block’s attention over earlier positions implicit". This is one of the most critical details for this method, as it is central to understanding the following:
- (1) How does the added residual stream via latent memory through the recursive depth axis interact with the sequence axis. Modeling information across a long sequence is typically what we refer to as "memory" and long-context modeling. The authors associate gains in long-context evaluations such as HotpotQA with the "memory" mechanism of the proposed method.
- (2) The claimed theoretical efficiency gains appear to stem from early-exiting but there is no mention of how early-exiting is handled in practice with respect to KV cache. Variable-depth recursion in looping transformers is tricky because early-exiting leaves gaps in the KV cache for subsequent iterations which are required for attention when a later token does not exit early. There are complex ways to account for this as shown in Mixture-of-Recursions [1] but I believe it is considered impractical:
vLLM does not currently support the adaptive exit feature ... the model will always execute the full number of total_ut_steps[2]True, we not use early exit in vllm. It's hard to handle kv cache here.[3]
Note on confidence: I have deep background in context management, KV cache, vanilla transformer architecture, and general knowledge of loop-based transformers. I am unfamiliar with RNN-style state-based models and fast-weight associative memory.
[1] https://arxiv.org/abs/2507.10524 [2] https://huggingface.co/ByteDance/Ouro-1.4B-Thinking [3] https://github.com/vllm-project/vllm/pull/27794#discussion_r2477168197
Critical strengths
- S1. I think that latent memory along the recursive depth axis to supplement the small representative capacity of the hidden state may be useful, and the proposed CPD method seems like a reasonable approach to help supervision. However, this differs from the motivation in the paper (see W1).
- S2. The accuracy gains and claimed efficiency gains are significant, but there are significant concerns (see W2, W3).
Critical weaknesses
- W1. The motivation mentioned in the early half of the introduction and the method do not align (see Paper Summary). It follows that there is insufficient justification for the results.
- W2. There are significant concerns regarding the soundness of the accuracy results. Notably:
- The reported GSM8K and MATH scores for Ouro 1.4 (74.6, 42.8) significantly differ from that of the Ouro paper's numbers on Table 7 (78.92, 82.40) [1]
- The comparison with text-based agentic memory methods (Search-R1, MemAgent, MEM1, Memory-R2) seem out of place because (1) the evaluation benchmarks do not involve handling large external context, (2) the Ouro base models are not trained with agentic capabilities, and (3) their relevance comes from the written motivation, which does not align with the actual method (W1).
- W3. There are significant concerns regarding the soundness of the efficiency results. Notably:
- It is unclear if the gains in FLOPS are attainable in real inference scenarios, as these seem to stem from early-exiting, for which adequate explanation of the exact mechanism is not provided (see Paper Summary). Details regarding early-exiting and handling of KV cache, detailed accounting cost components, e.g., prefill vs decode cost, would help support the soundness of the results.
- It is unclear how the wall time numbers are derived. The paper only mentions
We report FLOPs relative to the full-depth backbone and per-answer latency following Wang et al. (2025b), but the cited paper also does not provide clear explanation. Details regarding the measurement such as measurement protocol, hardware, software libraries (exact versions, options, kernels, etc) would clarify things and support the soundness of the results.
Questions for authors
See weaknesses.
5 comments
Author Response to Reviewer nZBT (Part 2/4)
W1b. How the motivation maps onto the method.
We respectfully disagree that the motivation and the method diverge. The abstract identifies two bottlenecks in text-based reasoning, reading long inputs and generating intermediate steps, and the latent actions address both.
On the input side, LatentHarness writes the prompt into fast weights once, and RECALL then reads evidence from memory instead of rereading the context. The read takes one matrix-vector product inside the loop and generates no text. On the output side, text-based reasoning spends tokens on intermediate steps. THINK instead runs another recurrent step in latent space, and EXIT ends the loop and emits the next token, so each output token gets only the computation it needs. In Table 1, LatentHarness improves on the strongest baselines in long-context F1 and general accuracy while using 48% and 62% of full-depth FLOPs, respectively. At 64k, 63.9% of evidence recalls hit supporting passages, and the share of recalls that read THINK-written states grows with input length (Table 4).
We tested retrieval beyond the attention window by extending the suites to 96k and 128k, past Ouro's 65,536-token positional range [4]. RLTT and LatentHarness attended only to the latest 64k prompt tokens, but LatentHarness also wrote every prompt token into memory in one streaming pass. Search-R1 retrieved from all passages, and Memory-R2 read the prompt in chunks, so both retained access to the full input.
| Method | 64k F1 | 96k F1 | 128k F1 |
|---|---|---|---|
| RLTT | 31.2 | 27.6 | 24.0 |
| Search-R1 | 38.1 | 36.5 | 35.2 |
| Memory-R2 | 39.4 | 37.8 | 36.9 |
| LatentHarness | 45.5 | 43.4 | 41.2 |
Answer F1 is pooled over HotpotQA, MuSiQue and 2WikiMultihopQA using the Section 3 scoring protocol.
The memory retrieved evidence outside the active attention window. At 128k, 71% of evidence recalls addressed passages outside that window, and 58.3% of all evidence recalls hit a supporting passage.
W2a. Why our Ouro numbers differ from those in the Ouro paper.
The scores come from different checkpoints and protocols, with Ouro's Table 7 [5] evaluating Ouro-1.4B base few-shot and our Table 1 evaluating the released Ouro-1.4B-Thinking zero-shot.
Ouro's Table 16 [5] specifies 3-shot CoT for GSM8K and 5-shot CoT for MATH500, with an in-house pipeline for MATH500. The Ouro paper reports neither score for the Thinking checkpoint. Our Table 1 labels MATH500 as MATH and uses the zero-shot Thinking chat template, with five sampled answers per question at temperature 0.6 and top-p 0.95 and a cap of 2,048 new tokens. It reports avg@5, or mean exact match across the five samples with one answer extractor for all methods, not greedy accuracy. The submission should have stated this protocol.
To further demonstrate that our evaluation is not the source of the gap, following the reviewer's concern, we reproduced the published base protocol and tested Ouro-1.4B-Thinking, RLTT and LatentHarness using Ouro's Thinking decoding settings (temperature 1.0, top-p 0.7), an 8,192-token budget and the mean over four samples per question.
| Checkpoint and protocol | GSM8K | MATH500 |
|---|---|---|
| Ouro-1.4B base, published Table 7 | 78.92 | 82.40 |
| Ouro-1.4B base, our Table 7 reproduction | 78.5 | 81.9 |
| Ouro-1.4B-Thinking, our Table 1 protocol | 74.6 | 42.8 |
| Ouro-1.4B-Thinking, 8,192-token sampling | 86.2 | 83.9 |
| RLTT, 8,192-token sampling | 86.8 | 84.7 |
| LatentHarness, 8,192-token sampling | 88.1 | 85.6 |
The first two rows use the base checkpoint and Ouro's few-shot protocol. The remaining rows use the Thinking checkpoint and the listed sampling protocols. All scores are exact-match accuracy.
Our evaluation recovered the published base scores within 0.5 points, so the gap reflects checkpoint and protocol differences. LatentHarness still led both baselines under the longer protocol, by smaller margins than in Table 1. The shorter cap truncated 34% of Ouro-1.4B-Thinking's MATH500 answers and 11% of LatentHarness's, making MATH500 sensitive to the cap, although the longer protocol also changed temperature and top-p.
Author Response to Reviewer nZBT (Part 3/4)
W2b. Why token-space memory agents are relevant baselines.
The motivation for these baselines lies in the long-context suites, where every method reads the full 4k to 64k distractor-padded context and every agent shares our backbone, data and RL steps.
Following the long-context setting of MemAgent [6] and Search-R1 [7], each method must find supporting passages among distractors. The baselines span the token-space ways of managing such input. Search-R1 retrieves passages, MemAgent and Memory-R2 read and write textual memory, and MEM1 keeps a compact state for memory and reasoning. LatentHarness reads memory inside the latent loop. None of the agents needs a separately pretrained controller, and all four gained 4.4 to 6.0 long-context points over memory-free GRPO on Ouro (Table 1). The W1b response above also tests them beyond the attention window, the setting the Introduction motivates.
To check that a backbone without agentic training does not handicap these recipes, we trained MemAgent, Memory-R2 and Search-R1 on Qwen3-1.7B, a size-matched dense backbone, keeping the same training data and 3k RL steps as on Ouro.
| Backbone | MemAgent | Memory-R2 | Search-R1 |
|---|---|---|---|
| Ouro-1.4B-Thinking | 43.6 (50.9) | 45.2 (52.6) | 44.1 (50.4) |
| Qwen3-1.7B | 45.1 (51.6) | 46.3 (53.0) | 45.0 (51.2) |
Each cell gives mean long-context answer F1, with mean general exact match in parentheses.
Switching to Qwen3-1.7B improved long-context F1 by only 0.9 to 1.5 points, suggesting that Ouro's recurrence did not limit the agents. LatentHarness on Ouro stayed 3.4 long-context and 2.8 general points ahead of the best Qwen3 agent.
W3b. Where the FLOPs savings arise and whether they hold in real inference.
The FLOPs savings come from running fewer recurrent steps per token, mostly in prefill on long inputs and in decoding on general reasoning, and they carry into wall-clock time, with LatentHarness using 48% of Ouro's FLOPs and 58% of its time on long-context reasoning.
Copy-forward fills the cache for tokens that exit early, so later tokens can attend to them without recomputing skipped steps (W1a and W3a). We split the FLOPs in Table 1 into prompt prefill and answer decoding. Each phase includes memory writes, read probes and policy operations, and FLOPs are relative to full-depth execution over the same tokens.
| Family | Method | Depth P/A | FLOPs P/A | Total FLOPs | Generated tokens |
|---|---|---|---|---|---|
| General | Ouro | 4.0/4.0 | 19.7/80.3% | 100% | 612 |
| General | RLTT | 3.4/3.5 | 17.0/69.0% | 86% | 598 |
| General | LatentHarness | 2.4/2.5 | 11.8/50.2% | 62% | 541 |
| Long | Ouro | 4.0/4.0 | 99.3/0.7% | 100% | 178 |
| Long | RLTT | 3.3/3.3 | 81.4/0.6% | 82% | 170 |
| Long | LatentHarness | 1.9/2.1 | 47.7/0.3% | 48% | 150 |
P/A denotes prompt prefill and answer decoding. Depth is the mean number of block-equivalent applications per position.
Most savings come from decoding on general reasoning. On long-context reasoning, prompts average 24.8k tokens and prefill accounts for most of the savings even with prompt memorization. On those inputs, wall-clock time falls less than FLOPs because block counts miss some overhead.
We also measured batched decoding throughput in tokens per second on one NVIDIA H200 141GB, with 4k-token inputs and all recurrent-step caches resident.
| Batch size | Ouro | RLTT | LatentHarness |
|---|---|---|---|
| 1 | 118 | 139 | 186 |
| 4 | 310 | 365 | 470 |
| 16 | 470 | 545 | 680 |
At batch 16, the largest size tested, LatentHarness still decoded faster at 680 tokens per second against 470 for Ouro.
Author Response to Reviewer nZBT (Part 1/4)
We thank the reviewer for recognizing the latent memory along the recursive depth axis and the CPD supervision, and address below the KV-cache mechanics of memory and early exit and the soundness of the accuracy and efficiency results.
LatentHarness keeps ordinary attention over the KV cache and copies an early-exited token's last computed keys and values into the skipped recurrent steps, adapting the widely used early-exit idea of copying hidden states forward, as in Depth-Adaptive Transformer [1] and CALM [2]. A separate fast-weight memory carries evidence across positions and reads a value for a query key from one d×d matrix instead of attending over all cached keys.
In the shared block described in Section 2.1, if a token exits after recurrent step t<N, later tokens that reach steps t+1,…,N still need its cache entries. We fill those entries in each layer with the keys and values from step t, a rule we call copy-forward. Later tokens can then attend to earlier positions without rerunning the block for an exited token. Mixture-of-Recursions [3] discusses sharing first-step keys and values or masking missing entries, while copy-forward keeps the last computed ones, and we apply it in prefill, decoding and training rollouts.
THINK writes a key-value pair to the matrix with a gated delta rule, and RECALL multiplies the matrix by a query key (Eqs. 2 and 3). A read adds no tokens to the cache and costs about 1% of one block application. The memory recovers evidence that attention misses at long range. At 64k, 63.9% of evidence recalls hit a supporting passage (Table 4), while the passage receiving the most attention was a supporting one at only 31.5% of answer positions, down from 58.7% at 4k.
We compared four cache rules on Ouro-1.4B-Thinking under the Section 3 evaluation protocol. The oracle computed the missing keys and values at deeper steps but still decoded at the selected exit step.
Copy-forward came within 0.1 general and 0.2 long-context points of the oracle, while masking scored 4.6 general and 6.7 long-context points below copy-forward.
We also retrained three variants for 3k steps from the same initialization and data to test what the memory needs to carry into later output positions.
General is mean exact match and long-context is mean answer F1. Support hit is the share of evidence recalls that address an annotated supporting passage.
Prompt memorization accounted for most of the gain. Resetting to zero brought F1 close to THINK and EXIT, and resetting to Mx cost 1.2 long-context points, showing an additional benefit from carrying memory across positions. We will replace the implicit-attention sentence of Section 2.1 with a description of the KV cache, the memory and copy-forward, and add the cache-rule comparison, the phase accounting and the timing protocol to Appendix B.3.