acceptodds
Under review as a conference paper at ICLR 2027

BRANCHMEM: Branching Reasoning with Persistent Latent Memory for Language Model Agents

Abstract

Agent memory enables LLM-based agents to improve by accumulating and reusing past experience. Latent memory provides a more compact representation of past experience than raw trajectories, but existing approaches often either absorb experience into shared trainable components or require additional optimization for each explicit latent memory. We introduce BRANCHMEM, a framework that uses a lightweight compressor to transform successful experiences into persistent latent memories. Each successful experience is encoded once into a memory that can later be retrieved and reused for subsequent tasks, without updating the underlying reasoner or optimizing each memory separately. We further find that different latent memories often alter generation only at a few critical points, yet these differences can redirect the entire reasoning process. Based on this observation, BRANCHMEM introduces Action Branching, which allows different latent memories to guide alternative reasoning paths at critical points while preserving the reasoning already performed, enabling more effective use of past experience for solving new problems. We evaluate BRANCHMEM across mathematical reasoning, code generation, and information-seeking tasks, demonstrating its ability to leverage past experience for new problems and showing how Action Branching further improves reasoning by exploiting alternative paths induced by latent memories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Latent Reasoning, LLM Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mGL3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BranchMem stores successful trajectories as latent memories for a frozen LLM reasoner. A shared compressor (Llama-3.2-1B) turns each trajectory into a fixed-length latent value, kept in a bank together with a key built from the task input. For a new query, the top-8 memories are retrieved, and each trajectory is conditioned on one memory at a time. The authors compare the retrieved memories under a fixed prefix, using teacher-forced next-token scoring against an anchor trajectory generated with the top-1 memory. Their aligned analysis reports agreement over shared prefixes and first divergences concentrated at low-margin positions. Within rescue cases, retrieval scores provide little separation between successful and unsuccessful candidates at ranks 2–8. Action Branching builds on this: it groups candidates by their first divergent position and alternative token, expands a representative branch, and re-probes the remaining memories in that group under the new prefix. Experiments use Llama-3.1-8B and Llama-3.2-3B on GSM8K, KodCode, TriviaQA, PopQA and StrategyQA, and report solution coverage, where a query counts as solved if any generated trajectory is correct.

Critical strengths
  1. Comparing different latent memories under the same reasoner and the same prefix is a good way to ask what a memory actually does, and the association between first divergence and low-margin positions is reported consistently across two models, two training objectives and two tasks (Table 2, Figure 3).

  2. The authors are fairly open about limits. Appendix B.3 shows where Action Branching falls short of independent generation from the same top-8 set, and reports the token savings under B=1 (81–86%). The retrieval analysis (RQ3) and the residual intervention (B.1) add useful context to the main table.

Critical weaknesses
  1. The main results do not separate the effect of the method from the effect of extra attempts. In Table 1, BranchMem is scored by whether any trajectory in a tree is correct, but the comparisons with Vanilla, TokMem and MemGen do not establish matched candidate counts or compute budgets. When I look at the one-trajectory comparison (BranchMem Top-1), the gain over existing memory methods is small. On Llama-3.1-8B, SFT Top-1 averages 63.27 against 63.81 for TokMem, and GRPO Top-1 averages 64.49 against 64.26 for MemGen-GRPO. On Llama-3.2-3B, SFT Top-1 is below both TokMem and MemGen-SFT. All four Top-1 variants have higher five-task averages than Vanilla, which supports the interface’s usefulness, but the averages do not establish a clear advantage over existing memory methods. The paper's own Table 7 makes the picture less favorable for branching. Independent generation from the same top-8 memories has higher coverage than the full tree in Table 1 in all 20 settings (by 0.07 to 3.05 points, 0.69 on average). These comparisons suggest a potential cost–coverage tradeoff rather than a coverage advantage over Independent Top-8. But the cost of the full tree is never reported, since Table 7 only covers B=1. Teacher-forced scoring of the remaining candidate memories at each node is not included in the token accounting either, and there is no no-memory baseline given the same number of samples. StrategyQA needs a separate note: answer coverage on a yes/no question becomes 100% whenever both valid answers appear in the tree. The rise from 65.93 to 83.41 therefore does not by itself establish better reasoning without the answer distribution or final-answer selection accuracy. The restricted B=1 results provide evidence of a token–coverage tradeoff, but the main-table gains do not establish more effective use of past experience at a matched total cost.

  2. It is not shown that the content of a memory matters, or that the "localized intervention" is specific to memory. The statistics in Table 2 condition on the first position where a candidate’s argmax differs from the anchor token. This selection may favor states where perturbations can change the decision. Although the measured log-probability changes and no-memory margins are not consequences of the definition alone, matched perturbation controls are needed to establish whether the pattern is specific to retrieved experience. Random or averaged memories, a learned constant prefix, or controlled prompt perturbations should be compared on both the branch-point statistics and task performance. The residual experiment in B.1 does show that the specific residual matters for which branch events are reproduced (100% for the own residual, 46.7% when shuffled). But it measures event recovery, not task success, so it does not show that memory content helps solve problems. There is also a training issue. By Eq. 3 and Appendix A.2, the latent is encoded from the same target trajectory that the model is then trained to reproduce (SFT) or to condition on (GRPO), whereas at test time the latent comes from a different problem. The interface may learn same-trajectory reconstruction rather than transferable experience, and the paper does not test this. Since the paper presents its contribution as reuse of past experience, and the motivation for Action Branching rests on this analysis, I think the point needs to be settled.

Overall, the aligned-comparison analysis is interesting, and the Top-1 five-task averages improve over Vanilla. But I cannot separate the main gains from the extra attempts, the cost of the full tree is missing, and the role of memory content has not been tested against matched controls. A cost-matched comparison and a content control could change my view.

Questions for authors
  1. Clarification: What decoding is used (greedy or sampling, temperature, top-p)? Under matched greedy decoding and stopping rules, what accounts for the difference between full-tree and Independent Top-8 coverage? If the anchor is sampled, how do you make sure the divergences measured against the argmax in Eq. 5 are not just sampling noise?
  2. Assessment-relevant comparison: What is the total cost of the full tree, counting both generated tokens and teacher-forced scoring passes, compared with Independent Top-8 and with no-memory sampling at the same number of samples?
  3. Assessment-relevant comparison: How do random, shuffled-across-questions and averaged memories compare with real retrieved ones, both in coverage and in the Table 2 statistics?
  4. Clarification and assessment-relevant comparison: Is the latent for a training example ever built from a different example's trajectory (cross-sample or leave-one-out)? If not, what happens when the interface is trained that way?
  5. Clarification and assessment-relevant comparison: On StrategyQA, in what fraction of trees do both answers appear, and what is the accuracy if one answer has to be chosen, for example by majority vote? Also, why does full SFT BranchMem on the 3B model fall well below Vanilla on GSM8K (69.90 vs 77.78)?
3 comments

Thank you for suggesting these controls. First divergences remain concentrated at low-margin positions after cross-query shuffling. Retrieving memories for the current query yields higher solution coverage. We also find that cross-example training achieves held-out coverage close to that of the original interface on GSM8K and KodCode.

W1 / Q1 · Attempt counts, decoding, and coverage

Decoding setup. The anchor, branch continuations, and Independent Top-8 under the same candidate pool all use greedy decoding in the submitted experiments. Temperature and top-pp do not affect those runs. Equation (5) compares the candidate memory's top token with the token chosen by the anchor under the same prefix. The anchor is generated greedily, so this comparison is free of sampling noise from the anchor.

Complete-tree and Independent Top-8 coverage. Under the same candidate pool, a complete grouped tree recovers the independently generated trajectories when next-token decisions and stopping rules agree. The submitted implementation capped the number of tokens scored during teacher-forced probing. A candidate whose first divergence fell beyond that limit was missed. Numerical differences between batched scoring and decoding can also change the top-ranked token when logits are nearly tied. In the rerun, we scored the full anchor and matched the candidate pool, greedy decoding, EOS handling, and generation limit. After the correction, the complete tree shares prefixes among candidates that choose the same actions and expands alternatives when their actions later diverge. The two methods attain the same solution coverage in all 20 settings.

Each cell below gives the solution coverage attained separately by both the complete tree and Independent Top-8 in that setting.

DatasetLlama-3.1-8B SFTLlama-3.1-8B GRPOLlama-3.2-3B SFTLlama-3.2-3B GRPO
GSM8K78.09%88.02%70.65%83.54%
KodCode71.95%68.00%69.65%62.85%
PopQA43.17%47.72%42.40%39.59%
StrategyQA74.23%86.46%72.19%71.61%
TriviaQA71.23%75.77%65.45%69.97%

Generation volume in Table 1. In Table 1, Vanilla, MemGen, TokMem, and BRANCHMEM Top-1 each generate one trajectory per query. The complete tree reuses existing prefixes to explore alternatives on GSM8K and KodCode with a similar number of newly decoded tokens to a single generation by Vanilla or MemGen-GRPO.

Mean newly decoded tokens per queryGSM8KKodCode
Vanilla, single sample174.7401.8
MemGen-GRPO, single generation176.1339.0
TokMem retrieval extension, single generation145.9131.0
Complete Action Branching tree199.7401.0
Independent Top-81,429.62,761.6
Reduction relative to Independent Top-886.03%85.48%
Vanilla, eight samples1,392.83,217.3

Independent Top-8 generates a complete trajectory from scratch for each candidate. Action Branching probes candidate next actions under the existing prefix and generates only a new suffix when an action diverges. Memories that propose the same alternative token at the same position share a branch point and are probed again under the new prefix. This sharing reduces the number of newly decoded tokens. The table counts the tokens newly decoded during complete tree construction, including the Top-1 anchor and the new suffixes of all branches. Each branch reuses the prefix before divergence.

On GSM8K and KodCode, Action Branching builds the complete tree with a newly decoded token budget comparable to a single generation from Vanilla or MemGen-GRPO. This supports the comparison with these baselines in Table 1. At a similar generation budget, Action Branching reuses the prefix before divergence and spends its new decoding on alternatives proposed by the candidate memories.

Coverage under the same attempt limit. We also compare solution coverage with eight attempts per query on the frozen Llama-3.2-3B Reasoner. Vanilla samples output tokens at T=0.2T=0.2 and top-p=1.0p=1.0. MemGen-GRPO applies Gaussian perturbations at scale 0.2 to the Weaver latents and decodes greedily. The TokMem retrieval extension generates from each of the eight highest-ranked memories. The complete tree returns up to eight trajectories. A query is covered if any trajectory is correct.

DatasetVanillaMemGen-GRPOTokMem retrieval extensionCorrected BRANCHMEM-SFT complete treeCorrected BRANCHMEM-GRPO complete tree
GSM8K81.72%79.45%74.83%70.65%83.54%
KodCode55.65%66.40%67.35%69.65%62.85%

On GSM8K, the corrected BRANCHMEM-GRPO complete tree reaches 83.54% coverage, compared with 81.72% for eight Vanilla samples. On KodCode, the BRANCHMEM-SFT complete tree achieves the highest coverage in the table. In the single-trajectory setting, all four BRANCHMEM Top-1 configurations in Table 1 have higher five-task averages than Vanilla. Comparisons with TokMem and MemGen vary with the training setup and Reasoner. The single-trajectory comparison shows the effect of the memory interface. Comparing the complete tree with Top-1 under the same interface quantifies the coverage gained by exploring alternatives within the candidate pool.

Q2 · Reasoner cost of the complete tree

The complete tree expands at most seven branches in addition to the anchor. With up to eight attempts per query, the complete tree decodes fewer new tokens than Independent Top-8 and eight Vanilla samples on both tasks (Q1). The table compares Reasoner cost on the same candidate pool and includes probing and prefix replay.

DatasetMethodTime, s/queryProcessed token positions per queryShare of time spent probing
GSM8KComplete tree5.142,393.812.51%
GSM8KBatched Independent Top-84.832,259.9—
KodCodeComplete tree10.525,448.713.09%
KodCodeBatched Independent Top-811.254,965.4—

Including probing and replay, the complete tree and batched Independent Top-8 have similar Reasoner times on the same pool.

W2 / Q3 · Memory content and retrieval

Retrieval and solution coverage. Retrieving memories for the current query improves coverage over cross-query shuffling by 3.13 and 4.56 percentage points on GSM8K and KodCode, respectively. Cross-query shuffling uses candidates retrieved for another query while retaining memories encoded from successful experiences. We also compare BGE semantic retrieval, random memories, and the memory-bank mean.

Memory conditionGSM8K solution coverageKodCode solution coverage
BRANCHMEM projected retrieval83.54%69.65%
BGE semantic retrieval83.32%68.88%
Random memories81.47%67.62%
Cross-query shuffled memories80.41%65.09%
Memory-bank mean77.18%63.41%

RQ3 and Table 6 in Appendix B.2 examine ranking within a retrieved pool. On TriviaQA and PopQA, among queries where Top-1 fails and Top-8 contains a correct solution, initial retrieval scores distinguish successful from unsuccessful candidates with a mean AUC of 0.498. RQ2 offers an explanation. Different memories can take the same actions along a shared prefix and propose different next steps only when they reach a particular reasoning state.

Branch-point statistics. To test whether the low-margin pattern in Table 2 depends on matching experiences to queries, we fix the anchor generated by the actual Top-1 retrieval for each query on both full test sets. We compare the other seven candidates retrieved for that query with seven candidates retrieved for a different query. The margin ratio below is the median no-memory Top-1/Top-2 margin at first divergence divided by the corresponding median at shared-prefix positions. AUC measures how well a low margin distinguishes these two sets of positions.

DatasetCandidate memoriesMargin ratioLow-margin AUC
GSM8KActual retrieval7.69%0.917
GSM8KCross-query shuffling7.35%0.916
KodCodeActual retrieval7.08%0.920
KodCodeCross-query shuffling7.47%0.914

Cross-query shuffling preserves the concentration of first divergences at low-margin positions and lowers solution coverage. The branch-point analysis compares actual retrieval with cross-query shuffling. The coverage table also includes random memories and the memory-bank mean. When we select low-margin positions using the no-memory Reasoner, the replacement effect on both tasks is still about three times that at random positions matched for progress through the trajectory.

Q4 · Cross-example training and generalization

Cross-example SFT achieves held-out solution coverage close to that of same-query pairing on both tasks. We freeze the Llama-3.2-3B Reasoner and train the interface from scratch on the original SFT training data for GSM8K and KodCode, separately. For each training query, BGE retrieves the most similar eligible query other than itself. We feed that other query's reference solution to the compressor, while the current query's reference solution remains the supervision target. KodCode uses the original length filter.

DatasetOriginal interface, Top-1Cross-example interface, Top-1Original interface, answer-pruned coverageCross-example interface, answer-pruned coverage
GSM8K65.57%64.44%69.90%69.90%
KodCode65.10%65.65%67.55%68.00%

Under the same answer-pruning protocol, training the interface with experiences from other queries yields held-out coverage close to the original interface. GSM8K remains at 69.90%, and KodCode changes from 67.55% to 68.00%, with a paired 95% bootstrap interval of [−0.65, +1.55] percentage points. These results show that cross-example pairing can train a memory interface that transfers to unseen queries.

Q5 · StrategyQA and SFT performance on GSM8K

Single-answer accuracy on StrategyQA. With Llama-3.1-8B GRPO, 54.59% of complete trees contain both Yes and No. StrategyQA uses the same agentic search setup as TriviaQA. Each trajectory can retrieve information and use that evidence to reach a yes-or-no judgment. Whether a branch reaches a correct conclusion therefore depends in part on the relevance and sufficiency of the evidence it retrieves. In this setting, coverage measures whether retrieval and reasoning yield at least one correct trajectory.

To assess whether that translates into better answer selection, we use a fixed rule to select one answer per query and then compare it with the reference answer. Each distinct trajectory with a valid answer casts one vote. Ties go to the first valid answer in traversal order, and we return Yes if no valid answer is available. With this rule, majority voting achieves 70.74% single-answer accuracy, exceeding the other single-generation methods with the same backbone in Table 1.

ReasonerContains both answersMajority-vote accuracyAnchor accuracy
Llama-3.1-8B54.59%70.74%67.39%

The 3B SFT gap on GSM8K. With Llama-3.2-3B, MemGen-SFT (66.18%) and TokMem (66.94%) also score below Vanilla (77.78%) and close to BRANCHMEM-SFT Top-1 (65.57%). Action Branching raises BRANCHMEM-SFT coverage to 69.90%. These similar results suggest that the reference solutions in the official GSM8K training split may provide limited direct supervision for reusing strategies across problems. With the same backbone and up to eight attempts per query, the corrected BRANCHMEM-GRPO tree achieves 83.54% coverage, compared with 81.72% for eight Vanilla samples. This gain shows that GSM8K remains informative for evaluating the benefit of BRANCHMEM.

Reviewer_oTE5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BRANCHMEM, which encodes successful trajectories once into persistent, independently retrievable latent memories via a shared compressor, without updating the frozen Reasoner. An aligned analysis shows that distinct memories behave identically over most of a trajectory and diverge only at a few low-margin decisions, and that retrieval scores barely separate useful from useless candidates. Building on this, Action Branching reuses the shared prefix and expands branches at memory-induced divergences. Across two frozen Reasoners and five benchmarks, it improves average coverage over the Top-1 variant by 4.01-7.84 points, and at B=1B{=}1 matches Independent Top-8 coverage within 2.65 points while reducing generated tokens by 81.44-85.76%.

Critical strengths
  1. The aligned characterization of latent-memory behavior is the main new knowledge: divergences concentrate at low-margin states and retrieval scores do not separate within-set utility (Tables 2, 6); both hold across all 8 configurations and directly motivate the method design.
  2. The gains are supported by like-for-like controls: against the Top-1 variant with the same interface and retrieval, branching never decreases coverage in any of the 20 settings and improves the average by 4.01-7.84 pp (Table 1); at B=1B{=}1 it nearly matches Independent Top-8 coverage at ~82-86% token reduction (Table 7).
Critical weaknesses
  1. The headline comparison conflates memory quality with trajectory count. Coverage counts a query as solved if any trajectory is correct (Section 4.1); in Table 1 BRANCHMEM produces a tree while baselines appear single-trajectory, and no budget-matched best-of-N control exists. My recomputation shows full-tree coverage is slightly below independent generation from the same candidate set in all 20 settings (Table 1 vs 7), so the gains cannot be credited to the method itself.
  2. The cost accounting is incomplete. The 81.44-85.76% token reduction excludes the teacher-forced probing overhead (Section 3.3), and the full-tree cost behind the main result is never reported -- only the B=1B{=}1 setting is accounted.
Questions for authors
  1. How many trajectories does each baseline generate in Table 1? What is the coverage of budget-matched no-memory best-of-N, and does BRANCHMEM still beat it?
  2. What is the average total generation cost per query (including probing) and the branch budget cap for the full trees in Table 1?
  3. Does the branch-point statistic remain significant when positions are selected independently of the divergence definition (e.g., pre-selected low-margin states)? The current selection rule itself inflates the ratio.
2 comments

W1 · Coverage of the corrected complete tree and Independent Top-8

The discrepancy you identified was between Tables 1 and 7 of the submitted manuscript. The submitted implementation capped the number of tokens scored during teacher-forced probing. When a candidate first diverged from the anchor beyond that limit, the probe missed the corresponding branch. In the rerun, we scored the full anchor and used the same candidate pool, greedy decoding, and stopping rules for both methods. The evaluation covered two Reasoners, two training methods, and five datasets. The corrected complete tree and Independent Top-8 attain the same solution coverage in all 20 settings. In each cell below, both methods were scored separately and obtained the reported value.

DatasetLlama-3.1-8B SFTLlama-3.1-8B GRPOLlama-3.2-3B SFTLlama-3.2-3B GRPO
GSM8K78.09%88.02%70.65%83.54%
KodCode71.95%68.00%69.65%62.85%
PopQA43.17%47.72%42.40%39.59%
StrategyQA74.23%86.46%72.19%71.61%
TriviaQA71.23%75.77%65.45%69.97%

W1–W2 / Q1–Q2 · Attempt counts, coverage, and inference cost

Generation volume in Table 1. In Table 1, Vanilla, MemGen, TokMem, and BRANCHMEM Top-1 each generate one trajectory per query. The complete tree can expand up to seven branches in addition to the anchor.

Mean newly decoded tokens per queryGSM8KKodCode
Vanilla, single sample174.7401.8
MemGen-GRPO, single generation176.1339.0
TokMem retrieval extension, single generation145.9131.0
Complete Action Branching tree199.7401.0
Independent Top-81,429.62,761.6
Reduction relative to Independent Top-886.03%85.48%

On GSM8K and KodCode, Action Branching builds the complete tree with a newly decoded token budget comparable to a single generation from Vanilla or MemGen-GRPO. This supports the comparison with these baselines in Table 1. At a similar generation budget, Action Branching reuses the prefix before divergence and spends its new decoding on alternatives proposed by the candidate memories.

Coverage under the same attempt limit. We also compare solution coverage with eight attempts per query on the frozen Llama-3.2-3B Reasoner. Vanilla samples output tokens at T=0.2T=0.2 and top-p=1.0p=1.0. MemGen-GRPO applies Gaussian perturbations at scale 0.2 to the Weaver latents and decodes greedily. The TokMem retrieval extension generates from each of the eight highest-ranked memories. The complete tree returns up to eight trajectories. A query is covered if any trajectory is correct.

DatasetVanillaMemGen-GRPOTokMem retrieval extensionCorrected BRANCHMEM-SFT complete treeCorrected BRANCHMEM-GRPO complete tree
GSM8K81.72%79.45%74.83%70.65%83.54%
KodCode55.65%66.40%67.35%69.65%62.85%

With up to eight attempts per query, the corrected GRPO tree reaches 83.54% coverage on GSM8K, compared with 81.72% for eight Vanilla samples. On KodCode, the SFT tree reaches 69.65%, above all three baselines in the table. The GRPO tree reaches 62.85%, above Vanilla and below MemGen-GRPO and the TokMem retrieval extension.

Inference cost of the complete tree. We measured Reasoner time for the corrected Llama-3.2-3B GRPO tree and batched Independent Top-8 on the same pool. Including probing and prefix replay, the two methods take 5.14 and 4.83 s per query on GSM8K and 10.52 and 11.25 s per query on KodCode, respectively. Teacher-forced probing accounts for 12.51% and 13.09% of complete-tree Reasoner time on the two tasks.

Q3 · Are low-margin positions more sensitive to memory replacement when selected independently?

After selecting positions independently, the memory replacement effect at low-margin positions is still about three times that of matched random positions on GSM8K and KodCode. Table 2 selects positions based on the observed first action divergence. To test whether this criterion amplifies the effect, we fix each generated anchor, use the no-memory Reasoner to select the 10% of positions with the smallest gap between the top two logits, and sample random controls from the same relative-position bins to match progress through the solution.

After selecting both sets of positions, we score the same prefix with the anchor memory and a candidate memory. We measure the absolute change DD in the original anchor token's log probability and record whether the two memories rank different tokens first. We include every selected position, including those where the highest-scoring token is unchanged. We average each metric within a query and then across queries.

MetricGSM8KKodCode
Mean DD, position-matched random positions0.0006360.000571
Mean DD, preselected low-margin positions0.0019130.001801
Paired difference in DD by query, 95% bootstrap interval[0.001254, 0.001298][0.001213, 0.001248]
Action disagreement rate, position-matched random positions0.056%0.048%
Action disagreement rate, preselected low-margin positions0.245%0.219%

The paired intervals lie above zero on both tasks. The no-memory Reasoner's margin can identify positions that are more sensitive to memory replacement before candidate actions are observed.

Reviewer_6XKiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work is related to using memory banks for utilizing experiences in producing better outcomes from reasoning-enabled LLMs (referred as frozen reasoners). Specifically, authors are describing an approach called BranchMem, which aims at improving the generation quality (measured over 5 benchmarks) via their ActionBranch approach. The main contribution as I see lies in the action-branch approach will tends to create branches to augment the next token generation conditioned on the prior retrieved latent reasoning. The validations are conducted over 2 open source models of different sizes (llama 3B, 8B) over five public benchmarks (gsm8k, KodCode, triviaQA, popQA, strategyQA). Although reviewer is not familiar with the the KodCode benchmark, and could not access its baseline performance and improvements quality.

Critical strengths
  1. Reviewer appreciate the branching method (Sec-3.4) and the proposed engineering to make it work. The improvements in performance as presented in the Table-1 is also acknowledged. I also have serious points related to it, please check the weakness for more discussions.
  2. I agree with the distinction in line 145-147 that action-branch is more data-driven instead of a pre-defined structure drive
Critical weaknesses
  1. The paper writing is lacking the threshold for publication at ICLR and needs some major rewriting. Here are my specific comments: a: The captions in the figures tell little to nothing about them and make the figure uninterpretable (for example, Fig-1 which is supposed to be main fig of the paper, and also other figures) b: The flow of the paper is broken at multiple places and required significant effort from reader to connect dots. For example "line -96-97" the mention of other memory just created a confusion of why using "other" when top1 is there. Only after reaching later sections, we see that this other memory comes from a K-pool. c: I dont agree with distinction in line 126-127 regarding claim of the continuous representation. Its not established anywhere in paper. d: What is "output projection" in line 218 e: For statement in line 290-291, is this a good approach? ranking based on diff of top-2 tokens? technically we are saying that select those generations which the diff b.w first and next token is high. Are we then penalizing diversity in subsequent branches? f: Till last para of Sec-3, I am not clear how the returned tree T is used for better generation, its not mentioned clearly g: The unnecessary and over usage of bold statements everywhere in the paper make the reader loose track of the flow of the paper

  2. A big claim that different reasoning trajectories remain same at most of the times and only differ at few locations is mentioned several times in the paper, even dedicated para line 416-418, but I cannot see from any table/figure that this is shown properly

  3. In Table1, main results only show quality comparison, however, one major issue I have is its complexity as well. Given iterative branches are getting created, how many forward passes do we need from the reasoning LLM and also additional context tokens we need to give to generate the outcome. A vanilla memory technique, may query the memory once -> augment -> generate. But here we will have multiple forward passes. Without this data, the comparison using only final quality is moot.

Questions for authors

Pls see the weakness section

3 comments

W1(a) · How should Figures 1–4 be read?

Figure 1 compares three latent-memory interfaces. TokMem optimizes independently retrievable memories one at a time. MemGen uses a shared Weaver to generate guidance vectors from the current reasoning state. BRANCHMEM uses a shared compressor to encode each successful trajectory once and stores the result with its retrieval key. New queries can retrieve these memories individually, allowing us to compare the actions they propose under the same reasoning prefix.

Figure 2 shows how these memories are used to solve a new problem. We retrieve once, generate an initial solution with the top-ranked memory, and use this solution as the anchor. We then check where each remaining memory would propose a different action along the anchor. To do this, we feed the generated anchor back to the model unchanged and use teacher forcing to score the next token at each position under one candidate memory at a time. With causal masking, each position sees only its prefix, so one forward pass gives that candidate’s scores along the entire anchor.

Suppose the anchor starts with (a₁, a₂, a₃). If a candidate agrees with the first two tokens but prefers b₃ after (a₁, a₂), position 3 is its first divergence. For now, we record b₃ as an alternative action. The probe at position 4 still uses the original prefix (a₁, a₂, a₃). When we expand the branch, we keep (a₁, a₂), insert b₃, and continue generation under that candidate memory. Candidates proposing the same alternative token at the same position share this branch point. We probe the remaining candidates in that group again along the new trajectory.

Figure 3 builds on Figure 2 by asking what kinds of reasoning states are associated with divergences such as choosing b₃ in place of a₃. Candidates first diverge at different positions, so we align each first divergence at position 0 to compare it with the preceding shared prefix. The solid curves show how much replacing the anchor memory changes the log probability of the original anchor token. The dashed curves show how close the top two token scores are when we keep the prefix fixed and remove the memory. Near the first divergence, the larger change in log probability and the smaller gap indicate that memory replacement has a stronger effect where the model’s own preference for the next token is weaker.

Figure 4 asks how early the resulting branches find a correct solution when the anchor is wrong. The left panel considers queries that a branch subsequently solves. Order 1 means the first expanded branch is already correct, which accounts for 47%–89% of these queries across settings. The right panel reports the mean number of branches expanded before reaching the first correct solution, counting a correct anchor as 0. We compute both statistics from the traversal order after building the complete tree.

W1(b) · Why consider memories beyond Top-1?

In Table 6, initial retrieval scores provide little separation between lower-ranked memories that do and do not solve the current query. On TriviaQA and PopQA, we consider queries for which the Top-1 answer is incorrect and at least one memory ranked 2–8 solves the query independently. We compare the initial retrieval scores of successful and unsuccessful candidates.

ReasonerTrainingTriviaQA AUCPopQA AUC
Llama-3.1-8BSFT0.5140.552
Llama-3.1-8BGRPO0.4630.422
Llama-3.2-3BSFT0.5100.503
Llama-3.2-3BGRPO0.4830.536

The mean AUC across the eight settings is 0.498, close to the random-ranking value of 0.5. For these queries, retrieval already found a memory that can solve the problem, but the query-level score provides little guidance about which one it is. RQ2 offers an explanation. Different memories can propose the same actions along a shared prefix and begin to differ only when the model reaches a particular reasoning state. From the perspective of memory selection, Action Branching acts as a second retrieval stage conditioned on the reasoning state. The initial query retrieves KK candidates from the bank, and the Top-1 memory generates the anchor. Action Branching then probes the remaining K−1K-1 memories from that same pool under the generated prefix. Keeping these candidates available lets the system use alternative actions they propose at later decisions, where the initial ranking offers limited guidance.

W1(c–d) · How are memories represented, retrieved, and used?

A continuous representation consists of a fixed number of real-valued vectors per memory. We encode a successful trajectory into a memory value and store it with the retrieval key derived from the original query. For a new query, the retrieval projection PkvP_{\mathrm{kv}} maps its key into the space of stored memory values, allowing cosine similarity to rank the candidates. Once we select a memory, the output projection PoutP_{\mathrm{out}} maps its latent vectors from the Compressor's representation space to the frozen Reasoner's input embedding space. The projected vectors are placed after the query and before the response, where they condition next-token prediction.

BRANCHMEM follows MemGen in conditioning the Reasoner on real-valued vectors. MemGen's Weaver produces guidance from the current reasoning state. BRANCHMEM encodes each successful experience once and stores it as an entry that can be retrieved and reused across queries. The distinction in lines 126–127 concerns how the vectors are obtained and reused. The output projection PoutP_{\mathrm{out}} in line 218 serves a role similar to latent-memory insertion in MemGen, where vectors from the Weaver condition subsequent generation by the frozen Reasoner.

W1(e) · Does branch ordering reduce diversity?

No. The gap between the two highest logits that you mention is the RQ2 diagnostic for local decision confidence. We order branch groups by the position of their first divergence. Within a group, Equation (8) selects the memory that continues first using the logit gap between the candidate’s preferred alternative token and the token actually used in the current trajectory, both scored under that candidate memory. At position 3, for example, the comparison is between b₃ and a₃, even if a₃ ranks below second under the candidate memory. When probing and continuation use the same greedy decisions and stopping rules, complete traversal preserves all paths reachable from the fixed candidate pool.

In the Figure 2 example, if two memories first choose b₃ at position 3, Equation (8) selects one to continue first. We probe the other memory again on the new trajectory, where it can propose a different action later. Candidates that first diverge at another position or propose a different token remain in separate groups at the parent node.

W1(f) · How is the returned tree used?

Action Branching returns a set of candidate solutions. The root is the Top-1 anchor. Each child retains its parent's prefix before the divergence and generates a new suffix from the alternative action. Table 1 measures solution coverage. We evaluate all trajectories after constructing the tree and count a query as covered if at least one trajectory is correct.

W1(g) · Readability

Thank you for flagging this. In a future revision, we will reduce boldface across full sentences and paragraphs and use it to highlight key terms and results.

W2 · How much of the prefix do independently generated trajectories share?

In the GSM8K comparison with Llama-3.2-3B GRPO, 58.43% of anchor–candidate pairs match token for token. Among pairs that diverge, the shared prefix spans 37.53% of the anchor length on average. Lines 416–418 of the original manuscript summarize the local analysis in Figure 3 and Table 2. Figure 3 aligns each pair's first action divergence at position 0. The solid curves show the change in the original token's log probability when the memory is replaced, and the dashed curves show the gap between the top two tokens when the memory is removed.

Table 2 quantifies this pattern. Across eight settings on GSM8K and KodCode, the mean memory replacement effect at first divergence is 53.9–959.0 times its level at shared-prefix positions. The mean no-memory logit margin falls to 5.2%–12.5% of its shared-prefix level. At these divergences, memory replacement affects next-token predictions more strongly, and the no-memory model has a weaker preference for its top action. To quantify trajectory overlap, we compare the Llama-3.2-3B GRPO Top-1 anchor with each independently generated candidate, token by token, on the full GSM8K test set.

Trajectory pairsCount (share)Mean shared prefix as a share of anchor length
Identical5,395 (58.43%)100%
Divergent3,838 (41.57%)37.53%

The task-level ablation examines whether matching memories to queries matters. Retrieval for the current query improves solution coverage over cross-query shuffling by 3.13 and 4.56 percentage points on GSM8K and KodCode, respectively. Many candidates follow the same path, while a change at a local decision can lead a divergent trajectory to a different solution.

W3 · Main-table comparison and complete-tree cost

Generation volume in Table 1. Table 1 compares solution coverage under each method's own inference procedure. On GSM8K and KodCode, the complete tree reuses pre-divergence prefixes to explore alternative solutions with a similar number of newly decoded tokens to a single generation by Vanilla or MemGen-GRPO.

Mean newly decoded tokens per queryGSM8KKodCode
Vanilla, single sample174.7401.8
MemGen-GRPO, single generation176.1339.0
TokMem retrieval extension, single generation145.9131.0
Complete Action Branching tree199.7401.0
Independent Top-81,429.62,761.6
Reduction relative to Independent Top-886.03%85.48%

On GSM8K and KodCode, Action Branching builds the complete tree with a newly decoded token budget comparable to a single generation from Vanilla or MemGen-GRPO. This supports the comparison with these baselines in Table 1. At a similar generation budget, Action Branching reuses the prefix before divergence and spends its new decoding on alternatives proposed by the candidate memories.

Inference cost of the complete tree. After generating the anchor, we use teacher forcing to score the other seven candidate memories. In a batch, all candidates use the same anchor and their own memory vectors, allowing the Reasoner to score them in parallel. The total number of Reasoner forward passes is the per-query average from the start of anchor generation through the end of tree construction. It includes autoregressive token prediction, prefix replay before branch continuation, and candidate probing. Each batch counts as one forward pass. Processed token positions are summed across all sequences in each batch.

Mean complete-tree cost per queryGSM8KKodCode
Total Reasoner forward passes217.21467.53
Processed token positions2,393.85,448.7
Total Reasoner time (s)5.14010.518
Teacher-forced probing time (s)0.6431.377

Each memory-conditioned sequence adds eight latent input positions. The newly decoded token comparison above measures the generation saved through prefix reuse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.