- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
RRMs scale computation by recursively refining a latent state, but the recursion depth is typically fixed or predicted by an auxiliary head. This paper argues that leveraging the Gaussian transition introduced in prior work makes the depth a learnable decision. Variational pretraining provides a starting point on the already-learned latent manifold distribution, and RL allows both the latent transitions and the halting action to be updated. As a result, the paper reports improvements of 1% on Sudoku and 4–5% on ARC.
Critical strengths
- Interesting results and analysis. The paper achieves the best performance on Sudoku and ARC-AGI-1/2 through RL post-training, and its analysis of how the reasoning trajectories change after post-training is informative.
- Well-designed ablations within a scoped method. Although the method itself is relatively scoped, the experiments within that scope are thorough. In particular, the depth extrapolation experiment (training with a depth cap of 16 and evaluating up to 1024) clearly shows the benefit of residual-conditioned halting over step-index halting and fixed depth. Each component is also ablated from the same Phase-1 checkpoint with an explicit stability criterion. An analogous experiment along the parallel (width) axis would have made the evaluation more complete.
Critical weaknesses
- Unclear problem statement. The abstract and introduction mainly argue that existing methods cannot learn adaptive depth and that RL solves this problem. However, TRM, HRM, and GRAM are all already trained with adaptive depth, and their depth can also be adjusted adaptively at inference time. It is unclear what benefit RL post-training brings on top of this, and the experiments do not clarify it either. In other words, it is unclear what problem the paper is trying to solve.
- Limited methodological novelty and contribution. Apart from the motivation, the paper does not provide reasons or experiments for why RL post-training must be run on top of an already pretrained GRAM. Does the latent transition need to come from a variationally trained model, or is a stochastic transition as a policy all that is needed? If the latter, it would have been better to run RL from the pretraining stage without a separate post-training phase, or to remove GRAM-specific details and train with only the minimal necessary components. Moreover, the RL part does not propose any particular novelty or new methodology; it merely applies existing RL algorithms. Although the paper shows improved performance on Sudoku, ARC, etc., I think this alone constitutes a low contribution to the ML community.
Questions for authors
- Beyond post-training on top of a pretrained GRAM, did you try training from scratch with RL?
- Have you tried applying RL post-training to methods other than GRAM?
- Adaptive depth is already possible in TRM, HRM, and GRAM. What is the advantage of achieving adaptive depth through RL?
- The paper does not clearly state how many samples were used for the reported results. Could you also provide a performance comparison across different numbers of samples?
8 comments
Response to Reviewer AwZb (2/2)
W2/Q1: Why RL must start from a pretrained generative prior — tested, not asserted.
You asked whether a stochastic transition as a policy is all that is needed, in which case one could run RL from scratch or strip the GRAM-specific parts. We ran the first version of that experiment. RL from scratch (same Phase 2, same reward, 30,000 updates, group width 16, no Phase-1 initialization) produces almost no learning on Sudoku, because the verifiable reward is effectively never positive at initialization and the group-relative advantage of a group in which every rollout fails is zero; on ARC-AGI-1 it solves only the trivially easy tasks. We then interpolated: start Phase 2 from Phase-1 checkpoints at 0%, 25%, 50%, and 100% of pretraining.
| Phase-1 pretraining before RL | Sudoku | ARC-1 |
|---|---|---|
| 0% (RL from scratch) | 1.9 | 5.8 |
| 25% | 93.8 | 45.2 |
| 50% | 96.9 | 52.3 |
| 100% | 98.4 | 56.8 |
The curve is monotone and steep at the low end. The reason is not that GRAM's architecture is special; it is that outcome-only RL needs a policy whose rollouts occasionally succeed, and variational pretraining is what supplies that support. This is also the reason the two phases are not merged: the ELBO supplies a supervision signal that does not require the reward to be non-zero, which is exactly what a cold start lacks. We state this as the design rationale in the revision rather than leaving it implicit, and we report the from-scratch numbers as a negative result that motivates the recipe.
The complementary evidence is already in Table 1: the TRM + grafted stochastic head + RL row adds a Gaussian head to a deterministic core and runs the same Phase 2, reaching 95.1/49.0/10.2 against 98.4/56.8/16.8. A stochastic head alone is not enough; the transition distribution has to be trained as one. That row is the closest available answer to "remove GRAM-specific details and train with the minimal components" — the minimal component that matters is the variationally pretrained transition, and removing it costs 7.8 ARC-1 points.
Q2: RL post-training on methods other than GRAM. The TRM + grafted head + RL row above is exactly this experiment for a deterministic recursive model, and SLPO (Table 1) applies outcome-RL to a latent reasoner with an empirical surrogate density. We have not applied Phase 2 to HRM or to non-recursive models; we say so in the limitations. The scope claim is precise: ReGRAM applies where the recursion exposes a normalized per-transition density, which in this family is GRAM. That is a real scope restriction, not a claim of generality.
W2: On methodological novelty.
We do not claim a new RL algorithm, and the revision does not present one. The contribution we do claim is the interface: a generative recursive model's transition distribution is already a normalized, reparameterizable policy with an exact log-density, so the standard machinery — clipped ratios, group baselines, KL anchors — becomes applicable without an empirical surrogate, and the pretraining distribution doubles as the trust region. The residual-conditioned halt action, the alive-at-step group baseline for variable horizons, and the reuse of the LPRM as a shaping potential are the pieces that make the interface work; each is ablated from the same checkpoint (Table 2). Whether this constitutes a sufficient contribution is a judgment we cannot settle by argument, but we have tried to make the claim precise enough that it can be judged on what it is.
Q4: Sample counts and the width axis.
You are right that the paper did not state the sampling budget per reported number; the revision adds it for every row (width 16, LPRM selector, deterministic halting, three seeds, except where noted). Following your suggestion, we also ran the width sweep on ARC-AGI-1:
| ARC-AGI-1, LPRM selector | |||||
|---|---|---|---|---|---|
| GRAM (Phase 1) | 42.9 | 47.8 | 51.4 | 52.8 | 53.5 |
| ReGRAM | 46.1 | 52.7 | 56.8 | 58.1 | 58.7 |
The gain does not close with more samples — it is 3.2 points at and 5.2 at — so post-training changes the distribution rather than merely adding sampling noise on top of it, and ReGRAM at width 4 matches GRAM at width 16 with a quarter of the rollouts. This is the parallel-axis analogue of the depth result, and we thank the reviewer for suggesting it; it is now in Sec. 6.
I have carefully read the general response and the individual responses to each reviewer. The authors’ willingness to accept the criticism and improve the paper is much appreciated.
First, I appreciate the comparison against the adaptive depth of existing models. However, the weakness has not been resolved. To reiterate, the paper still does not clearly explain what problem it aims to solve, nor sufficiently why ReGRAM is better, beyond “RL post-training made it work better.” The numbers already show that ReGRAM is a better model. But why is it better? Why should one use this approach? This explanation is still missing. Without it, it is difficult to provide evidence on whether the method works only in this particular setup or generalizes to other setups. I believe a research contribution should explain the underlying idea, not only demonstrate a better-performing model.
Regarding the second weakness, to be precise, my question is why it has to be GRAM. Showing through experiments that RL post-training should be performed on top of a pretrained reasoning model is demonstrating an expected result. It is the same inefficiency as training an LLM with RL alone without pretraining, especially since neither the correct trajectories nor the latent space they live in have been learned. To address this weakness, the authors need to show that the approach also works with other, more general generative priors beyond GRAM. The work should not be limited to a narrow extension specific to GRAM.
The width-sweep result is impressive. As I expected, post-training seems to have sharpened the prior so that it samples more toward correct answers. It would have been more interesting if the paper had directly addressed the distributional change induced by post-training, beyond a simple application of RL, or proposed improvements grounded in related theory.
Response to Reviewer AwZb (1/n)
We thank the reviewer for the detailed engagement, and in particular for naming precisely where the paper fails to persuade: the problem statement does not separate "predicting when to stop" from "optimizing what stopping achieves," and the method is not justified as requiring a pretrained generative prior. We take both criticisms seriously. We have not changed the paper into a claim of a new RL algorithm, because that would not be true — the RL machinery is standard. We have instead made the actual claim testable and, we believe, supported it with three new pieces of evidence: a decomposition showing the halting objective is not equivalent to the Phase-1 Q-head, a staged-pretraining study showing why RL from a random initialization does not substitute for the two-phase design, and the width-axis experiment you suggested.
New experiments: transition-only / halt-only / joint RL decomposition (Sec. R1); RL from 0/25/50/100% of Phase 1 pretraining (Sec. R2); width sweep with accuracy-matched compute accounting (Sec. R3). All use the same Phase-1 checkpoints and protocol as the paper.
W1/Q3: The problem is not "adaptive depth" in general — it is optimizing depth against the outcome, which the Q-head does not do.
We agree the submitted framing overstated the gap. TRM, HRM, and GRAM all have adaptive depth, and we have rewritten the introduction to say so plainly. The distinction we are actually claiming is narrower, and it is a distinction of objective rather than of capability:
This is a testable difference, and the decisive test is the halt-only ablation: keep the recursive weights θ completely frozen and update only the halt head ψ with the verifier advantage, starting from the same Phase-1 checkpoint.
The halt head has exactly the same inputs, the same architecture, and the same information as in Phase 1; the only thing that changes is the objective. That alone is worth 2.6 ARC-AGI-1 points. If the Q-head were already an optimal depth rule, this number would be zero. The Phase-1 head is a good predictor and a suboptimal controller, and the gap is what the paper is about.
We accept that the submitted abstract did not make this distinction; it argued against "fixed or auxiliary-head depth" in a way that read as denying that TRM/HRM/GRAM have adaptive depth. The revised abstract states the predict-versus-optimize distinction in the first two sentences.