acceptodds
Under review as a conference paper at ICLR 2027

Reinforced Generative Recursive Reasoning for Native Adaptive Computation Depth

Abstract

Recursive Reasoning Models scale computation by iteratively refining a latent state with shared transition functions, yet the amount of recursion is usually fixed externally or predicted by an auxiliary head. We show how a generative recursive prior turns this limitation into a learnable decision: its Gaussian transition is a normalized, reparameterizable policy with an exact log-density. Variational pretraining therefore provides an on-manifold starting distribution, while outcome reinforcement learning can update both latent transitions and a residual-conditioned halting action. A potential-based value signal supplies credit across the loop and a KL anchor keeps exploration near the pretrained manifold. On structured reasoning benchmarks, ReGRAM improves ARC-AGI-1/2 accuracy by 4.8/5.7 percentage points over GRAM, allocates more depth to harder inputs, and transfers from a training depth cap of 16 to inference budgets of 1024. The approach motivates a simple principle: adaptive depth is most natural when the computation itself is represented as a stochastic policy.

Primary area:
reinforcement learning
Keywords:
Looped Transformer, Recursive Reasoning Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AwZbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RRMs scale computation by recursively refining a latent state, but the recursion depth is typically fixed or predicted by an auxiliary head. This paper argues that leveraging the Gaussian transition introduced in prior work makes the depth a learnable decision. Variational pretraining provides a starting point on the already-learned latent manifold distribution, and RL allows both the latent transitions and the halting action to be updated. As a result, the paper reports improvements of 1% on Sudoku and 4–5% on ARC.

Critical strengths
  • Interesting results and analysis. The paper achieves the best performance on Sudoku and ARC-AGI-1/2 through RL post-training, and its analysis of how the reasoning trajectories change after post-training is informative.
  • Well-designed ablations within a scoped method. Although the method itself is relatively scoped, the experiments within that scope are thorough. In particular, the depth extrapolation experiment (training with a depth cap of 16 and evaluating up to 1024) clearly shows the benefit of residual-conditioned halting over step-index halting and fixed depth. Each component is also ablated from the same Phase-1 checkpoint with an explicit stability criterion. An analogous experiment along the parallel (width) axis would have made the evaluation more complete.
Critical weaknesses
  • Unclear problem statement. The abstract and introduction mainly argue that existing methods cannot learn adaptive depth and that RL solves this problem. However, TRM, HRM, and GRAM are all already trained with adaptive depth, and their depth can also be adjusted adaptively at inference time. It is unclear what benefit RL post-training brings on top of this, and the experiments do not clarify it either. In other words, it is unclear what problem the paper is trying to solve.
  • Limited methodological novelty and contribution. Apart from the motivation, the paper does not provide reasons or experiments for why RL post-training must be run on top of an already pretrained GRAM. Does the latent transition need to come from a variationally trained model, or is a stochastic transition as a policy all that is needed? If the latter, it would have been better to run RL from the pretraining stage without a separate post-training phase, or to remove GRAM-specific details and train with only the minimal necessary components. Moreover, the RL part does not propose any particular novelty or new methodology; it merely applies existing RL algorithms. Although the paper shows improved performance on Sudoku, ARC, etc., I think this alone constitutes a low contribution to the ML community.
Questions for authors
  1. Beyond post-training on top of a pretrained GRAM, did you try training from scratch with RL?
  2. Have you tried applying RL post-training to methods other than GRAM?
  3. Adaptive depth is already possible in TRM, HRM, and GRAM. What is the advantage of achieving adaptive depth through RL?
  4. The paper does not clearly state how many samples were used for the reported results. Could you also provide a performance comparison across different numbers of samples?
8 comments
AuthorsOpenReview ↗

Response to Reviewer AwZb (1/n)

We thank the reviewer for the detailed engagement, and in particular for naming precisely where the paper fails to persuade: the problem statement does not separate "predicting when to stop" from "optimizing what stopping achieves," and the method is not justified as requiring a pretrained generative prior. We take both criticisms seriously. We have not changed the paper into a claim of a new RL algorithm, because that would not be true — the RL machinery is standard. We have instead made the actual claim testable and, we believe, supported it with three new pieces of evidence: a decomposition showing the halting objective is not equivalent to the Phase-1 Q-head, a staged-pretraining study showing why RL from a random initialization does not substitute for the two-phase design, and the width-axis experiment you suggested.

New experiments: transition-only / halt-only / joint RL decomposition (Sec. R1); RL from 0/25/50/100% of Phase 1 pretraining (Sec. R2); width sweep with accuracy-matched compute accounting (Sec. R3). All use the same Phase-1 checkpoints and protocol as the paper.


W1/Q3: The problem is not "adaptive depth" in general — it is optimizing depth against the outcome, which the Q-head does not do.

We agree the submitted framing overstated the gap. TRM, HRM, and GRAM all have adaptive depth, and we have rewritten the introduction to say so plainly. The distinction we are actually claiming is narrower, and it is a distinction of objective rather than of capability:

  • A Q-head or ACT head is trained to predict whether continuing will help, using targets derived from a fixed policy's own rollouts (q^nhalt=1[y^(n)=y]\hat q^{halt}_n=\mathbf 1[\hat y^{(n)}=y], q^ncont=max⁡(qn+1halt,qn+1cont)\hat q^{cont}_n=\max(q^{halt}_{n+1},q^{cont}_{n+1})). It is a value estimate under the behavior policy that generated the supervised trajectories.
  • ReGRAM optimizes the halting decision together with the transition distribution against the verifiable outcome, under a reward that has no length term. The gradient is on the realized outcome of the sampled action, not on a regression to a stale target.

This is a testable difference, and the decisive test is the halt-only ablation: keep the recursive weights θ\theta completely frozen and update only the halt head ψ\psi with the verifier advantage, starting from the same Phase-1 checkpoint.

RL channel (same checkpoint)SudokuARC-1Mean HHρs\rho_s
none (Phase-1 Q-head)96.751.46.10.21
halt only (θ\theta frozen)97.554.08.30.34
transition only (ψ\psi frozen)98.155.37.40.26
joint (ReGRAM)98.456.89.60.43

The halt head has exactly the same inputs, the same architecture, and the same information as in Phase 1; the only thing that changes is the objective. That alone is worth 2.6 ARC-AGI-1 points. If the Q-head were already an optimal depth rule, this number would be zero. The Phase-1 head is a good predictor and a suboptimal controller, and the gap is what the paper is about.

We accept that the submitted abstract did not make this distinction; it argued against "fixed or auxiliary-head depth" in a way that read as denying that TRM/HRM/GRAM have adaptive depth. The revised abstract states the predict-versus-optimize distinction in the first two sentences.

AuthorsOpenReview ↗

Response to Reviewer AwZb (2/2)

W2/Q1: Why RL must start from a pretrained generative prior — tested, not asserted.

You asked whether a stochastic transition as a policy is all that is needed, in which case one could run RL from scratch or strip the GRAM-specific parts. We ran the first version of that experiment. RL from scratch (same Phase 2, same reward, 30,000 updates, group width 16, no Phase-1 initialization) produces almost no learning on Sudoku, because the verifiable reward is effectively never positive at initialization and the group-relative advantage of a group in which every rollout fails is zero; on ARC-AGI-1 it solves only the trivially easy tasks. We then interpolated: start Phase 2 from Phase-1 checkpoints at 0%, 25%, 50%, and 100% of pretraining.

Phase-1 pretraining before RLSudokuARC-1
0% (RL from scratch)1.95.8
25%93.845.2
50%96.952.3
100%98.456.8

The curve is monotone and steep at the low end. The reason is not that GRAM's architecture is special; it is that outcome-only RL needs a policy whose rollouts occasionally succeed, and variational pretraining is what supplies that support. This is also the reason the two phases are not merged: the ELBO supplies a supervision signal that does not require the reward to be non-zero, which is exactly what a cold start lacks. We state this as the design rationale in the revision rather than leaving it implicit, and we report the from-scratch numbers as a negative result that motivates the recipe.

The complementary evidence is already in Table 1: the TRM + grafted stochastic head + RL row adds a Gaussian head to a deterministic core and runs the same Phase 2, reaching 95.1/49.0/10.2 against 98.4/56.8/16.8. A stochastic head alone is not enough; the transition distribution has to be trained as one. That row is the closest available answer to "remove GRAM-specific details and train with the minimal components" — the minimal component that matters is the variationally pretrained transition, and removing it costs 7.8 ARC-1 points.

Q2: RL post-training on methods other than GRAM. The TRM + grafted head + RL row above is exactly this experiment for a deterministic recursive model, and SLPO (Table 1) applies outcome-RL to a latent reasoner with an empirical surrogate density. We have not applied Phase 2 to HRM or to non-recursive models; we say so in the limitations. The scope claim is precise: ReGRAM applies where the recursion exposes a normalized per-transition density, which in this family is GRAM. That is a real scope restriction, not a claim of generality.

W2: On methodological novelty.

We do not claim a new RL algorithm, and the revision does not present one. The contribution we do claim is the interface: a generative recursive model's transition distribution is already a normalized, reparameterizable policy with an exact log-density, so the standard machinery — clipped ratios, group baselines, KL anchors — becomes applicable without an empirical surrogate, and the pretraining distribution doubles as the trust region. The residual-conditioned halt action, the alive-at-step group baseline for variable horizons, and the reuse of the LPRM as a shaping potential are the pieces that make the interface work; each is ablated from the same checkpoint (Table 2). Whether this constitutes a sufficient contribution is a judgment we cannot settle by argument, but we have tried to make the claim precise enough that it can be judged on what it is.

Q4: Sample counts and the width axis.

You are right that the paper did not state the sampling budget per reported number; the revision adds it for every row (width 16, LPRM selector, deterministic halting, three seeds, except where noted). Following your suggestion, we also ran the width sweep on ARC-AGI-1:

ARC-AGI-1, LPRM selectorN=1N{=}1N=4N{=}4N=16N{=}16N=32N{=}32N=64N{=}64
GRAM (Phase 1)42.947.851.452.853.5
ReGRAM46.152.756.858.158.7

The gain does not close with more samples — it is 3.2 points at N=1N{=}1 and 5.2 at N=64N{=}64 — so post-training changes the distribution rather than merely adding sampling noise on top of it, and ReGRAM at width 4 matches GRAM at width 16 with a quarter of the rollouts. This is the parallel-axis analogue of the depth result, and we thank the reviewer for suggesting it; it is now in Sec. 6.

I have carefully read the general response and the individual responses to each reviewer. The authors’ willingness to accept the criticism and improve the paper is much appreciated.

First, I appreciate the comparison against the adaptive depth of existing models. However, the weakness has not been resolved. To reiterate, the paper still does not clearly explain what problem it aims to solve, nor sufficiently why ReGRAM is better, beyond “RL post-training made it work better.” The numbers already show that ReGRAM is a better model. But why is it better? Why should one use this approach? This explanation is still missing. Without it, it is difficult to provide evidence on whether the method works only in this particular setup or generalizes to other setups. I believe a research contribution should explain the underlying idea, not only demonstrate a better-performing model.

Regarding the second weakness, to be precise, my question is why it has to be GRAM. Showing through experiments that RL post-training should be performed on top of a pretrained reasoning model is demonstrating an expected result. It is the same inefficiency as training an LLM with RL alone without pretraining, especially since neither the correct trajectories nor the latent space they live in have been learned. To address this weakness, the authors need to show that the approach also works with other, more general generative priors beyond GRAM. The work should not be limited to a narrow extension specific to GRAM.

The width-sweep result is impressive. As I expected, post-training seems to have sharpened the prior so that it samples more toward correct answers. It would have been more interesting if the paper had directly addressed the distributional change induced by post-training, beyond a simple application of RL, or proposed improvements grounded in related theory.

Reviewer_NJrBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ReGRAM which is an extension of GRAM where the authors applied reinforcement learning to stochastic latent transitions and adaptive halting in recursive reasoning models.

Critical strengths

the question of learning both how to update latent reasoning states and when to stop reasoning is very timely and interesting + the I find the approach interesting The reported improvements over GRAM on ARC-AGI-1 (+4.8 percentage points) and ARC-AGI-2 (+5.7 percentage points) are promising

Critical weaknesses

I found the paper difficult to follow, especially the distinction between what is inherited from GRAM and what is actually new. The Gaussian policy formulation in Section 3 is unclear especially Each supervision step involves multiple stochastic transitions, so it is not obvious why the final state should follow a Gaussian distribution

The paper claims that adaptive depth emerges from RL, but the halting head is already pretrained with Q-target supervision. It is therefore difficult to tell how much is actually learned during RL. The experiments also do not clearly separate the benefits of optimizing latent transitions from those of learning when to halt.

Questions for authors

In Section 3, what exactly is considered an action in the policy? How is the Gaussian likelihood justified when a supervision step contains multiple transitions? How much adaptive stopping behavior is already present before RL? Could the authors provide separate ablations for RL on latent transitions only, halting only, and both jointly? Could the authors clarify how the claimed computation savings compare with fixed-depth baselines at matched accuracy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NJrB (1/n)

We thank the reviewer for the encouraging assessment of the results and for naming the two confusions that cost the most readability. Both are our fault, not the reader's: the paper never says in one place what is inherited from GRAM and what is new, and Sec. 3 compressed a per-transition density into a per-supervision-step definition. The revision adds an explicit inherited-versus-new component table, restates the policy at the transition level, and adds the three-way decomposition of RL effects (transition-only, halt-only, joint) that you asked for, together with a pre-RL measurement of how much adaptive stopping Phase 1 already provides.

W1: What is inherited from GRAM and what is new.

ComponentGRAM (Phase 1)ReGRAM
Encoder, recursive core fL,fHf_{\mathrm L},f_{\mathrm H}, decoderinherited unchangedinherited unchanged
Gaussian guidance μθ,σθ\mu_\theta,\sigma_\thetainherited; trained by the ELBOinherited; now optimized by policy gradient
Variational posterior qϕq_\phiinheriteddiscarded after Phase 1 (needs yy)
LPRM value head vψv_\psiinheritedreused as shaping potential and critic
Halt head inputhTh_T[hT;ρn][h_T;\rho_n], residual-conditioned (the only architectural change)
Halt head objectiveQ-target regression (TD)Bernoulli policy with verifier advantage
Inferencefixed depth, or Q-head haltinglearned halting, validated beyond the training horizon

The method is deliberately a wrapper: Phase 1 changes one head input, Phase 2 adds a rollout-and-PPO loop around the existing prior sampler. What is new is not an architecture but the observation that this transition distribution is a usable policy, plus the halting action, the residual conditioning, and the credit-assignment scheme that make it trainable.

W2/Q1: The Gaussian is per transition, not per supervision step.

You are right that a supervision step contains several stochastic transitions and its terminal state is not Gaussian. Our original Definition 3.1 said otherwise, and that was an error in the definition — the implementation always evaluated densities per high-level transition (App. A.2) and always computed PPO ratios per transition. The revised definition states this directly: the state is st−1=(ht−1,lt−1,ex)s_{t-1}=(h_{t-1},l_{t-1},e_x), the action is the next high-level state at=hta_t=h_t, and

πθ(at∣st−1,x)=N(at; mθ(st−1,x), Σθ(st−1,x)),mθ=uθ+μθ(uθ),\pi_\theta(a_t\mid s_{t-1},x)=\mathcal N\left(a_t;\,m_\theta(s_{t-1},x),\,\Sigma_\theta(s_{t-1},x)\right),\quad m_\theta=u_\theta+\mu_\theta(u_\theta),

which is normalized by the unit-Jacobian change of variables ϵt↦ht=ut+ϵt\epsilon_t\mapsto h_t=u_t+\epsilon_t. A supervision step is then a macro-action whose density is the product of TT such Gaussians. The deterministic low-level carry contributes no density but its parameters still appear in mθm_\theta, because the ratio recomputes the proposal from the recorded state.

Q2/W3: How much adaptive stopping exists before RL.

We measured the Phase-1 halt head directly on the same instances and difficulty quartiles:

ARC-AGI-1Q1Q2Q3Q4ρs\rho_sMean HH
Phase-1 halt head (pre-RL)4.95.76.57.30.216.1
ReGRAM (post-RL)5.87.610.012.40.439.6

Phase 1 carries a weak trend, as you suspected; RL doubles both the correlation and the spread. So the honest statement is that RL amplifies and reshapes an existing weak propensity rather than creating it from nothing, and the revision says exactly this instead of attributing the emergence curve entirely to RL.

AuthorsOpenReview ↗

Response to Reviewer NJrB (2/2)

Q3/W4: Separate ablations for transition-only, halt-only, and joint RL.

All three start from the same Phase-1 checkpoint with identical Phase-2 settings (width 16, no augmentation, three seeds); in the single-channel runs the other channel is frozen, and both use the same reward and advantage estimator.

RL channelSudokuARC-1Mean HHρs\rho_s(ARC-1)
none (Phase-1 checkpoint)96.751.46.10.21
halt only (θ\theta frozen)97.554.08.30.34
transition only (ψ\psi frozen at Phase-1 halting)98.155.37.40.26
joint (ReGRAM)98.456.89.60.43

Three things follow. First, both channels help on their own, and neither is redundant: halt-only gains 2.6 ARC-1 points with the transition distribution completely frozen, which means the Phase-1 Q-target head is a suboptimal stopping rule that outcome reward improves even without touching the reasoning weights. Second, transition-only gains 3.9 points but under-uses depth (mean HH 7.4), because a frozen halt head still stops as it was supervised to. Third, the joint gain (5.4) is larger than either alone but smaller than their sum (6.5), i.e. the two channels are complementary but sub-additive — the halt channel mostly contributes depth allocation, the transition channel mostly contributes accuracy. This is the cleanest evidence we have for why both should be updated, and the revision reports the table.

Q4: Computation savings at matched accuracy.

The 28.3$\times$ number compared mean depth to the maximum budget and should not be read as an efficiency claim; we have demoted it. The accuracy-matched comparison is:

ARC-AGI-1Mean depth HHAccuracy
fixed depth 646456.6
fixed depth 12812857.6
fixed depth 1024102445.2
step-index halting, budget 1024—49.8
residual halting, budget 102436.258.9

Against the best fixed budget (128), residual halting uses 3.5$\times$ fewer supervision steps at matched accuracy and is 1.3 points more accurate. Fixed depth degrades past 128; residual halting does not.

We have also added a width-axis experiment, which is the analogous check along the parallel direction: at every width, ReGRAM is above GRAM on ARC-AGI-1

ARC-AGI-1, LPRM selectorN=1N{=}1N=4N{=}4N=16N{=}16N=32N{=}32N=64N{=}64
GRAM (Phase 1)42.947.851.452.853.5
ReGRAM46.152.756.858.158.7

The gain is 3.2 points at N=1N{=}1 and grows to 5.2 at N=64N{=}64; post-training and width sampling are complementary rather than redundant, and ReGRAM at width 4 (52.7) already matches GRAM at width 16 (51.4) with a quarter of the rollouts.

Reviewer_G2iuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ReGRAM, an RL extension of GRAM. It treats stochastic latent updates as policy actions and adds a residual-conditioned stopping policy. Post-training combines PPO-style updates, a learned value potential, a KL penalty to the pretrained prior, and invalid-sample masking. The paper reports gains of 1.4 points on Sudoku-Extreme and 4.8/5.7 points on ARC-AGI-1/2. It also reports difficulty-dependent stopping, inference beyond the training depth cap, and a small loss of solution coverage. The intended contribution is joint optimization of recursive computation and stopping using outcome rewards.

Critical strengths
  1. The problem is well motivated. Learning both recursive updates and stopping could improve the use of inference computation. Starting from a pretrained stochastic model offers a practical way to investigate this question.

  2. The evaluation considers useful dimensions beyond final accuracy. It includes component controls, stopping depth, and solution coverage. Reporting the loss of diversity alongside improved correctness is helpful.

Critical weaknesses
  1. The policy formulation does not match the implemented recursion. Definition 3.1 assigns a single Gaussian density to an entire supervision step, although it contains three nonlinear stochastic transitions. Their endpoint marginal is generally not Gaussian. Appendix A.2 instead records densities for individual high-level updates, leaving the action definition and deterministic low-level carry unresolved. The shaping argument also has an indexing problem: Equation (10) gives G_n = R - Phi(s_n) when gamma=1, but s_n is produced by the current action a_n. Detaching Phi(s_n) does not make it an action-independent baseline for that transition. Finally, the additional pathwise gradient described in Section 4.3 is not specified in the complete objective. These gaps undermine the central claims of exact likelihoods and principled credit assignment.

  2. The headline gains and depth claims need clearer experimental support. Table 1 uses reported GRAM results, including Sudoku accuracy obtained with 20 trajectories, while the stated matched setting uses 16 (Please see GRAM Section 4.1: https://arxiv.org/html/2605.19376v2). ARC augmentation settings are also unclear: Figure 5 starts at the reported 52.0% baseline despite describing this analysis as using no augmentation. At depth 16, Figure 2b gives approximately 56.5% for step-index halting, whereas Table 2 reports 55.9%; any configuration difference needs explanation. Moreover, Phase 1 already supervises the halt head, so the later difficulty-depth correlation does not isolate what RL adds. Finally, the 28.3x step-count ratio against fixed depth 1024 does not establish an accuracy-matched efficiency gain. Matched evaluations and controls are needed to support these interpretations.

Questions for authors
  1. What are the exact state and action at which each PPO ratio is evaluated? Please reconcile the supervision-step action in Definition 3.1 with the three internal Gaussian actions in Appendix A.2. Which densities enter the loss, and how is parameter dependence through the deterministic low-level carry handled?
  2. Why is G_n = R - Phi(s_n) appropriate for the transition that produces s_n? Please give the complete hybrid loss, its coefficients, the critic target, and all stop-gradient locations. Does the implementation include the missing transition shaping reward or an explicit correction? Does the shaping ablation retain its gain with a consistent action indexing?
  3. Please map each main result to its checkpoint, split, augmentation count, trajectory count, selector, metric, and seed aggregation. Can the headline gains be reproduced by evaluating the same Phase-1 checkpoint before and after RL under one frozen protocol? Please also reconcile Figure 2b with Table 2 and specify the SLPO adaptation.
  4. How much difficulty-dependent stopping is already present after Phase 1? Please compare transition-only and halt-only RL, and report accuracy against executed compute for fixed depth, ACT, and residual halting. What fraction of trajectories actually continues beyond 16 steps? Is guidance clipping used in the experiments, and if so, how is its effect on the likelihood handled?
4 comments
AuthorsOpenReview ↗

Response to Reviewer G2iu (1/n)

We thank the reviewer for a careful and technically precise reading. W1 identifies two genuine errors in how the formulation was written — the granularity of the stochastic action in Definition 3.1, and the index at which the shaping term is credited in Eq. (10) — and W2 identifies genuine gaps in comparability. We have corrected both rather than reinterpreting them. Definition 3.1 is now stated per high-level transition, Eq. (10) is re-indexed so the potential is credited to the transition entering the scored state, the pathwise term appears explicitly in the complete objective, and every headline number is re-reported under one frozen protocol with the same Phase-1 checkpoint evaluated before and after RL.

The new evidence is: a frozen-protocol re-evaluation at matched trajectory count with no augmentation; a re-run of the shaping ablation under the corrected index; a pre-RL halting measurement that separates what Phase 1 already provides from what RL adds; an accuracy-versus-executed-depth comparison; a λpw\lambda_{\text{pw}} ablation; and a transition-only / halt-only decomposition. None of the corrections changes a headline conclusion, but they change what we are entitled to claim, and the revision states the narrower claim.


W1(a): The action definition conflated a transition with a supervision step — corrected.

The reviewer is right that a single Gaussian cannot describe a supervision step, which contains T=3T=3 nonlinear stochastic high-level transitions; the endpoint marginal of that composition is not Gaussian. The defect was in the definition, not in the implementation: the PPO ratios were always evaluated per transition and App. A.2 already records per-transition densities. The definition now says so.

Let t=1,…,Tt=1,\dots,T index high-level transitions within a supervision step. The state is st−1=(ht−1,lt−1,ex)s_{t-1}=(h_{t-1},l_{t-1},e_x), the action is the next high-level state at=hta_t=h_t, and

πθ(at∣st−1,x)=N ⁣(at; mθ(st−1,x), Σθ(st−1,x)),mθ=uθ+μθ(uθ),\pi_\theta(a_t\mid s_{t-1},x)=\mathcal N\!\left(a_t;\,m_\theta(s_{t-1},x),\,\Sigma_\theta(s_{t-1},x)\right),\qquad m_\theta=u_\theta+\mu_\theta(u_\theta),

with uθ=fH(ht−1,lt)u_\theta=f_{\mathrm H}(h_{t-1},l_t) and Σθ=diag(σθ2)\Sigma_\theta=\mathrm{diag}(\sigma_\theta^2). This density is normalized because ϵt↦ht=ut+ϵt\epsilon_t\mapsto h_t=u_t+\epsilon_t has unit Jacobian and the mean ut+μθ(ut)u_t+\mu_\theta(u_t) is carried inside it. A supervision step is a macro-action of TT such transitions with density ∏t=1Tπθ(at∣st−1,x)\prod_{t=1}^{T}\pi_\theta(a_t\mid s_{t-1},x) — a product of TT Gaussians, not one Gaussian. The ratio is evaluated per transition, and the advantage AnA_n of a supervision step is broadcast to the TT transitions it contains because they share one terminal outcome.

W1(b): Parameter dependence through the deterministic low-level carry.

The low-level refinement lt=fL(K)(lt−1,ht−1+ex)l_t=f_{\mathrm L}^{(K)}(l_{t-1},h_{t-1}+e_x) is deterministic given the conditioning state and therefore contributes no likelihood factor. Its parameter dependence is not dropped: when the ratio is recomputed, the carry is re-executed from the recorded state, so ∂mθ/∂θ\partial m_\theta/\partial\theta includes the recursive-core parameters of fLf_{\mathrm L} as well as fHf_{\mathrm H} and the guidance heads. This is why we parameterize by the next state rather than by ϵ\epsilon; holding ϵ\epsilon fixed would omit the change in the proposal mean. The carry is detached only between supervision steps, following Phase 1's truncation schedule, never within a transition. The revision states this in Sec. 4.3 and App. A.1.

AuthorsOpenReview ↗

Response to Reviewer G2iu (2/n)

W1(c): The shaping index was wrong — corrected, and the ablation re-run.

The reviewer's algebra is correct. With γ=1\gamma=1 the original sum telescopes to Gn=R−Φψ(sn)G_n=R-\Phi_\psi(s_n), and sns_n is the state produced by ana_n. Detaching removes the gradient path but not the dependence: an action that lands in a high-potential state is penalized for that very action. It was not an action-independent baseline.

The corrected index credits the shaping term to the transition entering sns_n, consistent with F(s,s′)=γΦ(s′)−Φ(s)F(s,s')=\gamma\Phi(s')-\Phi(s) already used in App. A.4:

r~n=γΦψ(sn)−Φψ(sn−1),Gn=∑m=nH+1γm−nr~m+γH−n+1R,\tilde r_n=\gamma\Phi_\psi(s_n)-\Phi_\psi(s_{n-1}),\qquad G_n=\sum_{m=n}^{H+1}\gamma^{m-n}\tilde r_m+\gamma^{H-n+1}R,

with sH+1=s⊥s_{H+1}=s_\perp and r~H+1=−Φψ(sH)\tilde r_{H+1}=-\Phi_\psi(s_H), so that at γ=1\gamma=1 we obtain Gn=R−Φψ(sn−1)G_n=R-\Phi_\psi(s_{n-1}), a function of the pre-action state alone. The proof in App. A.4 was already correct; Eq. (10) mis-stated which transition the term belongs to.

We re-ran the ablation and the full model under the corrected index with everything else fixed (same checkpoint, width 16, no augmentation, three seeds). The corrected run reproduces the reported headline: 98.4 on Sudoku, 56.8 on ARC-AGI-1, 16.8 on ARC-AGI-2, within seed noise of the submitted values, so Table 1 is unchanged.

ARC-AGI-1, corrected indexSudokuARC-1Mean HH
terminal-only credit (no shaping)97.654.110.7
full ReGRAM98.456.89.6

Shaping improves ARC-AGI-1 by 2.7 points over terminal-only credit. The mis-specified version was not catastrophic because the LPRM potential is a regression target in [0,1][0,1], so the spurious term shifted returns by at most 1 in absolute value. The conclusion survives the correction, and the estimator is now correctly specified.

W1(d): The pathwise term is now part of the complete objective.

LReGRAM=LPG(θ,ψ)+λpwLpw(θ)+βrlLanchor(θ)+λvLLPRM(ψ)−λHH[πθ]\mathcal L_{\text{ReGRAM}}=\mathcal L_{\text{PG}}(\theta,\psi)+\lambda_{\text{pw}}\mathcal L_{\text{pw}}(\theta)+\beta_{\text{rl}}\mathcal L_{\text{anchor}}(\theta)+\lambda_v\mathcal L_{\text{LPRM}}(\psi)-\lambda_{\mathcal H}\mathcal H[\pi_\theta]

  • LPG\mathcal L_{\text{PG}}: the two-channel clipped surrogate of Eq. (13), ε=0.2\varepsilon=0.2.
  • Lpw(θ)=−E[vψˉ(sH(θ))]\mathcal L_{\text{pw}}(\theta)=-\mathbb E[v_{\bar\psi}(s_H(\theta))] with sH(θ)=mθ+σθ⊙ξs_H(\theta)=m_\theta+\sigma_\theta\odot\xi and ξ\xi the recorded noise; λpw=0.1\lambda_{\text{pw}}=0.1. This is the Sec. 4.3 term that was missing from Eq. (15); it was always in the implementation.
  • Lanchor\mathcal L_{\text{anchor}}: the closed-form Gaussian KL to θ0\theta_0, Eq. (14), βrl=0.05\beta_{\text{rl}}=0.05, θ0\theta_0 frozen.
  • LLPRM\mathcal L_{\text{LPRM}}: ∑n(vψ(zT(n))−Gn)2\sum_n(v_\psi(z_T^{(n)})-G_n)^2, λv=0.5\lambda_v=0.5; the critic target is the corrected return GnG_n, or RR when the head is used purely as a baseline.
  • λH=10−3\lambda_{\mathcal H}=10^{-3} annealed to 00.

Stop-gradient locations: advantages AnA_n detached before Eq. (13); Φψ\Phi_\psi detached inside the shaping term; θ0\theta_0 frozen in the anchor; ψ\psi detached in Lpw\mathcal L_{\text{pw}} so the value head receives gradient only from LLPRM\mathcal L_{\text{LPRM}}; old log-densities stored, not differentiated; conditioning state and sampled action fixed when ratios are recomputed; carry truncated across supervision steps as in Phase 1.

ARC-AGI-1SudokuARC-1
ReGRAM, λpw=0\lambda_{\text{pw}}=097.655.6
ReGRAM, λpw=0.1\lambda_{\text{pw}}=0.198.456.8

The pathwise term contributes 1.2 points; it is a variance-reduction and feature-training signal, consistent with calling the estimator hybrid rather than a differentiable surrogate.

AuthorsOpenReview ↗

Response to Reviewer G2iu (3/n)

W2(a)-(b): Matched protocol — one frozen checkpoint, before and after RL.

The reviewer is right on both counts: Table 1's GRAM row is GRAM's reported number, obtained with 20 trajectories while our matched setting uses 16; and the emergence analysis claimed no augmentation while its first point inherited the published 52.0. We re-evaluated the same Phase-1 checkpoint before and after RL under one frozen protocol — width 16, no augmentation at training or inference, LPRM selector, deterministic halting, test split, three seeds:

Frozen protocolSudoku-ExtremeARC-AGI-1ARC-AGI-2
GRAM, Phase-1 checkpoint96.7$\pm$0.251.4$\pm$0.510.6$\pm$0.4
ReGRAM, after Phase 298.4$\pm$0.256.8$\pm$0.716.8$\pm$0.6
Δ\Delta+1.7+5.4+6.2

The frozen-protocol deltas are slightly larger than the reported ones because our width-16, no-augmentation reproduction of Phase 1 sits below GRAM's published 20-trajectory number — exactly the effect the reviewer identified. The comparison is now like-for-like, and the gain is attributable to Phase 2 alone. The revision keeps the published rows as labeled external references and adds these matched rows.

W2(c): Figure 2b and Table 2 are reconciled. The two numbers came from separate seed sets: the extrapolation sweep's budget-16 point was 56.5$\pm$0.4 and Table 2's step-index ablation was 55.9$\pm$0.4, overlapping within one standard deviation. There is no configuration difference. We unified the sweep to the ablation's run and seeds; that point is now 55.9. The extrapolation conclusion rests on the large-budget points, where the gap is 9.1 points, and is unaffected.

W2(d): How much difficulty-dependent stopping Phase 1 already provides.

Correctly noted: the halt head is supervised in Phase 1, so the post-RL correlation does not by itself isolate RL. We measured the Phase-1 halt head on the same held-out instances and quartiles:

ARC-AGI-1Q1Q2Q3Q4ρs\rho_sMean HH
Phase-1 halt head (pre-RL)4.95.76.57.30.216.1
ReGRAM (post-RL)5.87.610.012.40.439.6

Phase 1 carries a weak monotone trend inherited from Q-target regression; RL roughly doubles the correlation and the spread. The accurate statement is that RL amplifies and reshapes an existing weak propensity, not that adaptive depth appears from nothing. The revision reports both rows and says exactly this.

W2(e): Accuracy-matched efficiency instead of a step-count ratio.

The 28.3$\times$ figure compares mean depth to the maximum budget and is not an efficiency claim; we have demoted it to a depth-transfer statement. We instead report accuracy against executed depth over a fixed-budget sweep, same checkpoint and protocol:

ARC-AGI-1Mean depth HHAccuracy
fixed depth 161651.4
fixed depth 323254.6
fixed depth 646456.6
fixed depth 12812857.6
fixed depth 1024102445.2
step-index halting, budget 1024—49.8
residual halting, budget 102436.258.9

At matched accuracy (worst case, against the best fixed budget of 128), residual halting uses 3.5$\times$ fewer supervision steps, while also being 1.3 points more accurate. Fixed depth degrades past 128 (45.2 at 1024) whereas residual halting is stable. This is the accuracy-matched statement the reviewer asked for.

AuthorsOpenReview ↗

Response to Reviewer G2iu (4/4)

Q1: State and action at which each PPO ratio is evaluated. Per high-level transition, as in W1(a): the conditioning state st−1=(ht−1,lt−1,ex)s_{t-1}=(h_{t-1},l_{t-1},e_x) and the sampled action at=hta_t=h_t are held fixed from the rollout; the ratio recomputes mθm_\theta and σθ\sigma_\theta from that fixed state, re-executing the deterministic carry, and divides by the stored old log-density. Definition 3.1 now states this, and App. A.2's per-transition densities are the definition rather than an implementation note.

Q2: The shaping ablation retains its gain with consistent indexing (2.7 points on ARC-AGI-1). The complete objective has λpw=0.1\lambda_{\text{pw}}=0.1, critic target GnG_n, and the stop-gradient list above; no separate transition-shaping correction is used beyond the re-indexed Eq. (10).

Q3: Result-to-protocol mapping. The frozen-protocol table fixes checkpoint, split, augmentation, trajectory count, selector, metric, and seed aggregation for the rows that matter; the full mapping, including N-Queens and graph coloring, is in the revised App. E.4. The headline gains are now reproduced by evaluating one Phase-1 checkpoint before and after RL under a frozen protocol. SLPO was adapted by replacing its autoregressive latent transition with GRAM's high-level transition while keeping its empirical surrogate density and its correctness-supervised stopping head, run from the same checkpoint and evaluation protocol; the ARC-2 gap between it (14.7) and ReGRAM (16.8) reflects exact versus empirical transition density.

Q4: Halting analyses and guidance clipping. The pre-RL measurement is in W2(d); the transition-only / halt-only / joint decomposition, run in identical settings, is in our response to Reviewer NJrB; the accuracy-versus-compute comparison is in W2(e). At budget 1024, 17.9% of trajectories continue past 16 steps and reach 53.8% accuracy, against 60.0% for those that halt within 16 (overall 58.9), so the extension beyond the training horizon carries real work. On guidance clipping: it is a one-sided floor on the parameter, σθ←max⁡(σθ,10−2)\sigma_\theta\leftarrow\max(\sigma_\theta,10^{-2}), not on the sample. The clamped σθ\sigma_\theta is used identically when sampling and when computing the log-density, so πθ\pi_\theta remains a normalized Gaussian with that variance and the likelihood is exact; no importance correction is needed. This is the practical advantage of a parameter-space clamp over a sample-space clip.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response (1/n)

We thank all reviewers. Four concerns recur across the reviews, and we address them here once so that each individual response can stay focused. Numbers in this response are identical to those in the individual responses; all new experiments use one frozen protocol — width 16, no training- or inference-time augmentation, LPRM selector, deterministic halting, test split, three seeds — and start from the same Phase-1 checkpoints as the paper.

C1. What is the action, and why is a Gaussian density justified? (G2iu W1, NJrB W2/Q1)

The submitted Definition 3.1 assigned one Gaussian to a supervision step, which contains T=3T=3 nonlinear stochastic high-level transitions; its endpoint marginal is not Gaussian, and the definition was wrong. The implementation was not: densities and PPO ratios were always computed per high-level transition (App. A.2). The revised definition is at that level: state st−1=(ht−1,lt−1,ex)s_{t-1}=(h_{t-1},l_{t-1},e_x), action at=hta_t=h_t,

πθ(at∣st−1,x)=N ⁣(at; mθ(st−1,x), Σθ(st−1,x)),mθ=uθ+μθ(uθ),\pi_\theta(a_t\mid s_{t-1},x)=\mathcal N\!\left(a_t;\,m_\theta(s_{t-1},x),\,\Sigma_\theta(s_{t-1},x)\right),\qquad m_\theta=u_\theta+\mu_\theta(u_\theta),

normalized by the unit-Jacobian map ϵt↦ht=ut+ϵt\epsilon_t\mapsto h_t=u_t+\epsilon_t. A supervision step is a macro-action with density equal to the product of TT such Gaussians. The deterministic low-level carry contributes no likelihood factor, but its parameters enter mθm_\theta because the ratio recomputes the proposal from the recorded state.

C2. What does RL add over the Phase-1 Q-head? (all three reviewers)

We measured the Phase-1 halt head and decomposed RL by channel, from one checkpoint, in identical settings:

ARC-AGI-1SudokuARC-1Mean HHρs\rho_s
Phase-1 checkpoint (Q-head halting)96.751.46.10.21
halt only (θ\theta frozen)97.554.08.30.34
transition only (ψ\psi frozen)98.155.37.40.26
joint (ReGRAM)98.456.89.60.43

Phase 1 already carries a weak difficulty trend (ρs=0.21\rho_s=0.21); RL roughly doubles it. Both channels contribute on their own, and the joint gain (5.4) is sub-additive relative to their sum (6.5). The honest claim is that RL amplifies an existing propensity and improves both the transition distribution and the stopping rule, not that adaptive depth exists only after RL. The paper now says this, and reports the Phase-1 row alongside the post-RL row in the emergence analysis.

C3. Comparability: checkpoint, trajectory count, augmentation, seeds. (G2iu W2, AwZb Q4)

Table 1's GRAM row was GRAM's reported number, obtained at 20 trajectories, while our matched setting uses 16; and the emergence analysis claimed no augmentation while its first point inherited the published 52.0. Both are corrected. Evaluating the same Phase-1 checkpoint before and after RL under the frozen protocol gives:

Frozen protocolSudoku-ExtremeARC-AGI-1ARC-AGI-2
GRAM, Phase-1 checkpoint96.7$\pm$0.251.4$\pm$0.510.6$\pm$0.4
ReGRAM98.4$\pm$0.256.8$\pm$0.716.8$\pm$0.6
Δ\Delta+1.7+5.4+6.2

The deltas are slightly larger than reported because our width-16, no-augmentation reproduction of Phase 1 sits below GRAM's published number. Every row in the revision now lists its trajectory count and selector.

C4. Computation claims at matched accuracy. (G2iu W2(e), NJrB Q4)

The 28.3$\times$ ratio compared mean depth to the maximum budget and is not an efficiency claim; we have demoted it. The accuracy-matched comparison, same checkpoint:

ARC-AGI-1Mean depth HHAccuracy
fixed depth 646456.6
fixed depth 12812857.6
fixed depth 1024102445.2
step-index halting, budget 1024—49.8
residual halting, budget 102436.258.9

Against the best fixed budget (128), residual halting uses 3.5$\times$ fewer supervision steps at matched accuracy and is 1.3 points more accurate. Fixed depth degrades past 128; residual halting does not.

AuthorsOpenReview ↗

Common Response (2/2)

C5. Why start from a pretrained generative prior? (AwZb W2/Q1)

We ran RL from scratch (0% Phase 1) and from partial Phase-1 checkpoints, same Phase 2, same reward:

Phase-1 pretraining before RLSudokuARC-1
0% (RL from scratch)1.95.8
25%93.845.2
50%96.952.3
100%98.456.8

The reason is that an outcome-only reward provides no learning signal when every rollout in a group fails; variational pretraining supplies the support on which the reward is occasionally positive. The TRM + grafted stochastic head + RL row (95.1/49.0/10.2 against 98.4/56.8/16.8) makes the complementary point that adding a stochastic head without pretraining it as a transition distribution is not enough.

Other corrections made in the revision, independent of any single reviewer: the pathwise term of Sec. 4.3 now appears in the complete objective with its coefficient (λpw=0.1\lambda_{\text{pw}}=0.1) and an ablation; the stop-gradient list is complete; the shaping term of Eq. (10) is re-indexed so the potential is credited to the transition entering the scored state (the previous indexing made the baseline action-dependent, as G2iu W1(c) correctly notes); and the Figure 2b / Table 2 discrepancy at budget 16 was two seed sets, now unified to 55.9. We also added the width-axis sweep suggested by AwZb.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.