acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason from Feedback: Consistency-Aware Post-Training for Language Models

Abstract

Post-training has made substantial progress at optimizing the final outcome of reasoning tasks, but an equally important ability is rarely optimized as a target in its own right: how a model integrates newly acquired feedback into its subsequent reasoning and decisions. This becomes acute once the model must interact with external tools and execution environments, where it has to judge when additional feedback is warranted, absorb new information without derailing its existing line of reasoning, and tell which interactions actually contributed to success. We treat reasoning from feedback as an explicit post-training objective. To this end we introduce three complementary training signals: a contrastive representational constraint that keeps the model's reasoning state semantically coherent across an interaction while it absorbs the new observation; an uncertainty-based interaction policy that leads the model to seek feedback selectively rather than reflexively; and process-level credit assignment from counterfactual trajectory value estimation, which separates genuinely helpful interactions from redundant or harmful ones. The three are optimized jointly with Group Relative Policy Optimization (GRPO) and trained end-to-end on trajectories the model generates through interaction with a real execution environment. Instantiated in a sandboxed code-execution environment and evaluated on AIME 2024/2025, MATH-500, LiveCodeBench, and GPQA Diamond, our 7B model improves over the strongest same-size agentic RL baseline by 4.8 points on average. Analysis shows that the representational constraint improves feedback-utilization precision by 23%, while uncertainty-based gating reduces redundant interactions by 41%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training, reinforcement learning, feedback integration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nVrCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes consistency-aware post-training to improve how language models reason from tool feedback. It combines contrastive state regularization, uncertainty-based interaction gating, and approximate counterfactual credit assignment with GRPO. Experiments on five mathematics, coding, and science benchmarks report improved accuracy, better feedback utilization, and fewer redundant interactions.

Critical strengths
  1. The paper clearly formulates feedback seeking, integration, and attribution as training objectives, with relevant behavioral diagnostics beyond final-answer accuracy.
  2. Both backbones outperform the corresponding DemyAgent recipe on all five benchmarks. The behavioral analyses also show fewer redundant interactions.
  3. Component ablations support the training procedure, and the authors carefully distinguish the idealized theoretical statements from the practical approximations.
Critical weaknesses
  1. Eq. (4) lacks direct validation against actual counterfactual effects, leaving the reliability of interaction-level credit unclear.
  2. The feedback-integration analysis does not control for differences in requests, observations, and call counts.
  3. The gate lacks comparisons with simpler alternatives at matched call counts and an ablation of its entropy input.
  4. No GRPO comparison matches total training compute, including verifier training and target generation.
Questions for authors
  1. Does Eq. (4) correctly predict the signs and rankings of interaction effects measured through execute/skip rollouts?
  2. Does coherence training improve responses to identical feedback, particularly when reliable observations contradict an incorrect hypothesis?
  3. At matched call counts, does the gate outperform random suppression, entropy thresholds, and a hidden-state-only gate?
  4. Do the gains persist when GRPO receives the same total training budget, including all verifier-related costs?
4 comments
AuthorsOpenReview ↗

Response to W1 & Q1 (Part 1/4): Equation (4) vs. Execute/Skip Rollouts

Thank you for your rigorous review and falsifiable test designs. We number your four weaknesses/questions W1–W4 and Q1–Q4 in order, pairing each WkW_k with QkQ_k across four replies (three training seeds, mean ± SD under the paper's protocol unless noted; 5-bench / 4-bench Avg. match Tables 1 and 2).

Direct answer (W1 & Q1). On 612 interactions evaluated via execute/skip rollouts, Eq. (4) matches the empirical sign on 77.3% (249/322) of non-neutral interactions, achieves a 0.52 global Spearman rank correlation, and reaches 0.34 within trajectories. Its errors stem from two diagnosable structural biases; removing them with a replay-calibrated estimator raises performance to 88.2% / 0.71 / 0.59, while retraining the policy leaves main benchmark accuracy unchanged (63.6 ± 1.0 vs. 63.1 ± 1.2).

1. Execute/skip rollout evaluation & domain breakdown.

Across 150 training trajectories from steps 500 and 1,000 (50 math, 50 code, 50 science; 612 executed interactions), we sample 64 execute vs. 64 skip continuations from each pre-request prefix (skip matches bm=0b_m = 0: request unexecuted, no observation inserted, later calls allowed), defining Δm\Delta_m as their solve-rate difference: 253 helpful (Δm≥0.15\Delta_m \ge 0.15), 69 harmful (Δm≤−0.15\Delta_m \le -0.15), and 290 neutral (∣Δm∣<0.15|\Delta_m| < 0.15). Within-trajectory metrics cover the 130 trajectories with ≥3\ge 3 interactions (random top-1 hit rate: 30/130 = 23.1%; 32-vs.-32 split-half ceiling: 0.83 global, 0.74 within-trajectory):

| Credit estimator | Sign agreement (322 w/ ∣Δm∣≥0.15|\Delta_m| \ge 0.15) | Global Spearman ρ\rho (all 612) | Within-trajectory Spearman ρ\rho | Top-credited matches arg⁡max⁡Δm\arg\max \Delta_m (130 trajs) | |---|---|---|---|---| | Uniform split R(τ)/MR(\tau)/M | 61.8% (199/322) | 0.27 | 0.00 (constant) | 23.1% (random tie-break) | | Eq. (4) heuristic (paper) | 77.3% (249/322) | 0.52 | 0.34 | 46.2% (60/130) | | Replay-calibrated c^mcal\hat{c}_m^{\text{cal}} (new) | 88.2% (284/322) | 0.71 | 0.59 | 65.4% (85/130) |

By domain, Eq. (4) sign agreement is 79.5% (89/112, math), 77.4% (96/124, code), and 74.4% (64/86, science), rising to 89.3%, 88.7%, and 86.0% after calibration.

2. Structural biases & replay-calibrated retraining.

  • Two error sources: (i) Vψ(τ)V_\psi(\tau) conditions on the realized suffix, inflating credit in solved trajectories (52.6%, 101/192 of neutral calls get c^m>0\hat{c}_m > 0 when solved vs. 9.2%, 9/98 when unsolved; Section 5.5's human correlation of 0.73 is higher than 0.52 because annotators also saw the realized suffix); (ii) wmw_m penalizes large state updates (42.4%, 28/66 of helpful calls in the top wmw_m quartile get c^m≤0\hat{c}_m \le 0 vs. 14.4%, 27/187 elsewhere). Also, V^¬m>1\hat{V}_{\neg m} > 1 on 93/612 (15.2%) interactions ([0,1] clipping gives 250/322 = 77.6%).
  • Replay-calibrated estimator: Adding a skip-branch target head to VψV_\psi (6,000 prefixes × 32 skip rollouts, +50% offline rollouts) defines c^mcal=Vψ(post-obs)−Vψskip(pre-req)∈[−1,1]\hat{c}_m^{\text{cal}} = V_\psi(\text{post-obs}) - V_\psi^{\text{skip}}(\text{pre-req}) \in [-1, 1]—consistent under Appendix B.3 without suffix conditioning or wmw_m (+1 verifier pass/interaction; held-out evaluation above). Retraining yields 63.6 ± 1.0 5-bench Avg. (vs. 63.1 ± 1.2) and 3.7 vs. 3.4 productive AIME24 interactions. We will reframe Eq. (4) as a shaping heuristic and report c^mcal\hat{c}_m^{\text{cal}} and full replay breakdowns in Sections 4.2, 5.5, 6, Table 1, and new Appendix D.
AuthorsOpenReview ↗

Response to W2 & Q2 (Part 2/4): Identical-Feedback Probe with Contradicting Observations

Direct answer (W2 & Q2). We agree that Section 5.3 did not hold context fixed, and built a controlled identical-feedback probe. Given the exact same problem, prior reasoning, request, and call history, when a reliable observation contradicts an incorrect prior hypothesis, our full model adopts the observation and revises its reasoning on 80.4% (579/720) of cases vs. 72.1% (519/720) without the coherence constraint (+8.3 points, paired 95% CI [3.0, 13.6])—showing LcohL_{\text{coh}} does not lock the model into false hypotheses. When the observation itself is artificially corrupted, however, the full model trusts the corrupted output slightly more often, lowering accuracy within noise.

1. Controlled identical-feedback probe (240 contexts × 3 seeds = 720 evaluations).

From held-out MATH validation problems (disjoint from MATH-500), the shared SFT checkpoint generates 240 neutral contexts—each containing a problem, reasoning that explicitly states an expected intermediate value, one Python request computing that value, and the returned observation—edited in one place across three conditions:

  • Consistent: correct prior expectation, correct observation.
  • Contradicting: prior expectation edited to an incorrect value, observation kept correct (your exact test).
  • Corrupted: prior expectation kept correct, observation edited to an incorrect value.

Greedy continuations from all 3 seeds of each 7B method (allowing ≤2\le 2 further calls; models make 0.9, 0.8, 0.7, 0.7 follow-up calls on average) yield:

Metric (% and count / 720)Agentic GRPODemyAgent recipeOurs w/o LcohL_{\text{coh}}Ours (full)Full − w/o LcohL_{\text{coh}} [95% CI]
Consistent: final answer correct82.1 (591)84.0 (605)86.8 (625)87.2 (628)+0.4 [−3.3, 4.1]
Contradicting: adopts observation64.7 (466)68.2 (491)72.1 (519)80.4 (579)+8.3 [3.0, 13.6]
Contradicting: final answer correct55.7 (401)60.3 (434)64.2 (462)71.7 (516)+7.5 [1.9, 13.1]
Corrupted: adopts corrupted value31.8 (229)34.4 (248)37.4 (269)41.1 (296)+3.7 [−1.1, 8.5]
Corrupted: final answer correct51.4 (370)50.6 (364)49.7 (358)46.8 (337)−2.9 [−8.2, 2.4]

(Paired bootstrap CIs over the 240 contexts; vs. Agentic GRPO on Corrupted accuracy, Δ is −4.6 [−10.2, 1.0].) Because Eq. (2) constrains only sub-goal identity in a low-dimensional projection k≪dk \ll d, orthogonal directions remain free to update numerical beliefs—improving belief revision under reliable sandbox outputs while motivating explicit verification for noisy search/API tools.

2. Call-count normalization of Table 8 (AIME 2024).

Weighting Table 8 by accuracy gives overall per-trajectory failure rates of 22.2% (Agentic GRPO) vs. 5.4% (ours), a 4.1× gap that is confounded by trajectory length. Normalizing per 100 executed interactions narrows it to 2.9×, while the rate still rises from 1.4 to 2.4 without LcohL_{\text{coh}}:

Integration failures / 100 executed callsAgentic GRPODemyAgent recipeOurs w/o LcohL_{\text{coh}}Ours (full)
AIME 20244.03.12.41.4

We will make the identical-feedback probe primary in Section 5.3 and new Appendix H, add per-call rates to Table 8, and note corrupted-observation sensitivity in Section 6.

AuthorsOpenReview ↗

Response to W3 & Q3 (Part 3/4): The Gate vs. Simpler Alternatives at Matched Interaction Counts

Direct answer (W3 & Q3). At matched executed call counts, the learned gate substantially outperforms random suppression (+4.22 points on the 4-benchmark average) and a parameter-free entropy threshold (+2.60 points). Removing the explicit entropy scalar U~m\tilde{U}_m, however, leaves a hidden-state-only gate within 0.47 points of the full gate (64.18 ± 0.73 vs. 64.65 ± 0.73, well within seed noise). Thus, what drives the gain is a state-conditioned gate trained from counterfactual credit rather than the explicit vocabulary-entropy feature; we will revise our framing accordingly.

1. End-to-end training at matched interaction counts (7B, 3 seeds each).

All four gating rules share the exact same RL pipeline (LcohL_{\text{coh}}, c^m\hat{c}_m, and the 10%-step open-gate warm-up), changing only how bm∈{0,1}b_m \in \{0, 1\} is chosen at each request:

  • Random suppression: executes independently with fixed probability calibrated on the validation set to match the full gate's executed call count.
  • Entropy threshold: parameter-free rule bm=1[U~m>τH]b_m = \mathbf{1}[\tilde{U}_m > \tau_H], with τH\tau_H calibrated on the validation set in the same way.
  • Learned gate, hidden state only: removes U~m\tilde{U}_m from Eq. (5) so MLPϕ(hm−)\text{MLP}_\phi(h_m^-) is trained on sign(c^m)\text{sign}(\hat{c}_m) via Eq. (6).
  • Learned gate, hidden state + entropy (Full, Eq. 5): frozen inference thresholds μ\mu across the 3 seeds are 0.69, 0.76, 0.74 (mean 0.73).
Gating rule (7B, 3 seeds)AIME24 calls / trajProductive (%)AIME24MATH-500LCB v6GPQA-D4-bench Avg. ± SD
No gate (Tables 2 & 6)6.268.560.093.846.250.562.63 ± 0.62
Random suppression4.069.455.693.044.149.060.43 ± 0.72
Entropy threshold (U~m>τH\tilde{U}_m > \tau_H)3.973.158.993.645.550.262.05 ± 0.63
Learned gate, hm−h_m^- only4.381.662.295.047.352.264.18 ± 0.73
Learned gate, [hm−;U~m][h_m^-; \tilde{U}_m] (Full)4.182.963.395.247.652.564.65 ± 0.73

2. Key takeaways & inference-time gate swap.

  • Why random & entropy rules fall short: Random suppression prunes helpful and redundant calls alike (productive share 68.5% → 69.4%, accuracy −2.20 vs. no gate), confirming gains do not come from merely calling the tool less. Next-token entropy U~m\tilde{U}_m captures local phrasing ambiguity rather than whether computation will raise solve probability (productive share 73.1%, −0.58 vs. no gate).
  • Why hm−h_m^- suffices: Because the next-token logits are a linear readout of hm−h_m^-, MLPϕ(hm−)\text{MLP}_\phi(h_m^-) already extracts relevant uncertainty cues directly (Spearman ρ\rho between learned gmg_m and U~m\tilde{U}_m is only 0.38).
  • Inference-time swap (same trained policy): Replacing the full model's gate at test time with call-matched random suppression or an entropy threshold drops 4-bench Avg. from 64.65 to 61.9 and 62.7.

We will reframe "uncertainty-based gate" as "credit-trained state gate (with predictive entropy as an optional auxiliary feature)" in the Abstract, Introduction, and Section 4.3, and add these comparisons to Section 5.4, Table 2, and new Appendix I.

AuthorsOpenReview ↗

Response to W4 & Q4 (Part 4/4): GRPO at Matched Total Training Compute

Direct answer (W4 & Q4). Yes—the gains persist in full. Counting all offline verifier target generation and verifier training, our method uses 34.1% more total training compute than 1,000-step Agentic GRPO (Section 4.4 counted only the ~21% online RL overhead). Extending Agentic GRPO to match total compute (1,340 steps) raises its 5-benchmark average only from 50.9 ± 1.2 to 51.5 ± 1.0—still 11.6 points below our method (63.1 ± 1.2); extending the DemyAgent recipe to 1,340 steps moves it from 58.3 ± 0.9 to 58.7 ± 1.1 (−4.4 points below ours).

1. Complete end-to-end compute accounting (1,000-step Agentic GRPO = 100 units).

Measured on identical hardware and normalized to the baseline run, our method adds online RL overhead (+20.8%, the three terms Section 4.4 reports as ~5%, <1%, and ~15%) plus previously omitted offline verifier costs (+13.3%):

Compute stage (7B, per training seed)Agentic GRPOOurs
RL 1,000 steps (rollout execution & policy update)100.0100.0
Coherence constraint forward & backward (LcohL_{\text{coh}})0.05.0
Gate forward & update (LgateL_{\text{gate}})0.00.8
Online verifier forward passes during RL0.015.0
Verifier target generation (384,000 continuations, offline)0.012.4
Verifier training (12,000 prefixes × 2 epochs, offline)0.00.9
Total end-to-end compute100.0134.1

2. Baselines at matched total compute (7B, 3 seeds each).

Alongside running both baselines to 1,340 RL steps (134 units), we also test spending the offline verifier rollouts directly on 2 epochs of SFT over the 9,640 complete correct trajectories generated during target collection before 1,000 steps of Agentic GRPO:

Method (7B, 3 seeds)Total compute (rel.)AIME24AIME25MATH-500LCB v6GPQA-D5-bench Avg. ± SD
Agentic GRPO, 1,000 steps (Table 1)10043.336.791.238.644.950.9 ± 1.2
Agentic GRPO, 1,340 steps (compute-matched)13444.437.891.439.044.951.5 ± 1.0
Successful-rollout SFT → Agentic GRPO (1K steps)11446.740.091.840.547.053.2 ± 1.1
DemyAgent recipe, 1,000 steps (Table 1)10056.750.093.043.248.558.3 ± 0.9
DemyAgent recipe, 1,340 steps (compute-matched)13457.850.093.243.748.858.7 ± 1.1
Ours, 1,000 steps (Table 1)13463.356.795.247.652.563.1 ± 1.2

Extra RL steps add only +0.6 and +0.4 points because both outcome-only baselines have already plateaued past step 800 on the 500-problem held-out MATH validation set (89.2% / 90.6% / 91.0% / 91.2% at steps 600 / 800 / 1,000 / 1,340 for Agentic GRPO; 91.4% / 92.6% / 92.8% / 93.0% for DemyAgent recipe). Using the offline rollouts for SFT helps more (53.2) than extra RL steps (51.5), yet still trails by 9.9 points without step-level credit and selective gating. We will add the full compute ledger, compute-matched baselines, and 0→1,340 validation curves to Section 4.4, Table 1, Section 6, Appendix A, and new Appendix J. Thank you again for your thorough review.

Reviewer_H7NKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a tool-using model can learn when to seek feedback, how to use the returned information, and which interactions actually helped it solve a problem. It introduces a training signal for each of these three behaviors. In the authors' evaluation, the 7B model averages 63.1% across five benchmarks, compared with 58.3% and 50.9% for the two baselines in Table 1, while making fewer unhelpful tool calls. The paper also includes component ablations and discusses its limitations.

Critical strengths
  1. The paper breaks tool use into three concrete questions: when to obtain information, how to continue reasoning after receiving it, and which interaction was useful. This clear framing connects the proposed training signals to the experiments (Sections 3–5).
  2. The controlled comparison uses the same model, data, and tool environment, and the component ablations show that each signal contributes. Removing the interaction-credit reward, coherence constraint, or gate lowers the four-benchmark average in Table 2 by 4.53, 3.50, and 2.03 points, respectively. On AIME 2024, average tool calls fall from 6.8 to 4.1 per solution, largely through a reduction in unhelpful calls from about 3.2 to 0.7 (Section 5.3).
Critical weaknesses
  1. The estimate of whether an interaction helped is not checked against actually removing that interaction and letting the model continue. Equation (4) uses a continuation that has already seen the tool result. The correlation of 0.73 with human judgments (Section 5.5) is useful, but it does not establish the interaction's true effect on success.
  2. The stability of the reported gains is unclear. Table 1 reports means over three training runs without showing each run or a range of results. On the 30-question AIME tests, a difference of about 6.6–6.7 points is roughly two questions in a single run. The paper should show whether the gains remain across runs and test-question samples.
Questions for authors
  1. On a sample of interactions, could you remove one tool call, let the model continue from the same point, and compare the outcome with Equation (4)'s estimate? How often does the estimated success probability without that call fall outside [0, 1]?
  2. What are the results of the individual training runs in Tables 1 and 2, and how much do they vary? Do the gains remain across different training runs and test-question samples?
2 comments
AuthorsOpenReview ↗

Response to W1 & Q1 (Part 1/2): Removing a Tool Call and Letting the Model Continue vs. Equation (4)

Thank you for your thoughtful review and clear questions. We agree with both of your points (numbering your weaknesses W1–W2 and questions Q1–Q2 in order, pairing W1+Q1 here and W2+Q2 in Part 2).

Direct answer (W1 & Q1). We carried out both checks: (i) Compared against actually removing a tool call and letting the model continue, Eq. (4) gets the direction right (helpful vs. harmful) on 77.3% (249/322) of non-neutral calls with a rank correlation of 0.52—informative, though a rough approximation. (ii) Across all 4,920,320 executed calls logged during RL, Eq. (4)'s implied no-call success probability V^¬m\hat{V}_{\neg m} exceeds 1 on 826,614 calls (16.8%) and is <0< 0 on 0 calls (93/612 = 15.2% >1> 1 in our replay sample). Training a bounded estimator directly on call-removal outcomes eliminates out-of-bounds values and leaves our main benchmark results unchanged (63.6 ± 1.0 vs. 63.1 ± 1.2).

1. Removing one tool call and letting the model continue (Q1a).

On 150 training trajectories from mid-training (step 500) and end-of-training (step 1,000) containing 612 tool calls, we branched at each pre-call position: 64 continuations executing the call normally vs. 64 continuations skipping it (no tool output inserted; later calls still allowed). The difference in solve rates measures the call's actual effect: 253 calls helped (≥+0.15\ge +0.15), 69 hurt (≤−0.15\le -0.15), and 290 had no clear effect (∣diff∣<0.15|\text{diff}| < 0.15). Because 64 rollouts have finite-sample noise, splitting them 32 vs. 32 gives a 0.83 rank-correlation ceiling:

Credit methodCorrect sign on 322 helpful/harmful callsRank correlation across all 612 calls
Splitting final outcome equally (R(τ)/MR(\tau)/M)61.8% (199/322)0.27
Eq. (4) approximation (paper)77.3% (249/322)0.52
Bounded call-removal estimator (new)88.2% (284/322)0.71
  • Why human correlation (0.73) exceeded 0.52: Like Eq. (4), human annotators saw what happened after the call. Many calls that look useful afterward (e.g., double-checking a number already derived correctly) turn out to be optional when removed—confirming your point that 0.73 reflects post-hoc agreement rather than causal necessity.

2. Out-of-bounds rate & bounded call-removal estimator (Q1b).

  • Why V^¬m>1\hat{V}_{\neg m} > 1 occurs: Because Vψ∈[0,1]V_\psi \in [0, 1] (sigmoid output) and wm=1−cos⁡(hm+,hm−)≥0w_m = 1 - \cos(h_m^+, h_m^-) \ge 0, Eq. (4) adds two non-negative terms: it never falls below 0, but exceeds 1 when the model was already confident before the call (18 of the 93 out-of-bounds calls, 19.4%, still helped in replay yet were forced negative). Simply clipping V^¬m\hat{V}_{\neg m} to [0,1][0, 1] barely changes sign accuracy (77.3% → 77.6%, 250/322).
  • Bounded calibration: Training VψV_\psi on actual skipped-call continuations so credit is Vψ(with result)−Vψ(call skipped)V_\psi(\text{with result}) - V_\psi(\text{call skipped}) keeps both terms in [0,1][0, 1] and credit in [−1,1][-1, 1] (0% out-of-bounds), reaching 88.2% sign accuracy on held-out calls and 63.6 ± 1.0 5-bench Avg. upon retraining (vs. 63.1 ± 1.2 across 3 seeds). We will reframe Eq. (4) as a heuristic, report out-of-bounds rates, and make call-removal replay the main evidence in Sections 4.2, 5.5, and new Appendix D.
AuthorsOpenReview ↗

Response to W2 & Q2 (Part 2/2): Individual Runs and Stability of the Gains

Direct answer (W2 & Q2). Over same-setup Agentic GRPO, gains hold on every training run and every benchmark, and all 95% question-resampling intervals exclude zero. Over the DemyAgent recipe, the 5-benchmark average holds on every run (7B worst run 62.2 > baseline best 59.1; 4B 58.0 > 56.4) with a 7B question-resampling 95% CI of [1.0, 8.6], whereas individual CIs on 30-question AIME 2024/2025 and GPQA-D include zero—confirming your exact point about ~2-question AIME margins.

1. Individual training runs in Table 1 & question-resampling CIs.

Each cell lists the three training seeds (s1/s2/s3), whose mean equals Table 1:

MethodAIME24AIME25MATH-500LCB v6GPQA-D5-bench Avg. (s1/s2/s3; mean ± SD)
Agentic GRPO (7B)43.3/40.0/46.736.7/33.3/40.090.6/92.0/91.038.6/38.9/38.345.5/44.9/44.450.9/49.8/52.1 (50.9 ± 1.2)
DemyAgent recipe (7B)56.7/53.3/60.046.7/53.3/50.092.8/92.0/94.242.9/43.1/43.648.0/50.0/47.557.4/58.3/59.1 (58.3 ± 0.9)
Ours (7B)63.3/60.0/66.753.3/56.7/60.096.0/95.4/94.247.4/47.9/47.551.0/53.0/53.562.2/62.6/64.4 (63.1 ± 1.2)
DemyAgent recipe (4B)53.3/50.0/56.743.3/50.0/46.790.6/92.8/92.040.1/40.6/40.846.5/45.5/46.054.8/55.8/56.4 (55.7 ± 0.8)
Ours (4B)56.7/60.0/53.353.3/46.7/50.094.2/92.4/93.643.5/43.9/44.047.5/50.5/49.059.0/58.7/58.0 (58.6 ± 0.5)

Five-benchmark averages have zero seed overlap at both scales, while single 30-problem AIME runs touch at the boundary (e.g., 7B AIME24 solves 19/18/20 vs. 17/16/18 problems, touching at 18 = 60.0%). Paired bootstrap resampling of test questions (10,000 replicates, averaging each item across the 3 seeds; at 4B, 5-bench Δ vs. DemyAgent recipe is +2.9 [−0.7, 6.5], framed as directionally consistent):

7B Difference [95% question CI]AIME24AIME25MATH-500LCB v6GPQA-D5-bench Avg.
Ours − Agentic GRPO+20.0 [5.6, 34.4]+20.0 [4.4, 35.6]+4.0 [2.0, 6.2]+9.0 [5.5, 12.4]+7.6 [2.3, 13.0]+12.1 [7.7, 16.6]
Ours − DemyAgent recipe+6.6 [−5.6, 18.9]+6.7 [−6.7, 20.0]+2.2 [0.4, 4.0]+4.4 [1.5, 7.3]+4.0 [−0.8, 8.8]+4.8 [1.0, 8.6]

2. Individual training runs in Table 2 (4-benchmark Avg., 7B).

("Full method" and "w/o all three" match the exact 4-benchmark subset of the 7B rows above.)

Table 2 variantSeed 1 / 2 / 3Mean ± SD
Full method64.43 / 64.08 / 65.4864.65 ± 0.73
w/o gate63.29 / 62.05 / 62.5562.63 ± 0.62
w/o coherence constraint60.46 / 61.70 / 61.2961.15 ± 0.63
w/o counterfactual credit60.26 / 60.65 / 59.4860.13 ± 0.60
No gate warm-up57.76 / 61.46 / 60.1259.78 ± 1.87
w/o coherence and gate57.72 / 58.46 / 57.2257.80 ± 0.62
ℓ2\ell_2 alignment of raw states56.77 / 54.81 / 55.5255.70 ± 0.99
w/o all three (= Agentic GRPO)54.50 / 53.95 / 55.1054.50 ± 0.58

Our worst full seed (64.08) beats the best seed of any ablation (63.29), while "No gate warm-up" varies more (±1.87, early exploration collapse on seed 1). Because "w/o credit" and "w/o coherence" overlap slightly across 3 seeds and the coherence × gate interaction is −1.33 (SE 0.83), we will soften relative-ranking and sub-additivity claims in Sections 5.2 and 5.4, add SDs to Tables 1–2, and report all per-seed scores and CIs in new Appendix F. Thank you again for your careful review.

Reviewer_DB7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how language models seek and incorporate feedback from a code-execution environment during reasoning. It proposes three complementary training signals: a contrastive loss aligning projected hidden states before and after an interaction, a heuristic counterfactual credit estimator measuring the contribution of individual interactions, and an uncertainty-conditioned gate controlling whether proposed tool calls are executed. These signals are combined with GRPO and final-answer correctness rewards. Experiments on several benchmarks shows improved performance over baselines.

Critical strengths
  1. The analysis is detailed and examines productive versus redundant interactions, feedback-integration failures, and tool use across problem difficulty.
  2. I believe the paper identifies an important limitation of outcome-based training i.e final answer correctness provides limited information about whether individual tool calls were necessary or their outputs were incorporated appropriately. The decomposition into seeking, integrating, and attributing feedback provides a useful framework for studying these behaviors.
  3. The authors acknowledge that representation matching does not guarantee feedback utilization which is helpful.
  4. The method improves performance across the five evaluated benchmarks and both model backbones
Critical weaknesses
  1. The value function is trained using success rates from sampled continuations, but the paper does not clearly describe how it is refreshed during policy optimization. As the policy improves, both the distribution of encountered trajectories and the probability of success from a given prefix can change resulting in a distribution shift. Consequently, the value estimates may become off even for previously observed prefixes. The paper should clarify the verifier update procedure and evaluate the effect of freezing versus periodically refreshing it.
  2. The verifier relies on Monte Carlo targets from 32 continuations per prefix. On difficult problems, these continuations may contain few or no successes, yielding noisy or uninformative targets. It is unclear how the training data provide sufficient coverage of informative prefixes and how robust value estimation is achieved in this regime. Since, if we have really good on-policy succesful rollouts why not also SFT directly on them before GRPO, that can already solve the sparse reward problem to certain extent?
  3. The estimator uses the observed continuation to approximate what would happen without a tool call, although removing the call could lead the model down a substantially different reasoning path. For example, the model might solve the problem through an alternative approach without the tool, while also succeeding through the tool-assisted path. A change in hidden states or a successful observed continuation does not establish that the tool improved the probability of success. After the tool, lets say the observation changes but still that doesnt solve the real causal question and require much more ablations. Executed counterfactual rollouts are needed to validate whether the proposed approximation captures the intended causal contribution.
  4. The gate reward depends on the sign of the estimated counterfactual credit, which itself depends on the learned value function and the heuristic approximation. Errors in either component can therefore affect both policy shaping and tool selection, potentially suppressing useful interactions and reinforcing unhelpful ones. The gate does not provide an independent signal that corrects these errors. Overall, the method contains several tightly coupled components. Although the paper includes component-removal ablations, these do not fully explain how estimation errors propagate or which mechanisms drive the gains.
Questions for authors

Mentioned above

4 comments
AuthorsOpenReview ↗

Response to W1 (Part 1/4): Verifier Update Procedure — Frozen vs. Periodically Refreshed

Thank you for your careful review and constructive feedback. We number your four weaknesses W1–W4 in order and address each with new experiments (three training seeds, mean ± SD under the paper's protocol unless noted; 5-bench / 4-bench Avg. match Tables 1 and 2).

Direct answer (W1). Our verifier VψV_\psi is trained once before RL and frozen throughout RL (omitted in the submission). Distribution shift does occur—by step 1,000, the frozen verifier underpredicts empirical success by 0.12 and its rank correlation drops from 0.81 to 0.71—yet refreshing it every 250 steps changes final 5-bench accuracy only within seed noise (63.1 ± 1.2 → 63.5 ± 1.0).

1. Paper setup & drift measurement across RL.

VψV_\psi is initialized from the 3K-demonstration SFT checkpoint, which also generates its training data: 2 trajectories on each of 1,500 training prompts (500 math, 500 code, 500 science), retaining 4 random prefixes per trajectory (before requests and after observations; 12,000 prefixes × 32 continuations = 384,000 rollouts, trained for 2 epochs; Algorithm 1 has no ψ\psi update). To measure shift, we evaluate five RL checkpoints on 400 fresh training prefixes × 32 continuations each, comparing the frozen verifier against a periodically refreshed variant fine-tuned for 1 epoch at steps 250, 500, and 750 on 3,000 new on-policy prefixes (96,000 rollouts per refresh) mixed with the initial data (measured right after each refresh; step 1,000 uses the step-750 verifier):

RL step02505007501,000
Empirical prefix success rate0.380.460.550.590.62
Frozen VψV_\psi: mean bias (pred. − empirical)+0.02−0.02−0.08−0.09−0.12
Frozen VψV_\psi: Spearman ρ\rho with empirical0.810.790.760.730.71
Refreshed every 250 steps: mean bias+0.02−0.01−0.01−0.02−0.03
Refreshed every 250 steps: Spearman ρ\rho0.810.820.810.800.79

2. End-to-end RL: frozen vs. refreshed verifier.

Verifier schedule5-bench Avg. (%)AIME24 interactions (productive / total)Productive share (%)Extra offline rollouts
Frozen (paper, Table 1)63.1 ± 1.23.4 / 4.182.90 (384K base)
Refreshed every 250 steps63.5 ± 1.03.7 / 4.484.1+288K (+75%)
  • Why substantial drift has modest end-to-end impact: Credit c^m=Vψ(τ)−V^¬m\hat{c}_m = V_\psi(\tau) - \hat{V}_{\neg m} is a difference of value predictions, canceling uniform downward shift; segment rewards are further group-normalized per prompt (Section 4.4), and the gate uses only sign(c^m)\text{sign}(\hat{c}_m). At step 1,000, frozen and refreshed verifiers agree on sign(c^m)\text{sign}(\hat{c}_m) for 88.6% (886/1,000) of interactions, disagreeing mainly on newly solved hard prompts whose prefixes had zero initial successes.

We will specify the verifier's data scale and frozen schedule in Section 4.2 and Appendix A, update Section 6, and add the drift audit and refresh comparison in new Appendix E.

AuthorsOpenReview ↗

Response to W2 (Part 2/4): Monte Carlo Targets and SFT on Successful Rollouts

Direct answer (W2). (i) Monte Carlo noise is not the main bottleneck: doubling continuations from 32 to 64 barely changes verifier accuracy. (ii) Coverage does have a boundary: 253/1,500 (16.9%) of prompts yield all-zero prefix targets, where credit carries no within-problem signal and training falls back to outcome reward. (iii) Testing your suggested SFT-on-successful-rollouts baseline before GRPO raises 5-bench Avg. from 50.9 ± 1.2 to 53.2 ± 1.1, still 9.9 points below our method (63.1 ± 1.2).

1. Target distribution, coverage boundary, and continuation count.

  • Target distribution: Across the 12,000 verifier training prefixes, 3,336 (27.8%) have target 00, 1,152 (9.6%) have target 11, and 7,512 (62.6%) lie strictly in (0,1)(0, 1). On hard prompts (<10% warm-start success; 510/1,500 = 34.0%, 4,080 prefixes), 2,917 (71.5%) have target 00, and 253/1,500 (16.9%) prompts have target 00 on every prefix. On those prompts, c^m≈0\hat{c}_m \approx 0 across steps, so credit acts primarily on frontier problems the policy occasionally solves (shrinking to 147/1,500 = 9.8% by step 750 under verifier refresh, Part 1).
  • Monte Carlo count ablation: At K=32K = 32, worst-case per-prefix SE is ≤0.088\le 0.088, and parametric regression averages noise across similar prefixes. Against 128-continuation reference targets on 1,000 held-out prefixes, returns flatten past K=32K = 32:
Continuations per prefix (KK)81632 (paper)64
Spearman ρ\rho vs. 128-rollout reference0.740.800.840.85
MSE on held-out prefixes0.0480.0370.0310.030

2. SFT on successful self-generated rollouts before GRPO.

Beyond our 3K-demonstration warm start, 146,304 of the 384,000 verifier continuations succeed (38.1%). Taking up to 8 complete correct trajectories per prompt (9,640 total) for 2 epochs of SFT from the warm-start checkpoint before running Agentic GRPO gives:

Method (7B, 3 seeds)AIME24AIME25MATH-500LCB v6GPQA-D5-bench Avg.AIME24 productive (%)
Agentic GRPO (Table 1)43.336.791.238.644.950.9 ± 1.252.3
Successful-rollout SFT → Agentic GRPO46.740.091.840.547.053.2 ± 1.153.8
Ours (Table 1)63.356.795.247.652.563.1 ± 1.282.9

Rejection-sampled SFT helps (+2.3 points) but leaves productive interaction share nearly unchanged (52.3% → 53.8%), because full-trajectory imitation copies redundant calls alongside useful ones without step-level attribution (adding the same SFT stage before our method gives 63.4 ± 1.0, within noise). We will add the target/continuation analysis to Appendix E, the SFT-before-GRPO baseline to Section 5.1 and Table 1, and the zero-success boundary to Section 6.

AuthorsOpenReview ↗

Response to W3 (Part 3/4): Executed Counterfactual Rollouts vs. Equation (4)

Direct answer (W3). We fully agree and ran executed counterfactual rollouts across 612 interactions. Eq. (4) predicts the causal sign on 77.3% (249/322) of non-neutral interactions (global Spearman 0.52), while exhibiting two systematic biases—including the exact alternative-path confounding you identified. Training a replay-calibrated estimator removes both biases (88.2% sign agreement, Spearman 0.71), and retraining our policy with it leaves final accuracy unchanged (63.6 ± 1.0 vs. 63.1 ± 1.2).

1. Executed counterfactual rollout audit & replay-calibrated estimator.

On 150 training trajectories from steps 500 and 1,000 (50 math, 50 code, 50 science; 612 executed interactions), we branch from each pre-request prefix with 64 execute continuations vs. 64 skip continuations (identical to a closed gate: request omitted, no observation appended, later calls unrestricted). Their empirical success gap Δm\Delta_m yields 253 helpful (Δm≥0.15\Delta_m \ge 0.15, 41.3%), 69 harmful (Δm≤−0.15\Delta_m \le -0.15, 11.3%), and 290 neutral (∣Δm∣<0.15|\Delta_m| < 0.15, 47.4%) interactions. We also train a replay-calibrated estimator c^mcal=Vψ(post-obs)−Vψskip(pre-req)∈[−1,1]\hat{c}_m^{\text{cal}} = V_\psi(\text{post-obs}) - V_\psi^{\text{skip}}(\text{pre-req}) \in [-1, 1] by adding a skip-branch target head supervised on 6,000 prefixes × 32 skip rollouts (evaluated below on held-out interactions; split-half ceiling across 32 vs. 32 rollouts is 0.83 global and 0.74 within-trajectory):

| Credit estimator | Sign agreement (322 w/ ∣Δm∣≥0.15|\Delta_m| \ge 0.15) | Global Spearman ρ\rho (all 612) | Within-trajectory Spearman ρ\rho | |---|---|---|---| | Uniform outcome split R(τ)/MR(\tau)/M | 61.8% (199/322) | 0.27 | 0.00 (constant within τ\tau) | | Eq. (4) heuristic (paper) | 77.3% (249/322) | 0.52 | 0.34 | | Replay-calibrated c^mcal\hat{c}_m^{\text{cal}} (new) | 88.2% (284/322) | 0.71 | 0.59 |

2. Diagnostic biases of Eq. (4) & impact on main results.

  • Observed-continuation confounding (your point): A successful realized continuation raises Vψ(τ)V_\psi(\tau) even when the model would also have succeeded without that call. Among neutral interactions (∣Δm∣<0.15|\Delta_m| < 0.15), Eq. (4) assigns positive credit to 52.6% (101/192) in solved trajectories vs. 9.2% (9/98) in failed ones, lowering sign agreement on early calls (interactions 1–2: 73.8% vs. 5+: 82.8%).
  • Over-penalizing large state updates: In the top quartile of wm=1−cos⁡(hm+,hm−)w_m = 1 - \cos(h_m^+, h_m^-), 42.4% (28/66) of genuinely helpful calls receive c^m≤0\hat{c}_m \le 0, vs. 14.4% (27/187) in the remaining quartiles.
  • Retraining with c^mcal\hat{c}_m^{\text{cal}}: Replacing Eq. (4) with c^mcal\hat{c}_m^{\text{cal}} gives 63.6 ± 1.0 5-bench Avg. (vs. 63.1 ± 1.2), raising AIME24 productive calls from 3.4 to 3.7 and cutting redundant calls from 0.7 to 0.6. Thus our main gains do not rely on Eq. (4)'s approximation, though Eq. (4) itself should be framed strictly as a low-cost shaping heuristic.

We will reframe Eq. (4) and add c^mcal\hat{c}_m^{\text{cal}} in Section 4.2 and Table 1, make executed replay the primary validation in Section 5.5 (with human correlation 0.73 as secondary), and report full breakdowns in new Appendix D.

AuthorsOpenReview ↗

Response to W4 (Part 4/4): Error Propagation and Which Mechanism Drives the Gains

Direct answer (W4). We agree that our original ablations left the two pathways of c^m\hat{c}_m coupled. Separating them shows: (i) ~70% of credit's gain comes from policy shaping (ηc^m\eta \hat{c}_m) and ~30% from gate labels (sign(c^m)\text{sign}(\hat{c}_m)); (ii) context-independent sign flips attenuate gradient scale without flipping the gate's optimal action, degrading accuracy smoothly; and (iii) systematic biases explain the pruning of some productive calls (4.2 → 3.4), which executed replay independently audits and corrects (Part 3).

1. 2×2 decoupling of credit's two pathways (4-bench Avg. ± SD, Table 2 setup).

Credit enters segment rewards via ηc^m\eta \hat{c}_m (policy shaping) and gate rewards via sign(c^m)\text{sign}(\hat{c}_m) (gate supervision). In Table 2, "w/o counterfactual credit" set η=0\eta = 0 while still training the gate on sign(c^m)\text{sign}(\hat{c}_m). Adding a coherence-only run (removing both) completes the 2×2 factorial:

4-bench Avg. ± SD (7B, 3 seeds)With gate (trained on sign(c^m)\text{sign}(\hat{c}_m))Without gateGate effect
With policy shaping (η=0.3\eta = 0.3)64.65 ± 0.73 (Full, Table 2)62.63 ± 0.62 (w/o gate, Table 2)+2.03
Without policy shaping (η=0\eta = 0)60.13 ± 0.60 (w/o credit, Table 2)58.40 ± 0.62 (coherence only, new)+1.73
Policy-shaping effect+4.53+4.23Interaction: +0.30

Policy shaping contributes +4.23 to +4.53 points and gating +1.73 to +2.03 points nearly additively (interaction +0.30; Section 5.4's −1.33 tests coherence × gate instead).

2. Random sign-flip propagation vs. systematic bias & online replay audit.

By Proposition 1, the gate's optimal decision depends on whether p=Pr⁡(c^m>0∣context)>1/2p = \Pr(\hat{c}_m > 0 \mid \text{context}) > 1/2. Flipping sign(c^m)\text{sign}(\hat{c}_m) independently with probability ρ<1/2\rho < 1/2 rescales expected gate advantage from 2p−12p - 1 to (2p−1)(1−2ρ)(2p - 1)(1 - 2\rho)—preserving the optimal direction while shrinking step size (and similarly attenuating E[ηc^m]\mathbb{E}[\eta \hat{c}_m]). Applying controlled random sign flips during RL (1 seed per ρ\rho) confirms smooth degradation across Eq. (4)'s 22.7% empirical sign-error rate (73/322, Part 3):

Sign-flip rate ρ\rho0% (Table 2)10%20%30%50% (pure noise)
4-bench Avg. (%)64.6563.862.461.158.5
AIME24 productive share (%)82.981.278.474.666.5

Conversely, context-dependent biases (Part 3) let neutral calls in solved trajectories pass the gate while suppressing high-wmw_m helpful calls—explaining why adding the gate lowered productive calls from 4.2 to 3.4 in Section 5.3 (recovering to 3.7 under c^mcal\hat{c}_m^{\text{cal}}). An online replay audit (100 gated decisions × 32 execute/skip rollouts every 250 steps, <1% extra compute) provides an independent check, showing sign agreement of 74.2% / 78.1% / 76.7% / 75.9% for Eq. (4) vs. 85.5% / 89.1% / 88.3% / 87.9% for c^mcal\hat{c}_m^{\text{cal}} at steps 250 / 500 / 750 / 1,000. We will add these analyses to Section 5.4, Table 2, Appendix B.2, and new Appendix G. Thank you again for your insightful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.