- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes consistency-aware post-training to improve how language models reason from tool feedback. It combines contrastive state regularization, uncertainty-based interaction gating, and approximate counterfactual credit assignment with GRPO. Experiments on five mathematics, coding, and science benchmarks report improved accuracy, better feedback utilization, and fewer redundant interactions.
Critical strengths
- The paper clearly formulates feedback seeking, integration, and attribution as training objectives, with relevant behavioral diagnostics beyond final-answer accuracy.
- Both backbones outperform the corresponding DemyAgent recipe on all five benchmarks. The behavioral analyses also show fewer redundant interactions.
- Component ablations support the training procedure, and the authors carefully distinguish the idealized theoretical statements from the practical approximations.
Critical weaknesses
- Eq. (4) lacks direct validation against actual counterfactual effects, leaving the reliability of interaction-level credit unclear.
- The feedback-integration analysis does not control for differences in requests, observations, and call counts.
- The gate lacks comparisons with simpler alternatives at matched call counts and an ablation of its entropy input.
- No GRPO comparison matches total training compute, including verifier training and target generation.
Questions for authors
- Does Eq. (4) correctly predict the signs and rankings of interaction effects measured through execute/skip rollouts?
- Does coherence training improve responses to identical feedback, particularly when reliable observations contradict an incorrect hypothesis?
- At matched call counts, does the gate outperform random suppression, entropy thresholds, and a hidden-state-only gate?
- Do the gains persist when GRPO receives the same total training budget, including all verifier-related costs?
4 comments
Response to W2 & Q2 (Part 2/4): Identical-Feedback Probe with Contradicting Observations
Direct answer (W2 & Q2). We agree that Section 5.3 did not hold context fixed, and built a controlled identical-feedback probe. Given the exact same problem, prior reasoning, request, and call history, when a reliable observation contradicts an incorrect prior hypothesis, our full model adopts the observation and revises its reasoning on 80.4% (579/720) of cases vs. 72.1% (519/720) without the coherence constraint (+8.3 points, paired 95% CI [3.0, 13.6])—showing does not lock the model into false hypotheses. When the observation itself is artificially corrupted, however, the full model trusts the corrupted output slightly more often, lowering accuracy within noise.
1. Controlled identical-feedback probe (240 contexts × 3 seeds = 720 evaluations).
From held-out MATH validation problems (disjoint from MATH-500), the shared SFT checkpoint generates 240 neutral contexts—each containing a problem, reasoning that explicitly states an expected intermediate value, one Python request computing that value, and the returned observation—edited in one place across three conditions:
- Consistent: correct prior expectation, correct observation.
- Contradicting: prior expectation edited to an incorrect value, observation kept correct (your exact test).
- Corrupted: prior expectation kept correct, observation edited to an incorrect value.
Greedy continuations from all 3 seeds of each 7B method (allowing further calls; models make 0.9, 0.8, 0.7, 0.7 follow-up calls on average) yield:
| Metric (% and count / 720) | Agentic GRPO | DemyAgent recipe | Ours w/o | Ours (full) | Full − w/o [95% CI] |
|---|---|---|---|---|---|
| Consistent: final answer correct | 82.1 (591) | 84.0 (605) | 86.8 (625) | 87.2 (628) | +0.4 [−3.3, 4.1] |
| Contradicting: adopts observation | 64.7 (466) | 68.2 (491) | 72.1 (519) | 80.4 (579) | +8.3 [3.0, 13.6] |
| Contradicting: final answer correct | 55.7 (401) | 60.3 (434) | 64.2 (462) | 71.7 (516) | +7.5 [1.9, 13.1] |
| Corrupted: adopts corrupted value | 31.8 (229) | 34.4 (248) | 37.4 (269) | 41.1 (296) | +3.7 [−1.1, 8.5] |
| Corrupted: final answer correct | 51.4 (370) | 50.6 (364) | 49.7 (358) | 46.8 (337) | −2.9 [−8.2, 2.4] |
(Paired bootstrap CIs over the 240 contexts; vs. Agentic GRPO on Corrupted accuracy, Δ is −4.6 [−10.2, 1.0].) Because Eq. (2) constrains only sub-goal identity in a low-dimensional projection , orthogonal directions remain free to update numerical beliefs—improving belief revision under reliable sandbox outputs while motivating explicit verification for noisy search/API tools.
2. Call-count normalization of Table 8 (AIME 2024).
Weighting Table 8 by accuracy gives overall per-trajectory failure rates of 22.2% (Agentic GRPO) vs. 5.4% (ours), a 4.1× gap that is confounded by trajectory length. Normalizing per 100 executed interactions narrows it to 2.9×, while the rate still rises from 1.4 to 2.4 without :
| Integration failures / 100 executed calls | Agentic GRPO | DemyAgent recipe | Ours w/o | Ours (full) |
|---|---|---|---|---|
| AIME 2024 | 4.0 | 3.1 | 2.4 | 1.4 |
We will make the identical-feedback probe primary in Section 5.3 and new Appendix H, add per-call rates to Table 8, and note corrupted-observation sensitivity in Section 6.
Response to W3 & Q3 (Part 3/4): The Gate vs. Simpler Alternatives at Matched Interaction Counts
Direct answer (W3 & Q3). At matched executed call counts, the learned gate substantially outperforms random suppression (+4.22 points on the 4-benchmark average) and a parameter-free entropy threshold (+2.60 points). Removing the explicit entropy scalar , however, leaves a hidden-state-only gate within 0.47 points of the full gate (64.18 ± 0.73 vs. 64.65 ± 0.73, well within seed noise). Thus, what drives the gain is a state-conditioned gate trained from counterfactual credit rather than the explicit vocabulary-entropy feature; we will revise our framing accordingly.
1. End-to-end training at matched interaction counts (7B, 3 seeds each).
All four gating rules share the exact same RL pipeline (, , and the 10%-step open-gate warm-up), changing only how is chosen at each request:
- Random suppression: executes independently with fixed probability calibrated on the validation set to match the full gate's executed call count.
- Entropy threshold: parameter-free rule , with calibrated on the validation set in the same way.
- Learned gate, hidden state only: removes from Eq. (5) so is trained on via Eq. (6).
- Learned gate, hidden state + entropy (Full, Eq. 5): frozen inference thresholds across the 3 seeds are 0.69, 0.76, 0.74 (mean 0.73).
| Gating rule (7B, 3 seeds) | AIME24 calls / traj | Productive (%) | AIME24 | MATH-500 | LCB v6 | GPQA-D | 4-bench Avg. ± SD |
|---|---|---|---|---|---|---|---|
| No gate (Tables 2 & 6) | 6.2 | 68.5 | 60.0 | 93.8 | 46.2 | 50.5 | 62.63 ± 0.62 |
| Random suppression | 4.0 | 69.4 | 55.6 | 93.0 | 44.1 | 49.0 | 60.43 ± 0.72 |
| Entropy threshold () | 3.9 | 73.1 | 58.9 | 93.6 | 45.5 | 50.2 | 62.05 ± 0.63 |
| Learned gate, only | 4.3 | 81.6 | 62.2 | 95.0 | 47.3 | 52.2 | 64.18 ± 0.73 |
| Learned gate, (Full) | 4.1 | 82.9 | 63.3 | 95.2 | 47.6 | 52.5 | 64.65 ± 0.73 |
2. Key takeaways & inference-time gate swap.
- Why random & entropy rules fall short: Random suppression prunes helpful and redundant calls alike (productive share 68.5% → 69.4%, accuracy −2.20 vs. no gate), confirming gains do not come from merely calling the tool less. Next-token entropy captures local phrasing ambiguity rather than whether computation will raise solve probability (productive share 73.1%, −0.58 vs. no gate).
- Why suffices: Because the next-token logits are a linear readout of , already extracts relevant uncertainty cues directly (Spearman between learned and is only 0.38).
- Inference-time swap (same trained policy): Replacing the full model's gate at test time with call-matched random suppression or an entropy threshold drops 4-bench Avg. from 64.65 to 61.9 and 62.7.
We will reframe "uncertainty-based gate" as "credit-trained state gate (with predictive entropy as an optional auxiliary feature)" in the Abstract, Introduction, and Section 4.3, and add these comparisons to Section 5.4, Table 2, and new Appendix I.
Response to W4 & Q4 (Part 4/4): GRPO at Matched Total Training Compute
Direct answer (W4 & Q4). Yes—the gains persist in full. Counting all offline verifier target generation and verifier training, our method uses 34.1% more total training compute than 1,000-step Agentic GRPO (Section 4.4 counted only the ~21% online RL overhead). Extending Agentic GRPO to match total compute (1,340 steps) raises its 5-benchmark average only from 50.9 ± 1.2 to 51.5 ± 1.0—still 11.6 points below our method (63.1 ± 1.2); extending the DemyAgent recipe to 1,340 steps moves it from 58.3 ± 0.9 to 58.7 ± 1.1 (−4.4 points below ours).
1. Complete end-to-end compute accounting (1,000-step Agentic GRPO = 100 units).
Measured on identical hardware and normalized to the baseline run, our method adds online RL overhead (+20.8%, the three terms Section 4.4 reports as ~5%, <1%, and ~15%) plus previously omitted offline verifier costs (+13.3%):
| Compute stage (7B, per training seed) | Agentic GRPO | Ours |
|---|---|---|
| RL 1,000 steps (rollout execution & policy update) | 100.0 | 100.0 |
| Coherence constraint forward & backward () | 0.0 | 5.0 |
| Gate forward & update () | 0.0 | 0.8 |
| Online verifier forward passes during RL | 0.0 | 15.0 |
| Verifier target generation (384,000 continuations, offline) | 0.0 | 12.4 |
| Verifier training (12,000 prefixes × 2 epochs, offline) | 0.0 | 0.9 |
| Total end-to-end compute | 100.0 | 134.1 |
2. Baselines at matched total compute (7B, 3 seeds each).
Alongside running both baselines to 1,340 RL steps (134 units), we also test spending the offline verifier rollouts directly on 2 epochs of SFT over the 9,640 complete correct trajectories generated during target collection before 1,000 steps of Agentic GRPO:
| Method (7B, 3 seeds) | Total compute (rel.) | AIME24 | AIME25 | MATH-500 | LCB v6 | GPQA-D | 5-bench Avg. ± SD |
|---|---|---|---|---|---|---|---|
| Agentic GRPO, 1,000 steps (Table 1) | 100 | 43.3 | 36.7 | 91.2 | 38.6 | 44.9 | 50.9 ± 1.2 |
| Agentic GRPO, 1,340 steps (compute-matched) | 134 | 44.4 | 37.8 | 91.4 | 39.0 | 44.9 | 51.5 ± 1.0 |
| Successful-rollout SFT → Agentic GRPO (1K steps) | 114 | 46.7 | 40.0 | 91.8 | 40.5 | 47.0 | 53.2 ± 1.1 |
| DemyAgent recipe, 1,000 steps (Table 1) | 100 | 56.7 | 50.0 | 93.0 | 43.2 | 48.5 | 58.3 ± 0.9 |
| DemyAgent recipe, 1,340 steps (compute-matched) | 134 | 57.8 | 50.0 | 93.2 | 43.7 | 48.8 | 58.7 ± 1.1 |
| Ours, 1,000 steps (Table 1) | 134 | 63.3 | 56.7 | 95.2 | 47.6 | 52.5 | 63.1 ± 1.2 |
Extra RL steps add only +0.6 and +0.4 points because both outcome-only baselines have already plateaued past step 800 on the 500-problem held-out MATH validation set (89.2% / 90.6% / 91.0% / 91.2% at steps 600 / 800 / 1,000 / 1,340 for Agentic GRPO; 91.4% / 92.6% / 92.8% / 93.0% for DemyAgent recipe). Using the offline rollouts for SFT helps more (53.2) than extra RL steps (51.5), yet still trails by 9.9 points without step-level credit and selective gating. We will add the full compute ledger, compute-matched baselines, and 0→1,340 validation curves to Section 4.4, Table 1, Section 6, Appendix A, and new Appendix J. Thank you again for your thorough review.
Response to W1 & Q1 (Part 1/4): Equation (4) vs. Execute/Skip Rollouts
Thank you for your rigorous review and falsifiable test designs. We number your four weaknesses/questions W1–W4 and Q1–Q4 in order, pairing each Wk with Qk across four replies (three training seeds, mean ± SD under the paper's protocol unless noted; 5-bench / 4-bench Avg. match Tables 1 and 2).
Direct answer (W1 & Q1). On 612 interactions evaluated via execute/skip rollouts, Eq. (4) matches the empirical sign on 77.3% (249/322) of non-neutral interactions, achieves a 0.52 global Spearman rank correlation, and reaches 0.34 within trajectories. Its errors stem from two diagnosable structural biases; removing them with a replay-calibrated estimator raises performance to 88.2% / 0.71 / 0.59, while retraining the policy leaves main benchmark accuracy unchanged (63.6 ± 1.0 vs. 63.1 ± 1.2).
1. Execute/skip rollout evaluation & domain breakdown.
Across 150 training trajectories from steps 500 and 1,000 (50 math, 50 code, 50 science; 612 executed interactions), we sample 64 execute vs. 64 skip continuations from each pre-request prefix (skip matches bm=0: request unexecuted, no observation inserted, later calls allowed), defining Δm as their solve-rate difference: 253 helpful (Δm≥0.15), 69 harmful (Δm≤−0.15), and 290 neutral (∣Δm∣<0.15). Within-trajectory metrics cover the 130 trajectories with ≥3 interactions (random top-1 hit rate: 30/130 = 23.1%; 32-vs.-32 split-half ceiling: 0.83 global, 0.74 within-trajectory):
| Credit estimator | Sign agreement (322 w/ ∣Δm∣≥0.15) | Global Spearman ρ (all 612) | Within-trajectory Spearman ρ | Top-credited matches argmaxΔm (130 trajs) | |---|---|---|---|---| | Uniform split R(τ)/M | 61.8% (199/322) | 0.27 | 0.00 (constant) | 23.1% (random tie-break) | | Eq. (4) heuristic (paper) | 77.3% (249/322) | 0.52 | 0.34 | 46.2% (60/130) | | Replay-calibrated c^mcal (new) | 88.2% (284/322) | 0.71 | 0.59 | 65.4% (85/130) |
By domain, Eq. (4) sign agreement is 79.5% (89/112, math), 77.4% (96/124, code), and 74.4% (64/86, science), rising to 89.3%, 88.7%, and 86.0% after calibration.
2. Structural biases & replay-calibrated retraining.
[0,1]clipping gives 250/322 = 77.6%).