acceptodds
Withdrawn from ICLR 2027

The Real Bottleneck Is Not Reward Noise: Gradient Signal Degeneracy in Long-Horizon Agent RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards depends on an automatic verifier, and on single-turn tasks a considerable amount of verifier error has proven tolerable: noise rates of up to 15 percent leave peak accuracy within two points of a clean baseline. However, a long-horizon trajectory receives a single terminal reward while giving a faulty verifier dozens of opportunities to be wrong about it, so that tolerating the same rate per trajectory would demand a per-turn rate an order of magnitude lower, and long-horizon training should on this reasoning be the more fragile of the two. Contrary to that expectation, we measure no accuracy cost in a multi-turn search agent at false-positive rates up to 0.5, across a fourfold range of realized horizons and a sixfold range of training length. In stark contrast to what accuracy reports, however, the fraction of sampled groups contributing no reward-driven gradient rises from 0.57 to 1.00 over that same range, so that a policy whose batch contains no such gradient anywhere remains indistinguishable, on accuracy, from one trained against a clean verifier. Subsequently, we trace the discrepancy to verifier error that attaches to the task rather than to the answer and therefore removes whole groups at once, and we obtain a parameter-free account of the collapse which transfers across two environments, three model families, false negatives and KL regularization. Our work shows that a verifier cannot be certified by the performance of the policy it trains, and that the fraction of groups producing no reward-driven gradient, obtainable at no cost during training, should be monitored in its place.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Long-Horizon Agent, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_V9ywOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the effect of verifier errors on long-horizon RL with verifiable rewards. Contrary to the expectation that verifier errors should accumulate over long trajectories, the authors find that policy accuracy remains largely unchanged even under substantial false-positive rates. They attribute this apparent robustness to task-correlated verifier errors in group-relative policy optimization: such errors often make all samples in a group receive identical rewards, causing the entire group to contribute zero reward-driven gradient rather than introducing an incorrect gradient. The paper therefore argues that policy accuracy can be misleading as a measure of verifier quality, and proposes the fraction of degenerate groups as a more direct indicator of the effective training signal.

Critical strengths

The finding that task accuracy remains largely unchanged as the false-positive rate increases is interesting, and the underlying reasons warrant further investigation.

Critical weaknesses
  • The claim that a group’s advantage vanishes in the presence of verifier errors lacks sufficient theoretical justification and comprehensive empirical validation.

  • This paper lacks sufficient background statements, and the method section should be more focused on the main contributions and written in a more formal style.

  • The paper appears to have been written in haste. The authors should better articulate the core contributions and improve the manuscript's logical organization and overall clarity to make it easier to follow.

Questions for authors

Could the authors clarify the scope and limitations of the conclusions presented in the paper?

Reviewer_bGAPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how errors in the verifier affect RLVR methods. The experiments show that verifier error does not affect the accuracy of the trained model. Instead of measuring the accuracy, the paper proposes measuring the number of groups in GRPO which get the same advantage (contribute no gradient).

Critical strengths

None

Critical weaknesses
  1. Writing quality. I found it hard to understand the contributions and the claims the paper is making. It is written with lots of auxiliary clauses and convoluted sentences. Some examples: (with suggested improvements):

    • "However, a long-horizon trajectory receives a single terminal reward while giving a faulty verifier dozens of opportunities to be wrong about it, so that tolerating the same rate per trajectory would demand a per-turn rate an order of magnitude lower, and long-horizon training should on this reasoning be the more fragile of the two." -> "However, a long-horizon trajectory receives a single reward, meaning the verifier has dozens of opportunities to do a mistake. For example, a 15% trajectory error rate would mean an approx. 1% per-step error rate within 16 steps. In practice the per-step error rate is not small [citation needed?], and should lead to fragile long-horizon training."
    • "Contrary to that expectation, we measure no accuracy cost in a multi-turn search agent at false-positive rates up to 0.5, across a fourfold range of realized horizons and a sixfold range of training length." -> "Surprisingly, it does not lead to fragile training. We find that accuracy does not drop under a verifier with a false-positive rate of 0.5, which was tested on 4 different horizon lengths and 6 different tranining lengths."

    Unexplained terms:

    • Line 25-26: "verifier error that attaches to the task" -- what does attaching to the task mean?
    • What is the definition of ff and d0d_0? They are referred to in section 1, but only ff is introduced later in section 3.2.
    • Line 107+: what is a held-out condition?
    • Lines 207-208: is f meant to be the false-positive rate? What is natural? That doesn't seem to match with what's presented in Figure 2. Perhaps ff is the per-trajectory false-positive rate, and the other rate in Figure 2 is the per-step false-positive rate? That is my current guess.

    Unclear writing:

    • Line 146: delay, plateau, collapse in what? Delay in training? The loss/accuracy?
    • Line 146: what is an incomplete verifier?
    • Line 167: "Two gaps remain.", although it's not presented what they are.
    • Line 168: missing reference?
    • Figure 3: it is unclear what the figure is meant to represent. Is it the number of non-zero gradients from each of the GRPO steps? What kind of statement does it support? The lines look roughly similar, and it's hard to draw any conclusion without statistical confidence. Why aren't the left and middle parts superimposed so that comparison is easier?

    Mistakes:

    • Figure 1.1 Imperfect verifier should likely be a box around Reward model only, not both reward model and reference model; A_2 should likely be -0.5.
    • Figure 1.2, r_1 should be r_i; Figure 1.2(b) r_i should likely be 1, 1, 1, 1 (not 1, 0, 1, 0).

    The issues illustrated by these examples interfere with my ability to properly understand this paper and its contributions; there are likely entire sections I did not understand.

  2. Study details. There are many details missing from the paper. The abstract mentions testing over three model families, although there is no mention of what the models are. Neither architecture, nor training hyperparameters.

    The results are across many seeds, although it is unclear what is the variance among seeds. Is it possible to have confidence intervals for the presented results? Do the confidence intervals overlap? In Figure 3 it is mentioned, that "Faint markers are individual seeds", but no faint markers are presented.

    What is the system prompt used, mentioned in line 216?

    The corruption model is not described clearly enough. Is the verifier false-positive noise sampled independently for each trajectory (based on the simulated FP rate)? Reading further through section 4.4 it sounds like the false-positive corruption is marked for multiple trajectories (a GRPO group). Is this a realistic scenario?

Questions for authors
  1. What is the corruption model affecting the FP rate? (perhaps an algorithm showing how it works would be interpretable?)
  2. What are the study's details? (model architectures, pretrained checkpoints, training hyperparameters, confidence intervals across seeds, etc.)
Reviewer_jwxeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how marking wrong answers as correct affects GRPO training of an AI agent that answers questions through multiple search steps. Experiments use Qwen3.5-4B with LoRA on HotpotQA with a BM25 search tool, and rewards are assigned to final answers using substring matching while accuracy is evaluated using stricter exact matching. Verifier errors are injected in a controlled way, with target false-positive rates of up to 0.50.5 across all judgments. In each epoch, questions are selected pseudo-randomly, and every wrong answer to them is marked correct. All answers in an affected group then receive the same reward, so the group contributes no reward-driven gradient. The minimum number of retrievals is also varied, giving an average of 1.91.9-7.37.3 calls per trajectory, to examine how the effects depend on horizon. In the main retrieval sweep, the degenerate-group rate rises from 0.570.57 to 11 without a detectable reduction in accuracy, while no substantial amplification with horizon is reported. The expression d0+f(1−d0)d_0 + f (1 - d_0) (ff being the reported realized corruption rate), which is parameter-free, predicts a held-out condition within 0.0020.002 and is examined with other models, 2WikiMultihopQA, false negatives, and KL regularization. At a matched rate, answer-level corruption reduces accuracy and departs from the prediction by 0.160.16. In a separate mathematics experiment, only 29%29\% of the clean-training improvement survives at the same target rate of 0.50.5. A simulation shows that resampling an affected question cannot restore reward variation as long as its corruption decision remains fixed, whereas replacing questions can help, with a predicted sampling cost of 11−f\frac{1}{1 - f} times the clean-verifier cost. The paper, accordingly, argues that accuracy alone cannot establish verifier quality and recommends that the degenerate-group rate be monitored.

Critical strengths

CS1. The paper contains a useful demonstration that accuracy can miss lost training signal. In fact, were accuracy alone to be examined, the loss of reward-driven guidance could go unnoticed. Logging the fraction of zero-advantage groups is inexpensive, while the stricter evaluation verifier strengthens the accuracy comparison. The reviewer sees this as a critical strength because practitioners can identify potentially unproductive training despite stable performance.

CS2. The paper suggests a transparent intervention with an understandable mechanism. Whole-question corruption makes the zero-advantage mechanism explicit and supports a simple fixed-policy counting argument. The reported held-out condition, group-size controls, and additional settings provide useful checks, while the acknowledged limitations help define the scope. This makes the explanation and its empirical support easier to assess.

Critical weaknesses

The reviewer has some serious concerns about the paper, the two most important of which are given first, accompanied by detailed explanations, followed by some minor weaknesses under the umbrella of "Additional Weaknesses."

CW1. The first critical weakness is the fact that the intervention supports a substantially narrower conclusion than the paper's explanation of realistic verifier errors. This weakness is detailed hereafter.

  • The injected mechanism accepts every incorrect answer to a selected question. Ordinary task-dependent verifier failures, however, need not do this; a weak test suite can accept one class of incorrect programs while rejecting another, and a substring verifier can accept an incorrect answer containing the reference while rejecting another incorrect answer. Ergo, a group with true rewards (1,0,0)(1,0,0) can become (1,1,0)(1,1,0), retaining reward variation. Accordingly, it follows that task dependence alone does not imply whole-group saturation.
  • The equation in (1) has a sound interpretation only under explicit assumptions. For a fixed policy π\pi, let d(π)d(\pi) be its degenerate-group rate without injected errors, and let dmasked(π)d_{\mathrm{masked}}(\pi) be the rate after each question is independently switched to all-accept with probability qq. Then, we have dmasked(π)=d(π)+q [1−d(π)],d_{\mathrm{masked}}(\pi) = d(\pi) + q\,[1 - d(\pi)], which is the equation in (1) with d(π)d(\pi) in place of d0d_0 and qq in place of ff. However, applying this expression to a noisily trained policy requires its underlying degeneracy d(πnoisy)d(\pi_{\mathrm{noisy}}). The paper instead uses the degeneracy of the separately trained clean run, d0d_0, which is an empirical approximation. The paper recognizes baseline constancy as substantive, but removing groups does not establish it, nor does it guarantee an unharmed policy. At the same time, removing contributions can also change the direction of the realized batch gradient; e.g., contributions (1,0)(1,0) and (0,1)(0,1) sum to (1,1)(1,1), whilst removing the first leaves (0,1)(0,1). Expected direction preservation under uniform thinning at a fixed policy is a more limited statement than the paper's claim that such error "never redirects" the update.
  • Another concern is related to the horizon experiment. To be more precise, corruption is decided once per question and epoch, independently of the number of retrievals. Therefore, no additional corruption opportunities are introduced by increasing the horizon. This tests question-level saturation at different trajectory lengths, yet does not test the motivating mechanism in which each additional turn introduces a fresh opportunity for verifier error.
  • The contribution should also be positioned more narrowly against prior work. In fact, DAPO already discusses uniform-reward groups producing zero policy gradients and filters such groups, and He et al. already discuss collapse remaining invisible to conventional accuracy and loss curves. Whilst the paper acknowledges these works and states two gaps (Section 2), the second rests on a misreading: it says He et al. (2026) "find collapse visible in final accuracy under a clean verifier," whereas He et al. describe collapse as invisible in accuracy and loss curves. What remains new is treating the false-positive rate as the independent variable with the collapse rate as the dependent one, which is narrower than the paper presents.

CW2. The experimental reporting does not adequately substantiate the headline accuracy claim or the crucial larger-gain comparison:

  • The claim "no accuracy cost" is stronger than supported by the evidence. Section 4.2 and Appendix A acknowledge that effects below approximately five percentage points can remain undetected. Thus, the supported conclusion is that no reduction was detected at the experiments' resolution. If we want to assess how much improvement was available to lose, initial-policy accuracies and complete learning curves are needed. Also, the supplied 300-step trajectory plot compares clean training with target rate 0.100.10; Table 2 reports a realized rate of only 0.0100.010 for that target in the central reporting window, which does not demonstrate high-corruption robustness throughout a 300-step run.
  • The mathematics experiment is particularly important because it reportedly distinguishes the retrieval null result from substantial lost learning. Nonetheless, its dataset, subset-selection procedure, training/evaluation sizes, complete configuration, seed count, initial and final accuracies, and uncertainty are not adequately specified. Appendix C contains no accompanying experimental description, and the claimed gain-comparison plot appears to be absent. In Section 4.6, the sentence after the mathematics result states that "evaluated on 300 unseen questions the surviving fraction is 60.8%60.8\%, close to the 60%60\% measured during training." Since the mathematics result is 29%29\%, and 60%60\% is the retrieval figure given later in the same paragraph, it is unclear which experiment the 60.8%60.8\% refers to.
  • Importantly, definitions and evaluation endpoints need repair. Section 3.2 defines ff as Pr⁡(accept∣wrong)\Pr(\text{accept} \mid \text{wrong}), but Figure 2 reports f=0f = 0 in the clean condition despite acknowledged natural false positives. An underlying conditional false-positive rate aa and an independent injection probability qq give total conditional false-positive rate Pr⁡(accept∣wrong)=a+(1−a)q\Pr(\text{accept} \mid \text{wrong}) = a + (1 - a) q; this differs from the masking probability used in the counting identity. Moreover, Section 3.5 specifies the final ten steps, whereas Appendix E and Table 4 use steps 2020-4040 for the central results.

Additional Weaknesses.

  • In Figure 1, the top panel assigns an advantage of 0.50.5 to every sample, including the one with reward 00, although its centred advantage must be negative; and panel (b) shows rewards (1,0,1,0)(1,0,1,0) with zero variance and zero advantages, whereas under the all-accept injection they would be (1,1,1,1)(1,1,1,1).
  • Figures 3–6 do not match their captions, and the plot comparing training gains in mathematics and retrieval appears to be missing.
  • Section 4.3 states that the degenerate-group rate "moves sharply between rates of 0.200.20 and 0.400.40, from 0.6850.685 to 0.9030.903 to 1.0001.000"; however, 1.0001.000 is the value at 0.500.50 (Tables 2 and 4), and the value at 0.300.30 (0.8000.800) is omitted.
  • Line 168 contains an unresolved citation ("?").
Questions for authors
  1. What exactly does the reported ff measure? Please provide the corresponding counts and denominators, including natural errors, for each central condition.
  2. What are the initial-policy accuracies and complete clean/noisy learning curves on clearly identified evaluation questions? In particular, which 300-step runs maintain substantial realized corruption, and what uncertainty intervals support the accuracy comparison?
  3. Please provide the dataset and subset-selection rule, train/evaluation construction, configuration, seeds, initial/clean/noisy accuracies, realized corruption rates, and uncertainty behind the 29%29\% result. Which experiment does the subsequent 60.8%60.8\% unseen-question result describe?
  4. What is the exact implemented GRPO objective, including advantage normalization and loss aggregation? For the answer-keyed condition, identical answers within a group share the same hash and therefore receive the same decision; how often does this occur, and does the condition then provide the per-sample independent noise described in Section 3.2?
  5. Has the underlying uncorrupted reward degeneracy on groups sampled from the noisily trained policies been measured? Is there evidence that naturally occurring verifier errors produce the whole-group saturation assumed by the explanation?
  6. Please repair the final-ten-step and steps-20–40 endpoints, identify all models and their families, and supply the correctly matched figures and missing experimental descriptions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.