acceptodds
Under review as a conference paper at ICLR 2027

Resolvable-Effect Accounting for Tool-Calling Agents: Reward Bias and Evaluation Resolution

Abstract

Small gains from training tool-calling language models are difficult to assess when the training signal is affected by filtering and evaluation outcomes vary across runs. We introduce Resolvable-Effect Accounting (REA), a framework for examining reward logging and evaluation resolution, with candidate coverage and gradient statistics as supporting diagnostics. We show that, under group filtering in Group Relative Policy Optimization (GRPO), the retained-group reward is a conditional mean that need not move with the underlying success rate; logging the unfiltered mean avoids this selection effect for a fixed prompt distribution. In twelve-run HotpotQA evaluation panels, an estimated 69% of single-run variance is associated with run-level variation under the tested seed protocol, and same-seed pairing reduces comparison uncertainty when outcomes covary across conditions. In a Qwen3.5-2B case study, the observed checkpoint differences after 240 GRPO updates and 20 self-distillation updates are and percentage points, respectively, with 95% intervals that include zero. These results motivate explicit reporting of reward-selection rules and evaluation uncertainty. REA quantities can inform evaluation planning before optimizer updates, while their prospective value for training decisions remains to be assessed across matched configurations and additional settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, tool calling, reinforcement learning, on-policy distillation, GRPO, gradient signal-to-noise, evaluation methodology, empirical study

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_xtf6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces REA, a framework for examining reward logging and evaluation resolution in tool-calling agent training. It shows that averaging rewards only over retained GRPO groups with nonzero reward variance changes the quantity being estimated and can give a misleading impression of training progress. It also separates evaluation variability into question-level and run-level components to examine minimum detectable effects and the benefits of paired evaluation. Candidate coverage, gradient statistics, and conditional task-effect analyses provide additional diagnostics. A case study using Qwen3.5-2B on HotpotQA reports small checkpoint differences that remain statistically unresolved. The paper carefully distinguishes these observations from causal training effects and predictions of future training outcomes.

Critical strengths
  1. The analysis of retained-group rewards identifies a practical measurement problem and offers a simple remedy: report the unfiltered mean under a fixed prompt distribution. It clearly explains why the logged reward can diverge from the underlying success rate, while recognizing that this discrepancy alone does not establish optimizer failure.

  2. The repeated evaluations provide a more informative assessment than a single performance comparison. The paper clearly acknowledges the lack of repeated training runs, differences between configurations, confounding between seed and process effects, and the conditional nature of the reported intervals. The absence of a conclusive training improvement is not, by itself, a weakness.

Critical weaknesses
  1. The main projection of paired evaluation resolution relies on an idealized assumption whose applicability to the observed comparisons remains unclear. The curve from 0.44 to 0.12 percentage points assumes that a common run-level offset cancels under pairing, but the reported correlations between conditions do not establish complete cancellation. Although the appendix retains the covariance term and the text acknowledges this limitation, the main figure should also show projections based on the observed residual run-level variance, with uncertainty reflecting the limited sample of twelve runs. Without this analysis, the practical implications for evaluation sample size remain difficult to assess.

  2. The empirical validation is weakened by configuration mismatches. The gradient diagnostic uses a KL coefficient of 0 and stricter tool definitions, whereas the GRPO run uses a KL coefficient of 0.04. The base and trained evaluation panels also differ in their tool-schema descriptions. Although acknowledged, these differences limit the relevance of the gradient measurements to the actual training run and prevent the checkpoint comparisons from isolating training effects. A configuration-matched diagnostic panel and checkpoint comparison would provide stronger empirical support.

Questions for authors
  1. How do the projected paired minimum detectable effect curves change when residual run-level variance is estimated from the actual comparisons of the base model with GRPO and OPSD? Please report uncertainty or sensitivity analyses for these projections and distinguish them clearly from the idealized common-offset case.

  2. Could you recompute the gradient diagnostic using the actual GRPO training configuration and compare the base and trained checkpoints under identical evaluation settings? Would the main interpretations remain unchanged?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xtf6 — residual run-level variance and configuration-matched diagnostics

We thank the reviewer for the specific and constructive questions. We agree that the common-offset curve is idealized, and we have computed the residual-variance sensitivity from the existing 12 paired runs. For each condition pair we take the per-seed difference of run means. The observed paired sd is 0.347 pp for base-to-GRPO and 0.397 pp for base-to-OPSD. The estimated independent-binomial question-level component of a paired difference, using the per-question unbiased binomial variances in both panels, is about 0.496 pp. The method-of-moments residual run-level sd is therefore negative and truncated to zero in both panels. That is not an assumption of complete cancellation: it says the observed paired variance is at or below the independent question-level component, and with 12 runs the residual is bounded rather than identified. A chi-square interval on the paired variance (11 df) gives a 95% interval on the paired sd of [0.246, 0.589] pp for GRPO and [0.281, 0.674] pp for OPSD. Subtracting the question component gives a 95% upper bound on the residual run-level sd of 0.319 pp and 0.457 pp, with a lower bound of zero.

The point estimate of the residual is zero, so the central paired curve coincides with the current common-offset projection; what changes is the uncertainty band. At 7,405 questions, the paired MDE is 0.44 pp under the planning proxy, 0.52 pp at the GRPO upper bound, and 0.60 pp at the OPSD upper bound. At 100,000 questions, it is 0.12 pp under the idealized projection but 0.31 pp and 0.42 pp at the two upper bounds. So the qualitative claim—more questions alone cannot remove a run-level component, and pairing reduces but does not prove elimination—survives, while the 0.12 pp figure is optimistic if residual run-level variance is at the top of its 12-run interval. The current Figure 3(a) plots the idealized case; the sensitivity above is the relevant planning range. We also note that the observed paired sd is below the independent-binomial component, which suggests same-seed pairing also correlates question-level noise; the additive decomposition is an approximation.

On the configuration mismatch, the reviewer is right, and we do not defend the current gradient panel as matched. It used β_KL=0 and strict tool definitions, while the GRPO run used β_KL=0.04 and the training schema. The diagnostic was exploratory, its jackknife interval already includes zero, and it is not part of the two core claims. We cannot run a matched panel with the resources available for this response, so we treat the gradient result as exploratory and inconclusive; the main interpretations do not depend on it. The core reward-bias and resolution results come from the training run’s own reward logs and from within-panel variance decompositions, not from the gradient panel. The same applies to the checkpoint comparison: the panels share the question set, scorer, seeds, and agent limits, but differ in resource paths, model-config fields, and the tool-schema query description. The +0.098/+0.032 estimates should therefore be read as descriptive checkpoint-panel differences, not as configuration-isolated causal effects.

Reviewer_o78xOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Resolvable-Effect Accounting (REA), a framework for analyzing reward logging bias and evaluation uncertainty in RL for tool-calling agents. The framework focuses on two main issues: (1) selection bias introduced by filtering zero-variance rollout groups in GRPO, which can cause logged rewards to diverge from actual success rates, and (2) run-level variance in evaluation, which limits the ability to detect small performance improvements. REA further incorporates candidate coverage, gradient signal-to-noise analysis, and per-question effect variance as supporting diagnostics. Experiments are conducted using Qwen3.5-2B on HotpotQA, with 240 GRPO updates, 20 on-policy self-distillation (OPSD) updates, and repeated evaluations across 12 random seeds. Results demonstrate substantial discrepancies between filtered and unfiltered reward statistics and show that paired evaluation can reduce uncertainty. However, the observed performance improvements from GRPO and OPSD are small and statistically insignificant.

Critical strengths
  • The paper provides a systematic analysis of reward logging bias and evaluation uncertainty in agentic RL. The authors carefully distinguish observed results from statistically significant improvements and acknowledge the limitations of their experimental setup.
Critical weaknesses
  • The main contribution is largely applying established statistical analysis to agentic RL. While these analyses are valid and potentially useful, the proposed REA framework does not introduce substantially new methodological insights. More importantly, the paper does not demonstrate how these diagnostics can improve practical training decisions, such as selecting training configurations or allocating compute resources.

  • The experiments are restricted to a single model (Qwen3.5-2B) and task setting (HotpotQA), without independent training repetitions. Furthermore, the evaluation configurations are not fully consistent across checkpoints, and the exploratory gradient analysis uses different settings from the actual GRPO training. These limitations make it difficult to establish whether the reported diagnostics generalize to other models, tasks, and training configurations, or provide reliable guidance for broader agentic RL research.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer o78x — measurement accounting with explicit decision arithmetic

We appreciate the careful summary and agree with much of it. The individual ingredients—selection bias, variance decomposition, MDE—are established, and we do not claim otherwise. Our contribution is the accounting: a specific, free logging error in agentic RL that can reverse sign, and a separation of evaluation variance into a question component that shrinks with N and a run component that does not. The novelty is in the combination and the decision consequences, not in the statistical primitives.

The practical-utility concern is fair, and we should have made the decision consequences explicit. First, reward monitoring: switching from the filtered to the unfiltered mean changes the reading from 51.48% (apparently flat) to 35.00% (no reliable trend, +1.96 pp with 95% CI [−2.60, +6.42]) at zero cost; that is a different decision about whether to continue the run. Second, compute allocation: at 7,405 questions and 12 runs, unpaired evaluation has an MDE of 0.80 pp, while same-seed pairing gives 0.44 pp under the planning proxy—equivalent to about 40 unpaired runs for the same resolution, roughly a 3.3× saving. For a 0.2 pp target, 12 paired runs are still underpowered (about 59 runs would be needed), so the diagnostic says either add runs or do not expect detection. This is not a controlled study of researcher decisions; it is the decision arithmetic that follows from the measured variance components, and we present it as such. The same logic gives a configuration-selection rule: same-seed pairing is defensible for checkpoints sharing weight ancestry (r=0.84) but not across a protocol change (r=0.10), where independent panels are needed.

We agree on the empirical limits: one model, one task, no independent training repetitions, and manifests that are not fully configuration-identical. These are stated in the paper’s limitations section, and we restate them here rather than treating them as settled. The core claims do not depend on the cross-panel comparison: the reward-bias evidence comes from the GRPO run’s own rollout groups, and the resolution evidence comes from within each 12-seed panel. We cannot add a second model or task, and we do not claim generality from this case study. The gradient diagnostic is supporting and already inconclusive; a matched rerun is not available within the resources of this response, so we treat it as exploratory and do not rely on it for the core claims. The internal controls show the pipeline is not insensitive by construction: the ReAct protocol change is detected at −10.25 pp (23× the planning proxy), and the voting control predicts the observed +3.08 pp within 0.10 pp. The limitation is specifically sub-pp sensitivity, which is the regime of the training deltas.

Reviewer_Za9eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Resolvable-Effect Accounting (REA), a framework for auditing reward logging and evaluation uncertainty in tool-calling agents. It analyzes the impact of filtering zero-variance GRPO groups and the impact of run-level variance on evaluation resolution. Experiments on Qwen3.5-2B and HotpotQA illustrate these effects.

Critical strengths

The paper studies a practically relevant problem: distinguishing genuine training improvements from misleading reward statistics and evaluation noise.

Critical weaknesses
  1. The practical relevance of the reward-filtering analysis is unclear. Discarding zero-variance groups is an implementation-dependent choice rather than an inherent requirement of GRPO. Analyzing this specialized algorithm only does not show prevalence of the proposed method on other algorithms. Also, the paper does not demonstrate advantages beyond simply reporting original rewards.
  2. The validation of the proposed framework is insufficient. Experiments are limited to one model and dataset. The experiment is also limited to the search tool only, without analyzing other agent tools. Also, the paper does not demonstrate that REA leads to better training decisions or evaluation analysis compared with conventional multi-seed evaluation and reward monitoring.
Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer Za9e — reward filtering is a logging practice, not a GRPO requirement

We thank the reviewer for the direct assessment. We agree with the central point: discarding zero-variance groups is not part of the GRPO definition, and the paper should not have phrased it as though it were. Our claim is conditional on a logging practice, not on the algorithm. To be explicit: in the implementation we audit, as in several released GRPO recipes (for example DAPO’s dynamic sampling and slime’s check_reward_nonzero_std filter), zero-variance groups are discarded before the logged mean is computed. The same algebra applies to any group-relative method that logs over a variance-selected subset: the logged mean estimates E[q | retained], not E[q]. The object of analysis is the filter, not GRPO itself.

We also want to be clear about what this adds beyond “report the original reward.” Reporting the unfiltered mean is exactly our recommendation, and it costs nothing. The analysis is needed because the filtered statistic is not a constant-offset version of the original: it is a different functional of the success-probability distribution, and its response to a fixed +1 pp true improvement ranges from −3.93 to +20.77 pp across four plausible shift families. A team cannot correct this after the fact by subtracting a fixed bias. In our run the filtered reading is 51.48% and nearly flat across four 60-update segments, while the unfiltered reading is 35.00% and shows no reliable upward trend (+1.96 pp, 95% CI [−2.60, +6.42]). A monitor seeing only the first number would likely conclude the run is stable; the second says the training objective has not reliably improved. That is the decision difference, and it is not a claim that REA replaces standard evaluation.

On scope, we agree that one model, one dataset, and one tool cannot establish prevalence. The two core diagnostics are functionals of rollout logs and repeated evaluation panels, not properties of the search tool, so the case study validates the measurement rather than the generality. We did not run a second tool and do not imply otherwise. A replication needs group-level binary rewards for the logging diagnostic and repeated seeds on a fixed question set for the resolution diagnostic. On presentation, we agree that the current four-part organization makes the two core claims hard to see; the two claims are the filtered-mean bias and the run-level resolution floor, and candidate coverage and gradient signal are supporting evidence only.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.