- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces REA, a framework for examining reward logging and evaluation resolution in tool-calling agent training. It shows that averaging rewards only over retained GRPO groups with nonzero reward variance changes the quantity being estimated and can give a misleading impression of training progress. It also separates evaluation variability into question-level and run-level components to examine minimum detectable effects and the benefits of paired evaluation. Candidate coverage, gradient statistics, and conditional task-effect analyses provide additional diagnostics. A case study using Qwen3.5-2B on HotpotQA reports small checkpoint differences that remain statistically unresolved. The paper carefully distinguishes these observations from causal training effects and predictions of future training outcomes.
Critical strengths
-
The analysis of retained-group rewards identifies a practical measurement problem and offers a simple remedy: report the unfiltered mean under a fixed prompt distribution. It clearly explains why the logged reward can diverge from the underlying success rate, while recognizing that this discrepancy alone does not establish optimizer failure.
-
The repeated evaluations provide a more informative assessment than a single performance comparison. The paper clearly acknowledges the lack of repeated training runs, differences between configurations, confounding between seed and process effects, and the conditional nature of the reported intervals. The absence of a conclusive training improvement is not, by itself, a weakness.
Critical weaknesses
-
The main projection of paired evaluation resolution relies on an idealized assumption whose applicability to the observed comparisons remains unclear. The curve from 0.44 to 0.12 percentage points assumes that a common run-level offset cancels under pairing, but the reported correlations between conditions do not establish complete cancellation. Although the appendix retains the covariance term and the text acknowledges this limitation, the main figure should also show projections based on the observed residual run-level variance, with uncertainty reflecting the limited sample of twelve runs. Without this analysis, the practical implications for evaluation sample size remain difficult to assess.
-
The empirical validation is weakened by configuration mismatches. The gradient diagnostic uses a KL coefficient of 0 and stricter tool definitions, whereas the GRPO run uses a KL coefficient of 0.04. The base and trained evaluation panels also differ in their tool-schema descriptions. Although acknowledged, these differences limit the relevance of the gradient measurements to the actual training run and prevent the checkpoint comparisons from isolating training effects. A configuration-matched diagnostic panel and checkpoint comparison would provide stronger empirical support.
Questions for authors
-
How do the projected paired minimum detectable effect curves change when residual run-level variance is estimated from the actual comparisons of the base model with GRPO and OPSD? Please report uncertainty or sensitivity analyses for these projections and distinguish them clearly from the idealized common-offset case.
-
Could you recompute the gradient diagnostic using the actual GRPO training configuration and compare the base and trained checkpoints under identical evaluation settings? Would the main interpretations remain unchanged?
Response to Reviewer xtf6 — residual run-level variance and configuration-matched diagnostics
We thank the reviewer for the specific and constructive questions. We agree that the common-offset curve is idealized, and we have computed the residual-variance sensitivity from the existing 12 paired runs. For each condition pair we take the per-seed difference of run means. The observed paired sd is 0.347 pp for base-to-GRPO and 0.397 pp for base-to-OPSD. The estimated independent-binomial question-level component of a paired difference, using the per-question unbiased binomial variances in both panels, is about 0.496 pp. The method-of-moments residual run-level sd is therefore negative and truncated to zero in both panels. That is not an assumption of complete cancellation: it says the observed paired variance is at or below the independent question-level component, and with 12 runs the residual is bounded rather than identified. A chi-square interval on the paired variance (11 df) gives a 95% interval on the paired sd of [0.246, 0.589] pp for GRPO and [0.281, 0.674] pp for OPSD. Subtracting the question component gives a 95% upper bound on the residual run-level sd of 0.319 pp and 0.457 pp, with a lower bound of zero.
The point estimate of the residual is zero, so the central paired curve coincides with the current common-offset projection; what changes is the uncertainty band. At 7,405 questions, the paired MDE is 0.44 pp under the planning proxy, 0.52 pp at the GRPO upper bound, and 0.60 pp at the OPSD upper bound. At 100,000 questions, it is 0.12 pp under the idealized projection but 0.31 pp and 0.42 pp at the two upper bounds. So the qualitative claim—more questions alone cannot remove a run-level component, and pairing reduces but does not prove elimination—survives, while the 0.12 pp figure is optimistic if residual run-level variance is at the top of its 12-run interval. The current Figure 3(a) plots the idealized case; the sensitivity above is the relevant planning range. We also note that the observed paired sd is below the independent-binomial component, which suggests same-seed pairing also correlates question-level noise; the additive decomposition is an approximation.
On the configuration mismatch, the reviewer is right, and we do not defend the current gradient panel as matched. It used β_KL=0 and strict tool definitions, while the GRPO run used β_KL=0.04 and the training schema. The diagnostic was exploratory, its jackknife interval already includes zero, and it is not part of the two core claims. We cannot run a matched panel with the resources available for this response, so we treat the gradient result as exploratory and inconclusive; the main interpretations do not depend on it. The core reward-bias and resolution results come from the training run’s own reward logs and from within-panel variance decompositions, not from the gradient panel. The same applies to the checkpoint comparison: the panels share the question set, scorer, seeds, and agent limits, but differ in resource paths, model-config fields, and the tool-schema query description. The +0.098/+0.032 estimates should therefore be read as descriptive checkpoint-panel differences, not as configuration-isolated causal effects.