- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes EVA for allocating a fixed optimization budget across agent trajectories. IA-ELV measures held-out performance and decision-uncertainty changes through paired, budget-conserving reweighting updates. A learned predictor guides trajectory weighting without changing the underlying GiGPO surrogate. Experiments cover ALFWorld and WebShop with 1.5B and 7B models, alongside additional studies of semantic augmentation and state branching.
Critical strengths
-
Investigating what kinds of rewards can actually improve model performance is a meaningful research direction.
-
The gradient analysis is interesting, particularly the observation in line 47 that and are negatively correlated.
Critical weaknesses
-
The paper is somewhat difficult to read. The abstract and introduction introduce too many symbols and concepts at once, including “data-centric layer,” “unchanged ,” “meta-performance,” and “decision uncertainty.” The sudden reference to rewriting goals or branching in the third paragraph of the introduction is also confusing, as is the introduction of in lines 59–60. The fourth paragraph of the introduction, around line 56, contains too much information and is difficult to follow.
-
Figure 1 is not a vector graphic and appears blurry, making some of its text difficult to read. The figure is also overly dense; even in vector format, it would still be difficult to understand.
-
Rewriting is introduced in both the introduction and the main text without any transition. How does it relate to the preceding discussion of gradient allocation? Similarly, “Semantic-Aug” and “State-Branch” appear abruptly in line 166, without an explanation of what they are, why they are needed, or how they address the previously identified problem.
-
The underlying issue studied in this paper is that current RL credit assignment does not reflect what the model actually needs to learn. This observation is not new: prior work on reward-model training has long sought better proxy training objectives. I am not arguing that the paper has limited novelty, but rather that it should not make the following claim in line 40: “It has not asked a prior question. Once a batch has been collected, which trajectories should receive more of a fixed update budget, and which should be left alone or transformed?”
-
When selecting training weights, increasing the gradient contribution of one sample also effectively reduces the gradient contributions of other samples. How can one determine which of these changes is responsible for the performance improvement?
-
There is a concrete arithmetic inconsistency between the per-seed gains in Table 13 and the reported mean. The seven deltas are 2.0, 7.0, 3.5, 1.5, 4.6, 5.0, and 2.5, which sum to 26.1 and yield a mean of approximately 3.73. However, the change from 76.6 to 81.1 reported in the main text and table caption corresponds to a gain of +4.5.
-
The experiments use GiGPO as . Does the same phenomenon hold for standard GRPO? GiGPO already modifies reward allocation, so the reported phenomenon could stem from an issue with GiGPO’s reward allocation rather than from correctness-based GRPO more generally.
-
Training the predictor requires constructing a substantial dataset, with each data point requiring an actual evaluation. Ensuring robustness also requires a sufficiently large number of rollouts, and some samples will have performance differences too small to serve as training data. What is the cost of collecting these training data, and how many samples are needed to train the predictor?
Questions for authors
Please refer to the weakness
We thank the reviewer for the constructive feedback and for recognizing the importance of understanding which experiences contribute to policy improvement. We agree that the presentation, methodological interpretation, and experimental reporting need further clarification.
1. We agree that the abstract and introduction introduce too many concepts prematurely. We will reorganize the introduction around three questions: why return and advantage are insufficient to characterize learning value, how IA-ELV measures the effect of budget-preserving reallocation, and how EVA uses this signal for experience allocation and augmentation. We will simplify the notation, clarify the role of meta-performance and decision uncertainty, and move technical details to later sections.
2. We agree that Figure 1 is too dense and difficult to read. We will redraw it as a vector graphic and simplify the layout, separating the core valuation and allocation pipeline from the augmentation extension. We will also reduce in-figure text and clarify the relationship between components.
3. We will clarify that ELV measures the value of allocating more update budget to an existing experience, whereas EAG measures the additional benefit of transforming it. Semantic-Aug rewrites task descriptions while preserving intent; State-Branch explores alternative actions from decision states. Both may introduce semantic drift or invalid transitions, so their gains are evaluated separately. For legal recollection, generated branches are re-executed and receive fresh on-policy credit. We will introduce these operations and their motivation before presenting the four-action routing results, and clarify that the legal four-action experiment is limited to ALFWorld 1.5B Mid.
4. We agree that our original novelty claim was too broad. Prior work has studied reward modeling and proxy training objectives. Our contribution is more specific: measuring the marginal effect of reallocating a fixed update budget among already-collected on-policy experiences within a context. We will remove the claim that this question has not been studied before and better position IA-ELV relative to existing work.
5. We agree that increasing one trajectory's weight necessarily reduces the contribution of others. Thus, IA-ELV measures a joint reallocation effect, not an intrinsic contribution independent of other experiences. The target remains in both branches, while the additional weight is removed uniformly from other trajectories in its context. The paired difference therefore measures the effect of emphasizing the target relative to its context under this donor rule, rather than inclusion versus deletion. We will make this conditional interpretation explicit.
6. We thank the reviewer for identifying the arithmetic inconsistency. The seven reported gains sum to 26.1, yielding a mean of approximately +3.73, rather than +4.5. We will verify the original seed-level results and aggregation script, then correct the table and corresponding text based on the confirmed records.
7. Our experiments use GiGPO, whose hierarchical credit assignment differs from standard GRPO. Although IA-ELV is defined at the experience-allocation level and does not inherently require GiGPO, our current experiments cannot establish that the same phenomenon holds under GRPO or exclude the influence of GiGPO's credit assignment. We will explicitly limit our empirical claims to GiGPO and identify evaluation under standard GRPO as future work.
8. We agree that collecting predictor training data incurs substantial overhead. Each gold label requires paired counterfactual updates and matched evaluations. The current protocol uses n=64 trajectories per context, except ALFWorld 1.5B Early and Late, which use n=128. These are per-context probe sizes, not the total predictor dataset size. The predictor uses two lightweight ridge-regression heads for ΔJ and ΔH, evaluated through leave-one-context-out cross-validation to prevent context leakage. However, the manuscript does not report aggregate labeled samples, filtered probes, evaluation calls, refresh frequency, or additional GPU hours. We will audit the run records and report the actual dataset size and collection cost. We will also clarify that current results demonstrate predictability on the collected data, rather than establishing a universal minimum sample requirement.
We thank the reviewer again for these suggestions. We will improve the presentation, clarify the scope of the methodological contribution, correct the numerical inconsistency, and qualify claims that are not yet supported by experiments.