acceptodds
Under review as a conference paper at ICLR 2027

Not All Experiences Are Equal: Counterfactual Experience Valuation for Multi-Turn Agent RL

Abstract

On-policy reinforcement learning for multi-turn language agents largely optimizes the update rule (advantages, group comparisons, KL) while treating which experiences deserve update mass as a default. Return and advantage record how an episode ended and how strongly the current surrogate wants to reinforce it. Neither answers the counterfactual that matters under a conserved budget: whether giving one experience more update mass, taken from its neighbors, improves held-out capability. We introduce EVA (Experience Valuation and Augmentation), a data-centric layer in front of an unchanged , and its measurement core IA-ELV. IA-ELV scores each experience by a budget-preserving probe that returns gold coordinates , namely meta-performance change and decision-uncertainty reduction. On ALFWorld and WebShop, at 1.5B and 7B, learning value is sparse and peaks at mid-training, where return and advantage fail to recover . A leak-free predictor ranks the gold object with the same sign, so EVA can sit in front of GiGPO without a probe at every step. Gold Mid allocation, and the same rule under Mid-continue GiGPO, beat uniform weighting across both environments and both scales; Mid is the strongest stage. Soft probes keep learning value distinct from augmentation gain. Legally recollected four-action routing then adds held-out value beyond reweighting alone. EVA therefore separates outcome, credit, learning value, and augmentation value under explicit conservation constraints. Our code is available at https://anonymous.4open.science/r/eva-3B8C.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM Agents, Experience Valuation, Data Selection, Experience Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N7CJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes EVA for allocating a fixed optimization budget across agent trajectories. IA-ELV measures held-out performance and decision-uncertainty changes through paired, budget-conserving reweighting updates. A learned predictor guides trajectory weighting without changing the underlying GiGPO surrogate. Experiments cover ALFWorld and WebShop with 1.5B and 7B models, alongside additional studies of semantic augmentation and state branching.

Critical strengths
  1. Investigating what kinds of rewards can actually improve model performance is a meaningful research direction.

  2. The gradient analysis is interesting, particularly the observation in line 47 that RR and ∣A∣|A| are negatively correlated.

Critical weaknesses
  1. The paper is somewhat difficult to read. The abstract and introduction introduce too many symbols and concepts at once, including “data-centric layer,” “unchanged URLU_{RL},” “meta-performance,” and “decision uncertainty.” The sudden reference to rewriting goals or branching in the third paragraph of the introduction is also confusing, as is the introduction of δ\delta in lines 59–60. The fourth paragraph of the introduction, around line 56, contains too much information and is difficult to follow.

  2. Figure 1 is not a vector graphic and appears blurry, making some of its text difficult to read. The figure is also overly dense; even in vector format, it would still be difficult to understand.

  3. Rewriting is introduced in both the introduction and the main text without any transition. How does it relate to the preceding discussion of gradient allocation? Similarly, “Semantic-Aug” and “State-Branch” appear abruptly in line 166, without an explanation of what they are, why they are needed, or how they address the previously identified problem.

  4. The underlying issue studied in this paper is that current RL credit assignment does not reflect what the model actually needs to learn. This observation is not new: prior work on reward-model training has long sought better proxy training objectives. I am not arguing that the paper has limited novelty, but rather that it should not make the following claim in line 40: “It has not asked a prior question. Once a batch has been collected, which trajectories should receive more of a fixed update budget, and which should be left alone or transformed?”

  5. When selecting training weights, increasing the gradient contribution of one sample also effectively reduces the gradient contributions of other samples. How can one determine which of these changes is responsible for the performance improvement?

  6. There is a concrete arithmetic inconsistency between the per-seed gains in Table 13 and the reported mean. The seven deltas are 2.0, 7.0, 3.5, 1.5, 4.6, 5.0, and 2.5, which sum to 26.1 and yield a mean of approximately 3.73. However, the change from 76.6 to 81.1 reported in the main text and table caption corresponds to a gain of +4.5.

  7. The experiments use GiGPO as URLU_{RL}. Does the same phenomenon hold for standard GRPO? GiGPO already modifies reward allocation, so the reported phenomenon could stem from an issue with GiGPO’s reward allocation rather than from correctness-based GRPO more generally.

  8. Training the predictor requires constructing a substantial dataset, with each data point requiring an actual evaluation. Ensuring robustness also requires a sufficiently large number of rollouts, and some samples will have performance differences too small to serve as training data. What is the cost of collecting these training data, and how many samples are needed to train the predictor?

Questions for authors

Please refer to the weakness

1 comment

We thank the reviewer for the constructive feedback and for recognizing the importance of understanding which experiences contribute to policy improvement. We agree that the presentation, methodological interpretation, and experimental reporting need further clarification.

1. We agree that the abstract and introduction introduce too many concepts prematurely. We will reorganize the introduction around three questions: why return and advantage are insufficient to characterize learning value, how IA-ELV measures the effect of budget-preserving reallocation, and how EVA uses this signal for experience allocation and augmentation. We will simplify the notation, clarify the role of meta-performance and decision uncertainty, and move technical details to later sections.

2. We agree that Figure 1 is too dense and difficult to read. We will redraw it as a vector graphic and simplify the layout, separating the core valuation and allocation pipeline from the augmentation extension. We will also reduce in-figure text and clarify the relationship between components.

3. We will clarify that ELV measures the value of allocating more update budget to an existing experience, whereas EAG measures the additional benefit of transforming it. Semantic-Aug rewrites task descriptions while preserving intent; State-Branch explores alternative actions from decision states. Both may introduce semantic drift or invalid transitions, so their gains are evaluated separately. For legal recollection, generated branches are re-executed and receive fresh on-policy credit. We will introduce these operations and their motivation before presenting the four-action routing results, and clarify that the legal four-action experiment is limited to ALFWorld 1.5B Mid.

4. We agree that our original novelty claim was too broad. Prior work has studied reward modeling and proxy training objectives. Our contribution is more specific: measuring the marginal effect of reallocating a fixed update budget among already-collected on-policy experiences within a context. We will remove the claim that this question has not been studied before and better position IA-ELV relative to existing work.

5. We agree that increasing one trajectory's weight necessarily reduces the contribution of others. Thus, IA-ELV measures a joint reallocation effect, not an intrinsic contribution independent of other experiences. The target remains in both branches, while the additional weight is removed uniformly from other trajectories in its context. The paired difference therefore measures the effect of emphasizing the target relative to its context under this donor rule, rather than inclusion versus deletion. We will make this conditional interpretation explicit.

6. We thank the reviewer for identifying the arithmetic inconsistency. The seven reported gains sum to 26.1, yielding a mean of approximately +3.73, rather than +4.5. We will verify the original seed-level results and aggregation script, then correct the table and corresponding text based on the confirmed records.

7. Our experiments use GiGPO, whose hierarchical credit assignment differs from standard GRPO. Although IA-ELV is defined at the experience-allocation level and does not inherently require GiGPO, our current experiments cannot establish that the same phenomenon holds under GRPO or exclude the influence of GiGPO's credit assignment. We will explicitly limit our empirical claims to GiGPO and identify evaluation under standard GRPO as future work.

8. We agree that collecting predictor training data incurs substantial overhead. Each gold label requires paired counterfactual updates and matched evaluations. The current protocol uses n=64n=64 trajectories per context, except ALFWorld 1.5B Early and Late, which use n=128n=128. These are per-context probe sizes, not the total predictor dataset size. The predictor uses two lightweight ridge-regression heads for ΔJ\Delta J and ΔH\Delta H, evaluated through leave-one-context-out cross-validation to prevent context leakage. However, the manuscript does not report aggregate labeled samples, filtered probes, evaluation calls, refresh frequency, or additional GPU hours. We will audit the run records and report the actual dataset size and collection cost. We will also clarify that current results demonstrate predictability on the collected data, rather than establishing a universal minimum sample requirement.

We thank the reviewer again for these suggestions. We will improve the presentation, clarify the scope of the methodological contribution, correct the numerical inconsistency, and qualify claims that are not yet supported by experiments.

Reviewer_RzN2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EVA, a trajectory allocation layer for multi-turn agent RL. Its core estimator measures ΔJi\Delta J_i, the change in held-out performance when update weight is transferred from neighboring trajectories to trajectory ii, while conserving total weight. A lightweight predictor approximates these measurements for online reweighting without modifying the underlying GiGPO objective.

Critical strengths

Strengths

  • The paired interventions provide a concrete operational definition of local trajectory utility.
  • Fixed-pool comparisons help isolate allocation effects from changes in data collection.
  • The reported online gains are positive across two environments and two model scales.
Critical weaknesses

Weaknesses

  1. Limited methodological novelty. The relationship to validation-driven meta-reweighting and influence-based experience selection is not sufficiently resolved. Conserving total trajectory weight specifies an intervention direction, but the paper does not convincingly establish a substantial methodological advance over existing approaches.

  2. Insufficient comparisons with closely related methods. Uniform, reward-based, and absolute-advantage weighting are useful controls, but stronger influence-based and validation-driven reweighting baselines are needed. Moreover, weighting by ∣A∣|A| differs substantially from AWR-style weighting by exp⁡(A/β)\exp(A/\beta), so the absolute-advantage baseline does not establish superiority over advantage-weighted approaches more generally.

  3. Incomplete computational cost accounting. Gold-label generation requires additional policy updates and held-out evaluations. The main-training comparisons match rollout and update budgets, but do not establish the end-to-end cost of obtaining and maintaining the predictor. Label refresh frequency, aggregate evaluation cost, and total GPU hours should be reported. Algorithm 1 also appears to reuse the reference update, whereas the accompanying text states that it is recomputed for each pair; this discrepancy affects cost estimates.

Questions for authors
  1. Relationship to prior methods. What methodological capability does budget-preserving valuation provide beyond validation-driven meta-reweighting or influence-based experience selection? Can you compare against representative methods under matched total computational budgets?

  2. Advantage-weighting baseline. Why use ∣A∣|A| rather than signed-advantage weighting such as exp⁡(A/β)\exp(A/\beta)? Does the improvement persist against a suitably tuned signed-advantage baseline?

  3. End-to-end cost. How many gold probes are required initially, how frequently are labels refreshed, and what are the total additional GPU hours and environment interactions? Algorithm 1 appears to reuse the reference update, whereas the text states that it is recomputed for every pair. Which protocol produced the reported results?

  4. Online uncertainty and risk gating. How is σ^i\hat{\sigma}_i estimated for the lower-confidence-bound score? Since the predictor outputs only ΔJi^\widehat{\Delta J_i} and ΔHi^\widehat{\Delta H_i}, how is the PIG constraint enforced in the gated online experiments without access to probe outcomes?

  5. Separation of validation and final evaluation. Are the tasks used to generate ΔJi\Delta J_i labels disjoint from those used for final performance reporting? Please clarify the splits used for valuation, predictor validation, hyperparameter selection, and final evaluation.

2 comments

EVA is related to validation-driven reweighting and influence-based selection, but our contribution is a specific counterfactual estimand for on-policy multi-turn agent RL, rather than non-uniform weighting itself.

For a trajectory eie_i within a context CiC_i of rollouts sharing a task prompt, IA-ELV measures the held-out performance change when a fixed weight mass δ\delta is transferred from neighboring trajectories to eie_i. The reference and focus branches share the same checkpoint, optimizer state, update steps, learning rate, clipping configuration, and evaluation banks. This isolates the marginal effect of reallocating a conserved update budget, without adding optimization steps.

Unlike influence-based methods that estimate effects through gradients or parameter influence, or validation-driven reweighting that optimizes weights against a validation objective, IA-ELV measures a within-context, budget-preserving intervention on collected on-policy trajectories. It also records ΔH\Delta H and PIG as separate information and policy-displacement signals. We do not claim universal superiority or subsumption of prior methods; direct comparisons remain valuable future work.

We use absolute advantage ∣A∣|A| to test whether the magnitude of the local learning signal reflects marginal experience value, independently of its sign. At the Mid stage, both return and ∣A∣|A| are negatively correlated with gold ΔJ\Delta J. On the ALFWorld 1.5B Mid fixed pool, ∣A∣|A|-based weighting obtains 0.482, below Uniform (0.499), while gold IA-ELV reaches 0.592.

We acknowledge that this does not establish superiority over signed-advantage or AWR-style weighting, which requires specifying a non-negative transformation, normalization, and temperature. We will clarify the role of ∣A∣|A| as a magnitude-based baseline and acknowledge that tuned signed/AWR-style weighting remains unevaluated.

The probe protocol uses n=64n=64 trajectories per Mid context and n=128n=128 for ALFWorld 1.5B Early and Late. Each target requires a recomputed reference update paired with a focus update, ensuring paired stochastic effects. Although the appendix describes this protocol, Algorithm 1 places the reference update outside the loop. We will correct the pseudocode by moving it and its evaluation inside the target loop.

Thus, each valid target requires two updates and matched evaluations. The current manuscript does not report aggregate probe counts, evaluation calls, refresh frequency, or additional GPU hours. We will audit the run logs and report these costs relative to GiGPO training. The per-context batch size alone cannot represent total overhead. Gold IA-ELV is the measurement target; the predictor aims to reduce the need for paired probes on every online trajectory.

The strict online predictor has regression heads for ΔJ\Delta J and ΔH\Delta H, using trajectory outcomes, advantage statistics, length, termination, and within-context statistics. Gold PIG and other post-probe quantities are excluded. The conservative score ℓ=μ^−λσ^\ell=\hat{\mu}-\lambda\hat{\sigma} penalizes predictive uncertainty in ΔJ\Delta J; σ^\hat{\sigma} is not an estimate of PIG.

PIG, the mean Jensen–Shannon divergence between reference and focus action distributions on a frozen state bank, is available from paired probes. Therefore, the strict predictor cannot enforce a PIG threshold without an additional PIG predictor or online probe. We will clarify this distinction between gold-probe safety filtering and predictor-based conservative reweighting, and will not claim that the strict online predictor enforces PIG without a validated prediction mechanism. We will also specify how σ^\hat{\sigma} is estimated and λ\lambda calibrated.

Gold-label generation uses a frozen meta-validation task catalog MJM_J, with task identities and evaluation seeds held out from gold-label contexts. Predictor evaluation uses leave-one-context-out cross-validation, fitting preprocessing without the held-out context; trajectories sharing a task prompt are not split across training and validation. Gold ΔJ\Delta J, ΔH\Delta H, PIG, and post-probe constraint flags are excluded from deployable features.

We agree that the data roles need clearer documentation. The current manuscript does not fully specify a separate hyperparameter-selection split or establish the independence of every selection decision from final evaluation. We will provide a split table covering gold-label contexts, predictor folds, hyperparameter selection, and final evaluation tasks and seeds, claiming strict separation only where supported by the actual protocol.

We thank the reviewer for highlighting these issues. We will clarify that gold IA-ELV establishes a measurable, budget-preserving allocation signal, while the predictor and online safety interface require further validation. We will also correct the algorithm description, qualify baseline claims, and improve computational reporting.

Thank you for the clarifications. I’m keeping my Weak Reject. The gains over uniform weighting are encouraging, but it remains unclear whether EVA outperforms simpler methods once the cost of generating the labels is counted.

Reviewer_5X8wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes EVA, which estimates the learning value of individual trajectories under a fixed update budget and uses it to reweight agent training, reporting gains over uniform weighting on ALFWorld and WebShop with 1.5B and 7B models.

Critical strengths
  • The budget-preserving paired probe gives a clear operational definition of an experience’s learning value: whether shifting a fixed amount of update weight toward it improves held-out performance. This is more directly tied to the paper’s goal than ranking experiences by return or advantage.

  • The paper tests predicted-value reweighting in continued training, rather than relying only on gold-label analyses. The reported gains across ALFWorld and WebShop at two model sizes provide some evidence that the idea can transfer to an online training setting.

Critical weaknesses
  • The cost–benefit case is incomplete. Gold valuation requires repeated policy updates and held-out evaluations for each experience. The matched update budgets exclude the cost of generating labels and training or refreshing the predictor, so it is unclear whether the performance gains justify the full computational cost.

  • The predictor’s data requirements are unclear. The paper does not show how many gold labels are needed to achieve useful rankings, or how often the predictor must be refreshed as the policy changes. These omissions make the practicality of online weighting difficult to assess.

  • The closed-loop evidence is limited. Most reported gains are point estimates across two environments and one base optimizer; seed-level results are shown for only one main setting. This makes the reliability and generality of the gains hard to judge.

  • Presentation is extremely poor. Figure 1 is too dense to read easily, while several tables use considerable space for little detail; Table 2, for example, compresses four settings into a single row of ranges.

Questions for authors
  • How many paired probes and held-out evaluations were used to produce the gold labels, and what was their total wall-clock or GPU cost relative to GiGPO training?

  • How many gold labels are needed for the predictor to deliver the reported closed-loop gains, and was it retrained or refreshed as the policy changed?

  • How many seeds support each result in Table 4, and what is the variability of those gains? Could you also report the per-setting predictor results currently aggregated into ranges in Table 2?

2 comments

We thank the reviewers for their constructive feedback. We believe several concerns stem from an insufficient distinction between experience valuation, conventional advantage-based weighting, and online policy optimization. We clarify our core contribution and address these concerns below.

EVA studies a fundamental question: given a fixed on-policy batch and a limited optimization budget, how should update resources be allocated across experiences to improve policy performance? EVA is not a new policy-gradient algorithm. It is a valuation layer that operates before existing optimizers such as GiGPO.

Importantly, return and advantage do not directly measure the marginal benefit of allocating additional update resources to an experience. Return reflects observed outcomes, while advantage provides a local learning signal. Neither measures the effect of reallocating a conserved update budget. This is precisely what IA-ELV captures. By transferring update weight from competing experiences to a target experience while keeping the total budget unchanged, IA-ELV measures the resulting changes in held-out performance and decision uncertainty. Therefore, budget redistribution is not an experimental confound; it is the intervention that defines our research question.

Our experiments evaluate different stages of this framework and should not be conflated. Gold IA-ELV allocation improves final performance from 0.499 to 0.592 (+9.4 percentage points) on the frozen Mid-stage pool, demonstrating the potential of counterfactual experience valuation. However, the current online predictor and router do not consistently translate this potential into training gains: the seven-seed experiment shows an average change of approximately -0.7 percentage points, while the four-seed gated experiment is 8.2 points below uniform GiGPO. We explicitly acknowledge that stable online superiority has not been established. The gap between measuring experience value and reliably predicting and exploiting it online is an important limitation, not a result we intend to overclaim.

We also acknowledge that gold valuation incurs additional counterfactual probing and evaluation costs. Matched optimization budgets do not imply zero valuation overhead. We will clarify the computational requirements and report available measurements. The current predictor uses only pre-probe features; exploratory post-probe signals are not deployable and will be clearly distinguished. We further acknowledge that our experiments use absolute rather than signed advantages, and that generalization beyond GiGPO remains unverified.

Finally, the current study evaluates ALFWorld with Qwen2.5-1.5B-Instruct and GiGPO. We do not claim results on WebShop or 7B models. We will clarify the experimental scope, data splits, and implementation details, and recheck the numerical inconsistency identified in Table 13 against the original logs.

In summary, our contribution is not simply to reweight experiences, but to define and measure their context-dependent marginal learning value under a conserved update budget. The fixed-pool results support the value of this formulation, while the online results reveal the remaining challenges in prediction and deployment. We believe these two findings should be evaluated separately.

Thank you for the clarifications, but my core concerns remain unresolved. The response provides no quantitative accounting (e.g., the total valuation overhead, required label count, or predictor refresh frequency), which leaves the cost–benefit case unestablished. Also, consistent seed-level results are needed to clarify the conflicting descriptions of online gains and experimental scope.

Clarifications and promises of future reporting do not provide sufficient evidence to resolve these concerns. I therefore maintain my rating.

Comments on the paper

1 comment

Multi-turn agent RL faces a key challenge: trajectory returns and local advantages cannot fully capture an experience's contribution to future policy improvement. Experiences with similar outcomes may provide different learning signals, while low-return experiences may still help correct the policy. Thus, equal treatment or outcome-based ranking can waste limited update budgets.

Our core contribution is EVA, a counterfactual experience valuation and allocation framework for multi-turn agent RL. EVA estimates the marginal learning value of each experience and allocates a fixed update budget accordingly, rather than relying solely on trajectory returns or local advantages. It further distinguishes the value of learning from an existing experience (ELV) from the incremental value of transforming it (EAG). EVA operates as a valuation layer before an unchanged on-policy optimizer, not as a new RL algorithm.

A key design principle is to evaluate experiences by their contribution under a controlled learning intervention, rather than by assigning them static scores based on isolated properties. Specifically, EVA considers how the expected learning outcome changes when update resources are reallocated from competing experiences to a target experience while keeping the overall budget fixed. This counterfactual perspective makes experience value inherently relative to the current training context and competing data, allowing EVA to capture differences that cannot be fully reflected by return, advantage, or other local proxies.

Specifically, information-aware ELV (IA-ELV) estimates the effect of reallocating a conserved update budget toward a target experience relative to competing experiences. We evaluate this effect through paired, budget-preserving counterfactual updates, using held-out performance change and uncertainty reduction. Thus, redistribution is the intervention being studied, not an uncontrolled confound.

Importantly, experience value is context-dependent: the same trajectory may contribute differently under different policy states and competing experience pools. IA-ELV therefore incorporates the shared task context of on-policy experiences and estimates both their expected contribution to held-out performance and their effect on predictive uncertainty. Its online predictor uses training-time information to estimate these quantities without relying on future evaluation outcomes, enabling experience valuation to guide allocation during training. This design connects counterfactual evaluation with practical online decision-making while maintaining the on-policy data collection and update process.

Beyond prioritizing existing experiences, EVA also considers whether an experience can become more useful through augmentation. ELV and EAG address two complementary but distinct decisions: whether an existing experience deserves more learning resources, and whether transforming that experience can provide additional learning value. An experience with low ELV is not necessarily a good augmentation candidate, just as an experience with high ELV does not automatically benefit from further transformation. By evaluating these values separately, EVA avoids conflating the usefulness of existing data with the potential benefit of generating or modifying data. This distinction enables a more principled allocation of both training and augmentation resources, while accounting for the utility, validity, and cost of candidate transformations.

Our experiments use GiGPO with Qwen2.5-1.5B and 7B agents on ALFWorld and WebShop. Fixed-pool gold allocation yields positive gains across all four environment/model settings, with the largest gains at the Mid stage: +9.4 and +8.0 on ALFWorld, and +7.0 and +6.5 on WebShop. Under continued GiGPO training, EVA also improves performance; for example, ALFWorld 1.5B improves from 76.6 to 81.1 without gating and from 93.2 to 97.0 with gating. A legal four-action routing study further achieves +3.7 over Uniform and +2.1 over Downweight+Keep.

For the strict online predictor, Mid-stage results show active AUROC of 0.72–0.80 and Spearman correlation of 0.35–0.50, with overconfident errors below 8%. Performance is weaker at Early and Late stages, indicating that prediction quality depends on training stage. We will clarify this limitation and report computational overhead, including probe calls, held-out evaluations, labeling, refresh frequency, and additional GPU hours, based on actual logs.

Although EVA may extend to other policy-gradient optimizers, our experiments focus on GiGPO; transfer to methods such as GRPO remains future work. We will also clarify the distinction between final-step and best-over-training performance, improve the description of data separation and evaluation protocols, and verify the Table 13 aggregation against the original run logs, seed identifiers, and aggregation script before correcting any inconsistent values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.