acceptodds
Under review as a conference paper at ICLR 2027

RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in domains where task outcomes can be reliably evaluated, but long-horizon interaction remains challenging due to sparse terminal feedback and difficult credit assignment. Process rewards provide denser supervision, yet the capabilities most relevant for training can change as the policy evolves: a behavior that is easy to evaluate or frequently deficient need not be the bottleneck currently limiting task success. We introduce **RewardWeaver**, a self-evolving reward adaptation framework for language agents in long-horizon interaction. RewardWeaver maintains a semantically stable space of capability Rubrics and closes the loop between policy optimization, task evaluation, failure attribution, and reward adaptation. After each training stage, it performs outcome-grounded backward attribution on low-outcome trajectories, aggregates recurrent and policy-controlled capability bottlenecks, and dynamically selects and weights the corresponding process rewards for the next stage. Recurrent failures not covered by the existing capability space trigger a separate, controlled expansion procedure. We evaluate **RewardWeaver** on SOTOPIA, AmazonHistoryPrice, and a newly constructed Sales Benchmark. Across social interaction, bilateral bargaining, and domain-specific sales, **RewardWeaver** consistently establishes new state-of-the-art (SOTA) results. Ablations further demonstrate the importance of dynamic reward allocation, failure-grounded attribution, and stable reward semantics.

Primary area:
reinforcement learning
Keywords:
language agents, reinforcement learning, long-horizon interaction, failure attribution, reward adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qUQyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RewardWeaver for stage-wise reward adaptation in language agents performing long-horizon interaction. The method separates the fixed semantics of admitted capability rubrics and their evaluators from the set of capabilities prioritized for training at a given stage. After each training stage, it localizes failure evidence in low-outcome on-policy trajectories, aggregates recurrent capability bottlenecks attributable to the policy's behavior, and selects an equally weighted set of process rewards for the next stage. Failures outside the existing capability library trigger expansion through a separate validation procedure. The paper evaluates the framework on SOTOPIA, AmazonHistoryPrice, and a synthetic sales benchmark, with ablations examining dynamic selection, failure attribution, and stable capability semantics.

Critical strengths

The paper addresses a practical issue: policy bottlenecks can change during training, while fixed process rewards may cease to target them. Keeping admitted rubrics and evaluators fixed while adapting the active capability set is a coherent approach. Together with the requirement for policy-controllable evidence and the option to abstain from attribution, this design helps avoid automatically treating every low-outcome interaction as an agent deficiency and makes capability scores easier to compare across stages.

Critical weaknesses

The paper does not clearly describe the procedure of the proposed method. Although it provides an overall framework and pseudocode, key decisions in reward construction and capability expansion remain largely abstract: for example, when recurrent failures are judged to be uncovered by existing rubrics and trigger expansion, and which criteria determine acceptance of candidate capabilities, evaluators, and reward compositions (§2.5, Appendix A, Algorithms 1–2). These steps directly determine how rewards are constructed and updated. The lack of concrete rules makes it difficult for me to understand how the method was implemented in the experiments and affects its reproducibility.

Questions for authors
  1. In §2.5, how does the system determine that failures are recurrent and cannot be represented by the existing capability library, thereby triggering expansion? Please provide the decision rules actually used in the experiments and explain how genuine capability gaps requiring new rubrics are distinguished from retrieval or attribution errors.

  2. Please concretely define the validity criteria V in Algorithm 1 and the reward-publication condition ACCEPT(s) in Algorithm 2. Which metrics, calculations, and thresholds govern candidate capability/evaluator admission and reward-set publication, respectively? Please provide the criteria actually used in the experiments.

  3. Sales has an explicit 320/80 train–evaluation split, with evaluation data excluded from reward adaptation and checkpoint selection. However, I could not find corresponding information for SOTOPIA and AmazonHistoryPrice. Please specify the train–evaluation splits for these two benchmarks and clarify whether evaluation data are used for reward adaptation or checkpoint selection.

Reviewer_7C1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RewardWeaver, an adaptive reward framework for language agents in multi-turn interaction. It maintains a validated library of capability rubrics and, after each training stage, identifies recurrent, evidence-supported, policy-controlled deficiencies in low-outcome trajectories. The corresponding capabilities receive equal-weight process rewards in the next stage. Its central design separates changing training priorities from frozen capability definitions and evaluators, while allowing library expansion through a separate validation procedure.

Critical strengths

A clear and well-motivated mechanism for adapting rewards to changing training bottlenecks. The paper distinguishes whether a behavior can be evaluated reliably from whether it currently deserves optimization priority. Requiring attribution evidence, recurrence across trajectories, and policy controllability is a sensible response to the multiple sources of failure in interactive tasks. Separating reward selection from changes to admitted rubric semantics also provides an explicit design principle for limiting objective drift.

Critical weaknesses

The central mechanism is insufficiently specified and validated to establish reproducibility or the claimed validated. Algorithms 1–2 delegate crucial operations to attribution, aggregation, scheduling, and calibration routines without adequately specifying their models, prompts, priority statistics and coefficients, or admission and calibration thresholds. Appendix A supplies representative rubrics and one evaluator example, but no per-capability acceptance-set sizes, annotation sources, or quantitative reliability results. The underlying RL configuration and the aggregation and scaling of trajectory-level process scores are also underspecified. Meanwhile, Appendix D.2 describes a candidate evaluator being flagged for revision without an explicit update trace connecting this example to the frozen-evaluator principle. These omissions concern the mechanism that distinguishes the paper from generic adaptive reward schemes, rather than peripheral implementation details.

Questions for authors
  1. Under the same policy initialization, training-interaction budget, partner, and evaluator, how does RewardWeaver compare with a closely related evolving-rubric method such as EvoRubric? Could you also specify the additional model-call cost of reward generation, attribution, and calibration, and identify which Table 1 baselines were rerun under a common protocol?
  2. Could you provide the concrete configuration of Algorithms 1–2, including the attribution and scoring models, key prompts, priority computation and thresholds, and the aggregation and normalization of process scores in Equation (1)? For each of the 40 evaluators, how many independent acceptance examples were used, who supplied the reference labels, and which quantitative reliability criteria were actually met?
Reviewer_Kf7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RewardWeaver, a framework for dynamically adapting process rewards in long-horizon interactive agent training. It constructs a validated capability rubric library and periodically identifies recurring capability bottlenecks through failure attribution on low-outcome trajectories. The corresponding rubrics are then selected to guide subsequent RL training, while their definitions remain fixed. Experiments on SOTOPIA, AmazonHistoryPrice, and a newly constructed Sales Benchmark show improvements over several baselines.

Critical strengths
  • Clear motivation and practical framework. The paper addresses a relevant problem in long-horizon agent training: fixed process rewards may not reflect the policy's changing capability bottlenecks. The distinction between stable rubric semantics and adaptive training priorities is reasonable and well motivated.

  • The method shows improvements across multiple interactive benchmarks, and the ablations provide some evidence that failure-guided reward selection can outperform fixed or randomly selected rubrics.

Critical weaknesses
  • The central approach appears to be a relatively straightforward combination of existing ideas: rubric-based RL, failure attribution, and dynamic selection of training objectives. Prior work on evolving rubrics (e.g., EvoRubric) and failure-guided agent learning already explores closely related directions. The main distinction here is selecting a subset of predefined rubrics based on attributed failures, but its novelty and advantages over existing adaptive reward methods are not sufficiently established. More direct comparisons with these methods are needed.

  • The paper argues that failure attribution identifies the actual capability bottlenecks limiting task success, but the accuracy of these attributions is not independently validated. The reported correlation with subsequent failure reduction is relatively modest (Spearman ρ = 0.28), and the ablations do not consistently favor the proposed method across all metrics. Moreover, the main results lack multi-seed variance or statistical significance analysis. Consequently, it remains unclear whether the gains arise specifically from accurate bottleneck identification rather than simpler forms of dynamic reward selection.

Questions for authors
  • How does RewardWeaver compare with existing evolving-rubric approaches or simpler failure-driven training strategies under matched computational budgets? What is the specific advantage of the proposed failure-attribution-based selection?

  • Can the authors provide independent validation of the identified capability bottlenecks, such as human annotations or counterfactual interventions, to demonstrate that they are genuinely responsible for task failures?

  • Figure 2(c) describes capability expansion, but its role in the experiments is unclear. Were new rubrics actually added during training? If so, how much did they contribute to the performance gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.