- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces RewardWeaver for stage-wise reward adaptation in language agents performing long-horizon interaction. The method separates the fixed semantics of admitted capability rubrics and their evaluators from the set of capabilities prioritized for training at a given stage. After each training stage, it localizes failure evidence in low-outcome on-policy trajectories, aggregates recurrent capability bottlenecks attributable to the policy's behavior, and selects an equally weighted set of process rewards for the next stage. Failures outside the existing capability library trigger expansion through a separate validation procedure. The paper evaluates the framework on SOTOPIA, AmazonHistoryPrice, and a synthetic sales benchmark, with ablations examining dynamic selection, failure attribution, and stable capability semantics.
Critical strengths
The paper addresses a practical issue: policy bottlenecks can change during training, while fixed process rewards may cease to target them. Keeping admitted rubrics and evaluators fixed while adapting the active capability set is a coherent approach. Together with the requirement for policy-controllable evidence and the option to abstain from attribution, this design helps avoid automatically treating every low-outcome interaction as an agent deficiency and makes capability scores easier to compare across stages.
Critical weaknesses
The paper does not clearly describe the procedure of the proposed method. Although it provides an overall framework and pseudocode, key decisions in reward construction and capability expansion remain largely abstract: for example, when recurrent failures are judged to be uncovered by existing rubrics and trigger expansion, and which criteria determine acceptance of candidate capabilities, evaluators, and reward compositions (§2.5, Appendix A, Algorithms 1–2). These steps directly determine how rewards are constructed and updated. The lack of concrete rules makes it difficult for me to understand how the method was implemented in the experiments and affects its reproducibility.
Questions for authors
-
In §2.5, how does the system determine that failures are recurrent and cannot be represented by the existing capability library, thereby triggering expansion? Please provide the decision rules actually used in the experiments and explain how genuine capability gaps requiring new rubrics are distinguished from retrieval or attribution errors.
-
Please concretely define the validity criteria V in Algorithm 1 and the reward-publication condition ACCEPT(s) in Algorithm 2. Which metrics, calculations, and thresholds govern candidate capability/evaluator admission and reward-set publication, respectively? Please provide the criteria actually used in the experiments.
-
Sales has an explicit 320/80 train–evaluation split, with evaluation data excluded from reward adaptation and checkpoint selection. However, I could not find corresponding information for SOTOPIA and AmazonHistoryPrice. Please specify the train–evaluation splits for these two benchmarks and clarify whether evaluation data are used for reward adaptation or checkpoint selection.