acceptodds
Withdrawn from ICLR 2027

Auditing Reward-Model Selection on Tool Trajectories: Identification, Policy Utility, and Natural Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reward models often turn a candidate pool into one selected trajectory. The maximum score by itself does not show whether that trajectory is useful. We audit selected checkpoints on executable tool trajectories. The main evidence comes from 37 mixed-outcome -bench airline pools, with a secondary closure on AgentSuite airline and held-out retail cells. Five independently started fixed-process runs gave identical results on the original 370 candidates. By contrast, three semantically equivalent serializations agreed on the winner in only of pools (95% CI ). At the primary joint-evaluator seed, mixed-only excess utility was , , and in the three cells and varied across seeds. On the same 69 completed mixed pools, the follow-up sensitivity checks found full-cycle winner agreements after reversal of the primary orientation and agreements between padded and dynamic full cycles. A secondary matched-slot arm covered 69/102 gates and provides a feasibility-conditional, binary-outcome, policy-plus-graph diagnostic. Our conclusions are limited to the named checkpoints, policies, pools, serializations, graph constructions, and context protocols; they do not rank selector classes or establish general equivalence.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward modeling, agentic evaluation, tool-use trajectories, best-of-N selection, tie-aware evaluation, reward-model robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nBC3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether reward-model scores determine which single trajectory a pool policy returns. It audits 37 mixed-outcome τ²-bench airline pools (370 candidates, 185 successes), a secondary closure over three source/domain cells using a locally trained Qwen3-4B joint pointwise/pairwise diagnostic (three seeds), and implementation diagnostics (five deterministic repeats, three JSON serializations, graph/orientation, padding/length, pair support). Reported findings: five fixed-process repeats are identical, but three semantically equivalent serializations agree on the winner in only 57% of pools; named public checkpoints (D-Scalar, T-Verifier) show no detectable excess utility over chance at n=37; the primary-seed joint evaluator shows negative excess utility (-.284 original airline) that reverses sign at other seeds; pairwise/graph arms are highly sensitive to graph seed, orientation, and pair support. Validation is an internal audit chain plus a CPU/GPU release.

Critical strengths

1. The paper separates the deployed decision (which single trajectory the pool policy returns) from score quality , and studies it with executable reference outcomes on multi-turn tool trajectories. This is a useful framing, because mainstream reward-model evaluation reports accuracy or global correlation, which the paper shows can coexist with near-chance top-1 selection.

2. The evidence chain is unusually explicit: deterministic repeats, per-pool receipts, provenance tables, and clear labelling of which analyses were specified post hoc and which results are non-detections. I verified several headline numbers against the released JSON records and they match.

Critical weaknesses

1. The paper proposes no validated method. It explicitly disclaims a selector theory, a benchmark, and any ranking of selector classes, and states that its estimands "introduce no new algebra". The only prescriptive output is a "deployment receipt" whose thresholds the authors themselves label "prospective hypotheses, not promotion gates", and the single candidate mitigation (abstain-on-tie) covers 6/37 pools and selects failures in all six. No fallback policy is evaluated, so nothing here has been shown to improve selection.

2. The underlying observation is a known engineering property, not a research finding. That a scalar proxy reward need not order decisions consistently with the true objective is routine for any surrogate objective, and it is already documented for best-of-N judging by Landesberg (2026), which the paper cites as its closest prior work. The marginal additions (multi-turn tool trajectories, JSON key-order sensitivity) are domain transfers, and the headline number does not survive replication (-.284 at seed 20260920 versus +.014 and -.068 at the other two seeds, with AUC moving from .435 to .587).

Questions for authors

Q1. How much of the negative excess utility is attributable to the code-generation scaffold preference rather than a general top-1 selection failure? For example, what is excess utility computed within a single scaffold?

Q2. The AgentSuite cell of the secondary closure has no retained raw-snapshot hash, accession, or licence record. Can you release enough of that source for another group to reproduce even one cell, and if not, should the closure be read as evidence at all?

Reviewer_TZXyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits reward-model selection of tool-use trajectories. The main experiment evaluates selected checkpoints on 37 mixed-outcome airline candidate pools using executable success labels. Additional analyses examine tie-breaking, score precision, serialization, and pointwise versus pairwise selection. The main findings are that ranking quality does not necessarily yield better top-1 selection, and that equivalent JSON formats can change the selected trajectory despite deterministic repeated runs.

Critical strengths

The motivation is reasonable. Distinguishing unique score maxima, top-set quality, and selection utility helps evaluate the actual decision made by a reward-based selector. Explicit tie-breaking rules also make the reported results easier to interpret.

The serialization experiment is the strongest result. Five fixed-setting process repeats return identical results, but three semantically equivalent serializations agree on the winner in only 56.8% of pools. This shows that repeatability does not imply robustness to input format. Reporting outcome disagreement separately also helps distinguish changes in winner identity from changes in task success.

Critical weaknesses

The contribution is limited in scope. As the paper acknowledges, the distinction between ranking quality and selection utility is established in prior work. The serialization analysis adds a useful observation for tool trajectories, but the other experiments provide limited additional insight. The paper does not yet establish a broader finding beyond the behavior of the selected checkpoints and protocols.

Much of the remaining evidence is inconclusive or difficult to attribute. In the primary analysis, excess utility is −0.014 [−0.130, 0.108] for D-Scalar and 0.014 [−0.135, 0.159] for T-Verifier. These intervals do not establish either a meaningful benefit or its absence. The pairwise comparison is further limited by outcome-associated resource exclusions and sensitivity to graph orientation. Context extrapolation and the use of general preference models also make it difficult to separate selection failures from evaluator mismatch. The authors acknowledge these limitations, but they substantially restrict what the experiments establish.

Questions for authors

Does the serialization result persist with a task-appropriate evaluator operating within its supported context length?

Can the main utility analysis be extended to enough independent tasks to resolve a practically meaningful effect? The current intervals leave both improvement and degradation plausible.

On the feasible subset, how do the sparse pairwise results compare with complete all-pairs evaluation in both orientations? The low winner agreement after orientation reversal makes this reference important.

Reviewer_i3gQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper audits reward-model selection over pools of executable tool trajectories. It separates unique score maxima, tie policies and actual success of selected trajectory. Experiments show that deterministic scoring can still be sensitive to serialization, while pairwise selection also changes with graph construction and orientation.

Critical strengths

Paper evaluates success of selected trajectory rather than relying only on score correlation. Explicit tie policies expose how provenance-linked ordering can affect reported utility. Serialization and reversed-orientation checks also reveal instability that repeated runs with identical inputs would miss.

Critical weaknesses
  1. Evaluator and input mismatch limits interpretation. Main pointwise models include general preference checkpoints with extrapolated context, while local evaluators are tested far beyond training lengths; pairwise training also excludes same-outcome pairs that appear frequently at test time. These are valid stress tests, but without a strong task-matched baseline, it is difficult to tell what finding extends beyond these particular setups. Appendix’s ToolRM endpoint helps, but does not resolve this for main experiments.

  2. Evidence is small and difficult to reproduce independently. Primary natural analysis has 37 mixed pools, while secondary AgentSuite source lacks retained accession, raw-snapshot hash and licence record. Local adapters are also omitted from release. Numerical reconstruction from stored scores is useful, but cannot independently verify model behavior, especially when seed and serialization materially change results.

Questions for authors
  1. On complete trajectories fitting native context limits, do utility and serialization findings persist without context extrapolation?
  2. Can authors add a task-matched evaluator under its supported input protocol, keeping pools and scoring conditions fixed?
  3. Can local adapters and a fully traceable replacement for AgentSuite be released so key results can be reproduced from model inputs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.