- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether reward-model scores determine which single trajectory a pool policy returns. It audits 37 mixed-outcome τ²-bench airline pools (370 candidates, 185 successes), a secondary closure over three source/domain cells using a locally trained Qwen3-4B joint pointwise/pairwise diagnostic (three seeds), and implementation diagnostics (five deterministic repeats, three JSON serializations, graph/orientation, padding/length, pair support). Reported findings: five fixed-process repeats are identical, but three semantically equivalent serializations agree on the winner in only 57% of pools; named public checkpoints (D-Scalar, T-Verifier) show no detectable excess utility over chance at n=37; the primary-seed joint evaluator shows negative excess utility (-.284 original airline) that reverses sign at other seeds; pairwise/graph arms are highly sensitive to graph seed, orientation, and pair support. Validation is an internal audit chain plus a CPU/GPU release.
Critical strengths
1. The paper separates the deployed decision (which single trajectory the pool policy returns) from score quality , and studies it with executable reference outcomes on multi-turn tool trajectories. This is a useful framing, because mainstream reward-model evaluation reports accuracy or global correlation, which the paper shows can coexist with near-chance top-1 selection.
2. The evidence chain is unusually explicit: deterministic repeats, per-pool receipts, provenance tables, and clear labelling of which analyses were specified post hoc and which results are non-detections. I verified several headline numbers against the released JSON records and they match.
Critical weaknesses
1. The paper proposes no validated method. It explicitly disclaims a selector theory, a benchmark, and any ranking of selector classes, and states that its estimands "introduce no new algebra". The only prescriptive output is a "deployment receipt" whose thresholds the authors themselves label "prospective hypotheses, not promotion gates", and the single candidate mitigation (abstain-on-tie) covers 6/37 pools and selects failures in all six. No fallback policy is evaluated, so nothing here has been shown to improve selection.
2. The underlying observation is a known engineering property, not a research finding. That a scalar proxy reward need not order decisions consistently with the true objective is routine for any surrogate objective, and it is already documented for best-of-N judging by Landesberg (2026), which the paper cites as its closest prior work. The marginal additions (multi-turn tool trajectories, JSON key-order sensitivity) are domain transfers, and the headline number does not survive replication (-.284 at seed 20260920 versus +.014 and -.068 at the other two seeds, with AUC moving from .435 to .587).
Questions for authors
Q1. How much of the negative excess utility is attributable to the code-generation scaffold preference rather than a general top-1 selection failure? For example, what is excess utility computed within a single scaffold?
Q2. The AgentSuite cell of the secondary closure has no retained raw-snapshot hash, accession, or licence record. Can you release enough of that source for another group to reproduce even one cell, and if not, should the closure be read as evidence at all?