- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks how a fixed per-conversation inference budget (measured in bytes) should be allocated across four operations — SAMPLE, VERIFY, JUDGE, RETRY — to maximize pass$^k$, the probability that all attempts at a task succeed. Instead of live experiments, the authors replay 6,428 publicly released trajectories from τ-bench and τ²-bench through an exact evaluator: allocation policies are scored by recursion over the multiset of recorded draws, with no model queries. The headline finding is a "cost-over-accuracy rule": at deployment budgets, how cheaply a selector operates matters ~10× more than how accurately it selects. A perfect verifier that must pay to read full candidates is worth ~1 point of pass⁴ over a cheap plurality vote (43.6% vs 43.3–43.5%); the same perfect selector made free is worth ~10 points (53.2%). A write-set verifier reading only 2% of candidate bytes reaches 47.7%. The paper also reports that tuning for consistency rather than accuracy changes nothing, and that a learned controller (PATIENCE) matches but does not beat tuned rules or best-of-k.
Critical strengths
-
The evaluation methodology is the real contribution. Scoring arbitrary allocation policies exactly by recursion over recorded trajectory pools — with labels entering only at scoring and out-of-fold training targets — is clean, fast, reproducible, and eliminates provider-state confounds. The split-half validation (Table 6) directly addresses the plug-in bias concern and is exactly the kind of self-audit reviewers want. The replication on the original τ-bench release with different models, simulators, and year (Table 8) materially strengthens the claim that findings are about the setting, not a release.
-
The cost-over-accuracy rule is a genuinely useful operational message. "Selection accuracy is worth about a point; selection cost is worth about ten" is a crisp, falsifiable claim, and Proposition 2 (breakeven read cost) is tested against the read-cost sweep (Table 13) rather than left as theory. The write-set verifier is a constructive proof that cheap, weak-ish selection can beat expensive perfect selection — the paper practices what it preaches.
-
Intellectual honesty about negative results. Two preregistered-style expectations (consistency-tuned thresholds, learned controller beating rules) were refuted and reported as such. The authors state plainly they do not claim a new SOTA, quantify the gains (~2 points of pass⁴, ~4% relative), and flag what stands between 2 and 10 points. The limitations section is unusually candid (temperature-0 pools, pool-size caps, correlated-jury failure in 18% of sometimes-tasks).
-
Paired, cluster-bootstrapped design. The factorial over rule × menu × target with task-cluster bootstraps is well-constructed; half-widths near half a point define the resolution of nulls honestly rather than overclaiming equality.
Critical weaknesses
-
The central claim leans heavily on the cost model, which is acknowledged once and then treated as ground truth. The result that a perfect verifier ties a free vote depends entirely on charging verification the full byte cost of every candidate it reads. The paper calls this "a design choice, stated here once" — but it is the load-bearing assumption. A realistic LLM-as-judge verifier does not re-read the full transcript per candidate; a compact judge prompt, cached prefixes, or a verifier over extracted state would land at a very different point on the read-cost axis (Figure 6 shows the verdict is exquisitely sensitive to ρ: 100% → +0.1, 25% → +6.2). The write-set verifier result demonstrates the rule but is itself an existence proof engineered by the authors. I'm not sure whether the "rule" is a fact about inference-time allocation or a fact about one particular (and somewhat adversarial-to-verifiers) metering scheme. At minimum the paper should sweep verifier cost structures beyond "fraction of candidate bytes."
-
External validity of the replay is bounded in ways that matter for the main contrast. Because the released agents ran at temperature 0, a SAMPLE buys a different simulated user, not a different decode — the paper says this, but the consequence is deeper than stated: the study measures consistency under user-level variation only, which is plausibly the dominant variance source in τ²-bench's dual-control setting, but deployment sampling adds decode-level diversity that the vote's value (which rests on passing runs agreeing on write sets) may depend on. Pools of 4 (8 in replication) cap both the number of draws a policy can make and the diversity it can find; the ceiling of Theorem 1 is computed under the replay's own finite pools. A live, positive-temperature check — even a small one — would substantially raise confidence. I recognize this partially defeats the "no provider state" design, but one modest live arm would do a lot of work.
-
The theoretical framing is thinner than the empirical machinery. Proposition 1 is a constructed example (fine as a demonstration that objectives can differ), and Proposition 2 is near-tautological algebra: the verifier pays ρ per read so it gets m/(1+ρ) draws. Calling this a "rule" risks overstatement — it is a conditional trade-off curve parameterized by a metering assumption, not a law. Theorem 1 is standard union-of-failures reasoning. This is acceptable, but the abstract and intro should be calibrated accordingly.
-
The realistic verifiers are weak strawmen. The primary verifier is logistic regression on shallow observable features (AUC 0.63/0.63/0.88). The interesting regime — a moderately good verifier at moderate read cost — is covered only by the artificial "label + charged fraction" ladder. The paper would be stronger with one realistic mid-cost verifier (e.g., a small judge model over a trajectory summary) rather than interpolating between a strawman and an oracle.
-
The gains themselves are small, and the strongest domain effects are concentrated. Mean effects (~+1.8 points of pass⁴) are real by the bootstrap but concentrated in retail; airline is null or negative in several contrasts and has the widest intervals. The cross-release replication helps, but the operational prescription ("buy candidates and a vote") has its largest effect where the base agent is weakest — which is also where operators least need the advice. This is acknowledged in Appendix T but deserves a place in the main argument.
Questions for authors
-
Beyond charging a fraction of candidate bytes, did you test any realistic verifier cost structure (e.g., fixed judge prompt over a trajectory summary, cached-prefix pricing)? Does the cost-over-accuracy rule survive when the verifier reads, say, 20–30% of a transcript via a compact extraction?
-
For the write-set verifier (the constructive recommendation): its AUC is 0.65/0.70/0.93 by domain — is it not close to the telecom ceiling where a strong verifier should flip the recommendation (as Table 1 hints with AVA+menu ahead of best-of-k on telecom)? What read cost does the write-set verifier need before the vote wins again on telecom?
-
The 18% correlated-jury failure (plurality write set wrong despite a passing run in pool) seems like the true ceiling on voting rules. Did you quantify what the write-set verifier does on exactly those tasks?
-
With pools of 4, policies drawing ≥4 times with replacement face repeated candidates. How much does the vote's measured value depend on pool size? Would the split-half design support a pool-size sensitivity check on τ-bench (4 vs 8 trials)?
-
Why does the difficulty-gated rule underperform even with the oracle verifier (Table 9, 42.3)? Is that a budget artifact or a structural flaw of gating on the first candidate?
3 comments
Response to Reviewer iswP (part 2 of 2)
Q3, the correlated-jury tasks. We scored every selector on exactly the 63 tau2-bench tasks where the plurality write set is wrong although a passing run is in the pool (pass^4): base agent 1.9, vote 1.8, paid perfect verifier 8.5, write-set verifier 14.5, free perfect verifier 22.1. On every other task (n = 1,049): base 44.1, vote 46.4, paid oracle 46.0, write-set 50.1, free oracle 55.7. The write-set verifier recovers what the vote cannot on exactly those tasks, and it beats the paid oracle there because it keeps the draws the oracle spends on reads. The 18% is the vote's ceiling, not the write-set verifier's.
Q4, pool size. Using the eight-trial claude-3.5-sonnet pools against eight random four-trial subsamples of the same pools (pass^4):
| retail-v1 | pool 8 | pool 4, mean (sd) |
|---|---|---|
| Base | 49.7 | 53.5 (1.5) |
| Vote | 54.1 | 56.8 (1.5) |
| Vote minus base | +4.4 | +3.3 (0.8) |
| Verified minus vote | -2.7 | -2.4 (1.0) |
Airline-v1: vote minus base +1.2 at pool 8, +0.8 (0.9) at pool 4. Levels move about four points with pool size (the plug-in bias), the contrasts move by under a point, and the vote's advantage is larger at eight trials, so it is not an artifact of shallow pools.
Q5, the difficulty-gated rule. A budget artifact of its structure. The gate verifies the first candidate alone (its read plus the 1 KB call overhead). When the gate fails, the rule draws more, a new draw resets the verified flag, and the final verification re-reads the whole pool including the first candidate. A failed gate therefore costs one read plus two overheads, about a draw's worth of bytes at the retail median, which is why it trails even with the oracle (42.3 against 43.6). A gate that cached the first score would close that gap.
W4, strawman verifiers. The write-set verifier is our realistic mid-cost point: a trained classifier over the candidate's write actions, AUC 0.65/0.70/0.93 by domain, not an oracle, and it is the rule's recommendation. A judge model over a summary sits on the same read-price axis as the sweep above; we have not run one in the window, and the live arm's four-trial and eight-trial results above show how a weak selector's standing moves with the pool it faces.
W5, concentration. Leave-one-domain-out: dropping retail leaves PATIENCE minus base at +0.89 [+0.32, +1.48]; dropping airline +2.02 [+1.38, +2.68]; dropping telecom +2.29 [+1.54, +3.08]; dropping any one model leaves +1.58 to +2.06. On where the effect sits: the base agent is strongest on retail (56.5 pass^4) and weakest on telecom (about 24 to 33), and the gain is largest on retail (+3.03). So the operational effect concentrates where the agent is already strong, which is the deployment case, not the weak one. We now say this in the body.
W2, external validity. We ran the modest live arm the reviewer asked for, and it did a lot of work. With the benchmark's own harness, gpt-4.1-mini agent and the released gpt-4.1 simulator on retail, we recorded 8 trials per task at temperature 0.7 and 4 at temperature 0 (1,368 conversations, no failures). The temperature-0.7 pools have 2.13 distinct write-action sets per task over four trials; the released temperature-0 pools have 2.12. So at the benchmark's pool size the decode adds essentially no action diversity; the simulated user is the source. On the eight-trial 0.7 pool the vote beats base by +5.2 [+3.0, +7.4] (replay: +3.9), the paid perfect verifier is -1.1 [-3.3, +1.2] against the vote (the byte-meter tie), the write-set verifier is +0.9 [-2.4, +4.1] (and +4.3 [+0.9, +7.8] at temperature 0), and the free perfect verifier is +14.5. The one thing that did not replicate: under the dollar meter the logistic verifier beats the vote at four trials (+3.7) but loses on the eight-trial pool (-3.8); the perfect verifier wins either way (+12.4). We report it as a limit of the weak verifier, and the new appendix says so.
TL;DR
A note on the page header: the TL;DR field is locked and still shows the wording from before the revision, including the unconditional "ten times". The revised abstract and PDF state the claim as a function of the read price, and they are the version to read.
new tl;dr: Replaying 6,428 recorded agent runs exactly, we find that a perfect verifier ties a plain vote when reading costs as much as sampling and beats it by 8 points at list prices. The price of a read decides what an inference budget should buy.
Response to Reviewer iswP (part 1 of 2)
We thank the reviewer for a review that names what would change their assessment. We take the cost model first, because the reviewer is right that it is load-bearing.
W1, the cost model. We priced reads two more ways. (a) In dollars: from the per-conversation cost Sierra recorded in the released runs and the providers' list input prices, one read of a transcript costs 3 to 10% of producing it (rho 0.03 for claude-3.7-sonnet, 0.08 to 0.10 for gpt-4.1, 0.07 to 0.09 for gpt-4.1-mini, 0.04 to 0.06 for o4-mini). So the byte meter is the end of the curve that is hardest on verifiers. (b) We reran every rule at rho = 0.07 with the paper's logistic verifier (12-cell means, pass^4):
Operations beat samples by more at the dollar price (AVA+menu minus base +3.54 [+2.78, +4.38]), the vote no longer ties the verifier (verified minus vote +3.19 [+2.07, +4.29]), and the menu effect vanishes (AVA+menu minus AVA -0.09 [-0.49, +0.31]) because a cheap verifier alone suffices. (c) We charged the realistic verifiers a fraction rho of the full transcript per candidate:
Q1. Yes. At 20 to 30% of the transcript the write-set verifier is 2.3 to 3.0 points above the vote, and it stays above the vote until about rho = 0.85. A fixed judge prompt over a trajectory summary is a verifier that reads a fraction of the bytes, which is this sweep; cached-prefix pricing lowers rho further and moves along the same axis.
So the reviewer's question has a direct answer: the rule is a statement about the price of a read, with Proposition 2 giving the breakeven (0.68 predicted, about 0.85 measured for the write-set verifier). The revision states it that way in the abstract and introduction, and no longer says "ten times" unconditionally. What holds at every price is the ordering that a selector which reads 2% of the bytes beats one that reads them all when both are paid, and that is the constructive result.
W3, the theory. We accept the calibration. "Rule" stays as the name of a measured finding, stated as conditional on rho. Proposition 2 is simple algebra; its value is that it predicts the measured crossing within 0.8 points over the mid-range of the sweep (Table 13), which is the test a reviewer can check.
Q2, telecom. At its own 2% read the write-set verifier is 35.5 on telecom against the vote's 29.3. Charged a fraction of the full transcript it stays ahead until about rho = 0.9 (33.0 at 0.3, 31.3 at 0.5, 28.8 at 1.0). Its AUC of 0.93 there is not near a ceiling that flips the recommendation; the recommendation flips only when reading costs nearly as much as sampling.