acceptodds
Under review as a conference paper at ICLR 2027

ANOTHER SAMPLE OR A BETTER VERIFIER? WHAT INFERENCE BUDGET BUYS FOR CONSISTENT AGENTS

Abstract

An agent that handles refunds is useful only if it gets them right every time, not most of the time. Benchmarks call this consistency and measure it as passk, the chance that all k attempts at a task succeed. We ask what a fixed byte budget per conversation should buy to raise it, a sample, a verifier call, a vote, or a retry, and we find a rule. At the budgets operators run, how cheaply a candidate is selected matters ten times more than how accurately; we call this the cost-over-accuracy rule. The evidence comes from replaying 6,428 released τ-bench and τ 2-bench trajectories through an evaluator that scores any allocation policy exactly, running no model. Across four frontier models and three domains, a single attempt succeeds 56.3% of the time and all four attempts 41.4%. Buying operations rather than samples raises pass4 by +1.76 points at equal cost. A perfect verifier that must read every candidate reaches 43.6%, no better than a vote, because reading costs as much as sampling and cuts the draws in half; the same verifier made free reaches 53.2%, and a proposition predicts where the two cross. A verifier that reads only a candidate’s write actions, 2% of the bytes, follows the rule and reaches 47.7%. Two expectations written down before the study were refuted. Optimizing for consistency changed no allocation, and a learned controller matched but did not beat a tuned rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time compute, inference-time scaling, LLM agents, tool use, agent reliability, consistency, pass^k, budget-constrained allocation, verification, majority voting, self-consistency, retry, tau-bench, trace-driven evaluation, exact evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iswPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how a fixed per-conversation inference budget (measured in bytes) should be allocated across four operations — SAMPLE, VERIFY, JUDGE, RETRY — to maximize pass$^k$, the probability that all kk attempts at a task succeed. Instead of live experiments, the authors replay 6,428 publicly released trajectories from τ-bench and τ²-bench through an exact evaluator: allocation policies are scored by recursion over the multiset of recorded draws, with no model queries. The headline finding is a "cost-over-accuracy rule": at deployment budgets, how cheaply a selector operates matters ~10× more than how accurately it selects. A perfect verifier that must pay to read full candidates is worth ~1 point of pass⁴ over a cheap plurality vote (43.6% vs 43.3–43.5%); the same perfect selector made free is worth ~10 points (53.2%). A write-set verifier reading only 2% of candidate bytes reaches 47.7%. The paper also reports that tuning for consistency rather than accuracy changes nothing, and that a learned controller (PATIENCE) matches but does not beat tuned rules or best-of-k.

Critical strengths
  1. The evaluation methodology is the real contribution. Scoring arbitrary allocation policies exactly by recursion over recorded trajectory pools — with labels entering only at scoring and out-of-fold training targets — is clean, fast, reproducible, and eliminates provider-state confounds. The split-half validation (Table 6) directly addresses the plug-in bias concern and is exactly the kind of self-audit reviewers want. The replication on the original τ-bench release with different models, simulators, and year (Table 8) materially strengthens the claim that findings are about the setting, not a release.

  2. The cost-over-accuracy rule is a genuinely useful operational message. "Selection accuracy is worth about a point; selection cost is worth about ten" is a crisp, falsifiable claim, and Proposition 2 (breakeven read cost) is tested against the read-cost sweep (Table 13) rather than left as theory. The write-set verifier is a constructive proof that cheap, weak-ish selection can beat expensive perfect selection — the paper practices what it preaches.

  3. Intellectual honesty about negative results. Two preregistered-style expectations (consistency-tuned thresholds, learned controller beating rules) were refuted and reported as such. The authors state plainly they do not claim a new SOTA, quantify the gains (~2 points of pass⁴, ~4% relative), and flag what stands between 2 and 10 points. The limitations section is unusually candid (temperature-0 pools, pool-size caps, correlated-jury failure in 18% of sometimes-tasks).

  4. Paired, cluster-bootstrapped design. The factorial over rule × menu × target with task-cluster bootstraps is well-constructed; half-widths near half a point define the resolution of nulls honestly rather than overclaiming equality.

Critical weaknesses
  1. The central claim leans heavily on the cost model, which is acknowledged once and then treated as ground truth. The result that a perfect verifier ties a free vote depends entirely on charging verification the full byte cost of every candidate it reads. The paper calls this "a design choice, stated here once" — but it is the load-bearing assumption. A realistic LLM-as-judge verifier does not re-read the full transcript per candidate; a compact judge prompt, cached prefixes, or a verifier over extracted state would land at a very different point on the read-cost axis (Figure 6 shows the verdict is exquisitely sensitive to ρ: 100% → +0.1, 25% → +6.2). The write-set verifier result demonstrates the rule but is itself an existence proof engineered by the authors. I'm not sure whether the "rule" is a fact about inference-time allocation or a fact about one particular (and somewhat adversarial-to-verifiers) metering scheme. At minimum the paper should sweep verifier cost structures beyond "fraction of candidate bytes."

  2. External validity of the replay is bounded in ways that matter for the main contrast. Because the released agents ran at temperature 0, a SAMPLE buys a different simulated user, not a different decode — the paper says this, but the consequence is deeper than stated: the study measures consistency under user-level variation only, which is plausibly the dominant variance source in τ²-bench's dual-control setting, but deployment sampling adds decode-level diversity that the vote's value (which rests on passing runs agreeing on write sets) may depend on. Pools of 4 (8 in replication) cap both the number of draws a policy can make and the diversity it can find; the ceiling of Theorem 1 is computed under the replay's own finite pools. A live, positive-temperature check — even a small one — would substantially raise confidence. I recognize this partially defeats the "no provider state" design, but one modest live arm would do a lot of work.

  3. The theoretical framing is thinner than the empirical machinery. Proposition 1 is a constructed example (fine as a demonstration that objectives can differ), and Proposition 2 is near-tautological algebra: the verifier pays ρ per read so it gets m/(1+ρ) draws. Calling this a "rule" risks overstatement — it is a conditional trade-off curve parameterized by a metering assumption, not a law. Theorem 1 is standard union-of-failures reasoning. This is acceptable, but the abstract and intro should be calibrated accordingly.

  4. The realistic verifiers are weak strawmen. The primary verifier is logistic regression on shallow observable features (AUC 0.63/0.63/0.88). The interesting regime — a moderately good verifier at moderate read cost — is covered only by the artificial "label + charged fraction" ladder. The paper would be stronger with one realistic mid-cost verifier (e.g., a small judge model over a trajectory summary) rather than interpolating between a strawman and an oracle.

  5. The gains themselves are small, and the strongest domain effects are concentrated. Mean effects (~+1.8 points of pass⁴) are real by the bootstrap but concentrated in retail; airline is null or negative in several contrasts and has the widest intervals. The cross-release replication helps, but the operational prescription ("buy candidates and a vote") has its largest effect where the base agent is weakest — which is also where operators least need the advice. This is acknowledged in Appendix T but deserves a place in the main argument.

Questions for authors
  1. Beyond charging a fraction of candidate bytes, did you test any realistic verifier cost structure (e.g., fixed judge prompt over a trajectory summary, cached-prefix pricing)? Does the cost-over-accuracy rule survive when the verifier reads, say, 20–30% of a transcript via a compact extraction?

  2. For the write-set verifier (the constructive recommendation): its AUC is 0.65/0.70/0.93 by domain — is it not close to the telecom ceiling where a strong verifier should flip the recommendation (as Table 1 hints with AVA+menu ahead of best-of-k on telecom)? What read cost does the write-set verifier need before the vote wins again on telecom?

  3. The 18% correlated-jury failure (plurality write set wrong despite a passing run in pool) seems like the true ceiling on voting rules. Did you quantify what the write-set verifier does on exactly those tasks?

  4. With pools of 4, policies drawing ≥4 times with replacement face repeated candidates. How much does the vote's measured value depend on pool size? Would the split-half design support a pool-size sensitivity check on τ-bench (4 vs 8 trials)?

  5. Why does the difficulty-gated rule underperform even with the oracle verifier (Table 9, 42.3)? Is that a budget artifact or a structural flaw of gating on the first candidate?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iswP (part 1 of 2)

We thank the reviewer for a review that names what would change their assessment. We take the cost model first, because the reviewer is right that it is load-bearing.

W1, the cost model. We priced reads two more ways. (a) In dollars: from the per-conversation cost Sierra recorded in the released runs and the providers' list input prices, one read of a transcript costs 3 to 10% of producing it (rho 0.03 for claude-3.7-sonnet, 0.08 to 0.10 for gpt-4.1, 0.07 to 0.09 for gpt-4.1-mini, 0.04 to 0.06 for o4-mini). So the byte meter is the end of the curve that is hardest on verifiers. (b) We reran every rule at rho = 0.07 with the paper's logistic verifier (12-cell means, pass^4):

RuleByte meterDollar meter
Base agent41.441.4
Best-of-k43.343.3
Draw first, vote43.543.5
Draw first, verified42.846.7
Verify cascade42.545.9
AVA+menu43.244.8
PATIENCE (retrained)42.945.6

Operations beat samples by more at the dollar price (AVA+menu minus base +3.54 [+2.78, +4.38]), the vote no longer ties the verifier (verified minus vote +3.19 [+2.07, +4.29]), and the menu effect vanishes (AVA+menu minus AVA -0.09 [-0.49, +0.31]) because a cheap verifier alone suffices. (c) We charged the realistic verifiers a fraction rho of the full transcript per candidate:

rhoWrite-set verifierLogistic verifierVote
0.0547.646.943.5
0.2046.546.143.5
0.3045.845.443.5
0.5044.744.243.5
1.0043.042.843.5

Q1. Yes. At 20 to 30% of the transcript the write-set verifier is 2.3 to 3.0 points above the vote, and it stays above the vote until about rho = 0.85. A fixed judge prompt over a trajectory summary is a verifier that reads a fraction of the bytes, which is this sweep; cached-prefix pricing lowers rho further and moves along the same axis.

So the reviewer's question has a direct answer: the rule is a statement about the price of a read, with Proposition 2 giving the breakeven (0.68 predicted, about 0.85 measured for the write-set verifier). The revision states it that way in the abstract and introduction, and no longer says "ten times" unconditionally. What holds at every price is the ordering that a selector which reads 2% of the bytes beats one that reads them all when both are paid, and that is the constructive result.

W3, the theory. We accept the calibration. "Rule" stays as the name of a measured finding, stated as conditional on rho. Proposition 2 is simple algebra; its value is that it predicts the measured crossing within 0.8 points over the mid-range of the sweep (Table 13), which is the test a reviewer can check.

Q2, telecom. At its own 2% read the write-set verifier is 35.5 on telecom against the vote's 29.3. Charged a fraction of the full transcript it stays ahead until about rho = 0.9 (33.0 at 0.3, 31.3 at 0.5, 28.8 at 1.0). Its AUC of 0.93 there is not near a ceiling that flips the recommendation; the recommendation flips only when reading costs nearly as much as sampling.

AuthorsOpenReview ↗

Response to Reviewer iswP (part 2 of 2)

Q3, the correlated-jury tasks. We scored every selector on exactly the 63 tau2-bench tasks where the plurality write set is wrong although a passing run is in the pool (pass^4): base agent 1.9, vote 1.8, paid perfect verifier 8.5, write-set verifier 14.5, free perfect verifier 22.1. On every other task (n = 1,049): base 44.1, vote 46.4, paid oracle 46.0, write-set 50.1, free oracle 55.7. The write-set verifier recovers what the vote cannot on exactly those tasks, and it beats the paid oracle there because it keeps the draws the oracle spends on reads. The 18% is the vote's ceiling, not the write-set verifier's.

Q4, pool size. Using the eight-trial claude-3.5-sonnet pools against eight random four-trial subsamples of the same pools (pass^4):

retail-v1pool 8pool 4, mean (sd)
Base49.753.5 (1.5)
Vote54.156.8 (1.5)
Vote minus base+4.4+3.3 (0.8)
Verified minus vote-2.7-2.4 (1.0)

Airline-v1: vote minus base +1.2 at pool 8, +0.8 (0.9) at pool 4. Levels move about four points with pool size (the plug-in bias), the contrasts move by under a point, and the vote's advantage is larger at eight trials, so it is not an artifact of shallow pools.

Q5, the difficulty-gated rule. A budget artifact of its structure. The gate verifies the first candidate alone (its read plus the 1 KB call overhead). When the gate fails, the rule draws more, a new draw resets the verified flag, and the final verification re-reads the whole pool including the first candidate. A failed gate therefore costs one read plus two overheads, about a draw's worth of bytes at the retail median, which is why it trails even with the oracle (42.3 against 43.6). A gate that cached the first score would close that gap.

W4, strawman verifiers. The write-set verifier is our realistic mid-cost point: a trained classifier over the candidate's write actions, AUC 0.65/0.70/0.93 by domain, not an oracle, and it is the rule's recommendation. A judge model over a summary sits on the same read-price axis as the sweep above; we have not run one in the window, and the live arm's four-trial and eight-trial results above show how a weak selector's standing moves with the pool it faces.

W5, concentration. Leave-one-domain-out: dropping retail leaves PATIENCE minus base at +0.89 [+0.32, +1.48]; dropping airline +2.02 [+1.38, +2.68]; dropping telecom +2.29 [+1.54, +3.08]; dropping any one model leaves +1.58 to +2.06. On where the effect sits: the base agent is strongest on retail (56.5 pass^4) and weakest on telecom (about 24 to 33), and the gain is largest on retail (+3.03). So the operational effect concentrates where the agent is already strong, which is the deployment case, not the weak one. We now say this in the body.

W2, external validity. We ran the modest live arm the reviewer asked for, and it did a lot of work. With the benchmark's own harness, gpt-4.1-mini agent and the released gpt-4.1 simulator on retail, we recorded 8 trials per task at temperature 0.7 and 4 at temperature 0 (1,368 conversations, no failures). The temperature-0.7 pools have 2.13 distinct write-action sets per task over four trials; the released temperature-0 pools have 2.12. So at the benchmark's pool size the decode adds essentially no action diversity; the simulated user is the source. On the eight-trial 0.7 pool the vote beats base by +5.2 [+3.0, +7.4] (replay: +3.9), the paid perfect verifier is -1.1 [-3.3, +1.2] against the vote (the byte-meter tie), the write-set verifier is +0.9 [-2.4, +4.1] (and +4.3 [+0.9, +7.8] at temperature 0), and the free perfect verifier is +14.5. The one thing that did not replicate: under the dollar meter the logistic verifier beats the vote at four trials (+3.7) but loses on the eight-trial pool (-3.8); the perfect verifier wins either way (+12.4). We report it as a limit of the weak verifier, and the new appendix says so.

A note on the page header: the TL;DR field is locked and still shows the wording from before the revision, including the unconditional "ten times". The revised abstract and PDF state the claim as a function of the read price, and they are the version to read.

new tl;dr: Replaying 6,428 recorded agent runs exactly, we find that a perfect verifier ties a plain vote when reading costs as much as sampling and beats it by 8 points at list prices. The price of a read decides what an inference budget should buy.

Reviewer_JchqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to allocate a fixed inference budget across sampling, verification, write-action-set voting, and retrying to improve passk\mathrm{pass}^k, the probability that all repeated attempts succeed. It builds an exact recursive replay evaluator over finite public τ-bench/τ²-bench trajectory pools and compares rule-based controllers with the learned PATIENCE controller. Candidate-reading cost can matter more than selection accuracy under the chosen byte-cost model. A cheap write-set verifier outperforms a full-transcript perfect verifier, while learning does not clearly outperform tuned simple rules.

Critical strengths
  1. Reading candidates consumes resources, and this opportunity cost deserves explicit treatment in test-time-compute studies. Distinguishing passk\mathrm{pass}^k from average accuracy is also useful when considering repeated-service reliability.
  2. Within the stated finite-pool model, recursive evaluation avoids additional Monte Carlo noise in policy evaluation. Task-grouped fitting reduces label leakage for the verifier and controller.
  3. The write-set verifier is a concrete design response to the cost argument, rather than only an oracle comparison.
Critical weaknesses
  1. Section 3 evaluates p(π,τ)kp(\pi,\tau)^k from a finite pool for each task. This can be exact for the empirical replay model while still having finite-pool bias relative to the underlying trial distribution. The main text and Appendix I claim that the plug-in bias is the same across methods and does not affect policy comparisons. But different policies apply different nonlinear transformations to the empirical pool and need not share the same bias.
  2. The read-cost sweep is useful, but the conclusion still depends on transcript-byte accounting, the assumed price of reading candidates, exclusion of repeated system prompts, and whole-trajectory replacement.
  3. Section 5 states that the null effect of optimizing for consistency holds in every domain. Table 16 instead reports +0.92 [0.19,1.70] for retail and −1.48 [−2.99,−0.31] for airline;Likewise, the claim that another draw beats a perfect verifier for every voting rule conflicts with AVA+menu in Table 2 (+0.4 versus +2.3).
Questions for authors

Please distinguish exact values conditional on the empirical pool from estimates for the underlying trial distribution, justify or withdraw universal bias cancellation, and report split-pool/pool-size sensitivity with uncertainty for the principal contrasts.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Jchq

We thank the reviewer for three points that are each correct, and we have changed the paper on all three.

W1 and the question, plug-in bias. The reviewer is right that different rules apply different nonlinear maps to the pool and need not share a bias. We now distinguish two quantities throughout: the replay-exact value, which is the pass^k of a policy drawing from the recorded pool and is computed without Monte Carlo error, and the population value, which the split-half table estimates. The claim that the bias is "the same for every rule" is withdrawn and replaced by the measured biases. For the base agent the plug-in E[(c/n)^4] is biased up (Jensen): on the eight-trial pools the unbiased estimator C(c,4)/C(n,4) gives 46.2 against the plug-in 49.7 on retail-v1 and 22.4 against 25.3 on airline-v1. The vote is biased the other way: the split-half table already shows it under-predicted (59.9 predicted against 61.8 held out on retail) while the base agent is over-predicted (57.6 against 50.0). The bias therefore runs against the vote, and the reported vote advantage is conservative. Pool-size sensitivity, as requested: eight-trial pools against eight random four-trial subsamples give vote minus base +4.4 at pool 8 and +3.3 (sd 0.8, range +1.8 to +4.3) at pool 4 on retail-v1, and +1.2 against +0.8 (0.9) on airline-v1; verified minus vote -2.7 against -2.4 (1.0). Levels shift about four points, contrasts under one. Both tables are in the revised appendix.

W2, the byte accounting. We agree the conclusion depends on the read price and we now show it at two prices. From the cost fields Sierra recorded in the released runs and the providers' list input prices, one read costs 3 to 10% of a sample (rho 0.03 to 0.10 by model). Rerunning every rule at rho = 0.07: operations beat samples by +3.54 [+2.78, +4.38] (AVA+menu minus base), the verifier beats the vote by +3.19 [+2.07, +4.29], and the learned controller, retrained at that price, is +2.55 [+1.71, +3.42] over best-of-k while still 0.89 below the best fixed rule. The abstract now states the rule as a function of the read price rather than as "ten times". On the two accounting details: the repeated system prompt is identical for every candidate, so excluding it shifts every policy's cost by the same constant and cancels in all contrasts; whole-trajectory replacement is what the replay model can do and is stated as such in Section 3.

W3, the two inconsistencies. Both corrected. Section 5 now reads that the pooled null averages opposite signs, +0.92 [+0.19, +1.70] on retail and -1.48 [-2.99, -0.31] on airline, so tuning for consistency helped one domain and hurt another. The introduction sentence on Table 2 now gives the numbers per rule instead of a claim about every rule: for PATIENCE on retail one more draw is worth +1.6 points and a perfect verifier +0.5, while for AVA+menu, whose stopping depends on the verifier, the ranking reverses, +0.4 against +2.3. Both are within the band the rule claims and well below the free verifier's ten.

Thank you for the author's response and clarification. I tend to maintain my current positive evaluation.

Reviewer_KgqqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inference‑time compute allocation for consistent tool‑using agents, measured via passk\text{pass}^k: the probability that kk independent attempts on the same task all succeed. The authors propose the cost‑over‑accuracy rule: under realistic deployment‑level budgets, the monetary/compute cost of candidate selection matters far more than the raw accuracy of the selector itself. A perfect but expensive verifier that reads full trajectories delivers limited gains, while low‑cost voting (judge over write‑action tool calls) performs competitively. A lightweight verifier operating only on write‑action sets (≈2% of full transcript bytes) achieves substantially better pass4\text{pass}^4.

Critical strengths
  1. Instead of costly live online agent rollouts, the authors replay pre‑recorded benchmark trajectories to exactly compute passk\text{pass}^k for arbitrary allocation policies with zero LLM API calls. This eliminates variance from model provider state and enables exhaustive ablations over budgets, verifier designs, and rules. This evaluation framework is a valuable methodological advance for agent inference‑compute research and can be reused by the community.
  2. Prior literature prioritizes improving verifier accuracy; this paper highlights that verifier read cost can negate accuracy gains. The write‑action‑only verifier is a concrete, actionable design insight for production conversational agents. The theoretical propositions give interpretable break‑even conditions for when cheap voting beats high‑accuracy expensive verification.
Critical weaknesses
  1. All draws are sampled from a finite pool of pre‑generated trajectories with temperature 0. Real‑world agents generate fresh samples with non‑zero temperature, which would produce higher trajectory diversity. The authors acknowledge this limitation, but it remains unclear whether the cost‑over‑accuracy rule quantitatively holds for open‑ended on‑the‑fly sampling. The observed magnitude of performance gaps between voting and verifiers may shift under live generation. This substantially weakens confidence that the empirical numbers directly transfer to live agent deployments, which is the paper’s target setting.
  2. Even under favourable conditions, the absolute improvements in pass4\text{pass}^4 are on the order of +1.5‑2 percentage points. The learned PATIENCE controller does not beat simple heuristics like best‑of‑k. While the negative result is interesting scientifically, the paper delivers no new practical algorithm that meaningfully pushes agent consistency. For conference acceptance, one expects either larger empirical gains or a more thorough demonstration that the insights reliably translate to live agent behaviour. The paper leans heavily on theoretical and methodological merit, but practical upside remains limited.
Questions for authors
  1. Your replay uses finite pre‑computed trajectory pools with temperature 0. How would you expect the core results (cost‑over‑accuracy rule, relative ordering between voting and verifier‑based selectors) to change if trajectories were freshly sampled at non‑zero temperature? Can you provide qualitative or preliminary quantitative reasoning for whether the magnitude of gaps would grow or shrink?
  2. Theorem 1 defines a theoretical upper bound given a maximum number of draws mm. In your experiments, real policies sit far below this ceiling. What are the main bottlenecks preventing practical selectors from approaching this ceiling in multi‑turn agent tool‑use settings? Is it mostly selector quality, or are there task‑specific properties of the benchmark tasks?
  3. The write‑set verifier performs strongly in your offline replay. Have you identified any concrete failure modes for this verifier that could appear in live deployment, for example when semantically‑different correct tool sequences produce identical write‑action sets or vice‑versa?
  4. You observe that optimizing for passk\text{pass}^k versus pass@1 yields near‑identical allocation thresholds on this benchmark suite. To what degree do you attribute this to properties of the menu of available operations, versus properties of the τ2\tau^2-bench task distribution? Could there be other agent domains where optimizing for consistency would lead to drastically different allocation decisions?
1 comment
AuthorsOpenReview ↗

General response to all reviewers: read-price results, live arm, revised PDF

We thank the three reviewers for careful reviews. All three rate soundness as "minor or fixable", and they converge on one point we accept: the paper states its headline as if the byte meter, which prices a read at the cost of a sample, were the only meter. We have now priced reads in dollars from the cost fields Sierra recorded in the released runs and rerun every rule at that price. The findings survive and some strengthen; the headline sentence changes. The revised PDF contains the following, each detailed under the reviewer who asked for it.

  1. Read cost in dollars. At list input prices a read costs 3 to 10% of a sample (rho 0.03 to 0.10 by model), so the byte meter (rho = 1) is the pessimistic end for verifiers (revised Section 5).

  2. Every rule rerun at rho = 0.07 (revised Section 5, dollar-meter table). Operations beat samples by more than under bytes: AVA+menu minus base +3.54 [+2.78, +4.38] against +1.73. The vote no longer ties the verifier; the logistic verifier beats it by +3.19 [+2.07, +4.29]. The learned controller, retrained at that price, is +2.55 [+1.71, +3.42] over best-of-k and +0.95 [+0.32, +1.59] over AVA+menu, and still trails the best fixed rule by 0.89.

  3. Realistic verifiers across read price (revised Section 5, read-price figure). The write-set verifier beats the vote at every price up to about 0.85 of the transcript; Proposition 2 predicts the crossing at 0.68 for a perfect verifier.

  4. The abstract now states the rule as a function of the read price. Under the byte meter a free vote ties a paid perfect verifier and selection cost outweighs selection accuracy ten to one. At list prices a verifier that reads 2% of the bytes is worth +4 over the vote and a perfect one +8. "Ten times" is no longer stated unconditionally.

  5. Two sentences that contradicted their own tables (Section 5 on Table 16; the introduction on Table 2) and the split-half sentence on plug-in bias are corrected as Reviewer Jchq asked.

  6. New appendix tables: the correlated-jury tasks, pool size 8 against 4, leave-one-model-out and leave-one-domain-out, and the Theorem 1 gap decomposition.

  7. The live arm two reviewers asked for (new Appendix, "Live check at positive temperature"). We recorded 1,368 fresh retail conversations with the benchmark's own harness, gpt-4.1-mini agent and the released gpt-4.1 user simulator: 8 trials per task at agent temperature 0.7 and 4 at temperature 0, 114 tasks, zero infrastructure failures. Decode temperature adds almost no action-level diversity at the benchmark's pool size (2.13 distinct write-action sets per task over four trials at 0.7, against 2.12 in the released temperature-0 pools and 2.12 in our temperature-0 arm; the simulated user is where variation comes from). The ordering replicates in every live cell: vote minus base +5.2 [+3.0, +7.4] on the eight-trial 0.7 pool and +2.6 [+0.3, +5.1] at temperature 0; paid perfect verifier minus vote -1.1 [-3.3, +1.2] and +0.4 [-2.6, +3.5], the byte-meter tie; write-set verifier minus vote +0.9 [-2.4, +4.1] and +4.3 [+0.9, +7.8]; free perfect verifier minus vote +14.5 and +15.0. One caveat we report rather than hide: under the dollar meter the weak logistic verifier's edge over the vote holds at four trials (+3.7, +0.8) but not on the eight-trial pool (-3.8), while the perfect verifier's does (+12.4) and the write-set verifier ties there (-0.1) and wins at temperature 0 (+4.2). The result files, the fold-in script, and the two-line harness change are in the revised artifact.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer Kgqq

We thank the reviewer. We answer the two weaknesses and the four questions in order. W1 and Q1, temperature. We ran the live check. With the benchmark's own harness, gpt-4.1-mini agent and the released gpt-4.1 user simulator on retail, we recorded 8 trials per task at agent temperature 0.7 and 4 at temperature 0 (1,368 conversations, 114 tasks, no infrastructure failures). The quantitative reasoning the reviewer asked for now has numbers. (a) Diversity: at the benchmark's pool size, temperature 0.7 gives 2.13 distinct write-action sets per task over four trials against 2.12 in the released temperature-0 pools and 2.12 in our temperature-0 arm. The decode adds almost nothing at the action level; the simulated user is the source of trial-to-trial variation, which is what the replay already samples. Eight trials raise the count to 3.36. (b) Gaps: on the eight-trial 0.7 pool the vote beats base by +5.2 [+3.0, +7.4] and best-of-k by +4.8 [+2.8, +6.9], larger than the +3.9 and +3.2 of the released cell; at temperature 0 they are +2.6 [+0.3, +5.1] and +2.8 [+0.6, +5.2]. So the gaps hold or grow. (c) The ordering: the paid perfect verifier is -1.1 [-3.3, +1.2] against the vote at 0.7 and +0.4 [-2.6, +3.5] at 0, the byte-meter tie; the write-set verifier is +0.9 [-2.4, +4.1] and +4.3 [+0.9, +7.8]; the free perfect verifier +14.5 and +15.0. (d) Drift: our temperature-0 arm has 70.4% first-attempt success against 66.0% in the released run of the same agent and simulator, which is the day-to-day movement of hosted models that motivated exact replay in the first place. One caveat, reported in the new appendix: under the dollar meter the weak logistic verifier beats the vote at four trials (+3.7) but not on the eight-trial pool (-3.8), while the perfect verifier does either way (+12.4).

W2, small gains and no new algorithm. The +1.76 is measured under a byte meter that prices a read at the cost of a sample. At list prices a read costs 3 to 10% of a sample. Rerunning every rule at rho = 0.07, operations beat samples by +3.54 [+2.78, +4.38] (AVA+menu minus base), and the learned controller, retrained at that price, reaches 45.6 pass^4: +4.49 [+3.62, +5.45] over the base agent, +2.55 [+1.71, +3.42] over best-of-k, +0.95 [+0.32, +1.59] over AVA+menu. It still trails the best fixed rule (draw first, verified, 46.7) by 0.89 [0.28, 1.51], so the negative result stands in a milder form: learning beats the sampling rules once reads are priced realistically, but not a tuned verifier rule. The practical design the paper delivers is the write-set verifier, +4.2 over the vote at 2% of the bytes, which holds in every leave-one-model-out and leave-one-domain-out subset (+2.8 to +5.5).

Q2, the Theorem 1 ceiling. At the primary budget the 12-cell means are: logistic verifier 42.8, paid perfect verifier 43.6, free perfect verifier 53.2, Theorem 1 ceiling at three draws 59.4. The 16.6-point gap from the realistic verifier to the ceiling splits as 0.8 selector accuracy (42.8 to 43.6), 9.6 read cost (43.6 to 53.2), and 6.2 draw count (53.2 to 59.4), because tasks whose transcripts run longer than the median afford fewer than three draws. The ceiling itself is capped by tasks with no passing run in the pool, a task property. So the bottleneck is the price of a read, then the draws the budget buys, and selector quality last. We add this decomposition after Theorem 1.

Q3, write-set failure modes. Within a task, a write set that appears with both labels covers 3.4% of trajectories (216 of 6,428), so identical write sets with different correctness are rare in these benchmarks. The opposite is common: 32% of tasks with a passing run (356 of 1,115) have two or more distinct passing write sets, which is where a vote splits and a verifier need not. What the write set cannot see is a wrong answer given in text with correct database writes; that is why its AUC is 0.65 on retail and 0.93 on telecom, where the grader checks state. In deployment the same blind spot applies to any task graded on outputs rather than on state.

Q4, consistency against accuracy. Proposition 1 gives the condition: the two objectives disagree when operations lift different populations of tasks. On these benchmarks every operation lifts the same sometimes-tasks, so the pooled thresholds coincide. But the null is pooled, not universal: Table 16 shows +0.92 [+0.19, +1.70] on retail and -1.48 [-2.99, -0.31] on airline, with opposite signs, and we have corrected the sentence in Section 5 that said otherwise. Domains where a retry targets tool-error tasks that are hard while a verifier targets near-certain ones would separate the objectives; we name this in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.