- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks how oversight should change when a strong agent spends more test time compute trying to get a wrong answer past a weaker judge. It examines 2 cases: (1) best-of-N, where the agent picks the candidate it thinks the judge will most likely accept, and (2) longer thinking before a single answer. In both of these cases the agent never queries the judge. Using a Qwen3 (14B) agent and mostly Qwen3 (4B) judges, the paper finds that FAR rises 1.8-5.7x from N=1 to N=64 - and that judge failures concentrate on a minority of problems that the agent can find on its own. The main theorem bounds FAR at a large search budget using only FAR at a smaller one. SARCO uses this bound with standard risk-control tests to pick the cheapest verification protocol that meets both the false-acceptance and the honest-acceptance targets, or otherwise to abstain. On held out problems, every protocol SARCO returns meets both targets, though it returns 1 in only 27.5% of decisions at N=64. For longer thinking (which the theorem does not cover) protocols certified at a low reasoning budget can fail at higher budgets
Critical strengths
-
A correct transfer bound that gives the controller a very good guarantee: theorem 1 needs only the risk at the smaller budget, and the bound for this is very sharp. Precisely this is what lets SARCO certify a large search budget from a small calibration budget. As a result, its returned protocols come with a guarantee and don't depend on a forecast
-
Clear diagnostic findings: self-ranking raises FAR on every dataset (fig 2d), and the top 25% of problems hold most held-out failures (fig. 2c). Judge reasoning also raises FAR on GPQA (closed) and MMLU (fig. 4). These results are useful for anyone designing oversight evaluations - and the paper reports the negative results plainly
Critical weaknesses
Weakness 1: The worst-case bound makes SARCO too conservative in most settings.
The bound assumes all acceptances sits on the top-scored candidates, so it turns a 10% target at N=64 into 2.6% at N=16. Fig 2a suggests real FAR grows by significantly under 2x over that range - while the bound has to allow close to 4x. Consequently, this also sets a minimum number of test problems: appendix B4 requires 88 problems at N=64. GPQA, MMLU, and MATH have only about 29 to 45 test problems each, so SARCO can't return anything there at large N - no matter whatever the protocols are. The positive results, therefore, come only from logic and arithmetic. This problem is built into using a worst-case bound.
Weakness 2: the reliability gain over a valid baseline rests on very few decisions.
Table 1 appears to cover about 51 decisions at N=64. On that reading, SARCO’s 100% means 14 returns with no failures, and similarly direct high-N risk control’s 90% means 2 failures in 20 returns. Direct high-N calibrates at the deployment budget, so it is also a valid method, and 2 failures in about 51 decisions is within its own guarantee. Over the full grid (table 6), it fails in only 1.2% of returns and returns protocols more often. The paper does not show why SARCO is better than simply calibrating at the deployment budget. I believe that failure counts with intervals and a comparison at equal cost could fix this.
Questions for authors
- How many distinct incorrect candidates does each problem’s bank hold, and how often is it fewer than 64? Search is simulated by resampling from these banks. Because if they are small, FAR at N=64 may understate real best-of-64 risk, and direct high-N calibration may not need any extra generation
- Prompts A1 to A3 tell the agent that the reviewer “checks one randomly chosen step,” which describes the audit protocols. How does this fit the claim that the agent has no protocol information, and does the protocol ranking in figure 2b change with a neutral prompt? And also, how is the self-score S computed?
- Can you please report decisions, returns, and failures per selector with confidence intervals, and compare SARCO with direct high-N risk control at equal total cost? If direct high-N matches SARCO’s reliability with higher coverage, what is the remaining case for search transfer?
5 comments
Author Response - Part 2
[W2] The reliability gain over a valid baseline rests on very few decisions....
We agree completely with your statistical reading and have removed any suggestion that SARCO is more reliable than direct high- risk control. Both methods satisfy the same finite-sample guarantee:
1. Reliability parity with exact counts and confidence intervals. In the submitted Table 1 ( decisions at ), SARCO records 0 failures in 14 returns (0.0% [0.0%, 23.2%]) and direct high- risk control records 2 failures in 20 returns (10.0% [1.2%, 31.7%])—a non-significant difference (Fisher's exact ) that sits well within across 51 decisions. Our expanded 100-split evaluation across eight open-judge regimes (96 configurations, 9,600 decisions; Table R1 in the General Response) makes this parity precise: direct high- fails in 6 of 2,585 returns (0.2% [0.1%, 0.5%]) and SARCO () fails in 3 of 1,409 returns (0.2% [0.0%, 0.6%]). Neither method exceeds in any of the 96 configurations, whereas heuristic and forecasting selectors violate in 6 to 73 of 96 configurations.
2. Equal-cost calibration and the case for search transfer. When direct high- is restricted to the same total number of judged calibration candidates as SARCO (, reducing direct high-'s calibration problem count by ), SARCO achieves slightly higher return coverage. Direct high- is preferable when deployment can be generated and judged offline, whereas SARCO calibrates once at to certify larger budgets without extra candidate generation.
3. Why search-unaware risk control is unsafe in general (Table R12).
Although omitting Theorem 1 (Search-unaware risk control) happened to incur few violations on our real benchmarks (0.3% failures; 0/96 configurations ), that safety is purely empirical. On synthetic regimes near Theorem 1's worst case—where problems accept their highest-scored incorrect candidates—search-unaware risk control returns a protocol that violates the deployment target at in 98.5%–99.5% of 200 replicates, whereas SARCO and direct high- incur 0 violations (Table R12).
Table R12: Synthetic regimes near the Theorem 1 worst case (200 calibration replicates per row).
| Regime | Selector | Returns (of 200) | Replicates Missing Target at (of 200) [95% CI] |
|---|---|---|---|
| Worst case, no spread | Search-unaware risk control | 200 | 199 (99.5%) [97.2%, 100%] |
| Worst case, no spread | SARCO () | 158 | 0 (0.0%) [0.0%, 1.8%] |
| Worst case, no spread | SARCO, linear margin | 158 | 0 (0.0%) [0.0%, 1.8%] |
| Worst case, no spread | Direct high- risk control | 200 | 0 (0.0%) [0.0%, 1.8%] |
| Worst case, spread 0.5 | Search-unaware risk control | 200 | 197 (98.5%) [95.7%, 99.7%] |
| Worst case, spread 0.5 | SARCO () | 160 | 0 (0.0%) [0.0%, 1.8%] |
| Worst case, spread 0.5 | SARCO, linear margin | 160 | 0 (0.0%) [0.0%, 1.8%] |
| Worst case, spread 0.5 | Direct high- risk control | 200 | 0 (0.0%) [0.0%, 1.8%] |
| , test probs. | SARCO (sharp margin) | 200 | 0 (0.0%) [0.0%, 1.8%] |
| , test probs. | SARCO, linear margin | 0 | 0 (0.0%) [0.0%, 1.8%] |
4. Scaling deployment up to (Table R13). We also generated fresh 256-candidate banks on 100 problems per task and judged every candidate across the full menu (50 splits, 200 decisions per cell; Table R13). Because 15% of 100 problems leaves only 13–17 eligible test problems—below even at —SARCO and the other risk-controlled selectors abstain in of decisions with 0 failures (illustrating the boundary on small splits). By contrast, all four heuristic/forecasting selectors continue returning protocols whose failure rates climb steeply with : on GSM8K, from 37.0%–38.7% at to 87.6%–92.6% at .
Table R13: Failure rate among returned protocols as deployment search scales to .
| Task Selector | |||||
|---|---|---|---|---|---|
| GSM8K Outcome-based | 38.7% | 66.7% | 71.0% | 75.8% | 90.3% |
| GSM8K Tail-fit forecast | 37.6% | 60.3% | 64.1% | 75.6% | 92.6% |
| GSM8K Search-aware forecast | 38.2% | 60.9% | 66.5% | 71.8% | 87.6% |
| GSM8K Pooled | 37.0% | 56.0% | 63.4% | 71.1% | 91.6% |
| GSM8K SARCO, returns (failures) | 1 (0) | 0 | 0 | 0 | 0 |
| PrOntoQA Outcome-based | 15.0% | 18.0% | 21.0% | 32.0% | 40.0% |
| PrOntoQA Tail-fit forecast | 16.0% | 16.0% | 21.0% | 28.0% | 31.0% |
| PrOntoQA Search-aware forecast | 6.0% | 7.0% | 7.0% | 9.0% | 12.0% |
| PrOntoQA Pooled | 1.0% | 0.0% | 1.0% | 8.0% | 40.0% |
| PrOntoQA SARCO, returns (failures) | 0 | 0 | 0 | 0 | 0 |
Author Response - Part 3
[Q1] How many distinct incorrect candidates does each problem’s bank hold, and how often is it fewer than 64? Search is simulated by resampling from these banks. Because if they are small, FAR at N=64 may understate real best-of-64 risk, and direct high-N calibration may not need any extra generation
Formally, FAR is defined on the empirical retained-bank distribution (Appendix A.3), so Theorem 2's validity guarantee holds strictly for that endpoint. However, we agree with both of your practical points: bootstrap resampling from finite banks understates fresh best-of-64 generation, and in our cached-bank harness direct high- reused the same pre-generated banks without extra candidate generation.
1. Retained bank sizes and distinct draws (Table R14). With the Qwen3-14B attacker, 8.3% of GSM8K banks and 85.8% of PrOntoQA banks contain fewer than 64 retained incorrect candidates (median 64 on GSM8K and 56.5 on PrOntoQA, as candidate filtering discards valid or off-target derivations). Consequently, 64 draws with replacement visit an average of 40.6 distinct candidates on GSM8K and 36.4 on PrOntoQA (drawing the bank's single highest-scored candidate in 64% and 71% of draws, respectively). On our expanded 256-candidate banks, 0.0% of GSM8K and 1.0% of PrOntoQA banks have fewer than 64 candidates, and 64 draws with replacement visit 56.7 and 54.4 distinct candidates.
Table R14: Retained incorrect candidates per problem across banks. Distinct in 64 draws and Top candidate drawn in 64 reflect i.i.d. sampling with replacement at .
| Task | Attacker | Problems | Median | Min | Distinct in 64 Draws | Top Candidate Drawn () | ||
|---|---|---|---|---|---|---|---|---|
| GSM8K | Qwen3-14B | 600 | 64 | 55 | 0.0% | 8.3% | 40.6 | 64% |
| GSM8K | Phi-4 | 600 | 64 | 40 | 0.0% | 35.3% | 40.4 | 64% |
| GSM8K | GPT-4.1-mini | 600 | 64 | 50 | 0.0% | 7.0% | 40.6 | 64% |
| PrOntoQA | Qwen3-14B | 500 | 56.5 | 1 | 1.2% | 85.8% | 36.4 | 71% |
| PrOntoQA | Phi-4 | 500 | 29 | 12 | 0.8% | 100% | 25.6 | 89% |
| PrOntoQA | GPT-4.1-mini | 500 | 58 | 24 | 0.0% | 96.8% | 38.6 | 68% |
| GSM8K (256-cand. bank) | Qwen3-14B | 100 | 256 | 242 | 0.0% | 0.0% | 56.7 | 22% |
| PrOntoQA (256-cand. bank) | Qwen3-14B | 100 | 216 | 11 | 1.0% | 1.0% | 54.4 | 28% |
2. Measuring the resampling understatement gap on 256-candidate banks (Table R15).
Using the 256-candidate runs (Qwen3-14B Qwen3-4B), we compared FAR under three sampling schemes: (i) resampling with replacement from the first 64 candidates (64-bank), (ii) resampling with replacement from the full 256-candidate bank (Full bank), and (iii) drawing 64 fresh candidates without replacement (Fresh). Resampling from a 64-candidate bank understates fresh 64-candidate risk by at most 4 percentage points on GSM8K (21% vs. 25% under direct judgment; 0–3 points across audit and reasoning protocols) and at most 1 percentage point on PrOntoQA (8% vs. 9% under direct judgment; 0 points across all audit and reasoning protocols).
3. Shared banks vs. deployment generation cost for direct high-.
We explicitly note in Sec. 5 and Table 1 that in our benchmark evaluations, all selectors draw from shared pre-generated candidate banks—so Cal. draws in Table 1 measures judged draws per calibration problem from shared banks, and direct high- required no extra LLM generation in the experimental harness. In prospective deployment where 64-candidate banks do not already exist, direct high- requires generating, filtering, and judging fresh candidates per calibration problem, whereas SARCO requires only .
Table R15: FAR under 64-bank resampling, full-bank resampling, and fresh draws without replacement. Evaluated on the 256-candidate banks (Qwen3-14B Qwen3-4B; 100 GSM8K problems and the 99 PrOntoQA problems with retained candidates).
| Verification Protocol | GSM8K (64-bank) | GSM8K (Full bank) | GSM8K (Fresh) | PrOntoQA (64-bank) | PrOntoQA (Full bank) | PrOntoQA (Fresh) |
|---|---|---|---|---|---|---|
| Direct judgment | 21% | 24% | 25% | 8% | 9% | 9% |
| Judge + uniform audit () | 12% | 14% | 15% | 6% | 6% | 6% |
| Judge + optimized audit () | 7% | 9% | 9% | 3% | 3% | 3% |
| Judge + optimized audit () | 7% | 9% | 9% | 1% | 1% | 1% |
| Judge + uniform audit () | 8% | 8% | 8% | 3% | 3% | 3% |
| Judge reasoning () | 5% | 7% | 8% | 2% | 2% | 2% |
| Judge + every-step audit | 5% | 5% | 6% | 0% | 0% | 0% |
Manuscript revisions: We added Tables R14 and R15 to Appendix A.3, noted the finite-bank resampling understatement in Sec. 5, and clarified in Table 1's caption that Cal. draws counts judged draws from shared candidate banks.
Author Response - Part 1
Thank you for the thorough and constructive review. We are glad you valued Theorem 1's sharp distribution-free search-transfer guarantee and found our diagnostic results on query-free self-ranking, concentrated blind spots, and judge-reasoning failures informative, as well as our candid reporting of negative results.
Below, we address your concerns and hope these additional experiments and clarifications resolve your concerns and support a re-evaluation of our work. All updates have been incorporated into the revised manuscript and appendix.
We agree completely with your diagnosis of the empirical numbers: across our real evaluation pairings, self-ranked FAR grows by 1.0×–1.6× from N=16 to N=64 (Appendix C.3; Table R2 in our response to Reviewer JVsC), whereas Theorem 1 must guard against roughly a 3.85× increase (2.60%→10%), and our positive returns at large N come exclusively from logic (PrOntoQA) and arithmetic (GSM8K). We highlight three points clarifying what drives this behavior and when it is avoidable:
[Q1]). Table R12 demonstrates such regimes empirically, where selectors without the worst-case transfer margin fail on nearly every split.Table R11: Test-split size vs. zero-loss sample floor (m=16,α=0.10,δ=0.10) alongside SARCO's N=64 decisions. The statistical-test split comprises 15% of dataset problems prior to eligibility filtering (Appendix A.3).
Manuscript revisions: We updated Sec. 6 to explicitly state that positive returns come from logic and arithmetic, report coverage conditioned on clearing the zero-loss sample floor, and incorporate Table R11 (upgrading Table 8 in Appendix A.7).