acceptodds
Under review as a conference paper at ICLR 2027

ProbeCompress: Compressing Long-Horizon Agent Benchmarks with Task-Derived Probes

Abstract

Evaluating AI agents on long-horizon benchmarks consumes many tokens and substantial execution time because each task can require multiple model calls, tool invocations, and environment interactions. Task-subset methods evaluate fewer tasks, but each retained task still runs to completion. We introduce ProbeCompress, which predicts a new agent's full-benchmark score from short probes derived from the benchmark's own tasks. Each probe is built from the public context of one task and checks one requirement with its own verifier. Using responses from calibration models, ProbeCompress selects a small probe suite and fits a score predictor; a new model then runs only the selected probes. Across six agent benchmarks with up to 18 models, ProbeCompress predicts full-benchmark scores at a prespecified operating point with a mean absolute error of 0.028, while using 222.1× fewer tokens and 45.7× less summed execution work than full evaluation. The strongest complete-task baseline in our comparison reaches similar error while using more than ten times as many tokens. An interface fitted on 14 earlier models and reused unchanged on 4 later models reaches a mean absolute error of 0.044 without refitting. These savings exclude the one-time cost of building and calibrating the probe suite and apply to the evaluation setup used for calibration.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, benchmark compression, task-derived probes, long-horizon agents, efficient evaluation, score reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ccNQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

ProbeCompress predicts an agent's full long-horizon benchmark score from 64 short, LLM-generated probes derived from the benchmark's own tasks, using a ridge predictor fitted on ~17 calibration models. Across six benchmarks, it reaches 0.028 LOMO MAE with 222× fewer marginal tokens than full evaluation, and 0.044 MAE when the predictor is frozen and applied to 4 later models.

Critical strengths
  1. Practical problem. Full long-horizon agent evaluation is expensive, and subset methods still execute every retained task end-to-end. Changing the evaluation unit itself is a natural next step.

  2. Honest cost accounting. Marginal and setup costs are reported separately, wall-clock speedup is explicitly not claimed, and break-even points are given (Table S19).

  3. Thorough controls. Random probes, generic non-target probes, a two-parameter predictor, fully nested tuning, and all 3,060 14/4 splits rule out several obvious confounds.

Critical weaknesses
  1. The probes do not measure agent capability. By removing environment interaction, the worked examples reduce to static extraction a reasoning model with no agentic ability could solve. The method therefore works only through a reasoning–agency correlation in this frontier-only panel, and cannot detect differences that live in interaction, such as scaffold changes, which the intro names as a main use case. Parent association nearly vanishes on interaction-heavy OSWorld (Table S25: 0.058 vs. 0.21–0.26).

  2. This correlation breaks under realistic shift, undermining amortization. When fitted on earlier models and applied to later ones, MAE (0.044; 0.047 external) falls to the level of calibration-free Random-Subset Mean (0.047), stronger models are underestimated (Table S17), and close pairs are ranked at chance (Table S9). Yet setup costs 2,225 h (3.5× a full evaluation pass) and breaks even only after ~52 new systems (Table S19), assuming the interface stays valid, which the shift results suggest it does not.

Questions for authors
  1. What fraction of probes involve any environment interaction versus static extraction?
  2. Do probe scores change when the same model runs under a degraded scaffold or reduced step budget?
  3. How does a linear map from a cheap general-capability score compare?
  4. How much signal comes from the LLM rubric verifier, given deterministic verifiers raise MAE (Table S4)?
  5. Will the probe bank and code be released?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ccNQ (1/3)

Thank you for the detailed review. In short: (1) environment-action probes make up 31.4% of distinct selected probes, and excluding them raises macro MAE from 0.0280 to 0.0349. (2) A full-length MMLU-Pro LOMO baseline reaches 0.0385 MAE, versus 0.0280 for ProbeCompress (paired model-cluster 95% CI [+0.0018, +0.0205]). (3) The method is calibrated for new models under a fixed benchmark setup.

Weaknesses

1. What the probes measure.

A single probe does not measure end-to-end agentic competence. The suite predicts the aggregate benchmark score. ProbeCompress LOMO macro MAE is 0.0280, versus 0.0396 for non-target public-task probes (Table 3). Removing E probes and rerunning selection and fitting raises MAE to 0.0349 (OSW2: 0.0377 to 0.0511). The F+E-only condition reaches 0.0311, and E-only and S-only reach 0.0391 and 0.0409. Interaction and static evidence therefore both contribute. The full and Without-E conditions both keep K=64. E-only and S-only comparisons can also reflect smaller effective suites, and the F+E result is not K-matched.

2. Model diversity and general capability.

Our MMLU-Pro baseline uses all 12,032 questions zero-shot and fits a target-specific clipped affine map on calibration models only. Over all 102 development model-target cells, LOMO macro MAE is 0.0385 versus 0.0280 for ProbeCompress (paired difference +0.0105, 95% model-cluster CI [+0.0018, +0.0205]). ProbeCompress has lower MAE on all six targets. Eleven of eighteen models have a positive average paired difference, so model-level effects vary. Static probes alone do not beat this general-capability score (S-only 0.0409 vs 0.0385, with a smaller S-only suite on OSW2, Table R8). The advantage appears once file-read and environment-action probes are included (S+F 0.0349, F+E 0.0311, full suite 0.0280). A suite that worked only through static reasoning would not show this pattern. The frozen comparison is descriptive and is not part of this LOMO interval. Own-parent probe association is higher than re-paired association (0.441 vs 0.232, Table S25).

3. Scaffold changes.

We agree that the probe interface cannot detect scaffold changes. The probe runner uses a fixed instruction template, tool interface and execution limits, and shares only the model and decoding settings with the full-task agent. If the full-task scaffold or prompt changes, existing probe outcomes can be reused, but new full-task labels require target-dependent probe reselection and predictor refitting (Q2). We removed prompts and agent scaffolds from the motivating use cases and now state that the method targets new models under a fixed evaluation setup.

4. Weaker parent association on OSWorld.

OSW2 has 11 calibration systems per fold and the smallest usable bank (120 to 138 probes per fold, Table S30). The own-parent association stays above the re-paired value (0.362 vs 0.303), but the gap is the smallest in Table S25. OSW2 also shows the largest error increase when environment-action probes are removed. Its MAE rises from 0.0377 to 0.0511, and a bank of environment-action probes alone (0.0461 at K=56) outperforms the static-plus-file bank.

AuthorsOpenReview ↗

Response to Reviewer ccNQ (2/3)

Weaknesses (continued)

5. Reuse on later models.

On the five shared targets, ProbeCompress MAE is 0.0260 in LOMO and 0.0436 when the interface is fitted on 14 earlier-accessed models and applied unchanged to four later ones. This test is largely an extrapolation, since 13 of its 20 model-benchmark scores lie above the calibration maximum (point 6). All three calibrated methods in Table 4 lose accuracy relative to five-target LOMO, and ProbeCompress loses the least (+0.0176, against +0.0246 for EssenceBench-adapted and +0.0282 for PACE-Matched). Over all 3,060 possible 14/4 splits of the development panel, the median is 0.0416 (IQR 0.0400 to 0.0431, central 95% 0.0373 to 0.0461, Table S15), so the access-ordered split is not an outlier.

The 0.047 Random-Subset figure in the review is its value on the full development panel (Table 1). On the same four later models, Random-Subset Mean reaches 0.0524 (mean over 100 draws of 64 tasks). The difference from ProbeCompress is +0.0088, with a four-model interval of [−0.0038, +0.0219]. On the separate external holdout (four disjoint models, six targets), MAE is 0.0548 for Random-Subset and 0.0468 for ProbeCompress (difference +0.0080, interval [−0.0045, +0.0203]). Both intervals include zero, so these four-model comparisons do not establish an accuracy advantage. They do show that ProbeCompress is at least as accurate at a much lower cost. Random-Subset Mean runs 64 complete tasks per target, which takes about 44× more marginal tokens and 9× more summed work than ProbeCompress on the five frozen targets, and 57× and 12× on all six (Tables 2, S18, S24).

6. Underestimation above the calibration range.

On the frozen split, the 13 above-maximum cells have signed error −0.0193 and MAE 0.0542, versus −0.0021 and MAE 0.0239 for the seven within-range cells. On the external holdout, nine above-maximum cells have signed error −0.0278 and MAE 0.0668, versus +0.0012 and MAE 0.0380 for 13 within-range cells. Two below-minimum cells have signed error +0.0114 and MAE 0.0140. Appendix E.4 now reports these results. Since a new model's true position is unknown before full evaluation, we recommend full evaluation when a prediction lies near the calibration ceiling.

7. Close pairs.

For the 40 pairs whose reference scores differ by less than 0.02, all three methods score 48.75% (half credit for ties). A fresh full-benchmark run orders the same pairs correctly only 57.5% of the time, and 75.5% for gaps of 0.02 to 0.05. Near-tie ordering is therefore unstable in this sample even under full evaluation. This does not establish a general noise floor, and the weakness still matters for model selection with ProbeCompress. Table S9 uses the shared-interface pair holdout inside the development panel, not the frozen split. For the 695 pairs with gaps of at least 0.05, ProbeCompress reaches 94.68%. Section 4.4 now reports this gap dependence and when to run a full evaluation (see our response to Reviewer C9VJ).

8. Setup cost and break-even.

The figure of about 52 systems includes collecting three-pass historical labels for the calibration panel. Every calibrated method in our comparison needs these labels, including Original-Task Matched and EssenceBench-adapted. The setup specific to ProbeCompress costs 162.5M tokens and 2,225 h of summed work. It breaks even against one-pass full evaluation after 0.85 additional systems by tokens and 3.61 by summed work. With label collection included, the thresholds are 52.28 and 55.58 (Table S19). The Introduction now reports both cases next to the marginal ratios.

9. Keeping the interface current.

Reuse assumes that the relationship learned under the fixed evaluation setup still holds. Periodic full evaluations check for drift, and each fully evaluated system can be added to the calibration panel. Any change to the evaluation setup requires recalibration. Our response to Reviewer C9VJ (point 3) lists what can be reused for each type of change.

AuthorsOpenReview ↗

Response to Reviewer ccNQ (3/3)

Q1. Interaction versus static extraction.

Of 1,050 distinct selected probes, 323 (30.8%) are static, 397 (37.8%) read local files and 330 (31.4%) require environment actions. With tools disabled, E-probe pass rates fall from 50.70% to 0% for GPT-5.5 and from 28.27% to 0% for Qwen3.8-27B. This zero follows from the environment-state verification contract and is not separate predictive evidence. File-read pass rates fall from 63.28% to 40.27% and from 42.18% to 23.34%, and static rates change by at most 0.23 percentage points. Removing E probes raises macro MAE to 0.0349, while F+E-only reaches 0.0311. Appendix B.3.3 reports the executed WebArena-Pro probe WAPRO-0246-E1 (parent 20260501095702013727). The agent navigated Mattermost to Jane's direct messages and returned the header Jane and the route /mock-workspace/messages/@jane. Candidate identification and notifying Jane remain outside the probe. Section 3.2 now presents this executed probe in the main text, so the main-text examples are no longer only static.

Q2. Degraded scaffold or reduced step budget.

We have not run a separate scaffold-shift experiment. By design, changing only the full-task scaffold or its step budget does not change the fixed probe runner. The runner shares the model and decoding settings but has its own instructions, tool interface and execution limits. Existing calibration probe outcomes can therefore be reused, although a new stochastic run need not return identical outcomes. Full-task performance can change, so new full-task calibration labels, probe reselection, preprocessing and predictor refitting are required. If shared decoding settings change, both calibration probe responses and full-task labels must be regenerated. The tools-on/tools-off diagnostic in Q1 changes the probe runner itself and is not a test of full-task scaffold shift. The Introduction, Limitations and Appendix F.2 now state this scope.

Q3. A linear map from general capability.

We evaluate all 12,032 MMLU-Pro questions per development model with zero-shot, temperature-zero accuracy. A target-specific two-parameter clipped affine map is fitted on calibration models only, excluding the evaluated model in LOMO. Across 18 development models and 102 model-target cells, MMLU-Pro MAE is 0.0385 versus 0.0280 for ProbeCompress (paired difference +0.0105, 95% model-cluster bootstrap CI [+0.0018, +0.0205], 100,000 resamples). ProbeCompress has lower mean error on all six targets, and its model-level average error is lower for 11 of 18 models. With the same mapping, frozen 14→4 MAE is 0.0511 versus 0.0436. This four-model comparison is descriptive and is not covered by the LOMO interval. The baseline uses the complete question set, so it is not limited by a small evaluation budget. We did not test cheaper MMLU-Pro subsets or a five-shot chain-of-thought protocol. Table R6 in Appendix H.6 gives per-target results.

Q4. Contribution of the rubric verifier.

Accurate reconstruction remains possible with deterministic verification. Restricting the bank to deterministic probes and rerunning the pipeline raises macro MAE by 0.0030, from 0.0280 to 0.0309 (Table 3). Keeping the original probes and replacing rubric scoring with deterministic checks gives 0.0314 (Table S4). The deterministic-only result is still below Original-Task Matched (0.0337) and close to EssenceBench-adapted (0.0304), and both of these execute complete tasks. Rubric-verified probes are 276 of the 1,050 distinct selected probes (26.3%). Our response to Reviewer QFWM reports how often each verifier type can be fooled.

Q5. Release.

Upon publication, we plan to release the 2,665 accepted probe specifications, the code for generation, filtering, selection and calibration, the deterministic verifiers and frozen rubrics, the evaluation configurations and benchmark manifests, and the scripts and records we are allowed to redistribute. Proprietary model outputs and benchmark assets with redistribution restrictions are excluded. The reproducibility statement now specifies this scope.

Reviewer_QFWMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This is a very novel idea to solve a very relevant and common problem in AI evaluation. The paper proposes ProbeCompress - a way to use probes and estimate the model's score without running the evaluation end to end. The paper is very clear and thorough, with strong experimental evidence supporting the approach. They demonstrate their appraoch using around 6 different benchmarks and multiple models. I belive this is a very interesting idea and is worth broader discussion and dissemination among AI evaluation researchers.

Critical strengths

The idea behind this paper is novel and unique. Replacing tasks with task-derived surrogates is a very interesting idea. The experiments are very comprehensive in the sense that the authors used 6 benchmarks and 10+ models. They have been very thorough in this sense. The approach provides clear gains over other approaches like PACE.

Critical weaknesses

The paper lacks a clear study on the shortcomings of these probes - how broken are these probes and are these probes more easily reward hacked? Does that affect the quality of evaluation? It would also be nice to see some adversarial tests and better audit coverage of these probes.

Questions for authors

My main questions are around the validity and correctness of probes. I would have liked to see a more thorough understanding of the problems with these probes and how quality control can be used to improve them.

1 comment
AuthorsOpenReview ↗

Response to Reviewer QFWM

Thank you for asking us to examine probe quality directly. Appendix H.5 now reports a new audit of selected probes, adversarial verifier tests and a removal experiment.

1. Audit of selected probes.

A new audit of 150 of the 1,050 distinct selected probes, stratified by benchmark and verifier type, found 9 that fail the semantic criteria (6.0% of the audited sample, Wilson 95% interval 3.2% to 11.0%). The earlier pre-freeze audit covered a different population of candidates and rejected 8 of 240 (3.3%, Table S29), so the two audits are not a paired comparison. Two auditors independently assessed 30 shared probes and agreed on 28 (93.3%, Cohen's κ = 0.63). The nine failures were four ambiguous requests, two verifiers that accepted incorrect artifacts, one verifier that rejected a correct artifact, one requirement not supported by the public context and one incorrect input configuration.

2. Reward hacking and adversarial tests.

We tested 200 distinct selected probes (100 with rubric verifiers and 100 with deterministic verifiers), with three attack artifacts per probe: an injected instruction, an empty artifact and a copy of the input. Among the 600 attack artifacts there were 11 false accepts (1.83%) on nine distinct probes.

AttackRubric verifierDeterministic verifier
Prompt injection6/1000/100
Empty artifact1/1001/100
Copy of the input2/1001/100

Positive controls with 200 correct artifacts (100 per verifier type) produced four rubric and one deterministic false reject. All five occur on probes that are already in the 16-probe removal set below (SWEV-0446-F2, WAPRO-0187-E2, GAIA-VAL-093-E1, SWEV-0261-S1 and SWEMM-0112-F1). Deterministic verifiers accepted 0 of 100 prompt-injection artifacts, versus 6 of 100 for rubric verifiers. This does not establish immunity to other attacks, and full-task evaluators were not part of this test.

3. Effect on evaluation quality.

We removed the nine audit failures and the nine probes with false accepts. Two IDs overlap, which leaves 16 distinct probes, and this set also covers all five false-reject probes. We reran filtering, selection and fitting at K=64 in every fold. Macro MAE moved from 0.02796 to 0.02844 (+0.00048), and OSW2 MAE from 0.03769 to 0.03963. These results cover the identified defects only, not all possible verifier failures.

4. Execution noise and timeout sensitivity.

Three additional runs of the selected suites give macro MAE 0.0279, versus 0.0280 in the primary run. Across the three repeats, 7.26% of model-probe pairs have non-identical binary outcomes (benchmark mean, Table S28). The 120-second environment-execution cap was reached in 512 of 6,528 primary executions (7.84%) and 2,309 of 19,584 repeats (11.79%). Rerunning the 512 affected primary executions with the same cap gave 338 completions and 174 repeat timeouts. Replacing those outcomes with selection and predictor fixed changed macro MAE from 0.0280 to 0.0278 (OSW2: 0.0377 to 0.0369). A completed retry does not imply a passed verifier.

5. Quality control going forward.

Appendix B.4 now recommends adversarial acceptance checks for future banks. Each negative control must first be confirmed as incorrect under the frozen probe contract. An artifact that satisfies the contract is not a false accept just because it contains an instruction, is empty or copies the input. Probes whose verifiers accept invalid controls should be repaired and retested before the bank is frozen, or excluded. Correct artifacts serve as separate positive controls. For the current bank, Appendix H reports the sampled tests and the removal experiment next to the construction audit.

Reviewer_C9VJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProbeCompress estimates a model's full agent-benchmark score from short probes derived from the benchmark's public task context. Each probe tests a bounded requirement and has its own verifier. Responses from calibration models guide probe selection and a score predictor, after which the interface is frozen for new models. Across six benchmarks, the main 64-probe configuration reports 0.028 mean absolute error, with substantially lower marginal token consumption and summed execution work than full evaluation. The paper also evaluates nested tuning, simpler predictors, alternative probe sources, and reuse on held-out models. The resulting estimate is conditional on a fixed evaluation setup.

Critical strengths
  • The contribution is distinct from selecting fewer complete tasks. A retained coding or browser task can still be expensive, whereas these probes remove much of the interaction itself. The worked examples in Appendix B.3 make the change in evaluation unit explicit and identify which parts of the original task remain untested.

  • The experimental controls address several obvious alternative explanations. Random accepted probes and generic probes perform worse than the selected task-derived suite. The two-parameter predictor reaches 0.0290 MAE, close to the main result, and fully nested tuning reaches 0.0293. These results support the usefulness of the selected probe responses and reduce concern that the headline accuracy depends entirely on a flexible regression head.

  • The evaluation handles held-out systems carefully. Filtering, selection, scaling, and fitting are repeated within each outer model holdout. Pairwise and three-model ranking evaluations use a shared interface fitted without any of the compared systems. The frozen reuse and separate external holdout go beyond ordinary leave-one-model-out testing, although their small size still limits generalization claims.

Critical weaknesses
  • Table 9 reports pairwise accuracy of 0.488 when the true score gap is below 0.02, and 0.719 for gaps between 0.02 and 0.05. Of the 831 model pairs, 561 have gaps of at least 0.10, where ordering is much easier. The overall 0.898 pairwise accuracy is therefore a poor description of performance when comparing similar checkpoints. This is a limitation for model selection even though the paper provides the necessary diagnostic. The main text should state it more directly and explain when a full evaluation remains necessary.

  • Table 19 gives break-even points of about 52 additional systems by tokens and 56 by summed work when historical labels are included. If those labels already exist, the corresponding values are about one and four systems. Both calculations are useful, but they describe very different users. Since the interface is tied to a fixed benchmark and evaluation setup, the paper should explain when a change requires recalibration and which setup costs can be reused. The large marginal compression ratio alone does not answer that deployment question.

Questions for authors

Plz see the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer C9VJ

Thank you for the concrete suggestions on pairwise ordering, on when to run full evaluations and on presenting setup costs.

1. Pairwise ordering by true score gap.

Section 4.4 now states the close-pair limitation directly. Table S9 also includes a fresh full-benchmark run scored against the same reference order (half credit for ties).

True score gapPairsProbeCompressFresh full run
below 0.024048.75% (19.5/40)57.50% (23/40)
0.02 to 0.059671.88% (69/96)75.52% (72.5/96)
0.05 to 0.1013478.36% (105/134)79.85% (107/134)
0.10 or more56198.57% (553/561)98.22% (551/561)

Even a fresh full run orders pairs with gaps below 0.02 correctly only 57.5% of the time. Ordering of near-tied systems is unstable in this sample, but this does not establish a general noise floor or remove the limitation for ProbeCompress. For gaps of at least 0.05 (695 of 831 pairs, 83.6%), ProbeCompress reaches 94.68%.

2. When to run a full evaluation.

In a retrospective analysis, predicted-gap screening at τ=0.02 flags 145 of 831 comparisons (17.45%) and leaves 686 with 92.78% ordering accuracy. At τ=0.03, 218 are flagged (26.23%) and unflagged accuracy rises to 94.29%. At τ=0.05, 345 are flagged (41.52%) and unflagged accuracy is 96.40%. The overall ordering score stays at 746.5/831 for every threshold, because screening only marks which pairs need further evaluation. The flagged share counts comparisons, not distinct systems. For near-tied systems (true gap below 0.02), even a fresh full run reaches only 57.5%, so we recommend repeated evaluations or reporting the systems as indistinguishable. A prediction near the calibration ceiling is another reason for full evaluation, although we have not validated it as an extrapolation detector. We have not evaluated the accuracy or cost of a combined screening and full-evaluation policy. Section 4.4 now ends with a paragraph, "Close comparisons and when to run a full evaluation", that gives this guidance and the recalibration triggers in the main text.

3. Recalibration and reusable setup costs.

If historical labels already exist, the setup breaks even after 0.85 additional systems by tokens and 3.61 by summed work. If the labels must be collected, the thresholds are 52.28 and 55.58 (Table S19). Appendix F.2, referenced from the Introduction and Limitations, now lists what each change requires.

New model under the same setup. Run the selected probes, with periodic full evaluations to check that the interface still applies.

Full-task scaffold, prompt or execution limits change. The probe runner stays fixed, so compatible probe specifications and calibration probe outcomes are reused. Full-task labels for the calibration panel are recomputed, followed by probe reselection, preprocessing and predictor refitting.

Shared decoding settings change. Both probe outcomes and full-task labels change. Probe construction (245 h, including the audit) is reused, while calibration probes (1,980 h) and full-task labels are rerun before filtering, selection and fitting.

Benchmark release changes. Compatible specifications for unchanged tasks are reused, probes for changed tasks are rebuilt, and the affected labels and calibration outcomes are updated before reselection and fitting.

4. Interpreting the compression ratio.

The 222.1× token and 45.7× summed-work ratios compare marginal evaluation with a fresh full pass and exclude setup. The Introduction now gives both break-even cases next to these ratios.

5. Holdout size.

The access-ordered test has four later-accessed models on five targets, and the external test has four disjoint models on all six targets. On the external test, macro MAE is 0.0468 for ProbeCompress, 0.0587 for EssenceBench-adapted and 0.0698 for PACE-Matched (Table S16). With four models each, we treat both holdouts and their intervals as descriptive. As the Limitations section states, generalization to future systems remains open.

Reviewer_C9VJOpenReview ↗

Response

Thanks for addressing the concerns. I would like to keep my rating at 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all three reviewers. Changed text in the revised PDF is marked in blue. Tables added during the discussion are numbered R1 to R8, so the references in our responses stay stable. The main text stays within the 10-page limit, and the numbering of figures, Tables 1 to 5 and Tables S1 to S32 is unchanged.

Main text. The Introduction states that the interface targets new models under a fixed benchmark setup, no longer lists prompts or agent scaffolds as use cases, and gives both break-even cases next to the marginal ratios (ccNQ, C9VJ). Section 3.2 reports the shares of static, file-read and environment-action probes and presents an executed WebArena-Pro probe next to the static example (ccNQ). Section 4.3 adds the environment-action ablation, a full-length MMLU-Pro baseline and the effect of removing probes flagged by a new audit and adversarial tests (ccNQ, QFWM). Section 4.4 adds the matched Random-Subset comparison and a paragraph on close comparisons that explains when a full evaluation or recalibration is needed (C9VJ, ccNQ).

Presentation. Benchmark abbreviations and LOMO are defined at first use, dense passages in Sections 3.1 and 3.4 are simplified, and Table 5 uses one precision. The Limitations are grouped into scope, statistical evidence and use, and the Conclusion describes frozen reuse as descriptive evidence (all reviewers).

Appendix. B.3.3 describes the executed WebArena-Pro probe and B.4 recommends adversarial checks for future banks. E.3 and E.4 add matched Random-Subset comparisons (Table R2) and errors by calibration-range position (Table R3). F.2 covers deployment, recalibration triggers and reusable setup costs. H.5 reports the selected-probe audit, adversarial verifier tests and removal experiment (Tables R4 and R5). H.6 reports the MMLU-Pro baseline (Table R6), and H.7 the probe-class ablation and tools-on/tools-off controls (Tables R7 and R8). Table S9 adds a fresh full-run column, Table R1 reports predicted-gap screening, and Table S28 is followed by the timeout analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.