- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
ProbeCompress predicts an agent's full long-horizon benchmark score from 64 short, LLM-generated probes derived from the benchmark's own tasks, using a ridge predictor fitted on ~17 calibration models. Across six benchmarks, it reaches 0.028 LOMO MAE with 222× fewer marginal tokens than full evaluation, and 0.044 MAE when the predictor is frozen and applied to 4 later models.
Critical strengths
-
Practical problem. Full long-horizon agent evaluation is expensive, and subset methods still execute every retained task end-to-end. Changing the evaluation unit itself is a natural next step.
-
Honest cost accounting. Marginal and setup costs are reported separately, wall-clock speedup is explicitly not claimed, and break-even points are given (Table S19).
-
Thorough controls. Random probes, generic non-target probes, a two-parameter predictor, fully nested tuning, and all 3,060 14/4 splits rule out several obvious confounds.
Critical weaknesses
-
The probes do not measure agent capability. By removing environment interaction, the worked examples reduce to static extraction a reasoning model with no agentic ability could solve. The method therefore works only through a reasoning–agency correlation in this frontier-only panel, and cannot detect differences that live in interaction, such as scaffold changes, which the intro names as a main use case. Parent association nearly vanishes on interaction-heavy OSWorld (Table S25: 0.058 vs. 0.21–0.26).
-
This correlation breaks under realistic shift, undermining amortization. When fitted on earlier models and applied to later ones, MAE (0.044; 0.047 external) falls to the level of calibration-free Random-Subset Mean (0.047), stronger models are underestimated (Table S17), and close pairs are ranked at chance (Table S9). Yet setup costs 2,225 h (3.5× a full evaluation pass) and breaks even only after ~52 new systems (Table S19), assuming the interface stays valid, which the shift results suggest it does not.
Questions for authors
- What fraction of probes involve any environment interaction versus static extraction?
- Do probe scores change when the same model runs under a degraded scaffold or reduced step budget?
- How does a linear map from a cheap general-capability score compare?
- How much signal comes from the LLM rubric verifier, given deterministic verifiers raise MAE (Table S4)?
- Will the probe bank and code be released?
5 comments
Response to Reviewer ccNQ (2/3)
Weaknesses (continued)
5. Reuse on later models.
On the five shared targets, ProbeCompress MAE is 0.0260 in LOMO and 0.0436 when the interface is fitted on 14 earlier-accessed models and applied unchanged to four later ones. This test is largely an extrapolation, since 13 of its 20 model-benchmark scores lie above the calibration maximum (point 6). All three calibrated methods in Table 4 lose accuracy relative to five-target LOMO, and ProbeCompress loses the least (+0.0176, against +0.0246 for EssenceBench-adapted and +0.0282 for PACE-Matched). Over all 3,060 possible 14/4 splits of the development panel, the median is 0.0416 (IQR 0.0400 to 0.0431, central 95% 0.0373 to 0.0461, Table S15), so the access-ordered split is not an outlier.
The 0.047 Random-Subset figure in the review is its value on the full development panel (Table 1). On the same four later models, Random-Subset Mean reaches 0.0524 (mean over 100 draws of 64 tasks). The difference from ProbeCompress is +0.0088, with a four-model interval of [−0.0038, +0.0219]. On the separate external holdout (four disjoint models, six targets), MAE is 0.0548 for Random-Subset and 0.0468 for ProbeCompress (difference +0.0080, interval [−0.0045, +0.0203]). Both intervals include zero, so these four-model comparisons do not establish an accuracy advantage. They do show that ProbeCompress is at least as accurate at a much lower cost. Random-Subset Mean runs 64 complete tasks per target, which takes about 44× more marginal tokens and 9× more summed work than ProbeCompress on the five frozen targets, and 57× and 12× on all six (Tables 2, S18, S24).
6. Underestimation above the calibration range.
On the frozen split, the 13 above-maximum cells have signed error −0.0193 and MAE 0.0542, versus −0.0021 and MAE 0.0239 for the seven within-range cells. On the external holdout, nine above-maximum cells have signed error −0.0278 and MAE 0.0668, versus +0.0012 and MAE 0.0380 for 13 within-range cells. Two below-minimum cells have signed error +0.0114 and MAE 0.0140. Appendix E.4 now reports these results. Since a new model's true position is unknown before full evaluation, we recommend full evaluation when a prediction lies near the calibration ceiling.
7. Close pairs.
For the 40 pairs whose reference scores differ by less than 0.02, all three methods score 48.75% (half credit for ties). A fresh full-benchmark run orders the same pairs correctly only 57.5% of the time, and 75.5% for gaps of 0.02 to 0.05. Near-tie ordering is therefore unstable in this sample even under full evaluation. This does not establish a general noise floor, and the weakness still matters for model selection with ProbeCompress. Table S9 uses the shared-interface pair holdout inside the development panel, not the frozen split. For the 695 pairs with gaps of at least 0.05, ProbeCompress reaches 94.68%. Section 4.4 now reports this gap dependence and when to run a full evaluation (see our response to Reviewer C9VJ).
8. Setup cost and break-even.
The figure of about 52 systems includes collecting three-pass historical labels for the calibration panel. Every calibrated method in our comparison needs these labels, including Original-Task Matched and EssenceBench-adapted. The setup specific to ProbeCompress costs 162.5M tokens and 2,225 h of summed work. It breaks even against one-pass full evaluation after 0.85 additional systems by tokens and 3.61 by summed work. With label collection included, the thresholds are 52.28 and 55.58 (Table S19). The Introduction now reports both cases next to the marginal ratios.
9. Keeping the interface current.
Reuse assumes that the relationship learned under the fixed evaluation setup still holds. Periodic full evaluations check for drift, and each fully evaluated system can be added to the calibration panel. Any change to the evaluation setup requires recalibration. Our response to Reviewer C9VJ (point 3) lists what can be reused for each type of change.
Response to Reviewer ccNQ (3/3)
Q1. Interaction versus static extraction.
Of 1,050 distinct selected probes, 323 (30.8%) are static, 397 (37.8%) read local files and 330 (31.4%) require environment actions. With tools disabled, E-probe pass rates fall from 50.70% to 0% for GPT-5.5 and from 28.27% to 0% for Qwen3.8-27B. This zero follows from the environment-state verification contract and is not separate predictive evidence. File-read pass rates fall from 63.28% to 40.27% and from 42.18% to 23.34%, and static rates change by at most 0.23 percentage points. Removing E probes raises macro MAE to 0.0349, while F+E-only reaches 0.0311. Appendix B.3.3 reports the executed WebArena-Pro probe WAPRO-0246-E1 (parent 20260501095702013727). The agent navigated Mattermost to Jane's direct messages and returned the header Jane and the route /mock-workspace/messages/@jane. Candidate identification and notifying Jane remain outside the probe. Section 3.2 now presents this executed probe in the main text, so the main-text examples are no longer only static.
Q2. Degraded scaffold or reduced step budget.
We have not run a separate scaffold-shift experiment. By design, changing only the full-task scaffold or its step budget does not change the fixed probe runner. The runner shares the model and decoding settings but has its own instructions, tool interface and execution limits. Existing calibration probe outcomes can therefore be reused, although a new stochastic run need not return identical outcomes. Full-task performance can change, so new full-task calibration labels, probe reselection, preprocessing and predictor refitting are required. If shared decoding settings change, both calibration probe responses and full-task labels must be regenerated. The tools-on/tools-off diagnostic in Q1 changes the probe runner itself and is not a test of full-task scaffold shift. The Introduction, Limitations and Appendix F.2 now state this scope.
Q3. A linear map from general capability.
We evaluate all 12,032 MMLU-Pro questions per development model with zero-shot, temperature-zero accuracy. A target-specific two-parameter clipped affine map is fitted on calibration models only, excluding the evaluated model in LOMO. Across 18 development models and 102 model-target cells, MMLU-Pro MAE is 0.0385 versus 0.0280 for ProbeCompress (paired difference +0.0105, 95% model-cluster bootstrap CI [+0.0018, +0.0205], 100,000 resamples). ProbeCompress has lower mean error on all six targets, and its model-level average error is lower for 11 of 18 models. With the same mapping, frozen 14→4 MAE is 0.0511 versus 0.0436. This four-model comparison is descriptive and is not covered by the LOMO interval. The baseline uses the complete question set, so it is not limited by a small evaluation budget. We did not test cheaper MMLU-Pro subsets or a five-shot chain-of-thought protocol. Table R6 in Appendix H.6 gives per-target results.
Q4. Contribution of the rubric verifier.
Accurate reconstruction remains possible with deterministic verification. Restricting the bank to deterministic probes and rerunning the pipeline raises macro MAE by 0.0030, from 0.0280 to 0.0309 (Table 3). Keeping the original probes and replacing rubric scoring with deterministic checks gives 0.0314 (Table S4). The deterministic-only result is still below Original-Task Matched (0.0337) and close to EssenceBench-adapted (0.0304), and both of these execute complete tasks. Rubric-verified probes are 276 of the 1,050 distinct selected probes (26.3%). Our response to Reviewer QFWM reports how often each verifier type can be fooled.
Q5. Release.
Upon publication, we plan to release the 2,665 accepted probe specifications, the code for generation, filtering, selection and calibration, the deterministic verifiers and frozen rubrics, the evaluation configurations and benchmark manifests, and the scripts and records we are allowed to redistribute. Proprietary model outputs and benchmark assets with redistribution restrictions are excluded. The reproducibility statement now specifies this scope.
Response to Reviewer ccNQ (1/3)
Thank you for the detailed review. In short: (1) environment-action probes make up 31.4% of distinct selected probes, and excluding them raises macro MAE from 0.0280 to 0.0349. (2) A full-length MMLU-Pro LOMO baseline reaches 0.0385 MAE, versus 0.0280 for ProbeCompress (paired model-cluster 95% CI [+0.0018, +0.0205]). (3) The method is calibrated for new models under a fixed benchmark setup.
Weaknesses
1. What the probes measure.
A single probe does not measure end-to-end agentic competence. The suite predicts the aggregate benchmark score. ProbeCompress LOMO macro MAE is 0.0280, versus 0.0396 for non-target public-task probes (Table 3). Removing E probes and rerunning selection and fitting raises MAE to 0.0349 (OSW2: 0.0377 to 0.0511). The F+E-only condition reaches 0.0311, and E-only and S-only reach 0.0391 and 0.0409. Interaction and static evidence therefore both contribute. The full and Without-E conditions both keep K=64. E-only and S-only comparisons can also reflect smaller effective suites, and the F+E result is not K-matched.
2. Model diversity and general capability.
Our MMLU-Pro baseline uses all 12,032 questions zero-shot and fits a target-specific clipped affine map on calibration models only. Over all 102 development model-target cells, LOMO macro MAE is 0.0385 versus 0.0280 for ProbeCompress (paired difference +0.0105, 95% model-cluster CI [+0.0018, +0.0205]). ProbeCompress has lower MAE on all six targets. Eleven of eighteen models have a positive average paired difference, so model-level effects vary. Static probes alone do not beat this general-capability score (S-only 0.0409 vs 0.0385, with a smaller S-only suite on OSW2, Table R8). The advantage appears once file-read and environment-action probes are included (S+F 0.0349, F+E 0.0311, full suite 0.0280). A suite that worked only through static reasoning would not show this pattern. The frozen comparison is descriptive and is not part of this LOMO interval. Own-parent probe association is higher than re-paired association (0.441 vs 0.232, Table S25).
3. Scaffold changes.
We agree that the probe interface cannot detect scaffold changes. The probe runner uses a fixed instruction template, tool interface and execution limits, and shares only the model and decoding settings with the full-task agent. If the full-task scaffold or prompt changes, existing probe outcomes can be reused, but new full-task labels require target-dependent probe reselection and predictor refitting (Q2). We removed prompts and agent scaffolds from the motivating use cases and now state that the method targets new models under a fixed evaluation setup.
4. Weaker parent association on OSWorld.
OSW2 has 11 calibration systems per fold and the smallest usable bank (120 to 138 probes per fold, Table S30). The own-parent association stays above the re-paired value (0.362 vs 0.303), but the gap is the smallest in Table S25. OSW2 also shows the largest error increase when environment-action probes are removed. Its MAE rises from 0.0377 to 0.0511, and a bank of environment-action probes alone (0.0461 at K=56) outperforms the static-plus-file bank.