acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Benchmarks: An Active Testing Paradigm for Large Language Models

Abstract

Static benchmarks remain the dominant way to evaluate large language models, yet fixed questions are vulnerable to saturation, contamination, and overfitting. More fundamentally, their test distributions do not adapt to the model under evaluation, potentially spending evaluation budget on regions that are already well understood or largely solved. We propose Active Testing, an evaluation paradigm in which observed model behavior dynamically guides where to test next, with emphasis on regions that are both failure-prone and uncertain. To demonstrate its effectiveness, we instantiate this paradigm for web-search agents by turning Wikidata into a large, structured, and verifiable task space with 12 controllable dimensions. We realize the active tester with a random-forest surrogate, enabling sample-efficient search and targeted discovery of weak regions in a large discrete task space. Across six agents, accuracy drops by an average of 28.0 percentage points from the initial random baseline to the final attack round. Rankings by Active-Testing AUC also diverge from initial-accuracy rankings, revealing performance differences that the shared starting distribution obscures. The tester converges to different regions for different models, exposing model-specific weaknesses while also uncovering shared failure patterns across agents. These results motivate a shift from static benchmarks toward active testing protocols, offering a more proactive and diagnostically informative approach to evaluating frontier models.

Primary area:
datasets and benchmarks
Keywords:
Web-Search Agents, Active Testing, Weakness Discovery, Dynamic Task Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7YpfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an active testing framework for web-search agents. Within a Wikidata task space with 12 controllable dimensions, the system uses historical model performance to predict failure rates and uncertainty with a random forest, and combines these with diversity to select subsequent test configurations. Experiments on six agents show that adaptive task selection can lower accuracy in later rounds and change some models’ AT-AUC rankings.

Critical strengths
  • Combines structured task generation, knowledge-graph verification, and feedback-driven task selection into a complete workflow, with practical value for finding failures.

  • Task configurations are interpretable, making it easier to analyze performance across different domains, languages, and clue structures.

  • Provides experiments on multiple models and attempts to distinguish shared difficulties from model-specific weaknesses through cross-model replay.

Critical weaknesses
  • The paper requires the candidate set to contain more than one entity before the final clue is added, but the code provided in the authors’ attachment allows tasks where the first clue already uniquely determines the answer when MIN_FILTER=1, and later validation does not recheck this. Even if this constraint is enforced, it cannot rule out shortcuts where the final clue alone determines the answer or directly identifies the final entity. Therefore, the number of clues and labeled hops may not reflect the actual reasoning complexity.

  • Table 3 does not provide baselines for random selection over all rounds under the same budget, fixed difficult regions, or simple resampling based on error rates, nor does it separately test failure prediction, uncertainty, and diversity. The decline in accuracy after the test distribution shifts toward difficult regions is not, by itself, enough to demonstrate the added value of continued adaptive selection.

  • The code provided in the authors’ attachment requires RL and red-team baselines to strictly match the proposed task configurations, but allows RF to accept configuration changes, introducing a variable beyond the selection strategy. In addition, history-conditioned RL and RF use 1,400 and 300 initial tasks, respectively, and the comparison is conducted only on GPT-5.4 Mini. The acceptance rules need to be unified, budget accounting clarified, and efficiency comparisons across models added, including the cost of failed generation attempts.

  • Different models receive different adaptive test distributions, so AT-AUC reflects both model ability and how easily its weaknesses are discovered. Although the paper limits the interpretation of this metric, the ranking reversals involve differences of only 1.1 and 0.8 percentage points, without results from multiple random seeds or confidence intervals. The weakness-transfer experiments are also mainly limited to one-way replay between two models, which is not enough to rule out sampling noise or support stable model-specific weaknesses.

  • Uniqueness within the knowledge graph does not mean uniqueness on the open web; both semantic validation and scoring rely on GPT-5.4 Mini, which is also one of the evaluated models. The manual checks do not report error rates, agreement between annotators, or quality differences between early and late rounds, so the possibility that active selection amplifies generation or scoring errors cannot yet be ruled out.

  • The attachment lacks several model adapters, the RL baseline implementation, and the trajectory logs that the paper claims to provide; the main script also imports four symbols that do not exist in the generation module. This prevents independent reproduction of the main experiments and makes it impossible to verify the effects of the implementation differences above on the results.

Questions for authors

1.Could you fix and audit the prefix constraint, also check for shortcuts involving single clues, clue subsets, and direct identification of the final entity, and report the proportion of affected tasks?

2.Could you unify the task acceptance rules and budget accounting across strategies, and add the key baselines, component tests, and cross-model efficiency comparisons?

3.Could you provide results from multiple random seeds, confidence intervals, and two-way or multi-model evaluations on shared challenge sets to verify the stability of the rankings and model-specific weaknesses?

4.Could you add an independent manual audit and provide complete, runnable code, configurations, and trajectory logs consistent with the paper’s experiments?

7 comments

Thank you for recognizing the practical value of our structured task generation, interpretable configurations, and feedback-driven weakness discovery framework.

We address your concerns point by point below.

Q1 + W1: Could you fix and audit the prefix constraint, also check for shortcuts involving single clues, clue subsets, and direct identification of the final entity, and report the proportion of affected tasks?

Thank you for raising this question. We would like to clarify two important distinctions.

1. Objective of designed task space.

Our 12-dimensional task space systematically investigates how clue discriminativeness, evidence composition, reasoning topology, and domain knowledge affect agent performance, rather than merely constructing shortcut-free puzzles. As stated in Sec. 2.1: "Rather than predefining a small set of task families based on human priors, we factor the underlying knowledge-graph space into 12 low-level configuration parameters." This design maintains a meaningful difficulty gradient in the original task space.

2. Shortcut mitigation and empirical audit.

Dimensions such as Broad-first clue ordering control clue discriminativeness and candidate-set reduction, discouraging reliance on early discriminative clues. We additionally conducted a random audit of 55 tasks, examining single-clue and subset-only shortcuts.

Shortcut typeAffected tasksPercentage
Single-clue shortcut2 / 553.64%
Subset-only shortcut1 / 551.82%

These findings indicate that relatively few such shortcuts were identified in the audited sample. We will refine our statement in Section 2.2 and fix prefix constraint.

Q2 + W2: Does continued adaptive selection outperform random testing and fixed difficult-region selection? What is the contribution of each acquisition component?

Thank you for highlighting the need to isolate the contribution of sequential feedback.

To disentangle this effect from agent-side generation and search noise, we construct Oracle environments with known ground-truth failure landscapes, including nonlinear interactions among task dimensions, ensuring identical conditions and budgets across selectors.

We compare Online RF, Frozen RF, and Random using the same 12-dimensional task space and active testing rounds from the paper. Across 3 models with distinct weakness distributions and 3 independent seeds, we obtain:

CapabilityOnline RF AT-AUC ↓Frozen RF AT-AUC ↓Random AT-AUC ↓
Strong67.36 ± 2.49%71.08 ± 3.27%78.95 ± 0.78%
Medium56.78 ± 2.99%62.91 ± 3.82%70.79 ± 1.04%
Weak48.14 ± 2.59%53.01 ± 3.98%62.43 ± 1.07%
Mean57.43%62.33%70.72%

Online RF reduces mean AT-AUC by 4.90 pp relative to Frozen RF and 13.29 pp relative to Random, while maintaining higher diversity than Frozen RF.

These results support the value of sequential adaptation: in a large combinatorial task space, sequential feedback enables the selector to progressively discover vulnerable attribute combinations that frozen or random selection may miss.

Acquisition Component Ablation

We further examine failure prediction, uncertainty, diversity, and the cosine schedule through three independent runs.

MethodAT-AUC ↓Diversity ↑
Full Online RF49.48 ± 0.89%0.165
w/o Failure Prediction60.94 ± 1.23%0.188
w/o Uncertainty46.31 ± 2.80%0.159
w/o Diversity47.58 ± 2.02%0.156
w/o Cosine Schedule47.91 ± 2.24%0.158

Removing failure prediction raises AT-AUC by 11.46 pp, substantially weakening hard-task discovery. Uncertainty and diversity expose a meaningful trade-off: removing either may lower AT-AUC but also reduces diversity. These findings clarify the complementary roles of targeting difficult tasks and exploring the broader task space.

Q2 + W3: Are RF, RL, and red-teaming compared under consistent acceptance rules and evaluation budgets?

We appreciate the opportunity to clarify the implementation and budget accounting.

1. Configuration acceptance rules.

We would like to correct an implementation-related misunderstanding caused by the outdated code attachment.

In our experimental implementation, RF, RL, and Red-Team follow the same configuration-matching rule: a generated task is rejected if its realized configuration differs from the proposed configuration.

The permissive RF behavior identified in the earlier attachment does not reflect this shared acceptance rule. There is no intended RF-specific relaxation that allows configuration drift.

We apologize for the outdated attachment and have provided the synchronized implementation, including the common rejection logic, to make this directly verifiable.

2. Warm-start budget and sample efficiency.

The difference in warm-start sizes (RF: 300 vs. History-RL: 1,400) actually highlights the sample efficiency of our RF-based approach, which achieves stronger attack performance (23.2% vs. 28.9%) with substantially fewer initial samples.

As explained in Appendix C.2, history-conditioned RL performed poorly with smaller warm-start datasets, motivating its additional 1,100 training tasks.

To address the budget concern directly, we also evaluated History-RL under a strictly matched 300-task warm start:

MethodWarm-start tasksAccuracy ↓Diversity ↑
RF (Ours)30023.20%0.129
RL (History-Conditioned)30045.60%0.121
RL (History-Conditioned)1,40028.90%0.100

Under the same 300-task warm start, RF achieves substantially stronger attack performance and higher diversity.

We agree that the original 1,400-task RL comparison should not be presented as an equal-total-budget comparison. The additional training data will be accounted for explicitly.

Q3 + W4: How stable are the AT-AUC ranking changes, and do bidirectional replay results support model-specific weaknesses?

We address this concern through independent-run ranking checks and bidirectional cross-model replay.

1. Ranking Stability

Given the substantial cost of repeated multi-round web-agent evaluations, we prioritized DeepSeek V4 Flash and GPT-5.4 Mini, which exhibit closely matched initial performance, for three independent runs.

Due to resource constraints, we have not yet repeated the GLM-5.2–Gemini 3.1 Pro comparison.

RunGPT InitialDeepSeek InitialGPT AT-AUCDeepSeek AT-AUCΔ AT-AUC (DS − GPT)
Seed 4250.00%49.30%35.48%36.26%+0.78 pp
Seed 4353.70%52.50%30.69%35.10%+4.41 pp
Seed 4454.80%55.20%36.10%38.70%+2.60 pp
Mean ± SD52.83 ± 2.51%52.33 ± 2.95%34.09 ± 2.96%36.69 ± 1.84%+2.60 ± 1.82 pp

Consistent Directionality. Despite the stochasticity of multi-step web search and task execution, DeepSeek V4 Flash has a higher AT-AUC in all three runs, with a mean advantage of 2.60 pp.

Exposing Static Evaluation Blind Spots. AT-AUC captures performance persistence under adaptively targeted evaluation, a different diagnostic objective from accuracy on a fixed task distribution. As already clarified in Section 3.2, AT-AUC should not be interpreted as an estimate of model accuracy on a single shared distribution.

2. Bidirectional Cross-Model Replay

To test whether selected tasks merely reflect universal difficulty, we evaluate 509 GPT-targeted weak tasks on DeepSeek and 382 DeepSeek-targeted weak tasks on GPT, reporting mean ± SD across three independent runs.

Cross-Replay on GPT's Weak Regions

Metric / ModelHumanMuseumLow popularityFrenchAsteroid
DeepSeek Acc.43.3 ± 9.2%62.2 ± 4.3%46.5 ± 7.7%50.1 ± 6.6%63.4 ± 8.7%
GPT Acc.17.6 ± 3.6%26.4 ± 8.7%21.7 ± 3.6%18.7 ± 3.9%23.4 ± 3.3%

Cross-Replay on DeepSeek's Weak Regions

Metric / ModelShipFrenchLow popularityChineseHuman
DeepSeek Acc.10.3 ± 7.1%21.3 ± 5.3%24.4 ± 8.0%26.2 ± 5.2%34.4 ± 5.4%
GPT Acc.21.4 ± 4.9%25.8 ± 3.5%32.3 ± 5.1%26.6 ± 4.7%29.7 ± 3.6%

Key Findings:

  • Model-Specific Weakness Discovery: In 9 of 10 targeted marginal comparisons, the targeted agent performs worse than the replayed agent. The only exception is the Human marginal, already GPT-5.4 Mini's weakest top-1 marginal. This provides evidence that Active Testing identifies model-specific vulnerabilities rather than merely selecting universally difficult tasks.

  • Shared and Distinct Failure Regions: Our t-SNE visualization of final-round task configurations (Fig. 3(a)) reveals partially overlapping yet distinct regions across six agents, providing complementary descriptive evidence of shared and model-specific weakness patterns.

Together, these findings highlight the central diagnostic value of Active Testing: revealing model-specific capability boundaries and long-tail vulnerabilities that may remain hidden under a shared, fixed evaluation distribution.

Reviewer_LnjaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an adaptive evaluation framework called "Active Testing," designed to identify model weaknesses that lie beyond the scope of static benchmarks. The framework leverages 12 controllable dimensions to construct a vast, structured task space derived from Wikidata. Through a "reverse task generation" process—incorporating a SPARQL-based verification mechanism—it produces questions with verifiable answers. Subsequently, a Random Forest surrogate model is trained based on observed task configurations and model performance; this surrogate model then informs the selection of future test cases via an acquisition function that balances the predicted probability of failure, uncertainty, and diversity.

Critical strengths

This paper not only proposes the idea of ​​dynamically adjusting the test distribution based on model feedback, but also establishes a testing environment—leveraging Wikidata and SPARQL—that features well-defined task parameters and verifiable answers. Compared to relying entirely on an LLM to freely generate adversarial questions, this design offers superior controllability and interpretability; it also facilitates a systematic analysis of the correlations between various task attributes and model failures.

Critical weaknesses
  1. Comparability issues for models under adaptive test distributions The paper adopts AT-AUC to summarize the performance of different models across rounds of active testing and discusses shifts in model rankings based on this metric. This is indeed a novel and effective approach. Nevertheless, one core purpose of a benchmark is to provide a fair and comparable evaluation platform for models. Since AT-AUC is affected jointly by both the capability of the model under test and the task selection strategy of the tester, it constitutes a multi-factor variable. In my view, ablation studies targeting this comparability concern are warranted.

  2. Robustness of Active Testing The paper does not appear to include experiments verifying the reproducibility of results for a single model, which raises my concerns on this front.

Questions for authors

As noted in the Critical Weaknesses section: does this method suffer from the comparability issue under adaptive test distributions and the robustness issue of Active Testing? Additionally, what are the authors’ perspectives on these two concerns?

1 comment

Thank you for recognizing our active testing framework as a novel and effective approach. We appreciate this encouraging feedback. We address your concerns point by point below:

Q1. Does this method suffer from the comparability issue under adaptive test distributions? What are the authors’ perspectives on this concern?

  1. Paradigm Shift: We view this as a necessary shift from item-level standardization (static benchmarks) to protocol-level standardization (Active Testing). Traditional static benchmarks aim for identical test items, but they inherently suffer from saturation, data contamination, and overfitting, creating a deceptive illusion of comparability.

  2. Topographical Capability Mapping: A model's true capability is not a uniform score captured by a static test set, but a complex 'topographical map' of peaks and valleys. Active Testing provides a sample-efficient method to dynamically chart this map. Furthermore, by decomposing the task space into controllable dimensions, our framework yields a multi-faceted diagnostic toolkit (cross-dimensional interactions, marginal vulnerability profiles, and adversarial robustness) for a deeper understanding of modern agents.

  3. Protocol Standardization: Comparability is maintained by enforcing a strictly uniform evaluation protocol across all models—sharing the same 300-task random warm start, 12-dimensional Wikidata task space, acquisition/budget schedule, and round count. While adaptive selection introduces minor variance from the joint interaction between the model and the testing trajectory, our results in Q2 confirm that active testing reliably identifies weaknesses and achieves relatively robust AT-AUC performance.

Q2. Since AT-AUC is affected jointly by model capability and the task selection strategy, it constitutes a multi-factor variable. Ablation studies targeting this comparability concern are warranted.

To disentangle the effect of the selection strategy from agent-side generation and search noise, we constructed oracle environments with known ground-truth failure landscapes, including nonlinear interactions among task dimensions. This allows us to directly evaluate the effectiveness of different selection strategies under controlled capability distributions.

(1) Controlled Oracle Comparison: Online RF vs. Random

We compare Online RF and Random using the same 12-dimensional task space and active testing rounds. Across 3 models with distinct weakness distributions and 3 independent seeds, Online RF achieves a substantially lower mean AT-AUC (57.43%) than Random (70.72%). This demonstrates the value of sequential feedback: in a large combinatorial task space, the selector progressively identifies vulnerable attribute combinations that random sampling is less likely to discover.

CapabilityOnline RF (AT-AUC ↓ / Diversity)Random (AT-AUC ↓ / Diversity)
Strong67.36 ± 2.49% / 0.161 ± 0.00478.95 ± 0.78% / 0.187 ± 0.001
Medium56.78 ± 2.99% / 0.159 ± 0.00470.79 ± 1.04% / 0.188 ± 0.001
Weak48.14 ± 2.59% / 0.160 ± 0.00462.43 ± 1.07% / 0.188 ± 0.001

(2) Multi-Seed Evaluation: AT-AUC Stability

We further evaluate real agent DeepSeek V4 flash across 3 independent runs:

RunDeepSeek AT-AUCRandom Initialization
Seed 4236.26%49.30%
Seed 4335.10%52.50%
Seed 4438.70%55.20%
Mean ± SD36.69 ± 1.84%52.33 ± 2.95%

The standard deviation of DeepSeek's AT-AUC across runs is 1.84 pp, indicating that Active Testing produces relatively robust and comparable results across different random seeds.

Overall, we agree that AT-AUC should be interpreted in the context of a standardized evaluation protocol rather than as an item-wise accuracy score on an identical test set. Our goal is to go beyond static benchmarking and systematically discover model-specific weaknesses through feedback-driven exploration of a structured task space. We hope these additional controlled and multi-seed experiments address the reviewer’s concerns.

Reviewer_FL5eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Active Testing, in which observed agent behavior guides the selection of subsequent evaluation tasks. It implements this approach for web search agents using a Wikidata-based generator with 12 configurable dimensions. Tasks are constructed around a target entity, checked through executable SPARQL queries, and rendered into natural-language questions. A random forest surrogate selects configurations using predicted failure, tree disagreement, and diversity.

Six agents are evaluated using a shared 300-task random warm start followed by eight adaptive rounds targeting 200 tasks each. The paper reports an average accuracy decline of 28.0 percentage points and introduces AT-AUC to summarize performance persistence during testing. Two adjacent model pairs change order under this metric. Additional experiments examine cross-model transfer, shared failure patterns, search behavior, and alternative testers. On GPT-5.4 Mini, the RF tester achieves the lowest reported accuracy among the compared testers.

The structured generator and observed degradation are useful contributions. My main reservations concern whether the experiments isolate the benefit of sequential adaptation, establish reliable ranking changes, and support the claimed protection against task shortcuts.

Critical strengths
  1. The generator provides an interpretable and reusable evaluation space. Explicit configuration dimensions, executable queries, and fresh task instantiations make it possible to describe weaknesses in terms of task properties and revisit them with new questions. This is more informative than reporting an undifferentiated collection of failures.

  2. The study covers six agents, uses a common initial question set, reports complete testing trajectories, and compares several alternative testers. The substantial within-agent accuracy reductions support the narrower conclusion that the proposed procedure discovers considerably harder regions than the initial random sample.

  3. The paper acknowledges important limitations. It distinguishes AT-AUC from accuracy on a common fixed distribution, recognizes that marginal configuration associations are not causal, and states that graph-relative uniqueness does not guarantee uniqueness in the open world. These qualifications make the intended evaluation objective clearer.

Critical weaknesses
  1. The experiments do not isolate what sequential adaptation contributes.

Table 3 compares the RF tester with RL and LLM-based testers, but its random comparator is the initial warm-up rather than random testing continued for the same evaluation budget. There is also no selector trained on the common warm start and then held fixed. Consequently, the results do not separate the value of repeatedly incorporating new outcomes from the value of selecting difficult configurations once.

The acquisition function explicitly seeks failures. A declining trajectory demonstrates successful discovery of harder tasks, but does not by itself establish the contribution of sequential updating, uncertainty, diversity, or the cosine schedule. The reported comparisons support the complete RF procedure over the implemented alternative testers on GPT-5.4 Mini; they provide less support for the broader methodological claim.

Budget accounting also needs clarification. Appendix C.1 describes target evaluation counts, while generation failures and rejected proposals can affect realized counts. The history-conditioned RL baseline uses 1,400 warm-start tasks rather than the RF tester's 300. This extra information does not favor RF, but it means the comparison should not be described as having identical information budgets.

  1. The ranking and model-specificity claims are stronger than the reported evidence.

The two AT-AUC reversals involve gaps of only 1.1 percentage points for GLM-5.2 versus Gemini 3.1 Pro and 0.8 percentage points for DeepSeek V4 Flash versus GPT-5.4 Mini. The paper does not report uncertainty for these differences or variability across independent complete testing runs. Table A5 gives an RF seed, but does not establish ranking stability across the full adaptive procedure.

AT-AUC is a coherent measure of persistence under the stated protocol. However, its value depends on both the evaluated agent and the sequence of configurations discovered by the tester. For example, GLM-5.2's advantage over Gemini 3.1 Pro arises in intermediate rounds; Gemini is ahead initially and again in the final round. The present evidence does not distinguish persistent robustness differences from variation in how quickly the tester discovers difficult regions.

Table 1 provides useful evidence that difficult tasks transfer between agents, but limited evidence separating shared difficulty from model-specific targeting. Its 26.0% result is an unweighted average over five overlapping marginal groups, not overall accuracy on a single test distribution. It should therefore not be compared directly with overall round accuracy. The French and Human self-versus-transfer differences are 0.4 and 7.6 percentage points, respectively, without task counts or uncertainty. One-way replay leaves open whether one agent's selected tasks would also be harder for the other agent.

  1. The prefix check does not establish the claimed prevention of single-clue shortcuts.

Requiring every proper prefix to leave multiple candidates does not imply that multiple clues are needed to identify the final answer. Consider three clues whose individual candidate sets are {a,b,c}, {a,b}, and {a}. The successive intersections contain three, two, and one candidates, so the proper prefixes are non-singleton. Nevertheless, the final clue alone identifies the answer.

This is a limitation of the stated guarantee, not evidence that all generated tasks contain shortcuts. It matters because construction complexity is used to characterize the evaluation space. Individual-clue checks would be needed to support the narrower claim that no single clue suffices. Similarly, graph construction depth should be distinguished from reasoning depth that an agent must actually traverse.

  1. Residual task and grading errors are insufficiently quantified under adaptive selection.

The paper describes SPARQL verification and semantic validation, and its AI Use Statement reports manual inspection of generated task prose and validation outputs. My concern is therefore not that validation is absent. Rather, the paper does not report the inspection protocol, corrections made, or residual error rates across rounds and configurations.

If task ambiguity, rendering errors, or grading errors correlate with configuration, acquisition based on observed failure could preferentially revisit those regions. This is a potential confound, not a demonstrated explanation of the decline. Quantifying it would strengthen the interpretation of late-round failures as agent weaknesses.

The use of gpt-5.4-mini for both validation and judging, while also evaluating that model as an agent, makes agreement with independent judgments particularly useful to report. This does not itself demonstrate bias. Formatting violations should also be reported separately from semantic errors. The appendix's INVALID_OUTPUT example is not evidence of an otherwise-correct answer rejected solely for formatting: its answer also differs from the gold answer.

  1. Some explanatory claims exceed the observational evidence.

The finding that failed trajectories use more queries does not establish that search budget is not a bottleneck. Harder tasks can cause both additional searching and failure. Likewise, associations between clue selection, negative clues, additional layers, and accuracy do not identify the causal effect of changing those properties. Appendix B.3 acknowledges this limitation, but the main-text explanations should be brought into line with it.

My weak-reject recommendation is primarily driven by the missing controls for sequential adaptation, the uncertain ranking and specificity claims, and the gap between the prefix check and its stated guarantee. The validity concern is a request to quantify a remaining confound, not an allegation that the reported failures are invalid. Evidence addressing these issues would materially improve my assessment.

Questions for authors
  1. Contribution of adaptation: For one representative agent, could you compare the proposed procedure with a selector trained on the same 300-task warm start and then frozen for the remaining evaluation budget? The generator, validation pipeline, and outcome aggregation should be comparable. This would directly test what repeated updating adds beyond selecting difficult configurations once. A continued-random trajectory would provide another useful reference if available.

  2. Uncertainty and transfer: How many independent end-to-end runs underlie Figure 2, and what are the realized task counts? Please assess the uncertainty of the two AT-AUC reversals, accounting for the adaptive protocol and repeated configurations. If only one run was performed, please qualify the ranking claims accordingly. For Table 1, please provide cell counts, aggregation rules, and the task pool underlying the "GPT targeted" column. Reverse replay for the existing agent pair would help distinguish shared difficulty from model-specific targeting.

  3. Verification: What did the reported manual inspection cover, when was it performed, and what corrections resulted? A modest audit stratified by early versus late rounds and correct versus incorrect judgments would help quantify remaining task and grading errors. Sampling only failures would not support an estimate of overall validity-adjusted accuracy. Please separate ambiguity, rendering errors, answer-set errors, grading errors, and agent formatting violations. Also clarify whether singleton rejection is applied to the distinct projected answer set for every sub-property task, and whether any individual-clue shortcut check supplements the prefix condition.

  4. Reproducibility and reporting: Please specify, or point to the exact supplementary implementation of, the acquisition coefficients and cosine schedule, the diversity function and its normalization, configuration encoding, and batch-selection procedure. Please also report method-specific realized budgets and agent search/token limits. What aggregation does Table 3's 23.2% represent? Figure A1 contains GPT-4.1 Mini instead of GLM-5.2, and its GPT-5.6 Sol initial-accuracy line appears inconsistent with Figure 2. Please clarify whether these are labeling errors or results from a different experiment.

  5. Positioning: Section 5.3 already distinguishes the objective from fixed-distribution active testing; introducing this distinction earlier would improve clarity. The closest additional published comparisons include Ribeiro and Lundberg's "Adaptive Testing and Debugging of NLP Models" (ACL 2022) and Li et al.'s "AutoBencher: Towards Declarative Benchmark Construction" (ICLR 2025), which address target-model bug discovery and iterative benchmark construction. Please explain what the structured generator and acquisition procedure add relative to these approaches.

For adaptive ability measurement, relevant contrasts include Hofmann et al.'s "Fluid Language Model Benchmarking" (COLM 2025) and Zhuang et al.'s "Position: AI Evaluation Should Learn from How We Test Humans" (ICML 2025). These have a different objective from failure discovery and should be discussed accordingly. WebSailor (2025 preprint) is optional context for knowledge-graph-based task synthesis with information obfuscation, rather than a required comparison.

6 comments

Thank you for your thoughtful and constructive review, particularly for recognizing the value of our structured, verifiable task generation and evaluation experiments.

We appreciate your central question: whether our findings demonstrate the value of sequential adaptation, rather than simply selecting difficult tasks once. To address this, we have strengthened the evaluation along three axes:

  1. Sequential adaptation: Matched-budget oracle comparisons of Online, Frozen, and Random selection.
  2. Ranking stability and model specificity: Independent-run ranking checks and bidirectional cross-model replay.
  3. Evaluation reliability: Human audits, independent judging checks, and matched-budget RL comparisons.

We address each concern in detail below.


Q1 + W1: Does dynamic adaptation outperform one-shot difficult-task selection? Please provide matched-budget controlled comparisons—including a random baseline—to isolate the value of sequential feedback.

Thank you for highlighting the need to isolate the contribution of sequential feedback. Active testing suffers from agent-side generation and search noise, and multi-round seeds are costly. We therefore construct oracle environments with known ground-truth failure landscapes, including nonlinear interactions among task dimensions, ensuring identical conditions and budgets across selectors.

We compare Online RF, Frozen RF, and Random using the same 12-dimensional task space and active testing rounds from the paper. Across 3 models with distinct weakness distributions and 3 independent seeds, Online RF achieves a lower mean AT-AUC of 57.43%, compared with 62.33% for Frozen RF and 70.72% for Random, while maintaining higher diversity than Frozen RF.

These results support the value of sequential adaptation: in a large combinatorial task space, sequential feedback enables the selector to progressively discover vulnerable attribute combinations that frozen or random selection may miss.

Oracle Evaluation Results

CapabilityOnline RF (AT-AUC ↓ / Diversity ↑)Frozen RF (AT-AUC ↓ / Diversity ↑)Random (AT-AUC ↓)
Strong67.36 ± 2.49 / .161 ± .00471.08 ± 3.27 / .157 ± .00878.95 ± 0.78
Medium56.78 ± 2.99 / .159 ± .00462.91 ± 3.82 / .158 ± .01170.79 ± 1.04
Weak48.14 ± 2.59 / .160 ± .00453.01 ± 3.98 / .159 ± .01062.43 ± 1.07

Acquisition Component Ablation

We further examine failure prediction, uncertainty, diversity, and the cosine schedule through three independent runs.

Removing the failure-prediction component raises AT-AUC by 11.46 pp, substantially weakening hard-task discovery. Uncertainty and diversity expose a meaningful trade-off: removing either may lower AT-AUC, but also reduces coverage/diversity. These results clarify the complementary roles of targeting difficult tasks and exploring the broader task space.

MethodAT-AUC ↓Diversity ↑
Full Online RF49.48 ± 0.890.165
w/o Failure Prediction60.94 ± 1.230.188
w/o Uncertainty46.31 ± 2.800.159
w/o Diversity47.58 ± 2.020.156
w/o Cosine Schedule47.91 ± 2.240.158

Q2 + W2 — How stable are the ranking changes, and do two-way replay results support model specificity?

Ranking Stability

Given the substantial cost of repeated multi-round web-agent evaluations, we prioritized DeepSeek V4 Flash and GPT-5.4 Mini, which exhibit closely matched initial performance, for three independent runs. Due to resource constraints, we have not yet repeated the GLM-5.2–Gemini 3.1 Pro comparison.

Multi-Seed Evaluation Results

RunGPT InitialDeepSeek InitialGPT AT-AUCDeepSeek AT-AUCΔ AT-AUC
Seed 4250.00%49.30%35.48%36.26%+0.78 pp
Seed 4353.70%52.50%30.69%35.10%+4.41 pp
Seed 4454.80%55.20%36.10%38.70%+2.60 pp
Mean ± SD52.83 ± 2.51%52.33 ± 2.95%34.09 ± 2.96%36.69 ± 1.84%+2.60 ± 1.82 pp

Consistent Directionality: Despite the stochasticity of multi-step web search and task execution, DeepSeek V4 Flash has a higher AT-AUC in all three runs, with a mean difference of 2.60 pp.

Exposing Static Evaluation Blind Spots: AT-AUC captures robustness under adaptively targeted evaluation, a different diagnostic objective from accuracy on a fixed task distribution.

Bidirectional Cross-Model Replay

To test whether the selected tasks merely reflect universal difficulty, we replay 509 GPT-targeted weak tasks on DeepSeek and 382 DeepSeek-targeted weak tasks on GPT, reporting mean ± SD across three independent runs.

1. Cross-Replay on GPT's Weak Regions

Targeted byMetric / ModelHumanMuseumLow popularityFrenchAsteroid
GPTDeepSeek Acc.43.3 ± 9.2%62.2 ± 4.3%46.5 ± 7.7%50.1 ± 6.6%63.4 ± 8.7%
GPT Acc.17.6 ± 3.6%26.4 ± 8.7%21.7 ± 3.6%18.7 ± 3.9%23.4 ± 3.3%

2. Cross-Replay on DeepSeek's Weak Regions

Targeted byMetric / ModelShipFrenchLow popularityChineseHuman
DeepSeekDeepSeek Acc.10.3 ± 7.1%21.3 ± 5.3%24.4 ± 8.0%26.2 ± 5.2%34.4 ± 5.4%
GPT Acc.21.4 ± 4.9%25.8 ± 3.5%32.3 ± 5.1%26.6 ± 4.7%29.7 ± 3.6%

Key Findings:

  • Model-Specific Weakness Discovery: In 9 of 10 targeted marginal comparisons, the targeted agent performs worse than the replayed agent. The only exception is the Human marginal, already GPT-5.4 Mini's weakest top-1 marginal. This provides strong evidence that Active Testing identifies model-specific vulnerabilities rather than merely selecting universally difficult tasks.
  • Shared and Distinct Failure Regions: Our new t-SNE visualization of final-round task configurations (Fig. 3(a)) reveals partially overlapping yet distinct regions across six agents, providing complementary evidence of both shared and model-specific weakness patterns.

Together, these findings highlight the central diagnostic value of Active Testing: revealing model-specific capability boundaries and long-tail vulnerabilities that may remain hidden under a shared, fixed evaluation distribution.


Q3 + W3: Do prefix constraints prevent single-clue and subset shortcuts?

Thank you for your question. We clarify this from two perspectives:

1. Objective of designed task space. Our 12-dimensional task space systematically investigates how clue discriminativeness, evidence composition, reasoning topology, and domain knowledge affect agent performance, rather than merely constructing shortcut-free puzzles. As stated in Sec. 2.1: "Rather than predefining a small set of task families based on human priors, we factor the underlying knowledge-graph space into 12 low-level configuration parameters." This design maintains a meaningful difficulty gradient in the original task space.

2. Shortcut mitigation and empirical audit. Dimensions like Broad-first clue ordering and Min_filter control clue discriminativeness and candidate-set reduction, discouraging reliance on early discriminative clues. In a random audit of 55 tasks, only 3.64% exhibited single-clue shortcuts and 1.82% contained subset-only shortcuts, confirming that shortcuts remain rare and are not systematically exploited. Complete audit details and examples are in the supplementary material.

We appreciate this observation. We provided the audit procedure and examples in the supplement and will revise the claim about prefix constraints to state precisely what they enforce.


Q3 + W4: Could task-generation or grading errors be amplified by adaptation?

We appreciate this concern. To empirically address this, we conducted a human audit and analyzed error distributions across rounds.

1. Human Audit: Three human annotators independently audited 86 sampled tasks stratified across different models and testing rounds, achieving a high inter-annotator agreement of 93.41%. Complete audit logs (including items adjudicated by ≥ 2 annotators) are provided in the supplementary material. As summarized below across five fine-grained error subtypes, 96.51% of the audited failures stem from genuine LLM agent errors, whereas task-generation errors (2.33%) and judge errors (1.16%) remain minimal:

Error SourceSubtypeCountPercentage
Task Generation ErrorAmbiguity (unclear phrasing / non-unique answer)22.33%
Rendering (hallucination / altered clues)00.00%
LLM Answer ErrorSemantic / Entity / Reasoning7991.86%
Formatting / Output-Contract44.65%
Judge ErrorIncorrect Judge Decision11.16%

2. No Amplification Across Rounds: To verify whether task-generation errors accumulate under adaptive selection, we tracked their frequency across all nine evaluation rounds. The per-round error counts are [1, 0, 0, 0, 0, 0, 0, 1, 0], exhibiting no observed increasing trend with later rounds. We will integrate this audit summary and round-by-round analysis into the revised appendix.

Independent Judging Verification

Our ground truth is a unique entity verified by SPARQL; the judge performs semantic equivalence against that answer, rather than open-ended preference scoring. This substantially constrains the scope for subjective judging bias. In our cross-judging check over 50 trajectories, GPT-5.4 Mini and GLM reached full agreement.

Differences in warm-start sizes (RF: 300 vs. History-RL: 1,400)

The difference in warm-start sizes actually highlights the sample efficiency of our RF-based approach, which achieves stronger attack performance (23.2% vs 28.9% in Table 3) with substantially fewer initial samples.

As explained in Appendix C.2, the history-conditioned RL baseline performed poorly under smaller warm-starts, necessitating 1,400 tasks. Under a strictly matched 300-task warm start, RF achieves stronger attack performance (23.2% vs. 45.6% accuracy) while maintaining higher diversity (0.129 vs. 0.121). We will include this comparison in the revised paper.


W5 — Are the search-budget and task-attribute interpretations causal?

Thank you for your suggestion. We have revised Section 3.3 to refine the search-budget claim and now consistently frame attribute-level findings as diagnostic associations rather than causal conclusions. Additionally, in the next version, we will move the discussion on "Marginals Are Diagnostic, Not Causal" from the Appendix into the main paper.


Q4: Reproducibility and Reporting

Thank you for requesting greater implementation specificity.

We have provided the acquisition coefficients and cosine schedule, the diversity function and its normalization, the configuration encoding and batch-selection procedure, and method-specific realized budgets, together with the relevant code/configuration pointers.

We will also make the agent-side limits explicit:

  • 32 model calls.
  • 40 searches.
  • An 8,192-token per-call limit.
  • 32,768 total output tokens.
  • 2,000 characters per search query.

We hope this will make the selection procedure and experimental budgets easier to reproduce and compare.

Clarification of Table 3 and Figure A1

Thank you for pointing it out.

Table 3: The reported 23.2% represents the mean agent accuracy over the final two evaluation rounds. We will clarify this aggregation in the table caption. Figure A1 was retained from an earlier experimental run. We will replace it with the final results to ensure consistency with Figure 2.


Q5: Positioning and Related Work

Thank you for this helpful suggestion.

We agree that the distinction from active testing under a fixed evaluation distribution should appear earlier.

We will introduce this distinction in the Introduction and refine the Related Work discussion to clarify our central objective: adaptively discovering and characterizing model weaknesses across a structured task space, rather than estimating performance on a predefined fixed distribution.


Finally, we sincerely thank the reviewer for recognizing the motivation and potential of Active Testing as an evaluation paradigm. We hope that the new controlled comparisons, ranking and replay analyses, and independent audits help address your concerns about the experimental evidence. We would be happy to discuss any remaining questions or provide further clarifications.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarifying the Core Contribution and Addressing Key Concerns

Dear AC and Reviewers,

We sincerely thank the reviewers for recognizing the value of our work (all three assigned 2/3 for both Soundness and Informativeness) and would like to further clarify its central motivation. Static benchmarks suffer from saturation, contamination, and overfitting; more fundamentally, fixed test distributions cannot adapt to model behavior and may overlook critical weaknesses. We therefore propose shifting from standardized test items to a standardized active-testing protocol. Our goal is not merely to construct harder puzzles by eliminating seemingly simple cases (e.g., single-clue tasks), but to explore a broad, complex, yet verifiable task space through active testing, thereby uncovering valuable model-specific weaknesses alongside shared failure patterns.

The main concerns are sequential adaptation, result stability, and model specificity. Our rebuttal provides direct evidence: controlled oracle experiments isolate the value of adaptation (Online/Frozen/Random AT-AUC: 57.43/62.33/70.72%); three independent real-agent runs show consistent GPT–DeepSeek AT-AUC ordering; and bidirectional replay (509/382 tasks) finds targeted models weaker in 9/10 examined marginals.

We hope these results substantially address the central concerns and warrant reconsideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.