- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes an active testing framework for web-search agents. Within a Wikidata task space with 12 controllable dimensions, the system uses historical model performance to predict failure rates and uncertainty with a random forest, and combines these with diversity to select subsequent test configurations. Experiments on six agents show that adaptive task selection can lower accuracy in later rounds and change some models’ AT-AUC rankings.
Critical strengths
-
Combines structured task generation, knowledge-graph verification, and feedback-driven task selection into a complete workflow, with practical value for finding failures.
-
Task configurations are interpretable, making it easier to analyze performance across different domains, languages, and clue structures.
-
Provides experiments on multiple models and attempts to distinguish shared difficulties from model-specific weaknesses through cross-model replay.
Critical weaknesses
-
The paper requires the candidate set to contain more than one entity before the final clue is added, but the code provided in the authors’ attachment allows tasks where the first clue already uniquely determines the answer when MIN_FILTER=1, and later validation does not recheck this. Even if this constraint is enforced, it cannot rule out shortcuts where the final clue alone determines the answer or directly identifies the final entity. Therefore, the number of clues and labeled hops may not reflect the actual reasoning complexity.
-
Table 3 does not provide baselines for random selection over all rounds under the same budget, fixed difficult regions, or simple resampling based on error rates, nor does it separately test failure prediction, uncertainty, and diversity. The decline in accuracy after the test distribution shifts toward difficult regions is not, by itself, enough to demonstrate the added value of continued adaptive selection.
-
The code provided in the authors’ attachment requires RL and red-team baselines to strictly match the proposed task configurations, but allows RF to accept configuration changes, introducing a variable beyond the selection strategy. In addition, history-conditioned RL and RF use 1,400 and 300 initial tasks, respectively, and the comparison is conducted only on GPT-5.4 Mini. The acceptance rules need to be unified, budget accounting clarified, and efficiency comparisons across models added, including the cost of failed generation attempts.
-
Different models receive different adaptive test distributions, so AT-AUC reflects both model ability and how easily its weaknesses are discovered. Although the paper limits the interpretation of this metric, the ranking reversals involve differences of only 1.1 and 0.8 percentage points, without results from multiple random seeds or confidence intervals. The weakness-transfer experiments are also mainly limited to one-way replay between two models, which is not enough to rule out sampling noise or support stable model-specific weaknesses.
-
Uniqueness within the knowledge graph does not mean uniqueness on the open web; both semantic validation and scoring rely on GPT-5.4 Mini, which is also one of the evaluated models. The manual checks do not report error rates, agreement between annotators, or quality differences between early and late rounds, so the possibility that active selection amplifies generation or scoring errors cannot yet be ruled out.
-
The attachment lacks several model adapters, the RL baseline implementation, and the trajectory logs that the paper claims to provide; the main script also imports four symbols that do not exist in the generation module. This prevents independent reproduction of the main experiments and makes it impossible to verify the effects of the implementation differences above on the results.
Questions for authors
1.Could you fix and audit the prefix constraint, also check for shortcuts involving single clues, clue subsets, and direct identification of the final entity, and report the proportion of affected tasks?
2.Could you unify the task acceptance rules and budget accounting across strategies, and add the key baselines, component tests, and cross-model efficiency comparisons?
3.Could you provide results from multiple random seeds, confidence intervals, and two-way or multi-model evaluations on shared challenge sets to verify the stability of the rankings and model-specific weaknesses?
4.Could you add an independent manual audit and provide complete, runnable code, configurations, and trajectory logs consistent with the paper’s experiments?
7 comments
Q2 + W3: Are RF, RL, and red-teaming compared under consistent acceptance rules and evaluation budgets?
We appreciate the opportunity to clarify the implementation and budget accounting.
1. Configuration acceptance rules.
We would like to correct an implementation-related misunderstanding caused by the outdated code attachment.
In our experimental implementation, RF, RL, and Red-Team follow the same configuration-matching rule: a generated task is rejected if its realized configuration differs from the proposed configuration.
The permissive RF behavior identified in the earlier attachment does not reflect this shared acceptance rule. There is no intended RF-specific relaxation that allows configuration drift.
We apologize for the outdated attachment and have provided the synchronized implementation, including the common rejection logic, to make this directly verifiable.
2. Warm-start budget and sample efficiency.
The difference in warm-start sizes (RF: 300 vs. History-RL: 1,400) actually highlights the sample efficiency of our RF-based approach, which achieves stronger attack performance (23.2% vs. 28.9%) with substantially fewer initial samples.
As explained in Appendix C.2, history-conditioned RL performed poorly with smaller warm-start datasets, motivating its additional 1,100 training tasks.
To address the budget concern directly, we also evaluated History-RL under a strictly matched 300-task warm start:
| Method | Warm-start tasks | Accuracy ↓ | Diversity ↑ |
|---|---|---|---|
| RF (Ours) | 300 | 23.20% | 0.129 |
| RL (History-Conditioned) | 300 | 45.60% | 0.121 |
| RL (History-Conditioned) | 1,400 | 28.90% | 0.100 |
Under the same 300-task warm start, RF achieves substantially stronger attack performance and higher diversity.
We agree that the original 1,400-task RL comparison should not be presented as an equal-total-budget comparison. The additional training data will be accounted for explicitly.
Q3 + W4: How stable are the AT-AUC ranking changes, and do bidirectional replay results support model-specific weaknesses?
We address this concern through independent-run ranking checks and bidirectional cross-model replay.
1. Ranking Stability
Given the substantial cost of repeated multi-round web-agent evaluations, we prioritized DeepSeek V4 Flash and GPT-5.4 Mini, which exhibit closely matched initial performance, for three independent runs.
Due to resource constraints, we have not yet repeated the GLM-5.2–Gemini 3.1 Pro comparison.
| Run | GPT Initial | DeepSeek Initial | GPT AT-AUC | DeepSeek AT-AUC | Δ AT-AUC (DS − GPT) |
|---|---|---|---|---|---|
| Seed 42 | 50.00% | 49.30% | 35.48% | 36.26% | +0.78 pp |
| Seed 43 | 53.70% | 52.50% | 30.69% | 35.10% | +4.41 pp |
| Seed 44 | 54.80% | 55.20% | 36.10% | 38.70% | +2.60 pp |
| Mean ± SD | 52.83 ± 2.51% | 52.33 ± 2.95% | 34.09 ± 2.96% | 36.69 ± 1.84% | +2.60 ± 1.82 pp |
Consistent Directionality. Despite the stochasticity of multi-step web search and task execution, DeepSeek V4 Flash has a higher AT-AUC in all three runs, with a mean advantage of 2.60 pp.
Exposing Static Evaluation Blind Spots. AT-AUC captures performance persistence under adaptively targeted evaluation, a different diagnostic objective from accuracy on a fixed task distribution. As already clarified in Section 3.2, AT-AUC should not be interpreted as an estimate of model accuracy on a single shared distribution.
2. Bidirectional Cross-Model Replay
To test whether selected tasks merely reflect universal difficulty, we evaluate 509 GPT-targeted weak tasks on DeepSeek and 382 DeepSeek-targeted weak tasks on GPT, reporting mean ± SD across three independent runs.
Cross-Replay on GPT's Weak Regions
| Metric / Model | Human | Museum | Low popularity | French | Asteroid |
|---|---|---|---|---|---|
| DeepSeek Acc. | 43.3 ± 9.2% | 62.2 ± 4.3% | 46.5 ± 7.7% | 50.1 ± 6.6% | 63.4 ± 8.7% |
| GPT Acc. | 17.6 ± 3.6% | 26.4 ± 8.7% | 21.7 ± 3.6% | 18.7 ± 3.9% | 23.4 ± 3.3% |
Cross-Replay on DeepSeek's Weak Regions
| Metric / Model | Ship | French | Low popularity | Chinese | Human |
|---|---|---|---|---|---|
| DeepSeek Acc. | 10.3 ± 7.1% | 21.3 ± 5.3% | 24.4 ± 8.0% | 26.2 ± 5.2% | 34.4 ± 5.4% |
| GPT Acc. | 21.4 ± 4.9% | 25.8 ± 3.5% | 32.3 ± 5.1% | 26.6 ± 4.7% | 29.7 ± 3.6% |
Key Findings:
-
Model-Specific Weakness Discovery: In 9 of 10 targeted marginal comparisons, the targeted agent performs worse than the replayed agent. The only exception is the Human marginal, already GPT-5.4 Mini's weakest top-1 marginal. This provides evidence that Active Testing identifies model-specific vulnerabilities rather than merely selecting universally difficult tasks.
-
Shared and Distinct Failure Regions: Our t-SNE visualization of final-round task configurations (Fig. 3(a)) reveals partially overlapping yet distinct regions across six agents, providing complementary descriptive evidence of shared and model-specific weakness patterns.
Together, these findings highlight the central diagnostic value of Active Testing: revealing model-specific capability boundaries and long-tail vulnerabilities that may remain hidden under a shared, fixed evaluation distribution.
Thank you for recognizing the practical value of our structured task generation, interpretable configurations, and feedback-driven weakness discovery framework.
We address your concerns point by point below.
Thank you for raising this question. We would like to clarify two important distinctions.
1. Objective of designed task space.
Our 12-dimensional task space systematically investigates how clue discriminativeness, evidence composition, reasoning topology, and domain knowledge affect agent performance, rather than merely constructing shortcut-free puzzles. As stated in Sec. 2.1: "Rather than predefining a small set of task families based on human priors, we factor the underlying knowledge-graph space into 12 low-level configuration parameters." This design maintains a meaningful difficulty gradient in the original task space.
2. Shortcut mitigation and empirical audit.
Dimensions such as Broad-first clue ordering control clue discriminativeness and candidate-set reduction, discouraging reliance on early discriminative clues. We additionally conducted a random audit of 55 tasks, examining single-clue and subset-only shortcuts.
These findings indicate that relatively few such shortcuts were identified in the audited sample. We will refine our statement in Section 2.2 and fix prefix constraint.
Thank you for highlighting the need to isolate the contribution of sequential feedback.
To disentangle this effect from agent-side generation and search noise, we construct Oracle environments with known ground-truth failure landscapes, including nonlinear interactions among task dimensions, ensuring identical conditions and budgets across selectors.
We compare Online RF, Frozen RF, and Random using the same 12-dimensional task space and active testing rounds from the paper. Across 3 models with distinct weakness distributions and 3 independent seeds, we obtain:
Online RF reduces mean AT-AUC by 4.90 pp relative to Frozen RF and 13.29 pp relative to Random, while maintaining higher diversity than Frozen RF.
These results support the value of sequential adaptation: in a large combinatorial task space, sequential feedback enables the selector to progressively discover vulnerable attribute combinations that frozen or random selection may miss.
Acquisition Component Ablation
We further examine failure prediction, uncertainty, diversity, and the cosine schedule through three independent runs.
Removing failure prediction raises AT-AUC by 11.46 pp, substantially weakening hard-task discovery. Uncertainty and diversity expose a meaningful trade-off: removing either may lower AT-AUC but also reduces diversity. These findings clarify the complementary roles of targeting difficult tasks and exploring the broader task space.