- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark to evaluate how web agents manage their own perception. It allows agents to decide what info to acquire, at what level of detail and also when sufficient evidence has already been collected. This is unlike the conventional evaluation setups where the harness automatically provides the observations. The authors introduce two eval protocols: HSO where the accessibility info and screenshots are automatically provided and also ACP where the agent explicitly selects the perception operations.
Critical strengths
- This is an important gap in existing web-agent benchmarks which the paper addresses.
- The benchmark is quite comprehensive. It has 652 tasks across diverse scenarios, evidence types and also difficulty levels. I think it provides a useful testbed for evaluating perception decisions.
- The evaluation is also extensive and covers 18 models under 2 protocols. The results are satisfactory and show improved cost-success AUC for 17 models and also reduced token usage for 16.
- The detailed analysis provides useful insights into the limitations of current models and also some potential directions for improvement.
Critical weaknesses
- The two protocols differ in multiple ways. And this makes it difficult to determine whether the efficiency gains are coming from better perception decisions or maybe simply from reduced observation volume and context accumulation.
- I feel that the behavioral metrics may not fully capture perception quality. The repeated observations can sometimes be necessary.
- ACP also introduces some capabilities like 3D rendering which are unavailable under HSO which can limit the fairness of the direct protocol comparison the authors have.
- Although there is the token efficiency, 8 models experience lower task success under ACP which may raise some concerns about the reliability of this approach across different models and scenarios. For me though, this is a minor concern.
Questions for authors
- Could the authors somehow provide a more controlled comparison that can isolate the effect of agent-controlled perception from differences in observation volume and context accumulation and also the available capabilities like 3D rendering as I mentioned above?
- Can you please clarify how do you distinguish unnecessary repeated observations from legitimate re-verification? This is particularly in tasks involving dynamic state changes.
- I would also want to know what explains the substantial drop in success rates for models such as MiniMax-M3 and Kimi-K2.6 under ACP?
- Could the authors also clarify how alternative valid evidence-acquisition paths are accounted for when you compute evidence recall?
4 comments
Response to Reviewer ZcUT (PART 2)
Q3. Why do MiniMax-M3 and Kimi-K2.6 have lower success under ACP?
We examined the failure endpoints, observation patterns, and tool-aligned results for MiniMax-M3 and Kimi-K2.6. Their patterns differ: MiniMax-M3's decline is associated with extended-tool use, while Kimi-K2.6 shows difficulty stopping after gathering evidence.
-
Failure endpoints. Among failed episodes, 78.7% for MiniMax-M3 and 62.0% for Kimi-K2.6 end in forced submission at the step cap rather than voluntary submission.
-
Observation behavior. Both models repeat nearly 30% of their perception calls, the two highest rates in the field. They take 14.1 and 9.4 screenshots per episode on average, respectively. These patterns help explain their 9.5- and 9.3-point success declines: additional observations consume steps and context without reliably leading to timely submission.
-
Evidence from the ablation. MiniMax's Avg@2 rebounds to 0.479 under alignment, nearly matching HSO at 0.481 and well above full ACP at 0.386, suggesting that extended-tool use contributes substantially to its 9.5-point decline. Kimi reaches 0.379 under alignment, below 0.400 under full ACP and 0.493 under HSO; its drop is more consistent with weak observation self-management: after gathering the annotated evidence, it continues for a median of 39 additional turns. Despite their success declines, Kimi's ΔAUC is +0.028, while MiniMax is the only model with negative ΔAUC, at −0.008.
To characterize the 8 models with lower ACP success, we analyzed 23,463 recorded ACP episodes across 18 models and two runs, without additional evaluations.
Table 2. Forced submission and evidence available at the step cap
| Model | Forced submission | Complete evidence at cap |
|---|---|---|
| MiniMax-M3 | 57.1% | 65.1% |
| Kimi-K2.6 | 43.0% | 62.6% |
| GLM-5V-Turbo | 39.0% | 62.2% |
| Qwen3.5-Plus | 33.3% | 65.3% |
| GPT-5.6-terra | 8.4% | 53.6% |
| GPT-5.1 | 0.0% | n/a |
| Field | 18.6% | 53.1% |
GPT-5.1 never reaches the step cap, so evidence-at-cap does not apply. Forced submissions are concentrated among models with high repeat rates.
The two largest success declines coincide with the highest forced-submission rates. Among the other six models, five decline by at most 1.1 points, within observed run-to-run variation; Gemini-3.6-Flash declines by 4.7 points. About half of forced submissions occur after the complete annotated evidence set has been acquired, indicating that stopping behavior is an important failure mode.
Together, these findings identify model-specific perception management, particularly tool selection and stopping, as key factors behind the declines.
Q4. Evidence recall and alternative acquisition paths
We appreciate this insightful question.
First, evidence recall does not determine task success: the programmatic grader evaluates the submitted answer independently of the acquisition path. A valid alternative path cannot be marked unsuccessful for low recall.
As a diagnostic, recall measures coverage of one annotated minimum-sufficient evidence set, with anchor matching and a modality gate for pixel evidence. Other successful paths need not match those anchors; for example, one targeted visual read (ocr) may replace two structural reads. We assessed possible undercounting in two ways.
-
Coverage among successful trajectories. Of 8,551 successful ACP trajectories, 85.6% fully match the annotated set, 0.8% partially match it, and 13.6% have no anchor-level match. The zero-match cases cluster in task families with small answer spaces or opportunities for prior-knowledge shortcuts, such as naming a real business from world knowledge. They are a minority, concentrated in settings where prior knowledge may help.
-
Scene-level upper bound. Treating any visit to the relevant site as coverage raises field recall from 64.7% to 82.6%, while model rankings remain substantially correlated, with Spearman ρ = 0.83. Anchor-based recall is therefore conservative in absolute terms, but the broad ranking patterns persist.
Therefore, alternative paths do not affect success grading, while the relaxed coverage check indicates that comparative trends are not solely driven by exact anchor matching.
We are grateful for your detailed feedback. If any further questions remain, we would be happy to provide clarification and discuss them.
Thank you for the detailed rebuttal. The authors have addressed all my major concerns properly and I am completely satisfied with the responses. This is a really nice paper which addresses an important problem and presents nice findings for others to build upon. Considering this, I have increased my score and would encourage acceptance.
Thank You for Raising the Score: 3 → 4!
Dear Reviewer ZcUT,
Thank you very much for your thoughtful follow-up and encouraging reassessment of our work! We sincerely appreciate the time and effort you have devoted to reviewing our paper. Your constructive feedback and insightful questions have helped us strengthen our analysis and clarify several important aspects of our work.
We are grateful for your positive recognition and willingness to raise your score. Thank you again for your careful consideration, valuable feedback, and support!
Best regards, Authors
Response to Reviewer ZcUT (PART 1)
We sincerely thank you for the detailed and careful assessment. Your four questions, on the controlled comparison, repeated reads, success declines, and evidence recall, are all central; we address them one by one below.
Q1. Disentangling control, observation volume, context, and tool capabilities
We appreciate this important methodological comment. Our tool-aligned ablation addresses the difference in available primitives: most token savings persist without targeted tools, which also help preserve success. Observation volume and context growth remain policy-dependent.
Design. We restricted ACP to HSO's full-page accessibility-tree reads and full-viewport screenshots, disabling cropping, targeted visual reads (
ocr), and 3D rendering. With observation primitives aligned, the policy still determines observation timing and volume. Five models were evaluated on all 652 tasks twice, for 6,520 new episodes.Table 1. Tool-aligned ablation (Each cell lists HSO / aligned ACP / full ACP, in that order.)
Table 1 shows that aligned ACP retains substantial token savings across all five models: token use is roughly half to two-thirds of HSO, close to full ACP. Targeted tools additionally improve Avg@2 by 2–5 points for three models.
Two sensitivity considerations further support this interpretation.
Render-only tasks: Excluding the 42 render-only tasks leaves 16 of 18 models with higher ACP AUC; mean ΔAUC changes only from +0.060 to +0.061. The overall pattern is therefore not driven solely by render-only tasks.
Context accounting: Both protocols use the same context and memory mechanisms. Actual observation volume and context growth are policy outcomes, not independently fixed variables.
Q2. Repeated observations and legitimate re-verification
We agree that rereading after state changes can serve legitimate verification. Our main-text rule counts later reads with the same (tool, target, URL) signature, including some valid verification. We therefore examine stricter definitions. Two sensitivity checks and a rate comparison provide consistent results.
Content-identity rule. Counting a repeat only when the returned content is also identical yields a highly correlated model ranking, with Spearman ρ = 0.92.
Excluding controlled-state-change tasks. Removing 37 such tasks and applying the original rule to the remaining 615 yields ρ = 0.99.
Rate retention. Under the stricter rule, MiniMax and Kimi still have the highest repeat rates, retaining about 90% and 82% of their original rates, respectively; models with lower repeat rates show larger reductions.
Trajectory analysis also shows that repeat share increases with call position and that around half of ACP episodes reaching the step cap have already covered the complete annotated evidence set. The repeat-rate ranking and broad diagnostic pattern remain stable across the stricter checks.