acceptodds
Under review as a conference paper at ICLR 2027

PerceptArena: Benchmarking Perception as a Decision in Web Agents

Abstract

Computer-use benchmarks have proven sharp instruments for measuring an agent's execution ability: given an observation, can it select the right actions? Yet nearly all of them share a single harness design: the environment is rendered and fed to the agent at every step, so the agent itself never decides what to observe. We introduce PerceptArena, which returns that control over perception to the agent itself: how much to observe, at what fidelity, and when to stop observing. PerceptArena is a controllable, self-hosted web environment comprising nine scenarios and a cross-scenario layer with 652 tasks in total; each feasible task places its decisive evidence at particular locations and information carriers, making perception allocation part of the task rather than a fixed input. We evaluate 18 vision-language models spanning frontier and mid-tier APIs under both the mainstream Harness-Supplied Observation protocol (HSO) and our proposed Agent-Controlled Perception protocol (ACP). Seventeen of the eighteen models attain a higher cost-success AUC under ACP, and sixteen spend less metered tokens; the median model spends 59% of its HSO metered tokens, average success is essentially unchanged, moving 2.1 points field-wide and 0.9 points with the two degenerate baselines excluded, and evidence recall across the field rises from 60.0% to 64.7%. Nevertheless, ACP also exposes a shared weakness of frontier models in perception self-management: models struggle to target the decisive evidence, track what they already hold, and stop once the evidence suffices. These observations remain hidden under the traditional HSO protocol. PerceptArena thereby makes perception allocation a measured behavior, and turns what to look at, what to keep, and when to stop into explicit, trainable targets.

Primary area:
datasets and benchmarks
Keywords:
Computer Use Agent, Benchmark, Agent-Controlled Perception

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZcUTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark to evaluate how web agents manage their own perception. It allows agents to decide what info to acquire, at what level of detail and also when sufficient evidence has already been collected. This is unlike the conventional evaluation setups where the harness automatically provides the observations. The authors introduce two eval protocols: HSO where the accessibility info and screenshots are automatically provided and also ACP where the agent explicitly selects the perception operations.

Critical strengths
  • This is an important gap in existing web-agent benchmarks which the paper addresses.
  • The benchmark is quite comprehensive. It has 652 tasks across diverse scenarios, evidence types and also difficulty levels. I think it provides a useful testbed for evaluating perception decisions.
  • The evaluation is also extensive and covers 18 models under 2 protocols. The results are satisfactory and show improved cost-success AUC for 17 models and also reduced token usage for 16.
  • The detailed analysis provides useful insights into the limitations of current models and also some potential directions for improvement.
Critical weaknesses
  • The two protocols differ in multiple ways. And this makes it difficult to determine whether the efficiency gains are coming from better perception decisions or maybe simply from reduced observation volume and context accumulation.
  • I feel that the behavioral metrics may not fully capture perception quality. The repeated observations can sometimes be necessary.
  • ACP also introduces some capabilities like 3D rendering which are unavailable under HSO which can limit the fairness of the direct protocol comparison the authors have.
  • Although there is the token efficiency, 8 models experience lower task success under ACP which may raise some concerns about the reliability of this approach across different models and scenarios. For me though, this is a minor concern.
Questions for authors
  • Could the authors somehow provide a more controlled comparison that can isolate the effect of agent-controlled perception from differences in observation volume and context accumulation and also the available capabilities like 3D rendering as I mentioned above?
  • Can you please clarify how do you distinguish unnecessary repeated observations from legitimate re-verification? This is particularly in tasks involving dynamic state changes.
  • I would also want to know what explains the substantial drop in success rates for models such as MiniMax-M3 and Kimi-K2.6 under ACP?
  • Could the authors also clarify how alternative valid evidence-acquisition paths are accounted for when you compute evidence recall?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZcUT (PART 1)

We sincerely thank you for the detailed and careful assessment. Your four questions, on the controlled comparison, repeated reads, success declines, and evidence recall, are all central; we address them one by one below.


Q1. Disentangling control, observation volume, context, and tool capabilities

We appreciate this important methodological comment. Our tool-aligned ablation addresses the difference in available primitives: most token savings persist without targeted tools, which also help preserve success. Observation volume and context growth remain policy-dependent.

Design. We restricted ACP to HSO's full-page accessibility-tree reads and full-viewport screenshots, disabling cropping, targeted visual reads (ocr), and 3D rendering. With observation primitives aligned, the policy still determines observation timing and volume. Five models were evaluated on all 652 tasks twice, for 6,520 new episodes.

Table 1. Tool-aligned ablation (Each cell lists HSO / aligned ACP / full ACP, in that order.)

ModelAvg@2Tokens/episode (k)Cost-success AUC
GPT-5.6-terra0.551 / 0.500 / 0.541495 / 329 / 2640.509 / 0.494 / 0.546
GLM-5V-Turbo0.327 / 0.363 / 0.3571,299 / 737 / 6570.254 / 0.328 / 0.339
Kimi-K2.60.493 / 0.379 / 0.400946 / 526 / 4540.391 / 0.364 / 0.419
MiniMax-M30.481 / 0.479 / 0.386984 / 516 / 5540.372 / 0.445 / 0.364
GPT-5.10.333 / 0.285 / 0.331181 / 119 / 1150.371 / 0.319 / 0.384

Table 1 shows that aligned ACP retains substantial token savings across all five models: token use is roughly half to two-thirds of HSO, close to full ACP. Targeted tools additionally improve Avg@2 by 2–5 points for three models.

Two sensitivity considerations further support this interpretation.

  • Render-only tasks: Excluding the 42 render-only tasks leaves 16 of 18 models with higher ACP AUC; mean ΔAUC changes only from +0.060 to +0.061. The overall pattern is therefore not driven solely by render-only tasks.

  • Context accounting: Both protocols use the same context and memory mechanisms. Actual observation volume and context growth are policy outcomes, not independently fixed variables.


Q2. Repeated observations and legitimate re-verification

We agree that rereading after state changes can serve legitimate verification. Our main-text rule counts later reads with the same (tool, target, URL) signature, including some valid verification. We therefore examine stricter definitions. Two sensitivity checks and a rate comparison provide consistent results.

  1. Content-identity rule. Counting a repeat only when the returned content is also identical yields a highly correlated model ranking, with Spearman ρ = 0.92.

  2. Excluding controlled-state-change tasks. Removing 37 such tasks and applying the original rule to the remaining 615 yields ρ = 0.99.

  3. Rate retention. Under the stricter rule, MiniMax and Kimi still have the highest repeat rates, retaining about 90% and 82% of their original rates, respectively; models with lower repeat rates show larger reductions.

Trajectory analysis also shows that repeat share increases with call position and that around half of ACP episodes reaching the step cap have already covered the complete annotated evidence set. The repeat-rate ranking and broad diagnostic pattern remain stable across the stricter checks.

AuthorsOpenReview ↗

Response to Reviewer ZcUT (PART 2)

Q3. Why do MiniMax-M3 and Kimi-K2.6 have lower success under ACP?

We examined the failure endpoints, observation patterns, and tool-aligned results for MiniMax-M3 and Kimi-K2.6. Their patterns differ: MiniMax-M3's decline is associated with extended-tool use, while Kimi-K2.6 shows difficulty stopping after gathering evidence.

  1. Failure endpoints. Among failed episodes, 78.7% for MiniMax-M3 and 62.0% for Kimi-K2.6 end in forced submission at the step cap rather than voluntary submission.

  2. Observation behavior. Both models repeat nearly 30% of their perception calls, the two highest rates in the field. They take 14.1 and 9.4 screenshots per episode on average, respectively. These patterns help explain their 9.5- and 9.3-point success declines: additional observations consume steps and context without reliably leading to timely submission.

  3. Evidence from the ablation. MiniMax's Avg@2 rebounds to 0.479 under alignment, nearly matching HSO at 0.481 and well above full ACP at 0.386, suggesting that extended-tool use contributes substantially to its 9.5-point decline. Kimi reaches 0.379 under alignment, below 0.400 under full ACP and 0.493 under HSO; its drop is more consistent with weak observation self-management: after gathering the annotated evidence, it continues for a median of 39 additional turns. Despite their success declines, Kimi's ΔAUC is +0.028, while MiniMax is the only model with negative ΔAUC, at −0.008.

To characterize the 8 models with lower ACP success, we analyzed 23,463 recorded ACP episodes across 18 models and two runs, without additional evaluations.

Table 2. Forced submission and evidence available at the step cap

ModelForced submissionComplete evidence at cap
MiniMax-M357.1%65.1%
Kimi-K2.643.0%62.6%
GLM-5V-Turbo39.0%62.2%
Qwen3.5-Plus33.3%65.3%
GPT-5.6-terra8.4%53.6%
GPT-5.10.0%n/a
Field18.6%53.1%

GPT-5.1 never reaches the step cap, so evidence-at-cap does not apply. Forced submissions are concentrated among models with high repeat rates.

The two largest success declines coincide with the highest forced-submission rates. Among the other six models, five decline by at most 1.1 points, within observed run-to-run variation; Gemini-3.6-Flash declines by 4.7 points. About half of forced submissions occur after the complete annotated evidence set has been acquired, indicating that stopping behavior is an important failure mode.

Together, these findings identify model-specific perception management, particularly tool selection and stopping, as key factors behind the declines.


Q4. Evidence recall and alternative acquisition paths

We appreciate this insightful question.

First, evidence recall does not determine task success: the programmatic grader evaluates the submitted answer independently of the acquisition path. A valid alternative path cannot be marked unsuccessful for low recall.

As a diagnostic, recall measures coverage of one annotated minimum-sufficient evidence set, with anchor matching and a modality gate for pixel evidence. Other successful paths need not match those anchors; for example, one targeted visual read (ocr) may replace two structural reads. We assessed possible undercounting in two ways.

  1. Coverage among successful trajectories. Of 8,551 successful ACP trajectories, 85.6% fully match the annotated set, 0.8% partially match it, and 13.6% have no anchor-level match. The zero-match cases cluster in task families with small answer spaces or opportunities for prior-knowledge shortcuts, such as naming a real business from world knowledge. They are a minority, concentrated in settings where prior knowledge may help.

  2. Scene-level upper bound. Treating any visit to the relevant site as coverage raises field recall from 64.7% to 82.6%, while model rankings remain substantially correlated, with Spearman ρ = 0.83. Anchor-based recall is therefore conservative in absolute terms, but the broad ranking patterns persist.

Therefore, alternative paths do not affect success grading, while the relaxed coverage check indicates that comparative trends are not solely driven by exact anchor matching.

We are grateful for your detailed feedback. If any further questions remain, we would be happy to provide clarification and discuss them.

Thank you for the detailed rebuttal. The authors have addressed all my major concerns properly and I am completely satisfied with the responses. This is a really nice paper which addresses an important problem and presents nice findings for others to build upon. Considering this, I have increased my score and would encourage acceptance.

AuthorsOpenReview ↗

Thank You for Raising the Score: 3 → 4!

Dear Reviewer ZcUT,

Thank you very much for your thoughtful follow-up and encouraging reassessment of our work! We sincerely appreciate the time and effort you have devoted to reviewing our paper. Your constructive feedback and insightful questions have helped us strengthen our analysis and clarify several important aspects of our work.

We are grateful for your positive recognition and willingness to raise your score. Thank you again for your careful consideration, valuable feedback, and support!

Best regards, Authors

Reviewer_qgaUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper introduces PerceptArena, a benchmark for evaluating how computer-use agents manage perception: what information to acquire, at what granularity, what to retain, and when to commit. It provides 652 tasks with heterogeneous evidence carriers and evaluates 18 models under harness-supplied observation (HSO) and agent-controlled perception (ACP).

The reported results show generally improved token efficiency under the evaluated ACP interface, alongside relatively small changes in aggregate task success. The paper additionally analyzes evidence coverage, repeated observation, and termination behavior.

I think The contribution is a controlled benchmark for evaluating perception-allocation competence, rather than the introduction of ACP as a capability or interface paradigm. The main areas for improvement are positioning the contribution accordingly and validating that the benchmark distinguishes perception-policy quality from harness-design effects.

Critical strengths
  • An important evaluation objective beyond task success. An ACP interface gives agents control over observation, but does not establish that they exercise that control effectively. Evaluating whether agents acquire sufficient evidence economically is therefore valuable even when selective-observation mechanisms already exist. PerceptArena operationalizes this objective through explicit observation decisions, cost accounting, and process-level diagnostics. This complements end-task success evaluation and could support future development of better perception policies.

  • Substantial controlled environment and benchmark construction. The suite includes textual, visual, and rendered evidence, dynamic state changes, cross-source information requirements, and irreversible actions. These features provide a useful setting for studying observation decisions under controlled conditions.

Critical weaknesses
  • The contribution should be positioned as evaluating ACP competence, not introducing ACP paradigm as many harness such as Codex is already in ACP mode. The paper should more clearly distinguish three layers 1) An interface that enables agent-controlled observation. 2) A benchmark that creates tasks requiring effective observation choices. 3) Measurements that reliably distinguish better and worse perception policies. The contributions should be at the latter two layers. Existing selective-observation interfaces do not make this benchmark unnecessary. Instead, they make systematic evaluation more relevant. However, they weaken a motivation based on perception control itself being generally unavailable.

  • Harness design and perception-policy quality remain entangled. ACP and HSO differ in several interface dimensions, as the authors acknowledge. A stronger evaluation would use a competitive ACP harness such as a Codex-based browser setup and compare agent-controlled acquisition, harness-controlled acquisition, and a strong heuristic policy using comparable capabilities. Comparisons among policies within ACP are particularly important: disabling ACP tests access to control, whereas policy comparisons test the quality of exercising that control.

Questions for authors
  • Can the authors clarify whether the primary contribution is a new observation interface or a benchmark for evaluating competence within such interfaces?
  • How does agent-controlled acquisition compare with a strong fixed or heuristic policy using the same observation primitives?
3 comments
AuthorsOpenReview ↗

Response to Reviewer qgaU

We sincerely thank you for the thoughtful perspective on how the contribution should be positioned. Your three-layer distinction helped us state that positioning more clearly; we first address the contribution and then the same-primitive comparison.


Q1. Contribution: interface, benchmark, and measurement

We appreciate this helpful framing. Enabling agent-controlled observation and evaluating how effectively agents use it are distinct goals.

Agent frameworks such as Codex already provide varying degrees of observation control. Yet access does not guarantee effective use: an agent may capture screenshots, inspect regions, or retrieve page information and still miss decisive evidence or misjudge when to stop. This makes systematic evaluation of the kind this paper provides increasingly important. PerceptArena contributes a benchmark and measurement framework for perception allocation, beyond simply providing an interface or task collection:

  1. Task layer. We construct 652 tasks across nine scenarios and a cross-scenario layer, spanning heterogeneous evidence carriers, dynamic state changes, and irreversible actions that make perception decisions consequential for success.

  2. Formulation layer. We formulate perception allocation as an explicit decision process: what to observe, where and at what fidelity to acquire information, what to retain, and when to stop.

  3. Measurement layer. We introduce process-level measures of observation-cost attribution, evidence recall, redundant-perception analysis, and stopping behavior. Paired evaluations of 18 models under HSO and ACP reveal current capabilities and limitations.

The interface enables agent-controlled perception; PerceptArena measures how well agents use that control. These roles are complementary. We will make this distinction explicit in the introduction and related work.


Q2. Policy comparison with the same observation primitives

With matched primitives, aligned ACP uses roughly half to two-thirds as many tokens as HSO for all five models, while improving AUC for GLM and MiniMax-M3 by about +0.07.

In this tool-aligned comparison, HSO automatically injects a full-page accessibility tree and full-viewport screenshot at every step. Aligned ACP uses the same primitives, with cropping, targeted visual reads (ocr), and 3D rendering disabled. Tasks and cost accounting are matched; observation timing, volume, and context growth remain policy-dependent. Five models were evaluated on 652 tasks with two runs each. Table 1 reports the results.

Table 1. Tool-aligned ablation (Each cell lists HSO / aligned ACP / full ACP, in that order.)

ModelAvg@2Tokens/episode (k)Cost-success AUC
GPT-5.6-terra0.551 / 0.500 / 0.541495 / 329 / 2640.509 / 0.494 / 0.546
GLM-5V-Turbo0.327 / 0.363 / 0.3571,299 / 737 / 6570.254 / 0.328 / 0.339
Kimi-K2.60.493 / 0.379 / 0.400946 / 526 / 4540.391 / 0.364 / 0.419
MiniMax-M30.481 / 0.479 / 0.386984 / 516 / 5540.372 / 0.445 / 0.364
GPT-5.10.333 / 0.285 / 0.331181 / 119 / 1150.371 / 0.319 / 0.384

Table 1 highlights the benefit of observation control with matched tools.

  • Token cost: All five models use roughly half to two-thirds as many tokens as HSO under aligned ACP.

  • AUC: GLM and MiniMax-M3 improve by approximately +0.07 over HSO.

  • MiniMax-M3: Its aligned AUC reaches 0.445, above HSO at 0.372 and full ACP at 0.364, consistent with unhelpful extended-tool use under full ACP.

These results demonstrate the token savings enabled by autonomous observation selection with matched primitives, together with AUC improvements for two models.

We sincerely appreciate your suggestions. If any further questions arise, we would be pleased to clarify and continue the discussion.

Thanks to the authors for providing the additional results and clarifications. These help better position the paper's contributions, particularly in evaluating ACP competence rather than introducing perception control as a new paradigm.

I have increased my soundness score and maintain my overall rating.

AuthorsOpenReview ↗

Thank You for Your Feedback and Reassessment

Dear Reviewer qgaU,

Thank you very much for your thoughtful follow-up and positive reassessment! We sincerely appreciate your constructive feedback, which has helped us better clarify our contributions and strengthen our analysis.

We are grateful for your recognition of our additional results and clarifications, as well as your decision to increase the soundness score. Thank you again for your time, careful consideration, and valuable suggestions!

Best regards, Authors

Reviewer_jQohOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PerceptArena, a benchmark for evaluating whether web agents can decide what to observe, how much detail to collect, and when they have enough information to act. The benchmark contains 652 tasks across nine scenarios and a cross-scenario setting. The paper evaluates 18 vision-language models under two protocols: Harness-Supplied Observation (HSO), where the harness provides observations automatically, and Agent-Controlled Perception (ACP), where the agent chooses its own observations. ACP improves the cost-success AUC for 17 of the 18 models and reduces token usage for 16.

Critical strengths
  • The paper addresses an important problem that is not addressed by existing web-agent benchmarks. Adding perception to the actions required within the harness is useful.
  • The benchmark covers a good range of scenarios and evidence types, including screenshots, scanned documents, spreadsheets, photographs, and 3D renders. Also, including changing environment states and tasks that require irreversible actions also makes the evaluation more challenging than simple information retrieval.
  • The analysis of missed evidence, repeated reads, and stopping behavior helps explain why agents struggle under ACP and provides useful directions for future work.
  • Appendices and code link provide good details and help in reproducibility.
Critical weaknesses
  • The comparison between HSO and ACP changes more than who controls perception. HSO always provides a full-page accessibility tree and screenshot, while ACP allows targeted reads, visual question answering on cropped regions, and 3D rendering. The 3D tool is not available under HSO, and 42 tasks can only be solved through that tool. The authors acknowledge that this is not a causal comparison, but it still makes it difficult to determine how much of the improvement comes from agent-controlled perception itself and how much comes from differences in the available tools and observation format. -The paper counts repeated calls based on the tool, target, and URL, but some tasks involve changes in availability, document contents, or other environment states. In those cases, reading the same page again may be necessary.
  • The cost-success AUC is based on token budgets, but tokenization and image-token cost differ across model APIs. The appendix acknowledges that absolute costs are expressed in each API's own units. This limits direct comparisons between models and makes it harder to interpret the AUC as a general measure of cost efficiency.

Minor Paper writing/presentation can be improved. For example, captions are short and not very information, and abstract contains many numbers that are hard to read without understanding the experimental setup.

Questions for authors
  • How would the results look if the available perception tools and modalities are kept the same across HSO and ACP?
  • Since token accounting differs across APIs, can the authors report latency and monetary cost alongside token usage, or clarify how the AUC should be interpreted when comparing different models?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jQoh

We sincerely thank you for the careful reading and the constructive suggestions. Your points on the same-tool comparison, repeated reads, and how AUC should be read are all constructive, and they helped us further improve the experiment and its interpretation. We address them in turn.


W1/Q1. Tool-aligned comparison: control or tools?

We ran a tool-aligned ablation that restricts ACP to HSO's two observation sources, full-page accessibility-tree reads and full-viewport screenshots, with cropping, targeted visual reads (ocr), and 3D rendering disabled. The observation primitives are aligned, while observation timing, volume, and context growth remain policy-dependent. We evaluated five models on all 652 tasks twice, yielding 6,520 new episodes.

With aligned primitives, all five models retain substantial token savings, and targeted tools further improve AUC and task success for several models.

Table 1. Tool-aligned ablation (Each cell lists HSO / aligned ACP / full ACP, in that order.)

ModelAvg@2Tokens/episode (k)Cost-success AUC
GPT-5.6-terra0.551 / 0.500 / 0.541495 / 329 / 2640.509 / 0.494 / 0.546
GLM-5V-Turbo0.327 / 0.363 / 0.3571,299 / 737 / 6570.254 / 0.328 / 0.339
Kimi-K2.60.493 / 0.379 / 0.400946 / 526 / 4540.391 / 0.364 / 0.419
MiniMax-M30.481 / 0.479 / 0.386984 / 516 / 5540.372 / 0.445 / 0.364
GPT-5.10.333 / 0.285 / 0.331181 / 119 / 1150.371 / 0.319 / 0.384
  • Token efficiency: Aligned ACP uses roughly half to two-thirds as many tokens as HSO for every model, retaining at least 70% of the savings achieved by full ACP.
  • Cost-success AUC: GLM and MiniMax gain +0.074 and +0.073 over HSO under alignment, and full ACP also exceeds aligned ACP for four of the five models.
  • Task success: Full ACP improves Avg@2 by 2–5 points over aligned ACP for three models.

Excluding the 42 render-only tasks, which are 6.4% of the suite, still leaves 16 of 18 models with higher ACP AUC, and the mean ΔAUC moves only from +0.060 to +0.061. The overall trend therefore does not depend on render-only tasks.


W2. Repeated reads after state changes

We acknowledge that repeated reads can be legitimate after state changes, and our main-text rule marks any later read with the same tool, target, and URL as repeated. We test this definition with two stricter checks.

  1. Content-identity rule. A read counts as repeated only if the returned content is also identical, and the 18-model ranking remains highly correlated, with Spearman ρ = 0.92.
  2. Tasks without state changes. Excluding all state-changing tasks yields ρ = 0.99.
  3. Rate retention. Under the stricter rule, MiniMax and Kimi still have the highest repeat rates, at about 90% and 82% of their original rates, while models with lower repeat rates show larger reductions.

The ranking and the diagnostic pattern are therefore robust to stricter definitions.


W3/Q2. Interpreting AUC, monetary cost, and latency

We thank the reviewer for raising this important issue. Token-based AUC: The improvement reported for 17 of 18 models compares ACP with HSO within each model, using the same 652 tasks, two runs per protocol, and consistent API-specific token accounting. It describes paired protocol differences rather than absolute cost rankings across model APIs.

Monetary cost: We repriced 46,861 recorded episodes offline using published API prices and caching rules. Of 16 models with available prices, 14 yield non-degenerate comparisons, and ACP has lower estimated monetary cost in 13 of them, with a median ACP/HSO cost ratio of 0.561, close to the 59% token ratio. When AUC is recomputed under a common monetary budget, it improves in 9 of the 14 comparisons.

Latency: The median ACP/HSO end-to-end latency ratio is 1.105, so ACP is about 10.5% slower by this measure. Fewer tokens do not necessarily mean shorter wall-clock time, because API round trips and queueing also contribute.

The most consistent finding is lower token consumption and estimated monetary cost under ACP. Token-based AUC should be read as a within-model protocol comparison, while monetary AUC and latency provide complementary efficiency views. We thank the reviewer for this suggestion, which helped us enrich the paper's efficiency analysis along the token, monetary-cost, and latency dimensions.


Writing and presentation

We will make figure and table captions self-contained by specifying the protocol, metric, and comparison baseline, and the new Table 1 will follow this standard. We will also streamline the abstract by introducing the evaluation setup before the key findings and reducing numerical clutter.

We sincerely appreciate your thoughtful feedback and would be glad to discuss any remaining questions.

Thank you for the rebuttal. The tool-aligned and repeated reads experiments are helpful. Based on the tool-aligned experiment, it seems that some AUC and success gain come from the targeted tools, so I would suggest to clarify that within the contribution claims "We evaluate 18 vision-language models under both harnesses and find that seventeen of them attain a higher cost-success AUC under ACP". It would be also be good to clarify the monetary cost and fix the paper presentation/captions in the revision. My initial rating was positive, so I am keeping the same rating.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Dear Reviewer jQoh,

Thank you very much for your thoughtful follow-up and constructive feedback! We sincerely appreciate your recognition of our additional experiments and clarifications.

We will carefully incorporate your suggestions into the revision, particularly by clarifying the contribution of targeted tools to the reported gains, refining the monetary cost analysis, and improving the paper's presentation and captions.

Thank you again for your time, valuable suggestions, and continued support!

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

For AC, SAC and PC: Summary of Rebuttal Process (Submission #4797)

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

As the discussion concludes, we sincerely thank the reviewers for their feedback and you for overseeing our submission.

PerceptArena establishes a benchmark for measuring how web agents manage perception as a decision, rather than treating observations as fixed inputs. Its 652 tasks require agents to choose what evidence to gather, at what fidelity, what to retain, and when to stop. Across 18 models evaluated under Harness-Supplied Observation (HSO) and Agent-Controlled Perception (ACP), we find substantial token savings alongside persistent difficulties in evidence selection, redundant reads, and stopping. PerceptArena makes these previously overlooked behaviors measurable, providing clear targets for more capable and efficient agents.

1. Reviewer ZcUT (Rating: 3 → 4)

  • Main concerns: The reviewer praised the importance of the problem, comprehensive benchmark design, extensive evaluation, and insightful analysis. Their questions concerned fairness between observation protocols, legitimate repeated reads, performance declines under ACP, and alternative evidence-acquisition paths.

  • Our response: We conducted a tool-aligned comparison, showing that substantial token savings persist even without additional targeted tools. We also verified the robustness of our repeated-read findings under stricter definitions, analyzed trajectories to explain model-specific difficulties with tool selection and stopping, and clarified how alternative evidence paths are handled in evaluation. These results further reinforce the reliability and diagnostic value of PerceptArena.

  • Reviewer's follow-up: We are especially grateful that the reviewer stated, “The authors have addressed all my major concerns properly and I am completely satisfied with the responses.” The reviewer also described our work as “a really nice paper which addresses an important problem and presents nice findings for others to build upon,” and raised the rating from 3 to 4, explicitly encouraging acceptance. We deeply appreciate this encouragement.

2. Reviewer qgaU (Rating: 3)

  • Main concerns: The reviewer valued evaluation beyond task success and our substantial benchmark construction. They encouraged us to clarify that our contribution lies in measuring how effectively agents use perception control, and to better distinguish perception-policy effects from harness differences.

  • Our response: We clarified that PerceptArena contributes tasks and process-level measures of perception quality, rather than merely access to observation tools. We also compared protocols using matched observation tools, showing that agent-controlled observation retains substantial efficiency benefits. This helped sharpen our contribution as a benchmark for perception-allocation competence, rather than a claim to invent observation control.

  • Reviewer's follow-up: The reviewer acknowledged that our results and clarifications better positioned the paper, increased the soundness assessment, and maintained the positive overall rating. We sincerely appreciate this thoughtful reassessment.

3. Reviewer jQoh (Rating: 3)

  • Main concerns: The reviewer praised the research question, diverse evidence, failure analysis, and reproducibility. Their questions concerned differences in perception tools, repeated observations after state changes, and the interpretation of token-based cost across model APIs.

  • Our response: We conducted tool-aligned experiments and a sensitivity analysis excluding render-only tasks, confirming that substantial efficiency benefits persist with matched observation tools and that the overall findings are not driven by render-specific tasks. We also tested stricter repeated-read definitions and performed additional monetary-cost and latency analyses. Furthermore, we clarified that token-based AUC is best interpreted as a within-model protocol comparison, and that targeted tools contribute to some gains rather than attributing everything to observation control alone.

  • Reviewer's follow-up: The reviewer found our experiments helpful and maintained the positive rating, while suggesting clearer presentation of the findings. We are grateful for this continued support.

Overall, all three reviewers maintained positive assessments, with ZcUT explicitly encouraging acceptance, qgaU increasing the soundness assessment, and jQoh recognizing the additional experiments. Their feedback prompted stronger controls and more precise interpretations, further supporting our central contribution: making perception allocation an explicit, measurable capability in web agents.

Thank you for your time and thoughtful consideration. We sincerely hope PerceptArena merits favorable consideration.

Best regards,

The Authors of Submission #4797

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.