acceptodds
Under review as a conference paper at ICLR 2027

Omni2Web: Benchmarking Audiovisual Web Editing with Weak References

10 views

Abstract

Screen-recorded web editing requests contain weak deictic expressions such as "this" and "there," whose referents depend on speech, cursor trajectories, page state, and edit history. Such requests require intent recovery beyond the explicit specifications assumed by many existing web-editing benchmarks. We introduce Omni2Web, a bilingual benchmark of 918 instances spanning 13,907 edit steps. It defines three complementary tracks: Direct Editing evaluates webpage editing from recordings, Instruction Recovery measures explicit intent recovery, and Instruction Utility tests whether recovered instructions can drive a fixed code executor. We evaluate 17 open- and closed-source models. The best models attain 51.17 on the Edit Fidelity Score (EFS) for Direct Editing and 49.14 on the Instruction Recovery Score (IRS); under the fixed executor, the strongest recovered instructions reach 51.08 EFS, still far below the 89.69 EFS obtained with oracle instructions. Step-level analyses show that correct grounding does not guarantee successful edits, while some Omni models recover instructions that the fixed coding model executes substantially better than their direct edits. Controlled ablations further demonstrate the value of temporally aligned audiovisual evidence, while alternative judges preserve the leader and broad ordering. Together, these findings reveal substantial headroom in multimodal intent recovery and code execution and highlight the promise of pairing Omni rewriters with coding models.

Primary area:
datasets and benchmarks
Keywords:
Audiovisual Understanding, Deictic Grounding, Web Editing, Omni Vibe Coding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pRJDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Omni2Web, a benchmark that evaluates how LLMs would handle fuzzy/unclear instructions. It devides the task into three subtasks: end-to-end capability, intent recovery and instuction utility. And it also proposed two major evaluation metrics: Edit Fidelity Score(EFS) and Instruction Recovery Score(IRS) to better compare the performance across different models.

Critical strengths
  1. This paper identifies an important challenge in modular AI systems: while specialized models are responsible for executing specific tasks, accurately interpreting and communicating user intent is essential for effective collaboration. The proposed benchmark provides a valuable evaluation setting for this problem.

  2. The experimental evaluation covers a wide range of mainstream model families, including both open-source and closed-source models.

  3. The ablation studies are thorough and well-designed, particularly those investigating the contributions of different input modalities and audiovisual temporal alignment.

Critical weaknesses
  1. Insufficient justification of the benchmark's distinctiveness. Although the authors demonstrate that the three evaluation tracks capture different aspects of model performance, it remains unclear whether their combination reveals model capability differences that cannot be adequately evaluated by existing benchmarks. A more systematic comparison with closely related benchmarks would help establish the necessity and novelty of the proposed evaluation framework.

  2. Limited discussion of the dataset's ecological validity. According to Section 3.2, the editing requests are based on predefined scripts rather than naturally occurring user interactions. It is unclear to what extent these scripted requests, particularly their frequent weak references and multi-step revisions, reflect realistic webpage editing scenarios. Further discussion or empirical evidence regarding the representativeness of the collected data would strengthen the benchmark's practical relevance.

  3. Potential executor dependency in Instruction Utility evaluation. Track C primarily relies on a single fixed coding model, DeepSeek-v4-pro, to evaluate the utility of recovered instructions. Consequently, the results may reflect not only the quality of instruction recovery but also the compatibility between the recovered instructions and the chosen executor. Although the authors provide partial self-execution experiments, evaluating additional executors would help establish whether the reported findings generalize across different coding models.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer pRJD (Part 1/2)

We thank the reviewer for recognizing the importance of interpreting and communicating intent in modular systems, the broad model coverage, and the modality/temporal-alignment ablations.

Corresponding manuscript revisions are highlighted in red.

1. Distinctiveness and the value of three tracks

Omni2Web evaluates how models recover requested edits from synchronized speech and cursor references, and how that intent translates into code. Our original Related Work distinguishes editing supplied HTML from recreating a webpage. We have expanded this comparison in Section 2 with a feature matrix.

We recognize the context already studied by related benchmarks: EDIT-Bench includes highlighted code and studies IDE cursor context; FronTalk provides multi-turn textual or visual feedback; WebVR and IWR-Bench reconstruct webpages from reference videos and assets.

BenchmarkCode editingImage/videoSpoken requestRecoveryUtility
EDIT-Bench✓✗✗✗✗
FronTalk✓✓✗✗✗
WebVR✗✓✗✗✗
IWR-Bench✗✓✗✗✗
Omni2Web✓✓✓✓✓

Editing means modifying existing code, including later FronTalk turns. Recovery separately scores model-recovered intent; Utility executes that text with a common coding model. These columns describe benchmark settings and evaluation targets.

Omni2Web's recordings and annotations are newly collected around existing source HTML. Models must resolve which page elements the speaker means, what should change, and which changes remain after revisions. The shared-instance A/B/C protocol then connects this intent recovery to editing outcomes.

What the combined evaluation reveals. Qwen3.5-Omni-Plus obtains 29.56 EFS in direct editing, while its recovered instructions yield 51.08 EFS with the fixed coding model. Both use the same outcome metric on the same instances. Its 49.14 IRS separately measures recovered target/action content. Thus, direct-editing performance alone would miss useful recovered intent, while text recovery alone would leave its practical editing utility untested.

Conversely, Gemini 3.5 Flash grounds the target correctly but fails to fulfill the edit on 16.68% of scored active Track A steps. The three tracks expose these different understanding/execution profiles on shared requests, with execution capability held fixed across rewriters in Track C.

Paper revisions. Section 2 (Related Work) and the new Table 1 now present the systematic comparison and explain the diagnostic role of the three tracks.

2. Ecological validity and the benchmark's intended scope

Omni2Web deliberately focuses on weakly referential editing to distinguish models' intent-recovery capabilities. When a user says "this" while pointing, the system must infer a concrete target and action from the shared audiovisual context. Our benchmark is designed to assess how well models handle these requests.

Why this setting matters. In the original evaluation, the same DeepSeek-v4-pro executor reaches 89.69 EFS with explicit oracle instructions, versus 51.08 with the strongest recovered instructions, on the same 918 instances (Table 3). This shows substantial ability to execute specified edits alongside a large gap when the instructions must first be recovered from recordings. It motivates evaluating the recovery of weakly expressed intent in addition to code execution from explicit specifications.

The scripts provide checkable targets, actions, and undo dependencies, while annotators record the speech and cursor pointing themselves. Deliberate coverage of weak references and revisions supports systematic comparison of the capability we aim to evaluate.

Scope and paper revisions. We have not measured the frequency of weak references or revisions in naturally occurring editing sessions. We have added Appendix A.17 (Limitations) to state that these frequencies are benchmark design choices and that our results characterize performance within this interaction setting. Naturalistic data would be needed to measure that prevalence and assess generalization to everyday requests. Section 3.2 and Appendix A.14 describe the collection procedure; Tables 25–26 retain the complete bilingual examples.

AuthorsOpenReview ↗

Response to Reviewer pRJD (Part 2/2)

3. Dependence on the Track C executor

Thank you for suggesting an additional executor. The ordering of the five tested rewriters and the large recovered-to-oracle gap persist with a second shared executor. We have completed a GLM-5.3 comparison on the same fixed 300 instances covering 129 source webpages, using the original recovered instructions and one shared oracle condition: 1,800 evaluated entries in total. GLM-5.3 is outside the original 17-model evaluation pool.

Role of a common executor. It holds implementation capability fixed across rewriters, allowing their recovered instructions to be compared through the same coding model. We additionally calibrate utility against each executor's own oracle on the same instances. The new comparison keeps the source HTML, recovered instructions, instance IDs, and EFS protocol fixed; historical DeepSeek outputs are reused and GLM outputs were collected later.

Instruction sourceDeepSeek EFSDeepSeek NIUGLM EFSGLM NIU
Qwen3.5-Omni-Plus50.3256.3250.6156.26
Gemini 3.5 Flash49.3255.2049.6455.18
GPT-5.6 Sol46.1051.5946.1651.30
Grok 4.636.7541.1339.0843.44
Qwen3.8-27B28.2131.5829.2632.52
Oracle89.35100.0089.96100.00

EFS uses a 0–100 scale; NIU is the percentage of each executor's own oracle EFS. GLM columns show the completed budget-extension condition below.

What the results show. The observed EFS point-estimate ordering of the five rewriters is identical across executors (Spearman = 1.00), with Qwen3.5-Omni-Plus and Gemini 3.5 Flash remaining the top two. This ordering also holds for GLM before the budget extension. The best recovered instructions leave oracle gaps of 39.03 EFS points with DeepSeek and 39.35 with GLM. Thus, both the ordering and substantial instruction-utility headroom are reproduced with another common coding model. Absolute scores can still vary: Grok's extended GLM score exceeds its historical DeepSeek score by 2.33 EFS, with a 95% source-page bootstrap interval of [1.00, 3.67]. The paired intervals use 2,000 source-page replicates, holding generated outputs and judge scores fixed.

Generation budget and complete coverage. GLM initially used a 65,536-token output budget, including reasoning tokens. We retried only the 192 length failures at 131,072 tokens, including 34 Oracle cases, and retained the other 1,608 scores unchanged. Each retry keeps its first terminal model outcome. The final 1,800 entries include five historical upstream-input zeros and one Oracle length-budget zero. The reported GLM scores reflect this targeted retry condition.

Complementary self-execution control. Our existing control also compares DeepSeek with each rewriter executing its own instructions, including an oracle for each executor. All values below are EFS on the same 300 instances.

RewriterDeepSeek: recovered / oracleSelf: recovered / oracle
Gemini 3.5 Flash49.32 / 89.3553.48 / 90.45
Qwen3.5-Omni-Plus50.32 / 89.3545.20 / 67.43

Substantial recovered-to-oracle gaps also remain under self-execution. Together, the two controls support the instruction-utility findings across the tested execution settings. We report utility relative to each executor, with the shared-executor ranking comparison scoped to these five rewriters and 300 instances.

Paper revisions. New Appendix A.5 and Table 14 report the shared-executor EFS and own-oracle NIU comparison and its implications. Appendix A.4 links this comparison to the existing self-execution control in Table 13.

Reviewer_MWjhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Omni2Web, a bilingual benchmark for screen-recorded webpage editing requests. Building on 129 source webpages reused from WebVR, the authors collect new editing recordings and annotations, resulting in a benchmark comprising 918 instances and 13,907 edit steps. The benchmark defines three evaluation tracks: Direct Editing, Instruction Recovery, and Instruction Utility, which respectively assess models' ability to directly edit webpages given the source HTML and recordings, recover the user's actual editing intent from recordings, and perform webpage edits using the recovered instructions and source HTML. The authors evaluate 17 open- and closed-source models on this benchmark. The results indicate substantial room for improvement in multimodal intent recovery and code execution.

Critical strengths
  1. Introducing a new benchmark for webpage editing. The benchmark focuses on screen-recorded webpage editing requests, requiring models to understand user intent by combining speech, cursor positions, and page context. It provides a new evaluation setting for webpage editing capabilities.
  2. The paper is clearly and concisely written and easy to follow.
  3. The authors evaluate 17 open- and closed-source models on the benchmark. The results show clear performance differences across models, demonstrating the benchmark's discriminative ability.
Critical weaknesses
  1. Insufficient justification for evaluation weights and sensitivity of some model rankings to weight choices. The Edit Fidelity Score (EFS) combines multiple evaluation dimensions using a weighted sum, but the choice of weights is not sufficiently justified. Although the authors conduct a weight sensitivity analysis, they do not provide the complete set of 26 EFS weights or explain how these configurations were selected. Furthermore, under one No-damage-heavy weight configuration, recalculating the rankings using the results in Table 1 shows that 40% of the models on Track A shift by at least three positions, indicating that some model rankings are sensitive to the choice of weights.

  2. Lack of consistency validation for the final rendered webpage. Although the editing requests rely on visual information, the final editing results are primarily evaluated by comparing the original and modified HTML, without systematic validation of their consistency with the actual rendered results.

  3. Limited diversity of editing operations. The existing operations mainly focus on changes to page styles, layouts, and content, while lacking common webpage editing operations such as element insertion and modifications to interactive behavior. This limits the benchmark's coverage of real-world webpage editing scenarios.

  4. The current evaluation requires models to output the complete HTML document, which limits the direct evaluation of common editing approaches that generate only patches.

  5. Insufficient details on dataset construction. The paper describes the dataset construction, edit script design, annotation, and review processes mainly at a high level, without providing sufficiently detailed descriptions of the procedures. Besides, although some editing examples are presented, the paper lacks complete representative editing instances, making it difficult for readers to understand the correspondence between the original editing requests, annotations, and final webpage modifications.

  6. A dedicated ethics statement and data availability information should be provided. For a newly introduced benchmark, it's better that the authors include a separate ethics statement to provide more complete disclosure of the data collection process, annotators' informed consent, and data usage. In addition, the paper should clarify whether there are plans to publicly release the benchmark data and evaluation code, and how these resources can be accessed.

Questions for authors

Please see the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer MWjh (Part 1/3)

We thank the reviewer for recognizing the practical value of audiovisual webpage editing, the clarity of the presentation, and the benchmark's ability to distinguish current models.

Corresponding manuscript revisions are highlighted in red.

1. EFS weights and ranking sensitivity

EFS provides a fixed overall comparison; component scores support different application priorities. Table 2 already reports G/E/N/R alongside EFS. Readers can directly assess edit fulfillment, preservation, grounding, or revision handling according to their intended use.

Default weights. Completing requested edits is the primary objective, so E receives 0.50. N and R receive 0.20 each; G receives 0.10 because locating a target alone does not complete an edit. This prioritizes fulfilling edits over preservation alone.

Interpreting rank changes. Reweighting changes the evaluation priorities. Qwen3.8-27B scores 82.94 on No-damage but 7.56 on Edit Fulfillment (Table 2), so emphasizing preservation improves its relative position. This is an interpretable tradeoff; a rank change alone does not invalidate the metric. It motivates reading EFS together with the component scores.

Sensitivity and disclosure. Table 18 gives all 26 vectors: 25 points on a 0.1-spaced grid, with components in [0.1,0.6], sum one, and E largest or tied-largest; plus equal weights. Crossing these with α∈{0.05,0.10,0.15,0.20,0.25,0.30} yields 156 configurations. They cover fulfillment-prioritizing choices and a balanced reference.

ConfigurationGENR
Default0.100.500.200.20
Equal weights0.250.250.250.25
No-damage-heavy example0.100.400.400.10

In the original model set, all 156 configurations preserve the A/C leaders, and 150 retain Track A Spearman ρ≥0.9. In the reviewer's No-damage-heavy case (α=0.15), 6 of 15 Track A models (40%) move at least three places (ρ=0.743). This sensitivity to changed priorities coexists with stable leaders and broad ordering in most tested configurations. The component breakdown makes these tradeoffs explicit.

Paper revisions. Appendix A.8 explains the default weights and ranking sensitivity; Table 18 gives the complete grid and selection rule. The full component breakdown remains in Table 2.

2. Validation against rendered webpages

Our existing human calibration includes direct inspection of rendered source and edited webpages for 50 Track A outputs. These are the Track A portion of the original calibration study, which includes 50 Track A and 50 Track B outputs.

One evaluator inspected the pages in a browser and answered the corresponding grounding, edit-fulfillment, and revision rubric items. We use the evaluator's first responses, recorded before the automatic decisions were revealed. Thus, the human judgments incorporate the rendered outcome, while the automatic judgments use the benchmark's HTML/rubric evaluation.

We compare the two evaluations at two levels. Item agreement is the percentage of paired rubric decisions that agree. For output-level correlation, we aggregate each component within each output and compute Pearson's r between the human and automatic component scores across outputs. The comparison uses the same semantic requirements on both sides.

ComponentPaired itemsAgreementPearson r
Grounding65075.69%.867
Edit fulfillment52876.52%.840
Revision12268.85%.361

Agreement with rendered outcomes. Human and automatic assessments are strongly correlated for target grounding (r=0.867r=0.867) and edit fulfillment (r=0.840r=0.840), directly linking the HTML/rubric scores to the requested changes visible in the browser. As discussed in the paper, the judge is more conservative on edit fulfillment while preserving relative variation across outputs.

The human evaluator checks the same target, edit, and revision requirements on the rendered source and output pages. This directly tests the correspondence raised by the reviewer: whether the automatic rubric scores reflect the requested editing outcome. Together with the alternative-judge results in Section 6.2, this existing Track A browser calibration supports using the automatic evaluator as a consistent and reproducible basis for comparing editing performance.

Existing manuscript evidence. Section 6.2, Appendix A.7, and Table 16 report the original study with 50 Track A and 50 Track B outputs. The rendering comparisons above use its 50 Track A outputs; Table 16 and the original calibration statistics are unchanged.

AuthorsOpenReview ↗

Response to Reviewer MWjh (Part 2/3)

3. Scope of editing operations

Our present benchmark targets audiovisual intent recovery for edits to existing pages, and we have made the boundary of that scope explicit.

The 12 operation categories include visual and content edits as well as structural moves/swaps, layout changes, image replacement, region-level changes, cross-element styling, and revisions. These operations require resolving what the user points to, what should change, and which earlier changes remain active. The benchmark therefore tests dependencies and structural edits in addition to isolated text or style substitutions.

Element insertion and interactive-behavior modification are not covered as dedicated categories. We agree that both would broaden the benchmark's coverage. They would also require additional collection and evaluation: insertion needs annotations for the intended new element and placement, while behavior changes require checks of the resulting interactions. We do not treat the current 12 categories as covering all front-end development.

This scope is sufficient for studying the specific speech/cursor grounding and multi-step intent-recovery problem introduced in the paper. Extending operation coverage is a valuable next step, and the revised discussion explicitly identifies these missing categories so that readers can interpret the results within the evaluated task.

Paper revisions. Appendix A.14 now describes the current operation scope and explicitly notes that insertion and interactive-behavior changes need new data and functional checks.

4. Evaluating patch-based editing systems

Thank you for raising this interface concern. Complete HTML is the evaluator's final input; a model can use patches to produce that input. Given source HTML HH and a predicted patch PP, the system submits H^=apply⁡(H,P)\widehat H=\operatorname{apply}(H,P) to the existing rubric, image-check, and no-damage pipeline. The benchmark thus supports patch-based systems through a deterministic application step.

Added experiment. We completed 900 Patch evaluations: GPT-5.6 Sol, Gemini 3.5 Flash, and Qwen3.5-Omni-Plus each on the same 300 instances spanning 129 source pages, using their original Track A inputs without timestamp augmentation. Models directly generate SEARCH/REPLACE blocks, which are applied sequentially through unique exact matches, without fuzzy matching or repair. Generation and application failures score zero within the full 300-instance denominator. Explicit NO_CHANGES outputs are evaluated as unchanged source pages.

ModelFull HTML EFSPatch EFSPatch minus HTML
Gemini 3.5 Flash50.5245.59-4.93
Qwen3.5-Omni-Plus29.1023.19-5.91
GPT-5.6 Sol42.6641.79-0.87

Full-HTML outputs and scores are reused from the original evaluations on exactly these IDs. This experiment demonstrates a working evaluation path for model-generated patches: the same EFS measures the final pages while retaining patch-application failures. The differences are descriptive comparisons with historical full-HTML runs, rather than an isolated causal effect of output format.

Paper revisions. Section 4.1 now states how patch-based systems use the evaluator. Appendix A.15 and Table 27 add the application protocol, the three-model results, and the historical-baseline interpretation.

AuthorsOpenReview ↗

Response to Reviewer MWjh (Part 3/3)

5. Construction details and complete examples

Thank you for requesting further procedural detail and complete examples.

The original manuscript already describes the construction workflow and annotation structure. Section 3.2 explains the selection of 129 WebVR source pages, the design of page-specific scripts covering 12 operation types, script review before recording, natural speech with cursor pointing, manual transcription and rubric annotation, and consistency review before inclusion. In particular, the scripts specify the requested changes and their order, and the final review checks consistency in targets and edits.

Section 3.1 specifies the instance contents and how each ordered step is represented by paired grounding and edit-fulfillment/revision items, with an additional no-damage item. It also states that explicit reference instructions are constructed deterministically from the paired rubric anchors. These sections describe how recordings and annotations form evaluable editing instances.

Additional detail in Appendix A.14. Building on this description, we explain the annotation guidance and correspondence between stored fields. Target anchors identify the relevant element or region; expected effects specify the requested text, colors, layout, image subject, or state to restore. The review checks consistency among the requested targets, edits, and ordered annotations. The appendix also distinguishes collection guidance from measured frequencies in the final dataset.

The stored before_anchor identifies the target, after_expected_change specifies the required action, and the target item's distractor_note retains the spoken request. Operation labels and ordered indices link these fields to the paired rubric items and explicit reference instructions. This makes the connection between the original request and the evaluation criteria explicit.

Complete instances in Tables 25–26. The added examples show every step of one Chinese and one English ten-step instance, tracing spoken requests through resolved targets/actions and undo links to the required final state.

Spoken languageInstanceIllustrative undo links
Chineseb0804_human_101_004Step 5 undoes step 1; step 10 undoes steps 3 and 7
Englishb0810_human_192_002Step 4 undoes step 3; step 10 undoes step 9

We selected the lexicographically first ten-step instance in each language without consulting model outputs or scores, and reproduce the Chinese requests in their original language. Readers can follow which changes are cancelled, which remain active, and what the final webpage must satisfy. These semantic requirements allow multiple correct HTML implementations.

Paper revisions. The additional construction and annotation details are consolidated in Appendix A.14, with complete instances in Tables 25–26, complementing the workflow and annotation structure already described in Sections 3.1–3.2.

6. Ethics and public data/code availability

Thank you for recommending a dedicated ethics statement.

Ethics and consent. Section 3.2 already describes the team of 15 paid annotators, standardized training, and informed consent for research use and public data release. The 129 source webpages reused from WebVR are fictional, constructed webpages. The recordings show the webpages and cursor movements, retain the participants' original voices, and contain no footage of their faces. We obtained signed informed consent forms from every recording participant, covering research use and public data release.

Data and evaluation code. We have already fully open-sourced the benchmark data and evaluation code. To preserve anonymity during double-blind reviewing, we cannot include identifying links in this response. We will include links to the code and evaluation data in the final published version of the paper.

Paper revisions. We have added a dedicated Ethics Statement in Appendix A.16, documenting the nature of the webpages, the recording content, and written informed consent.

Reviewer_HexmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Omni2Web, a benchmark for understanding user intent and editing webpages from screen recordings containing speech, cursor pointing, and weak deictic expressions. It comprises 918 bilingual instances, 129 source webpages, and 13,907 historical edit steps, with three evaluation tracks: Direct Editing, Instruction Recovery, and Instruction Utility under a fixed executor. The authors evaluate 17 open- and closed-source multimodal models. Modality and temporal-alignment ablations, cross-track comparisons, and robustness analyses reveal differences between intent recovery and code implementation capabilities, as well as substantial room for improvement.

Critical strengths
  1. The paper is clearly written and well motivated. It addresses a practical webpage editing task in which users express requests through speech and cursor-based references. The task background, research objectives, and evaluation design connect naturally.

  2. The experiments are comprehensive, with extensive robustness analyses. Evaluating 17 multimodal models spanning Omni and VL systems provides a useful performance reference for the community. Alternative judges, human calibration, 156 EFS configurations, source-page cluster bootstrap, and sensitivity analyses for invalid outputs strengthen confidence in the results.

  3. The three evaluation tracks offer useful diagnostic value. Direct Editing, Instruction Recovery, and Instruction Utility provide complementary views of model capabilities. In particular, instruction recovery and the fixed executor help reveal differences between models' ability to understand intent and implement changes in code.

  4. The dataset represents substantial construction effort and includes multi-step dependencies and revisions. Omni2Web contains 918 bilingual instances and 13,907 historical edit steps. Each instance includes 10–20 steps and undo operations, allowing it to examine instruction dependencies beyond isolated single-step edits.

Critical weaknesses
  1. Track B does not sufficiently account for step order. The current matching score does not penalize ordering errors, although the order of edits and undo operations can affect the final result. IRS may therefore miss such errors, which warrants further validation.

  2. The advantage of Omni models needs to be distinguished from the effect of temporal alignment. VL performance improves substantially when timestamps are added (Table 10), suggesting that leaderboard gaps may partly reflect differences in temporal information. Including this setting as a main baseline would make the comparison more complete.

  3. Correct HTML edits do not necessarily produce correctly rendered pages. Tracks A/C are scored primarily from HTML and rubrics, with insufficient validation of the rendered results. Rendering checks or human assessment would help establish whether the scores reflect actual page quality.

  4. The visual sampling differences between VL and Omni models need clarification. Some VL models receive frames at 1 FPS, whereas Omni models receive native video, potentially preserving different amounts of temporal detail. The sampling settings should be reported, alongside a controlled comparison with visual input conditions matched as closely as possible.

Questions for authors
  1. Can the current IRS detect errors in step order or undo dependencies that affect the final result?
  2. How would the conclusions about Omni models' advantage change if a VL baseline with timestamped transcripts were included?
  3. Could the authors provide evidence of agreement between HTML-based scores and the quality of pages actually rendered in a browser?
  4. What controllable frame rates, frame counts, and resolutions are used for each model, and do the performance differences persist when these conditions are matched?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Hexm (Part 1/4)

We thank the reviewer for recognizing the clear motivation, comprehensive experiments, diagnostic value of the three tracks, and effort involved in collecting multi-step requests.

Corresponding manuscript revisions are highlighted in red.

1. IRS and order/undo dependencies

We deliberately keep IRS as a content metric, because a single reference order is not a unique correct representation of the user's intent.

For example, changing a heading's color and changing an unrelated button's text can be performed in either order. Requiring the reference's exact total order would penalize such valid rewrites. The appropriate temporal requirement is to preserve dependencies where they affect meaning or execution. Track B retains the full historical instruction sequence, including revisions; it does not reduce the reference to only the final net edits.

Temporal reasoning is still necessary for content recovery: the model must determine which element "this" refers to and which earlier change an undo targets. Recovering the wrong target or action can lose IRS credit. We agree with your specific concern that a harmful reordering of otherwise correct target/action content can retain credit under IRS. Tracks A/C provide complementary outcome evidence, but do not guarantee detection of every such error.

Additional analysis. We conducted a text-only dependency audit for five models under their original input conditions, covering all 918 IDs per model and 4,590 evaluation entries in total. The original target/action matching and reported IRS are frozen. The auditor examines reference and recovered instruction text together with that matching; it receives no audiovisual inputs.

Dep. removes previously earned action credit for steps flagged by the LLM audit as having harmful order/undo dependencies. Dep.+amb. also removes the corresponding credit for ambiguous cases. Neither calculation rematches outputs or awards new credit. All scores below use the 0–100 scale.

ModelOriginal IRSDep.Dep.+amb.
Gemini 3.5 Flash47.4647.2147.20
Qwen3.5-Omni-Plus49.1448.9548.95
GPT-5.6 Sol48.3447.9747.96
Claude Opus 548.9148.6048.57
Grok 4.640.8340.5040.49

IRS decreases by 0.19–0.37 points, or 0.19–0.38 when ambiguous credit is also removed, and all five model ranks remain unchanged. Thus, the tested dependency correction has a small effect on this content comparison. This supports retaining IRS while reporting dependency sensitivity separately. These are LLM-audit sensitivity results, not human-validated error rates or bounds on undetected ordering errors.

Paper revisions. Section 4.2 now explains the order-invariant content objective. Appendix A.12 and Table 23 add the dependency protocol, results, and scope.

AuthorsOpenReview ↗

Response to Reviewer Hexm (Part 2/4)

2. Timestamp-augmented VL baselines

Thank you for suggesting this comparison. Our primary leaderboard focuses on models' own intent recovery under the original input protocol. +TS adds an external alignment aid and therefore evaluates an assisted VL system. We have added full-set Track A and Track B comparisons for all three VL models, together with two additional strong native Omni models.

All rows use the same 918 instance IDs. For Omni models, base means native audio/video input. For VL models, base means 1 FPS frames plus the original manual transcript. The frames already carry their timestamps; +TS additionally attaches externally derived speech-interval timestamps to the same transcript text. The frame files, source pages, and scoring protocol are retained. A reports EFS and B reports IRS; † denotes an additional Omni evaluation.

ModelA baseA +TSB baseB +TS
Gemini 3.5 Flash51.17—47.46—
Gemini 3.8 Flash†56.95—50.75—
Qwen3.5-Omni-Plus29.56—49.14—
Qwen3.8-Omni-Flash†56.65—51.62—
GPT-5.6 Sol42.8653.9248.3458.71
Claude Opus 547.3757.1448.9156.60
Grok 4.641.4558.4040.8355.67

Native audiovisual advantage. The results support the native audiovisual advantage under the original input protocol: Omni models lead A/B in the original 17-model evaluation, and both additional native Omni models exceed all three plain-transcript VL baselines on A/B. They achieve these results directly from the recording's audio/video, without an external speech-to-frame alignment stage.

What +TS adds. The timestamps are derived from the same recording through external audio processing and ASR-assisted alignment. The speech-interval annotations make temporal correspondence explicit in the VL input, transferring part of the temporal-alignment work to preprocessing. The +TS score therefore evaluates the VL model together with an external alignment stage. This is additional assistance supplied to the model, rather than temporal correspondence recovered by the model from the recording itself.

To keep the primary comparison focused on models' own intent-recovery capability under the original input interfaces, Table 2 retains results without timestamp assistance; Appendix A.3 (Table 12) reports the assisted systems separately.

The assisted results are strong: the best +TS VL result exceeds all four native Omni references on A, and all three +TS VL models exceed them on B. These higher assisted scores do not overturn the native Omni advantage observed under the original protocol. They show the performance attainable when an external alignment stage is added. Historical and new API evaluations occurred on different dates, so these score differences alone do not isolate a causal effect of timestamps.

Paper revisions. Appendix A.3 and Table 12 present the full comparison and explicitly distinguish native audiovisual intent recovery from externally assisted VL performance. The revised discussion explains why the primary leaderboard retains the original inputs. Section 5.1 links to this comparison.

AuthorsOpenReview ↗

Response to Reviewer Hexm (Part 3/4)

3. Agreement with browser-rendered pages

Our existing human calibration includes direct inspection of rendered source and edited webpages for 50 Track A outputs. These are the Track A portion of the original calibration study, which includes 50 Track A and 50 Track B outputs.

One evaluator inspected the pages in a browser and answered the corresponding grounding, edit-fulfillment, and revision rubric items. We use the evaluator's first responses, recorded before the automatic decisions were revealed. Thus, the human judgments incorporate the rendered outcome, while the automatic judgments use the benchmark's HTML/rubric evaluation.

We compare the two evaluations at two levels. Item agreement is the percentage of paired rubric decisions that agree. For output-level correlation, we aggregate each component within each output and compute Pearson's r between the human and automatic component scores across outputs. The comparison uses the same semantic requirements on both sides.

ComponentPaired itemsAgreementOutputs for rPearson r
Grounding65075.69%47.867
Edit fulfillment52876.52%47.840
Revision12268.85%50.361

Three outputs contain unknown grounding/edit labels and are excluded from the corresponding output-level correlations. The table preserves the actual denominators rather than treating every component as having 50 complete output scores.

Agreement with rendered outcomes. Human and automatic assessments are strongly correlated for target grounding (r=0.867r=0.867) and edit fulfillment (r=0.840r=0.840), directly linking the HTML/rubric scores to the requested changes visible in the browser. As discussed in the paper, the judge is more conservative on edit fulfillment while preserving relative variation across outputs. This shows that a more conservative judgment can still track the editing outcomes observed by the human evaluator.

The human evaluator checks the same target, edit, and revision requirements on the rendered source and output pages. This directly tests the correspondence raised by the reviewer: whether the automatic rubric scores reflect the requested editing outcome. Together with the alternative-judge results in Section 6.2, this existing Track A browser calibration supports using the automatic evaluator as a consistent and reproducible basis for comparing editing performance.

Existing manuscript evidence. Section 6.2, Appendix A.7, and Table 16 report the original study with 50 Track A and 50 Track B outputs. The rendering comparisons above use its 50 Track A outputs; Table 16 and the original calibration statistics are unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Author Responses

To the best of our knowledge, Omni2Web is the first benchmark to evaluate whether models can edit existing webpages from screen recordings containing spoken requests, cursor pointing, and weak deictic references (e.g., "this" and "there"). On the same set of requests, we evaluate Direct Editing (Track A), Instruction Recovery (Track B), and Instruction Utility (Track C). Fixing the code executor in Track C disentangles two failure modes: failing to recover user intent versus understanding the intent but failing to implement the edit correctly.

We sincerely thank all reviewers and the Area Chair (AC) for their thoughtful feedback. All three reviewers currently lean toward acceptance with overall ratings of 3 / 3 / 3. During the discussion period, Reviewer MWjh raised their rating from 2 to 3, and Reviewer Hexm explicitly confirmed maintaining their positive rating.

  • Reviewer Hexm (3, maintained): We thank Reviewer Hexm for recognizing our clear motivation, thorough experiments, and the diagnostic value of the three tracks. In response, we added a controlled 1 FPS (644px) visual-sampling comparison confirming that Gemini 3.5 Flash retains its Track A lead, and evaluated speech-timestamped VL baselines separately alongside two newer native Omni models to distinguish native audiovisual capability from external temporal alignment. Our step-order and undo dependency audit shifts IRS by only 0.19–0.37 points, with zero rank changes across all five evaluated models.
  • Reviewer MWjh (2 → 3): We thank Reviewer MWjh for recognizing the practical value of audiovisual webpage editing and the benchmark's discriminative power. We disclosed all 156 EFS configurations (150/156 retain Track A Spearman ρ ≥ 0.9), clarified the rationale for prioritizing edit fulfillment alongside reporting the full G/E/N/R breakdown, and added 900 patch-based editing evaluations, complete 10-step bilingual examples, and dedicated Ethics/Limitations sections. Following Reviewer MWjh's follow-up, we also transparently documented the DOM-level scope of No-damage and the single-evaluator limitation of our browser-based human calibration.
  • Reviewer pRJD (3): We thank Reviewer pRJD for highlighting the importance of intent recovery in modular AI systems and appreciating our broad model coverage and modality/temporal-alignment ablations. We added a systematic comparison with existing benchmarks and re-evaluated Track C using a second fixed code executor (GLM-5.3). The ranking of all five instruction rewriters remains identical (Spearman = 1.00), and under either executor, the strongest recovered instructions still trail oracle instructions by ~39 EFS points.

Thank you again for reviewing our submission.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.