- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Omni2Web, a benchmark that evaluates how LLMs would handle fuzzy/unclear instructions. It devides the task into three subtasks: end-to-end capability, intent recovery and instuction utility. And it also proposed two major evaluation metrics: Edit Fidelity Score(EFS) and Instruction Recovery Score(IRS) to better compare the performance across different models.
Critical strengths
-
This paper identifies an important challenge in modular AI systems: while specialized models are responsible for executing specific tasks, accurately interpreting and communicating user intent is essential for effective collaboration. The proposed benchmark provides a valuable evaluation setting for this problem.
-
The experimental evaluation covers a wide range of mainstream model families, including both open-source and closed-source models.
-
The ablation studies are thorough and well-designed, particularly those investigating the contributions of different input modalities and audiovisual temporal alignment.
Critical weaknesses
-
Insufficient justification of the benchmark's distinctiveness. Although the authors demonstrate that the three evaluation tracks capture different aspects of model performance, it remains unclear whether their combination reveals model capability differences that cannot be adequately evaluated by existing benchmarks. A more systematic comparison with closely related benchmarks would help establish the necessity and novelty of the proposed evaluation framework.
-
Limited discussion of the dataset's ecological validity. According to Section 3.2, the editing requests are based on predefined scripts rather than naturally occurring user interactions. It is unclear to what extent these scripted requests, particularly their frequent weak references and multi-step revisions, reflect realistic webpage editing scenarios. Further discussion or empirical evidence regarding the representativeness of the collected data would strengthen the benchmark's practical relevance.
-
Potential executor dependency in Instruction Utility evaluation. Track C primarily relies on a single fixed coding model, DeepSeek-v4-pro, to evaluate the utility of recovered instructions. Consequently, the results may reflect not only the quality of instruction recovery but also the compatibility between the recovered instructions and the chosen executor. Although the authors provide partial self-execution experiments, evaluating additional executors would help establish whether the reported findings generalize across different coding models.
Questions for authors
None
2 comments
Response to Reviewer pRJD (Part 2/2)
3. Dependence on the Track C executor
Thank you for suggesting an additional executor. The ordering of the five tested rewriters and the large recovered-to-oracle gap persist with a second shared executor. We have completed a GLM-5.3 comparison on the same fixed 300 instances covering 129 source webpages, using the original recovered instructions and one shared oracle condition: 1,800 evaluated entries in total. GLM-5.3 is outside the original 17-model evaluation pool.
Role of a common executor. It holds implementation capability fixed across rewriters, allowing their recovered instructions to be compared through the same coding model. We additionally calibrate utility against each executor's own oracle on the same instances. The new comparison keeps the source HTML, recovered instructions, instance IDs, and EFS protocol fixed; historical DeepSeek outputs are reused and GLM outputs were collected later.
| Instruction source | DeepSeek EFS | DeepSeek NIU | GLM EFS | GLM NIU |
|---|---|---|---|---|
| Qwen3.5-Omni-Plus | 50.32 | 56.32 | 50.61 | 56.26 |
| Gemini 3.5 Flash | 49.32 | 55.20 | 49.64 | 55.18 |
| GPT-5.6 Sol | 46.10 | 51.59 | 46.16 | 51.30 |
| Grok 4.6 | 36.75 | 41.13 | 39.08 | 43.44 |
| Qwen3.8-27B | 28.21 | 31.58 | 29.26 | 32.52 |
| Oracle | 89.35 | 100.00 | 89.96 | 100.00 |
EFS uses a 0–100 scale; NIU is the percentage of each executor's own oracle EFS. GLM columns show the completed budget-extension condition below.
What the results show. The observed EFS point-estimate ordering of the five rewriters is identical across executors (Spearman = 1.00), with Qwen3.5-Omni-Plus and Gemini 3.5 Flash remaining the top two. This ordering also holds for GLM before the budget extension. The best recovered instructions leave oracle gaps of 39.03 EFS points with DeepSeek and 39.35 with GLM. Thus, both the ordering and substantial instruction-utility headroom are reproduced with another common coding model. Absolute scores can still vary: Grok's extended GLM score exceeds its historical DeepSeek score by 2.33 EFS, with a 95% source-page bootstrap interval of [1.00, 3.67]. The paired intervals use 2,000 source-page replicates, holding generated outputs and judge scores fixed.
Generation budget and complete coverage. GLM initially used a 65,536-token output budget, including reasoning tokens. We retried only the 192 length failures at 131,072 tokens, including 34 Oracle cases, and retained the other 1,608 scores unchanged. Each retry keeps its first terminal model outcome. The final 1,800 entries include five historical upstream-input zeros and one Oracle length-budget zero. The reported GLM scores reflect this targeted retry condition.
Complementary self-execution control. Our existing control also compares DeepSeek with each rewriter executing its own instructions, including an oracle for each executor. All values below are EFS on the same 300 instances.
| Rewriter | DeepSeek: recovered / oracle | Self: recovered / oracle |
|---|---|---|
| Gemini 3.5 Flash | 49.32 / 89.35 | 53.48 / 90.45 |
| Qwen3.5-Omni-Plus | 50.32 / 89.35 | 45.20 / 67.43 |
Substantial recovered-to-oracle gaps also remain under self-execution. Together, the two controls support the instruction-utility findings across the tested execution settings. We report utility relative to each executor, with the shared-executor ranking comparison scoped to these five rewriters and 300 instances.
Paper revisions. New Appendix A.5 and Table 14 report the shared-executor EFS and own-oracle NIU comparison and its implications. Appendix A.4 links this comparison to the existing self-execution control in Table 13.
Response to Reviewer pRJD (Part 1/2)
We thank the reviewer for recognizing the importance of interpreting and communicating intent in modular systems, the broad model coverage, and the modality/temporal-alignment ablations.
Corresponding manuscript revisions are highlighted in red.
1. Distinctiveness and the value of three tracks
Omni2Web evaluates how models recover requested edits from synchronized speech and cursor references, and how that intent translates into code. Our original Related Work distinguishes editing supplied HTML from recreating a webpage. We have expanded this comparison in Section 2 with a feature matrix.
We recognize the context already studied by related benchmarks: EDIT-Bench includes highlighted code and studies IDE cursor context; FronTalk provides multi-turn textual or visual feedback; WebVR and IWR-Bench reconstruct webpages from reference videos and assets.
Editing means modifying existing code, including later FronTalk turns. Recovery separately scores model-recovered intent; Utility executes that text with a common coding model. These columns describe benchmark settings and evaluation targets.
Omni2Web's recordings and annotations are newly collected around existing source HTML. Models must resolve which page elements the speaker means, what should change, and which changes remain after revisions. The shared-instance A/B/C protocol then connects this intent recovery to editing outcomes.
What the combined evaluation reveals. Qwen3.5-Omni-Plus obtains 29.56 EFS in direct editing, while its recovered instructions yield 51.08 EFS with the fixed coding model. Both use the same outcome metric on the same instances. Its 49.14 IRS separately measures recovered target/action content. Thus, direct-editing performance alone would miss useful recovered intent, while text recovery alone would leave its practical editing utility untested.
Conversely, Gemini 3.5 Flash grounds the target correctly but fails to fulfill the edit on 16.68% of scored active Track A steps. The three tracks expose these different understanding/execution profiles on shared requests, with execution capability held fixed across rewriters in Track C.
Paper revisions. Section 2 (Related Work) and the new Table 1 now present the systematic comparison and explain the diagnostic role of the three tracks.
2. Ecological validity and the benchmark's intended scope
Omni2Web deliberately focuses on weakly referential editing to distinguish models' intent-recovery capabilities. When a user says "this" while pointing, the system must infer a concrete target and action from the shared audiovisual context. Our benchmark is designed to assess how well models handle these requests.
Why this setting matters. In the original evaluation, the same DeepSeek-v4-pro executor reaches 89.69 EFS with explicit oracle instructions, versus 51.08 with the strongest recovered instructions, on the same 918 instances (Table 3). This shows substantial ability to execute specified edits alongside a large gap when the instructions must first be recovered from recordings. It motivates evaluating the recovery of weakly expressed intent in addition to code execution from explicit specifications.
The scripts provide checkable targets, actions, and undo dependencies, while annotators record the speech and cursor pointing themselves. Deliberate coverage of weak references and revisions supports systematic comparison of the capability we aim to evaluate.
Scope and paper revisions. We have not measured the frequency of weak references or revisions in naturally occurring editing sessions. We have added Appendix A.17 (Limitations) to state that these frequencies are benchmark design choices and that our results characterize performance within this interaction setting. Naturalistic data would be needed to measure that prevalence and assess generalization to everyday requests. Section 3.2 and Appendix A.14 describe the collection procedure; Tables 25–26 retain the complete bilingual examples.