- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ViRE-Bench, a benchmark of 1,000 frontend code-editing tasks built from 202 real repositories across HTML, React, Vue and Angular. Each task gives a coding agent an existing repository and an annotated screenshot. The agent must infer the requested edit, locate the relevant code and implement the change, sometimes across several pages that share a component. The tasks cover eight edit operations. They are constructed from structured edit specifications, from which both the reference implementation and the visual feedback are generated. Evaluation combines validity and operation-specific intent checks with visual measures of target fidelity and preservation of unrelated content. Among six models, the best reaches an intent satisfaction rate of 70.2%.
Critical strengths
The paper targets a practical and underexplored setting, editing real multi-file repositories from annotated screenshots, which complements existing benchmarks on design-to-code generation and text-instructed editing. The benchmark is carefully constructed. The reference implementation and the visual feedback are generated from the same specification, external resources are frozen to reduce rendering variability, and unstable regions are identified through repeated captures. Scoring regions are also derived independently of agent outputs. The evaluation is thoughtfully designed: it separates validity, intent satisfaction, target fidelity and preservation, and its intent checks accept alternative implementations. A human study supports both the tasks and the metrics, with high agreement on intent judgments and a strong correlation with human quality ratings. The experiments are solid, covering six models on all tasks with detailed breakdowns by operation, technology and cross-page consistency.
Critical weaknesses
W1. Tasks and feedback are generated automatically and may differ from real user feedback. Edit targets and parameters are selected by Gemini-3.7-Flash (Sec. 3.2), and the annotations are drawn programmatically from the VES in a uniform style. The Limitations section acknowledges that performance on independently collected human feedback remains untested. Real users often annotate more loosely and ambiguously, and may request several changes in one image.
In addition, for text changes, style changes and component additions, the feedback image includes a short textual note, such as “Give this a deep blue background” (App. E, Table 5). These three operations account for 360 tasks, or 36% of the benchmark, and receive relatively high scores. In contrast, movement, resizing and alignment, which rely on geometric marks only, receive the lowest scores. The differences across operations may therefore partly reflect whether a textual note is present, rather than the difficulty of the operation itself.
W2. Gemini both generates the edit intents and achieves the best results. Gemini-3.7-Flash chooses the targets and parameters for all tasks. It also leads the evaluation by about 10 points over the second model and about 23 points over GPT-5.6-Sol, with especially large gaps on alignment and reordering. With explicit intent, Gemini and GPT both reach about 95% intent satisfaction (Fig. 5), so the gap between them arises mainly when inferring intent from the screenshot. This is exactly the step where a model may benefit from familiarity with edits it proposed itself.
W3. The ambiguity of the visual feedback is not quantified, and there is no human baseline. Intent satisfaction is about 70% under standard input and about 95% with explicit intent. It is unclear how much of this gap reflects model limitations and how much reflects feedback that does not determine a unique edit, especially for numeric parameters such as movement distance or target size. The human study judges request clarity as a binary majority vote, with 95 of 96 tasks approved (App. I). Such a judgment says little about whether a person could recover the exact target, operation and parameters from the image.
W4. The motivation for evaluating agents without execution feedback is unclear. Agents cannot run commands, build the project or preview the rendered page (Sec. 4.1). The paper explains that this standardizes tool access across models. However, practical frontend development relies heavily on checking the rendered result, especially for visual edits such as movement and resizing. Without execution, agents cannot detect build failures or verify whether an edit looks as intended.
W5. The scoring is lenient on incomplete cross-page edits and on damage to unrelated content. For shared-component tasks, intent is checked only on the primary page (Sec. 4.3, App. H), and the primary page receives half the weight in target fidelity. In the example of Fig. 12, the model edits only the primary page and leaves three affected pages unchanged. By my estimate, the overall score would still exceed 80 if preservation stays near its typical value. Yet consistent cross-page editing is a central challenge the benchmark aims to capture.
In addition, conditional preservation lies between 99.4 and 99.8, and target fidelity between 93.7 and 95.9. Both are computed on a different subset for each model, so they add little to the ranking, and the overall score largely tracks intent satisfaction, as the paper notes. Averaging preservation over whole pages may also dilute damage to individual components.
Questions for authors
Q1. On automatically generated feedback. Could the authors report the fraction of tasks with textual notes for each operation? It would help to report validity and intent satisfaction rates per operation, split by whether a note is present. Finally, a small set of real user annotations would show whether the main findings carry over to natural feedback, for example by asking several people to annotate the same edits in their own way.
Q2. On possible bias from intent generation. Could the authors regenerate the intents for a subset of tasks with another model or with human annotators, and check whether model rankings remain stable?
Q3. On ambiguity and a human baseline. I would suggest a human intent-inference study. Annotators would see only the annotated screenshot and write down the target, operation and parameters, and the authors would report agreement with the VES for each operation. This would give an upper bound on how well the intent can be recovered. It would also help interpret the model results, particularly for movement and resizing.
Q4. On the no-execution setting. Could the authors explain the reasons for this choice in more detail, for example cost, safety or reproducibility? It would also be informative to evaluate a subset with build access and a tool that returns a screenshot of the rendered page, and to report whether intent satisfaction and model rankings change.
Q5. On cross-page consistency and preservation. Could the authors apply intent checks on all affected pages for shared-component tasks, or report the fraction of submissions that pass on every page? It would also help to compare target fidelity and preservation on a common subset of tasks that all models pass.