acceptodds
Under review as a conference paper at ICLR 2027

1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World

Abstract

Design-to-code translates high-fidelity UI designs into executable front-end implementations, but progress remains hard to compare due to inconsistent datasets, toolchains, and evaluation protocols. We introduce 1D-Bench, a benchmark grounded in real e-commerce workflows, where each instance provides a reference rendering and an exported intermediate representation that may contain extraction errors. 1D is short for one day, representing the efficient completion of design-to-code tasks in less than one day. Models take both as input, using the intermediate representation as structural cues while being evaluated against the reference rendering, which tests robustness to intermediate representation defects rather than literal adherence. 1D-Bench requires generating an executable React codebase under a fixed toolchain with an explicit component hierarchy, and defines a multi-round setting in which models iteratively apply component-level edits using execution feedback. Experiments on commercial and open-weight multimodal models show that iterative editing generally improves final performance by increasing rendering success and often improving visual similarity. We further conduct a pilot study on post-training with synthetic repair trajectories and reinforcement learning based editing, and observe limited and unstable gains that may stem from sparse terminal rewards and high-variance file-level updates. The data and scripts used in this study are available in an anonymized repository at https://anonymous.4open.science/r/d2c-benchmark-A9C4.

Primary area:
datasets and benchmarks
Keywords:
design-to-code, benchmark, MLLM, coding agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n3VrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces 1D-Bench, which uses reference images and potentially flawed exported intermediate representations (IRs) from real e-commerce design workflows to evaluate models’ ability to generate executable React projects. The benchmark includes both single-round generation and multi-round refinement using build logs, rendered images, and difference heatmaps. The authors also explore supervised fine-tuning on synthetic repair trajectories and reinforcement learning to assess whether post-training improves model performance.

Critical strengths
  1. Using both reference images and potentially flawed exported IRs as inputs is valuable and better reflects real design-to-code workflows.
  2. The data come from an internal e-commerce workflow. The evaluation set is stratified by IR quality, platform, language, and complexity.
  3. The authors establish a fixed React toolchain and execution environment. Reporting rendering success rates separately from visual similarity on successfully rendered samples helps distinguish executability from visual quality.
Critical weaknesses
  1. Although imperfect IRs are a key feature of the benchmark, the paper does not report human annotations of defect types or severity. It also lacks a comparison of image-only, IR-only, and image-plus-IR inputs for the same model.
  2. Multi-round generation allows more model calls, code revisions, and rendering feedback, but the paper does not include a single-round resampling baseline matched for tokens, tool calls, or time. In Table 2, Gemini 3 Pro’s final score decreases slightly from 79.6 to 79.5, while its rendering success rate falls from 100% to 97.7%. This suggests that refinement can also damage previously successful outputs.
  3. Figure 4 shows that the SFT loss decreases, but similarity during GRPO does not improve consistently. The authors attribute this to factors such as sparse rewards without experiments that isolate their effects. Results for SFT-only and SFT-plus-GRPO variants should be reported.
Questions for authors
  1. What are the total token usage, number of tool calls, number of renders, and wall-clock time for multi-round generation?
  2. How well do the automated metrics agree with human expert preferences on webpages generated by the evaluated models?
  3. What are the test-set scores for SFT-only and SFT-plus-GRPO models?
1 comment

Response to Reviewer n3Vr

We thank Reviewer n3Vr for the questions on IR defects, cost accounting and the training objective. The revision separates the failure audit from the IR-defect taxonomy and reports the missing numbers.

W1. No human annotation of IR defect types or severity; no image-only / IR-only / image+IR comparison.

The input comparison is in R1-W2. We have added a defect taxonomy for IRs (missing attributes, incorrect grouping, noisy style values, missing nodes) and are applying it to a sampled subset of imperfect-IR instances, with counts by type, severity ratings and inter-annotator agreement. The sampling frame and coding rules are in the revision; the counts follow.

Section 4.4 and new Table 7 keep the failure audit separate from IR defects. The five models produce 380 initial-round rendering failures on the 264 instances. The audit covers a 50% sample of them, 190 cases: 182 (95.8%) are missing or invalid files and imports, and 8 (4.2%) are TypeScript, JSX or CSS build errors. In the 182 cases a failed or abnormal tool call passes an empty path or empty content to WriteTool. These are tool-use failures at the file-writing interface. (Section 4.4, new Table 7)

W2. No budget-matched single-round resampling baseline; refinement can damage outputs.

Both points are covered in R2-W2 and R2-Q3. The matched-budget single-round condition is running with model calls, tokens and wall-clock time held equal and execution feedback removed. Section 3.2 reports the Gemini 3 Pro regression explicitly (FinalScore 79.6 to 79.5, RSR 100% to 97.7% on 264 instances, paired t = -0.267, n.s.), with per-round trajectories and best-of-round scores alongside it.

W3. Figure 4 attributes GRPO instability without isolating factors; SFT-only and SFT+GRPO results missing.

The training objective learns initial generation and multi-round repair jointly, and both trajectory types are mixed throughout SFT. Figure 4 has been redrawn and its caption rewritten. Joint multi-task instruction tuning mitigates the forgetting that a sequential recipe can cause (He et al., 2023). Held-out scores for base, SFT-only and SFT+GRPO will be reported in Section 4.4 as part of R1-W4, and the reward-density and edit-granularity experiments in R2-W3 are running where the compute budget allows. (Section 4.4, Figure 4)

Q1. Token usage, tool calls, renders, wall-clock time for multi-round.

One complete trajectory takes about 8 minutes per instance at about USD 5.2 under Claude Sonnet 4.5, with roughly 0.95M input and 0.16M output tokens, about 9 components per project, about 12 component-level WriteTool calls per repair round, and four model rounds (one initial plus at most three repairs, K = 3 excluding initial), each followed by a build and render pass. Most of the latency comes from dependency installation, compilation, browser rendering and visual comparison rather than inference. New Table 5 tabulates these quantities per model and setting. (New Table 5)

Q2. Agreement between automated metrics and human expert preference on model outputs.

See R1-W3. Section 2.3 and the Limitations state the scope of the 50-pair, 20-annotator study, and the model-output study is running with the design given there.

Q3. Test-set scores for SFT-only and SFT+GRPO.

See R1-W4.

Reviewer_4vzgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper provides 1D-bench, a real-world UI code generation benchmark for e-commerce scenarios, which evaluates executable React generation using both a reference rendering and an imperfect exported IR as input. The authors define a multi-round protocol for iterative UIs code generation, construct an evaluation set including 204 instances and conduct single-round and multi-round evaluations on 5 models. It also explores SFT and GRPO based on synthetic repair trajectories. The results show that multi-round repairs generally improve final performance.

Critical strengths
  • The paper provides a real-world UI code generation benchmark from e-commerce development workflows.
  • The paper defines a multi-round protocol for iterative UIs code generation.
Critical weaknesses
  • Limited contributions. I think the existing design-to-code datasets can also support multi-round evaluation by adding rendering, feedback, and modification loops, so the proposed multi-round protocol is not sufficient to demonstrate the necessity of the new baseline.

  • Unclear experiments setup. Comparing single-round and multi-round increases the computational budget and feedback information, but lacks control experiments with the same budget and different types of feedback.

  • Insufficient validation. The paper mainly discusses that the GRPO returns are limited and unstable through training curves, but does not report a complete comparison of baselines and the models after post-training.

Questions for authors
  • Actually, I can't understand the challenges of e-commerce UIs generation. What are the differences between general UI code generation and UI code generation for e-commerce?
  • Why do existing benchmarks not support multi-round evaluation, while 1D-Bench and MRWeb can? If the main difference is the additional provision of IR, could existing benchmarks also extract or construct similar structural information and incorporate the same execution feedback loop?
  • Table 2 shows that the RSR of some LLMs is lower under multi-round settings compared to single-round settings. What could be the possible reasons? Is the metric reasonable?
  • What is the performance of the model after post-training?
2 comments

Response to Reviewer 4vzg

We thank Reviewer 4vzg for the questions on what the benchmark contributes and whether the setup can be reproduced. We have narrowed the claim and added the missing definitions and measurements.

W1. Limited contributions; multi-round could be added to existing benchmarks.

The contribution is the combination of industrial e-commerce inputs with exported, imperfect IRs, a persistent workspace in which later edits depend on component boundaries from earlier rounds, and a fixed React toolchain with rendering success as a first-class metric.

Section 1 and Section 2.3 now say that a multi-round loop can be added to existing image-to-HTML datasets, and position 1D-Bench as complementary. We have extended Table 1 with three columns marking, for each benchmark, support for persistent multi-round editing, noisy structural inputs, and executable output under a fixed toolchain. MRWeb is an image-to-HTML benchmark with multi-page context and no exported structure in its inputs; Section 1 states in one sentence that it is not an executable method under our React toolchain. (Section 1, Section 2.3, Table 1)

W2. Unclear setup; no budget-matched or feedback-type controls.

A trajectory has four model calls: one initial generation and at most three repair rounds. K = 3 counts the repair segments and excludes the initial generation. Section 2.2 and Section 4.3 state this in the same words.

New Table 5 reports wall-clock time, tokens, tool calls and round structure per model and setting. One complete trajectory takes about 8 minutes per instance at about USD 5.2 under Claude Sonnet 4.5, with roughly 0.95M input and 0.16M output tokens at USD 3 and USD 15 per million. A project has about 9 components, and each repair round makes about 12 component-level WriteTool calls. Dependency installation, compilation, headless rendering and image comparison account for much of the 8 minutes. (Section 2.2, Section 4.3, new Table 5)

Both controls are running at the same round budget: a single-round condition matched on model calls, tokens and wall-clock time with execution feedback removed, and feedback-type controls covering build logs only, rendered image only, diff heatmap only, and the full feedback the harness returns.

W3. Insufficient validation of post-training.

See R1-W4. Section 4.4 gives the mechanism the curves support: terminal rewards are sparse, failure is discontinuous, and WriteTool overwrites whole files, so each action is a high-variance macro edit and credit for a terminal reward is assigned across several rounds. Two isolation experiments are running: reward density (terminal-only against intermediate scores used as partial reward) and edit granularity (whole-file overwrite against localized patch actions). (Section 4.4)

Q1. What is specific to e-commerce UI generation?

Consumer business pages contain animations, gradients, interaction states and encapsulated business components, with layout and style values that pixels do not carry. The 264 tasks were stratified from 984 retained instances, themselves filtered from 5,856 candidates, balancing IR reliability, platform, language and complexity. New Table 3 reports component counts, text density, language mix and the IR node-count distribution, with the stratum boundaries used for sampling. (New Table 3)

Q2. Could existing benchmarks construct IRs and support multi-round?

They can in principle, and Section 2.3 and the Limitations say so. Our IR is not inferred from the reference screenshot: both inputs come from the same uploaded design file, and the IR is what the production export emits. It carries encapsulated business components, exact style values, assets and hierarchy. Re-deriving an equivalent IR from screenshots changes the noise profile the benchmark studies. (Section 2.3, Limitations)

Q3. Why does RSR sometimes drop in multi-round, e.g., Gemini 3 Pro 100% to 97.7%?

Refinement can damage an output that already worked. Over the 264 instances, FinalScore improves significantly for every model except Gemini 3 Pro (paired t = -0.267, n.s., n = 264), while paired similarity on the 258 instances that render in both rounds does improve for Gemini 3 Pro (t = 2.635, p < 0.01). Section 3.2 states both results: the pages that still render are closer to the reference, and a few that used to render no longer do, which FinalScore charges for.

We have added per-round RSR trajectories, an analysis of the edits that cause regressions, and a best-of-round secondary metric. The tests evaluate within-model improvement. Claude Sonnet 4.5, Gemini 3 Pro and GPT-5.2 are reported as a top group by point estimates. (Section 3.2)

Q4. Performance after post-training?

See R1-W4.

Reviewer_1J1jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. Introduces a design-to-code benchmark using industrial screenshots and potentially defective design representations.
  2. Models generate executable React projects and revise them using visual and execution feedback.
  3. Evaluates five models and includes an exploratory supervised and reinforcement-learning study.
Critical strengths
  1. Realistic inputs. Industrial designs and imperfect exports reflect practical development challenges.
  2. Useful execution-based evaluation. Separating rendering success from visual similarity distinguishes different failure types.
Critical weaknesses
  1. Results are internally inconsistent. Table 2 success rates do not match integer counts over 204 instances. Figure 3B also conflicts with GLM’s reported FinalScores.
  2. Input robustness is untested. Screenshot-only, deterministic-rendering, and input-quality comparisons are missing.
  3. Metric validation is limited. Human evaluation uses synthetic perturbations rather than model outputs, with no numerical agreement statistics.
  4. Post-training outcomes are missing. Training curves do not establish held-out performance before training, after supervised training, and after reinforcement learning.
Questions for authors
  1. Can you reconcile the evaluated counts, Table 2 success rates, and GLM’s Figure 3B curve?
  2. How do screenshot-only generation and deterministic rendering compare across perfect and imperfect inputs?
  3. What is metric–human agreement on actual model outputs, including uncertainty?
  4. What are the held-out results for the base, supervised, and reinforcement-learning checkpoints?
4 comments

Response to Reviewer 1J1j

We thank Reviewer 1J1j for the detailed reading, in particular the check on Table 2 and Figure 3B. W1 to W4 are addressed below.

W1. Table 2 / Figure 3B are internally inconsistent.

Table 2 and Figure 3B came from different runs, and the set size was printed incorrectly. The evaluation set has 264 instances; Table 1, Section 2.1, Appendix A and the Data Availability statement now say so. The counts are:

ModelInitial roundFinal roundFinalScore
GLM-4.6V18/264 = 6.8%18/264 = 6.8%3.7 to 5.2
Qwen3-VL-235B-A22B-Instruct250/264 = 94.7%257/264 = 97.3%59.0 to 61.9
Claude Sonnet 4.5240/264 = 90.9%258/264 = 97.7%74.0 to 80.4
GPT-5.2168/264 = 63.6%246/264 = 93.2%49.8 to 79.1
Gemini 3 Pro264/264 = 100.0%258/264 = 97.7%79.6 to 79.5

We have added k / 264 beside each percentage. All ten RSR cells now reproduce from an integer count over 264; the Qwen3-VL-235B multi-round cell read 97.4%, which has no integer numerator over 264, and now reads 97.3% (257/264). FinalScore uses unrounded mean similarity and RSR, so it can differ by up to 0.1 from the product of the printed values. Figure 3B has been regenerated from the same result file as Table 2.

Section 3.2 states three denominators. FinalScore and RSR use 264. Mean similarity uses the instances that render in the round scored, from 18 for GLM-4.6V to 264 for Gemini 3 Pro initially. The paired FinalScore test covers all 264 instances, with failures scoring zero; the paired similarity test uses the two-round intersection, 258 for Gemini 3 Pro because all 264 render initially. (Section 2.1, Section 3.2, Table 2, Figure 3B)

W2. Input robustness is untested.

The IR and the screenshot come from the same design file. The IR is the output of the production export pipeline and carries hierarchy, assets, encapsulated business components and exact style values. Section 2.1 cites the Figma2Code controlled comparison: visual fidelity of 0.5653 with image-only input and 0.6923 with metadata.

We have added Table 4, which splits the existing FinalScore results by the pre-computed IR-quality label.

ModelPerfect-IR FinalScoreImperfect-IR FinalScore
GLM-4.6V7.44.5
Qwen3-VL-235B-A22B-Instruct59.462.7
Claude Sonnet 4.581.180.2
GPT-5.272.381.2
Gemini 3 Pro79.979.4

The two subsets contain different tasks, and the difference changes sign across models. We report the split descriptively, with per-subset counts. The label is computed before model evaluation, and Section 3.3 records that the strata and the score share a metric family. A controlled test needs matched perfect and imperfect IR variants of the same instance; our production records contain no such pairs.

The image-only and IR-only conditions are running under the identical harness on the 264 instances, split by IR quality, and will be reported with per-subset counts and confidence intervals. (Section 2.1, new Section 3.3, new Table 4)

W3. Metric validation is limited.

Appendix C lists all components and weights: 0.5 image, 0.3 completeness, 0.2 layout, and within the image term 0.8 LPIPS, 0.1 SSIM, 0.1 pixel difference. The weights were fixed before the five-model evaluation.

Section 2.3 now states the scope of the 50-pair, 20-annotator study, which calibrates the metric under controlled DOM and style perturbations, and gives one example for each failure mode that motivated the composite. CLIP and LPIPS stay high when text blocks or buttons are missing or misplaced, and move sharply when a large decorative image changes although layout is intact. Pixel scores over-penalize font rasterization and antialiasing. The OCR completeness, element matching, layout, perceptual and pixel terms address these cases.

The human study on model outputs is running: outputs sampled across the score range, expert judgments, absolute agreement, inter-annotator agreement, and a composite-versus-LPIPS-only comparison with bootstrap confidence intervals. The design is in Section 2.3. (Section 2.3, Appendix C, Limitations)

W4. Post-training held-out outcomes are missing.

The abstract, Section 4.4 and the Limitations now describe the post-training pilot as exploratory and state that the training curves do not establish a reliable gain. Held-out evaluation on the 264 instances is running with the three checkpoints behind Figure 4, reporting mean similarity, RSR and integer counts for base Qwen3-VL-32B, SFT-only and SFT+GRPO as Table 6. (Section 4.4, Limitations, new Table 6)

Q1 to Q4

Q1 to Q4 are covered by W1 to W4. Section 3.2 now explains why multi-round can reduce RSR for models with high initial RSR: at 100% there are no failures left to repair, so a regression has no offsetting gain. It also presents the Gemini 3 Pro paired similarity result for that case.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.