- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces 1D-Bench, which uses reference images and potentially flawed exported intermediate representations (IRs) from real e-commerce design workflows to evaluate models’ ability to generate executable React projects. The benchmark includes both single-round generation and multi-round refinement using build logs, rendered images, and difference heatmaps. The authors also explore supervised fine-tuning on synthetic repair trajectories and reinforcement learning to assess whether post-training improves model performance.
Critical strengths
- Using both reference images and potentially flawed exported IRs as inputs is valuable and better reflects real design-to-code workflows.
- The data come from an internal e-commerce workflow. The evaluation set is stratified by IR quality, platform, language, and complexity.
- The authors establish a fixed React toolchain and execution environment. Reporting rendering success rates separately from visual similarity on successfully rendered samples helps distinguish executability from visual quality.
Critical weaknesses
- Although imperfect IRs are a key feature of the benchmark, the paper does not report human annotations of defect types or severity. It also lacks a comparison of image-only, IR-only, and image-plus-IR inputs for the same model.
- Multi-round generation allows more model calls, code revisions, and rendering feedback, but the paper does not include a single-round resampling baseline matched for tokens, tool calls, or time. In Table 2, Gemini 3 Pro’s final score decreases slightly from 79.6 to 79.5, while its rendering success rate falls from 100% to 97.7%. This suggests that refinement can also damage previously successful outputs.
- Figure 4 shows that the SFT loss decreases, but similarity during GRPO does not improve consistently. The authors attribute this to factors such as sparse rewards without experiments that isolate their effects. Results for SFT-only and SFT-plus-GRPO variants should be reported.
Questions for authors
- What are the total token usage, number of tool calls, number of renders, and wall-clock time for multi-round generation?
- How well do the automated metrics agree with human expert preferences on webpages generated by the evaluated models?
- What are the test-set scores for SFT-only and SFT-plus-GRPO models?
Response to Reviewer n3Vr
We thank Reviewer n3Vr for the questions on IR defects, cost accounting and the training objective. The revision separates the failure audit from the IR-defect taxonomy and reports the missing numbers.
W1. No human annotation of IR defect types or severity; no image-only / IR-only / image+IR comparison.
The input comparison is in R1-W2. We have added a defect taxonomy for IRs (missing attributes, incorrect grouping, noisy style values, missing nodes) and are applying it to a sampled subset of imperfect-IR instances, with counts by type, severity ratings and inter-annotator agreement. The sampling frame and coding rules are in the revision; the counts follow.
Section 4.4 and new Table 7 keep the failure audit separate from IR defects. The five models produce 380 initial-round rendering failures on the 264 instances. The audit covers a 50% sample of them, 190 cases: 182 (95.8%) are missing or invalid files and imports, and 8 (4.2%) are TypeScript, JSX or CSS build errors. In the 182 cases a failed or abnormal tool call passes an empty path or empty content to WriteTool. These are tool-use failures at the file-writing interface. (Section 4.4, new Table 7)
W2. No budget-matched single-round resampling baseline; refinement can damage outputs.
Both points are covered in R2-W2 and R2-Q3. The matched-budget single-round condition is running with model calls, tokens and wall-clock time held equal and execution feedback removed. Section 3.2 reports the Gemini 3 Pro regression explicitly (FinalScore 79.6 to 79.5, RSR 100% to 97.7% on 264 instances, paired t = -0.267, n.s.), with per-round trajectories and best-of-round scores alongside it.
W3. Figure 4 attributes GRPO instability without isolating factors; SFT-only and SFT+GRPO results missing.
The training objective learns initial generation and multi-round repair jointly, and both trajectory types are mixed throughout SFT. Figure 4 has been redrawn and its caption rewritten. Joint multi-task instruction tuning mitigates the forgetting that a sequential recipe can cause (He et al., 2023). Held-out scores for base, SFT-only and SFT+GRPO will be reported in Section 4.4 as part of R1-W4, and the reward-density and edit-granularity experiments in R2-W3 are running where the compute budget allows. (Section 4.4, Figure 4)
Q1. Token usage, tool calls, renders, wall-clock time for multi-round.
One complete trajectory takes about 8 minutes per instance at about USD 5.2 under Claude Sonnet 4.5, with roughly 0.95M input and 0.16M output tokens, about 9 components per project, about 12 component-level WriteTool calls per repair round, and four model rounds (one initial plus at most three repairs, K = 3 excluding initial), each followed by a build and render pass. Most of the latency comes from dependency installation, compilation, browser rendering and visual comparison rather than inference. New Table 5 tabulates these quantities per model and setting. (New Table 5)
Q2. Agreement between automated metrics and human expert preference on model outputs.
See R1-W3. Section 2.3 and the Limitations state the scope of the 50-pair, 20-annotator study, and the model-output study is running with the design given there.
Q3. Test-set scores for SFT-only and SFT+GRPO.
See R1-W4.