- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this study, the authors propose a method for constructing a synthetic training dataset, an adaptive GRPO, and an evaluation using VLM-as-a-judge for the task of generating web design code from sketch inputs. The experimental results demonstrate that the proposed data synthesis pipeline and adaptive GRPO training improve performance on the sketch-to-code task compared to baseline methods.
Critical strengths
- Whereas previous work focused primarily on benchmark development, they present a pipeline for generating synthetic data that can be used for training.
- They have confirmed that the proposed training method achieves better performance than the baseline.
Critical weaknesses
- Since the evaluation criteria and reward criteria (as described in Table 3) are essential for understanding the paper, I think they should be included in the main text rather than the appendix.
- The reward used during training and the criteria used during evaluation are exactly the same, and evaluation of the test data is limited to the VLM-as-a-judge approach. I believe the reliability of the results would be further enhanced by conducting manual evaluations even on a small sample set to assess the reliability of the VLM-as-a-judge approach.
- Regarding the VLM-as-a-judge model used for evaluation, I think evaluating it not only with the open model but also with a new version of the higher-performing closed model would improve the robustness of the evaluation.
- There are some unclear points regarding the specifications of the synthetic data. While the paper states that textual requirements are used as input data in addition to sketch images, it is unclear how detailed these textual requirements are. For example, if a reference contains a long text passage, to what extent is that text provided as input? If the text itself is not provided as input, it would be nearly impossible to reproduce that element exactly as it appears in the reference. How are such points taken into account during evaluation? Also, how is non-text content—such as photographs and other images—handled? Is it provided as additional input, generated as a placeholder, or what is the assumption here?
- I think it would be helpful to include examples of textual requirements in the Appendix so we can better understand the details.
Questions for authors
- Regarding the quality of the synthesized data, Section 5.1 states that 348 out of 400 samples passed the quality check. What kinds of errors were found in the filtered data? What differences exist between the synthesized sketches and the hand-drawn sketches? Also, were the textual requirements checked?
- Wouldn’t it be more accurate to directly compare the generated text with the reference HTML text—using character-by-character matching or similar methods—rather than using VLM-as-a-judge to assess text similarity?
- It appears that training was conducted using only GRPO without an SFT step, but I think it is common practice to perform SFT before GRPO. Was the baseline performance sufficient enough this time that SFT was not necessary? Did you conduct a comparison between cases with and without SFT?
- How is ρ in Equation (8) determined?
- Will the synthetic data and the manually created data be made public?
7 comments
Comment 4: Textual Requirements and Non-Text Content
There are some unclear points regarding the specifications of the synthetic data. While the paper states that textual requirements are used as input data in addition to sketch images, it is unclear how detailed these textual requirements are. For example, if a reference contains a long text passage, to what extent is that text provided as input? If the text itself is not provided as input, it would be nearly impossible to reproduce that element exactly as it appears in the reference. How are such points taken into account during evaluation? Also, how is non-text content—such as photographs and other images—handled? Is it provided as additional input, generated as a placeholder, or what is the assumption here? I think it would be helpful to include examples of textual requirements in the Appendix so we can better understand the details.
Response
Textual requirements: We generate detailed requirements from both the reference webpage rendering and its HTML source code. These requirements cover layout, colors, typography, and the specific wording of headings, body text, navigation items, and buttons. For longer passages, the requirement-generation prompt likewise requests the actual text rather than a summary of its topic. The model therefore obtains textual content from the requirements and spatial structure from the sketch.
Image assets: We provide the paths to the corresponding image assets as part of the input and instruct the model to reference these paths directly in the generated HTML, allowing the original assets to be loaded during rendering. For the Sketch2Code benchmark, we follow its setting and replace the original images with placeholder images.
Example requirement (abridged): “Use a light-blue background (#e6f5fb) and dark-blue headings (#114a8b). Place the heading ‘The Refreshing Alternative’ and subtitle ‘Full flavour without the sugar.’ on the left, with a product illustration on the right. Include a ‘BUY NOW’ button. Use the supplied image
3316a98ae0.jpgas a splash-texture layer behind the product.”
Comment 5: Synthetic Data Quality and Quality-Check Results
Regarding the quality of the synthesized data, Section 5.1 states that 348 out of 400 samples passed the quality check. What kinds of errors were found in the filtered data? What differences exist between the synthesized sketches and the hand-drawn sketches? Also, were the textual requirements checked?
Response
We clarify the filtering criteria, sketch differences, and textual requirement verification below.
Errors identified during filtering: We exclude samples with the following issues:
- Broken images: Images fail to load, leaving broken-image icons, alt text, or empty boxes and placeholders where image content should appear.
- Broken layouts: Severe element overlap, unreadable overlapping text, substantial off-screen content, or obvious horizontal overflow and layout collapse.
- Nearly blank pages: Pages consist predominantly of a uniform background with very little visible content, suggesting incomplete rendering.
- Non-webpage content: Pages dominated by a single image, poster, or video cover, without a meaningful interface layout to represent as a sketch.
Synthesized versus hand-drawn sketches: Synthesized sketches are generally more regular in geometry and alignment. Human-drawn sketches exhibit greater variation in strokes, positions, and proportions, and photographed sketches may additionally contain perspective distortion, shadows, and other capture noise.
Verification of textual requirements: After generating the requirements, we perform an additional model-based verification step to check whether the required textual content from the reference webpage is included. If omissions are detected, the model revises the requirements, and this process repeats until the verification identifies no missing required text.
Comment 6: Direct Text Matching
Wouldn’t it be more accurate to directly compare the generated text with the reference HTML text—using character-by-character matching or similar methods—rather than using VLM-as-a-judge to assess text similarity?
Response
Direct text matching can provide a more objective measure of textual fidelity. Our rendering-based evaluation additionally considers how text is actually presented: correct text in the HTML may still be obscured, clipped, or positioned outside the visible area because of layout errors. Such cases are not fully captured by matching the underlying text alone. We also assess similarities in font family, size, style, and weight, extending the evaluation beyond textual content accuracy to its visual presentation.
To address this concern, we additionally evaluate our method and the newly added baselines using the static metrics introduced by Sketch2Code. These include a text-matching metric and other rule-based metrics for block matching, position, color, and layout, alongside CLIP similarity. We apply these metrics to both benchmarks; the results are shown below.
Scores range from 0 to 1, with higher values indicating better performance. Visual is the arithmetic mean of Block, Text, Position, Color, and CLIP. Total is an additional summary defined here as (Layout + Visual) / 2.
Benchmark Model Layout Visual Block Text Position Color CLIP Total SketchFactory-Bench Qwen3-VL-8B + DARR (ours) 0.1605 0.6565 0.5560 0.7519 0.6583 0.6615 0.6548 0.4085 SketchFactory-Bench Qwen3.6-35B-A3B 0.0540 0.3716 0.1549 0.4766 0.4042 0.4349 0.3873 0.2128 SketchFactory-Bench Qwen3.5-9B 0.0498 0.3516 0.1545 0.4527 0.3706 0.3898 0.3904 0.2007 SketchFactory-Bench GLM-4.6V-Flash 0.0089 0.0478 0.0335 0.0571 0.0496 0.0517 0.0469 0.0283 SketchFactory-Bench GLM-4.6V 0.0353 0.1537 0.1072 0.1875 0.1587 0.1613 0.1535 0.0945 Sketch2Code Qwen3-VL-8B + DARR (ours) 0.2469 0.8508 0.7618 0.9490 0.8207 0.8558 0.8668 0.5489 Sketch2Code Qwen3.6-35B-A3B 0.1673 0.6471 0.4720 0.7564 0.6530 0.6754 0.6788 0.4072 Sketch2Code Qwen3.5-9B 0.1497 0.6445 0.4623 0.7635 0.6333 0.6666 0.6967 0.3971 Sketch2Code GLM-4.6V-Flash 0.1060 0.4183 0.3674 0.4797 0.3946 0.4195 0.4302 0.2621 Sketch2Code GLM-4.6V 0.1556 0.5682 0.4992 0.6458 0.5488 0.5687 0.5787 0.3619 Our method achieves the highest text-matching score and the highest scores on the other reported static metrics among these models on both benchmarks, providing complementary evidence beyond VLM-based evaluation.
Comment 7: SFT Before GRPO
It appears that training was conducted using only GRPO without an SFT step, but I think it is common practice to perform SFT before GRPO. Was the baseline performance sufficient enough this time that SFT was not necessary? Did you conduct a comparison between cases with and without SFT?
Response
The base models already possess basic sketch-to-code generation capabilities, providing a starting point for direct RL. Moreover, multiple HTML implementations can produce the same rendered appearance. SFT on a single reference implementation may bias the model toward a particular coding style, whereas our goal is to improve the rendered result. We therefore directly optimize the models through RL using rewards based on the generated webpage renderings.
To examine the effect of SFT, we additionally evaluated SFT and SFT+RL on Qwen2.5-VL-7B. The comparison with the base model and direct RL is shown below.
SketchFactory-Bench — GLM-4.6V judge
Training setting Visual structure Color & aesthetic Textual content User interface Overall Base 3.37 3.38 4.03 3.07 3.46 SFT 4.68 4.67 4.98 4.43 4.69 SFT + RL 4.81 4.97 5.56 4.45 4.94 Direct RL 5.08 5.11 5.75 4.83 5.19 Sketch2Code — GLM-4.6V judge
Training setting Visual structure Color & aesthetic Textual content User interface Overall Base 4.68 4.81 5.96 4.34 4.95 SFT 5.84 5.86 6.67 5.61 5.99 SFT + RL 6.10 5.96 6.88 5.87 6.20 Direct RL 6.11 6.16 7.17 5.83 6.32 Direct RL slightly outperforms SFT+RL.
We thank you for the careful reading of our paper and for the constructive comments. Regarding your concerns, I have the following clarifications.
Comment 1: Include Evaluation and Reward Criteria in the Main Text
Comment 2: Reward-Evaluation Overlap and Human Evaluation
Comment 3: A Stronger Judge Model