acceptodds
Under review as a conference paper at ICLR 2027

SketchFactory: A Unified Framework for Learning and Evaluating Sketch-to-Code Generation

Abstract

Generating runnable front-end code from hand-drawn sketches and textual requirements can accelerate interface development. Existing work on sketch-to-code has primarily focused on benchmark development. However, a unified framework integrating training data synthesis, task-specific evaluation, and reinforcement learning algorithm remains lacking. To address this gap, we introduce SketchFactory, a pipeline that synthesizes high-quality sketch-to-code data by deriving sketches and textual requirements from rendered webpages. We further propose an evaluation strategy that uses a vision-language model to judge rendered pages along interpretable quality dimensions, accommodating geometric imprecision in sketches. Building on these multidimensional scores, we develop difficulty-adaptive reward reweighting DARR, a reinforcement learning algorithm that shifts reward weights toward the policy's weaker dimensions based on recent dimension-wise performance. Experiments on two benchmarks under two independent judges show that trained on SketchFactory data, DARR-Qwen3-8B raises the overall score of its base model by – points on a – scale across the four benchmark–judge settings.

Primary area:
generative models
Keywords:
MLLM, sketch to code generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ANrhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this study, the authors propose a method for constructing a synthetic training dataset, an adaptive GRPO, and an evaluation using VLM-as-a-judge for the task of generating web design code from sketch inputs. The experimental results demonstrate that the proposed data synthesis pipeline and adaptive GRPO training improve performance on the sketch-to-code task compared to baseline methods.

Critical strengths
  • Whereas previous work focused primarily on benchmark development, they present a pipeline for generating synthetic data that can be used for training.
  • They have confirmed that the proposed training method achieves better performance than the baseline.
Critical weaknesses
  • Since the evaluation criteria and reward criteria (as described in Table 3) are essential for understanding the paper, I think they should be included in the main text rather than the appendix.
  • The reward used during training and the criteria used during evaluation are exactly the same, and evaluation of the test data is limited to the VLM-as-a-judge approach. I believe the reliability of the results would be further enhanced by conducting manual evaluations even on a small sample set to assess the reliability of the VLM-as-a-judge approach.
  • Regarding the VLM-as-a-judge model used for evaluation, I think evaluating it not only with the open model but also with a new version of the higher-performing closed model would improve the robustness of the evaluation.
  • There are some unclear points regarding the specifications of the synthetic data. While the paper states that textual requirements are used as input data in addition to sketch images, it is unclear how detailed these textual requirements are. For example, if a reference contains a long text passage, to what extent is that text provided as input? If the text itself is not provided as input, it would be nearly impossible to reproduce that element exactly as it appears in the reference. How are such points taken into account during evaluation? Also, how is non-text content—such as photographs and other images—handled? Is it provided as additional input, generated as a placeholder, or what is the assumption here?
  • I think it would be helpful to include examples of textual requirements in the Appendix so we can better understand the details.
Questions for authors
  • Regarding the quality of the synthesized data, Section 5.1 states that 348 out of 400 samples passed the quality check. What kinds of errors were found in the filtered data? What differences exist between the synthesized sketches and the hand-drawn sketches? Also, were the textual requirements checked?
  • Wouldn’t it be more accurate to directly compare the generated text with the reference HTML text—using character-by-character matching or similar methods—rather than using VLM-as-a-judge to assess text similarity?
  • It appears that training was conducted using only GRPO without an SFT step, but I think it is common practice to perform SFT before GRPO. Was the baseline performance sufficient enough this time that SFT was not necessary? Did you conduct a comparison between cases with and without SFT?
  • How is ρ in Equation (8) determined?
  • Will the synthetic data and the manually created data be made public?
7 comments

We thank you for the careful reading of our paper and for the constructive comments. Regarding your concerns, I have the following clarifications.

Comment 1: Include Evaluation and Reward Criteria in the Main Text

Since the evaluation criteria and reward criteria (as described in Table 3) are essential for understanding the paper, I think they should be included in the main text rather than the appendix.

Response

Thank you for the suggestion. We will move the evaluation and reward criteria in Table 3 from the appendix to the main text.

Comment 2: Reward-Evaluation Overlap and Human Evaluation

The reward used during training and the criteria used during evaluation are exactly the same, and evaluation of the test data is limited to the VLM-as-a-judge approach. I believe the reliability of the results would be further enhanced by conducting manual evaluations even on a small sample set to assess the reliability of the VLM-as-a-judge approach.

Response

For evaluation beyond VLM-as-a-judge, we adopt the evaluation protocol of the Sketch2Code benchmark, which uses a set of rule-based metrics. The results are reported in our response to Comment 6. We randomly sampled 300 cases comparing outputs from Qwen3-VL-32B and our Qwen3-VL-8B + DARR model, with 150 cases from SketchFactory-Bench and 150 from Sketch2Code. For each case, we used the overall scores assigned by GLM-4.6V and human judge to determine which output was better, and measured agreement between their preferences.

BenchmarkCasesMatching preferencesAgreement
SketchFactory-Bench15011878.67%
Sketch2Code15012482.67%
Combined30024280.67%

This analysis assesses consistency across model judges and human evaluation.

Comment 3: A Stronger Judge Model

Regarding the VLM-as-a-judge model used for evaluation, I think evaluating it not only with the open model but also with a new version of the higher-performing closed model would improve the robustness of the evaluation.

Response

Our limited evaluation budget does not allow us to run large-scale evaluation with additional closed-source models. We therefore supplement the existing evaluations with another capable judge, Tencent HY 4, to examine robustness across judges. The results for the original-paper models with completed HY4 evaluations on both benchmarks are reported below.

Scores range from 0 to 10. F denotes SketchFactory-Bench and S denotes Sketch2Code.

ModelF StructureF ColorF TextF UIF OverallS StructureS ColorS TextS UIS Overall
LLaVA-1.5-13B0.370.780.790.240.550.391.001.080.240.68
LLaVA-NeXT Vicuna-13B0.000.020.000.000.000.030.230.060.010.08
LLaVA-NeXT-34B0.020.330.020.020.100.431.050.750.370.65
Gemma-3-12B3.223.815.142.613.693.554.836.033.684.52
Gemma-3-27B5.615.926.165.215.735.306.356.725.395.94
InternVL3.5-14B5.326.246.075.205.715.066.216.575.235.77
InternVL3.5-38B5.856.556.485.606.125.256.406.735.445.95
Qwen2.5-VL-32B5.926.596.545.686.185.566.706.795.736.20
Qwen3-VL-32B4.795.575.034.715.025.776.946.676.026.35
Qwen2.5-VL-7B + fixed-weight GRPO5.296.636.295.195.855.026.796.655.435.97
Qwen2.5-VL-7B + DARR5.396.776.295.255.925.316.986.785.666.18
Qwen3-VL-8B + fixed-weight GRPO5.896.956.285.826.235.747.246.916.126.50
Qwen3-VL-8B + DARR6.207.296.536.116.535.837.356.976.226.59

HY4 also assigns higher overall scores to DARR than to fixed-weight GRPO for both backbones on both benchmarks. Qwen3-VL-8B + DARR achieves the highest overall score among the models shown on each benchmark, supporting the robustness of these conclusions across judges.

Comment 4: Textual Requirements and Non-Text Content

There are some unclear points regarding the specifications of the synthetic data. While the paper states that textual requirements are used as input data in addition to sketch images, it is unclear how detailed these textual requirements are. For example, if a reference contains a long text passage, to what extent is that text provided as input? If the text itself is not provided as input, it would be nearly impossible to reproduce that element exactly as it appears in the reference. How are such points taken into account during evaluation? Also, how is non-text content—such as photographs and other images—handled? Is it provided as additional input, generated as a placeholder, or what is the assumption here? I think it would be helpful to include examples of textual requirements in the Appendix so we can better understand the details.

Response

Textual requirements: We generate detailed requirements from both the reference webpage rendering and its HTML source code. These requirements cover layout, colors, typography, and the specific wording of headings, body text, navigation items, and buttons. For longer passages, the requirement-generation prompt likewise requests the actual text rather than a summary of its topic. The model therefore obtains textual content from the requirements and spatial structure from the sketch.

Image assets: We provide the paths to the corresponding image assets as part of the input and instruct the model to reference these paths directly in the generated HTML, allowing the original assets to be loaded during rendering. For the Sketch2Code benchmark, we follow its setting and replace the original images with placeholder images.

Example requirement (abridged): “Use a light-blue background (#e6f5fb) and dark-blue headings (#114a8b). Place the heading ‘The Refreshing Alternative’ and subtitle ‘Full flavour without the sugar.’ on the left, with a product illustration on the right. Include a ‘BUY NOW’ button. Use the supplied image 3316a98ae0.jpg as a splash-texture layer behind the product.”

Comment 5: Synthetic Data Quality and Quality-Check Results

Regarding the quality of the synthesized data, Section 5.1 states that 348 out of 400 samples passed the quality check. What kinds of errors were found in the filtered data? What differences exist between the synthesized sketches and the hand-drawn sketches? Also, were the textual requirements checked?

Response

We clarify the filtering criteria, sketch differences, and textual requirement verification below.

Errors identified during filtering: We exclude samples with the following issues:

  • Broken images: Images fail to load, leaving broken-image icons, alt text, or empty boxes and placeholders where image content should appear.
  • Broken layouts: Severe element overlap, unreadable overlapping text, substantial off-screen content, or obvious horizontal overflow and layout collapse.
  • Nearly blank pages: Pages consist predominantly of a uniform background with very little visible content, suggesting incomplete rendering.
  • Non-webpage content: Pages dominated by a single image, poster, or video cover, without a meaningful interface layout to represent as a sketch.

Synthesized versus hand-drawn sketches: Synthesized sketches are generally more regular in geometry and alignment. Human-drawn sketches exhibit greater variation in strokes, positions, and proportions, and photographed sketches may additionally contain perspective distortion, shadows, and other capture noise.

Verification of textual requirements: After generating the requirements, we perform an additional model-based verification step to check whether the required textual content from the reference webpage is included. If omissions are detected, the model revises the requirements, and this process repeats until the verification identifies no missing required text.

Comment 6: Direct Text Matching

Wouldn’t it be more accurate to directly compare the generated text with the reference HTML text—using character-by-character matching or similar methods—rather than using VLM-as-a-judge to assess text similarity?

Response

Direct text matching can provide a more objective measure of textual fidelity. Our rendering-based evaluation additionally considers how text is actually presented: correct text in the HTML may still be obscured, clipped, or positioned outside the visible area because of layout errors. Such cases are not fully captured by matching the underlying text alone. We also assess similarities in font family, size, style, and weight, extending the evaluation beyond textual content accuracy to its visual presentation.

To address this concern, we additionally evaluate our method and the newly added baselines using the static metrics introduced by Sketch2Code. These include a text-matching metric and other rule-based metrics for block matching, position, color, and layout, alongside CLIP similarity. We apply these metrics to both benchmarks; the results are shown below.

Scores range from 0 to 1, with higher values indicating better performance. Visual is the arithmetic mean of Block, Text, Position, Color, and CLIP. Total is an additional summary defined here as (Layout + Visual) / 2.

BenchmarkModelLayoutVisualBlockTextPositionColorCLIPTotal
SketchFactory-BenchQwen3-VL-8B + DARR (ours)0.16050.65650.55600.75190.65830.66150.65480.4085
SketchFactory-BenchQwen3.6-35B-A3B0.05400.37160.15490.47660.40420.43490.38730.2128
SketchFactory-BenchQwen3.5-9B0.04980.35160.15450.45270.37060.38980.39040.2007
SketchFactory-BenchGLM-4.6V-Flash0.00890.04780.03350.05710.04960.05170.04690.0283
SketchFactory-BenchGLM-4.6V0.03530.15370.10720.18750.15870.16130.15350.0945
Sketch2CodeQwen3-VL-8B + DARR (ours)0.24690.85080.76180.94900.82070.85580.86680.5489
Sketch2CodeQwen3.6-35B-A3B0.16730.64710.47200.75640.65300.67540.67880.4072
Sketch2CodeQwen3.5-9B0.14970.64450.46230.76350.63330.66660.69670.3971
Sketch2CodeGLM-4.6V-Flash0.10600.41830.36740.47970.39460.41950.43020.2621
Sketch2CodeGLM-4.6V0.15560.56820.49920.64580.54880.56870.57870.3619

Our method achieves the highest text-matching score and the highest scores on the other reported static metrics among these models on both benchmarks, providing complementary evidence beyond VLM-based evaluation.

Comment 7: SFT Before GRPO

It appears that training was conducted using only GRPO without an SFT step, but I think it is common practice to perform SFT before GRPO. Was the baseline performance sufficient enough this time that SFT was not necessary? Did you conduct a comparison between cases with and without SFT?

Response

The base models already possess basic sketch-to-code generation capabilities, providing a starting point for direct RL. Moreover, multiple HTML implementations can produce the same rendered appearance. SFT on a single reference implementation may bias the model toward a particular coding style, whereas our goal is to improve the rendered result. We therefore directly optimize the models through RL using rewards based on the generated webpage renderings.

To examine the effect of SFT, we additionally evaluated SFT and SFT+RL on Qwen2.5-VL-7B. The comparison with the base model and direct RL is shown below.

SketchFactory-Bench — GLM-4.6V judge

Training settingVisual structureColor & aestheticTextual contentUser interfaceOverall
Base3.373.384.033.073.46
SFT4.684.674.984.434.69
SFT + RL4.814.975.564.454.94
Direct RL5.085.115.754.835.19

Sketch2Code — GLM-4.6V judge

Training settingVisual structureColor & aestheticTextual contentUser interfaceOverall
Base4.684.815.964.344.95
SFT5.845.866.675.615.99
SFT + RL6.105.966.885.876.20
Direct RL6.116.167.175.836.32

Direct RL slightly outperforms SFT+RL.

Reviewer_VvzyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SketchFactory, a unified framework that integrates training data synthesis, task-specific evaluation, and reinforcement learning to generate runnable front-end code from sketches and requirements. It further introduces Difficulty-Adaptive Reward Reweighting (DARR), a reward reweighting strategy for reinforcement learning that targets the policy’s weaker dimensions. Experimental results show improvements on two benchmarks.

Critical strengths
  1. The research question is interesting, and the task design is clear. The data synthesis pipeline addresses the data bottleneck, while the subsequent training strategy alleviates imbalance across multiple dimensions.
  2. The experimental results are consistent. DARR improves performance on both benchmarks under both judges.
Critical weaknesses
  1. Methodological novelty is limited. The main contribution lies in integrating these steps into a practical pipeline. The data synthesis pipeline and benchmark are useful and practical, while DARR is a relatively incremental reward reweighting strategy for reinforcement learning.
Questions for authors

No further questions.

4 comments

We thank you for the careful reading of our paper and for the constructive comments. Regarding your concerns, I have the following clarifications.

Comment 1: Incremental Methodological Novelty

Methodological novelty is limited. The main contribution lies in integrating these steps into a practical pipeline. The data synthesis pipeline and benchmark are useful and practical, while DARR is a relatively incremental reward reweighting strategy for reinforcement learning.

Response

Thank you for recognizing the contributions of our data synthesis pipeline and benchmark. Regarding methodological novelty, prior webpage generation methods use fixed reward weights or predefined training stages, which do not adapt reward priorities to the model’s evolving learning state. DARR dynamically reweight reward dimensions based on the model’s training state, addressing uneven learning progress across dimensions in sketch-to-code generation. Our experiments further show that improvements in structural understanding are accompanied by gains in text-related performance, suggesting positive interactions across dimensions. For example, even accurately reproduced text may be obscured or clipped by an incorrect layout and therefore fail to appear correctly on the rendered webpage. By directing learning toward weaker structural dimensions, DARR helps alleviate such issues, allowing improvements in layout to also benefit text presentation.

To further assess the effectiveness of our approach, we expanded the comparison with four additional competitive baselines and evaluated them using both GLM-4.6V judgments and the static metrics introduced by Sketch2Code.

GLM-4.6V evaluation. Scores range from 0 to 10. F denotes SketchFactory-Bench and S denotes Sketch2Code. Each benchmark Overall is the unweighted mean of its four category scores. Total is the equal-weight average of the two benchmark Overall scores.

ModelF StructureF ColorF TextF UIF OverallS StructureS ColorS TextS UIS OverallTotal
Qwen3-VL-8B + DARR (ours)6.056.016.365.836.066.676.637.436.406.786.42
Qwen3.6-35B-A3B5.955.756.085.715.876.636.477.196.406.676.27
Qwen3.5-9B4.714.574.844.384.625.205.035.924.875.264.94
GLM-4.6V-Flash5.915.736.315.675.906.005.896.915.826.156.03
GLM-4.6V6.005.796.265.785.966.326.127.116.136.426.19

Static evaluation. Scores range from 0 to 1, with higher values indicating better performance. Visual is the arithmetic mean of Block, Text, Position, Color, and CLIP. Total is defined here as (Layout + Visual) / 2.

BenchmarkModelLayoutVisualBlockTextPositionColorCLIPTotal
SketchFactory-BenchQwen3-VL-8B + DARR (ours)0.16050.65650.55600.75190.65830.66150.65480.4085
SketchFactory-BenchQwen3.6-35B-A3B0.05400.37160.15490.47660.40420.43490.38730.2128
SketchFactory-BenchQwen3.5-9B0.04980.35160.15450.45270.37060.38980.39040.2007
SketchFactory-BenchGLM-4.6V-Flash0.00890.04780.03350.05710.04960.05170.04690.0283
SketchFactory-BenchGLM-4.6V0.03530.15370.10720.18750.15870.16130.15350.0945
Sketch2CodeQwen3-VL-8B + DARR (ours)0.24690.85080.76180.94900.82070.85580.86680.5489
Sketch2CodeQwen3.6-35B-A3B0.16730.64710.47200.75640.65300.67540.67880.4072
Sketch2CodeQwen3.5-9B0.14970.64450.46230.76350.63330.66660.69670.3971
Sketch2CodeGLM-4.6V-Flash0.10600.41830.36740.47970.39460.41950.43020.2621
Sketch2CodeGLM-4.6V0.15560.56820.49920.64580.54880.56870.57870.3619

On SketchFactory-Bench and Sketch2Code, our method achieves the highest VLM-as-a-judge score among the models. To further assess performance beyond VLM-based evaluation, we also evaluate the models using the static metrics introduced by Sketch2Code. Our method achieves the highest scores across all reported static metrics on both benchmarks, with larger performance margins. Notably, our 8B model outperforms the 106B GLM-4.6V model under both evaluation approaches.

Dear Reviewer,

Thank you for your constructive comments. We have submitted our responses to your questions. If you have any remaining concerns, we would be happy to provide further clarifications.

Thank you for the detailed response, which have addressed my main concerns. I have read the rebuttal and the other reviews.I will keep my rating.

Thank you for confirming that our response has addressed your main concerns and for taking the time to consider our rebuttal and the other reviews.

We will incorporate the clarifications and additional experimental results into the revised manuscript.

Reviewer_8k2uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SketchFactory, a unified framework for sketch-to-code generation that integrates training data synthesis, task-specific evaluation, and a reinforcement learning algorithm. Experiments across two benchmarks demonstrate the potential of the proposed approach to improve sketch-to-code generation, although the reported performance gains are incomplete in the provided results.

Critical strengths
  1. The proposed SketchFactory is of practical value for the research of sketch-to-code generation. The SketchFactory framework provides a potentially useful pipeline for improving sketch-to-code generation.
Critical weaknesses
  1. The technical contribution is limited. The proposed SketchFactory is a combination of existing approaches (training data synthesis, vision-language model evaluation, and reinforcement learning).
  2. The presentation should be further improved by providing a more detailed and accessible explanation of the mathematical formulations.
  3. It would be better to include more recent and competitive baselines for sketch-to-code generation. In particular, the use of Qwen2.5-VL and GLM 4.1V may not adequately reflect the capabilities of the latest available models.
Questions for authors

Please refer to Critical Weaknesses.

4 comments

We thank you for the careful reading of our paper and for the constructive comments. Regarding your concerns, I have the following clarifications.

Comment 1: Limited Technical Novelty

The technical contribution is limited. The proposed SketchFactory is a combination of existing approaches (training data synthesis, vision-language model evaluation, and reinforcement learning).

Response

SketchFactory goes beyond combining existing approaches: it introduces task-specific designs for three challenges in sketch-to-code generation.

1. Training data synthesis. The high cost of manual sketch annotation makes it difficult to construct large-scale, high-quality datasets. Consequently, existing work has provided only small evaluation benchmarks, without datasets at the scale needed for model training. We address this limitation with an automated pipeline that synthesizes sketches. Combining edge detection with DOM geometry preserves layout while replacing text and images with sketch placeholders. This enables scalable data construction while maintaining structural consistency between the generated sketches and the original webpage renderings.

2. Evaluation. Existing sketch-to-code evaluation inherits webpage reconstruction metrics from design-to-code, where close spatial correspondence is expected. Hand-drawn sketches contain positional and proportional deviations, so these metrics can penalize valid implementations. The Introduction explicitly identifies this limitation: “these metrics may penalize geometric differences even when generated pages faithfully capture the intended layout.” Our evaluation uses a VLM to assess multiple interpretable dimensions, allowing layout intent to be assessed beyond strict geometric matching.

3. Reinforcement learning. Prior webpage generation methods use fixed reward weights or separate training stages, which, as stated in the Introduction, “do not adapt to uneven progress across reward dimensions during training.” This matters for sketch-to-code because visual structure remains harder to learn than textual content, as shown in our training analysis. DARR dynamically assigns greater weight to dimensions with lower recent scores, adapting optimization to the model’s changing weaknesses. With the same training data and backbone, DARR outperforms fixed-weight GRPO across both backbones, both benchmarks, and both judges.

Comment 2: Mathematical Presentation

The presentation should be further improved by providing a more detailed and accessible explanation of the mathematical formulations.

Response

We will refine the presentation to make the mathematical formulations clearer and more accessible. If the reviewer could indicate which specific parts are unclear, we would be happy to provide more detailed clarification.

Comment 3: Stronger and More Recent Baselines

It would be better to include more recent and competitive baselines for sketch-to-code generation. In particular, the use of Qwen2.5-VL and GLM 4.1V may not adequately reflect the capabilities of the latest available models.

Response

We have expanded the comparison with four additional baselines: Qwen3.6-35B-A3B, Qwen3.5-9B, GLM-4.6V-Flash (9B), and GLM-4.6V (106B). Below, we report their results on both benchmarks alongside our Qwen3-VL-8B + DARR model, using GLM-4.6V as the judge and the Sketch2Code static metrics.

GLM-4.6V evaluation. Scores range from 0 to 10. F denotes SketchFactory-Bench and S denotes Sketch2Code. Each benchmark Overall is the unweighted mean of its four category scores. Total is the equal-weight average of the two benchmark Overall scores.

ModelF StructureF ColorF TextF UIF OverallS StructureS ColorS TextS UIS OverallTotal
Qwen3-VL-8B + DARR (ours)6.056.016.365.836.066.676.637.436.406.786.42
Qwen3.6-35B-A3B5.955.756.085.715.876.636.477.196.406.676.27
Qwen3.5-9B4.714.574.844.384.625.205.035.924.875.264.94
GLM-4.6V-Flash5.915.736.315.675.906.005.896.915.826.156.03
GLM-4.6V6.005.796.265.785.966.326.127.116.136.426.19

Static evaluation. Scores range from 0 to 1, with higher values indicating better performance. Visual is the arithmetic mean of Block, Text, Position, Color, and CLIP. Total is defined here as (Layout + Visual) / 2.

BenchmarkModelLayoutVisualBlockTextPositionColorCLIPTotal
SketchFactory-BenchQwen3-VL-8B + DARR (ours)0.16050.65650.55600.75190.65830.66150.65480.4085
SketchFactory-BenchQwen3.6-35B-A3B0.05400.37160.15490.47660.40420.43490.38730.2128
SketchFactory-BenchQwen3.5-9B0.04980.35160.15450.45270.37060.38980.39040.2007
SketchFactory-BenchGLM-4.6V-Flash0.00890.04780.03350.05710.04960.05170.04690.0283
SketchFactory-BenchGLM-4.6V0.03530.15370.10720.18750.15870.16130.15350.0945
Sketch2CodeQwen3-VL-8B + DARR (ours)0.24690.85080.76180.94900.82070.85580.86680.5489
Sketch2CodeQwen3.6-35B-A3B0.16730.64710.47200.75640.65300.67540.67880.4072
Sketch2CodeQwen3.5-9B0.14970.64450.46230.76350.63330.66660.69670.3971
Sketch2CodeGLM-4.6V-Flash0.10600.41830.36740.47970.39460.41950.43020.2621
Sketch2CodeGLM-4.6V0.15560.56820.49920.64580.54880.56870.57870.3619

On SketchFactory-Bench and Sketch2Code, our method achieves the highest VLM-as-a-judge score among the models. To further assess performance beyond VLM-based evaluation, we also evaluate the models using the static metrics introduced by Sketch2Code. Our method achieves the highest scores across all reported static metrics on both benchmarks, with larger performance margins. Notably, our 8B model outperforms the 106B GLM-4.6V model under both evaluation approaches.

Dear Reviewer,

Thank you for your constructive comments. We have submitted our responses to your questions. If you have any remaining concerns, we would be happy to provide further clarifications.

Dear Reviewer,

We apologize for reaching out again as the discussion period is coming to an end. We would be grateful for any follow-up feedback on whether our responses and additional experiments have addressed your concerns. A brief indication of any remaining concerns would help us identify what further clarification may be useful before the discussion closes.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. In response to the reviews, we have added five sets of experiments covering stronger baselines, static evaluation, human assessment, an additional judge, and training strategies. These experiments address the reviewers' concerns about competitiveness, evaluation reliability, and the choice of direct reinforcement learning.

  1. Comparison with four additional baselines. We evaluated Qwen3.6-35B-A3B, Qwen3.5-9B, GLM-4.6V-Flash (9B), and GLM-4.6V (106B) on both SketchFactory-Bench and Sketch2Code. Under the GLM-4.6V judge, our Qwen3-VL-8B + DARR model obtains overall scores of 6.06 and 6.78, respectively, exceeding all four additional baselines. For comparison, GLM-4.6V (106B) scores 5.96 and 6.42, and Qwen3.6-35B-A3B scores 5.87 and 6.67.

  2. Static evaluation on both benchmarks. We additionally evaluated our 8B DARR model and the four new baselines using the Sketch2Code metrics: layout, block matching, text matching, position, color, and CLIP similarity. Our model achieves the highest scores on every reported static metric among these models on both benchmarks. Its text-matching scores are 0.7519 and 0.9490, and its aggregate Visual scores are 0.6565 and 0.8508, on SketchFactory-Bench and Sketch2Code, respectively. These results provide complementary evidence beyond VLM judgments and directly address the request for text matching.

  3. Human evaluation on 300 cases. We randomly sampled 150 cases from each benchmark to compare outputs from Qwen3-VL-32B and our Qwen3-VL-8B + DARR model. Human and GLM-4.6V preferences agree on 118/150 cases (78.67%) from SketchFactory-Bench and 124/150 cases (82.67%) from Sketch2Code, yielding 80.67% agreement overall. This study assesses the consistency of VLM judgments with human preferences for the sampled comparisons.

  4. Evaluation with an additional judge, HY4. We evaluated 13 model/training configurations from the original paper on both benchmarks with HY4. DARR outperforms fixed-weight GRPO in overall score for both backbones on both benchmarks. Our 8B DARR model achieves the highest overall scores among the evaluated configurations: 6.53 and 6.59, compared with 6.23 and 6.50 for its fixed-weight counterpart. These results support the consistency of DARR's gains across judges.

  5. SFT and SFT+RL comparisons. We added SFT and SFT+RL experiments on Qwen2.5-VL-7B and compared them with the base model and direct RL under GLM-4.6V. On SketchFactory-Bench, the four settings score 3.46 / 4.69 / 4.94 / 5.19; on Sketch2Code, they score 4.95 / 5.99 / 6.20 / 6.32, in the same order. Direct RL achieves the highest overall score on both benchmarks, supporting our training choice in this setting.

We also clarify SketchFactory's contributions: scalable synthesis of aligned sketch–requirement–code triplets, evaluation that accommodates geometric imprecision, and DARR's adaptation to uneven learning progress across reward dimensions. The existing controlled comparisons with fixed-weight GRPO isolate the benefit of adaptive weighting.

We thank the reviewers again for their constructive feedback. We hope that these additional experiments and clarifications address the concerns raised. If any concerns remain, we would be happy to provide further clarification and discuss them in detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.