SketchFactory: A Unified Framework for Learning and Evaluating Sketch-to-Code Generation

Generating runnable front-end code from hand-drawn sketches and textual requirements can accelerate interface development. Existing work on sketch-to-code has primarily focused on benchmark development. However, a unified framework integrating training data synthesis, task-specific evaluation, and reinforcement learning algorithm remains lacking. To address this gap, we introduce SketchFactory, a pipeline that synthesizes high-quality sketch-to-code data by deriving sketches and textual requirements from rendered webpages. We further propose an evaluation strategy that uses a vision-language model to judge rendered pages along interpretable quality dimensions, accommodating geometric imprecision in sketches. Building on these multidimensional scores, we develop difficulty-adaptive reward reweighting DARR, a reinforcement learning algorithm that shifts reward weights toward the policy's weaker dimensions based on recent dimension-wise performance. Experiments on two benchmarks under two independent judges show that trained on SketchFactory data, DARR-Qwen3-8B raises the overall score of its base model by $1.42$–$5.05$ points on a $0$–$10$ scale across the four benchmark–judge settings.