- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses credit assignment in RL for visual code generation, where rewards come from rendered pixels but actions are code tokens. It proposes Structure-Grounded Credit Assignment (SGCA), which uses the correspondence between rendered nodes (DOM elements or Matplotlib artists) and their source spans to assign finer-grained credit. Spatial credit rescales the GRPO advantage across tokens according to each block’s rendering error, oriented by the advantage sign and preserving the mean advantage, without learned models or extra rollouts. Temporal credit, for multi-turn repair, scores each edit by its return-to-go minus a learned node-conditioned value, which can also be used to choose the next component to edit.
Critical strengths
- The paper observes that each rendered visual part (e.g., a DOM element) maps to the code span that produced it. It uses this correspondence to weight tokens by the rendering error of their region, instead of broadcasting one advantage per response. This grounding of credit in visual-part–code correspondence is new, cheap (no learned model or extra rollouts), and generalizes from web pages to charts.
Critical weaknesses
-
Lack of clarity of region-to-code mapping. The mapping relies on heuristics: source blocks are paired with DOM nodes in document order, and the discrepancy is measured inside the generated node’s box, which assumes positional alignment with the target. Displaced elements, missing elements, and CSS in <style> blocks can all misattribute credit. The only quantitative localization test uses single-fault injection on 100 pages, achieving 67% top-1 accuracy, which does not establish robustness under multiple compounding errors. The paper does not report how often the gate fires, how often pairing fails, or what fraction of tokens fall outside all blocks. This matters because the block-shuffled control shows that misplaced weights actively hurt structure (−3.7 HTML score).
-
The generality claim is not supported beyond static rendering. Pages are rendered with scripts disabled, and both the reward and all evaluation metrics lack interactive evaluations. Functional behavior, such as whether a button works when clicked, is therefore neither rewarded nor measured, and the paper includes no dynamic or interaction-based evaluation. Yet the conclusion presents SGCA as a general recipe for learning from visual feedback (line 485). In interactive settings, the relevant code (event handlers, scripts) usually lies outside the element's source span, and errors only appear in specific UI states, so the node-to-code correspondence becomes many-to-many and state-dependent.
Questions for authors
-
Could the authors explain the region-to-code mapping in more detail and report how reliable it is in practice? For example, how often does source-to-DOM pairing fail, when the browser repairs malformed HTML, and how is credit assigned when an element is displaced or missing from the generated page?
-
How would the SGCA method handle interactive elements such as buttons? For these, the relevant code (event handlers, scripts) usually lies outside the element’s source span, and errors only appear after an interaction. Interactive elements are very important for webpages, but it seems the authors only use static screenshots to evaluate and disabled scripts when rendering. Evaluation benchmarks are also limited to static webpage evaluation. Could you provide results on WebGen-Bench as you mentioned in line 91?
-
Have the authors tried SGCA with a newer or different parameter scale base model (e.g., the Qwen3-VL series or Qwen2.5-VL-3B)?
3 comments
Thank you for the detailed response and additional experiments. The new results address most of my concerns. I would suggest explicitly discussing the limitations of SGCA in handling dynamic and interactive webpages in the Limitations section, particularly the challenges of state-dependent credit assignment.
Overall, the rebuttal is appreciated and I will maintain weak acceptance for this paper.
Thank you for your constructive feedback and continued support. We will explicitly discuss SGCA’s limitations in handling dynamic and interactive webpages in the Limitations section, with particular attention to the challenges of state-dependent credit assignment.
Response to Reviewer KXB5
Thank you for your careful review and constructive comments.
Changes are marked in blue; section, table, and figure numbers refer to the revised manuscript.
W1 + Q1: Mapping and reliability.
Single-turn attribution proceeds as follows:
<head>/<style>and unpaired blocks, receive the mean block weight. Eq. 2 applies sign orientation and normalization.In repair, blocks are visible children of the replaced component; spatial weights apply to the edit payload, and the gate retains SGCA-ST's temporal advantage.
Reliability. Pairing is truncated on 4.6% of training responses and fails outright on 2.7%; outright failures trigger the gate. Browser repairs occur on 6.8% of responses, declining from 12.1% to 3.4%. Of these repairs, 31% change block structure; the gate catches 93% of those, leaving 0.15% of responses with uncaught browser-repair misalignment. Overall gate rate is 11.4%. Outside-block tokens account for 23.5% of tokens, 81% of them in
<head>/<style>.Harder faults. Displacement can produce error within the generated element's own box. On 500 pages, explicit displacement reaches 63% Top-1 versus 4% chance. A missing element has no source span, so no token is blamed directly; its error falls in the parent's or a neighbour's box, and one of them ranks first on 54% of pages. With two/three simultaneous faults, Recall@k (k = fault count) is 56.5%/51.0% versus 8%/12% chance. Shared-CSS attribution identifies affected blocks with 31% Top-1 and 57% Top-3. The rule's tokens receive the default weight; linking rules to matched nodes would extend credit to the rule itself.
Misplaced weights. The original −3.7 HTML result uses full permutation. Permuting each response's weights with probability 0.25 retains 75% of the CLIP gain and yields 33.21 HTML versus GRPO's 31.91.
Revision: Sections 4.1–4.2 and Appendix C.1 clarify mapping and fallback; Appendix F.3 (Table A13) reports reliability statistics, harder faults, and the partial-permutation runs.
W2 + Q2: Interactive behaviour and generality.
We evaluate the existing single-turn checkpoints on WebGen-Bench with its official functionality and appearance pipeline. Each instruction is given as text-only input without an agent framework, and the model generates a single HTML file in one pass (three seeds).
SGCA-S improves appearance by 0.15 over GRPO. Functional accuracy is on par (28.6 versus 28.1); we do not claim the 0.5-point difference as a gain. Training uses static renders with scripts disabled and no interaction reward, so spatial credit targets appearance.
We agree that the original conclusion overreached, and we have removed the statement that structure-grounded credit is "a general recipe for learning from visual feedback". The claim is now that the same credit rules are reused through task-specific structural mappings, shown for static HTML and Matplotlib renders. For buttons, credit must connect UI state, event handler, and visual outcome; rendering after interaction events and linking handlers to target elements through selectors would extend the interface to state-conditioned records. We leave this to future work.
Revision: Section 5.1 introduces this evaluation; Appendix F.5 (Table A15) reports results. Section 6 removes the general-recipe claim and states the static-render scope.
Q3: Newer and smaller models.
We add Qwen2.5-VL-3B and Qwen3-VL-8B using Table A2's single-turn settings (Design2Code evaluation, three seeds). The 7B row is from the original paper.
Both new models improve, with confidence intervals that exclude zero. The absolute CLIP gain shrinks as the base model improves, while the relative LPIPS reduction stays between 13% and 16%. All three policies are Qwen-VL models; we have not tested another family.
Revision: Sections 5.1–5.2 and Appendix F.2 (Table A12) report model settings, results, and the model-family scope.
We thank the reviewer again for the thoughtful comments.