acceptodds
Under review as a conference paper at ICLR 2027

AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

Abstract

Generating information-rich visual artifacts such as slides, posters, and dashboards as code is appealing because the outputs are structured, editable, and verifiable. Yet code models cannot see how their choices of layout and color come together on the canvas, often limiting spatial planning and aesthetic quality. Image generators excel at visual design but can misrender requested content, making their outputs unreliable targets for direct reproduction. We therefore introduce AesCode, which generates code grounded in the content prompt while using an image generated from the same prompt as an aesthetic guidance. To do so, AesCode represents the target as a design graph that describes the whole canvas and makes each property attributable. From this graph, we derive complementary cross-modal rewards: deterministic verifiers score properties parsable from code and rendering, while a VLM judge assesses the non-parsable ones with a sample-specific Visual Graph Rubric (VGR) tied to graph elements and relations. We optimize these decoupled rewards through reinforcement learning with per-channel normalization to jointly improve spatial planning, aesthetic quality, and verifiable correctness. To evaluate this setting, we build a benchmark that scores verifiable correctness and visual quality separately, with visual scores highly consistent with human judgments. We find that reference images substantially improve the visual quality of code generated by existing models, and ablations show that AesCode benefits from references both as model input and as a source of visual rewards. Relative to same-scale Qwen3-VL baselines given the same references, AesCode-8B and AesCode-32B improve visual quality by 31.1 and 22.4 percentage points, reaching scores comparable to those of closed-source frontier models such as GPT-5.5 and Claude Opus 4.8.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, VLM, Multimodal Code Generation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q3h3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AesCode, a framework for generating HTML visual artifacts from content prompts and generated reference images. It uses a design graph to separate content requirements from visual cues and construct complementary rewards. Qwen3-VL models are trained through SFT followed by reinforcement learning with per-channel reward normalization. Evaluation on a new 300-prompt benchmark and three external benchmarks show improvements in content correctness and visual quality over the base models, with visual quality comparable to the evaluated frontier models.

Critical strengths
  1. Thorough evaluation. The paper tests diverse open-source and proprietary models, trains both 8B and 32B variants, and includes ablations, external benchmarks, and human evaluation.

  2. Gains persist without reference images. Removing the reference at inference reduces AesCode-8B’s Visual score by only 1.0 point, supporting its usefulness in prompt-only generation.

Critical weaknesses

1. The evaluation does not directly test selective use of noisy references. A central premise is that generated references may contain incorrect text or data, so AesCode should borrow their design while following the prompt’s content requirements. However, the experiments mainly assess overall correctness and visual quality, without directly testing whether the model rejects erroneous reference content when it conflicts with the prompt.

2. The handling of conflicting reward signals is unclear. Text and Table/Chart rewards enforce prompt-derived requirements, while Semantic Layout, Whitespace, and VGR reward reference-derived properties. These objectives can conflict when correcting content requires changing the layout—for example, adding an element omitted from the reference. The paper should clarify how such conflicts are resolved.

Questions for authors

How does AesCode behave on references with content errors? how are reference-derived graph constraints and rewards adjusted to avoid penalizing correct outputs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Q3h3

We sincerely thank the reviewer for the detailed and thoughtful review, and for recognizing the thoroughness of the evaluation and that the gains persist without the reference at inference.

W1, W2, and Q: How does AesCode handle references whose content contradicts the prompt, both in the reward channels during training and in the behavior of the trained model?

The three points concern the same question, so we answer them together. We ran a full study on such conflicts, reported in the General Response (G1–G4), and summarize below the parts relevant to W2 and then W1.

1. How the reward channels handle a conflict (W2, and the second part of Q)

Only two channels compare the output with the reference. VGR is not purely reference-derived, since its design graph and questions follow the prompt wherever it specifies content or structure, and take only layout and visual style from the reference.

Reward channels at a conflict point (G2, Tables 2 and 3).

Reward channelCompared withAt a conflict point
Text, Table/Chartthe prompt onlyrewards following the prompt (100% of paired data conflicts)
VGRthe prompt, and the reference for layout and stylefavors the prompt at 75.0% of the points, and the reference at 0.0%
Semantic Layout, Whitespacethe referenceonly a weak pull toward the reference
  1. No hand-written rule or adjustment is needed. GDPO normalizes each channel within the group of rollouts before adding them, so what matters is how strongly each channel separates two rollouts. At a conflict point, a VGR question favors the prompt-following rollout by about 0.7σ0.7\sigma, while Semantic Layout favors the copy of the reference by only 0.030.03–0.55σ0.55\sigma (G2), because both layout channels are soft by construction.
  2. In the reviewer's example, adding an element that the reference omits is rewarded. In Fig. C3 (first row), the prompt asks for a row of four buttons ending with "Strategy Partnerships", while the reference draws only three. AesCode-8B draws all four and keeps the color palette, border style, and section layout of the reference. The prompt-derived checks reward the added button, and since the row stays in place, the layout channels change little. Beyond this example, across the 110 controlled inputs in which the reference draws too few or too many elements, AesCode-8B follows the prompt more often than its base model (62.3% vs. 52.2% on counts, 47.4% vs. 35.7% on missing or extra elements; Table 5).

2. How the trained model behaves on conflicts (W1, and the first part of Q)

Noisy references are already part of our evaluation. The G1 audit finds natural conflicts in 19.7% of all samples, the benchmark included, and all reported results include them. Following the reviewer's suggestion, we also test the conflict point directly on a controlled set of 391 inputs (G3, examples in Fig. C2).

  1. AesCode-8B follows the prompt over a wrong reference more often than its base model on all eight conflict types (57.3% vs. 39.2%), most clearly on the text (91.7% vs. 62.5%) and data (65.9% vs. 29.1%) errors the reviewer mentions, and approaches GPT-5.5 on the subtler natural conflicts (53.9% vs. 61.2%; G3, Table 5). Fig. C3 also shows outputs where the reference swaps two statistic tiles, reverses an arrow, or uses a different title, and AesCode-8B restores what the prompt asks for while keeping the design of the reference.
  2. Our reward design remains robust when conflicts are frequent in training. With 49.5% instead of 19.5% conflict samples in training, prompt adherence on the controlled set improves further (62.5% vs. 57.3%; last row of Table 5), and overall performance on the original benchmark is virtually unchanged (82.53 vs. 82.94; G4, Table 7).

We have added the controlled conflict study and these examples to Appendix C.4 (Table C2, Figs. C2 and C3), and stated the source of each reward channel in §3.2 (marked in blue). We hope this study addresses the reviewer's concerns, and would be grateful if the reviewer would take it into account in the final assessment.

Dear Reviewer Q3h3, thank you again for your detailed review. Following your suggestion, the General Response and our reply now test the conflict points directly on a controlled set of 391 inputs, explain how each reward channel treats a conflict so that correct outputs are not penalized, and add this study to Appendix C.4 of the revised paper. We would be grateful if you could let us know whether this resolves your concerns, and we would welcome the chance to address any remaining questions.

Thanks for your efforts in addressing the concerns provided. I have increased my score accordingly.

Reviewer_np3pOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on generating visual code (such as slides and posters) that is information-rich, editable, and verifiable. The paper proposes AesCode. This approach uses an image generated from the same prompt as a visual layout reference, while employing a design representation to define the target. The goal is to align the layout and colors specified in the prompt with the visual aesthetics provided by the reference image. The authors constructed a benchmark of 300 examples to evaluate both verifiable correctness and visual quality. The results report that AesCode-8B/32B significantly outperforms the Qwen3-VL baseline under identical reference image conditions, achieving visual quality comparable to closed-source models like GPT-5.5 and Claude Opus 4.8.

Critical strengths
  1. The benchmark comprehensively covers both content and visual aspects, ensuring a reasonable evaluation. On your proposed benchmark, the model demonstrates significant improvement compared to the baseline (Qwen-VL). Furthermore, when provided with reference images, the model outperforms advanced closed-source models in most capabilities.
  2. The method design is rational and effective. The "SFT + GDPO" paradigm successfully enhances the model's capabilities. Specifically, SFT improves the model's code generation ability, while the reward design within GDPO is well-conceived, effectively optimizing the visual aesthetics of the generated code.
Critical weaknesses

The paper claims to achieve a decoupling of content and design; however, there is a lack of experimental validation to support this assertion. Specifically, if the input reference image contains incorrect content, does the model inadvertently reference this content, thereby negatively affecting the subsequent generation?

Questions for authors
  1. Can you demonstrate that your model has truly achieved a decoupling of content and design?
  2. What are the specific models used for generating the final evaluation checklist and for the scoring process, respectively?
  3. Is the performance gain of VGR over direct rubrics stable across repeated training runs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer np3p (1/2)

We sincerely thank the reviewer for the constructive review, and for finding the benchmark comprehensive and the SFT + GDPO design, including the rewards, rational and effective.

W1 / Q1: How does the model behave when the reference contains content that contradicts the prompt?

In AesCode, decoupling refers to the supervision: content and design are optimized as separate objectives, through separate reward channels. Content is rewarded only against the prompt (Text, Table/Chart, and the content and structure questions of VGR), while the reference, as one possible realization that may contain errors, mainly guides layout and visual style (G2, Table 2). It is a decoupling of the reward channels, and it does not require the model to disentangle content and design internally.

Because all three reviewers raised related questions, we ran a supplementary study, reported in full in the General Response (G1–G4), and summarize the parts on content conflicts here.

By design, training follows the prompt when the two conflict. The training prompt states that the generation prompt is the primary contract and the reference is visual guidance only. The content rewards (Text, Table/Chart) check the output only against the prompt, and VGR favors the prompt at 75.0% of the natural conflict points and never favors the reference (G2, Tables 2 and 3).

We then test the trained model on both the natural conflicts in the benchmark and a controlled set of 391 conflict inputs across eight categories (G3), and also stress-test training by raising the share of conflict samples from 19.5% to 49.5% (G4). As shown in Tables 5 and 7 of the General Response:

  1. AesCode-8B learns to follow the prompt over a wrong reference. It follows the prompt far more often than its base model across both controlled and natural conflicts, approaching GPT-5.5 on the natural ones (Table 5). The gains are largest specifically on the content conflicts the reviewer asks about, namely text (reaching 91.7%) and data (more than doubling the base model).

  2. Our reward design remains robust even when conflicts are frequent in training. Because a conflict typically affects only one or two local details in a dense infographic (G1), raising the training conflict rate to 49.5% leaves overall benchmark performance virtually unchanged (Table 7), while further improving prompt adherence on conflict inputs (last row of Table 5).

We have clarified the definition of decoupling in §3.1 and added the controlled conflict study to Appendix C.4 (marked in blue).

Q2: What are the specific models used for generating the final evaluation checklist and for the scoring process?

  1. Authoring. GPT-5.5 (high reasoning) writes the checklist questions from the prompt and its parsed requirements. As described in Section 3.4, we developed this step with humans in the loop on a separate 50-instance set: annotators reviewed the questions and ranked generation pairs, and we revised the pipeline until agreement exceeded 90%, then froze it for the 300-instance benchmark.

  2. Scoring. GPT-5.5 answers each frozen question from the rendering of the candidate output, and every answer must cite the region it was read from. Replacing it with Gemini-4-argon gives highly consistent scores (96.2%); we use GPT-5.5 mainly because its API is more accessible to us. Both steps are now stated in §3.4 and Appendix E.1.

Since GPT-5.5 is also one of the compared systems, we validated this judge against human judgments. In the blinded human study in the paper, the human majority agrees with the checklist judgments on 93.3% of the sampled questions, and the Visual score agrees with direct human pairwise choices in 80.66% of the comparisons.

AuthorsOpenReview ↗

Response to Reviewer np3p (2/2)

Q3: Is the performance gain of VGR over direct rubrics stable across repeated training runs?

1. Repeated run. We repeated both trainings with a different seed.

RewardRunTextBoundaryTable/ChartRuleContentLayoutStyleVisual
+VGROriginal (paper)94.0688.3687.7990.0786.4187.8053.2175.80
+Direct rubricOriginal (paper)94.1088.1287.9490.0583.1786.7647.8972.61
+VGRNew seed94.4289.2188.6590.7685.2286.1151.6174.31
+Direct rubricNew seed94.0388.0587.8689.9882.1585.6446.2371.34

The new runs start from the same SFT checkpoint and use the same data, hyperparameters and number of steps as the original ones; only the random seed differs. We were only able to afford one additional run per method. Training the 8B model takes about 3.5 days on 8 B200 GPUs, because each rollout has a long multimodal input and a long HTML output, and every rollout has to be rendered in a headless browser and then verified and judged asynchronously. We acknowledge that two runs per method are not enough to estimate the variance precisely. VGR is ahead of the direct rubric in both runs, by 3.19 and 2.97 Visual points, with similar Rule scores, and the lower VGR run (74.31) is still above the higher direct-rubric run (72.61).

2. Why VGR helps. We compared the two question sets on the same outputs.

VGRDirect rubric
Subjective questions25%67%
Compound questions50%96%
Style questions14%35%
Defect questions (overlap, clipping)15%1%
Rollouts with a perfect score5%35%
Questions answered differently by two rollouts of a prompt14%7%

Direct-rubric questions are mostly subjective and compound, lean toward style, and almost never ask about defects. Most outputs pass such questions, so scores saturate: 35% of the rollouts get a perfect score, against 5% for VGR. GDPO normalizes rewards within a group of 8 rollouts, so only questions answered differently within the group produce a training signal; VGR has twice as many (14% vs. 7%).

Dear Reviewer np3p, thank you again for your constructive review. Beyond the conflict study in the General Response (G1–G4), our reply clarifies that decoupling refers to the supervision, shows that AesCode-8B follows the prompt over a wrong reference more often than its base model, names the checklist authoring and scoring models, and reports a repeated VGR training run with a new seed. We would appreciate hearing whether these address your concerns, and we are glad to provide any further details.

Reviewer_xBuGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a new reliable way to generate accurate infographics. It uses a design graph to organize supervision, combining programmatic checks for content and structural correctness with a graph-based visual rubric.
It’s based model is Qwen3-VL models and its trained through supervised fine-tuning followed by multi-reward reinforcement learning using GDPO, each with a few thousand examples. Evaluation includes a 300-instance benchmark, three external benchmarks, component ablations and a blinded human study.
The results show substantial improvements over the base models and stronger verifiable correctness with visual quality comparable to the evaluated frontier models.

Critical strengths

The paper addresses a useful problem with a sensible approach. The separation between what an artifact needs to communicate and how it should look is a nice direction. For slides and infographics, a visually appealing result is of limited use if its labels, data, or relationships are wrong. The proposed combination of content checks and visual guidance is well motivated, and the results suggest that it improves correctness while retaining competitive visual quality.

The empirical study is a strong. The comparisons with alternative visual rewards and reconstruction-based training helps explain the choices behind the method. The human evaluation also makes the visual-quality results more convincing, and the external benchmarks show that the improvements extend beyond the authors’ own evaluation. Taken together, these experiments support a useful empirical contribution.

Critical weaknesses

The handling of structurally incorrect references is not yet well established. The prompt-derived content checks provide a clear safeguard against some reference errors, and both graph-authoring stages receive the prompt. My concern is narrower: when the reference contradicts a requested ordering or connection, it appears there's a conflict on rewards and which structure the training rewards going to favor.

Questions for authors

When a reference contradicts prompt-required ordering, grouping, or connectivity, how are the graph, rubric questions, and reference-based layout rewards handled?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xBuG

We sincerely thank the reviewer for the careful reading, the encouraging assessment, and the insightful question. As the reviewer notes, the content checks and both graph-authoring stages see the prompt, which narrows the question to how structural conflicts are resolved.

Q: When a reference contradicts prompt-required ordering, grouping, or connectivity, how are the graph, rubric questions, and reference-based layout rewards handled, and which structure do the training rewards favor?

Short answer: the graph and the rubric questions take the structure from the prompt. Only the two layout rewards pull toward the reference, and the pull is weak, so the training rewards favor the structure in the prompt.

All three reviewers asked related questions, so we ran a supplementary study on references that contradict the prompt and report it in full in the General Response. Below we summarize its four parts (G1–G4), focusing on structural conflicts.

PartWhat we didFinding for structural conflicts
G1Audited all 7,708 references against the requirements in their promptsThe reference violates 0.40% of the structural requirements; 7.1% of the samples contain such a violation (Table 1)
G2Analyzed how each reward channel treats a conflictThe graph and VGR follow the prompt; Semantic Layout and Whitespace pull toward the reference, but weakly (Tables 2 and 3)
G3Built a controlled test set whose references are regenerated with a wrong order, grouping, connection, or label binding, and tested the trained modelAesCode-8B follows the prompt more often than its base model on all four structural types, with the largest gains on order and connection (Table 5)
G4Replaced 30% of the training set with new controlled conflict samples as a stress testThe model trained on this data follows the prompt more often on order, grouping, and label binding, and its overall score on the original benchmark is almost unchanged (last row of Table 5; Table 7)

For the three components in the question:

  1. The graph and the rubric questions take structure from the prompt. Both graph-authoring stages read the prompt and the reference, follow the prompt wherever it specifies content or structure, and borrow layout and visual style from the reference. The rubric questions are then written from this graph. If the reference draws a timeline in the wrong order, the questions still check the order in the prompt. On natural conflicts, VGR favors the prompt at 75.0% of the conflict points and never favors the reference (Table 3).

  2. The reference-based layout rewards are where the conflict arises, but they are soft. Semantic Layout applies a soft Gaussian to text-block positions, so a block displaced by 5% of the canvas still scores about 0.95, and it does not see arrows, containers, or label bindings at all. Whitespace only checks grid occupancy.

  3. Together, the rewards favor the structure in the prompt. Under GDPO's within-group normalization across 8 rollouts, a VGR question on a conflicting structure gives the prompt-following rollout an advantage of about 0.7σ0.7\sigma, whereas Semantic Layout favors the copy of the faulty reference by only 0.030.03–0.55σ0.55\sigma, most on ordering (G2). The prompt-aligned structural signal therefore dominates.

As summarized in the G3 and G4 rows of the table above (General Response, Tables 5 and 7), the trained model reflects this reward design. It follows the prompt over a faulty reference structure more often than its base model, and remains robust when the share of conflict samples in training is raised to 49.5%. We have added the controlled conflict study to Appendix C.4, and clarified in §3.2 and Appendix D.4 that the graph and the rubric questions follow the prompt wherever it specifies structure (marked in blue).

Dear Reviewer xBuG, thank you again for your thoughtful review. In the General Response we added a study of references that contradict the prompt (G1–G4), and in our reply to you we explain how the graph, the rubric questions, and the layout rewards handle structural conflicts, with the trained model following the prompt more often than its base model on all four structural types. We would be grateful to know whether this addresses your question, and we are happy to clarify anything further.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all reviewers for their careful and constructive reviews. AesCode separates what an infographic must communicate, taken from the prompt, from how it should look, guided by a generated reference image, and trains with verifiable content rewards and a graph-based visual rubric (VGR). We are glad that the reviewers find the problem useful and the approach well motivated (xBuG, np3p), the gains over the base models clear with visual quality comparable to frontier models (xBuG, np3p, Q3h3), and the evaluation thorough (xBuG, np3p, Q3h3).

What this rebuttal adds. All three reviewers raised one common and insightful question: what happens when the reference image contradicts the prompt? Generated references are imperfect by nature, and AesCode is built with this in mind. It rewards content and structure against the prompt and takes layout and style from the reference. Arbitrary multimodal conflicts remain an open problem for future work, and here we test how far our current design holds in a supplementary study (G1–G4) on the unmodified pipeline. Its main findings are:

  1. Real references are imperfect, but only locally. Across all 7,708 samples, references satisfy 99.1% of the checkable prompt requirements (generated by GPT-Image-2 from detailed prompts); although 19.7% of the samples contain a conflict, it typically affects only 1 or 2 of about 34 requirements (G1).
  2. The reward design ties content and structure to the prompt. Text and Table/Chart rewards use only the prompt, and VGR favors the prompt at 75.0% of the conflict points and never the reference (G2).
  3. AesCode-8B learns to follow the prompt over a wrong reference. It does so more often than its base model, both on the natural conflicts in the benchmark (53.9% vs. 34.6%) and on a controlled conflict test set (57.3% vs. 39.2%) (G3).
  4. Our decoupled reward design remains robust when conflicts are much more frequent in training. When the share of conflict samples in the training set rises from 19.5% to 49.5%, overall benchmark performance is virtually unchanged (82.53 vs. 82.94), and prompt adherence on conflict inputs further improves (G4).

What we revised (marked in blue in the revised paper). We added the controlled conflict study to Appendix C.4, clarified decoupling and the roles of the prompt and the reference (§3.1, §3.2, Appendix D.4), and named the checklist authoring and scoring models (§3.4, Appendix E.1).

G1. How often does the reference contradict the prompt?

This audit is independent of training and evaluation and follows IGenBench [1], which checks whether generated images satisfy their prompts. Taking the prompt as ground truth, GPT-5.5 first splits it, without seeing the reference, into requirements that can each be checked as yes or no against an image (about 34 per prompt, e.g., "there are eight step circles"); it then judges each requirement against the reference. A conflict is a requirement that the reference clearly violates. We audit all 7,708 samples (7,408 training, 300 test), manually verify a stratified sample of 477, and correct the rates accordingly.

Table 1. Conflict types and how often they occur.

TypeWhat counts as a conflictExampleRequirements violatedSamples affected
A. Element setA1 Count: a stated count is wrong. A2 Missing/extra: a required unit is missing, or an unrequested one is drawn.8 steps requested, 7 drawn0.27%7.4%
B. Logical structureB1 Order: wrong order. B2 Grouping: an element in the wrong group. B3 Connection: an arrow links the wrong elements or points the wrong way. B4 Label binding: a label on the wrong element.timeline drawn as 4 → 6 → 5 → 70.40%7.1%
C. Texta required line of text is missing, or prompt instructions are drawn as textfooter lacks the contact line0.08%2.5%
D. Datavalues differ from the given data or sit in the wrong category, a stated trend is reversed, or table columns are added or missingbars shifted by one category6.1%6.3%
Any0.89%19.7%

"Requirements violated" is the rate within each type (Any: over all requirements); "Samples affected" is the share of samples with at least one such conflict. The contradictions are mostly matters of detail, or cases that need stronger logical ability than current generators have, such as a count off by one, inexact chart values, and long chains of ordered or connected elements. This agrees with IGenBench [1], which reports 0.90 question-level and 0.49 infographic-level accuracy for the strongest generator it evaluates, and also finds data-related requirements to be the main bottleneck.

[1] Tang et al. IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation. arXiv:2601.04498, 2026.

AuthorsOpenReview ↗

General Response (2/3)

G2. How does the training pipeline handle conflict samples?

By design, the model must follow what the prompt specifies. The reference shows one possible realization, which may contain errors, so it mainly serves what the prompt leaves open, such as layout and visual style. The training system prompt and every user message state that the generation prompt is the primary contract and the reference image is visual guidance only. VGR is built the same way. Its design graph and questions are built from both the prompt and the reference, and follow the prompt wherever it specifies content or structure. Two channels, Semantic Layout and Whitespace, do compare the output with the reference; point 3 below shows that both are soft. Table 2 groups the reward channels by where their ground truth comes from.

Table 2. Reward channels and their dependence on the reference.

GroupChannelsGround truthEffect of a wrong reference
Directly verifiableText, Table/Chartprompt onlynone by construction; a correct output is rewarded even if the reference is wrong
VGRVGRprompt (what it specifies) + reference (layout and style the prompt leaves open)rewards the prompt in most conflicts (Table 3)
Soft, reference-basedSemantic Layout, Whitespacereference imagesoft by construction, so the effect of a local error is limited
Reference-freeExecution, Boundaryrenderingnone
  1. Text and Table/Chart check the output against requirements extracted from the prompt, so a wrong number or missing text in the reference cannot lower the reward of a correct output. In a paired test on data conflicts, Table/Chart favors the prompt-following output in 100% of the pairs.

  2. For VGR, we take the conflicts in element set and logical structure that occur naturally in our data, and check which side the VGR questions favor (Table 3). Text and data are left out here, because by design they are checked by the Text and Table/Chart verifiers. VGR favors the prompt at 75.0% of the conflict points and never favors the reference. At 18.3% of the points no question covers the conflict, since we limit each sample to 10–20 questions so that the VLM judge can answer them reliably, which leaves out some small local details.

Table 3. How VGR questions treat naturally occurring conflicts between the prompt and the reference.

Outcome at the conflict pointDefinitionShare of conflict points
Favors the prompta VGR question passes only if the output follows the prompt75.0%
Favors the referencea VGR question passes only if the output copies the reference0.0%
Neutrala VGR question covers the point, and both versions pass it6.7%
Not askedno VGR question covers the point18.3%
  1. Semantic Layout and Whitespace compare the output with the reference, but both are soft by construction. Semantic Layout scores each text block with a Gaussian of its distance to the matched reference block and averages over all blocks, so a block displaced by 5% of the canvas still scores about 0.95. It does not look at arrows, containers, or which label belongs to which element. Whitespace compares coarse occupancy grids, so adding one element or swapping two changes only a few cells.

  2. Because GDPO normalizes each reward channel by its within-group standard deviation across the 8 rollouts of a sample, a channel's influence depends on its relative variance within the group. When two rollouts differ only at a conflict point, a triggered VGR question creates a gap of about 0.7σ0.7\sigma in favor of the prompt, whereas Semantic Layout favors copying the faulty reference by only 0.030.03–0.55σ0.55\sigma, depending on the conflict type. Consequently, the prompt-aligned gradient dominates.

G3. How does the trained model handle contradictory inputs?

Natural conflicts are uncontrolled, so we construct a controlled conflict test set in which the type and the location of each conflict are known: 391 inputs built from the 300 benchmark prompts, each with a reference that contradicts the prompt at one known target point (Table 4). We build them in two ways:

  1. For the structural types (B1–B4), the reference must be a real image with the wrong structure. We rewrite the prompt P into P* with the altered structure, generate an image I* from P*, and pair the original prompt with it, (P, I*).
  2. For A1, A2, C, and D, we keep the image and edit the prompt, (P*, I).

Table 4. Controlled conflict test set.

A1 CountA2 Missing/extraB1 OrderB2 GroupingB3 ConnectionB4 Label bindingC TextD DataTotal
#Samples5357514935505145391
AuthorsOpenReview ↗

General Response (3/3)

Every input is admitted only after an independent check confirms that the reference clearly violates the prompt at the target point, and, for (P, I*), that I* actually draws the altered structure. Inputs that fail are discarded. For example, in a B1 (order) input, the prompt asks for a timeline of milestones in 2004, 2010, 2016, and 2024, and the reference I* draws them as 2004 → 2016 → 2010 → 2024. In an A2 (missing/extra) input, the edited prompt asks for a three-step service flow, while the reference I draws four steps. More examples are in Figs. C2 and C3 in Appendix C.4. Fig. C2 shows inputs built in both ways, with the conflict point marked in the reference. Fig. C3 shows AesCode-8B outputs on four of these inputs (count, order, connection, and text), where at each conflict point the output follows the prompt while the rest keeps the design of the reference.

We run each model on the benchmark inputs that contain a natural conflict and on the controlled conflict inputs (3 outputs per input), and use GPT-5.5 to judge, at the conflict point, whether the output follows the prompt. An output that copies the reference at that point, or gets the point wrong in any other way, counts as not following the prompt. The judgment concerns only whether the output follows the prompt at that point, not how well the output is designed.

Table 5. Share of outputs that follow the prompt at the conflict point (%), by model.

ModelA1 CountA2 Missing/extraB1 OrderB2 GroupingB3 ConnectionB4 Label bindingC TextD DataAll controlledNatural conflicts
GPT-5.552.985.788.177.525.989.7100.072.776.161.2
Qwen3-VL-8B52.235.730.740.119.036.762.529.139.234.6
AesCode-8B62.347.462.146.333.742.791.765.957.353.9
(Used in G4) AesCode-8B, +30% conflicts69.251.267.251.532.149.397.272.962.556.7

AesCode-8B learns to follow the prompt over a wrong reference, and does so far more often than its base model on both natural conflicts (53.9% vs. 34.6%) and the controlled set (57.3% vs. 39.2%). The gain is clearest for text and data, which verifiers check during training. Connections and exact counts remain hard for every model, including GPT-5.5. Since natural conflicts are subtler, all models do worse on them; even GPT-5.5 follows the prompt in only 61.2% of the outputs and copies the error from the reference in most of the rest.

G4. What if the training data contained many more conflicts?

To test how far the design holds, we replace 30% of the training samples (2,220 of 7,408) with controlled conflict samples, each with one injected conflict of a known type (Table 6). They are constructed and checked in the same way as the test set in G3, and they replace samples that the audit found free of natural conflicts. After the replacement, 49.5% of the training samples contain at least one conflict, up from 19.5% originally. We train AesCode-8B on this data with the same configuration as in the paper, and evaluate it with the same protocol as in G3 (last row of Table 5) and on the original benchmark (Table 7).

Table 6. Conflict samples for replaced trianing by type.

A1 CountA2 Missing/extraB1 OrderB2 GroupingB3 ConnectionB4 Label bindingC TextD DataTotal
#Samples3051972782782772773043042,220

Table 7. Original benchmark performance.

AesCode-8B training dataTextBoundaryTable/ChartRuleContentLayoutStyleVisualOverall
Original (paper)94.0688.3687.7990.0786.4187.8053.2175.8082.94
30% replaced by conflict samples94.0987.9386.2789.4386.2286.9753.6975.6382.53
Δ+0.03-0.43-1.52-0.64-0.19-0.83+0.48-0.17-0.41

Even when nearly half the training references contain errors (49.5% vs. 19.5%), our pipeline maintains its overall benchmark performance (82.53 vs. 82.94; Table 7) while further strengthening conflict resolution. Across almost every conflict category, prompt adherence increases, rising from 57.3% to 62.5% on the controlled set and from 53.9% to 56.7% on natural conflicts (last row of Table 5), which approaches the 61.2% natural-conflict accuracy of GPT-5.5. Rather than requiring synthetic conflict injection in standard practice, this stress test demonstrates that our decoupled reward design is inherently robust. Even under severe reference degradation, it prevents visual errors from corrupting code generation and actively teaches the model to prioritize the prompt.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.