acceptodds
Withdrawn from ICLR 2027

Reinforcement Learning from Intermediate Renders for Image-to-Code Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Reinforcement learning is increasingly used to post-train vision-language models for image-to-code generation, such as generating SVG code from a reference image, by optimizing rewards computed from the final rendered output. However, relying on a single terminal reward provides sparse feedback that is poorly aligned with the contribution of individual tokens. A generated program may contain operations that accurately reproduce some parts of the target image alongside others that introduce errors, yet all tokens are trained from the same final outcome. We observe that many intermediate code prefixes are not only executable, but already produce meaningful partial renders that reflect progress toward the target. This property provides a natural source of denser supervision during generation. Based on this observation, we introduce , an RL framework with a token-level render-progress reward that turns changes between intermediate renders into localized feedback for the generated sequence. We evaluate our approach on Image-to-SVG and Image-to-TikZ generation. Across both tasks, our method improves over supervised fine-tuning and standard GRPO, yielding new state-of-the-art open-source models. This shows that intermediate rendering provides a simple and effective source of process supervision for RL post-training of image-to-code models. Code and model weights will be released upon publication.

Primary area:
reinforcement learning
Keywords:
image-to-code generation, reinforcement learning, token-level reward, image-to-SVG, image-to-TikZ, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vjhgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes IR4RL, a reinforcement learning method for image-to-code generation. Instead of relying only on a single reward computed from the final rendered program, the method exploits the fact that many intermediate code prefixes can be closed into executable programs and rendered. After each completed drawing command, the prefix is rendered and scored against the target image, and the change in visual score between consecutive renders serves as a render-progress reward. These segment-level rewards are propagated backward to preceding tokens with exponential decay and combined with the standard group-relative outcome advantage in GRPO. The method is evaluated on Image-to-SVG, using OmniSVG-4B on MMSVGBench, and on Image-to-TikZ, using DeTikZify-v2 on DaTikZ-v3. It improves over SFT, RAFT, and outcome-only RL baselines, achieves state-of-the-art results among open-source models, and produces substantially shorter programs.

Critical strengths
  1. The paper introduces an appealing process-level reward for image-to-code generation by exploiting renderable code prefixes, without requiring human annotations or a separately trained reward model. Using changes in visual similarity between consecutive renders provides localized feedback on whether newly generated segments improve or degrade the reconstruction.
  2. Compared with Render-in-the-Loop [1], the method leaves the inference procedure unchanged, requiring no step-wise rendering or visual feedback at inference time.

[1] Liang G, Wang Z, Hu J, et al. Render-in-the-loop: Vector graphics generation via visual self-feedback[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 229-245.

Critical weaknesses
  1. The paper motivates image-to-code generation through structured editing, yet its reward measures only visual similarity and does not explicitly encourage editable SVG structure. A single complex path and a collection of independently editable semantic elements can produce identical renders while offering substantially different editing experiences. Per-command scoring does not directly address this distinction. The evaluation reports visual quality and token length but does not examine path structure, semantic decomposition, or practical editing operations, leaving unclear whether the gains come at the expense of editability. Moreover, the path-based representation appears to exclude primitives such as <rect>, <circle>, and <text>, which expose properties that are often easier to manipulate than equivalent generic paths. Without evaluating these structural limitations, the paper provides insufficient support for the editability benefit motivating the task. Shorter code is not necessarily easier to edit.

  2. The authors render after every completed SVG path command, but these boundaries do not necessarily correspond to meaningful visual changes. For example, a move command typically adds no visible content, while the effects of some geometric operations become apparent only after subsequent commands are completed. Repeatedly rendering and scoring such prefixes may therefore increase computational overhead while providing limited useful supervision. However, the paper does not quantify how rendering frequency affects training time. Moreover, the rendering-frequency ablation reports only test reward, without showing how this choice affects the individual reconstruction metrics reported in Table 1. Further analysis is needed to establish whether per-command rendering offers sufficient benefits to justify its computational cost.

  3. The presentation lacks clarity, making it difficult to understand how the method actually works. The method hinges on the granularity of intermediate rendering and on how prefixes are closed, yet the main text describes both only vaguely. It merely states that a boundary is placed after every completed "drawing command". In SVG, however, "drawing command" may refer either to a graphical element such as <path> or to an individual command within a path, so readers can easily interpret the method as rendering at the element level. Only Appendix A.2 and Fig. A.2 make clear that boundaries are in fact placed after every move, line, curve, arc, or close command within a single path. Moreover, unfinished paths are closed back to their starting point and filled with a default black color. In addition, OmniSVG's tokenizer does not generate the outer <svg> and <path> tags, so the closure operator must supply them. These details directly determine what the intermediate renders look like and what the reward actually measures, yet they too are described only in the appendix.

Questions for authors
  1. Since the reward does not constrain how SVG elements are organized, could the authors report structural statistics of the generated SVGs, such as the number of paths per image and commands per path, compared with the SFT and outcome-only baselines? Does the reduction in token length come from more compact geometry, or from merging separate visual parts into fewer, more complex paths that are harder to edit?
  2. How does training time vary across the rendering frequencies in the ablation, and what fraction of it is spent on rendering and scoring intermediate prefixes? Could the authors also report the full evaluation metrics on the test set for this ablation, rather than test reward alone?
  3. For the Image-to-SVG setting, could the authors clarify in the method section what a "drawing command" refers to, i.e., whether boundaries are placed after each SVG element such as <path>, or after each command within a path (e.g., M, L, C)?
Reviewer_c1nxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces IR4RL, a dense-reward method for RL post-training of image-to-code VLMs. The method closes intermediate code prefixes into partial renders and uses them to measure progress toward the target. The score change between consecutive renders, Δj=Fj−Fj−1\Delta_j = F_j - F_{j-1}, serves as a render-progress reward. This reward is propagated backward over tokens with decay λ\lambda and added to the GRPO advantage with weight α\alpha.

The method is evaluated on image-to-SVG and image-to-TikZ benchmarks:

  • Image-to-SVG: using an OmniSVG-4B backbone on MMSVGBench, IR4RL outperforms vanilla GRPO from the same base model on every metric and produces shorter programs. A user study supports these results (Tab. 1, 2).
  • Image-to-TikZ: IR4RL improves DeTikZify-v2 on DaTikZ-v3, although the gains over baselines are less pronounced than on MMSVGBench.

The paper also ablates the process supervision (reward composition, propagation and render sparsity) and evaluates best-of-KK sampling (Fig. 6, 7).

Critical strengths

The method's performance on MMSVGBench is strong. Among learned methods, it achieves the best results on all visual-quality metrics, outperforming outcome-only GRPO, SFT and RAFT under the same OmniSVG-4B base model and data. It also rivals or exceeds costlier general-purpose multimodal LLMs such as GPT-5.2, Gemini 3 Flash, Claude Sonnet 5 and Qwen3-VL-235B. Moreover, performance consistently degrades as renders become sparser (Fig. 6c), which ties the gains to the proposed mechanism.

Critical weaknesses
  • Unreported compute cost. IR4RL renders every prefix of every rollout. The conclusion acknowledges that this adds training-time computation (lines 480–481), but the cost is not quantified. Given the same compute, outcome-only GRPO could use more rollouts or training steps.
  • Restrictive assumptions. By the paper's own description (lines 248–250), the method requires every prefix to be renderable and newly generated elements to persist, i.e., purely additive drawing. The conclusion acknowledges this limitation (lines 478–480), but transfer beyond SVG and TikZ is not tested. The assumption excludes scenarios such as subtractive CAD operations and layouts where later elements reflow earlier ones (HTML/CSS).
  • Weaker transfer to TikZ. On DaTikZ-v3, C-BLEU drops from 7.47.4 with outcome-only RL to 3.13.1 with IR4RL (Table 3), even though Sec. 5.2 claims improvement over outcome-only RL "across all metrics".
  • Per-domain engineering. Each new domain requires a closure operator, with design choices such as SVG's black fill for unfinished paths, a similarity metric, and tuning of α\alpha, λ\lambda and NN. Because λ\lambda decays per token, its effective reach depends on the tokenizer and the typical command length. The TikZ settings are not reported, so the tuning cost is unknown.
  • Myopic default reward. With λ=1\lambda = 1, the deltas telescope to FM−Fj−1F_M - F_{j-1}, which is potential-based shaping [1] and preserves the outcome reward's optimal policy. The default per-token λ=0.9\lambda = 0.9 breaks this property and favours locally visible gains. Yet the policy-preserving λ=1\lambda = 1 performs worse (Fig. 6b), which the paper explains only as a benefit of locality.

[1] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999.

Questions for authors
  1. [Clarification] What are the per-step wall-clock time and the number of renders per rollout for outcome-only RL versus IR4RL? Does IR4RL still outperform an outcome-only baseline given matched compute (e.g., more rollouts or training steps)?
  2. [Clarification] With the default per-token λ=0.9\lambda = 0.9, the process reward no longer telescopes and favours locally visible gains. Can you clarify why λ=1\lambda = 1 performs worse?
  3. [Clarification] How much tuning did TikZ require? Which α\alpha, λ\lambda and NN were used, and do they differ from the SVG settings?
  4. [Clarification] How sensitive are the results to the choice of closure operator?
  5. [Suggestion / future work] Can the method handle non-additive constructs such as SVG clip-paths and masks?
Reviewer_AjnaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces IR4RL, an RL post-training framework for image-to-code generation (Image-to-SVG and Image-to-TikZ) that uses intermediate renders as a token-level process reward. The method exploits the fact that prefixes of drawing code are executable and renderable: it places boundaries after each drawing command, closes truncated prefixes with a closure operator, scores consecutive intermediate renders with a visual similarity metric, and uses the per-command difference Δ as a process reward propagated back to tokens with exponential decay (λ), combined with the outcome reward under GRPO. The authors evaluate on two heterogeneous tasks and report state-of-the-art open-source results. Overall, this is a clean and valuable contribution to RL post-training for visual code generation. The motivation is strong, the method is simple and easy to reproduce, and the experimental validation is thorough, including ablations that directly support the mechanism. However, several aspects — the unexamined assumption that monotonically increasing render similarity is a safe learning signal, and the unexplored theoretical connection to potential-based reward shaping — need further clarification.

Critical strengths
  1. Timely and clean first systematic use of the renderer as a free process oracle for training-time supervision in image-to-code, requiring no human labels and no learned reward model, which keeps the method simple and highly reproducible.
  2. Transparent and thorough experimental validation across two heterogeneous representations (SVG/TikZ) with different syntax, rendering pipelines, and similarity metrics, including user studies and a VLM-as-judge evaluation in addition to automatic metrics.
  3. The ablations directly pin down the mechanism: α, λ, and render-frequency (N) are each isolated, and the finding that the process reward alone accounts for most of the gain supports the paper's core claim rather than peripheral effects.
  4. Honest positioning of the contribution: the authors explicitly acknowledge that RL reweights rather than creates capability (citing Yue et al.), and report best-of-K and base-model-size results that truthfully bound the gains.
Critical weaknesses
  1. The claim of novelty relative to the process-supervision literature is not fully supported: the Δ reward is formally equivalent to [1]potential-based reward shaping (Ng et al., 1999) with the renderer score as the potential, and this connection is neither cited nor discussed; relatedly, at λ=1 the process reward telescopes into the outcome signal, which could explain the reported drop at λ=1 (Fig. 6b) but is left unexplained.
  2. The assumption that monotonic progress equals the optimal trajectory is unexamined. There is no analysis of how Δ is distributed along sequence position, of non-monotone (corrective redrawing) trajectories, or of sensitivity to drawing order — all plausible failure modes of the process reward.
  3. It is not clear how the closure-snapshot distortion affects the signal: F_j is scored on a truncated program closed with default syntax, which differs most from the actual continuation early in the sequence; the resulting bias in early Δ estimates is not quantified.
  4. The evaluation lacks detail on two points: Table 1 labels the post-trained rows as OmniSVG-8B while the text states the 4B base model is used (labeling inconsistency to be reconciled), and the SVG training set is only 700 samples with no out-of-domain generalization reported.

[1] Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of ICML.

Questions for authors
  1. Can the authors provide an analysis of Δ along sequence position, along with the fraction of non-monotone trajectories and their final quality — i.e., evidence that monotone progress is indeed a safe assumption for the reward?
  2. How sensitive is the process reward to drawing order? For the same final image produced under two valid orderings, how different are the Δ sequences, and does the reward implicitly favor order over content?
  3. How are the closure-snapshot distortion and the per-position credit imbalance quantified, and what proportion of the overall gain is attributable to the process term versus the outcome term as a function of sequence position?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.