acceptodds
Under review as a conference paper at ICLR 2027

Behavior-Aligned Trust Regions for SVG Reinforcement Learning

Abstract

Reinforcement learning from verifiable rewards (RLVR) for SVG generation typically constrains policy updates in token space. Yet token-space constraints can be poorly aligned with visual behavior: the same drawing may admit many syntactically distinct programs, while a small textual edit can substantially alter the rendering. This mismatch creates opportunities for reward hacking, where the policy exploits shortcuts that preserve a proxy reward while degrading the drawing. We study whether a behavior-aligned representation space can complement tokenlevel constraints in SVG RLVR, and instantiate this space with a variational latent model built on a frozen SVG-finetuned LLM encoder. The Gaussian posterior provides a probabilistic local representation with closed-form divergences and uncer- tainty estimates, while a lightweight metric adapter calibrates the local geometry using programmatic SVG transformations. We first show that frozen encoder hidden states do not reliably track rendering similarity, motivating the learned metric. We then combine the calibrated behavior representation with a token-KL syntax anchor in a hybrid constraint, using an uncertainty-adaptive local radius. On SVG RLVR, the resulting method reduces the two studied collapse behaviors (viewBox and length collapse) while preserving or improving generation quality, and the representation remains useful on unseen sources and out-of-domain SVGs, while performance is lower on held-out transformation families.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning from Verifiable Rewards (RLVR), Trust Region / Policy Constraint, Variational Latent Space (VAE), Scalable Vector Graphics (SVG) Generation, Semantic Representation / Metric Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VVkrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether SVG reinforcement learning benefits from constraining changes in rendering behavior rather than relying only on token-space constraints. It combines a frozen SVG-finetuned LLM encoder, a variational representation trained to reconstruct DINOv2 rendering features, and a metric adapter trained using equivalent and corrupted SVG pairs. The resulting behavior-distance penalty complements token-KL regularization, with a posterior-based adaptive radius. Validation includes representation probes, SVG generation/refinement benchmarks, and ablations against reward patching, a DINOv2 render anchor, and a non-variational metric. The method improves reported generation quality and collapse-detection rates, although generalization to held-out transformation families is weaker.

Critical strengths
  1. Useful empirical analysis of representation geometry. The probe-pair experiments demonstrate that raw frozen LLM representations can place rendering-equivalent rewrites farther apart than visually disruptive edits. This provides useful evidence that representations used for behavioral constraints require explicit validation, rather than assuming task-finetuned hidden states already provide an appropriate distance metric.
  2. Meaningful controls and improvements beyond the proposed failure detectors. The paper includes reward-patching, DINOv2 render-anchor, and non-variational controls, with three-seed results. It also reports improvements in established generation metrics: for example, FID improves from 11.43 for token-KL alone to 9.37 for Full. These results provide evidence of practical benefits beyond merely reducing the paper’s own collapse statistics.
Critical weaknesses

1. The central reward-hacking claims are not sufficiently disentangled from the reward design and detector definitions.

Length collapse is defined solely by Lgen<0.5LgtL_{\rm gen} < 0.5L_{\rm gt}, without requiring any accompanying visual degradation. This definition may therefore conflate legitimate compression with an actual failure mode. Moreover, ReffR_{\rm eff} is constant once the generated length falls below this threshold, so continued shortening cannot be explained simply by progressively increasing efficiency reward. The use of GT length as the reference is also sensitive to the serialization of the ground-truth code and requires paired code-length information to be available during training.

For viewBox collapse, reproducible rollout examples and more precise rendering/preprocessing details are needed to establish the proposed exploit mechanism. In addition, the Fixed-reward control changes both the rendering setup and the length penalty, making it difficult to attribute the observed effect to either factor individually. The manuscript also does not report a matched ablation that removes ReffR_{\rm eff} while keeping the remaining setup fixed, nor does it quantify the incremental benefit of Full under the patched setting.

As a result, the generality of these failure modes and the causal explanation offered for them remain unclear.

2. The necessity of the additional representation-learning machinery is not fully established.

Render-anchor directly uses DINOv2 features, whereas Full additionally introduces a frozen code encoder together with trained representation- and metric-learning components. This comparison changes both the representation source and the amount of task-specific training, so it does not isolate whether the gains specifically require code-side encoding, or whether similar improvements could be obtained by calibrating the visual representation using the same supervision.

The existing zinvz_{\rm inv}-only control is informative and already performs close to Full. However, the incremental contributions of the VAE, posterior uncertainty, and adaptive radius would benefit from more tightly matched ablations.

This is particularly important because Full introduces additional training complexity and also incurs higher rollout-time cost than the simpler render-anchor baseline.

Questions for authors

1. Clarification and validation of the failure detectors.

Among the outputs classified as length collapse, what fraction actually exhibit loss of necessary visual content, as opposed to being valid but more compact programs? Please provide an independent visual validation of the detector, together with sensitivity analyses over the collapse threshold.

For viewBox collapse, please provide representative RL rollouts including the generated SVG code, rasterization dimensions, target-image preprocessing, rendered outputs, and individual reward components. In particular, does the failure mode persist when the evaluation resolution is fixed independently of the generated SVG content?

2. Evidence needed to establish benefits beyond the specific reward design.

What happens when only ReffR_{\rm eff} is removed, while all other reward weights and training settings are kept unchanged? Does Full continue to improve output quality under both this setting and the patched reward/rendering setup? Please also provide the exact formulation of the patched length reward.

These comparisons are important for determining whether the proposed method addresses a broader optimization issue or primarily compensates for failure modes induced by the particular reward configuration used in the current experiments.

3. Clarification of the GT-length assumption.

Why is one-half of the length of a single GT program an appropriate efficiency threshold, particularly in Text-to-SVG, where multiple programs of substantially different lengths can produce equally valid outputs? Are the reward values and collapse labels stable if the GT program is replaced by a rendering-equivalent but compressed or expanded version?

Please also clarify how the proposed training setup applies when paired GT SVG code is unavailable.

4. Evidence needed to isolate the representation contribution.

For Render-anchor, please specify the exact distance function, reference construction, penalty coefficient, and radius-calibration procedure. How does a DINOv2-based metric head trained with the same pairwise supervision compare?

Given the strong performance of the zinvz_{\rm inv}-only variant, please clarify which matched experiments isolate the incremental contributions of the VAE and the uncertainty-based radius. It would also be helpful to report whether these incremental gains are statistically reliable across runs.

Reviewer_JcMWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the problem of the trust region​ in SVG generation RL, designs a representation space aligned with SVG rendering behavior, and establishes the trust region in the variational latent space of a VAE​ based on a frozen encoder from an SVG-finetuned LLM. In addition, it contribute a probe-pair benchmark and a ranking-consistency protocol for evaluating rendering-aligned representations

Critical strengths
  1. The research problem is clearly defined and well-motivated: the trust region in SVG generation RL requires a specially designed representation space, as standard approaches fail to align with SVG rendering behavior.

  2. The ablations are comprehensive and provide clear evidence that the proposed variational latent space is indeed aligned with SVG rendering behavior, isolating the contribution of each design component.

Critical weaknesses
  1. All results are from a single encoder family (InternSVG-8B/Qwen2.5-7B) and remain unvalidated for other SVG encoder families.

  2. The authors emphasize the importance of perceptual DINOv2 reconstruction, but do not ablate other semantic features (such as SigLIP)

Questions for authors
  1. Have different SVG-finetuned LLMs been compared? Is the approach effective only with a specific model?
  2. Has the method been evaluated across different RL algorithms?
Reviewer_7ye4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that token-space proximity is poorly aligned with rendered SVG behavior. It builds a frozen-encoder variational representation, calibrates it using programmatic equivalence/corruption pairs, and adds a latent-distance hinge penalty to GRPO while retaining token KL. An uncertainty-dependent radius adapts the behavioral anchor. Experiments evaluate representation geometry, SVG generation quality, and two collapse behaviors.

Critical strengths

A well-motivated distinction between source syntax and executed visual behavior, with explicit counterexamples and representation probes. The controls include both a patched reward and rendering-feature regularization; the paper does not merely compare against token KL. The method improves multiple quality measures while substantially lowering the two reported collapse rates. Held-out transformations expose a realistic generalization limitation, and the paper openly acknowledges shared-feature blind spots.

Critical weaknesses

The discriminative-only head achieves FID 9.71±0.35 and collapse rates 1.2±0.5%/2.4±0.9%, compared with 9.37±0.28 and 1.0±0.4%/2.0±0.7% for the full model. These small differences over three seeds do not establish a robust incremental benefit. A paired comparison and simpler deterministic uncertainty/control baseline would help.

The automatic detector flags output below half the ground-truth token length, while the efficiency reward favors compression. Valid concise SVGs can satisfy this criterion. The existing human/probe checks are useful, but per-detector precision/recall and threshold sweeps on rollout outputs are needed to interpret collapse rates.

Clarify whether the EMA is prompt-conditioned, how multiple valid drawings are represented, the EMA coefficient and fallback trigger, and how calibrated variance receives supervision. Otherwise the anchor could penalize legitimate alternative compositions or make radius widening unreliable.

Held-out transformation AUROC falls to 0.807, and the policy is not explicitly optimized to exploit the adapter. The frozen metric and DINO-based calibration can share blind spots with the reward. Claims should remain about the two tested failure modes until adversarial stress tests are supplied.

Labeling the low-beta, high-KL setting “posterior collapse” is confusing under the usual meaning of posterior collapse (posterior approaching the prior). Define the diagnostic actually measured.

Questions for authors

It seems that a good idea to save SVG RL training from collapse. Is there any evidence could show how the svg changes with RL training steps., when i use behavior align trust region or not, and is the training more robust?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.