acceptodds
Under review as a conference paper at ICLR 2027

StyleCraft: Latent Augmentations as a Step Toward Generalizable Reinforcement Learning

Abstract

Reinforcement learning agents often perform well in their training environments but degrade under visual distribution shifts. Existing approaches commonly address this problem with predefined pixel augmentations, whose effectiveness depends on how well the selected transformations cover unseen visual variation. We propose StyleCraft, a two stage framework that combines task style disentanglement with learned visual augmentation. During the first stage, pixel augmentations together with gradient reversed reconstruction encourage the policy visible task representation to discard appearance specific information while a separate style representation captures visual variation. Once the generative model has learned a sufficiently structured style space, the second stage uses interpolation between style representations to construct new visual observations while keeping the task representation fixed. These generated observations replace fixed pixel augmentations in the invariance objective and provide additional supervision for learning a more visually robust task representation. Experiments across KAGE Bench, DCS based locomotion, RL4VLA, and ManiSkill show that StyleCraft improves robustness to unseen visual conditions across controlled visual shifts, continuous control, and robotic manipulation. We additionally reproduce the ManiSkill scene on a real robot, where StyleCraft retains nonzero success under unseen table colors while the PPO baseline fails.

Primary area:
reinforcement learning
Keywords:
visual generalization, reinforcement learning, disentangled representations, latent augmentation, style interpolation, representation learning, zero shot generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1HL1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes StyleCraft for visual RL generalization. It aims to separate task and appearance representations, then uses a frozen copy of the learned encoders and decoder to generate training views by interpolating style codes. A continuing task encoder is trained to produce consistent representations across these views. The paper reports results on KAGE-Bench, distracting control tasks, RL4VLA, and ManiSkill, including a real-robot evaluation.

Critical strengths
  • Relevant objective and potentially useful augmentation mechanism. Using a learned appearance representation to generate additional training views is a meaningful extension of representation-invariance approaches, if the generated views preserve task content.
  • Broad evaluation scope. The inclusion of multiple visual-shift settings and a physical-robot experiment makes the empirical question consequential, although the current design limits what can be concluded about the proposed mechanism.
Critical weaknesses
  1. The claimed task–style disentanglement is not demonstrated. The paper assumes that its two encoders learn task-relevant content and visual style separately, but the training objectives do not guarantee this separation, and the paper provides no direct evidence that it occurs. There is no systematic factor-swapping analysis, task-preservation measurement, or visualization across the evaluated tasks. Final policy returns alone cannot establish that the proposed representation or generated views behave as claimed.

  2. The experimental methodology and reporting are seriously inadequate.

  • The paper contains no ablation isolating Stage 1, style interpolation, or the individual losses, so its central mechanism cannot be assessed.
  • The headline RL4VLA and ManiSkill experiments each use only one training seed. More seriously, the implementation does not consistently follow the method presented in the main text: gradient reversal and style KL appear only in KAGE-Bench; ManiSkill is listed as single-phase; and RL4VLA runs Stage 2 separately, alongside additional color-diverse frames, a reverse curriculum, and SIL whose treatment in the baselines is unclear. The stage configurations and augmentation settings are insufficiently explained.
  • Reporting discrepancies further undermine confidence: the KAGE improvement is stated as both 12.3% and 19%, while ManiSkill is described as having three held-out colors in the main text but 12 in Appendix Table 3.
  • The reference list contains only 16 entries despite the breadth of the topic and omits directly relevant work.

Taken together, these are substantive obstacles to reproducing the experiments and attributing the reported gains, rather than minor presentation issues.

  1. The novelty relative to prior work is insufficiently established. Focus On What Matters: Separated Models For Visual-Based RL Generalization (SMG) already uses separate branches to learn task-relevant and task-irrelevant visual representations, together with reconstruction and consistency objectives, without relying on StyleCraft’s two-stage training scheme. SMG is neither cited nor discussed. Style interpolation may be a meaningful extension, but the paper does not isolate its contribution or explain convincingly what it adds beyond this closely related approach.
Questions for authors
  • What evidence shows that the task and style encoders capture the intended factors, and that interpolated views preserve the original task state? In particular, how diverse are style codes paired within a minibatch?
  • What is the performance of Stage 1 alone under the same total training budget? How do results change without style interpolation, or if pixel augmentation continues throughout training?
  • Which stages, losses, data sources, and auxiliary training procedures are used in each benchmark? For RL4VLA, do PPO and RAD receive the same additional color-diverse frames, reverse curriculum, and SIL?
  • Please clarify the uncertainty of the RL4VLA and ManiSkill results, each trained with one seed, and reconcile the reporting discrepancies: KAGE’s stated improvements of 12.3% and 19%, and ManiSkill’s three held-out colors in the main text versus 12 in Appendix Table 3.
2 comments

Thank you for your review and feedback. We address your questions point by point in the appendix of the revised paper, where we clarify the experimental protocols and provide the relevant supporting results.

Thank you for the response. The additional probes and KAGE ablations address some of my concerns, but substantial issues remain. In particular, the three-seed KAGE ablation does not resolve the lack of independent training runs for the headline RL4VLA and ManiSkill results. Different environment seeds and manual object placements measure evaluation variability, not variability across independently trained policies. The conceptual advance over SMG also remains insufficiently established.

Overall, I consider the paper substantially below the standard required for acceptance at ICLR. The work needs a clearer justification of its novelty, more robust experimental validation, and more consistent methodological reporting. I therefore maintain my original score and encourage the authors to undertake a substantial revision.

Reviewer_cSAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper is about visual distribution shift in pixel-based RL. The main idea is to split the representation into a task part and a style part, trained with pixel-view consistency, a gradient-reversed reconstruction loss, and a Gaussian prior on style. After that the teacher is frozen and new consistency views are generated by interpolating style means. The method is evaluated on KAGE-Bench, DCS, RL4VLA, ManiSkill, and a real robot. One thing to note up front is that the appendix shows the actual recipe differs quite a lot between these settings (pp. 4-6, Eqs. 2-8; pp. 12-13, Tables 2-3).

Critical strengths
  1. The KAGE results are encouraging. Ten training seeds, several shift axes, mean of -502 vs -655 for RAD, best mean in 7 of 10 configurations. I checked the printed averages against the normalized summary and they match (pp. 2, 7-8).
  2. The generator is only used at training time, nothing extra at deployment, which is practical. Taking a learned augmentation approach across multiple environments and onto a real robot is a good amount of work, and Tables 2-3 do give a lot of architecture and training detail (pp. 9, 12-13).
Critical weaknesses
  1. My main problem is that the experiments don't actually isolate the two-phase interpolation mechanism that the paper is built around. ManiSkill is single-phase and uses action/value consistency under permuted styles instead. SimplerEnv adds a reverse curriculum and SIL. Three settings drop the style KL entirely (Table 3). The authors do disclose all of this, which I appreciate, but then it is not clear which result is supposed to support the headline mechanism and whether baselines got the same extras. The real robot result supports "decoder-based augmentation helps" in general, it does not really show that frozen-teacher Phase-II interpolation is what helps. The script excerpts also don't tell me which entry point produced which result.
  2. There is no controlled comparison separating interpolation from just continuing pixel consistency, or from self-reconstruction views, or from gradient reversal alone (pp. 7-9, 12-13). Also, fixing the task code doesn't guarantee the task state is preserved if the style code happens to carry task information. I am not saying the observations are corrupted, just that this needs to be checked directly. On top of that, Table 1 picks the best eval return over training, the manipulation runs use a single seed, and the real-robot trial counts are not given. Together these make the generalization evidence weaker than it looks.
Questions for authors
  1. Which entry point, config, checkpoint, and budget produced each result? Did the baselines get the same curriculum, SIL, architecture, and data access?
  2. How does the single-phase ManiSkill variant (and its three held-out colors in the plot) relate to the main method and to the twelve colors in Table 3? For the real robot, what are the successes/trials per color, and what do the error bars mean for single-seed runs?
  3. Do Phase-I-only and alpha=0 reconstruction-view controls at matched budget still lose to the full method? Is there any direct check that the task state survives interpolation? Harder environments can be future work.
1 comment

Thank you for your review and feedback. We address your questions point by point in the appendix of the revised paper, where we clarify the experimental protocols and provide the relevant supporting results.

Reviewer_1HW7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  1. The paper demonstrates that the results generalize to a real robot in Figure 6.
  2. StyleCraft is evaluated on a variety of benchmarks, demonstrating its broad applicability.
  3. StyleCraft achieves strong empirical performance relative to baselines. Many of the experiments also report results across multiple random seeds to establish the reliability of the results.
Critical strengths
  1. The paper demonstrates that the results generalize to a real robot in Figure 6.
  2. StyleCraft is evaluated on a variety of benchmarks, demonstrating its broad applicability.
  3. StyleCraft achieves strong empirical performance relative to baselines. Many of the experiments also report results across multiple random seeds to establish the reliability of the results.
Critical weaknesses
  1. My primary concern with the paper is the lack of ablations that clearly establish the primary contributions. The paper does not evaluate which components of StyleCraft are important for performance and does not test whether latent augmentations improve performance, despite this being a central claim. The paper also does not ablate the second training phase, the reconstruction objectives, or interpolation compared to simply swapping in the style latent from a different observation.
  2. The paper provides limited evidence that StyleCraft actually disentangles style and task representations. Nothing in the objective explicitly enforces this separation. The paper should quantitatively support these claims by demonstrating that varying the style latent preserves key task properties, such as joint poses.
  3. The paper discusses and cites the ALDA approach but does not empirically compare against it, despite ALDA being a more recent baseline than the other compared approaches.
  4. DCS only reports results for PPO and RAD, omitting the other baseline approaches from Table 1 designed for training robust agents. RL4VLA similarly evaluates against only a few baselines. This is particularly problematic because some of the largest improvements over baselines are reported on these benchmarks, making a more comprehensive comparison important. Likewise, the real-robot experiments only compare against PPO, which does not explicitly train for visual robustness. This is an unfair comparison against StyleCraft, which specifically optimizes for this setting, as do other baselines that are omitted from this experiment.
  5. The paper does not compare StyleCraft against modern pretrained image encoders. While the paper evaluates against several approaches for training visually robust RL agents from 2019–2021, more recent and successful techniques for learning visual control policies have emerged and should be considered. A strong baseline would be a policy using a pretrained visual encoder, either frozen or fine-tuned. Given the success of vision-language-action models that leverage pretrained encoders to achieve strong visual robustness, it is unclear whether task-specific visual augmentation training is necessary.
  6. Table 1 reports the best evaluation performance achieved over the course of training, which provides an overly optimistic estimate. While this evaluation procedure is shared across methods, it can obscure their true relative performance. Instead, the paper should either present evaluation curves, as in Figure 4, or use a consistent checkpoint selection strategy with separate validation and test sets.
Questions for authors

See the weaknesses section.

2 comments

Thank you for your review and feedback. We address your questions point by point in the appendix of the revised paper, where we clarify the experimental protocols and provide the relevant supporting results.

Thank you for the response. I still believe many of my stated weaknesses remain unaddressed, so I maintain my original score. (1) requires significant additional empirical comparisons beyond what is provided in the rebuttal. I remain concerned about (6), despite this being the KAGE-Bench evaluation protocol. I can accept the justification for (5), but encourage the authors to include some discussion of this in the main paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.