acceptodds
Under review as a conference paper at ICLR 2027

Latent Space Perturbations Improve Visual Generalization in Reinforcement Learning

Abstract

Visual generalization in reinforcement learning (RL) means keeping the same task performance when the visuals change, for example the background, lighting, or object appearance, while the dynamics and rewards stay the same. Humans handle such changes well, but RL agents often lose a lot of performance under these visual shifts. Inspired by efficient coding theory from neuroscience, we study how visual shifts change the encoder latent space. We find that encoders trained in fixed visual settings learn a mostly low-dimensional latent space: out-of-distribution (OOD) visuals mainly cause mean and variance shifts along the top principal components of the training latent space, instead of creating new directions. We validate this on our diagnostic environment (LSP-Env) with controlled tests for background changes, distractors, and agent appearance changes, and we also validate this on Procgen. Based on these results, we propose Latent Space Perturbation (LSP), a PCA-guided method that adds variance-scaled Gaussian noise along dominant latent directions during training. Across LSP-Env, Procgen, and RL-ViGen, LSP improves OOD performance without hurting in-distribution performance, and it does so without changing the encoder or using pixel-level augmentation.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, visual generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QNTDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual generalization in reinforcement learning from the perspective of latent representation geometry. The authors observe that visual distribution shifts tend to alter latent representations primarily along the dominant principal components of the training distribution, rather than introducing entirely new directions. Motivated by this finding, they propose Latent Space Perturbation (LSP), which injects variance-scaled Gaussian noise along these principal directions during training to improve policy robustness to visual shifts. Experiments on LSP-Env, five Procgen environments, and two RL-ViGen tasks show consistent improvements in OOD performance. Additional analyses with different encoders and stronger visual shifts further examine when the proposed latent-space assumption holds and where the method begins to lose effectiveness.

Critical strengths
  1. The paper establishes a clear link between its latent-space analysis and the proposed method. The authors show that visual OOD shifts are largely concentrated in the dominant training subspace, and use this observation to motivate PCA-aligned perturbations rather than generic latent noise.
  2. LSP is simple, lightweight, and easy to integrate without modifying the policy architecture. Its gains on both Procgen and RL-ViGen suggest that the method extends beyond the controlled LSP-Env and has practical value across different visual RL settings.
Critical weaknesses
  1. The latent-shift analysis needs a cleaner state-matched control. The training and evaluation latents appear to use different state distributions, so the measured mean and variance shifts may reflect both state changes and visual changes. Since this analysis is the main motivation for LSP, the authors should compare the same underlying states under training and OOD appearances and recompute the key shift statistics.

  2. The experimental setup contains several inconsistencies. For RL-ViGen Door, Table 2 reports 5×1055 \times 10^5 total training steps, while the selected LSP schedule starts at 5×1065 \times 10^6 steps. The LSP-Env experiment is also described as PPO in some places and SAC in others. The authors should clarify the exact configuration used for each reported result and make the settings consistent throughout the paper.

Questions for authors
  1. Can the authors clarify how task states were sampled for the training and evaluation latent sets in Section 5.2? As written, the two sets appear to use different state distributions, which may confound state variation with appearance variation. If so, could the authors recompute the key mean- and variance-shift statistics using the same underlying states rendered under different appearances? This control seems important for validating the main geometric motivation behind LSP.

  2. Can the authors clarify the exact configurations used for Figures 9 and 10? For the Door experiment, Table 2 reports a total horizon of 5×1055 \times 10^5 steps, while the selected LSP schedule activates at 5×1065 \times 10^6 steps. Please clarify the activation time and its unit used in the actual run. Similarly, please confirm whether Figure 9 uses SAC or PPO and correct the inconsistent labels across the main text, figure, and appendix.

3 comments

Dear Reviewer,

Thank you for your careful evaluation and constructive feedback.

We have addressed your concerns through additional experiments and a thorough verification of our experimental configurations.

We kindly ask you to download the updated PDF of our submission, where we have included a dedicated Rebuttal section at the end of the Appendix, containing a detailed response to your comments and the new experimental results.

We believe these additions strengthen the paper and would greatly appreciate it if you could reconsider your assessment based on the updated evidence.

Thank you again for your time and consideration.

Dear Reviewer,

We hope the new evidence addresses your original concerns. If any questions or issues remain, we would be happy to discuss them and provide further clarification.

Thank you for the detailed response and additional experiments. The state-matched analysis substantially addresses my first concern, providing much stronger evidence that visual shifts are concentrated in the dominant latent directions independently of task-state variation.

The clarification of the SAC/PPO configurations and the actual Door training horizon and perturbation onset also addresses my second concern. However, please ensure that these corrections are consistently reflected in the main text and tables, and that the Door results from different schedules are clearly distinguished rather than presented as independent replications.

Overall, my original concerns have been largely addressed, and I appreciate the additional effort.

Reviewer_cGBtOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes how task-independent visual transformation affect the latent representation of the image in RL agents. The main finding is that OOD visual changes are mainly projected along the principal PCA directions of the training features. Thus, it proposes Latent Space Perturbation (LSP), a data augmentation technique that works in the latent space by adding Gaussian noise along the leading principal directions. The method is evaluated on a custom environmnet, as well as Procgen games and two RL-ViGen manipulation tasks.

Critical strengths
  1. The method is simple, doesn't add much training overhead, and it is RL algorithm agnostic.
  2. The reported improvements over the base agents are consistent across the three benchmarks
  3. The authors try to evaluate what type of task-agnostic visual change doesn't respect their assumption (e.g., color inversion), and show that the alignment between training and test subspaces is worse in these cases.
Critical weaknesses
  1. LSP is essentially performing data augmentation in latent space: adding rescaled Gaussian noise along the main directions. However, the paper does not extensively compare it with latent-augmentation methods. The only latent baseline considered is IBAC-SNI, and it appears only on Procgen. It also does not discuss work that makes policies robust to transformations of the latent space, such as Policy Stitching (Jian et al.).

  2. The paper doesn't provide enough evidence to support the claim that the proposed noising scheme is clearly better than the alternatives:

    • The only noise ablation is isotropic, and its magnitude is not rescaled as done in LSP
    • Other missing ablations are: adding noise of with the same magnitude along directions orthogonal to the leading ones, along random ones, and along the lowest-variance ones. All these should perform worse.
    • In general, it does not demonstrate that alignment with the principal directions is responsible for the gains.
  3. All agents use small CNNs trained from scratch with RL. A natural baseline is an agent trained on top of pretrained visual encoder such as DINOv2 or DINOv3 (frozen or fine-tuned). Those may already be resilient to visual changes.

  4. The noise schedule and strength are tuned per task, and it is not clear if directly on the test conditions. Either the baselines should have a similar tuning, or the tuning of these noising schemes should be test-agnostic.

  5. It is unclear why task-irrelevant changes should fall along the leading directions of the training features, and it is shown only for a few types of shift. The paper itself shows some visual variations (e.g., color inversion) do not respect this assumption.

  6. Moreover on the visual changes: being task-irrelevant is defined only through the assumption that dynamics and rewards do not change, and most importantly nothing guarantees benchmarks respect this assumption (e.g., Procgen's held-out levels differ in layout as well as appearance).

Questions for authors
  • How does LSP compare with latent augmentation adding noise of the same rescaled magnitude along directions orthogonal to the leading ones, along the lowest-variance principal directions, random ones or isotropic? How does performance vary with the number of perturbed directions?

  • How does LSP compare with an agent trained on pretrained encoders (e.g., DINOv2 or DINOv3, frozen and fine-tuned)? Does LSP still help on top of one?

  • How were the per-task noise settings selected, and on which data? Were the baselines tuned in the same way?

I am willing to raise my score if: (a) there is more supporting evidence that this noising scheme is clearly better than the alternatives, and (b) using a pre-trained visual encoder is indeed worse or can be improved with this noising scheme.

3 comments

Dear Reviewer,

Thank you for your detailed review and valuable suggestions.

We have taken your concerns seriously and conducted several additional experiments, including new controlled comparisons and ablation studies, to address the questions you raised.

We kindly ask you to download the updated PDF of our submission, where we have included a dedicated Rebuttal section at the end of the Appendix, containing a detailed response to your comments and the new experimental results.

We hope these additional findings address your main concerns and would greatly appreciate your reconsideration of the paper's evaluation.

Thank you again for your time and consideration.

Thank you for the new experiments. The new controls answer my question on the noise directions, and with a frozen DINOv2 encoder LSP seems to help on backgrounds, with no difference on agent appearance. Both experiments are run only on LSP-Env, the paper's diagnostic environment, and do not cover Procgen or the robot tasks. On those benchmarks, the evidence that this noising scheme is clearly better than the alternatives still comes from the main comparisons with the baselines. In my review I asked how the per-task noise settings were selected and whether the baselines were tuned in the same way. The authors state that the settings of LSP were chosen on the evaluation environments, without a separate validation split, but not how the baselines were tuned. Overall, I will keep my score.

Dear Reviewer,

Thank you for your follow-up and for acknowledging that our new experiments address your concerns regarding the choice of perturbation directions.

We would like to clarify two important points about the scope of our work and the fairness of our baseline comparisons.

1. Pretrained visual encoders and the scope of our work.

Our work primarily focuses on improving visual generalization in online RL, where compact visual encoders are trained from scratch jointly with the policy, without relying on large-scale visual pretraining.

This distinction is important. Our CNN encoder contains approximately 0.405M parameters, whereas even the smallest DINOv2 ViT-S/14 backbone contains approximately 21M parameters, making it about 52 times larger. Moreover, DINOv2 benefits from extensive prior visual pretraining, whereas our encoder learns its representations entirely through RL interactions.

Consequently, pretrained foundation encoders represent a substantially different training regime rather than a directly matched alternative to the lightweight CNN setting investigated in our paper.

Nevertheless, we conducted the additional frozen-DINOv2 experiments specifically in response to your suggestion, as we agree that exploring the interaction between LSP and pretrained representations is interesting. We consider these experiments complementary to our main contribution rather than a prerequisite for validating LSP within its intended setting.

We will also make this scope more explicit in the revised manuscript, emphasizing lightweight visual representations learned from scratch in online RL.

2. Baseline implementations and hyperparameter selection.

Regarding your question about baseline tuning, all competing baselines were evaluated using the original implementations provided by their respective authors, together with their published training configurations and hyperparameter settings.

We deliberately followed these established, author-validated training recipes rather than introducing our own modifications or independently retuning competing methods. We consider this a reproducible and appropriate reference for comparison with existing approaches.

We acknowledge that the original LSP configurations were selected using the reported evaluation environments, and we have disclosed this explicitly. Our additional fixed-configuration experiments with disjoint validation and test appearances were introduced to provide complementary evidence without this selection bias.

3. Experimental evidence beyond LSP-Env.

We agree that the new direction-control ablations are conducted in LSP-Env. This is intentional: its controlled appearance transformations allow us to isolate the effect of perturbation direction without confounding changes in task dynamics or level structure.

Our Procgen and RL-ViGen experiments serve a different purpose: evaluating whether the resulting method can improve performance beyond this controlled diagnostic setting.

We believe these two levels of evaluation are complementary: controlled experiments investigate the underlying mechanism, while established benchmarks assess its practical applicability.

Taken together, we believe the new energy-matched ablations, the additional pretrained-encoder comparison, and the broader benchmark results provide substantially stronger evidence for the contribution of LSP within its intended scope.

We would appreciate your reconsideration of the paper in light of these clarifications and the additional experimental evidence.

Reviewer_iNA7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Latent Space Perturbation (LSP), which injects variance-scaled noise along the principal PCA directions of encoder representations during training. Through latent space analysis and experiments on LSP-Env, Procgen, and RL-ViGen, the paper investigates and validates its effectiveness in improving visual generalization.

Critical strengths

The paper proposes a simple training method based on a clearly defined hypothesis about latent space geometry, which can be easily integrated into existing reinforcement learning algorithms. The experiments cover controlled visual variations, procedurally generated games, and robotic manipulation tasks, with results reported across multiple random seeds.

Critical weaknesses

The sampling relationship in the geometric analysis remains unclear. Section 5.2 and Figure 4 describe variations in task states among training samples, whereas the test samples are generated by fixing the task state and varying only the visual appearance. However, it is not explicitly stated whether the two sets of samples are paired based on the same task states. If their state distributions differ, the observed differences in means and variances may reflect both task-state variations and visual changes. Without clarification, it is difficult to attribute these statistical differences entirely to visual out-of-distribution (OOD) shifts, which affects my assessment of the paper’s central empirical findings.

There is an inconsistency in a key configuration of the robotic experiments. Table 2 on page 16 lists the total number of training steps for RL-ViGen (Door) as 10^5, while setting the point at which the perturbation is abruptly introduced to 10^6. According to the definition in Table 3 on page 18, if both values use the same unit of training steps, the perturbation would not be activated before training ends. This may simply be a reporting typo, but the currently reported configuration does not appear consistent with the claimed benefits of LSP. Clarification of the actual experimental setup is needed to reliably assess and reproduce these results.

Questions for authors
  1. In §5.2, are the training and test representations obtained from paired renderings of the same underlying states, with identical state weights? If not, could the authors provide an additional analysis using state-matched pairs to disentangle the effects of visual changes and state distribution shifts on the geometric metrics?

  2. Is the inconsistency between the total number of training steps and the perturbation onset step in RL-ViGen a typographical error, or do these values use different units? Could the authors clarify the actual experimental settings and specify when the perturbation begins to take effect?

3 comments

Dear Reviewer,

Thank you for your thoughtful review and constructive feedback.

We have carefully addressed your concerns and conducted additional experiments to provide stronger evidence for our findings.

We kindly ask you to download the updated PDF of our submission, where we have included a dedicated Rebuttal section at the end of the Appendix, containing a detailed response to your comments and the new experimental results.

We would greatly appreciate it if you could reconsider your assessment in light of this additional evidence.

Thank you again for your time and consideration.

Thank you for your response and clarifications. I have noted the updates.

Dear Reviewer,

Thank you for your acknowledgment.

We would like to emphasize that we have conducted additional experiments specifically to address both of your concerns, including a new state-matched latent analysis and a detailed verification of the RL-ViGen experimental configurations.

Since these additions directly address the issues raised in your review, we would appreciate a reassessment of your original rating in light of the new evidence.

We look forward to your feedback and reconsideration of the score.

Comments on the paper

1 comment

Dear Reviewers,

We would like to sincerely thank all reviewers for their constructive feedback and for the time devoted to evaluating our work.

During the rebuttal period, we made a substantial effort to address the concerns raised in all three reviews, including conducting new experiments specifically designed to answer your questions.

In particular:

  • For iNA7 and QNTD, we conducted a new state-matched latent-space analysis using 2,000 identical simulator states rendered under different visual conditions. This directly addresses the potential confounding between task-state and appearance variation and provides substantially stronger evidence for our central geometric hypothesis. We also audited the RL-ViGen configurations, verified the actual perturbation schedules, and clarified the SAC/PPO inconsistencies.

  • For cGBt, we conducted extensive additional ablations comparing PCA-aligned perturbations against energy- and spectrum-matched alternatives, including random, orthogonal, bottom-PC, and isotropic noise, as well as experiments varying the number of perturbed directions. We also evaluated a pretrained DINOv2 encoder with and without LSP on two visual protocols, clarified our baseline hyperparameter selection procedures, and further explained the scope of our contribution.

All these additional experiments, results, and detailed responses are included in the dedicated Rebuttal section at the end of the updated PDF.

We believe these additions substantially strengthen the experimental foundation of our paper and directly address the principal methodological concerns raised during the review process.

We are particularly grateful that the new state-matched analysis and configuration clarifications have been recognized as addressing the original concerns in the subsequent discussion.

We sincerely hope that the additional evidence will be taken into account in the final assessment of our submission.

If any concerns remain, we would greatly appreciate further feedback and would be happy to provide additional clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.