acceptodds
Under review as a conference paper at ICLR 2027

Understanding Robustness to Observation Corruptions through Actor-Critic Information Specialization

Abstract

Observation corruptions can disrupt reinforcement learning (RL) by altering an agent's inputs and subsequent decisions. However, how Actor-Critic architecture shapes the resulting information and output responses remains poorly understood. We study shared and decoupled PPO trained on clean observations across Procgen, DMC, and Robust Gymnasium. Our framework combines information-theoretic decompositions, representation-to-output sensitivity, and performance-difference relations. Paired diagnostics and closed-loop evaluations address four questions concerning control performance, information specialization, output sensitivity, and responses across corruption types. Decoupled PPO achieves higher corrupted returns in most evaluated tasks, while return-related specialization exhibits greater proportional erosion than action-related specialization. The results distinguish information retention from feature and output stability, supporting a joint assessment of robustness that accounts for corruption type. Our code is included in the supplementary material and will be released on GitHub.

Primary area:
reinforcement learning
Keywords:
Visual Reinforcement Learning, Actor–Critic Representations, Observation Corruptions, Representation Analysis, Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aRreOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an analysis on how observation corruptions (noise, blur, missing information, etc.) affect reinforcement learning agents, and whether having (or not) separate networks for the actor and the critic makes them more robust. The authors compare two PPO architectures: shared-PPO (shared Actor-Critic encoder) and Decoupled-PPO (separate encoders) across 18 tasks (Procgen, DMC and Robust Gymnasium). Both are trained on clean observations and evaluated on corrupted ones. The paper investigates the questions of performance improvement, information specialization, representation sensitivity and robustness under this framework. It also provides an information-theoretic analysis relating information loss, representation changes and policy performance. The paper shows that decoupled PPO gets higher returns under corruption on 14 out of 18 tasks, that return-related specialization is more affected than action-related specialization, and that representation drift alone doesn't explain the changes in policy outputs or performance.

Critical strengths
  • Exhaustive empirical analysis. The paper evaluates shared and decoupled PPO across 18 environments that covers both visual and state-based observations. It also considers multiple corruption types and proposes additional experiments with PPG and DrAC.
  • Robustness analysis: Rather than focusing only on final returns, the paper also investigates how corruption affects information specialization, representations and network outputs. This makes the proposed analysis more complete and the finding that information retention and control performance do not necessarily need the same architecture is interesting and novel to me.
  • The paper also provides theoretical analysis connecting information loss, representation changes and policy behavior. The derivations seems sound to me.
Critical weaknesses
  • Difficulty of isolating if decoupling actually improves robustness. The authors report that decoupled PPO achieves higher corrupted returns on 14/18 tasks. But, decoupled PPO often already outperforms shared PPO on clean observations (see table 3). Also, the decoupled architecture uses two separate encoders, while shared PPO only uses one. This introduces a difference in terms of parameters count as well as capacity that is not controlled, nor checked in the experiments. Thus, it is unclear to me whether the higher corrupted returns come from the better robustness (due to representation coupling, greater model capacity) or simply because it had a stronger policy before corruption.
  • The conclusion are somehow mostly descriptive. The paper provides interesting observations about information specialization under corruption but it is unclear to me how robust these findings are to the mutual information estimation procedure (in particular the use of 8 dimensional PCA projections). Also the theoretical analysis doesn't directly explain the observed differences between shared and decoupled PPO.
Questions for authors

1 - Since decoupled PPO uses two separate encoders, how would the results change when comparing against a shared architecture with a matched parameter count? Could the authors also compare policies with similar clean performance?

2 - Given that the critic is not used for action selection at inference, does return-related specialisation provide any additional predictive information about robustness beyond actor-side metrics like policy KL ? This would help clarify why the authors use the critic information analysis.

3 - How sensitive are the mutual information results to the PCA dimensionality and the choice of the estimator?

4 - Does the measured information-specialization decline actually correlate with performance drop across individual tasks and corruption types? this would help clarify whether the proposed information-theoretic analysis provides useful insights into robustness beyond descriptive observations.

5 - How sensitive are the conclusion to more or less drastic observation corruptions? The experiments use fixed severity settings, and it would be interesting to see whether the relative advantages of shared and decoupled PPO remain consistent under weaker or stronger corruptions.

5 comments
AuthorsOpenReview ↗

Response to aRre

Thank you for recognizing the value of our analysis beyond final returns and for your helpful questions about robustness. We have uploaded a revised manuscript and would appreciate your review of the additions below.

Q1 / W1. Encoder design, capacity, and clean performance.

Matching encoder capacity, total parameter count, and clean-policy quality simultaneously is challenging. Our main comparison therefore fixes encoder design and size within each benchmark: one encoder for both heads in Shared PPO and two independent encoders of the same size in Decoupled PPO. This keeps per-encoder capacity and feature dimensionality consistent for specialization and feature analyses (Table 4).

Sec. 5.2, "Scaling Shared and Decoupled PPO," clarifies this design and adds Procgen comparisons at approximately 0.62M, 1.25M, and 2.50M active parameters by varying CNN widths, with 256-dimensional features, three seeds, and 24.99M environment steps. Decoupled PPO has higher task-averaged normalized corrupted return at each budget (Fig. 2a; App. C.1.2; Table 28).

We pair final checkpoints by task, budget, and seed, selecting solely by similar clean returns (Table 29). At 5%/10% symmetric clean-gap tolerances, 5/45 and 10/45 pairs qualify; median corrupted-return gains are 1.40%/2.65%, while mean retention differences are +1.62/-1.00 percentage points.

Q2. Why analyze Critic information?

During PPO training, Critic estimates enter GAE for Actor updates; Shared PPO also updates its encoder through value-loss gradients (Table 4). Our analysis examines the resulting representations: Sec. 5.3 (Fig. 4) shows greater proportional erosion of return than action specialization across the three benchmark averages. In Procgen, the decoupled Critic has greater feature drift but smaller value drift (Sec. 5.4; Table 2), consistent with the head-weighted mapping in Sec. 4.3. During frozen evaluation, Critic MI therefore characterizes the sensitivity of learned return-related information alongside feature changes, output changes, and policy KL.

In the small two-task study in Sec. 5.5, adding return-specialization loss to output and Actor-MI metrics improves held-out-corruption return-loss prediction hit rates by 2.08 percentage points on both tasks (error tolerance: 10 points). Results vary when both type and seed are held out, so we view this as preliminary evidence. Given the limited rebuttal timeline, future work will include broader evaluations, further theoretical derivations, and targeted representation interventions to clarify specialization's predictive and causal roles.

Q3 / W2. MI sensitivity and estimation.

Sec. 5.7, Fig. 6, and App. C.6 vary PCA dimension d={4,8,12,16} and KSG nearest-neighbor count k={3,5,7} across four DMC tasks. Return-specialization erosion exceeds action-specialization erosion in all 12 macro-average point estimates. At (d,k)=(8,5), 20,000 seed-bootstrap resamples give a 95% interval of [8.50,60.83] percentage points for return-minus-action proportional erosion.

We calibrate MI using ten within-collector target shuffles per condition. After subtracting the shuffled-target baselines, projected Critic return MI is 0.19092 nats on clean inputs and 0.16085 nats on corrupted inputs, supporting the observed information loss. Procgen uses KSG for continuous returns and Ross for discrete actions. The added DMC tests vary PCA dimensions and KSG's nearest-neighbor count to assess the stability of the specialization findings.

Q4 / W2. Specialization changes and return loss.

Fig. 5 aligns specialization changes and return retention across 16 Procgen corruption categories. Across these task-aggregated categories, greater return-specialization erosion is associated with greater proportional return loss (Spearman rho=0.791, computed from the reported values). The four-channel MI analysis identifies each branch's contribution to changes in the specialization gap.

Secs. 4.2-4.4 characterize information changes, map feature displacements through trained heads to outputs, and relate policy changes to returns via the performance-difference identity. Secs. 5.3-5.5 examine these levels empirically. In Procgen, Decoupled PPO has smaller policy KL and fewer action flips, while Shared PPO retains proportionally more return information (Fig. 4; Table 2). These analyses connect specialization to control robustness and explain why information retention, output stability, and return should be assessed jointly.

Q5. Corruption severity.

Sec. 5.2, "Performance across corruption severities," evaluates all 16 visual corruptions at levels 1-5 (App. C.1.3; Table 30). Decoupled PPO has higher corruption-averaged return at every level on all five Procgen and four DMC tasks. The additional experiments show an approximately linear overall decline in performance as severity increases. We therefore use severity 3 as a common intermediate setting for the paired information, feature, and output analyses.

I would first like to thank the authors for the detailed response and the work done during the rebuttal! A big part of my questions have been answered with new experiments. Please find some detailed comments below:

Addressed concerns

  • Q3: Sensitivity of the MI estimates: Convincing sweeps for both PCA dimensionality and KSG neighborhood. thanks for that, it clarifies my concerns.
  • Q5: Corruption severity: Convincing sweep showing that decoupled PPO gets higher return at every severity level. Maybe you can report it next to Table 30, as at severity 5, retention is mixed.
  • Q2 / Q4: Predictive value of information specialization: The gain from adding return specialization loss is quite small (+2.08 pp, i.e., about 2 out of 96 predictions), and the correlation for specialization erosion (ρ\rho = 0.791) is similar to the one for policy KL and action flips (0.70–0.82). So for now, to me, specialization describes rather than adds predictive power over output-level metrics. I think stating this explicitly in the paper would be good. Also, the ρ\rho = 0.791 value is not in the revised PDF (or maybe I missed it), so please include it if that's not the case

And I appreciate these clarifications and additional experiments. However, several issues and question still remain to me.

Remaining points

  • Q1: Capacity and clean performance: The parameter-matched comparison is a nice addition thank you for that. I have another comment after going through this new part of the appendix. App. C.1.2 states that Shared PPO uses three joint epochs per batch, while Decoupled PPO uses one Actor epoch and nine Critic epochs. So the number of gradient updates per batch is not matched, could the authors comment on this?
  • Clean-matched pairs (Table 29): The one brings to me the most concern. When looking at the table, the retention difference is close to 0 across all tolerances level. To me, it suggests that both architectures lose a similar fraction of performance under corruption. When you match the clean returns, the gain in corrupted return is small (<3%) and we observe larger gains only when the clean performance increase (around 11%) In my opinion, the current results do not clearly show that decoupling improves robustness beyond having a stronger clean policy. This limitation should be added and acknowledge and the claims reframed in light of these new results (I think you should focus on higher corrupted returns rather than robustness gains from decoupling itself).
  • W2: Theory: I don't think my point as been fully addressed during the rebuttal. Adding a short discussion of what Section 4.2-4.4 predicts or not about the architecture would help clarifying this.

Conclusion

I have also read the other reviews and the authors' responses to them, and I did not see anything that would change my assessment. I appreciate the effort done during the rebuttal to add new experiments, but given Table 29 results as well as the other remaining points (see above), the new controls do not show that decoupling improves robustness beyond a stronger clean policy. So I keep my current score.

AuthorsOpenReview ↗

Response to aRre

Thank you for the follow-up. Your comments helped us clarify what the capacity and clean-return comparisons establish, and where the analytical framework offers a concrete explanation of the observed responses.

1.Decoupled PPO achieves higher corrupted return in the large majority of the parameter-matched comparisons: 39 of 45 checkpoint pairs (87%) and 13 of 15 task–budget combinations after averaging over seeds. The advantage spans all three model sizes, with Decoupled PPO leading on 5/5, 4/5, and 4/5 tasks, respectively. Thus, at comparable model size, the evaluated decoupled configuration more often learns policies that deliver higher return under observation corruption.

The 5% and 10% clean-return tolerances in Table 29 examine a narrower question and cover only 5/45 (11%) and 10/45 (22%) of the full comparison. Even in these small subsets, Decoupled PPO has positive median corrupted-return gains of 1.40% and 2.65%. With wider tolerances of 15% and 20%, it leads in 12/16 and 14/19 pairs, with median corrupted-return gains of 11.22% and 11.86%. We will clarify in Sec. 5.2 and the conclusion that higher attained corrupted return is the predominant outcome across the matched-capacity comparison, while the two tightest clean-return subsets show similar proportional retention (+1.62 and −1.00 percentage points).

2.The original Procgen schedules follow Garcin et al. (ICLR 2025, Table 5), with three joint epochs for Shared PPO and one Actor epoch plus nine Critic epochs for Decoupled PPO. The reference comparison holds encoder design, width per encoder, and representation dimension fixed. We tested sensitivity to the update schedule on CoinRun at approximately 1.25M parameters. With three passes per loss in both architectures, Decoupled PPO attains a higher mean corrupted return on CoinRun across two paired seeds (6.42 vs. 5.18), retains more of its clean return (90.40% vs. 81.55%), and leads on 14 of 16 corruption types.

3.We will clarify what Sections 4.2–4.4 imply for the architecture comparison. Under the assumptions of Sec. 4.2, the observation-channel loss is common to both architectures when evaluated on the same paired observations and targets. The remaining information-retention terms depend on the learned encoder. Sharing supplies the same encoder output to both heads, whereas decoupling allows branch-specific responses. Given a trained head and a feature displacement, Sec. 4.3 determines the output change. Sec. 4.4 connects changes in the executed policy to return through action advantages and state visitation. The closed-loop return ordering therefore depends on the learned policies and the task.

To explain the Critic response in Sec. 5.4, we decomposed mean absolute value drift on 30 original Procgen checkpoints into the value-head norm and the mean absolute feature displacement along its unit readout direction. On all five tasks, Decoupled PPO has a larger mean head norm and a smaller mean readout-direction displacement. Across tasks, these factors are 3.050 vs. 1.110 and 0.219 vs. 1.323, respectively, for Decoupled and Shared PPO. The smaller readout-direction displacement dominates in the paired analysis, yielding lower mean value drift (0.537 vs. 1.099). This explains how the larger normalized Critic feature drift reported in Sec. 5.4 can coexist with a smaller change in value output.

Overall, we would appreciate it if you could reconsider your assessment in light of the broader matched-capacity evidence, the new update-schedule comparisons, and the more concrete explanation of the representation-to-output response.

Reviewer_K7UcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies observation-corruption robustness in shared and decoupled Actor-Critic architectures. The authors compare PPO agents across Procgen, DMC, and Robust Gymnasium. The experiments show that Decoupled PPO achieves higher corrupted returns in most tasks, while action- and return-related information respond differently to observation corruptions.

Critical strengths
  1. The experiments cover 18 tasks across three benchmarks with different observation corruptions.
  2. The paper examines both policy performance and internal representations, including information specialization and changes in Actor and Critic outputs.
Critical weaknesses
  1. The experiments do not isolate the effect of Actor-Critic decoupling. Decoupled PPO has more parameters and often achieves much higher clean returns, while Shared PPO frequently shows better relative return retention under corruption. The performance differences cannot be attributed to decoupling without controlling for model capacity and clean performance.
  2. The information analysis does not explain its relationship to robustness. MI is estimated from eight-dimensional PCA projections rather than full representations, and the effect of this projection is not examined. The paper also does not show how the measured information specialization affects control performance.
Questions for authors
  1. Does Decoupled PPO still show an advantage under observation corruptions when model capacity and clean performance are controlled?
  2. How sensitive are the information-specialization results to the PCA projection dimension? What evidence connects the measured specialization changes to the differences in control performance?
3 comments
AuthorsOpenReview ↗

Response to K7Uc

Thank you for recognizing our broad evaluation and analysis of internal representations. We have uploaded a revised manuscript addressing your concerns about fair comparisons and the link between specialization and control, and would appreciate your review of the additions.

Q1 / W1. Encoder design, model capacity, and clean-policy quality.

Our main analyses use the same encoder architecture, per-encoder width, and output dimension within each benchmark. Shared PPO uses one encoder, while Decoupled PPO uses two separately parameterized encoders of the same design. This keeps the feature-extractor structure and representation size consistent when comparing information specialization and feature responses. We retain this configuration for the main analyses and now explain the rationale in Sec. 5.2, “Scaling Shared and Decoupled PPO.”

We also compare Shared and Decoupled PPO with similar total parameter counts. By adjusting convolutional widths, we evaluate approximately 0.62M, 1.25M, and 2.50M active-parameter budgets in Procgen, retaining 256-dimensional representations, three training seeds, and the same 24.99M environment-step budget. Decoupled PPO achieves higher mean normalized corrupted return across Procgen tasks at each matched budget (Fig. 2a; App. C.1.2; Table 28).

We further pair final checkpoints by task, parameter budget, and training seed, then select pairs with similar clean returns. At 5%/10% symmetric clean-return tolerances, 5/45 and 10/45 pairs qualify, with median corrupted-return gains of 1.40%/2.65% (Table 29). These comparisons show modest positive gains among policies with closely matched starting performance.

Mean retention differences in the same 5%/10% subsets are +1.62/-1.00 percentage points. We report both corrupted return and retention to distinguish achieved performance from preservation relative to each policy's clean baseline.

Q2 / W2. PCA sensitivity and the relationship between specialization and control.

Sensitivity of the information measurements. Each PCA projection is fitted on clean data and fixed across clean/corrupted inputs. We use d=8 as a practical compromise between retained feature variation and finite-sample estimation. We now vary d={4,8,12,16} and KSG neighborhood sizes k={3,5,7} on four DMC tasks, holding checkpoints, targets, paired observations, and corruptions fixed (Sec. 5.7; Fig. 6; App. C.6). Return-specialization erosion exceeds action-specialization erosion in all 12 macro-average point estimates. At (d,k)=(8,5), 20,000 seed-bootstrap resamples yield a 95% interval of [8.50,60.83] percentage points for their proportional-erosion difference, entirely above zero. The shared-versus-Critic return-MI retention ordering holds in 10/12 settings. After shuffled-target calibration, Critic return MI still decreases from 0.19092 nats (clean) to 0.16085 nats (corrupted).

Information specialization and performance. Sec. 4.2 separates observation-channel information loss from changes in encoder retention. A smaller return-specialization gap can reflect a Critic return-MI decrease or an Actor return-MI increase. Sec. 5.3 and Fig. 4 show that shared representations retain a larger fraction of projected return-related MI than the decoupled Critic, although Decoupled PPO achieves higher corrupted return in most tasks. This difference motivates examining how the trained heads respond to the retained representations.

Feature changes and output responses. Sec. 4.3 analyzes the original, unprojected final-head features and derives exact mappings from their displacement to logit/value changes. Output changes depend on displacement magnitude, direction, and the trained head's weights. Consistent with this analysis, Decoupled PPO shows larger Critic feature drift but smaller value-output drift, together with lower policy KL and action-flip rates (Sec. 5.4; Table 2). Thus, feature stability and output stability can differ.

Output changes and closed-loop return. Sec. 4.4 connects policy changes to return through the performance-difference identity, accounting for action consequences and induced state visitation. Across Procgen corruption types, Spearman correlations with proportional return loss are 0.697/0.809 for policy KL and 0.715/0.818 for action flips (Shared/Decoupled PPO; Sec. 5.5; App. Fig. 17).

Figure 5 further shows that defocus/Gaussian blur leave projected observation–return MI approximately unchanged, yet reverse the mean return-specialization gap and retain only 40.47%/44.24% of clean return. Translation/cutout reduce projected observation–return MI more but retain 75.07%/88.66% of return. These results show why specialization, trained-head responses, and closed-loop performance need to be examined together.

Given the limited rebuttal timeline, we will pursue further theoretical derivations, broader evaluations, and targeted representation interventions to study when specialization changes affect control.

Thank you for the detailed response and the additional experiments. The parameter-matched comparisons address my concern about total model size in Procgen, and the PCA and estimator sensitivity analyses address my concern that the specialization result might depend on the original eight-dimensional projection.

My main remaining concern is whether decoupling improves return retention under observation corruption when clean performance is comparable. In Table 29, only 5 and 10 of the 45 checkpoint pairs meet the 5% and 10% clean-return tolerances, respectively. Decoupled PPO attains slightly higher corrupted returns in both subsets, but the mean difference in return retention changes sign between them. I would state this limitation explicitly when interpreting the robustness benefit of decoupling.

The new prediction analysis also provides preliminary evidence that mutual information measures can help predict return loss beyond output-level metrics. The additional value of Critic information varies when both corruption type and training seed are held out, so how much information specialization explains the performance difference between the two architectures remains open.

Considering the points above, I maintain my original rating.

AuthorsOpenReview ↗

Response to K7Uc

Thank you for the follow-up. We appreciate your assessment that the parameter-matched comparisons and PCA/KSG sweeps address your earlier concerns. We will clarify the scope of the performance and information-analysis claims as follows.

1.Across the full parameter-matched Procgen comparison, Decoupled PPO attains higher corrupted return in 39 of 45 checkpoint pairs and 13 of 15 task–budget combinations after averaging over seeds. It leads on 5/5, 4/5, and 4/5 tasks at the three respective budgets. This is a practical advantage at comparable model size. Across a range of tasks and budgets, the evaluated decoupled configuration more often learns policies that achieve higher return under observation corruption.

2.The 5% and 10% clean-return tolerances in Table 29 cover only 5/45 (11%) and 10/45 (22%) of the same checkpoint pairs. Even in these restricted subsets, the median corrupted-return gains are positive (+1.40% and +2.65%). At the 15% and 20% tolerances, Decoupled PPO attains higher corrupted return in 12/16 and 14/19 pairs, with median gains of +11.22% and +11.86%. As you point out, the mean retention differences at the two tightest tolerances are small. We will state both findings explicitly in Sec. 5.2 and the conclusion. The full comparison shows a broad advantage in attained corrupted return, and the close-clean subsets show similar proportional retention when the starting returns are close.

3.Adding MI measures improves held-out return-loss prediction on CoinRun and StarPilot. The additional contribution of Critic information varies by task and holdout, so we view it as preliminary evidence for a supplementary diagnostic. We also analyzed the original frozen Critic checkpoints across all five Procgen tasks. Decoupled PPO has a larger mean value-head weight norm (3.050 vs. 1.110) and smaller mean feature displacement along the value readout direction (0.219 vs. 1.323). Both patterns occur on all five tasks. The smaller displacement accounts for the lower mean value-output drift in the paired analysis (0.537 vs. 1.099). This identifies a concrete difference in how the trained networks translate feature changes into outputs.

We would appreciate it if you could reconsider your assessment in light of these additional results and the clarified scope of our conclusions.

Reviewer_tkzBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how sharing or separating the actor and critic networks affects reinforcement-learning agents when their observations are corrupted. The authors train PPO agents on clean inputs and test them on visual and state-vector changes across Procgen, DeepMind Control, and Robust Gymnasium. They examine task performance, information about reference actions and returns, changes in internal features, and changes in the networks' outputs. The results show that separate actor and critic encoders achieve higher corrupted returns in 14 of 18 tasks, although retaining a larger fraction of clean performance does not consistently favor separation. The study also finds that corruption tends to reduce the critic's relative advantage in representing returns more than the actor's relative advantage in representing actions. The main contribution is an empirical analysis showing why information retention, feature stability, and actual control performance should not be treated as the same measure of robustness.

Critical strengths
  1. The paper addresses a useful question and makes several important distinctions. A representation can change considerably without changing the chosen action, and retaining information does not necessarily mean that the trained output head can use it reliably. The discussion of the output heads gives a concrete explanation for why measuring feature change alone is insufficient.

  2. The evaluation covers both image-based and state-vector tasks, uses three training seeds, and examines several types of corruption. I appreciate the combination of paired-input measurements and actual environment rollouts: the former isolates the network's immediate response, while the latter captures the consequences of changed actions. The PPG and DrAC comparisons also provide useful context beyond the basic PPO architecture comparison.

  3. The authors generally avoid overstating the measurements. They distinguish absolute corrupted return from the fraction of clean return retained, explain that a shrinking actor-critic information gap can result from changes in either branch, and acknowledge that information measured after reducing features is not the information in the full representation. The corruption-specific examples in Figure 6 are helpful because they show that an average result can hide quite different responses.

Critical weaknesses
  1. The separation between corruption robustness and clean-policy quality and model capacity. Section 5.2 acknowledges that proportional retention can favor sharing, which is important. To support a stronger conclusion about architecture, I would like to see a parameter-matched shared baseline and comparisons between checkpoints with similar clean performance.

  2. The corruption settings cover several types but a limited range within each type. The visual experiments use one severity per corruption. In the state-vector experiments, the Gaussian condition adds the same sampled scalar to every coordinate, and the other conditions add constant offsets. If the authors can provide any results on severity sweep and independently perturbed, scale-normalized vector coordinates, which would help establish whether the main patterns persist beyond the selected settings.

  3. Some specialization gaps in Table 28 are small, so estimation error and the choice of retained dimensions could materially affect the conclusions. It would be beneficial to vary the feature dimension and estimator settings, report uncertainty for the information gaps, and include a shuffled-target check.

Questions for authors

The authors mention that the observed associations as non-causal. I am wondering whether adding the information measures improves prediction of performance loss on held-out tasks or corruption types beyond using output changes alone.

2 comments
AuthorsOpenReview ↗

Response to tkzB

Thank you for these helpful suggestions, which have improved the clarity and empirical support of our study. We have uploaded a revised manuscript and would be grateful if you could review the manuscript.

Q1: Predictive value of information measures and causality.

Following your suggestion, we conducted a small-scale exploratory study on CoinRun and StarPilot (Sec. 5.5). Task-specific ridge models predict proportional return loss using clean return, architecture, policy KL, action flips, and value drift. Adding action-MI loss in the policy representation and return-MI loss in the value representation raises the fraction of predictions within 10 percentage points of the observed loss from 48.96% to 55.21% and from 41.67% to 59.38%, respectively, under held-out corruption types. Holding out both corruption type and training seed gives 48.96% to 54.17% and 36.46% to 53.13%. The absolute hit rates remain around 53–59%, so we view the predictive evidence as preliminary.

These preliminary results complement our analysis of information changes, final-head feature-to-output mappings, and policy-to-return relations (Secs. 4.2–4.4), examined empirically in Secs. 5.3–5.5. Given the limited rebuttal timeline, future work will combine further theoretical derivations, broader held-out evaluations, and targeted representation interventions to clarify specialization's predictive and causal roles.

W1: Fair comparison, model capacity, and clean-policy quality.

Fair comparison across architectures, tasks, and corruptions requires specifying what is held fixed. Our original comparison uses the same encoder architecture and per-encoder width within each benchmark. This controls per-encoder capacity and architectural inductive bias when examining shared and separate representations. This rationale was insufficiently explicit and is now clarified in Sec. 5.2, “Scaling Shared and Decoupled PPO.”

We adjust convolutional widths to match active parameter counts at approximately 0.62M, 1.25M, and 2.50M, retaining a 256-dimensional representation and the same 24.99M environment-step budget. Decoupled PPO achieves higher mean normalized corrupted return across Procgen tasks at each budget (Fig. 2a; App. C.1.2; Table 28).

To further account for clean-policy quality, we pair final checkpoints by task, parameter budget, and training seed, then select pairs with similar clean returns. At 5%/10% symmetric clean-return tolerances, 5/45 and 10/45 pairs qualify, with median corrupted-return gains of 1.40%/2.65% (Table 29). This complements parameter matching by comparing policies with similar starting performance. We report attained return and proportional retention separately throughout.

W2. Corruption severity and independent vector perturbations.

Sec. 5.2, “Performance across corruption severities,” adds levels 1–5 for all 16 visual corruptions (Fig. 2b; Table 30). The results show roughly linear overall declines in corruption-averaged return. Decoupled PPO maintains higher corruption-averaged return on all nine visual tasks at every level.

For vectors, we add independent Gaussian noise across coordinates and timesteps, with SD=10% of each coordinate's clean SD and a common scale for both architectures before frozen normalization. Decoupled PPO attains higher corrupted return on 6/9 tasks (Table 31). At marginal SD=0.005, scalar-broadcast and independent noise yield the same architectural ranking on all nine tasks (Table 32).

For subsequent analyses, we retain visual severity 3 as an intermediate reference and the three vector conditions: scalar Gaussian noise (SD=0.005) and constant shifts (+0.005, +0.4), covering random fluctuations and systematic biases.

W3. Reliability of the information estimates.

We added PCA/KSG sensitivity, uncertainty estimates, and shuffled-target checks (Sec. 5.7; Fig. 6; App. C.6). Across four DMC tasks, we vary d={4,8,12,16} and k={3,5,7}. Return-specialization erosion exceeds action-specialization erosion in all 12 macro-average point estimates.

At the main setting (d,k)=(8,5), 20,000 seed-bootstrap resamples yield 95% intervals of [8.50,60.83] percentage points for return-minus-action proportional erosion and [0.00606,0.01828] nats for the return-gap decrease, both entirely above zero. For each channel and condition, we subtract the mean MI estimated from ten within-collector target shuffles. Calibrated Critic return MI still decreases from 0.19092 nats (clean) to 0.16085 nats (corrupted). Together, these checks strengthen the evidence for asymmetric specialization erosion and show that the Critic return-information loss persists after calibration.

I thank the authors for the efforts on the rebuttal. I will keep the current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.