- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an analysis on how observation corruptions (noise, blur, missing information, etc.) affect reinforcement learning agents, and whether having (or not) separate networks for the actor and the critic makes them more robust. The authors compare two PPO architectures: shared-PPO (shared Actor-Critic encoder) and Decoupled-PPO (separate encoders) across 18 tasks (Procgen, DMC and Robust Gymnasium). Both are trained on clean observations and evaluated on corrupted ones. The paper investigates the questions of performance improvement, information specialization, representation sensitivity and robustness under this framework. It also provides an information-theoretic analysis relating information loss, representation changes and policy performance. The paper shows that decoupled PPO gets higher returns under corruption on 14 out of 18 tasks, that return-related specialization is more affected than action-related specialization, and that representation drift alone doesn't explain the changes in policy outputs or performance.
Critical strengths
- Exhaustive empirical analysis. The paper evaluates shared and decoupled PPO across 18 environments that covers both visual and state-based observations. It also considers multiple corruption types and proposes additional experiments with PPG and DrAC.
- Robustness analysis: Rather than focusing only on final returns, the paper also investigates how corruption affects information specialization, representations and network outputs. This makes the proposed analysis more complete and the finding that information retention and control performance do not necessarily need the same architecture is interesting and novel to me.
- The paper also provides theoretical analysis connecting information loss, representation changes and policy behavior. The derivations seems sound to me.
Critical weaknesses
- Difficulty of isolating if decoupling actually improves robustness. The authors report that decoupled PPO achieves higher corrupted returns on 14/18 tasks. But, decoupled PPO often already outperforms shared PPO on clean observations (see table 3). Also, the decoupled architecture uses two separate encoders, while shared PPO only uses one. This introduces a difference in terms of parameters count as well as capacity that is not controlled, nor checked in the experiments. Thus, it is unclear to me whether the higher corrupted returns come from the better robustness (due to representation coupling, greater model capacity) or simply because it had a stronger policy before corruption.
- The conclusion are somehow mostly descriptive. The paper provides interesting observations about information specialization under corruption but it is unclear to me how robust these findings are to the mutual information estimation procedure (in particular the use of 8 dimensional PCA projections). Also the theoretical analysis doesn't directly explain the observed differences between shared and decoupled PPO.
Questions for authors
1 - Since decoupled PPO uses two separate encoders, how would the results change when comparing against a shared architecture with a matched parameter count? Could the authors also compare policies with similar clean performance?
2 - Given that the critic is not used for action selection at inference, does return-related specialisation provide any additional predictive information about robustness beyond actor-side metrics like policy KL ? This would help clarify why the authors use the critic information analysis.
3 - How sensitive are the mutual information results to the PCA dimensionality and the choice of the estimator?
4 - Does the measured information-specialization decline actually correlate with performance drop across individual tasks and corruption types? this would help clarify whether the proposed information-theoretic analysis provides useful insights into robustness beyond descriptive observations.
5 - How sensitive are the conclusion to more or less drastic observation corruptions? The experiments use fixed severity settings, and it would be interesting to see whether the relative advantages of shared and decoupled PPO remain consistent under weaker or stronger corruptions.
5 comments
I would first like to thank the authors for the detailed response and the work done during the rebuttal! A big part of my questions have been answered with new experiments. Please find some detailed comments below:
Addressed concerns
- Q3: Sensitivity of the MI estimates: Convincing sweeps for both PCA dimensionality and KSG neighborhood. thanks for that, it clarifies my concerns.
- Q5: Corruption severity: Convincing sweep showing that decoupled PPO gets higher return at every severity level. Maybe you can report it next to Table 30, as at severity 5, retention is mixed.
- Q2 / Q4: Predictive value of information specialization: The gain from adding return specialization loss is quite small (+2.08 pp, i.e., about 2 out of 96 predictions), and the correlation for specialization erosion ( = 0.791) is similar to the one for policy KL and action flips (0.70–0.82). So for now, to me, specialization describes rather than adds predictive power over output-level metrics. I think stating this explicitly in the paper would be good. Also, the = 0.791 value is not in the revised PDF (or maybe I missed it), so please include it if that's not the case
And I appreciate these clarifications and additional experiments. However, several issues and question still remain to me.
Remaining points
- Q1: Capacity and clean performance: The parameter-matched comparison is a nice addition thank you for that. I have another comment after going through this new part of the appendix. App. C.1.2 states that Shared PPO uses three joint epochs per batch, while Decoupled PPO uses one Actor epoch and nine Critic epochs. So the number of gradient updates per batch is not matched, could the authors comment on this?
- Clean-matched pairs (Table 29): The one brings to me the most concern. When looking at the table, the retention difference is close to 0 across all tolerances level. To me, it suggests that both architectures lose a similar fraction of performance under corruption. When you match the clean returns, the gain in corrupted return is small (<3%) and we observe larger gains only when the clean performance increase (around 11%) In my opinion, the current results do not clearly show that decoupling improves robustness beyond having a stronger clean policy. This limitation should be added and acknowledge and the claims reframed in light of these new results (I think you should focus on higher corrupted returns rather than robustness gains from decoupling itself).
- W2: Theory: I don't think my point as been fully addressed during the rebuttal. Adding a short discussion of what Section 4.2-4.4 predicts or not about the architecture would help clarifying this.
Conclusion
I have also read the other reviews and the authors' responses to them, and I did not see anything that would change my assessment. I appreciate the effort done during the rebuttal to add new experiments, but given Table 29 results as well as the other remaining points (see above), the new controls do not show that decoupling improves robustness beyond a stronger clean policy. So I keep my current score.
Response to aRre
Thank you for the follow-up. Your comments helped us clarify what the capacity and clean-return comparisons establish, and where the analytical framework offers a concrete explanation of the observed responses.
1.Decoupled PPO achieves higher corrupted return in the large majority of the parameter-matched comparisons: 39 of 45 checkpoint pairs (87%) and 13 of 15 task–budget combinations after averaging over seeds. The advantage spans all three model sizes, with Decoupled PPO leading on 5/5, 4/5, and 4/5 tasks, respectively. Thus, at comparable model size, the evaluated decoupled configuration more often learns policies that deliver higher return under observation corruption.
The 5% and 10% clean-return tolerances in Table 29 examine a narrower question and cover only 5/45 (11%) and 10/45 (22%) of the full comparison. Even in these small subsets, Decoupled PPO has positive median corrupted-return gains of 1.40% and 2.65%. With wider tolerances of 15% and 20%, it leads in 12/16 and 14/19 pairs, with median corrupted-return gains of 11.22% and 11.86%. We will clarify in Sec. 5.2 and the conclusion that higher attained corrupted return is the predominant outcome across the matched-capacity comparison, while the two tightest clean-return subsets show similar proportional retention (+1.62 and −1.00 percentage points).
2.The original Procgen schedules follow Garcin et al. (ICLR 2025, Table 5), with three joint epochs for Shared PPO and one Actor epoch plus nine Critic epochs for Decoupled PPO. The reference comparison holds encoder design, width per encoder, and representation dimension fixed. We tested sensitivity to the update schedule on CoinRun at approximately 1.25M parameters. With three passes per loss in both architectures, Decoupled PPO attains a higher mean corrupted return on CoinRun across two paired seeds (6.42 vs. 5.18), retains more of its clean return (90.40% vs. 81.55%), and leads on 14 of 16 corruption types.
3.We will clarify what Sections 4.2–4.4 imply for the architecture comparison. Under the assumptions of Sec. 4.2, the observation-channel loss is common to both architectures when evaluated on the same paired observations and targets. The remaining information-retention terms depend on the learned encoder. Sharing supplies the same encoder output to both heads, whereas decoupling allows branch-specific responses. Given a trained head and a feature displacement, Sec. 4.3 determines the output change. Sec. 4.4 connects changes in the executed policy to return through action advantages and state visitation. The closed-loop return ordering therefore depends on the learned policies and the task.
To explain the Critic response in Sec. 5.4, we decomposed mean absolute value drift on 30 original Procgen checkpoints into the value-head norm and the mean absolute feature displacement along its unit readout direction. On all five tasks, Decoupled PPO has a larger mean head norm and a smaller mean readout-direction displacement. Across tasks, these factors are 3.050 vs. 1.110 and 0.219 vs. 1.323, respectively, for Decoupled and Shared PPO. The smaller readout-direction displacement dominates in the paired analysis, yielding lower mean value drift (0.537 vs. 1.099). This explains how the larger normalized Critic feature drift reported in Sec. 5.4 can coexist with a smaller change in value output.
Overall, we would appreciate it if you could reconsider your assessment in light of the broader matched-capacity evidence, the new update-schedule comparisons, and the more concrete explanation of the representation-to-output response.
Response to aRre
Thank you for recognizing the value of our analysis beyond final returns and for your helpful questions about robustness. We have uploaded a revised manuscript and would appreciate your review of the additions below.
Q1 / W1. Encoder design, capacity, and clean performance.
Matching encoder capacity, total parameter count, and clean-policy quality simultaneously is challenging. Our main comparison therefore fixes encoder design and size within each benchmark: one encoder for both heads in Shared PPO and two independent encoders of the same size in Decoupled PPO. This keeps per-encoder capacity and feature dimensionality consistent for specialization and feature analyses (Table 4).
Sec. 5.2, "Scaling Shared and Decoupled PPO," clarifies this design and adds Procgen comparisons at approximately 0.62M, 1.25M, and 2.50M active parameters by varying CNN widths, with 256-dimensional features, three seeds, and 24.99M environment steps. Decoupled PPO has higher task-averaged normalized corrupted return at each budget (Fig. 2a; App. C.1.2; Table 28).
We pair final checkpoints by task, budget, and seed, selecting solely by similar clean returns (Table 29). At 5%/10% symmetric clean-gap tolerances, 5/45 and 10/45 pairs qualify; median corrupted-return gains are 1.40%/2.65%, while mean retention differences are +1.62/-1.00 percentage points.
Q2. Why analyze Critic information?
During PPO training, Critic estimates enter GAE for Actor updates; Shared PPO also updates its encoder through value-loss gradients (Table 4). Our analysis examines the resulting representations: Sec. 5.3 (Fig. 4) shows greater proportional erosion of return than action specialization across the three benchmark averages. In Procgen, the decoupled Critic has greater feature drift but smaller value drift (Sec. 5.4; Table 2), consistent with the head-weighted mapping in Sec. 4.3. During frozen evaluation, Critic MI therefore characterizes the sensitivity of learned return-related information alongside feature changes, output changes, and policy KL.
In the small two-task study in Sec. 5.5, adding return-specialization loss to output and Actor-MI metrics improves held-out-corruption return-loss prediction hit rates by 2.08 percentage points on both tasks (error tolerance: 10 points). Results vary when both type and seed are held out, so we view this as preliminary evidence. Given the limited rebuttal timeline, future work will include broader evaluations, further theoretical derivations, and targeted representation interventions to clarify specialization's predictive and causal roles.
Q3 / W2. MI sensitivity and estimation.
Sec. 5.7, Fig. 6, and App. C.6 vary PCA dimension d={4,8,12,16} and KSG nearest-neighbor count k={3,5,7} across four DMC tasks. Return-specialization erosion exceeds action-specialization erosion in all 12 macro-average point estimates. At (d,k)=(8,5), 20,000 seed-bootstrap resamples give a 95% interval of [8.50,60.83] percentage points for return-minus-action proportional erosion.
We calibrate MI using ten within-collector target shuffles per condition. After subtracting the shuffled-target baselines, projected Critic return MI is 0.19092 nats on clean inputs and 0.16085 nats on corrupted inputs, supporting the observed information loss. Procgen uses KSG for continuous returns and Ross for discrete actions. The added DMC tests vary PCA dimensions and KSG's nearest-neighbor count to assess the stability of the specialization findings.
Q4 / W2. Specialization changes and return loss.
Fig. 5 aligns specialization changes and return retention across 16 Procgen corruption categories. Across these task-aggregated categories, greater return-specialization erosion is associated with greater proportional return loss (Spearman rho=0.791, computed from the reported values). The four-channel MI analysis identifies each branch's contribution to changes in the specialization gap.
Secs. 4.2-4.4 characterize information changes, map feature displacements through trained heads to outputs, and relate policy changes to returns via the performance-difference identity. Secs. 5.3-5.5 examine these levels empirically. In Procgen, Decoupled PPO has smaller policy KL and fewer action flips, while Shared PPO retains proportionally more return information (Fig. 4; Table 2). These analyses connect specialization to control robustness and explain why information retention, output stability, and return should be assessed jointly.
Q5. Corruption severity.
Sec. 5.2, "Performance across corruption severities," evaluates all 16 visual corruptions at levels 1-5 (App. C.1.3; Table 30). Decoupled PPO has higher corruption-averaged return at every level on all five Procgen and four DMC tasks. The additional experiments show an approximately linear overall decline in performance as severity increases. We therefore use severity 3 as a common intermediate setting for the paired information, feature, and output analyses.