- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RENDER, a framework for diagnosing and repairing harmful shortcut dependencies in visual control policies after deployment. It uses a few target failure trajectories to identify candidate causes, verifies them through interventions in the source simulator, and trains adaptors while keeping the original policy frozen.
Critical strengths
I find the problem meaningful, and I appreciate the clear separation between causal diagnosis and policy repair. The theoretical motivation is reasonable, and the experimental evaluation covers several different visual control settings, including ForkRoad, Atari, and CARLA. I also appreciate the ablation studies and the evaluation of source performance preservation.
Critical weaknesses
- I think the main limitation is the reliance on a predefined intervention library and a source simulator capable of reproducing candidate deployment failures. I am not sure how well the approach would generalize when the responsible mechanism is unknown or cannot be reproduced in the simulator.
- I also find the real-world validation somewhat limited. The experiments are conducted in simulated environments and mainly involve individual, interpretable nuisance factors. More evidence would be helpful to understand whether the approach remains effective under complex or interacting distribution shifts.
- Each benchmark uses only one frozen base policy, while the five reported seeds mainly vary the repair process. Since shortcut dependencies can differ across independently trained policies, I am not sure whether the current results are sufficient to demonstrate consistent diagnosis and repair across different policy checkpoints.
Questions for authors
- How sensitive is the diagnosis to the coverage and quality of the predefined intervention library?
- Could the authors evaluate scenarios involving multiple simultaneous nuisance changes?
- Could the authors provide additional experiments with independently trained policies?
6 comments
We are very grateful for your insightful and constructive feedback (2/2)
Q3: Could the authors provide additional experiments with independently trained policies?
A3: Thank you for this suggestion. In the paper, Freeway and Breakout use publicly released pixel PPO checkpoints (Appendix C). Since only one trained policy is released per game, each game has a single frozen base policy. To address your concern, we trained three additional Breakout policies (seeds 1 to 3) with the same recipe as the released checkpoint (10M decisions), changing only the training seed. One more run did not learn the game (native return 30.4), so we leave it out.
We then reran our method from scratch on each of the three policies. The results are summarized below in the same format as Figure 5, with standard errors over the 20 evaluation games for Frozen and over five repair seeds for RENDER.
| Method | Source ↑ | Target ↑ | Response (%) ↓ |
|---|---|---|---|
| Released policy | |||
| Frozen | 185.40 ± 19.10 | 55.00 ± 16.43 | 27.9 |
| RENDER | 200.08 ± 4.02 | 125.18 ± 6.67 | 18.4 ± 0.4 |
| Seed 1 | |||
| Frozen | 216.25 ± 20.17 | 34.10 ± 10.22 | 29.2 |
| RENDER | 206.60 ± 12.79 | 141.81 ± 13.10 | 20.0 ± 0.8 |
| Seed 2 | |||
| Frozen | 159.10 ± 27.84 | 4.35 ± 0.92 | 43.7 |
| RENDER | 143.42 ± 9.89 | 73.23 ± 9.21 | 23.3 ± 0.4 |
| Seed 3 | |||
| Frozen | 167.20 ± 20.15 | 87.05 ± 15.65 | 25.6 |
| RENDER | 193.20 ± 8.57 | 126.52 ± 9.52 | 18.0 ± 0.3 |
The results are consistent across policies. For all three seeds, our method identifies the same mechanism as for the released policy, namely the paddle and ball color, and Step 3 confirms that it harms the task. The repair then raises the return under the red paddle and lowers the response, while the native return stays within episode noise. In addition, the action delay, which lies outside the library, is rejected for every policy, and no wrong mechanism is accepted. We have added these results to the updated manuscript according to your great suggestion (Section 4.2, L405-407; Appendix D.12, L1869-1921). Please let us know if anything is still not fully clear.
Rebuttal Follow-up
Thanks again for your insightful and constructive feedback. We have submitted our rebuttal with new experiments and expanded discussions addressing the points raised in your review.
If you have time during the discussion period, we would greatly appreciate it if you could take a look at our response. We would be happy to clarify any aspects of the paper or response that may still be unclear.
Thank you very much for your time and consideration.
We are very grateful for your insightful and constructive feedback (1/2)
Dear Reviewer sopr,
We are very grateful for your insightful and constructive feedback. We are glad that you found the problem meaningful and the framework clear. In light of the suggestions, we have added new experiments in the updated manuscript. Please find our detailed point-by-point responses below.
Q1: How sensitive is the diagnosis to the coverage and quality of the predefined intervention library?
A1: Thank you for the question. Coverage and quality decide whether our method can name a mechanism, not whether the mechanism it names is correct: when no candidate reproduces the deployment change, our method abstains. We note coverage as the main limitation in the Discussion (L519-520):
This requirement is mild, however. Our method only needs a simulator that can apply task preserving changes to the source environment, such as changing the color of a game object, the brightness, or the camera position, and its library is built from such changes. Many simulators already provide them, such as textures, lighting, and camera pose in domain randomization [1], weather and lighting in CARLA [2], and object properties in CausalWorld [3]. We also considered real robot tasks, where these changes must be applied in the robot's environment, and leave them to future work.
To test whether our method can reject interventions outside its coverage, we removed the true family from the library on the paper's five diagnosis blocks, so that the deployment change lies outside it, and reran calibration and Steps 1 and 2 on the same target failures.
Our method abstains in all 15 cases and never accepts a wrong family, even when the nearest remaining candidate is itself a harmful mechanism, such as the red paddle for the white brick target. Quality, that is, how closely a candidate reproduces the actual change, has the same effect. When the Freeway cars take intermediate gray levels and the library contains white cars, the closest candidate of the right family, our method abstains in all 10 diagnoses instead of accepting it. Please let us know if anything is still not fully clear.
[1] Domain randomization for transferring deep neural networks from simulation to the real world. IROS 2017
[2] CARLA: An open urban driving simulator. CoRL 2017
[3] CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning. ICLR 2021
Q2: Could the authors evaluate scenarios involving multiple simultaneous nuisance changes?
A2: Thank you for the question. We discuss this as a limitation and as future work in the Discussion (L521-523):
Multiple simultaneous nuisance changes can interact in complex ways, which makes them hard to diagnose. In Breakout, for example, the red paddle alone lowers the frozen policy's return to 55.0, adding a HUD mask lowers it further to 23.4, and adding blue bricks instead raises it to 93.3, although both added changes are harmless on their own. In supervised learning, concurrent shifts usually hurt more than single ones but not always [1], and attributing a performance change to several co-occurring shifts requires knowing the causal structure among them [2]. As far as we know, no prior work diagnoses which of several simultaneous nuisance changes causes the failure of a deployed policy. We see this as an open question that needs more time and careful study, and we leave it to future work.
[1] An analysis of model robustness across concurrent distribution shifts. TMLR 2025
[2] "Why did the model fail?": Attributing model performance changes to distribution shifts. ICML 2023