acceptodds
Withdrawn from ICLR 2027

What drives counterfactual-resoponse gains in policy distillation?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can teacher interpretability improve policy distillation beyond ordinary off-policy sampling? We study two routes: selecting a decision tree's input vocabulary and augmenting its training data with matched state pairs drawn along response-validated features. Across 27 PPO and SAC teachers on four control tasks, counterfactual augmentation improves intervention consistency (IC), a descriptive statistic of teacher–student response agreement, by on fresh pairs at equal tree size and a 20,000-label budget. Sampling controls recover most of this gain; the increment over random pairs is . A post-hoc expansion across 33 teachers evaluates IC and action error on identical endpoints: the scores are strongly associated, and only 58 of 2,376 method comparisons survive endpoint-error, return and competence matching, with a median absolute IC gap of . This does not establish equivalence, but provides no demonstrated incremental diagnostic utility for IC. Restricting the vocabulary to validated features can omit information needed for control; a predictive sufficiency check repairs the Pendulum failure without establishing a general repair. The study combines a pre-registered core with explicitly post-hoc controls and sensitivities. Its contribution is an empirical decomposition of response gains and the limits of this gate and feature library, without an established interpretability-specific control or robustness advantage.

Primary area:
interpretability and explainable AI
Keywords:
Policy distillation, interpretable reinforcement learning, counterfactual evaluation, decision tree policies, imitation learning, off-policy sampling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Ls9EOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether counterfactual-response agreement in policy distillation can be improved through interpretability-guided feature selection and data augmentation. It introduces an operational feature-validation gate and intervention consistency (IC), a metric measuring teacher–student agreement in action changes across matched state pairs. Experiments across PPO and SAC teachers show that counterfactual augmentation improves IC, but ordinary off-policy sampling controls recover most of the gain. A subsequent endpoint-matched analysis finds strong agreement between IC and conventional action error, without establishing additional diagnostic utility. The paper also shows that restricting surrogate inputs to validated concepts can exclude control-relevant information, while a predictive sufficiency check provides only a task-specific repair. Overall, the contribution is primarily an empirical analysis of the limitations of interpretability-guided policy distillation.

Critical strengths
  1. Careful experimental controls. The study includes matched-budget comparisons, multiple sampling controls, fresh counterfactual pairs, and teacher-level statistical analyses that help distinguish ordinary distribution coverage from handle-specific effects.

  2. Transparent reporting of negative findings. The authors explicitly acknowledge the limited practical benefits, metric limitations, implementation corrections, and post-hoc analyses, providing a useful empirical assessment rather than overstating the results.

Critical weaknesses
  1. Limited independent utility of IC. Large IC differences mainly arise between policies with substantially different control competence. The endpoint-matched analysis shows strong correlation with ordinary action error and no established additional diagnostic value.

  2. Limited interpretability-specific gains. CF augmentation improves fresh IC by 0.097, but random pairs already achieve 0.077, without a demonstrated advantage in closed-loop return or robustness. The novelty also needs clearer positioning relative to prior counterfactual and concept-based policy distillation [1,2].

  3. Insufficiently validated feature selection. The response-based gate can exclude control-critical variables, substantially degrading Pendulum performance. The proposed sufficiency check repairs this case but does not establish a general solution, while matched state pairs do not isolate individual causal effects.

  4. Restricted generality and incomplete cost accounting. The conclusions rely on small teachers and mostly CART surrogates, including Hopper trees that fail closed-loop, without testing augmentation on stronger surrogate classes. Equal fitting-label budgets also exclude substantial additional teacher access, and several important controls are post hoc.

[1] Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation

[2] Interpretable Deep Reinforcement Learning Via Concept-Based Policy Distillation

Questions for authors

Given the substantially stronger performance of the mixture-of-linear-experts baseline on some tasks, is there evidence that the sampling attribution results extend beyond CART trees to more competent surrogate policies?

Reviewer_QTsiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Supposidly better evaluation metric for RL policy distillation than standard teacher-student agreement. Nice experiments. Typo in the title. Very hard to understand the setting and contributions as most terms are not defined.

Critical strengths

Very strong empirical methodology. The idea of choosing carefully designed states to measure response agreement is sound. However it is so badly explained that it is hard to understand the paper.

Critical weaknesses

Writing is clearly mostly AI without human checking.

Questions for authors

Please write clearer. Dont hesitate to add schematics for new concepts like validated handles.

Reviewer_AdCjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors focus on improving policy distillation by exploiting the interpretability of the teacher. Specifically, they investigate two approaches: selecting relevant input features and improving sampling techniques through counterfactual data augmentation. They evaluate these approaches through extensive experiments, analyzing the problem from different angles and introducing the intervention consistency metric. Their results suggest that most of the improvements can be attributed to better state coverage rather than counterfactual sampling itself.

Critical strengths
  • Very extensive experimental setup, attempting to analyze the problem from several different angles.

  • Honest about the results, even when they are not particularly strong.

Critical weaknesses
  • Very difficult to follow and read. Sentences are fragmented and disconnected, and the text lacks cohesion. It is very dense, without enough high-level context for a reader to properly understand it.

  • The abstract and introduction should set the stage for the paper, but on my first read, I could barely understand what the paper was about from a general perspective. Both could be improved significantly.

  • It feels more like a report of the experiments conducted rather than a paper with a specific goal in mind, a clear set of experiments, and clear outcomes. Of course, additional experiments can be included in the appendix, but at least in the main paper, the storyline should be linear, with a clear hypothesis, a clear setup, and clear results. That's it. Instead, it feels more like an experimental report that has been iteratively improved, with additional patches introduced to fix issues.

  • The hypothesis of using counterfactual inputs to improve policy distillation is interesting. However, even if the proposed approach slightly improves intervention consistency, most of the gains can be attributed to simpler sampling strategies, and the improvements do not translate into a clear practical advantage.

  • Overall, it feels like a very incremental contribution to the community, which mainly learns that an interesting hypothesis was tested without yielding major results. Beyond this, what are the main generalisable insights that the community should take away from this study?

Questions for authors

See concerns above.

Comments on the paper

1 comment

We're very grateful to the reviewers for their time and for the careful reading they've given the paper. The paper doesn't clearly state its question or key takeaways, and several terms are used before being defined. We agree with that and have decided to withdraw the paper and rewrite it.

Regarding the ethics flag, we used an LLM assistant and disclosed this in the AI use statement (p. 10), as required by ICLR policy. All reported numbers were checked against the experimental outputs, and all references were checked against the sources. We accept that the writing fell short of the standard reviewers should expect, and the revised version will be rewritten by the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.