- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a simple but effective modification to Contrastive Reinforcement Learning : conditioning the critic on action chunks. The method is evaluated across online and offline benchmarks, it shows performance gains on both. The authors show that conditioning the critic on information from an action chunk tells more about the goal than using a single action, and it generates measurably better representations for critic.
Critical strengths
the idea is simple and easy to understand: use action chunks in Contrastive Reinforcement Learning.
Evaluated on both offline and online benchmarks, showed improvments on both.
Critical weaknesses
Though the experiments are promising, the theoretical analysis is weak, and are mainly based on intuition. For example, why the mutual information between the state-action input and the goal can only increase with action chunk length, I(s, a1:H ; g) ≥ I(s, a; g)?
I sometimes find the arguments in the paper too aggressive, for example, they mentioned 'because distillation can only lose information'. Actually there are papers like [1] showed that distilled stundent performs better than teacher. I know it is from generative models, but it would be better if the conclusions are made from the experiment observations.
[1] Zhou, Mingyuan, et al. "Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation." Forty-first International Conference on Machine Learning. 2024.
Questions for authors
To make the experiment more convincing, have you ever tried to distill a single-action critic into a new single-action one?
3 comments
Dear Reviewer,
We wanted to follow up before the discussion period ends on November 18. Given our clarifications, we would highly appreciate it if you’d consider increasing your score. Please let us know if you have any remaining questions -- we’d be happy to provide more details. Thank you again for taking the time to carefully review our paper and for the helpful suggestions!
Thank the author for the reply. However, I still think the paper lack theoretical analysis. I therefore maintain my score.
We thank the reviewer for the constructive feedback, the reference, and the suggested experiment, which we have run and added to the paper!
We kindly note that our work is primarily an empirical study, which we believe is appropriate as existing theory cannot yet fully characterize representation learning in deep RL. Our proposed hypothesis is nonetheless consistent with a basic information-theoretic result, I(s,a1:H;g)≥I(s,a1;g). This follows from the chain rule of mutual information, I(s,a1:H;g)=I(s,a1;g)+I(a2:H;g∣s,a1)≥I(s,a1;g), where the inequality holds because conditional mutual information is non-negative, I(a2:H;g∣s,a1)≥0. Alternatively, the statement follows from the data processing inequality, which states that for any function f, I(f(X);Z)≤I(X;Z), i.e., processing X cannot increase its information about Z. Applying it with X=(s,a1:H), Z=g, and f dropping the future actions, f(s,a1:H)=(s,a1), gives I(s,a1;g)≤I(s,a1:H;g).
To make the scope of our paper explicit, we have added the following limitation to the Conclusions section: “Our explanation is empirical rather than theoretical; a theoretical analysis of when chunked critics learn better representations remains an open problem.”. Moreover, we have added the derivation above to Appendix B.1.
We agree the original wording was phrased more strongly than intended, and have revised it to: “we attribute the surviving gain to the critic’s representations — and because the distilled critic may not capture everything the chunked critic learned, it likely understates their benefit”. We hope it clarifies the statement. If there are any other such occurrences, we are happy to rephrase them accordingly.
We thank the reviewer for proposing this experiment. To answer this question, we ran an experiment where a single-action CRL critic is distilled from another single-action CRL critic, using the same procedure (Eq. 8, Appendix E.7) as for distilling CRL + AC into a single-action critic, on two OGBench datasets: scene-noisy-v0 and cube-single-noisy-v0 (σ = 1.0). For all distillations we use κ=0.5 (mean regression), following Reviewer tDHD's suggestion.
cube-single-noisy-v0 (σ=1), 10 seeds per method
scene-noisy-v0, 6 seeds per method
The distilled CRL critic does not consistently improve over standard CRL: it underperforms on cube-single-noisy (peak −15%, final −5%), and on scene-noisy it matches standard CRL at peak (+2%), improving only final performance (+24%). In contrast, the critic distilled from CRL + AC outperforms standard CRL on both datasets, at both peak (+20%, +20%) and final performance (+18%, +53%). This indicates that distillation itself is not what improves the critic distilled from CRL + AC. The corresponding learning curves are shown in Figure 5 in Appendix E.1 of the revised manuscript. We believe this control strengthens the representational benefit finding. Thank you again for proposing this experiment!
We hope that our responses, including the single-action critic into a new single-action one experiment, have addressed reviewer’s concerns and increased confidence in our paper’s scope and results. If so, we kindly ask the reviewer to consider increasing their score. We’re happy to answer any further questions.