acceptodds
Under review as a conference paper at ICLR 2027

Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL

25 views

Abstract

While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.

Primary area:
reinforcement learning
Keywords:
contrastive reinforcement learning, action chunking, representation learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6pHAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a simple but effective modification to Contrastive Reinforcement Learning : conditioning the critic on action chunks. The method is evaluated across online and offline benchmarks, it shows performance gains on both. The authors show that conditioning the critic on information from an action chunk tells more about the goal than using a single action, and it generates measurably better representations for critic.

Critical strengths

the idea is simple and easy to understand: use action chunks in Contrastive Reinforcement Learning.

Evaluated on both offline and online benchmarks, showed improvments on both.

Critical weaknesses

Though the experiments are promising, the theoretical analysis is weak, and are mainly based on intuition. For example, why the mutual information between the state-action input and the goal can only increase with action chunk length, I(s, a1:H ; g) ≥ I(s, a; g)?

I sometimes find the arguments in the paper too aggressive, for example, they mentioned 'because distillation can only lose information'. Actually there are papers like [1] showed that distilled stundent performs better than teacher. I know it is from generative models, but it would be better if the conclusions are made from the experiment observations.

[1] Zhou, Mingyuan, et al. "Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation." Forty-first International Conference on Machine Learning. 2024.

Questions for authors

To make the experiment more convincing, have you ever tried to distill a single-action critic into a new single-action one?

3 comments

We thank the reviewer for the constructive feedback, the reference, and the suggested experiment, which we have run and added to the paper!

Though the experiments are promising, the theoretical analysis is weak, and are mainly based on intuition [...] For example, why the mutual information between the state-action input and the goal can only increase with action chunk length, I(s, a1:H;g) ≥ I(s, a; g)?

We kindly note that our work is primarily an empirical study, which we believe is appropriate as existing theory cannot yet fully characterize representation learning in deep RL. Our proposed hypothesis is nonetheless consistent with a basic information-theoretic result, I(s,a1:H;g)≥I(s,a1;g)I(s, a_{1:H}; g) \geq I(s, a_1;g). This follows from the chain rule of mutual information, I(s,a1:H;g)=I(s,a1;g)+I(a2:H;g∣s,a1)≥I(s,a1;g)I(s, a_{1:H}; g) = I(s, a_1; g) + I(a_{2:H}; g \mid s, a_1) \geq I(s, a_1; g), where the inequality holds because conditional mutual information is non-negative, I(a2:H;g∣s,a1)≥0I(a_{2:H}; g \mid s, a_1) \geq 0. Alternatively, the statement follows from the data processing inequality, which states that for any function ff, I(f(X);Z)≤I(X;Z)I(f(X); Z) \leq I(X; Z), i.e., processing XX cannot increase its information about ZZ. Applying it with X=(s,a1:H)X = (s, a_{1:H}), Z=gZ = g, and ff dropping the future actions, f(s,a1:H)=(s,a1)f(s, a_{1:H}) = (s, a_1), gives I(s,a1;g)≤I(s,a1:H;g)I(s, a_1; g) \leq I(s, a_{1:H}; g).

To make the scope of our paper explicit, we have added the following limitation to the Conclusions section: “Our explanation is empirical rather than theoretical; a theoretical analysis of when chunked critics learn better representations remains an open problem.”. Moreover, we have added the derivation above to Appendix B.1.

I sometimes find the arguments in the paper too aggressive, for example, they mentioned 'because distillation can only lose information'.

We agree the original wording was phrased more strongly than intended, and have revised it to: “we attribute the surviving gain to the critic’s representations — and because the distilled critic may not capture everything the chunked critic learned, it likely understates their benefit”. We hope it clarifies the statement. If there are any other such occurrences, we are happy to rephrase them accordingly.

To make the experiment more convincing, have you ever tried to distill a single-action critic into a new single-action one?

We thank the reviewer for proposing this experiment. To answer this question, we ran an experiment where a single-action CRL critic is distilled from another single-action CRL critic, using the same procedure (Eq. 8, Appendix E.7) as for distilling CRL + AC into a single-action critic, on two OGBench datasets: scene-noisy-v0 and cube-single-noisy-v0 (σ\sigma = 1.0). For all distillations we use κ=0.5\kappa = 0.5 (mean regression), following Reviewer tDHD's suggestion.

cube-single-noisy-v0 (σ=1\sigma = 1), 10 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.508–0.318–
CRL + AC -> single-action CRL0.610+20%0.376+18%
CRL -> single-action CRL0.433−15%0.304−5%

scene-noisy-v0, 6 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.232–0.137–
CRL + AC -> single-action CRL0.277+20%0.211+53%
CRL -> single-action CRL0.236+2%0.170+24%

The distilled CRL critic does not consistently improve over standard CRL: it underperforms on cube-single-noisy (peak −15%-15\%, final −5%-5\%), and on scene-noisy it matches standard CRL at peak (+2%+2\%), improving only final performance (+24%+24\%). In contrast, the critic distilled from CRL + AC outperforms standard CRL on both datasets, at both peak (+20%+20\%, +20%+20\%) and final performance (+18%+18\%, +53%+53\%). This indicates that distillation itself is not what improves the critic distilled from CRL + AC. The corresponding learning curves are shown in Figure 5 in Appendix E.1 of the revised manuscript. We believe this control strengthens the representational benefit finding. Thank you again for proposing this experiment!


We hope that our responses, including the single-action critic into a new single-action one experiment, have addressed reviewer’s concerns and increased confidence in our paper’s scope and results. If so, we kindly ask the reviewer to consider increasing their score. We’re happy to answer any further questions.

Dear Reviewer,

We wanted to follow up before the discussion period ends on November 18. Given our clarifications, we would highly appreciate it if you’d consider increasing your score. Please let us know if you have any remaining questions -- we’d be happy to provide more details. Thank you again for taking the time to carefully review our paper and for the helpful suggestions!

Thank the author for the reply. However, I still think the paper lack theoretical analysis. I therefore maintain my score.

Reviewer_tDHDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies a simple modification, action chunking, to contrastive RL (CRL), a self-supervised goal-conditioned RL method, in both online and offline settings, and observes large performance gains. The authors then run ablations on what drives these gains and conclude that action chunking gives the contrastive critic more information, yielding better representations, an effect that seems particularly relevant to CRL.

Critical strengths
  1. A simple method that works.
  2. Clear motivation and structure make it easy to follow.
  3. Broad, careful evaluation across 21 offline and 11 online environments.
  4. Real attempts to explain the gains, including distillation, state-only critics, noise sweeps, and depth and replanning ablations.
Critical weaknesses
  1. The paper's main claimed contribution is the explanation of the gains, since action chunking in RL is already established. However, this explanation has three gaps, namely (i) the distillation ablation is confounded by implicit policy improvement, (ii) the information argument is close to tautological, and (iii) the representational benefit is not shown online. I expand on each in the questions below.

  2. Benchmark coverage is uneven. There is no online manipulation, and offline navigation is limited to antmaze-explore, where the gain is negligible.

Questions for authors
  1. Expectile distillation. The single-action distillation (Eq. 8, Appendix E.1) uses upper-expectile regression (κ=0.95\kappa = 0.95), which values roughly the best continuation of a chunk rather than the behavior policy's. This is itself a policy-improvement step, and it would help most on noisy data, which is exactly where the gains concentrate. The claims that the surviving gain "must originate in the critic's representations" and that distillation gives a lower bound on the representational benefit therefore do not follow. How does single-action distillation perform with κ=0.5\kappa = 0.5, i.e., mean rather than upper-expectile regression? Does the gain over CRL survive?

  2. Information argument. Adding inputs can never reduce mutual information, so I(s,a1:H;g)≥I(s,a;g)I(s, a_{1:H}; g) \geq I(s, a; g) holds trivially, and the added inputs here are future actions. Higher categorical accuracy is therefore expected and does not show why the policy improves. Can you separate "more information in the critic" from "the critic conditions on future actions", e.g., by conditioning on a chunk of past actions, or on future actions shuffled across trajectories? An analysis of what the chunked critic estimates (the value of committing to HH actions, then following the behavior policy) and why that helps action selection would also strengthen this argument.

  3. The largest headline gain (+93.1%) comes from the online setting, yet replanning every step drops CRL + AC below CRL (Fig. 9), and the authors acknowledge that open-loop execution is necessary there. These gains may therefore come from temporally extended execution, one of the standard explanations the paper sets out to go beyond. What evidence supports a representational benefit online?

  4. How does CRL + AC perform on JaxGCRL manipulation (arm) tasks online, and on OGBench navigate or stitch datasets offline?

  5. Why is H=3H = 3 generally best, and does the optimal HH depend on control frequency, episode length, or noise level? Relatedly, how accurate is the long-HH critic on the actor's own chunks rather than on dataset chunks? Could the collapse at large HH reflect critic exploitation rather than extraction difficulty?

5 comments

We thank the reviewer for their positive, detailed feedback and thoughtful actionable questions!

Q1: Expectile distillation

We thank the reviewer for this insightful observation, with which we agree. To answer the question directly, we ran single-action distillation with κ=0.5\kappa = 0.5, which corresponds to mean regression and involves no improvement step, on cube-single-noisy (σ=1.0\sigma = 1.0), and scene-noisy-v0 to expand evaluation. Below we present the results:

cube-single-noisy-v0 (σ=1\sigma = 1), 10 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.508–0.318–
CRL + AC -> single-action CRL0.610+20%0.376+18%

scene-noisy-v0, 6 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.232–0.137–
CRL + AC -> single-action CRL0.277+20%0.211+53%

The gain over CRL remains with mean regression, confirming that performance is not induced by expectile regression. We have added these results (Figure 5) to the paper in Appendix E.1.

Q2: Can you separate "more information in the critic" from "the critic conditions on future actions", e.g., by conditioning on a chunk of past actions, or on future actions shuffled across trajectories? [...] An analysis of what the chunked critic estimates (the value of committing to actions, then following the behavior policy) and why that helps action selection would also strengthen this argument.

On cube-single-noisy (σ=1.0\sigma = 1.0), we compare the validation categorical accuracy of single-action CRL, CRL + AC (H=5H = 5), and two controls with the same input dimension as CRL + AC: a critic conditioned on a1a_1 plus the H−1H - 1 preceding (past) actions, and a critic conditioned on a1a_1 plus future actions a2:Ha_{2:H} shuffled across trajectories. We report the gain in accuracy over the state-only critic V(s)V(s):

Critic inputGain over V(s) (pp)
CRL+4.18
CRL + AC, past actions+3.99
CRL + AC, future actions shuffled across trajectories+4.32
CRL + AC+7.56

Both controls perform on par with single-action CRL (+3.99 and +4.32 pp vs. +4.18 pp), whereas CRL + AC nearly doubles the gain (+7.56 pp). With the same input dimension, the improvement therefore requires information that more strongly correlates with the agent’s future; shuffling future actions or providing past actions do not provide this signal. We have included this experiment in the manuscript in Figure 6 in Appendix E.2, as we believe it strengthens the information argument, and thank the reviewer for suggesting it!

Regarding what the chunked critic estimates and why it helps action selection: Qβ(s,a1:H,g)Q^\beta(s, a_{1:H}, g) is the value of committing to a1:Ha_{1:H} and then following the behavior policy. Greedy extraction from it therefore corresponds to an HH-step lookahead on the behavior value function, which, in deterministic MDPs with per-step replanning, tightens the standard one-step policy improvement guarantee of CRL from 2γ1−γϵβ\frac{2\gamma}{1-\gamma}\epsilon_\beta to 2γH1−γϵβ\frac{2\gamma^H}{1-\gamma}\epsilon_\beta (the hh-greedy policy of [3], with the bound of [4]), where ϵβ=∥V⋆−Vβ∥∞\epsilon_\beta = \|V^\star - V^\beta\|_\infty. We have added a small formal statement to Appendix B.2.

Q3: What evidence supports a representational benefit online?

Thank you for this question. Offline, where per-step replanning performs on par with open-loop execution, the representational benefit is supported by our distillation results (Q1). Online, however, we find open-loop execution to be necessary, which makes a clean isolation of the representational benefit challenging, as noted in Section 5.3. As each run collects a different set of trajectories, it is also hard to separate improvements in exploration and in representations. We have updated our discussion of limitations in Conclusions to further acknowledge this challenge.

Q4: How does CRL + AC perform on JaxGCRL manipulation (arm) tasks online, and on OGBench navigate or stitch datasets offline?

As suggested, we have extended our results on OGBench navigation datasets (pointmaze-large-navigate-v0, humanoidmaze-medium-navigate-v0) and stitch (antmaze-medium-stitch-v0), 6 seeds each:

DatasetCRLCRL + AC
humanoidmaze-medium-navigate-v00.070.00
pointmaze-large-navigate-v00.560.76
antmaze-medium-stitch-v00.490.51

And on JaxGCRL manipulation (arm) tasks online (5 seeds):

EnvironmentCRLCRL + AC (H=3H=3)CRL + AC (H=5H=5)
arm_binpick_easy0.150.080.07
arm_push_hard0.210.190.21

CRL + AC improves substantially on pointmaze-large-navigate (0.56 -> 0.76) and matches CRL on antmaze-medium-stitch, while on humanoidmaze-medium-navigate both methods perform poorly and action chunking does not help. On arm_binpick_easy CRL + AC underperforms CRL and on arm_push_hard it performs comparably, with all algorithms performing poorly on both tested JaxGCRL manipulation tasks. We have included the results in the paper in Appendices E.3 and E.6.

Q5: Why is generally best, and does the optimal depend on control frequency, episode length, or noise level?

We think these would be very helpful ablations. We find H=3H=3 to be a robust default, and Figure 3 indicates that the chunk length H trades off critic accuracy against the difficulty of policy extraction, but, as noted in our Conclusion, the optimal HH may vary across environments. To test the dependence on noise, we compare H=3H=3 and H=5H=5 on cube-single-noisy across noise levels σ\sigma, with per-step replanning (3 seeds; note that Figure 4, right, uses open-loop execution):

σ\sigmaH=3H=3H=5H=5Gap between H=5H=5 and H=3H=3
00.9680.992+0.024
0.010.9830.977-0.006
0.030.9510.973+0.022
0.10.9470.931-0.016
0.30.7760.829+0.053
1.00.6150.716+0.101

At low noise (σ≤0.1\sigma \leq 0.1), the two chunk lengths perform similarly, with differences within about 2 points and no consistent sign. At higher noise, longer chunks are clearly better (+5 points at σ=0.3\sigma = 0.3, +10 at σ=1.0\sigma = 1.0). These results suggest that longer chunks become more beneficial as noise increases

We additionally test varying the control frequency in the online JaxGCRL Ant environment (5 seeds):

Control FrequencyCRLCRL + AC (H=3H=3)CRL + AC (H=5H=5)
10.550.970.94
½0.090.900.54
⅓0.130.660.43

Decreasing the control frequency leads to a substantial performance degradation across all algorithms. However, the relative ordering of algorithm performance remains unchanged so the optimal action chunk length does not depend on control frequency in this setting. We report these results in the paper in Appendix E.5. As online results are confounded by exploration, we are happy to extend this experiment in the offline setting for the camera-ready version.

Q5: Relatedly, how accurate is the long- critic on the actor's own chunks rather than on dataset chunks?

This is a good question. Measuring this accuracy requires executing the actor's chunk in the simulator, continuing with the behavior policy to sample future states (positives and in-batch negatives), and computing the categorical accuracy of Figure 3. Given the implementation effort required to do this reliably, we prioritized the other requested experiments during the rebuttal period, and are happy to add this evaluation for the camera-ready version.

Q5: Could the collapse at large reflect critic exploitation rather than extraction difficulty?

Our extraction argument follows DQC [2], which attributes the failure of long-chunk policies to extraction difficulty. The picture may be more nuanced, and extraction difficulty may involve critic exploitation. To test this, we compare the critic's score of the actor's chunk with that of the dataset chunk and of a chunk produced by a BC policy, using as the goal a future state from the same trajectory (cube-single-noisy-v0 (σ=1.0\sigma = 1.0), 4096 states each):

HHDataset chunkBC policyActor% states with Actor > Dataset chunk)
5−12.63−12.90−11.7168%
30−13.07−13.96−6.1596%

Since the actor maximizes the critic, its chunks are expected to score somewhat higher than dataset chunks. However, at H=30H = 30, the actor's chunks score is far above the dataset and BC chunks, despite near-zero success. This indicates that extraction at large HH might fail through critic exploitation: the actor finds out-of-distribution chunks that the critic overestimates, even though the critic is informative on dataset chunks (as distillation in Figure 3 shows). We have added the experiment to Appendix E.4 and revised the paper to make this explicit and thank the reviewer for the question.


We are genuinely grateful for the reviewer's careful technical questions and insightful suggestions, which we believe have substantially strengthened our paper. We hope the new experiments (mean-regression distillation, the shuffled-action and past-action controls, results on navigation, stitch, and arm tasks, the dependence of the optimal HH on noise level and control frequency, and the actor-vs-dataset critic scores) have addressed the reviewer's concerns. If so, we would be grateful if the reviewer would consider raising their score. We are happy to answer any further questions.

[2] Qiyang Li, Seohong Park, Sergey Levine. Decoupled Q-Chunking. ICLR 2026. arXiv:2512.10926

[3] Yonathan Efroni, Gal Dalal, Bruno Scherrer, Shie Mannor. Beyond the One-Step Greedy Approach in Reinforcement Learning. ICML 2018. arXiv:1802.03654

[4] Satinder P. Singh, Richard C. Yee. An Upper Bound on the Loss from Approximate Optimal-Value Functions. Machine Learning, 16(3):227–233, 1994. doi:10.1007/BF00993308

Reviewer_ADTGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work extends the standard contrastive reinforcement learning (CRL) formulation by conditioning the learned state-action value function on a sequence of multiple actions rather than on a single action. This practice is referred to as action chunking, and the proposed method is referred to as CRL + AC. The work finds that action chunking improves performance on a number of problems in both online and offline RL problems drawn from OGBench and JaxGCRL. In testing why action chunking helps, the authors find that previously proposed explanations alone are insufficient to explain the improvements imparted by action chunking and propose a new explanation. They additionally perform ablations that assess how network depth, replanning, and the degree of noise in demonstrations affects the proposed method.

Critical strengths

1. The paper is well written and easy to follow. The introduction states the research questions and overviews of the main findings without ambiguity. The related work section does a good job of situating the work within the broader literature. The background is concise while still providing the required mathematical foundations.

2. The paper is well motivated. The research questions are clearly stated, and each research question naturally leads to the next without leaving any obvious gaps.

Critical weaknesses

1. The methodology contains several mistakes and unclear procedures.

I will use Table 1 as an example. The reported results use three seeds, which is not enough to compute a reliable bootstrapped confidence interval. Strange inconsistencies in the results are unexplained, like the fact that CRL performs better in noisy versions of cube-single and puzzle-3x3 rather than the no-noise version of the environments. I suspect that these inconsistencies are results of the above issues. The process for computing confidence intervals should be explained (even in the appendix) rather than referring to a library used to compute the confidence interval (RLiable). The table also compares incomparable quantities: the OGBench paper states that the reported \pm values are standard deviations, but this work reports confidence intervals. Similar problems are persistent throughout figures in the paper.

It is not clear how the headline numbers (31.7% on OGBench manipulation suite, 69.4% on suboptimal OGBench datasets, and +93.1% on online JaxGCRL locomotion and navigation) are computed. Is there a need for these numbers? I find the more nuanced statements about the results (e.g. no improvements on some environments like cube and puzzle normal, but significant improvements in others like noisy puzzle and scene) to be more compelling. They help think about why this method helps in a way the headline numbers do not help.

2. A central claim of the paper is that action chunking improves performance in CRL + AC in part because it “adds information about the goal” (line 315), but it is not clear what this means. At first I assumed it meant that more actions provides the critic with information that makes it easier to infer the goal g, but g is already an input to the value function, so it is unclear why this would be useful.

The plots in Figure 3 are offered as evidence of the proposed explanation, but I fail to see how it is related. The left plot shows that as the state-action value function is conditioned on more future actions, it becomes more accurate. It should be expected that providing information about the future will make predictions of the future more accurate. If there is a link to the proposed explanation, this does not clarify it.

An alternative explanation for the gap in performance between CRL + AC and the distilled critic is the distillation process itself. Why would we expect the hypothesized good critic representation learned by the CRL + AC to survive distillation and not learn a new representation? Why does the distillation method use quantile regression instead of regression of the mean? The distilled critic is trained to learn something different, and that difference may be the source of the differing results.

I would consider raising my score if both points are addressed.

Questions for authors
  1. Is there a connection between the proposed explanation on why action chunking is useful in CRL and the results presented in Figure 3 that you think I am missing?
  2. Is the objective of the loss in Equation 3 minimization? If so, should the sign of the equation be flipped?
  3. Do you expect a network with recurrent connections to perform similarly to action chunking? Have you tried this? (I am not suggesting you add this to the paper, but only asking out of curiosity)
4 comments

We thank the reviewer for their positive, detailed review!

W1: The reported results use three seeds, which is not enough to compute a reliable bootstrapped confidence interval.

Thank you for raising this point. We used three seeds in the offline setting to prioritize breadth of evaluation (11 online benchmarks, 21 offline datasets with per-dataset α∈\alpha \in {1,3,10} tuning). We have used more seeds for additional experiments we include in the revision, and we will extend all the offline experiments to at least 6 seeds for the camera-ready version, which we could not do within the rebuttal window due to computational constraints.

W1: Strange inconsistencies in the results are unexplained, like the fact that CRL performs better in noisy versions of cube-single and puzzle-3x3…

These trends are consistent with the original evaluation in OGBench [1] (Tab. 5), where CRL performs better on noisy datasets. A likely reason is the state coverage that increases with higher noise injection. We have added a pointer to the original evaluation in our manuscript to clarify this.

W1: The process for computing confidence intervals should be explained [...] The table also compares incomparable quantities: the OGBench [...]

We have added a description of our confidence-interval procedure to Appendix D.1. In short, in figures, we report 95% percentile intervals from a stratified bootstrap: seeds are resampled with replacement within each task, the mean success rate is recomputed, and this is repeated 2,000 times, and 2.5th and 97.5th percentiles are reported.

Thank you for catching the Table 1 inconsistency. We now report mean and standard deviation over seeds for all methods, including group averages. We describe how the standard deviations, including those of group averages, are computed in Appendix D.1.

It is not clear how the headline numbers [...] are computed. I find the more nuanced statements about the results [...] to be more compelling.

The headline numbers are relative gains of CRL + AC over CRL in aggregated success rate (details in Appendix C.3). We have now referenced this in the caption of Figure 1. We agree that nuanced statements are more interesting, which is why we center our analysis on per-task success rates (Table 1) and learning curves (Appendix F); the aggregated metrics are only meant as an overall summary of trends. We have extended the discussion of Table 1 (line 269) to describe where gains are largest (noisy Puzzle and Scene) and where CRL + AC performs on par with CRL (play variants of Cube and Puzzle).

W2/Q1: A central claim of the paper is that action chunking [...] “adds information about the goal” (line 315), but it is not clear what this means. At first I assumed it meant that more actions provides the critic with information that makes it easier to infer the goal g [...] It should be expected that providing information about the future will make predictions of the future more accurate [...] Is there a connection between the proposed explanation on why action chunking is useful in CRL?

Thank you for raising this important point. The CRL critic factorizes as Q(s,a,g)=ϕ(s,a)⊤ψ(g)Q(s, a, g) = \phi(s, a)^\top \psi(g). Although Q(s,a,g)Q(s, a, g) is conditioned on gg, the state-action encoder ϕ\phi never sees gg, so it must infer the goal from its own input to score the true gg above the negatives in the batch. "Adding information about the goal" means that ϕ(s,a1:H)\phi(s, a_{1:H}) can in principle infer the true goal gg better than ϕ(s,a1)\phi(s, a_1).

In other words, as the reviewer notes, information about the future can, in principle, only make goal predictions equally or more accurate; however, arbitrary information is not necessarily helpful. To verify this, on cube-single-noisy (σ=1.0\sigma = 1.0), we compare the validation categorical accuracy of single-action CRL, CRL + AC (H=5H = 5), and a control with the same input dimension as CRL + AC: a critic conditioned on a1a_1 and future actions a2:Ha_{2:H} sampled from random later timesteps of the same trajectory. We report the gain over the state-only critic V(s)V(s):

Critic inputGain over V(s)V(s) (pp)
CRL+4.18
CRL + AC, random future actions+4.31
CRL + AC+7.56

The control performs on par with single-action CRL, whereas CRL + AC nearly doubles the gain. The improvement therefore comes specifically from the immediate future actions, rather than from additional inputs or general information about the trajectory. Furthermore, Figure 3 shows that the gain over V(s)V(s) grows further with chunk length (up to H=30H = 30), indicating that longer chunks add more information about the goal. We have added this experiment to Appendix E.2 (Figure 6) of the revised manuscript. We are happy to clarify any further questions.

W2: An alternative explanation for the gap in performance between CRL + AC and the distilled critic is the distillation process itself. Why would we expect the hypothesized good critic representation learned by the CRL + AC to survive distillation and not learn a new representation?

Part of the representation is directly inherited: the goal encoder ψ\psi is shared with the CRL + AC critic and kept fixed during distillation (Eq. 8, Appendix E.7); only the state-action encoder ϕ’\phi’ is distilled.

Still, distillation is a complex process that may not fully preserve representations. Thus, to test whether distillation itself drives the gain, we distill a single-action CRL critic from a standard CRL critic, using the same procedure (Eq. 8, Appendix E.7) as for distilling CRL + AC into a single-action critic, on two OGBench datasets: scene-noisy-v0 and cube-single-noisy-v0 (σ=1.0\sigma = 1.0). For all distillations we use κ=0.5\kappa = 0.5 (mean regression). The results are presented in tables below:

cube-single-noisy-v0 (σ=1.0\sigma = 1.0), 10 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.508–0.318–
CRL + AC -> single-action CRL0.610+20%0.376+18%
CRL -> single-action CRL0.433−15%0.304−5%

scene-noisy-v0, 6 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.232–0.137–
CRL + AC -> single-action CRL0.277+20%0.211+53%
CRL -> single-action CRL0.236+2%0.170+24%

The distilled CRL underperforms on cube-single-noisy, and on scene-noisy it matches standard CRL at peak, improving only final performance. In contrast, the critic distilled from CRL + AC outperforms standard CRL on both datasets. This suggests that gain comes from the source of the distillation process, rather than from the distillation process itself -- i.e. from what the chunked critic learned. The resulting learning curves are presented in Figure 5 in Appendix E.1 of the revised manuscript.

W2: Why does the distillation method use quantile regression instead of regression of the mean? The distilled critic is trained to learn something different, and that difference may be the source of the differing results.

Our distillation followed the standard DQC procedure, which uses upper-expectile regression (high κ\kappa values). With perfect critics, κ→1\kappa \to 1 approaches the score of the best in-data continuation. In contrast, with κ=0.5\kappa = 0.5 the expectile loss reduces to squared error, whose minimizer is the conditional mean over in-data continuations. To isolate the representational effect, we ran distillation with κ=0.5\kappa = 0.5 on cube-single-noisy (σ=1\sigma = 1) and, to further strengthen the experiment, on scene-noisy-v0:

cube-single-noisy-v0 (σ=1\sigma = 1), 10 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.508–0.318–
CRL + AC -> single-action CRL0.610+20%0.376+18%

scene-noisy-v0, 6 seeds per method

MethodPeakPeak gain over CRLFinalFinal gain over CRL
CRL0.232–0.137–
CRL + AC -> single-action CRL0.277+20%0.211+53%

With κ=0.5\kappa = 0.5, the distilled critic still outperforms CRL by 20% at peak on each, and by 18% (cube-single-noisy) and 53% (scene-noisy) in final performance, suggesting that expectile regression is not driving performance. We have added these results in Appendix E.1 (Figure 5). Thank you for suggesting this experiment, which offers a clean comparison and further supports our results.

Q2: Is the objective of the loss in Equation 3 minimization? If so, should the sign of the equation be flipped?

Yes, thank you for catching this. We have flipped the sign in Equation 3 so that it is minimized.

Q3: Do you expect a network with recurrent connections to perform similarly to action chunking? Have you tried this?

Thank you for the interesting question, we have not tried this. If recurrent connections are used to embed past states, performance may still improve for other reasons (e.g., handling partial observability), although in our controls conditioning the critic on past actions did not improve its goal prediction (Appendix E.2). Moreover, a recurrent encoder may not be easier to train; we believe that action chunks are a more direct way to produce a more informative critic, but we think this remains an interesting direction for future work


We hope our response has addressed both points the reviewer raised. If so, we would be grateful if the reviewer would consider raising their score. We are happy to answer any further questions.

[1] Seohong Park, Kevin Frans, Benjamin Eysenbach, Sergey Levine. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025. arXiv:2410.20092

Dear Reviewer,

As the discussion period ends on November 18, we wanted to kindly follow up on our response. If it has addressed your concerns, we would be grateful if you would consider raising your score. We are happy to answer any remaining questions. Thank you once more for your careful review of our paper and your valuable suggestions!

Thank you for the thorough response! My major concerns have all been addressed, so I am raising my soundness score from 2 to 3 and overall score from 3 to 4.

I only have one small comment remaining. My point on the bootstrapped CI with 3 seeds was actually meant to be less focused on the seeds and more so on the use of a confidence interval (though more seeds would also be an improvement). Bootstrapped CIs are generally unreliable with only 3 seeds and often underestimate the range. They become more reliable with around 20-30 seeds. I would just avoid CIs all together if you are only reporting 3 seeds. You could report the standard deviation or the minimum and maximum instead as the table now does.

This is overall a solid work. Thank you for the interesting read.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.