acceptodds
Under review as a conference paper at ICLR 2027

PACE: Discovering Options via Self-Distillation

Abstract

Canonical reinforcement learning (RL) policies require reading an observation before sampling an action. However, running a policy at each step for a single decision can be expensive. Should an agent be able to emit n actions at a time, this would allow reducing the number of observations it has to read and the number of inference passes it has to run during a decision process, unlocking large computational savings. In this work, we argue that many decision problems contain states where one observation carries enough information for the policy to emit more than one optimal action at a time, making intermediate state reads an avoidable cost. We propose PACE (Per-state Adaptive Commitment): while Proximal Policy Optimisation (PPO) trains as usual, PACE distills it into a multi-step policy that reads one observation and emits several consecutive actions from it alone. Every time this policy reads an observation from the environment, it also predicts the number of actions to output by measuring how far that replay can be trusted, and committing exactly that far. On random tasks with deterministic transitions, PACE reduces the number of required observations by up to 8x, and the compression shrinks as transition stochasticity rises. Furthermore, we show that, in open-ended settings, recursively augmenting the action space of a Markov Decision Process with the options discovered by PACE drastically improves success rates in environments where both curriculum and tabula-rasa learning fail.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, options, action chunking, self-distillation, inference efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MQVCOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PACE (Per-state Adaptive Commitment) aims to reduce how often an RL agent must observe its environment during execution. The idea is that, in predictable situations, one observation may contain enough information to choose several consecutive actions. PACE trains a standard PPO policy and uses its recorded experience to train two additional components alongside it through distillation. The first learns to predict the actions PPO would choose over the next few steps, using only the starting observation. The second predicts how many actions the agent can execute before those predictions are likely to disagree with PPO. Experiments, primarily on MiniGrid tasks, report up to 8× fewer observations while maintaining success rates close to PPO’s. The paper also reuses discovered action sequences as additional actions when learning harder tasks. Combining these options with curriculum learning increases success on the final Room Chain level from 30.3% to 85.0%.

Critical strengths

A useful separation of learning objectives.

PPO learns to solve the task by choosing one action at a time and observing after each step. Alongside it, PACE’s sequence predictor learns to imitate PPO’s future action choices using only the starting observation. PPO’s predictions provide specific teaching targets for learning these sequences. This is an appealing design because it allows PPO to learn the task without requiring it to explore through long action sequences executed without intermediate observations.

Relevant results within the evaluated setting.

Table 1 shows substantial observation reductions without the large success-rate losses suffered by several chunking baselines. The Room Chain results also demonstrate a substantial benefit from combining options with parameter transfer.

Critical weaknesses

W1 Observation savings do not establish computational savings.

This distinction matters because reduced inference cost is a central motivation. PACE evaluates three branches and generates actions autoregressively: one observation therefore does not correspond to the same computation as one PPO forward pass. Appendix D.7 reports 3.3× more deployment parameters, while Appendix E.1 reports 132 seconds of training versus 11 seconds for PPO on DK8. The paper provides no deployment latency or FLOP measurements.

W2 Comparisons with related methods must be strengthened.

The SEAR baseline is adapted to PPO, changing its critic and training procedure. The authors explain that their implementation of original SEAR failed on these sparse-reward tasks, which provides a reasonable justification for the adaptation. Nevertheless, the reported comparison primarily establishes an advantage over the adapted version only. The paper should also discuss temporal distillation (Avé et al., 2025), which learns to predict an action and its repetition length from a teacher policy. PACE extends this idea to sequences containing different actions, and a direct comparison would help establish the benefit of that additional flexibility. These gaps limit the strength of the comparative claims.

Questions for authors

Q1 : Is PACE faster than PPO after training? Please compare their execution times on the same tasks and hardware, including the time spent running all parts of PACE.

Q2 : How does PACE compare with the earlier temporal-distillation method? That method learns to repeat one action several times. A comparison on the same tasks would help show the benefit of PACE’s ability to predict sequences containing different actions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer MQVC (part1)

We thank reviewer MQVC for the review and for the helpful feedback to improve the manuscript.

We are grateful to read that the review recognises that PACE “is an appealing design because it allows PPO to learn the task without requiring it to explore through long action sequences executed without intermediate observations”, that PACE “shows substantial observation reductions without the large success-rate losses suffered by several chunking baselines”, and that “The Room Chain results also demonstrate a substantial benefit from combining options with parameter transfer.”

We also understand the concerns raised, and we have addressed each of them below.


W1, Q1: Observation savings do not establish computational savings. This distinction matters because reduced inference cost is a central motivation. PACE evaluates three branches and generates actions autoregressively: one observation therefore does not correspond to the same computation as one PPO forward pass. Appendix D.7 reports 3.3× more deployment parameters, while Appendix E.1 reports 132 seconds of training versus 11 seconds for PPO on DK8. The paper provides no deployment latency or FLOP measurements.

and

Is PACE faster than PPO after training? Please compare their execution times on the same tasks and hardware, including the time spent running all parts of PACE.

We have added a new analysis that helps better understand the costs of running PACE compared to those of PPO at deployment/inference time. We report FLOPs and Wall Clock Time, both per decision and per episode, in Table 10, Appendix E.1, and the corresponding per-episode ratios in Table 2, Section 6.1, with the full ratios in Table 11, Appendix E.1. Wall Clock Time is measured on an RTX 5090.

Table 10 FLOPs / decision

MethodDK5DK8DO6-N3P2SC11-N5
PPO19,27234,24823,49656,136
PACE51,70596,63364,377162,297
no codebook93,159138,087105,831203,751

Table 10 Wall Clock Time / decision (μs), measured on RTX5090

MethodDK5DK8DO6-N3P2SC11-N5
PPO113.62 ± 10.41110.86 ± 8.48113.42 ± 11.41108.19 ± 4.32
PACE115.58 ± 11.04112.47 ± 6.65114.86 ± 10.88110.40 ± 5.17

Table 10 Wall Clock Time per episode (ms), measured on RTX5090

MethodDK5DK8DO6-N3P2SC11-N5
PPO1.077 ± 0.0141.846 ± 0.0191.358 ± 0.0352.048 ± 0.032
PACE0.136 ± 0.0000.304 ± 0.0670.883 ± 0.0400.290 ± 0.021

Table 11 Per-episode ratios relative to PPO

DK5DK8DO6-N3P2SC11-N5
Observations0.12 ± 0.000.16 ± 0.040.64 ± 0.030.14 ± 0.01
FLOPs0.33 ± 0.000.46 ± 0.101.76 ± 0.080.40 ± 0.03
Wall Clock Time0.13 ± 0.000.16 ± 0.040.65 ± 0.030.14 ± 0.01
FLOPs, no codebook0.60 ± 0.010.65 ± 0.152.89 ± 0.140.50 ± 0.03

The tables show that a single PACE decision costs more FLOPs than a PPO decision. However, PACE takes less decisions than PPO, thanks to its multi-step policy, overall making PACE faster to run at deployment in all the environments, and cheaper to run in all environments except DO6-N3P2, due to its stochasticity, as described in Section 6.2.

We have also revised the contribution in the main-text to “reducing the number of observations/decisions”. Section 4 now explains that the greedy continuations are computed once after training and stored in the options table. At deployment, a decision runs the base policy, termination function and encoder, followed by a lookup; there is no sequential decoding at deployment. Section 6.1 now reports the deployment-cost results above.

At training time, PACE is more expensive than PPO alone. Appendix D.7 reports the parameter counts. Appendix E.1 explains the training cost: the distillation loss evaluates the teacher and decoder at every position. The deployment measurements above account for the larger deployed network. This behaviour of more expensive training but cheaper inference is akin to that of Mixture of Experts models in LLM settings, where training costs are traded for inference costs (despite them being completely different methods, and different purposes).

AuthorsOpenReview ↗

Response to reviewer MQVC (part2)


W2: The SEAR baseline is adapted to PPO, changing its critic and training procedure. The authors explain that their implementation of original SEAR failed on these sparse-reward tasks, which provides a reasonable justification for the adaptation. Nevertheless, the reported comparison primarily establishes an advantage over the adapted version only.

We agree that the original manuscript did not provide direct evidence that the adapted SEAR outperforms the original formulation. We now label the PPO port as SEAR† and describe the adaptation and its rationale in Appendix C.3.

To justify the adaptation we have also included the pre-port SEAR baseline as described in the original paper in Table 1. Results show that the original, pre-port SEAR achieves 7.5% ± 4.7% success rate on average on all environments. We report below the results with SEAR, the SEAR† port, and PACE for your convenience.

DK5Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
SEAR0.04 ± 0.101.59 ± 0.24×28.85 ± 4.566.07 ± 0.914.75 ± 0.20
SEAR†1.00 ± 0.003.60 ± 0.38×18.79 ± 1.462.66 ± 0.267.08 ± 0.29
PACE (Table 1)1.00 ± 0.008.01 ± 0.00×9.43 ± 0.001.18 ± 0.007.97 ± 0.00
DK8Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
SEAR0.00 ± 0.000.83 ± 0.01×100.00 ± 0.0020.00 ± 0.005.00 ± 0.00
SEAR†0.51 ± 0.172.79 ± 0.16×38.56 ± 2.456.00 ± 0.346.43 ± 0.07
PACE (Table 1)0.99 ± 0.016.34 ± 0.85×16.56 ± 0.112.70 ± 0.606.30 ± 0.83
DO6-N3P2Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
SEAR0.26 ± 0.162.59 ± 0.60×22.25 ± 6.314.90 ± 1.294.44 ± 0.18
SEAR†0.63 ± 0.021.05 ± 0.06×11.49 ± 0.7211.49 ± 0.721.00 ± 0.00
PACE (Table 1)0.78 ± 0.021.56 ± 0.07×12.04 ± 0.307.69 ± 0.351.57 ± 0.04
SC11-N5Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
SEAR0.00 ± 0.000.47 ± 0.01×200.00 ± 0.0040.00 ± 0.005.00 ± 0.00
SEAR†0.22 ± 0.108.40 ± 1.16×21.81 ± 3.482.30 ± 0.379.47 ± 0.14
PACE (Table 1)0.92 ± 0.017.22 ± 0.49×18.90 ± 0.202.63 ± 0.197.21 ± 0.48
All environmentsSuccess rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
SEAR0.08 ± 0.051.37 ± 0.16×87.78 ± 1.9517.74 ± 0.394.80 ± 0.07
SEAR†0.59 ± 0.053.96 ± 0.31×22.66 ± 1.145.61 ± 0.236.00 ± 0.08
PACE (Table 1)0.92 ± 0.015.78 ± 0.25×14.23 ± 0.093.55 ± 0.185.76 ± 0.24

Table 1 (b). Success rate, compression, steps to goal, observations per episode, and commitment length of SEAR, the SEAR† port, and PACE on the tasks of Table 1; mean ± standard deviation over seeds. All environments: mean across the four tasks.

AuthorsOpenReview ↗

Response to reviewer MQVC (part3)

W2, Q2: The paper should also discuss temporal distillation (Avé et al., 2025), which learns to predict an action and its repetition length from a teacher policy. PACE extends this idea to sequences containing different actions, and a direct comparison would help establish the benefit of that additional flexibility.

and

How does PACE compare with the earlier temporal-distillation method? That method learns to repeat one action several times. A comparison on the same tasks would help show the benefit of PACE’s ability to predict sequences containing different actions.

We have added a discussion of Avé et al. (2025) to Section 2 and results for temporal distillation (TempDistill) and observation pricing (ObsPrice) to Table 1. The implementation details of both are in new Appendices C.6 and C.7. We report below the results with only TempDistill, ObsPrice and PACE for your convenience, using 32 seeds.

DK5Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
TempDistill1.00 ± 0.001.20 ± 0.06×9.44 ± 0.067.89 ± 0.451.20 ± 0.06
ObsPrice0.98 ± 0.091.81 ± 0.19×9.42 ± 0.135.29 ± 0.541.80 ± 0.19
PACE (Table 1)1.00 ± 0.008.01 ± 0.00×9.43 ± 0.001.18 ± 0.007.97 ± 0.00
DK8Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
TempDistill0.90 ± 0.251.60 ± 0.11×16.66 ± 0.2210.45 ± 0.751.60 ± 0.10
ObsPrice0.99 ± 0.021.11 ± 0.05×16.58 ± 0.1215.04 ± 0.661.10 ± 0.05
PACE (Table 1)0.99 ± 0.016.34 ± 0.85×16.56 ± 0.112.70 ± 0.606.30 ± 0.83
DO6-N3P2Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
TempDistill0.78 ± 0.011.17 ± 0.06×11.94 ± 0.4710.27 ± 0.501.16 ± 0.02
ObsPrice0.77 ± 0.021.08 ± 0.04×11.46 ± 0.3311.13 ± 0.431.03 ± 0.03
PACE (Table 1)0.78 ± 0.021.56 ± 0.07×12.04 ± 0.307.69 ± 0.351.57 ± 0.04
SC11-N5Success rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
TempDistill0.90 ± 0.021.03 ± 0.16×19.08 ± 0.5118.57 ± 1.741.04 ± 0.16
ObsPrice0.89 ± 0.081.22 ± 0.11×19.07 ± 0.3715.70 ± 1.461.22 ± 0.11
PACE (Table 1)0.92 ± 0.017.22 ± 0.49×18.90 ± 0.202.63 ± 0.197.21 ± 0.48
All environmentsSuccess rate ↑Compression ↑Steps to goalObservations per episode ↓Commitment length
TempDistill0.90 ± 0.061.25 ± 0.05×14.28 ± 0.1811.80 ± 0.501.25 ± 0.05
ObsPrice0.91 ± 0.031.31 ± 0.06×14.13 ± 0.1311.79 ± 0.441.29 ± 0.06
PACE (Table 1)0.92 ± 0.015.78 ± 0.25×14.23 ± 0.093.55 ± 0.185.76 ± 0.24

Table 1 (c). Success rate, compression, steps to goal, observations per episode, and commitment length of the two new baselines and PACE on the tasks of Table 1; mean ± standard deviation over seeds. All environments: mean across the four tasks.


We would like to thank the reviewer again for the useful feedback and would be grateful if the reviewer would reconsider their score to suggest acceptance, should the new results and clarification fill the gap identified in the review.

Reviewer_v1D5OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores a middle ground between the relatively well-studied notions of action repetition and option discovery in RL. In particular, the paper considers options that are defined by a fixed sequence of primitive actions. To discover such options, a behavior policy is trained via vanilla PPO (maximizing return) and its realized actions are used to distill useful sequences of actions to use as options. The experimental evaluation shows that the proposed method, PACE, significantly reduces the decision-making frequency under deterministic dynamics, gracefully shortens option execution horizons to handle stochastic environments, and learns a library of generally useful options in a meta-learning study in family of gridworlds.

Critical strengths
  1. The paper explores option discovery in RL by learning fixed action chunks and the appropriate number of timesteps to commit to them from a particular state. This is an important intermediate between the relatively well-studied notions of “action repetition” and general options in hierarchical RL. The high-level approach (self-distillation) is intuitive and interesting.

  2. The experimental evaluation that shows that PACE can appropriately select shorter action chunks to adapt to more stochastic environments is interesting and hopefully will inspire future works to explore additional ways to use uncertainty quantification to determine action chunk length, both in RL and in imitation learning.

Critical weaknesses
  1. The policy architecture and training schemes are complicated (multiple learned embeddings, use of a codebook) and these design decisions are not ablated in the experiments. For example, the decision to discover options using the “codebook bottleneck” (Section 4.2) is not adequately justified. I would like to see ablation experiments in which the “encoder” is removed and the “decoder” is conditioned just on the current state and the (embeddings of) the actions generated in the chunk so far. It would also be good to include in ablations of the other learned embeddings (of actions, of positions indices)- would a simpler architecture in which the “decoder” work well? In my mind the core idea of the paper is “let vanilla RL guide the discovery of useful frozen sequences of actions, and dynamically determine how long to commit to executing them”; the paper should justify every nonstandard technique needed to realize this idea (either with ablation experiments, or with citations to prior RL papers that justify these choices).

  2. The use of formal symbols is not streamlined. For example, the start of Section 3 introduces standard notions of an option as a tuple “on option ω\omega is a tuple (I,πω,β)(I, \pi_{\omega}, \beta)” and of termination condition “beta : S -> [0,1]” but these symbols are used to mean different things later. The paper later says that its notion of “option” is a sequence of primitive actions “\omega = (\omega_0, …, \omega_)$, which is inelegant because you have re-used tuple notation. Additionally, the paper redefines β\beta to characterize the distribution of lengths of option chosen from a particular state. This is a different notion than the β\beta introduced in the “canon” section. In fact, to make PACE’s β\beta fit into the canonical options framework, one would have to consider PACE options to be “semi-markov options” (See discussion of “timeouts” in Section 4 of [1]). But I am not convinced that you actually need to introduce all of the options framework symbols for your paper; you could use a more streamlined notation and merely explain in text the relationship to the options framework. Similarly, the POMDP formalism is not used in the paper (even though the first-person grid experiments are partially observed).

[1]. Sutton, Precup, and Singh. Between MDPs and Semi-MDPs: Learning, Planning, and Representing Knowledge at Multiple Temporal Scales. 1998.

Questions for authors
  1. Additional experiments: is the complicated architecture and use of learned embeddings critical to achieve PACE’s performance? If the authors provide relevant ablations (addressing “Weakness #1”) then I would consider increasing my score.

  2. Clarification: For the codebook, how do you decide how many codes “M” to include? Is this an algorithmic hyperparameter? Or does the learning process determine how many distinct codes are included?

  3. Clarification: What is the role of the “Final objective” (Equation 9)? The preceding prose gave me the impression that each of the different losses were used for disjoint parameters sets. Why do you show them all added together?

Lower priority questions/suggestions:

  1. Clarification: Why did you change from the full observations in “Experiment 1” (Section 6) to first-person, partial observations for “Experiment 2” (Section 7)? I only realized you used two different observation spaces while reading Appendix D.2; you mention “first person” in the lead-in to 7.2 but did not say any details about the observations earlier in section 6. My guess is that this is to keep the same observation space viable for multiple different environments, so that the learned observation encodings can transfer to different environments better. What is your justification? In a revised version of your manuscript you should make the observation space clear for all experiments, and clearly justify why you need each detail of your experimental design to answer your experimental questions.

5.Clarification: It would be good to include [1] in related work.

[1] Helen Zhou, Audrey Huang, Kamyar Azizzadenesheli, Rachel Childers, Zachary C. Lipton. “Timing as an Action: Learning When to Observe and Act.” 2024.

7 comments
AuthorsOpenReview ↗

Response to Reviewer v1D5 (part1)

We thank reviewer v1D5 for the constructive review, for suggesting acceptance with a score of 3 and confidence of 4, and for proposing to increase the score, should Weakness 1 be addressed with the new ablations.

We are grateful that the review recognises that PACE “is an important intermediate between the relatively well-studied notions of “action repetition” and general options in hierarchical RL”, that “The high-level approach (self-distillation) is intuitive and interesting”, and that it is interesting that PACE “can appropriately select shorter action chunks to adapt to more stochastic environments”. We also appreciate the reviewer’s hope that PACE “will inspire future works to explore additional ways to use uncertainty quantification to determine action chunk length”.

We have carefully reviewed the comments, and acknowledge the concerns raised, including the codebook and action/position embeddings ablations. To address these, we have added two new ablations and provided the relevant clarifications both in the revised manuscript and in this rebuttal. We detail each one below.


W1, Q1: I would like to see ablation experiments in which the “encoder” is removed and the “decoder” is conditioned just on the current state and the (embeddings of) the actions generated in the chunk so far.

and

these design decisions are not ablated in the experiments.

and

the decision to discover options using the “codebook bottleneck” (Section 4.2) is not adequately justified.

We have added a new experiment on the DoorKey Ladder and RoomChain that ablates the codebook to better understand its impact on the method. We report the results in Appendix E.2.

Without a codebook, the decoder reads the current state, together with position and action-prefix embeddings. More importantly, when removing the encoder and the codebook, the next actions in an option can no longer be precomputed into a lookup table, so the decoder must run at every position of a generated chunk even at deployment. This greatly increases computation.

We report deployment FLOPs and Wall Clock Time in Tables 10 and 11, Appendix E.1, and the fraction of primitive frames executed through options discovered in previous tasks in Table 12, Appendix E.2. Numbers in Table 11 show the amount of resources PACE uses with respect to PPO (e.g., 0.12 observations means that PACE uses 12% of PPO’s observations). We discuss the codebook ablation in Appendix E.2. We also report the deployment costs below, for your convenience.

Table 10 FLOPs / decision

MethodDK5DK8DO6-N3P2SC11-N5
PPO19,27234,24823,49656,136
PACE51,70596,63364,377162,297
no codebook93,159138,087105,831203,751

Table 10 Wall Clock Time / decision (μs), measured on RTX 5090

MethodDK5DK8DO6-N3P2SC11-N5
PPO113.62 ± 10.41110.86 ± 8.48113.42 ± 11.41108.19 ± 4.32
PACE115.58 ± 11.04112.47 ± 6.65114.86 ± 10.88110.40 ± 5.17

Table 10 Wall Clock Time per episode (ms), measured on RTX 5090

MethodDK5DK8DO6-N3P2SC11-N5
PPO1.077 ± 0.0141.846 ± 0.0191.358 ± 0.0352.048 ± 0.032
PACE0.136 ± 0.0000.304 ± 0.0670.883 ± 0.0400.290 ± 0.021

Table 11 Per-episode ratios relative to PPO

DK5DK8DO6-N3P2SC11-N5
Observations0.12 ± 0.000.16 ± 0.040.64 ± 0.030.14 ± 0.01
FLOPs0.33 ± 0.000.46 ± 0.101.76 ± 0.080.40 ± 0.03
Wall Clock Time0.13 ± 0.000.16 ± 0.040.65 ± 0.030.14 ± 0.01
FLOPs, no codebook0.60 ± 0.010.65 ± 0.152.89 ± 0.140.50 ± 0.03

The ablation shows that adding the codebook, which compresses the options into a finite, discrete set, does not degrade success rates. However, removing the codebook forces the decoder to compute the hmax⁡−1h_{\max} - 1 continuation actions online. This adds a net 41,454 FLOPs per decision across all four tasks because their action spaces and decoder dimensions are identical (Table 10).

AuthorsOpenReview ↗

Response to reviewer v1D5 (part2)

Furthermore, to better understand the effects of the codebook on the quality of the learned options, we run the DoorKey Ladder and RoomChain experiments on PACE with the ablated codebook. We look at the policy at convergence, and measure the share of actions that it takes that are learned options. We report results in Table 12, Appendix E.2, and below.

DoorKey Ladder

LevelWith codebookWithout codebook
DK1060.4 ± 9.354.1 ± 3.2
DK1271.8 ± 1.357.4 ± 5.3
DK1469.6 ± 0.767.3 ± 2.2
DK1677.1 ± 1.769.4 ± 4.1

Room Chain

LevelWith codebookWithout codebook
L277.4 ± 3.957.2 ± 11.1
L368.8 ± 4.163.8 ± 1.3
L478.4 ± 3.369.2 ± 7.5
L578.1 ± 1.274.2 ± 3.1

We observe that removing the codebook reduces the fraction of primitive frames executed through the learned options on both task ladders. This suggests that the codebook fosters the learning of options that are more reusable in these settings.

Overall, the codebook helps both reduce computation and distill options that are more valuable to reuse.


W1, Q1: It would also be good to include in ablations of the other learned embeddings (of actions, of positions indices)- would a simpler architecture in which the “decoder” work well?

and

Additional experiments: is the complicated architecture and use of learned embeddings critical to achieve PACE’s performance? If the authors provide relevant ablations (addressing “Weakness #1”) then I would consider increasing my score.

We have added ablations for both position and action embeddings on the decoder, and measured success rate and compression rate. We report results in Table 13, Appendix E.2, using three seeds and the hyperparameters of Experiment 1. We report them also below for your convenience.

DK5Success rateCompression
PACE (Table 1)1.00 ± 0.008.01 ± 0.00
Without position embedding1.00 ± 0.003.48 ± 0.37
Without action embedding1.00 ± 0.002.01 ± 0.38
Without both embeddings1.00 ± 0.001.86 ± 0.00
DK8Success rateCompression
PACE (Table 1)0.99 ± 0.016.34 ± 0.85
Without position embedding1.00 ± 0.012.59 ± 0.15
Without action embedding0.99 ± 0.012.49 ± 0.11
Without both embeddings0.99 ± 0.022.34 ± 0.40
DO6-N3P2Success rateCompression
PACE (Table 1)0.78 ± 0.021.56 ± 0.07
Without position embedding0.80 ± 0.011.48 ± 0.07
Without action embedding0.80 ± 0.001.42 ± 0.03
Without both embeddings0.78 ± 0.011.47 ± 0.01
SC11-N5Success rateCompression
PACE (Table 1)0.92 ± 0.017.22 ± 0.49
Without position embedding0.89 ± 0.024.13 ± 0.29
Without action embedding0.93 ± 0.004.72 ± 0.32
Without both embeddings0.90 ± 0.014.89 ± 0.28
All environmentsSuccess rateCompression
PACE (Table 1)0.92 ± 0.015.78 ± 0.25
Without position embedding0.92 ± 0.012.92 ± 0.12
Without action embedding0.93 ± 0.002.66 ± 0.13
Without both embeddings0.92 ± 0.012.64 ± 0.12

We observe that, while the success rate doesn’t vary much, removing either embedding degrades the compression rate. This is because, without both embeddings, the decoder receives the same code at every continuation position, so greedy decoding produces the same action repeatedly. This reduces the set of possible options to learn, from the composition of any action to only the repetition of the same action.

AuthorsOpenReview ↗

Response to reviewer v1D5 (part3)


Q2: Clarification: For the codebook, how do you decide how many codes “M” to include? Is this an algorithmic hyperparameter? Or does the learning process determine how many distinct codes are included?

M is a fixed hyperparameter, and we use M = 16 throughout (Appendix D.5), chosen as a reasonable size for a reusable action set. We did not finetune M, and did not open the door to investigate how PACE scales with M. Our intuition suggests that there are interesting scaling laws in there, but this work is already very dense and does not give the space that such an investigation requires. For this reason, we plan to carry on with these investigations for future work, in their own study.

W2: The use of formal symbols is not streamlined.

Thank you for catching the inconsistencies in the notation and the opportunity to simplify the narration. We now introduce options only in prose in Section 3, and use β only for the commitment-length distribution and ω for action sequences. We replaced the unused POMDP formalism with a sentence explaining the partial observation in Experiment 2.

Q3: Clarification: What is the role of the “Final objective” (Equation 9)? The preceding prose gave me the impression that each of the different losses were used for disjoint parameters sets. Why do you show them all added together?

Yes, each loss trains its own parameter group, and we agree that Equation 9 is misleading. We have removed Equation 9 and now state this directly in Section 4. The groups only share a global gradient-norm clip (Appendix E.7).

Q4: Clarification: Why did you change from the full observations in “Experiment 1” (Section 6) to first-person, partial observations for “Experiment 2” (Section 7)?

and

In a revised version of your manuscript you should make the observation space clear for all experiments, and clearly justify why you need each detail of your experimental design to answer your experimental questions.

We have revised the manuscript to include a clear description of the observation space for all experiments in Section 5.1.

To add more details on the change of observation space, the experiment in Section 6 is designed to test whether PACE can reproduce PPO’s behaviour and how much it can compress the required observations and decisions. Full-grid observations let us test this without the additional challenge of partial observability. Section 5.1 now explains this purpose.

In Section 7 we transition to partial observations to keep the same observation space viable across levels within each task ladder, such that we can reuse the same parameters and architecture (i.e., curriculum settings), as the review notes. Section 5.1 now explains that the first-person observations have fixed dimensions within each ladder, allowing the curriculum arms to inherit parameters.

Q5: Clarification: It would be good to include [1] in related work.

We have now included a discussion of [1] in Section 2, including the work in the observation-pricing subset of related work methods.

We would like to thank the reviewer again for the invaluable feedback to improve our manuscript. We have added the requested ablations, and clarified the unclear points both in the rebuttal and the revised manuscript. We would appreciate it if you would increase your score to reflect this, should the new material fill the gaps identified.

Reviewer_28ABOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACE, a method for learning open-loop temporal action sequences, which are called options, alongside standard PPO, so that a deployed agent reads fewer observations. The PPO policy is trained unchanged. A separate option network is trained to predict the next h_max−1 actions of the PPO policy from the initiating state alone. It encodes the state into one of 16 vector-quantized codes and decodes actions autoregressively, trained by forward-KL distillation against the PPO policy's distribution at the states actually reached. A third network is trained to predict where the option's greedy action first disagrees with PPO's greedy action to act as termination function of the options. At deployment the agent commits to as many actions as the predicted probability is above 0.9.

The main caveat is that the savings are measured in observation reads, not computation. Deployment latency and FLOPs are not reported.

Critical strengths
  • The core design is simple and well motivated. Learning the multi-step policy by imitating a one-step teacher, rather than from multi-step returns, sidesteps the difficulty that makes return-based chunking degrade with chunk length. The paper shows that degradation directly for Fixed and SEAR (Fig 5). I like that PACE also recovers PPO exactly which makes it a strict add-on to a standard pipeline.
Critical weaknesses
  • The computational-savings motivation is not substantiated by the experiments reported. The abstract and introduction frame the contribution as reducing inference passes and computation. The metric reported throughout is observation reads, which is not the same thing. Each read runs the base actor, the termination network, the encoder, and h−1 sequential decoder steps. I would either reframe the contribution around observation cost or add an end-to-end inference cost comparison. The closing claim about language models is also not supported by anything in the paper.

  • Aside from deployment latency or FLOPs, which is the gap behind the "fewer inferences" question, the paper also does not report learning curves or sample efficiency to prove whether the learned options are helping. It would also help to show some options actually learned by the proposed method.

Questions for authors
  • Train and deploy mismatch: During training, both the prefix and the reached states come from the PPO policy's sampled actions. At deployment, the prefix is the decoder's own greedy output. Have you measured agreement or termination calibration on greedy rollouts? Does training on greedy-prefix windows change anything?

  • Inference cost: Can you report deployment wall-clock or FLOPs per episode for PPO and PACE, including the sequential decoder steps? Under what per-observation cost does PACE come out ahead?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 28AB (part1)

We thank reviewer 28AB for the valuable feedback and for suggesting acceptance with a score of 3.

We are glad that the review found “the core design is simple and well motivated”, that PACE “sidesteps the difficulty that makes return-based chunking degrade with chunk length”, and that “recovering PPO exactly, which makes it a strict add-on to a standard pipeline”.

We also acknowledge the concerns raised in the review, including the substantiation of the claim on computational savings, and not reporting the learning curves.

Following your comments, we have made the updates described below. To address these, we have added analyses on the computational and observation costs of PACE, reframed the contribution as reducing the number of observations/decisions, included learning curves, and clarified the unclear points. We provide details for each action below.


W1, Q2: The computational-savings motivation is not substantiated by the experiments reported. The abstract and introduction frame the contribution as reducing inference passes and computation. The metric reported throughout is observation reads, which is not the same thing.

and

I would either reframe the contribution around observation cost or add an end-to-end inference cost comparison.

and

Inference cost: Can you report deployment wall-clock or FLOPs per episode for PPO and PACE, including the sequential decoder steps?

We have added a new analysis that helps better understand the costs of running PACE compared to those of PPO at deployment/inference time. We report FLOPs and Wall Clock Time, both per decision and per episode, in Table 10, Appendix E.1, and the corresponding per-episode ratios in Table 2, Section 6.1, with the full ratios in Table 11, Appendix E.1. Wall Clock Time is measured on an RTX 5090.

Table 10 FLOPs / decision

MethodDK5DK8DO6-N3P2SC11-N5
PPO19,27234,24823,49656,136
PACE51,70596,63364,377162,297
no codebook93,159138,087105,831203,751

Table 10 Wall Clock Time / decision (μs), measured on RTX5090

MethodDK5DK8DO6-N3P2SC11-N5
PPO113.62 ± 10.41110.86 ± 8.48113.42 ± 11.41108.19 ± 4.32
PACE115.58 ± 11.04112.47 ± 6.65114.86 ± 10.88110.40 ± 5.17

Table 10 Wall Clock Time per episode (ms), measured on RTX5090

MethodDK5DK8DO6-N3P2SC11-N5
PPO1.077 ± 0.0141.846 ± 0.0191.358 ± 0.0352.048 ± 0.032
PACE0.136 ± 0.0000.304 ± 0.0670.883 ± 0.0400.290 ± 0.021

Table 11 Per-episode ratios relative to PPO

DK5DK8DO6-N3P2SC11-N5
Observations0.12 ± 0.000.16 ± 0.040.64 ± 0.030.14 ± 0.01
FLOPs0.33 ± 0.000.46 ± 0.101.76 ± 0.080.40 ± 0.03
Wall Clock Time0.13 ± 0.000.16 ± 0.040.65 ± 0.030.14 ± 0.01
FLOPs, no codebook0.60 ± 0.010.65 ± 0.152.89 ± 0.140.50 ± 0.03

The tables show that a single PACE decision costs more FLOPs than a PPO decision. However, PACE takes less decisions than PPO, thanks to its multi-step policy, overall making PACE faster to run at deployment in all the environments, and cheaper to run in all environments except DO6-N3P2, due to its stochasticity, as described in Section 6.2.

We have also revised the contribution in the main-text to “reducing the number of observations/decisions”. Section 4 now explains that the greedy continuations are computed once after training and stored in the options table. At deployment, a decision runs the base policy, termination function and encoder, followed by a lookup; there is no sequential decoding at deployment. Section 6.1 now reports the deployment-cost results above.

At training time, PACE is more expensive than PPO alone. Appendix D.7 reports the parameter counts. Appendix E.1 explains the training cost: the distillation loss evaluates the teacher and decoder at every position. The deployment measurements above account for the larger deployed network. This behaviour of more expensive training but cheaper inference is akin to that of Mixture of Experts models in LLM settings, where training costs are traded for inference costs (despite them being completely different methods, and different purposes).

AuthorsOpenReview ↗

Response for reviewer 28AB (part2)


W2: The paper also does not report learning curves or sample efficiency to prove whether the learned options are helping.

We have now reported learning curves for both experiments in Figures 4–6, Appendix A.5.

Figure 4 shows that, measured in primitive frames, PACE converges at nearly the same rate as PPO on three tasks and more slowly on DK8. Figure 5 shows that augmenting the action space with the discovered options improves convergence and reduces variance across seeds compared with the control baselines. Figure 6 shows the same benefit when the discovered options are combined with curriculum learning on the Room Chain.


W2: It would also help to show some options actually learned by the proposed method.

We have added Table 22 to Appendix F.1, listing the eight options added at every level of both task chains, with deployment counts. Examples include LF, RFF and RTFF on DoorKey, and TFF and RFTFF on the Room Chain (L/R: turn left/right; F: forward; T: toggle). These sequences combine turning or toggling with forward movement, so they represent short behaviours that cannot be expressed by repeating a single action.


Q1: Train and deploy mismatch: During training, both the prefix and the reached states come from the PPO policy's sampled actions. At deployment, the prefix is the decoder's own greedy output. Have you measured agreement or termination calibration on greedy rollouts? Does training on greedy-prefix windows change anything?

We have added measurements on greedy deployment rollouts with 32 seeds per task to Appendix E.8. Figure 12 compares agreement with PPO's greedy actions by position under training and greedy prefixes. Table 15 reports measurements on the executed commitments; entries are mean ± standard deviation over 32 seeds.

MeasurementDK5DK8DO6-N3P2SC11-N5
Agreement of executed actions0.999±0.0040.996±0.0080.989±0.0050.976±0.018
Commitments that fully agree0.995±0.0270.994±0.0080.985±0.0080.966±0.025
Predicted survival of commitments0.997±0.0040.975±0.0040.972±0.0020.973±0.004

The termination function is trained to predict the first disagreement between the option policy and the base policy. It shortens unreliable continuations and returns control to the base policy, limiting how long errors from the decoder’s own prefix can accumulate. This limits exposure bias. On greedy deployment rollouts, 97.6–99.9% of the executed actions agree with PPO.

We have not trained on greedy-prefix windows because this would require additional environment interaction to reach the states induced by the decoder’s greedy prefixes. Our approach instead reuses PPO’s collected trajectories and per-state action distributions to distill the option policy.


W1: The closing claim about language models is also not supported by anything in the paper.

We agree that this claim is unsupported by the experiments. We now present it as a potential direction for future work in the conclusion.


We would like to thank the reviewer again for the insightful discussion and the feedback on the manuscript. In light of providing the new evidence requested and the clarifications, we would appreciate it if you would consider increasing your score, should the new results and clarification fill the gap identified in the review.

Reviewer_28ABOpenReview ↗

Response to authors

Thank you for the detailed response and new experiments. They address most of my concerns. A few points for the final version:

Deployment cost. The FLOP and latency results answer my question. Please state in Section 4 that the continuation lookup is exact: the decoder conditions only on the code, prefix and position, so the table has at most M × |A| = 112 entries. The submitted Algorithm 1 still decodes sequentially.

Per-episode wall-clock equals observations × per-decision latency on every task (e.g., DK5: 1.18 × 115.58 μs = 0.136 ms), so it seems computed rather than measured, and excludes environment steps. Also, per-decision latency is nearly the same for PPO and PACE despite 2.7–2.9× the FLOPs, so wall-clock here mainly tracks decision count. The FLOP ratios (0.33–0.46× on deterministic tasks, 1.76× on DO6-N3P2) better reflect compute savings, and I would report those alongside the 6–8× observation reduction. “Fewer forward passes” in the general response should read “fewer decisions”.

Learning curves. Table 7 shows PACE’s base policy uses a different champion configuration from PPO on DK8 and DO6-N3P2. Does this explain the slower DK8 convergence? If so, please note it, and clarify that agreement in Table 15 is with PACE’s own base policy.

Remaining points. The greedy-rollout agreement (97.6–99.9%), the option examples in Table 22, and moving the language-model claim to future work resolve my other concerns.

If these clarifications go into the final version, I will raise my soundness score to 3 and keep my recommendation for acceptance.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 28AB

We thank reviewer 28AB for the detailed follow-up and for confirming that the new experiments and revisions address the other concerns. We are glad that the reviewer found the FLOP and latency results helpful in answering the question about deployment cost.

We also acknowledge the remaining points concerning the exact continuation lookup, the interpretation of policy inference time, and the learning-curve comparison.

Following your comments, we have clarified these points in the revised manuscript. The changes made in response to this follow-up are marked in blue. We provide the details below.


Deployment cost. The FLOP and latency results answer my question. Please state in Section 4 that the continuation lookup is exact: the decoder conditions only on the code, prefix and position, so the table has at most M × |A| = 112 entries. The submitted Algorithm 1 still decodes sequentially.

We have now clarified the lookup procedure in Section 4 and Algorithm 1 in Appendix B. Under greedy deployment, the action prefix develops deterministically, so the code and first action uniquely determine the maximal continuation. The M∣A∣M|\mathcal{A}| continuations are computed once after training and stored in a table. With M=16M=16 and ∣A∣=7|\mathcal{A}|=7, the table has 112112 entries. Algorithm 1 now constructs T[e,a]T[e,a] before the deployment loop and reads the stored continuation inside the loop instead of decoding it sequentially.


Deployment cost. Per-episode wall-clock equals observations × per-decision latency on every task (e.g., DK5: 1.18 × 115.58 μs = 0.136 ms), so it seems computed rather than measured, and excludes environment steps. Also, per-decision latency is nearly the same for PPO and PACE despite 2.7–2.9× the FLOPs, so wall-clock here mainly tracks decision count. The FLOP ratios (0.33–0.46× on deterministic tasks, 1.76× on DO6-N3P2) better reflect compute savings, and I would report those alongside the 6–8× observation reduction. “Fewer forward passes” in the general response should read “fewer decisions”.

Table 10 (Appendix E.1) measures per-decision Policy Inference Time after JIT compilation and warm-up, then aggregates episode values with Table 1's mean decision counts. We use this protocol because direct timing does not isolate policy-side cost at either scale: fixed overhead masks PACE's 2.72.7--2.9×2.9\times per-decision FLOPs, while environment and system overhead hides the effect of PPO making up to 8×8\times as many decisions in end-to-end episode time. The aggregate isolates cumulative policy-call time but mainly follows decision count because per-decision latencies are similar.

To clarify this, we have renamed “Wall Clock Time” to “Policy Inference Time”, marked decision-level values as “measured” and episode-level values as “aggregated”, and stated in the Table 10 caption that environment execution is excluded. Observation reduction remains the core contribution; calculated policy-side FLOPs support the main experiment in Table 2, while timing is only supplementary implementation information in Table 10. We have also replaced “fewer forward passes” with “fewer observations and decisions” in the General Response.


Learning curves. Table 7 shows PACE’s base policy uses a different champion configuration from PPO on DK8 and DO6-N3P2. Does this explain the slower DK8 convergence? If so, please note it, and clarify that agreement in Table 15 is with PACE’s own base policy.

We cannot clearly attribute the slower convergence on DK8 to either the hyperparameters or the method. Each method uses its own champion configuration on each task, selected by the highest end-of-training success rate in a 2424-configuration, 33-seed sweep. The current Table 8 in Appendix D.6 reports these configurations. We have revised the caption of Figure 4 to state that every curve uses the method's own champion configuration and to point to Table 8.

The agreement analysis compares PACE's decoded actions with the greedy actions of PACE's own base policy at the reached states. It does not compare PACE with the separate PPO arm in Table 1. We have made this explicit in Figure 12 and the current Table 14 in Appendix E.8, and updated the training-prefix legend accordingly.


Remaining points. The greedy-rollout agreement (97.6–99.9%), the option examples in Table 22, and moving the language-model claim to future work resolve my other concerns.

We thank the reviewer for confirming that these additions and revisions resolve the other concerns. They remain reported in Table 14, Appendix E.8; Table 21, Appendix F.1; and Section 8, respectively.


We would like to thank the reviewer again for the careful reading and for indicating a willingness to raise the soundness score. We hope that the explanations and revisions above address the remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their careful and constructive reviews. We summarise below the strengths they identified, their shared concerns, and our revisions.

We are glad that the reviewers found the core design of PACE "simple and well motivated" (28AB), and that PACE "sidesteps the difficulty that makes return-based chunking degrade with chunk length" (28AB), making it "a strict add-on to a standard pipeline" (28AB). The reviewers also saw PACE as "an important intermediate between the relatively well-studied notions of 'action repetition' and general options in hierarchical RL", with a self-distillation approach that is "intuitive and interesting" (v1D5), hoping it "will inspire future works to explore additional ways to use uncertainty quantification to determine action chunk length" (v1D5). We are glad that reviewer MQVC noted “relevant results within the evaluated setting” with "substantial observation reductions without the large success-rate losses suffered by several chunking baselines" and "a substantial benefit from combining options with parameter transfer" on the Room Chain (MQVC).

The reviewers raised two shared concerns. First, the fact that fewer observations do not by themselves mean less computation (MQVC, 28AB). A PACE decision runs more networks than a PPO forward pass, and the paper reported no deployment FLOPs or latency. Second, the comparisons and design choices needed more support. MQVC asked for a comparison with temporal distillation (Avé et al., 2025) and noted that we compare against an adapted version of SEAR. v1D5 asked us to justify the codebook and the learned embeddings with ablations. The reviewers also asked for clearer notation and observation spaces (v1D5), and for direct evidence that the learned options help, including examples of them (28AB).

We made the following revisions:

  • Deployment cost (MQVC, 28AB). We added policy-side FLOPs and Policy Inference Time for PACE and PPO (Table 2, Section 6.1; Table 10, Appendix E.1). Observation reduction is the core contribution, FLOPs support the main comparison, and timing is supplementary implementation information. Section 4 and Algorithm 1 clarify the exact continuation lookup.
  • New baselines and related work (MQVC, v1D5). We added the baselines requested, temporal distillation (TempDistill), and observation pricing (ObsPrice) to Table 1, with their implementation details in Appendices C.6 and C.7. At equal or higher success rates, PACE compresses observations by 1.56–8.01×, against at most 1.81× for either baseline. Section 2 now discusses Avé et al. (2025) and Zhou et al. (2024).
  • Original SEAR (MQVC). Table 1 now reports the original SEAR and labels the PPO version SEAR†; Appendix C.3 explains the adaptation. The original SEAR has far lower success rates than the adapted baseline.
  • Architecture ablations (v1D5). We have added two new experiments ablating the codebook and the position and action embeddings in the option policy decoder. Removing the codebook substantially increases FLOPs per decision because continuations can no longer be precomputed into a table. It also lowers the share of actions executed through discovered options on both task ladders (Table 11), suggesting that the options learned without the codebook are less reusable. Removing the action or position embeddings (Table 12) leaves the average success rate at 0.92–0.93, but compression falls from 5.78× to 2.64–2.92×. Without either embedding, the decoder receives the same code at every continuation position, so greedy decoding produces the same action repeatedly.
  • Learning curves (28AB). We added learning curves for both experiments (Figures 4–6). Results show that augmenting the action space with the discovered options speeds up convergence and reduces variance across seeds. Table 21 (Appendix F.1) lists the options PACE discovers at each level of both task ladders. On greedy deployment rollouts (Appendix E.8, Table 14), 97.6–99.9% of executed actions agree with PACE's own base policy.
  • Presentation (v1D5, 28AB). We streamlined the notation: options are now introduced in prose, β is used only for the commitment-length distribution and ω only for action sequences, and the unused POMDP formalism is removed. We removed Equation 9, since each loss trains its own parameters. Section 5.1 now states and justifies the observation space of every experiment. We state that M = 16 is a fixed hyperparameter. The claim about language models is now presented as a direction for future work.

We hope these changes address the reviewers' concerns and would be grateful if they would consider raising their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.