- Rating
- 2(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PACE (Per-state Adaptive Commitment) aims to reduce how often an RL agent must observe its environment during execution. The idea is that, in predictable situations, one observation may contain enough information to choose several consecutive actions. PACE trains a standard PPO policy and uses its recorded experience to train two additional components alongside it through distillation. The first learns to predict the actions PPO would choose over the next few steps, using only the starting observation. The second predicts how many actions the agent can execute before those predictions are likely to disagree with PPO. Experiments, primarily on MiniGrid tasks, report up to 8× fewer observations while maintaining success rates close to PPO’s. The paper also reuses discovered action sequences as additional actions when learning harder tasks. Combining these options with curriculum learning increases success on the final Room Chain level from 30.3% to 85.0%.
Critical strengths
A useful separation of learning objectives.
PPO learns to solve the task by choosing one action at a time and observing after each step. Alongside it, PACE’s sequence predictor learns to imitate PPO’s future action choices using only the starting observation. PPO’s predictions provide specific teaching targets for learning these sequences. This is an appealing design because it allows PPO to learn the task without requiring it to explore through long action sequences executed without intermediate observations.
Relevant results within the evaluated setting.
Table 1 shows substantial observation reductions without the large success-rate losses suffered by several chunking baselines. The Room Chain results also demonstrate a substantial benefit from combining options with parameter transfer.
Critical weaknesses
W1 Observation savings do not establish computational savings.
This distinction matters because reduced inference cost is a central motivation. PACE evaluates three branches and generates actions autoregressively: one observation therefore does not correspond to the same computation as one PPO forward pass. Appendix D.7 reports 3.3× more deployment parameters, while Appendix E.1 reports 132 seconds of training versus 11 seconds for PPO on DK8. The paper provides no deployment latency or FLOP measurements.
W2 Comparisons with related methods must be strengthened.
The SEAR baseline is adapted to PPO, changing its critic and training procedure. The authors explain that their implementation of original SEAR failed on these sparse-reward tasks, which provides a reasonable justification for the adaptation. Nevertheless, the reported comparison primarily establishes an advantage over the adapted version only. The paper should also discuss temporal distillation (Avé et al., 2025), which learns to predict an action and its repetition length from a teacher policy. PACE extends this idea to sequences containing different actions, and a direct comparison would help establish the benefit of that additional flexibility. These gaps limit the strength of the comparative claims.
Questions for authors
Q1 : Is PACE faster than PPO after training? Please compare their execution times on the same tasks and hardware, including the time spent running all parts of PACE.
Q2 : How does PACE compare with the earlier temporal-distillation method? That method learns to repeat one action several times. A comparison on the same tasks would help show the benefit of PACE’s ability to predict sequences containing different actions.
5 comments
Response to reviewer MQVC (part2)
W2: The SEAR baseline is adapted to PPO, changing its critic and training procedure. The authors explain that their implementation of original SEAR failed on these sparse-reward tasks, which provides a reasonable justification for the adaptation. Nevertheless, the reported comparison primarily establishes an advantage over the adapted version only.
We agree that the original manuscript did not provide direct evidence that the adapted SEAR outperforms the original formulation. We now label the PPO port as SEAR† and describe the adaptation and its rationale in Appendix C.3.
To justify the adaptation we have also included the pre-port SEAR baseline as described in the original paper in Table 1. Results show that the original, pre-port SEAR achieves 7.5% ± 4.7% success rate on average on all environments. We report below the results with SEAR, the SEAR† port, and PACE for your convenience.
| DK5 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| SEAR | 0.04 ± 0.10 | 1.59 ± 0.24× | 28.85 ± 4.56 | 6.07 ± 0.91 | 4.75 ± 0.20 |
| SEAR† | 1.00 ± 0.00 | 3.60 ± 0.38× | 18.79 ± 1.46 | 2.66 ± 0.26 | 7.08 ± 0.29 |
| PACE (Table 1) | 1.00 ± 0.00 | 8.01 ± 0.00× | 9.43 ± 0.00 | 1.18 ± 0.00 | 7.97 ± 0.00 |
| DK8 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| SEAR | 0.00 ± 0.00 | 0.83 ± 0.01× | 100.00 ± 0.00 | 20.00 ± 0.00 | 5.00 ± 0.00 |
| SEAR† | 0.51 ± 0.17 | 2.79 ± 0.16× | 38.56 ± 2.45 | 6.00 ± 0.34 | 6.43 ± 0.07 |
| PACE (Table 1) | 0.99 ± 0.01 | 6.34 ± 0.85× | 16.56 ± 0.11 | 2.70 ± 0.60 | 6.30 ± 0.83 |
| DO6-N3P2 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| SEAR | 0.26 ± 0.16 | 2.59 ± 0.60× | 22.25 ± 6.31 | 4.90 ± 1.29 | 4.44 ± 0.18 |
| SEAR† | 0.63 ± 0.02 | 1.05 ± 0.06× | 11.49 ± 0.72 | 11.49 ± 0.72 | 1.00 ± 0.00 |
| PACE (Table 1) | 0.78 ± 0.02 | 1.56 ± 0.07× | 12.04 ± 0.30 | 7.69 ± 0.35 | 1.57 ± 0.04 |
| SC11-N5 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| SEAR | 0.00 ± 0.00 | 0.47 ± 0.01× | 200.00 ± 0.00 | 40.00 ± 0.00 | 5.00 ± 0.00 |
| SEAR† | 0.22 ± 0.10 | 8.40 ± 1.16× | 21.81 ± 3.48 | 2.30 ± 0.37 | 9.47 ± 0.14 |
| PACE (Table 1) | 0.92 ± 0.01 | 7.22 ± 0.49× | 18.90 ± 0.20 | 2.63 ± 0.19 | 7.21 ± 0.48 |
| All environments | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| SEAR | 0.08 ± 0.05 | 1.37 ± 0.16× | 87.78 ± 1.95 | 17.74 ± 0.39 | 4.80 ± 0.07 |
| SEAR† | 0.59 ± 0.05 | 3.96 ± 0.31× | 22.66 ± 1.14 | 5.61 ± 0.23 | 6.00 ± 0.08 |
| PACE (Table 1) | 0.92 ± 0.01 | 5.78 ± 0.25× | 14.23 ± 0.09 | 3.55 ± 0.18 | 5.76 ± 0.24 |
Table 1 (b). Success rate, compression, steps to goal, observations per episode, and commitment length of SEAR, the SEAR† port, and PACE on the tasks of Table 1; mean ± standard deviation over seeds. All environments: mean across the four tasks.
Response to reviewer MQVC (part3)
W2, Q2: The paper should also discuss temporal distillation (Avé et al., 2025), which learns to predict an action and its repetition length from a teacher policy. PACE extends this idea to sequences containing different actions, and a direct comparison would help establish the benefit of that additional flexibility.
and
How does PACE compare with the earlier temporal-distillation method? That method learns to repeat one action several times. A comparison on the same tasks would help show the benefit of PACE’s ability to predict sequences containing different actions.
We have added a discussion of Avé et al. (2025) to Section 2 and results for temporal distillation (TempDistill) and observation pricing (ObsPrice) to Table 1. The implementation details of both are in new Appendices C.6 and C.7. We report below the results with only TempDistill, ObsPrice and PACE for your convenience, using 32 seeds.
| DK5 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| TempDistill | 1.00 ± 0.00 | 1.20 ± 0.06× | 9.44 ± 0.06 | 7.89 ± 0.45 | 1.20 ± 0.06 |
| ObsPrice | 0.98 ± 0.09 | 1.81 ± 0.19× | 9.42 ± 0.13 | 5.29 ± 0.54 | 1.80 ± 0.19 |
| PACE (Table 1) | 1.00 ± 0.00 | 8.01 ± 0.00× | 9.43 ± 0.00 | 1.18 ± 0.00 | 7.97 ± 0.00 |
| DK8 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| TempDistill | 0.90 ± 0.25 | 1.60 ± 0.11× | 16.66 ± 0.22 | 10.45 ± 0.75 | 1.60 ± 0.10 |
| ObsPrice | 0.99 ± 0.02 | 1.11 ± 0.05× | 16.58 ± 0.12 | 15.04 ± 0.66 | 1.10 ± 0.05 |
| PACE (Table 1) | 0.99 ± 0.01 | 6.34 ± 0.85× | 16.56 ± 0.11 | 2.70 ± 0.60 | 6.30 ± 0.83 |
| DO6-N3P2 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| TempDistill | 0.78 ± 0.01 | 1.17 ± 0.06× | 11.94 ± 0.47 | 10.27 ± 0.50 | 1.16 ± 0.02 |
| ObsPrice | 0.77 ± 0.02 | 1.08 ± 0.04× | 11.46 ± 0.33 | 11.13 ± 0.43 | 1.03 ± 0.03 |
| PACE (Table 1) | 0.78 ± 0.02 | 1.56 ± 0.07× | 12.04 ± 0.30 | 7.69 ± 0.35 | 1.57 ± 0.04 |
| SC11-N5 | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| TempDistill | 0.90 ± 0.02 | 1.03 ± 0.16× | 19.08 ± 0.51 | 18.57 ± 1.74 | 1.04 ± 0.16 |
| ObsPrice | 0.89 ± 0.08 | 1.22 ± 0.11× | 19.07 ± 0.37 | 15.70 ± 1.46 | 1.22 ± 0.11 |
| PACE (Table 1) | 0.92 ± 0.01 | 7.22 ± 0.49× | 18.90 ± 0.20 | 2.63 ± 0.19 | 7.21 ± 0.48 |
| All environments | Success rate ↑ | Compression ↑ | Steps to goal | Observations per episode ↓ | Commitment length |
|---|---|---|---|---|---|
| TempDistill | 0.90 ± 0.06 | 1.25 ± 0.05× | 14.28 ± 0.18 | 11.80 ± 0.50 | 1.25 ± 0.05 |
| ObsPrice | 0.91 ± 0.03 | 1.31 ± 0.06× | 14.13 ± 0.13 | 11.79 ± 0.44 | 1.29 ± 0.06 |
| PACE (Table 1) | 0.92 ± 0.01 | 5.78 ± 0.25× | 14.23 ± 0.09 | 3.55 ± 0.18 | 5.76 ± 0.24 |
Table 1 (c). Success rate, compression, steps to goal, observations per episode, and commitment length of the two new baselines and PACE on the tasks of Table 1; mean ± standard deviation over seeds. All environments: mean across the four tasks.
We would like to thank the reviewer again for the useful feedback and would be grateful if the reviewer would reconsider their score to suggest acceptance, should the new results and clarification fill the gap identified in the review.
Response to Reviewer MQVC (part1)
We thank reviewer MQVC for the review and for the helpful feedback to improve the manuscript.
We are grateful to read that the review recognises that PACE “is an appealing design because it allows PPO to learn the task without requiring it to explore through long action sequences executed without intermediate observations”, that PACE “shows substantial observation reductions without the large success-rate losses suffered by several chunking baselines”, and that “The Room Chain results also demonstrate a substantial benefit from combining options with parameter transfer.”
We also understand the concerns raised, and we have addressed each of them below.
and
We have added a new analysis that helps better understand the costs of running PACE compared to those of PPO at deployment/inference time. We report FLOPs and Wall Clock Time, both per decision and per episode, in Table 10, Appendix E.1, and the corresponding per-episode ratios in Table 2, Section 6.1, with the full ratios in Table 11, Appendix E.1. Wall Clock Time is measured on an RTX 5090.
Table 10 FLOPs / decision
Table 10 Wall Clock Time / decision (μs), measured on RTX5090
Table 10 Wall Clock Time per episode (ms), measured on RTX5090
Table 11 Per-episode ratios relative to PPO
The tables show that a single PACE decision costs more FLOPs than a PPO decision. However, PACE takes less decisions than PPO, thanks to its multi-step policy, overall making PACE faster to run at deployment in all the environments, and cheaper to run in all environments except DO6-N3P2, due to its stochasticity, as described in Section 6.2.
We have also revised the contribution in the main-text to “reducing the number of observations/decisions”. Section 4 now explains that the greedy continuations are computed once after training and stored in the options table. At deployment, a decision runs the base policy, termination function and encoder, followed by a lookup; there is no sequential decoding at deployment. Section 6.1 now reports the deployment-cost results above.
At training time, PACE is more expensive than PPO alone. Appendix D.7 reports the parameter counts. Appendix E.1 explains the training cost: the distillation loss evaluates the teacher and decoder at every position. The deployment measurements above account for the larger deployed network. This behaviour of more expensive training but cheaper inference is akin to that of Mixture of Experts models in LLM settings, where training costs are traded for inference costs (despite them being completely different methods, and different purposes).