- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CAPI formulates actor-loss weighting as an entropy-regularized allocation problem with a cap on each sample's normalized weight share. It derives capped-exponential weights and applies a tempered, clipped version as detached gates in TD-MPC2. The method also uses conservative critic scores and, on HumanoidBench, planner-likelihood regularization. Experiments cover 18 control tasks with five seeds.
Critical strengths
-
Bounding normalized shares gives capacity control a precise meaning that raw-weight clipping does not provide. The allocation derivation is clear. The paper also explicitly limits its policy-improvement theorem to an ideal statewise operator, rather than claiming a guarantee for each implemented actor step.
-
The results extend across a substantial set of difficult tasks. Average improvements over TD-M(PC)² are about 18.6% on DMControl and 21.0% on HumanoidBench, with particularly large gains on some tasks. The breadth of these results supports a weak acceptance, although the mechanism needs cleaner isolation.
Critical weaknesses
-
The allocation comparison changes both relative weights and total actor-loss scale. Table 2 reports mean gates around 0.12–0.19 for CAPI, while normalized exponential and uniform weights average 1. Under Eq. (13), this leaves an important confound. It may also matter for the κ=∞ ablation, whose practical gates include both capacity-dependent scaling and a clipping floor.
-
The full method's gains are not yet cleanly attributable to capacity. The existing allocation and planner-regularization ablations are useful, but planner regularization changes H1hand-walk return from 331.8 to 934.9. Also, the unweighted Dog-run curve is substantially above the main-table TD-MPC2 result. This may reflect other retained components or configuration differences; it needs clarification before that comparison can establish the capacity rule's contribution.
Questions for authors
-
Clarification: How is κ=∞ implemented in Figure 5(a), and what gate values result? Is it equivalent to the unweighted variant up to an overall scale?
-
Clarification: What differs between the unweighted ablation and main-table TD-MPC2? Were the principal baselines rerun under the same seeds and evaluation protocol?
-
Targeted check: On the existing ablation tasks, does capacity still outperform the other allocation rules when all variants have the same mean weight and share the same scores and auxiliary terms? This is the check most likely to change my recommendation.
3 comments
Thank you for your comprehensive response. I have raised my score to 4.
Thank you for the updated score and helpful suggestions. We have revised the experimental section accordingly. We greatly appreciate your time and valuable input in improving our work.
We appreciate the reviewer’s careful assessment and constructive feedback. We have updated the manuscript, with the revisions highlighted for ease of reference.
1. Isolating the Contribution of Capacity
Response. We reran the allocation ablation with uniform, exponential, and UWAC controls fixed to CAPI’s reference mean weights of 0.36 on Dog-run and 0.23 on H1hand-walk. We added UWAC weighting following another reviewer’s suggestion. CAPI retains its original batch-dependent gate scale. All variants share the model architecture, critic update, planner, and per-sample actor objective within each task. The exponential control uses the same scores and weight temperature as CAPI. Weights are fixed during backpropagation and scale the full per-sample actor loss, including the optional planner-likelihood term.
We have revised Section 5.3 to describe these controls. Figure 5(b) reports the following returns at one million environment steps, with mean ± sample SD across training runs.
We also control for planner guidance. In Figure 5(b), the planner-likelihood coefficient is fixed at λ=0 on Dog-run and λ=1 on H1hand-walk across all weighting rules. The complementary ablation in Appendix B.1 (Table 3 and Figure 6) retains CAPI weighting and varies λ to evaluate the role of planner-likelihood regularization.
2. Implementation and Interpretation of the Infinite-Capacity Variant
Response. Thank you for asking us to clarify this point. Increasing κ relaxes the upper bound on each sample's allocation. For a batch of n samples, this bound becomes redundant once κ≥n, since κ/n≥1. For fixed finite scores, the normalization threshold then stays constant, while the practical gate still contains −logκ. Increasing κ further lowers the log weights until every gate reaches the floor e−ℓmax, which is approximately 0.0498 under our default setting. Thus, reaching κ=n removes the allocation constraint and the common-floor limit requires sufficiently large κ. In that limit, the complete actor loss is a constant multiple of the unweighted loss within the same framework. Without the floor, the gates would instead approach zero while retaining exponential relative weights.
We had already recognized the limiting behavior described above and intended to exclude κ=∞ from the comparison, but its curve was inadvertently included in the figure. We have removed it from revised Figure 5(a), while leaving the discussion of the finite-capacity ablation unchanged.
3. Baseline Configurations and Evaluation Protocol
Response. Uniform assigns the same constant weight to every sample and shares TD-MPC2’s world model and planning framework. In our implementation, replay is updated with recent trajectory fragments during data collection, as described in Section 4.3, while the reference TD-MPC2 implementation adds trajectories after an episode ends. The uniform control retains the other CAPI training choices, so the allocation ablation compares weighting rules within a common implementation.
We compare all algorithms using episode returns at the 1M-step training budget, with the same training seed IDs 0,1,2,3,4. For TD-MPC2, SAC, and DreamerV3, we combine our own runs with publicly released runs. Results for TD-M(PC)² and CAPI are based on our own runs. Publicly released runs retain their original evaluation settings. Source repository links are provided in Appendix B.4.
Thank you again for the constructive feedback.