acceptodds
Under review as a conference paper at ICLR 2027

Capacity-Aware Policy Improvement for Model-Based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) can combine a learned policy with online planning for data-efficient continuous control. However, in planner-guided learning, the planner collects experience, whereas actor updates evaluate actions sampled from the nominal policy. This policy-planner mismatch can expose actor updates to inaccurate value estimates for actions poorly covered by replay data. Critic-based weighting can prioritize promising actions, but concentrating weight on a few samples can amplify the influence of errors in their value estimates. We propose Capacity-Aware Policy Improvement (CAPI), which formulates weighting within a finite actor batch as an entropy-regularized allocation problem with an explicit capacity constraint. The resulting allocation reweights the TD-MPC2 actor objective, prioritizing high-scoring actions while limiting the normalized weight share assigned to each sample. Experiments on high-dimensional continuous-control tasks from the DeepMind Control Suite and HumanoidBench demonstrate that CAPI improves data efficiency and final performance compared to strong baselines.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, continuous control, model predictive control, entropy-regularized allocation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_xQrSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAPI formulates actor-loss weighting as an entropy-regularized allocation problem with a cap on each sample's normalized weight share. It derives capped-exponential weights and applies a tempered, clipped version as detached gates in TD-MPC2. The method also uses conservative critic scores and, on HumanoidBench, planner-likelihood regularization. Experiments cover 18 control tasks with five seeds.

Critical strengths
  1. Bounding normalized shares gives capacity control a precise meaning that raw-weight clipping does not provide. The allocation derivation is clear. The paper also explicitly limits its policy-improvement theorem to an ideal statewise operator, rather than claiming a guarantee for each implemented actor step.

  2. The results extend across a substantial set of difficult tasks. Average improvements over TD-M(PC)² are about 18.6% on DMControl and 21.0% on HumanoidBench, with particularly large gains on some tasks. The breadth of these results supports a weak acceptance, although the mechanism needs cleaner isolation.

Critical weaknesses
  1. The allocation comparison changes both relative weights and total actor-loss scale. Table 2 reports mean gates around 0.12–0.19 for CAPI, while normalized exponential and uniform weights average 1. Under Eq. (13), this leaves an important confound. It may also matter for the κ=∞ ablation, whose practical gates include both capacity-dependent scaling and a clipping floor.

  2. The full method's gains are not yet cleanly attributable to capacity. The existing allocation and planner-regularization ablations are useful, but planner regularization changes H1hand-walk return from 331.8 to 934.9. Also, the unweighted Dog-run curve is substantially above the main-table TD-MPC2 result. This may reflect other retained components or configuration differences; it needs clarification before that comparison can establish the capacity rule's contribution.

Questions for authors
  1. Clarification: How is κ=∞ implemented in Figure 5(a), and what gate values result? Is it equivalent to the unweighted variant up to an overall scale?

  2. Clarification: What differs between the unweighted ablation and main-table TD-MPC2? Were the principal baselines rerun under the same seeds and evaluation protocol?

  3. Targeted check: On the existing ablation tasks, does capacity still outperform the other allocation rules when all variants have the same mean weight and share the same scores and auxiliary terms? This is the check most likely to change my recommendation.

3 comments

We appreciate the reviewer’s careful assessment and constructive feedback. We have updated the manuscript, with the revisions highlighted for ease of reference.

1. Isolating the Contribution of Capacity

On the existing ablation tasks, does capacity still outperform the other allocation rules when all variants have the same mean weight and share the same scores and auxiliary terms?

Response. We reran the allocation ablation with uniform, exponential, and UWAC controls fixed to CAPI’s reference mean weights of 0.36 on Dog-run and 0.23 on H1hand-walk. We added UWAC weighting following another reviewer’s suggestion. CAPI retains its original batch-dependent gate scale. All variants share the model architecture, critic update, planner, and per-sample actor objective within each task. The exponential control uses the same scores and weight temperature as CAPI. Weights are fixed during backpropagation and scale the full per-sample actor loss, including the optional planner-likelihood term.

We have revised Section 5.3 to describe these controls. Figure 5(b) reports the following returns at one million environment steps, with mean ± sample SD across training runs.

Weighting ruleDog-runH1hand-walk
CAPI669.1 ± 58.0929.9 ± 6.5
Uniform weighting438.3 ± 125.8895.2 ± 16.3
Exponential weighting481.4 ± 131.3870.9 ± 33.8
UWAC weighting638.6 ± 15.0829.0 ± 14.6

We also control for planner guidance. In Figure 5(b), the planner-likelihood coefficient is fixed at λ=0\lambda=0 on Dog-run and λ=1\lambda=1 on H1hand-walk across all weighting rules. The complementary ablation in Appendix B.1 (Table 3 and Figure 6) retains CAPI weighting and varies λ\lambda to evaluate the role of planner-likelihood regularization.

2. Implementation and Interpretation of the Infinite-Capacity Variant

How is κ=∞ implemented in Figure 5(a), and what gate values result? Is it equivalent to the unweighted variant up to an overall scale?

Response. Thank you for asking us to clarify this point. Increasing κ\kappa relaxes the upper bound on each sample's allocation. For a batch of nn samples, this bound becomes redundant once κ≥n\kappa\geq n, since κ/n≥1\kappa/n\geq1. For fixed finite scores, the normalization threshold then stays constant, while the practical gate still contains −log⁡κ-\log\kappa. Increasing κ\kappa further lowers the log weights until every gate reaches the floor e−ℓmax⁡e^{-\ell_{\max}}, which is approximately 0.0498 under our default setting. Thus, reaching κ=n\kappa=n removes the allocation constraint and the common-floor limit requires sufficiently large κ\kappa. In that limit, the complete actor loss is a constant multiple of the unweighted loss within the same framework. Without the floor, the gates would instead approach zero while retaining exponential relative weights.

We had already recognized the limiting behavior described above and intended to exclude κ=∞\kappa=\infty from the comparison, but its curve was inadvertently included in the figure. We have removed it from revised Figure 5(a), while leaving the discussion of the finite-capacity ablation unchanged.

3. Baseline Configurations and Evaluation Protocol

What differs between the unweighted ablation and main-table TD-MPC2? Were the principal baselines rerun under the same seeds and evaluation protocol?

Response. Uniform assigns the same constant weight to every sample and shares TD-MPC2’s world model and planning framework. In our implementation, replay is updated with recent trajectory fragments during data collection, as described in Section 4.3, while the reference TD-MPC2 implementation adds trajectories after an episode ends. The uniform control retains the other CAPI training choices, so the allocation ablation compares weighting rules within a common implementation.

We compare all algorithms using episode returns at the 1M-step training budget, with the same training seed IDs 0,1,2,3,4\\{0,1,2,3,4\\}. For TD-MPC2, SAC, and DreamerV3, we combine our own runs with publicly released runs. Results for TD-M(PC)² and CAPI are based on our own runs. Publicly released runs retain their original evaluation settings. Source repository links are provided in Appendix B.4.

Thank you again for the constructive feedback.

Thank you for your comprehensive response. I have raised my score to 4.

Thank you for the updated score and helpful suggestions. We have revised the experimental section accordingly. We greatly appreciate your time and valuable input in improving our work.

Reviewer_ypMGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a novel reweighting technique in model-based RL to counter policy-planner mismatch. This is done by avoiding the amplification of errors by solely reweighting updates with the critic. Instead, this work introduces a technique for weighting updates as an allocation problem using entropy regularized capacity constraints. Altogether, this places more emphasis on high value state-action pairs relative to the action's proposal density. This new reweighting technique is then deployed within an algorithmic framework derived from the prior TD-MPC algorithm. Experimental results on DeepMindControl and HumanoidBench demonstrate the efficacy of the CAPI framework, showing notable performance improvement.

Critical strengths

The choice of treating the weighting of the batch as an allocation problem was informative and I believe to be an appropriate use of the planner. This capacity-aware reweighting of informative to, possibly, erroneous updates is novel as far as I am aware.

I found the empirical analysis of CAPI to be convincing. While there are some experiments and/or analyses that I feel are lacking (see below) the range of ablation studies help to augment and improve upon the results presented in Section 5.

Critical weaknesses

My primary complaint about the technical presentation of this work is the depth and purpose of the ablations and complementary analyses presented in the paper. The ablations shared in Figure 5 are a good start but I still am left wondering whether or how well CAPI truly addresses policy-planner mismatch. This is a core motivation for contemporary MBRL but there is zero analysis of whether the proposed approaches are actually addressing the problem. Sure the benchmark scores improve with CAPI but that doesn't necessarily convey that the source of the conceptual problem is being solved. More development and analysis that directly addresses the major frame of reference for this work would greatly strengthen the paper.

Second, the presentation in Section 4 is quite difficult to get through. There is fairly little attention paid to how the reader is supposed to connect concepts and understand the core technical development that underlies the proposed technique. Section 4.2 reads as a bulleted list in paragraph form with little connectivity between concepts and ideas. There's no thread that motivates the development or justifies the choices of each. I felt that Theorems 1 and 2 weren't connected at all and were unnecessary at worst. The paper wouldn't be changed if they weren't included at all. Part of this is a lack of concrete justification or motivation of the underlying concepts to develop CAPI which leaves me with the impression that much of Section 4.2 is written to levy an element of "mathiness" which serves to muddle the presentation of an otherwise strong set of ideas in the development of CAPI. As a note, there is zero mention of how how the capacity thresholds are actually computed in the main body of the paper. Sure there's some reference to an optimization that needs to be done in Theorem 1 but this isn't helpful for the audience of this work. A vague reference to the algorithm used and a Section in the appendix is not sufficient.

Questions for authors

None that go above or beyond my comments.

3 comments

We thank the reviewer for the thoughtful feedback and helpful suggestions. The revisions are highlighted in the updated manuscript.

1. Addressing Policy–Planner Mismatch

Sure the benchmark scores improve with CAPI but that doesn't necessarily convey that the source of the conceptual problem is being solved.

Response. We appreciate your comment, which gets to the heart of the policy–planner mismatch motivating our work. We have added a controlled evaluation of the learned actor's contribution to planning.

Planner-likelihood regularization reduces the distance to the stored planner reference from 8.239 to 1.440 on H1hand-walk (Appendix B.1, Table 3). In a separate comparison at frozen checkpoints, capacity reduces the maximum sample share from 2.31% to 1.12% on Dog-run and from 16.06% to 1.80% on H1hand-walk.

The new experiment in Section 5.2, Table 1 directly tests the actor's contribution to MPPI. We froze the trained CAPI and exponential models on H1hand-walk, both using λ=1\lambda=1, and compared 24 explicit actor trajectories with a control that replaces them with Gaussian trajectories. Both conditions retain 512 candidates per iteration and the terminal actor.

MPPI iterationsMethodWithout actor proposalsWith actor proposalsMean gain
2CAPI32.2 ± 23.1751.9 ± 132.5+719.7
2Exponential211.8 ± 134.6333.3 ± 65.9+121.4
4CAPI199.7 ± 211.6906.9 ± 37.7+707.2
4Exponential691.5 ± 87.9708.6 ± 94.4+17.0
8CAPI394.9 ± 447.9928.2 ± 16.9+533.2
8Exponential731.6 ± 65.3808.1 ± 25.2+76.5

Actor proposals occupy 4.7% of the candidate pool and improve CAPI's mean return at all three search budgets. At the primary two-iteration budget, they increase CAPI's return by 719.7, compared with 121.4 for exponential weighting. MPPI uses highly valued candidates to refine its sampling distribution, allowing actor proposals to guide subsequent search iterations. These controlled comparisons demonstrate that the actor learned within CAPI supplies useful candidates for planning under a limited search budget.

2. Technical Presentation and Design Rationale

Section 4.2 reads as a bulleted list in paragraph form with little connectivity between concepts and ideas.

Response. We agree that the original Section 4.2 did not clearly explain how the allocation problem, weighting rule, and policy improvement analysis fit together. We have reorganized the section and added explanations to clarify these connections.

Following the motivation in Section 4.1, we first formulate an allocation problem that favors promising actions while limiting weight concentration within a finite actor batch. Theorem 1 derives the optimal weighting rule based on this allocation problem. We then describe the practical gate and actor objective to explain how this rule is implemented. Finally, Theorem 2 examines policy improvement under an idealized state-wise update, providing a policy-level interpretation of the allocation.

3. Connection and Role of the Two Theorems

I felt that Theorems 1 and 2 weren't connected at all and were unnecessary at worst.

Response. Thank you for raising this point. Our goal is to prioritize promising actions while limiting the share of update weight assigned to any sample. Theorem 1 characterizes the optimal weights for this constrained allocation problem, including the shared threshold required for normalization. This solution provides the basis for the practical gate used in actor learning.

We then examine the implications of capacity constraints for policy improvement. At the policy level, the corresponding constraint limits the increase in action density relative to the current policy. Theorem 2 establishes that, with exact policy values and an update solved exactly at every state, the new policy’s soft value does not decrease. This connects the weight construction to a policy improvement guarantee for the corresponding idealized update. We have clarified this connection in Section 4.2.

4. Computing the Capacity Threshold

A vague reference to the algorithm used and a Section in the appendix is not sufficient.

Response. We have revised Section 4.2 to explain how the shared batch threshold is computed at each rollout depth by solving a monotone scalar normalization equation using 25 bisection iterations. We have also revised Algorithm 1 to present threshold computation and actor-weight construction as separate steps. Appendix B.2, Algorithm 2 provides the search interval and full pseudocode.

Thank you again for the constructive feedback. We welcome any further questions about these points.

Thank you, I'm pleased with the responses and updates to the paper. I feel that it is substantively improved following the suggestions and advice from the reviewers. As such, I'm happy to raise my score to "Accept".

Thank you. We’re glad the revisions addressed your concerns. We greatly appreciate your time and valuable input in improving our work.

Reviewer_g385OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces capacity-constrained weighting within TD-MPC2 actor batches, combined with pessimistic value scores and optional planner-likelihood regularization.

Critical strengths

Results on 18 tasks, with five seeds per task, show meaningful improvements; the allocation-rule and capacity ablations provide additional support. The method and hyperparameters are relatively clearly described. The authors explicitly delimit the theoretical guarantee for the ideal operator and provide an anonymous code link.

Critical weaknesses
  1. Missing comparisons with directly related actor-weighting methods. Additional sample weighting in Q-driven actor updates has precedents, including UWAC [1] and ReW-PE-SAC [2]. The paper neither discusses these directly related methods nor compares their weighting rules, leaving the additional value of its particular weighting design unclear. The unweighted and ordinary exponential-weighting baselines in Figure 5(b) do not replace comparisons with relevant prior methods.

  2. The capacity design's novelty and mechanism need clearer explanation. DCEM [3] and CVaR policy optimization [4] already use similar limits on normalized weight shares. Their purposes differ from CAPI, so they do not establish duplication. However, the paper should explain what new design or understanding comes from applying such a constraint to its actor updates. The motivation is that erroneously high scores may give a few samples excessive weight and harm training. Yet the paper does not directly show whether its high-weight samples are overestimated, what harm follows, or whether capacity reduces it. Figure 5(b) shows a performance gain, but does not validate this explanation. Theorem 1 computes the weights; Theorem 2 assumes exact values and guarantees ideal policy improvement. Neither analyzes harm from estimation errors or how capacity mitigates it.

  3. Unexplained discrepancy between the reported weight statistics and the default formulas. For H1hand-walk with λ = 1, Table 2 reports a mean gate of 0.12 and a fraction of samples at the minimum weight of 76%. Under Eqs. (8), (9), and (12) and the default parameters in Table 3, if both statistics are computed over the same sample collection, the mean gate must satisfy w ≥ 1 8 + 0.76( e −3 − e −15 ) ≈ 0.163.

The threshold-score processing in Appendix B.2 does not remove this lower bound, and ordinary rounding cannot explain the discrepancy. Please reconcile the actual parameters, statistical definitions, or implementation. This discrepancy alone is insufficient to invalidate the reported training returns.

Questions for authors

Please report the actual capacity ratio, weight temperature, and clipping range used for Table 2, together with the definitions of the mean gate and the fraction of samples at the minimum weight. Specify whether these statistics use the same training window, rollout depths, and sample collection. If there is additional processing beyond Eq. (12), please provide the actual computation steps.

4 comments

We sincerely thank the reviewer for the constructive comments. We have revised the manuscript and highlighted the changes.

1. Comparison with Related Weighting Methods

Additional sample weighting in Q-driven actor updates has precedents, including UWAC [1] and ReW-PE-SAC [2].

Response. UWAC [1] and ReW-PE-SAC [2] use sample reweighting to reduce unreliable updates. CAPI derives weights from a capacity-constrained allocation over the actor batch, explicitly limiting how much normalized weight any sample can receive. The distinction is between weighting samples by estimated reliability or training utility and constraining the distribution of influence across the batch. We have expanded the Weighted Policy Improvement discussion in Section 2 (Related Work) to explicitly cover UWAC and ReW-PE-SAC.

We have also added UWAC weighting to Ablation of the allocation rule (Section 5.3) and updated Figure 5(b). This baseline adapts UWAC’s actor-weighting rule to the CAPI framework. We compute inverse-variance weights from the existing critic ensemble and apply the detached weights to the full actor objective, retaining the world model, critic update, and planner. Following another reviewer’s suggestion to control actor-loss scale, we use a common fixed mean actor weight for uniform, exponential, and UWAC weighting within each task. The following pseudocode defines the UWAC weights at each rollout depth.

# q_values: [num_critics, batch_size], omega_bar: fixed mean weight
with torch.no_grad():
    variance = q_values.var(dim=0, unbiased=False)
    inv_var = (variance + epsilon).reciprocal()
    weights = omega_bar * inv_var / inv_var.mean()

UWAC’s inverse-variance coefficients favor samples with low predictive uncertainty [1, Eq. (5)]. However, aligned critic gradients can yield low ensemble variance even for out-of-distribution actions [5]. In our ensemble adaptation, such actions can consequently receive large weights, offering a possible explanation for the H1hand-walk gap. The table reports mean returns at 1M environment steps. CAPI achieves the highest mean final return among these four weighting rules on both tasks.

Weighting ruleDog-runH1hand-walk
CAPI669.1 ± 58.0929.9 ± 6.5
Uniform weighting438.3 ± 125.8895.2 ± 16.3
Exponential weighting481.4 ± 131.3870.9 ± 33.8
UWAC weighting638.6 ± 15.0829.0 ± 14.6

2. Novelty of Capacity-Constrained Actor Weighting

DCEM and CVaR policy optimization already use similar limits on normalized weight shares.

Response. Thank you for highlighting these connections. DCEM [3] advances differentiable search by smoothing elite selection, while predictive CVaR policy gradients [4] provide an effective way to optimize tail risk through weighted costs. CAPI uses the related capacity constraint to control the distribution of weights within an actor update. Unlike DCEM's binary-entropy smoothing for fitting a search distribution, our Shannon-entropy allocation yields exponential weights. Their threshold and saturation structure is used to weight actor losses, with the weights detached during optimization. The scores determine which samples are favored, while capacity limits the normalized weight any sample can receive, including samples with unusually high critic scores. We have added these connections to Section 2 (Related Work).

[1] Yue Wu et al. Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning. ICML, 2021.

[2] Wenzhen Huang et al. Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning. AAAI, 2021.

[3] Brandon Amos and Denis Yarats. The Differentiable Cross-Entropy Method. ICML, 2020.

[4] Ju-Hyun Kim and Seungki Min. Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient. ICML, 2024.

[5] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. NeurIPS, 2021.

3. Mechanism and Theoretical Support

Yet the paper does not directly show whether its high-weight samples are overestimated, what harm follows, or whether capacity reduces it.

Response. Thank you for raising this point. We examined whether samples receiving the maximum CAPI weight had larger mean estimation errors Q−Q^MCQ-\widehat Q_{\mathrm{MC}} than the remaining samples. Using one 1M-step checkpoint per task, we evaluated actions sampled at 256 replay states, with MC returns averaged over 16 environment rollouts per action under the frozen actor. Gates reached the upper bound for 21/256 samples on Dog-run and 29/256 samples on H1hand-walk.

TaskMean error, gate = 1Mean error, gate < 1
Dog-run87.0682.43
H1hand-walk18.060.41

Samples at the maximum gate have higher mean overestimation on both tasks, with a particularly clear difference on H1hand-walk. This supports the concern that overestimated samples can receive the strongest weighting preference, motivating explicit control over their normalized shares.

To examine the effect of this control, we removed the upper gate bound while retaining the same samples, scores, temperature, weight floor, and total weight at the frozen checkpoint.

Task / Weighting ruleMaximum normalized shareESS / batch size
Dog-run / Without upper bound2.31%48.69%
Dog-run / CAPI1.12%57.50%
H1hand-walk / Without upper bound16.06%10.13%
H1hand-walk / CAPI1.80%31.73%

On H1hand-walk, the cap reduces the largest sample share from 16.06% to 1.80% and substantially broadens the allocation. The Dog-run comparison shows the same direction. These diagnostics complement the updated allocation comparison in Section 5.3, Figure 5(b).

The capacity constraint also gives us the following simple bound. For the allocation problem in Section 4.2, let s^=s+δ\widehat s=s+\delta, and let ω⋆\omega^\star and ω^\widehat\omega be the optimal allocations under the reference and estimated scores, respectively, with the same capacity constraint. For Φs(ω)=⟨ω,s⟩+H(ω)\Phi_s(\omega)=\langle\omega,s\rangle+\mathcal{H}(\omega),

\Phi_s(\omega^\star)-\Phi_s(\widehat\omega) \leq \frac{\kappa}{n} \min_{c\in\mathbb{R}}\lVert \delta-c\mathbf{1}\rVert_1.$$ This means that capacity also bounds how much scoring errors can cost in the allocation objective. A common score offset does not change the allocation, so the bound captures differences in errors across samples. For example, if errors affect $m$ samples and each error is at most $\epsilon$, the loss is bounded by $\kappa m\epsilon/n$. This complements **Theorem 2** by explicitly accounting for score errors and explains the role of capacity without requiring the algorithm to identify erroneous samples in advance. ### 4. Consistency of the Weight Statistics in Table 3 > Please report the actual capacity ratio, weight temperature, and clipping range used for Table 2, together with the definitions of the mean gate and the fraction of samples at the minimum weight. Specify whether these statistics use the same training window, rollout depths, and sample collection. **Response.** Thank you for identifying this discrepancy in the submitted **Table 2**, now **Table 3 (Appendix B.1)**. The mean gate and floor fraction were computed incorrectly, and we apologize for this calculation error. We checked the retained training logs and confirmed that the recorded settings match those reported in **Table 4**, with $\kappa=8$, a weight temperature of $5$, and log-gate clipping to $[-3,0]$. We reran CAPI on Dog-run and H1hand-walk with $\lambda=0$ and $\lambda=1$. During these runs, we recorded the gate coefficients actually used in the actor loss to obtain directly verifiable statistics. For each seed, we collected these paired statistics over 800k–1M environment steps, using all 256 samples at each of the three rollout depths in every actor update. We then averaged each statistic across seeds. We have updated **Table 3** and replaced the corresponding Dog-run and H1hand-walk results in **Table 2** and return curves with the rerun results. These updates lead to modest changes in aggregate benchmark performance. The corrected H1hand-walk results from **Table 3** are reproduced below. All metrics are averaged over 800k–1M environment steps. | Metric | $\lambda=0$ | $\lambda=1$ | |---|---:|---:| | Mean return | 249.9 | 911.8 | | Mean L2 distance $d_{\mathrm{AP}}$ | 8.239 | 1.440 | | Mean gate | 0.286 | 0.232 | | Floor fraction $\rho_{\downarrow}$ | 29.87% | 54.47% | Thank you again for the constructive feedback. We welcome any further questions about these points.

Thank you for the detailed response and revisions. I am raising my rating from Weak Reject to Weak Accept.

Thank you for the updated score and constructive feedback. We have revised the related work and experimental sections accordingly. We sincerely appreciate your time and valuable suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and appreciation

We appreciate all reviewers for their comments. Your feedback helped us improve both the explanation of CAPI and its empirical support. We summarize the main revisions below.

  • We clarified the capacity constraint on each sample’s normalized weight share and rewrote Section 4.2 to connect the weighting rule, threshold computation, and actor update. We also explained how the solution for a finite batch relates to the ideal policy improvement analysis.
  • We revised the Related Work to explain CAPI’s connections to and differences from UWAC, DCEM, and policy optimization based on CVaR, highlighting capacity constraints for allocating weights in actor learning.
  • We added exponential, uniform, and UWAC-inspired actor-weighting controls within the same framework, with their weight scales and retained components specified explicitly.
  • Based on the fixed checkpoint, we compared planning with and without explicit actor proposals, using paired initial states and the same total candidate budget. CAPI achieved higher mean returns with proposals at each tested planning budget, providing direct evidence of their contribution to planning.

We appreciate your time and the suggestions that helped improve this work again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.