acceptodds
Under review as a conference paper at ICLR 2027

Calibrated Candidate Replacement under Imperfect World Models via Baseline-Relative Learning

Abstract

World-model errors can make candidate scores unreliable. A higher predicted return does not always justify replacing the agent's incumbent action. We propose Baseline-Relative Value Learning (BRVL), a post-training selector for frozen world-model agents. BRVL learns candidate–incumbent return differences from paired offline simulator rollouts under a common baseline continuation. Its objective combines relative-return regression with a candidate-set loss for forgone gains and harmful replacements. At deployment, BRVL replaces the incumbent only when an alternative has a positive calibrated lower score. We evaluate BRVL with DreamerV3 and TD-MPC2 on DMControl, Meta-World, and ManiSkill2. BRVL improves mean closed-loop performance and frozen-set net gain over matched-supervision regression baselines in all six settings. The gain–harm trade-off varies across settings.

Primary area:
reinforcement learning
Keywords:
reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_tmdFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes BRVL, an action selection method that combines prior methods such as relative return-based action selection, decision-aware learning, and split-conformal calibration to replace incumbent actions if the predicted return improvements derived from world-model features are sufficiently large when compared to those of the incumbent actions. The authors show performance improvements over prior work across both DreamerV3 and TD-MPC2 in the DMC, Meta-World, and ManiSkill2 benchmarks.

Critical strengths

The authors demonstrate improvements in performance across two modern model-based RL methods and three benchmark suites (DMC, Meta-World, and ManiSkill2). When compared to cases without any of their additions, the BRVL method achieves up to a 20 percentage point higher success rate in Meta-World and up to a 35 percentage point increased success rate in ManiSkill2. This indicates that the combination of these methods does provide generalized performance across different agent algorithms and environment types.

The inclusion of ablations for BRVL that show the relative improvements and contributions of each component to the overall performance of the method are very useful. They isolate which parts are most important and demonstrate how each contributes to the overall performance.

Critical weaknesses

The primary weakness of this paper is the limited novelty of the technical contributions. The paper combines a number of existing ideas like relative returns (advantage estimates), an antisymmetric head, decision-aware learning, and split-conformal calibration. None of these components are novel, and there is an argument to be made that adding them together is a novel application, but from my understanding these components were 1. already used for similar purposes previously (meaning there isn’t novelty in their utilization or in resolving another underlying problem) and 2. in several cases, were already used in similar RL and decision-making benchmark settings. Under those two conditions the novelty of application seems limited, and their combination appears to be an incremental improvement rather than a substantial novel contribution.

There does seem to be some evidence of improved performance being achieved through combining the methods into BRVL, but without empirical insights into why the combination works better, this seems to be demonstrating A + B + C leads to better performance, rather than something impactful for the RL or world model community. Overall, further discussion of the novelty of combining the methods, not just their performance relative to the ablated baselines from which these methods inherited, is necessary to improve this paper.

I think the writing in this paper is very poor. In just the first page there are numerous cases where sentences fail to build a coherent argument. The connections between ideas throughout this paper are poorly developed at worst and require substantial reader effort to draw connections properly. For example the first three sentences of second paragraph in the introduction list prior work and citations in a very formulaic and enumerated way without making it clear their relationship or relevance to the paper’s central contributions and themes. It should be made clear to the reader through the flow of the paper, not left to the reader to read between the lines.

This same patterns is repeatedly used throughout the paper and makes a substantial revision necessary to establish a coherent narrative flow and make the technical contributions more digestible. Similarly in the results section the authors frequently restate the results from their tables with very little additional insight or explanation.

Scientific writing requires more than presenting technically correct statements and writing out experimental results in prose. This paper does quite a poor job at this fundamental science communication task. There is some interesting content here, but as written I don't think this paper is ready for acceptance.

Questions for authors

Unless I am missing something, prior work already motivated why each individual component that makes up this approach is useful for addressing individual issues. What about the combined effect of BRVL is novel? Some greater intuition and additions to the paper to describe why the combination of these methods produces better performance, and whether that benefit is simply additive. What scientific insight are we to gain from this work?

BRVL uses additional compute resources to generate the 2400 branched trajectories from the simulator per task and checkpoint. Could comparable gains be made by simply using these additional trajectories directly for further training of the agent or the world model? It would seem this is the most relevant missing algorithmic baseline.

Reviewer_fWKDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BRVL, a selector that decides whether to replace a frozen world-model agent’s action (the incumbent) with one of N generated alternatives. It learns candidate-minus-incumbent return differences from paired rollouts in a resettable simulator, trains them with regression plus a decision-aware cost, and calibrates the predictions with split conformal prediction. It replaces the incumbent only when a calibrated lower score is positive. The paper proves relevant theoretical guarantees, and evaluates on DMControl, Meta-World and ManiSkill2 with strong results.

Critical strengths

A sensible, conservative formulation. Predicting improvement over the agent’s own action, rather than absolute values, and defaulting to that action unless evidence is strong is well motivated and makes the gain–harm trade-off explicit.

Ablations show that each component contributes.

Critical weaknesses

The paper is framed around world-model errors, yet all supervision comes from a resettable ground-truth simulator; the world model only supplies features and candidates. This makes the role of model-based RL unclear.

Given that simulator and the extra rollouts, it is unclear whether the same interaction and compute would be better spent improving the agent itself than training an external selector. No experiment addresses this, since every baseline shares the frozen-agent assumption.

Questions for authors

Q1: Does BRVL apply to model-free agents? If so, what does the world model contribute?

Q2: How does BRVL compare with spending the same rollout budget (or compute) on fine-tuning the agent itself? Please also report the total supervision budget relative to base-agent training.

Minor: The writing obscures the problem. The introduction reads as a list of citations, and the problem definition is buried under method jargon.

Reviewer_niJVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Baseline-Relative Value Learning (BRVL), a post-training action selector for frozen world-model agents. It learns candidate–incumbent return differences from paired simulator rollouts that execute different initial actions and then follow the same baseline controller. An antisymmetric ensemble is trained with relative-return regression and a candidate-set decision loss that penalizes forgone gains and harmful replacements. Blockwise split-conformal calibration supplies a margin, and the selector replaces the incumbent only when an alternative has a positive calibrated lower score.

The analysis establishes simultaneous coverage of fresh finite-rollout measurements under exchangeability and a marginal expected harmful-loss bound under additional assumptions. Experiments combine DreamerV3 and TD-MPC2 with DMControl, Meta-World, and ManiSkill2. BRVL achieves higher mean closed-loop performance than the matched relative-regression baseline in all six settings, although the statistical evidence and gain–harm trade-off vary across settings.

Critical strengths

The matched-supervision comparison provides meaningful evidence for decision-aware learning. Rel-MSE+Gate uses the same relative targets, inputs, ensemble architecture, and calibration procedure, while removing the decision-aware loss. This comparison controls for the additional simulator supervision. Table 2 reports substantial closed-loop improvements, including 38.0 and 39.1 return points on DMControl for DreamerV3 and TD-MPC2, respectively, with confidence intervals excluding zero. This is the strongest evidence that the proposed objective contributes beyond obtaining better supervision.

The evaluation and analysis distinguish several quantities that are easily conflated. The paper separates frozen-set selection quality from repeated closed-loop deployment, and measured harmful loss from loss in expected return. Source-disjoint partitions, shared frozen candidates, paired episode evaluations, and dependence-aware resampling strengthen the comparisons. The theoretical discussion also explicitly acknowledges that offline measurement coverage does not establish pointwise expected improvement or closed-loop safety. This precision makes the contribution easier to assess within its intended scope.

Critical weaknesses

The independent measurement audit is not clearly supported by the displayed results. Section 4.6 introduces an independent (R_{\mathrm{audit}}=20) audit on ten DMControl tasks and refers to Figure 3 for its findings. However, Figure 3’s caption identifies the main (R=3) evaluation, its displayed DreamerV3 BRVL net gain is 0.029 rather than the 0.028 stated in Section 4.6, and its legend contains “Oracle ref. (placeholder)”. Appendices A.9 and D.4 describe the audit protocol, but do not resolve which reported results come from that audit. This could be a figure assembly error, but it currently prevents checking whether the measurement-sensitive harm estimates and reference gaps survive independent, higher-repetition evaluation.

Reproducibility and calibration sensitivity are insufficiently established. Section 3.2 describes the candidate descriptor only as containing action and world-model features; the exact contents are not enumerated. The number and identities of the frozen base-agent checkpoints are also unclear. Moreover, the reported intervals condition on fixed fitting and calibration data, while each calibration group contains only 20 blocks and uses the second-largest residual at the primary coverage level. The three ensemble initializations therefore do not establish stability across independently collected calibration sets. These gaps matter because both the learned comparison and the replacement operating point are central to the reported gains.

Questions for authors

Clarification of existing results: Please reconcile Figure 3 with Section 4.6. Which quantities use the primary (R=3) measurements, and which use the independent (R_{\mathrm{audit}}=20) audit? Please provide the audit’s net gain, harmful loss, severe-harm rate, and cross-fitted reference with uncertainty, and explain the “placeholder” label. A comparison on the same audit roots with fixed selected actions would clarify whether measurement noise changes the conclusions.

Implementation clarification: What exactly enters (\phi(z,u)) for each agent family? How many independently trained base-agent checkpoints are evaluated per task, and what training budgets and checkpoint-selection rules produced them?

Robustness evidence: How sensitive are replacement rates, measured coverage, and BRVL’s advantage over Rel-MSE+Gate to independently resampling the fitting/calibration source episodes, particularly the 20 calibration blocks? The current uncertainty analysis explicitly conditions on these data; evidence about this additional source of variation would materially affect my confidence in the results.

Reviewer_pk1WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Baseline-Relative Value Learning (BRVL), a post-training action selector for frozen world-model agents (DreamerV3 and TD-MPC2). For each decision state, a fixed generator supplies the incumbent action and nine alternatives; paired rollouts from the restored simulator state measure the return difference between each alternative and the incumbent under the frozen baseline continuation. An antisymmetric ensemble is trained on these differences with a regression loss plus a softmax decision cost for forgone gains and harmful replacements, and a split-conformal quantile of the normalized residuals is computed on held-out source blocks. At deployment the incumbent is replaced only when an alternative has a positive calibrated lower score. On 39 DMControl, 50 Meta-World and 5 ManiSkill2 tasks, BRVL improves closed-loop performance and frozen-set net gain over matched-supervision baselines in all six agent–benchmark settings.

Critical strengths

Clear decision-theoretic framing: the incumbent is a zero-cost fallback, replacement requires a positive calibrated lower bound on the relative return, and the paper is precise about what the guarantees do and do not cover. Large evaluation: 94 tasks, two agent families, matched-supervision controls (Rel-MSE+Gate isolates the decision loss), adapted RVL and DAE references, and ablations that separate antisymmetry, the decision loss, the calibration margin, and fallback.

Critical weaknesses

The selector's environment budget is not compared against the right baseline. Each task/checkpoint needs 2,400 branch trajectories for fitting and calibration; Figure 2(b) shows 15–35M fitting-plus-calibration transitions on DMControl and 10–20M on Meta-World, and the paper never says how this compares with the budget used to train the base agents. The only reference is the frozen agent. The obvious comparison is to spend the same interactions on the agent itself: continue training DreamerV3/TD-MPC2, or fit the actor and critic to the paired Monte-Carlo returns (e.g., advantage-weighted regression on the branch data). Without it, the closed-loop gains cannot be attributed to the selector design rather than to the extra data.

The method is per-task and does not transfer. Selectors use task/checkpoint-specific projections and per-group calibration, and evaluation is on new episodes of the same tasks and checkpoints. "Post-training selector" therefore means a task-specific correction layer that requires a restorable simulator, which the authors acknowledge, but it narrows the contribution considerably.

Calibration contributes little to performance although it is the headline. BRVL-Mean (no margin) reaches 742.5/899.4 vs 749.0/906.0 for BRVL (Table 3), and the decision-aware loss accounts for most of the gain (36 return points). The guarantees (Eqs. 8–9, Appendix C) are marginal over calibration blocks for R = 3 finite measurements, not for expected returns and not for closed-loop visitation. The measured harm is tiny for every gated method (D̂3 = 0.003 vs 0.006 in normalized units), so "calibrated" replacement is not what drives the results.

The gap between frozen-set and closed-loop gains is unexplained. Single-intervention net gain is about 0.03 of the task range, yet closed-loop success on ManiSkill2 with DreamerV3 rises from 23.7% to 58.0%. The paper only notes that the two use different state distributions; there is no closed-loop replacement rate, per-task breakdown, or analysis of where the compounding comes from. This matters because the theory covers single interventions and the authors themselves note that repeated intervention can shift the visited distribution.

Statistics. The main table (Table 1) has no intervals; there are three selector initializations and 20 closed-loop episodes per task; the TD-MPC2 net-gain intervals on Meta-World and ManiSkill2 include zero (Table 2).

Presentation. The prose is telegraphic and hard to follow; Figure 2 contains fifteen small panels with overlapping labels; the legend of Figure 3 contains "Oracle ref. (placeholder)"; "Eq. equation 5" and similar broken references appear repeatedly; Hoeffding's inequality is cited as "(?)" in Appendix C.1.

Questions for authors

What is the total number of environment transitions used to obtain the BRVL agent (base training, roots, branches), and how do continued training of the base agent or actor fine-tuning on the branch returns perform under the same budget?

Why do small single-step gains translate into very large closed-loop gains on ManiSkill2 with DreamerV3? What is the closed-loop replacement rate, and does the harm statistic hold on closed-loop states?

Can a selector fitted on one set of checkpoints or tasks be used on others, or must the full branch collection be repeated for each deployment?

Please add confidence intervals to Table 1 and fix the figure legend and equation-reference issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.