acceptodds
Under review as a conference paper at ICLR 2027

RG-TAIL: Reliability-Gated Trust-Aware Imagination Learning under Limited Budgets

Abstract

Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned world models to generate synthetic experience. However, model-generated transitions can be harmful when model errors accumulate especially in data-scarce settings, where reliable synthetic experience is essential due to the high cost of real-world interaction. Our large-scale empirical study, (RoI), reveals that while synthetic data can substantially improve reinforcement learning, uncertainty estimates do not necessarily align with the long-term utility of synthetic transitions. Based on this observation, we propose (), a framework that combines transition-level trust estimation with a higher-level reliability gate to determine whether synthetic experience could be used in policy optimization. We perform extensive experiments across diverse environments, reinforcement learning algorithms, model configurations, and data budgets. Compared with internal baselines and representative MBRL methods, RG-TAIL achieves the best overall performance under both extremely low- and high-budget settings, improving task success rates by approximately % on average.

Primary area:
reinforcement learning
Keywords:
Model-Based Reinforcement Learning, World Models, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N5dYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to use model-generated experience effectively when real interaction data are limited. TAIL learns transition-level trust from world-model prediction errors and uses it to weight synthetic experience. RG-TAIL adds a reliability gate that determines whether synthetic experience contributes to both actor and critic updates, critic updates only, or neither. The manuscript reports experiments across six control environments, three reinforcement learning algorithms, and multiple data budgets, together with ablations, failure cases, and statistical analyses.

Critical strengths
  1. The method makes a useful distinction between actor and critic training. Synthetic data may help estimate values while giving poor directions for improving the policy. Allowing critic-only updates addresses this possibility, and using only real data when reliability is low gives the method a clear fallback.

  2. The experiments cover a useful range of settings and report failures as well as gains. The study uses multiple algorithms, data budgets, and ten random seeds, with ablations and threshold-sensitivity tests. The statistical appendix reports that the 5K comparisons with TD-MPC2 and DreamerV3 are not statistically significant. This helps readers distinguish the measured gains from differences that remain uncertain.

Critical weaknesses
  1. The success-rate metric is too coarse to support the headline performance claim. Section 4 counts an episode as successful whenever its return is above zero, across all six tasks. The meaning of that cutoff depends on the environment. For example, Table 18 reports a 100% success rate for every method on Hopper, although their mean returns range from 71.92 to 578.63. This metric treats substantially different levels of performance as equally successful.

    The abstract emphasizes an average success-rate improvement of approximately 19 percentage points. Interpreting this number requires a task-specific justification for the zero-return cutoff, a named reference baseline, and an explicit averaging rule. The supplied scripts also use different success-rate conventions, so the authors should provide the shared postprocessing code used for the tables. These details determine what the headline improvement actually measures.

  2. The paper and code use different definitions of the proposed method. Equations (5)–(6) compute reliability from disagreement between the deterministic and probabilistic world models, together with rollout length. The PPO implementation follows this definition. The SAC and TD3 implementations replace model disagreement with the mean learned transition trust. This changes the signal that decides whether synthetic data are used for actor updates, critic updates, or neither. The authors need to identify which calculation was used for the reported SAC and TD3 results.

    The trust calculation also differs. Equation (2) divides prediction error by its median, while the SAC and TD3 code use the standard deviation and the PPO code uses the larger of the median and standard deviation. These choices change the resulting trust weights. In addition, Section 6.2 describes No Gate as equivalent to TAIL, but the supplied SAC No Gate and standalone TAIL implementations use different weighting choices. The authors should specify the code and settings behind these comparisons and explain how the trust-training data were kept separate from world-model fitting. Until these points are resolved, the tables cannot be tied to one clearly specified implementation.

Questions for authors
  1. For the SAC and TD3 results in Tables 1–2, was reliability computed from WM–PWM disagreement as in Equations (5)–(6), or from mean transition trust as in the supplied code? Please identify the function and settings used for those runs.

  2. Section 6.2 states that No Gate is equivalent to TAIL. Were their trust normalization, actor and critic weights, and other training settings identical in the reported comparison? Please explain the differences between the supplied SAC implementations.

  3. What does a return above zero represent in each task, and which common postprocessing script produced the reported success rates? Please give the reference baseline and averaging rule used to obtain the approximately 19-percentage-point headline improvement.

3 comments
AuthorsOpenReview ↗

Response to Reviewer N5dY

We thank the reviewer for their valuable feedback.

Weakness 1 & Question 3: Success Matrix

We defined an episode as successful when its return was greater than zero because several of the six environments have no official task-completion criterion. Our intent was to provide a shared metric across all six environments. We agree that this threshold is an inappropriate proxy for task completion.

MountainCarContinuous has an explicit goal-reaching event, and the LunarLander documentation considers an episode solved when its return is at least 200. The four MuJoCo locomotion tasks have no explicit completion event. Hopper and Walker2d can terminate early when the agent enters an unhealthy state, while HalfCheetah and Swimmer run to a time limit. In all four tasks, the episode return captures forward progress and control costs.

In the revision, we will report mean episodic return with 95% confidence intervals across training seeds as the primary metric for all six environments. We will also report the MountainCarContinuous goal-reaching rate and the fraction of LunarLander episodes that meet the documented 200-point threshold. We will remove the success rates based on positive returns.

The original average +19% gain can be reproduced from the Average rows of Tables 1, 2, and 19. We treated 5K and 10K as extremely low budgets and 500K as an extremely high budget. At each of these budgets, we subtracted each reported external baseline’s positive-return rate from RG-TAIL’s rate:

BudgetSourceMBPOTD-MPC2DreamerV3
5KTable 1+17.0 pp+6.0 pp+1.7 pp
10KTable 2+17.5 pp+13.1 pp+9.8 pp
500KTable 19+51.7 pp+39.9 pp—

The eight comparisons received equal weight, yielding 156.7 ÷ 8 = 19.5875 percentage points.

Weakness 2 & Question 1&2: Inconsistency Issues

We checked the training code and confirmed the inconsistency. We developed RG-TAIL with PPO as the primary backbone and introduced a different reliability calculation when porting it to SAC and TD3. We deeply apologize for this mistake. We have started rerunning the experiments for PPO, SAC, and TD3 with a shared implementation of the WM–PWM agreement-based reliability calculation in Eqs. (5)–(6). We will update the affected tables and analyses using the rerun results.

Additional: 5K Budget Results from Rerunning

We have rerun RG-TAIL (PPO, SAC, TD3) and three baselines under 5K budget. As shown in the tables below, RG-TAIL with SAC achieves the highest raw return, and averagely RG-TAIL achieves the highest normalized return.

Raw returns at 5K

TaskRG-TAIL (PPO)RG-TAIL (SAC)RG-TAIL (TD3)MBPOTD-MPC2DreamerV3
MountainCarContinuous−66.98 (0.05%)−1.28 (0.00%)−1.84 (0.00%)−0.10 (0.00%)−0.05 (0.00%)−47.87 (3.90%)
Hopper192.1637.2727.6326.9136.38108.32
HalfCheetah−346.18−35.20−440.66−45.28−117.29−276.23
Walker2d177.12−21.523.6649.292.55151.92
Swimmer26.5217.7728.66−8.36−4.4617.61
LunarLander−405.06 (0.00%)−164.87 (1.00%)N/A−865.00 (0.00%)−183.85 (0.00%)−161.77 (0.00%)
Average−70.40−27.97−76.51−140.42−44.45−34.67

Normalized returns at 5K

TaskRG-TAIL AverageMBPOTD-MPC2DreamerV3
MountainCarContinuous0.5090.9930.9940.000
Hopper0.1280.0000.0210.177
HalfCheetah0.0230.1250.0930.022
Walker2d0.1690.1560.0000.500
Swimmer0.9640.0000.1150.766
LunarLander0.7250.0000.8510.879
Average0.4200.2120.3460.391

Thank you for clarifying how the 19-percentage-point figure was calculated and for providing the new 5K results. Given that the consistency of the gains across the remaining reruns and the gate’s contribution under a matched comparison have yet to be established, I will maintain my original rating of 2 for now.

AuthorsOpenReview ↗

Thank you again

We sincerely thank the reviewer for their thoughtful assessment and recognition of our responses. We completely agree that additional experiments and reruns are needed to resolve the issues. We deeply appreciate your time in this work, and we will continue to refine the study and present a stronger paper.

Reviewer_UnDNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of the proper time of model-generated experience participating in reinforcement learning under limited real-data budgets. The paper presents RG-TAIL, a framework that combines a transition-level trust predictor, supervised by held-out one-step prediction errors, with a global reliability gate. The global gate utilizes world model disagreement, synthetic rollout length, and return variability. Experiments across six simulated environments and six data budgets, and ten seeds show the improvement in aggregated returns and success rates.

Critical strengths
  • The paper provides an approach RG-TAIL to estimate transition-level trustworthiness from held-out real-world prediction errors.
  • It introduces a transition-level trust predictor.
  • The paper provides experimental results to validate the performance of the proposed RG-TAIL.
Critical weaknesses
  • The central idea of the paper introduces a new component called trust predictor. However, it is not clear how to train this predictor. If it is assessed as the difference between the real state and the predicted one, this has been widely studied, consequently leading to questionable novelty of the paper.
  • This paper emphasizes world models. However, it is still the model-based reinforcement learning. It is hard to find any unique features related to world models, making the paper confusing.
  • The trust target is a transformation of one-step prediction error, while reliability integrates both model agreement and rollout duration. These are predicted values, and the paper does not show their reliability of identifying beneficial or harmful synthetic updates on held-out configurations.
Questions for authors
  1. Please clarify why positive return should represent task completion in all six environments.
  2. What are the exact critic losses and synthetic transition weights in RG-TAIL?
  3. There are some grammar issues and typos within the paper. Please proofread the whole paper carefully.
  4. It would be better to have an overview subsection to describe RG-TAIL.
4 comments
AuthorsOpenReview ↗

Response to Reviewer UnDN (1/2)

We thank the reviewer for their valuable feedback.

Weakness 1: Trust Predictor

We apologize for the lack of clarity in the Methodology section. Section 3.1.1 describes the trust predictor and its supervision target, and Appendix I.2 provides the architecture and training hyperparameters. In brief, we measure the one-step prediction error by comparing the predicted next state with the observed next state in a real transition. We map that error to a trust target that decreases as the error grows and train the predictor by supervised regression. We will bring these details together in Section 3.1.1.

We agree that using prediction error as a supervision signal is an established technique. As we note at the end of Related Work, “Unlike these approaches, RG-TAIL distinguishes local transition-level trust from policy-conditioned global reliability, rather than relying on a single uncertainty signal to regulate synthetic experience.” Our contribution is this two-layer gating mechanism. We first developed TAIL around local trust, which weights individual synthetic transitions and stops low-trust rollouts. Our mechanistic analysis of TAIL then motivated adding a global reliability gate that decides whether synthetic data should update both the actor and critic, the critic alone, or neither. The gate is designed to make the use of synthetic data more robust across environments and downstream learners. We will clarify this contribution in the revision.

Weakness 2: World Models

We apologize that our presentation gave the impression that the paper emphasizes world model design. The Abstract, Introduction, and Related Work frame RG-TAIL within model-based reinforcement learning. We use standard deterministic and probabilistic dynamics models to generate synthetic experience. Their predictions also supply inputs to the trust estimator and, through model agreement, contribute to the reliability score used for routing. We will further clarify this scope across the paper.

Weakness 3: Utility Alignment

Our central empirical claim concerns the performance of RG-TAIL as a complete method relative to the evaluated baselines. To evaluate the utility alignment, we ran additional experiments with PPO and the probabilistic world model (PWM) at the 5K and 10K real-interaction budgets, using 20 paired training seeds per environment and budget.

For the correlation analysis, we measured the downstream return gain from adding the full candidate batch to the fixed real dataset while holding the number of PPO updates constant. We then computed the Spearman correlation across seeds between this gain and the batch mean of each signal. We used negative predictive variance so that a higher score represents lower uncertainty. Predicted trust had a higher correlation point estimate in 3 of 6 environments at 5K and 5 of 6 at 10K.

For the matched filtering comparison, we ranked the same 4,096 candidate transitions by predicted trust or by negative predictive variance and retained the top 2,048 under each rule. The two conditions used the same fixed real dataset, PPO update budget, and evaluation seeds. Trust-based filtering yielded higher mean returns in 5 of 6 environments at both 5K and 10K. The unadjusted 95% confidence intervals for the return difference were above zero for HalfCheetah at 5K and Walker2d at 10K, and included zero for the other comparisons.

5K real-interaction budget

EnvironmentPredicted trust ρNegative predictive variance ρTrust filter − low-variance filter return [95% CI]
LunarLander+0.170+0.045+12.17 [−6.22, +33.12]
MountainCarContinuous−0.484−0.427−0.44 [−1.03, +0.16]
Hopper+0.164+0.146+3.37 [−10.65, +16.49]
HalfCheetah−0.195+0.311+34.35 [+13.86, +54.75]
Walker2d+0.099+0.083+9.76 [−8.36, +28.64]
Swimmer−0.289−0.263+0.92 [−0.29, +2.51]

10K real-interaction budget

EnvironmentPredicted trust ρNegative predictive variance ρTrust filter − low-variance filter return [95% CI]
LunarLander−0.092−0.238+18.76 [−0.90, +39.76]
MountainCarContinuous−0.002−0.006+0.65 [−0.86, +2.10]
Hopper−0.186−0.229+0.02 [−1.85, +1.92]
HalfCheetah+0.050−0.090−3.47 [−11.57, +5.12]
Walker2d−0.090−0.200+26.87 [+12.33, +42.62]
Swimmer−0.242−0.230+0.13 [−4.20, +3.85]
AuthorsOpenReview ↗

Response to Reviewer UnDN (2/2)

Question 1: Success Matrix

We defined an episode as successful when its return was greater than zero because several of the six environments have no official task-completion criterion. Our intent was to provide a shared metric across all six environments. We agree that this threshold is an inappropriate proxy for task completion.

MountainCarContinuous has an explicit goal-reaching event, and the LunarLander documentation considers an episode solved when its return is at least 200. The four MuJoCo locomotion tasks have no explicit completion event. Hopper and Walker2d can terminate early when the agent enters an unhealthy state, while HalfCheetah and Swimmer run to a time limit. In all four tasks, the episode return captures forward progress and control costs.

In the revision, we will report mean episodic return with 95% confidence intervals across training seeds as the primary metric for all six environments. We will also report the MountainCarContinuous goal-reaching rate and the fraction of LunarLander episodes that meet the documented 200-point threshold. We will remove the success rates based on positive returns.

Question 2: Critic Weight and Loss

Critic weight. We set wic=clip⁡(τi/τˉ,0.05,1)w_i^c = \operatorname{clip}(\tau_i / \bar{\tau}, 0.05, 1) for admitted synthetic critic samples when reliability is below 0.40 and the variance of their trust scores within the minibatch exceeds 0.01. Here τˉ\bar{\tau} is the mean trust of those samples. All other admitted samples receive unit weight.

Critic loss. Let BcB_c denote the size of the critic minibatch after gating. For PPO, we minimize

LcPPO=12Bc∑i=1Bcwic(Vϕ(si)−yi)2\mathcal{L}_c^{\mathrm{PPO}} = \frac{1}{2B_c}\sum_{i=1}^{B_c} w_i^c \left(V_\phi(s_i)-y_i\right)^2

where yiy_i is the pre-update value estimate plus the unnormalized GAE advantage for transition ii. For SAC and TD3, we minimize

LcSAC/TD3=1Bc∑i=1Bcwic∑j=12(Qϕj(si,ai)−yi)2\mathcal{L}_c^{\mathrm{SAC/TD3}} = \frac{1}{B_c}\sum_{i=1}^{B_c} w_i^c \sum_{j=1}^{2}\left(Q_{\phi_j}(s_i,a_i)-y_i\right)^2

SAC uses an entropy-regularized Bellman target, and TD3 uses a Bellman target with target policy smoothing. The weight wicw_i^c scales the prediction error for each sample and leaves the target yiy_i unchanged.

Question 3: Grammar Issues

Thank you so much for pointing out the grammar issues. We will proofread and revise the whole paper carefully.

Question 4: Missing RG-TAIL Overview

We apologize again for the lack of clarity in the Methodology section. We introduced TAIL first for two reasons. First, it reflected the actual development of the project, as our mechanistic analysis of TAIL motivated the design of the reliability gate. Second, we also included TAIL as an internal baseline. Presenting the methods in that order made it harder to see how RG-TAIL, our final method, works as a whole.

In retrospect, a standalone comparison with an earlier version of our method adds limited insight. In the revision, we will remove the standalone presentation and comparison of TAIL from the main paper. The Methodology section will open with an end-to-end overview of RG-TAIL, showing how it generates synthetic rollouts, assigns transition-level trust, and routes synthetic data to actor and critic updates.

We recognize that the paper needs a major revision to address these issues and improve readability. We sincerely thank the reviewer for their detailed and constructive feedback.

Additional: 5K Budget Results from Rerunning

We have rerun RG-TAIL (PPO, SAC, TD3) and three baselines under 5K budget. As shown in the tables below, RG-TAIL with SAC achives the highest raw return, and averagely RG-TAIL achieves the highest normalized return.

Raw returns at 5K

TaskRG-TAIL (PPO)RG-TAIL (SAC)RG-TAIL (TD3)MBPOTD-MPC2DreamerV3
MountainCarContinuous−66.98 (0.05%)−1.28 (0.00%)−1.84 (0.00%)−0.10 (0.00%)−0.05 (0.00%)−47.87 (3.90%)
Hopper192.1637.2727.6326.9136.38108.32
HalfCheetah−346.18−35.20−440.66−45.28−117.29−276.23
Walker2d177.12−21.523.6649.292.55151.92
Swimmer26.5217.7728.66−8.36−4.4617.61
LunarLander−405.06 (0.00%)−164.87 (1.00%)N/A−865.00 (0.00%)−183.85 (0.00%)−161.77 (0.00%)
Average−70.40−27.97−76.51−140.42−44.45−34.67

Normalized returns at 5K

TaskRG-TAIL AverageMBPOTD-MPC2DreamerV3
MountainCarContinuous0.5090.9930.9940.000
Hopper0.1280.0000.0210.177
HalfCheetah0.0230.1250.0930.022
Walker2d0.1690.1560.0000.500
Swimmer0.9640.0000.1150.766
LunarLander0.7250.0000.8510.879
Average0.4200.2120.3460.391

Thank you so much for working hard on my comments. I am still having concerns about the global gate, like the added values, right update, success metrics, etc. Thus, I would like to maintain my rating.

AuthorsOpenReview ↗

Thank you again

We sincerely thank the reviewer for their thoughtful assessment and recognition of our responses. We completely agree that additional experiments and reruns are needed to resolve the issues. We deeply appreciate your time in this work, and we will continue to refine the study and present a stronger paper.

Reviewer_qUNvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies when synthetic experience generated by learned world models should be used for reinforcement learning under limited real-data budgets.

The proposed method, Reliability-Gated Trust-Aware Imagination Learning (RG-TAIL), combines two mechanisms. First, a transition-level trust model is trained to predict an exponentially transformed one-step dynamics error using held-out real transitions. Its predictions guide rollout initialization, actor-loss weighting, and synthetic-rollout termination. Second, a higher-level reliability gate determines whether synthetic transitions contribute to both actor and critic updates, only critic updates, or neither.

The reliability score combines agreement between deterministic and probabilistic world models with the realized length of synthetic rollouts. When reliability is low, the coefficient of variation of synthetic episode returns determines whether critic learning should also be blocked.

Experiments span six environments, PPO/SAC/TD3, six real-data budgets, and ten random seeds. The paper reports improvements over internal baselines and comparisons with MBPO, TD-MPC2, and DreamerV3, alongside component ablations, threshold sensitivity, failure analysis, and runtime measurements.

Critical strengths
  1. A useful distinction between local trust and global permission to learn from synthetic data.

    RG-TAIL recognizes that continuously downweighting individual transitions cannot always prevent harmful synthetic updates. The ability to disable actor updates while retaining critic updates, or to exclude synthetic data entirely, provides an interpretable fallback mechanism. The reported LunarLander failure also illustrates the trade-off between conservative filtering and suppressing useful experience.

  2. A substantial collection of internal analyses.

    The paper includes trust-function ablations, removal of individual components, threshold sensitivity, alternative routing designs, dynamic recalibration, budget scaling, and runtime profiling. Appendix E additionally reports hierarchical-bootstrap comparisons with multiple-comparison correction and acknowledges that improvements over TD-MPC2 and DreamerV3 are not statistically significant at the 5K budget. These analyses are valuable, although they do not resolve the broader evaluation concerns.

Critical weaknesses
  1. The evaluation protocol and external-baseline results do not yet support a reliable comparison.

    Section 4 states that real transitions are collected using an online PPO policy and subsequently used to train fixed world models. It is unclear whether downstream learners receive a fixed dataset, continue collecting real experience, or follow different protocols across configurations and external baselines. This distinction is fundamental: training from a fixed behavior dataset and online model-based learning pose different distribution-shift and exploration problems.

    RG-TAIL results are also averaged over applicable algorithms and configurations, whereas external baselines are individual methods. This comparison does not isolate the contribution of the proposed mechanism under a common backbone.

    Several baseline patterns warrant implementation validation. TD-MPC2 remains near the same low returns across budgets, while MBPO’s HalfCheetah return drops from 4,351.49 at 100K to -374.29 at 500K. Such behavior is not impossible, but it cannot by itself justify the conclusion that additional synthetic experience becomes unnecessary or harmful. Validated reference implementations, complete learning curves, and matched data and training protocols are needed.

  2. The success metric and aggregation choices substantially weaken the headline conclusions.

    The paper defines success uniformly as an episode return greater than zero. This is not a validated task-completion criterion across these environments. For example, TD-MPC2 receives 100% success on Walker2d with a return of approximately 41, whereas RG-TAIL has a much higher return of approximately 162 but only 73.4% success at 5K. This demonstrates that the metric can rank limited positive-return behavior above stronger control performance.

    Raw average returns combine environments with very different reward scales, while the alternative min–max normalization depends on the compared methods. Neither resolves the need for meaningful per-task evaluation and uncertainty estimates.

    The central motivation also remains insufficiently tested. Deterministic models outperforming probabilistic models does not directly establish that uncertainty fails to predict synthetic-transition utility. Furthermore, the learned trust target is itself a transformed prediction error, rather than a downstream-utility target. A direct comparison of uncertainty, prediction-error-based trust, and measured learning benefit is missing.

After Rebuttal: The rebuttal is helpful, which addresses most of my initial concerns so I keep the score of 3

Questions for authors
  1. Data and budget accounting: Is downstream training offline on a fixed PPO-collected dataset, online with continued interaction, or a mixture? Please specify this separately for every configuration and external baseline. Are collection, calibration, validation, and evaluation interactions accounted for consistently?

  2. Matched baseline validation: Are MBPO, TD-MPC2, and DreamerV3 based on official implementations or reimplementations? Can their behavior be validated under standard online protocols, and can RG-TAIL be compared with an otherwise identical backbone under matched data access and update budgets?

  3. Evaluation metrics: Can the results be reported using task-appropriate completion criteria and standard return-based summaries with confidence intervals? How is the headline approximately +19% improvement calculated?

  4. Utility alignment: How strongly do probabilistic uncertainty, predicted trust, and global reliability correlate with measured downstream benefits from synthetic data on held-out conditions? Can prediction-error-based trust be compared directly with an uncertainty-based filter using the same model and rollout protocol?

  5. Reward and state scaling: How are state dimensions normalized before computing prediction errors and model disagreement? How sensitive is the gate to reward offsets? In particular, synthetic-return CV can change under reward shifts without corresponding changes in dynamics-model accuracy.

  6. Implementation and reporting: Please define the critic trust weights and reconcile the rollout-termination thresholds and conflicting LunarLander results. How are rewards and termination conditions obtained for imagined transitions, and how are PPO likelihood ratios and advantages handled when using stored real transitions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qUNv (1/3)

We thank the reviewer for their valuable feedback.

Weakness 1 & Question 1: Unclear Evaluation Protocol

Thank you for pointing out the ambiguity in our description. We will revise Section 4 to clarify when real experience is collected, how it is used in downstream training, and how evaluation is kept separate.

For each task and budget, an online PPO collector gathers a fixed dataset that is shared across RoI, TAIL, and RG-TAIL. We fit the world models on this dataset and freeze them before downstream training. No downstream learner collects additional real experience for training. Real-only configurations train on the fixed dataset. Imagined-only configurations train on synthetic rollouts initialized from states in that dataset. Hybrid configurations train on both real and synthetic experience.

By contrast, MBPO, TD-MPC2, and DreamerV3 collect their own experience online, and the common budget caps training interactions with the real environment. We view this project as a competition where each method is given the same fixed budget. Therefore, each external method retains its own learning and control procedure, including model rollouts, planning, and update schedules as applicable. Evaluation episodes are run separately and are never used for training.

Since each external baseline uses its own downstream RL algorithm, we consider it reasonable to use RG-TAIL’s average performance across all applicable backbones and configurations at the same real-interaction budget as the basis for comparison. However, Table 11 also reports the per-algorithm averages for RoI, TAIL, and RG-TAIL.

Weakness 2 & Question 3: Success Matrix

We defined an episode as successful when its return was greater than zero because several of the six environments have no official task-completion criterion. Our intent was to provide a shared metric across all six environments. We agree that this threshold is an inappropriate proxy for task completion.

MountainCarContinuous has an explicit goal-reaching event, and the LunarLander documentation considers an episode solved when its return is at least 200. The four MuJoCo locomotion tasks have no explicit completion event. Hopper and Walker2d can terminate early when the agent enters an unhealthy state, while HalfCheetah and Swimmer run to a time limit. In all four tasks, the episode return captures forward progress and control costs.

In the revision, we will report mean episodic return with 95% confidence intervals across training seeds as the primary metric for all six environments. We will also report the MountainCarContinuous goal-reaching rate and the fraction of LunarLander episodes that meet the documented 200-point threshold. We will remove the success rates based on positive returns.

The original average +19% gain can be reproduced from the Average rows of Tables 1, 2, and 19. We treated 5K and 10K as extremely low budgets and 500K as an extremely high budget. At each of these budgets, we subtracted each reported external baseline’s positive-return rate from RG-TAIL’s rate:

BudgetSourceMBPOTD-MPC2DreamerV3
5KTable 1+17.0 pp+6.0 pp+1.7 pp
10KTable 2+17.5 pp+13.1 pp+9.8 pp
500KTable 19+51.7 pp+39.9 pp—

The eight comparisons received equal weight, yielding 156.7 ÷ 8 = 19.5875 percentage points.

Question 2: Matched Baseline Validation

We also observed the unusually flat TD-MPC2 returns across budgets and the sharp drop in MBPO’s HalfCheetah return from 100K to 500K. We cannot yet determine whether these patterns arose from the real-interaction budget, implementation details, or other aspects of training. We agree that these observations alone are insufficient to conclude that additional synthetic experience becomes unnecessary or harmful, and we will revise the discussion accordingly.

The submitted results for MBPO, TD-MPC2, and DreamerV3 used our local reimplementations. Appendix I.2 documents their main hyperparameters. We have started rerunning all three baselines using their official implementations under their standard online training protocols. The revision will update the baseline results and include learning curves against cumulative real-environment interactions, uncertainty across seeds, and details of the code versions, warm-up, and update schedules.

AuthorsOpenReview ↗

Response to Reviewer qUNv (2/3)

Question 4: Utility Alignment

As we acknowledged in Appendix A (Q4), the resulting trust score is only indirectly connected to policy improvement, and our current design does not fully close the gap between model-level reliability and task-level utility. We agree that the manuscript overstated the extent to which RG-TAIL closes this gap and the evidence that prediction-error-based trust is a better filter than probabilistic uncertainty. We will revise those claims.

Our central empirical claim concerns the performance of RG-TAIL as a complete method relative to the evaluated baselines. To evaluate the utility alignment, we ran additional experiments with PPO and the probabilistic world model (PWM) at the 5K and 10K real-interaction budgets, using 20 paired training seeds per environment and budget.

For the correlation analysis, we measured the downstream return gain from adding the full candidate batch to the fixed real dataset while holding the number of PPO updates constant. We then computed the Spearman correlation across seeds between this gain and the batch mean of each signal. We used negative predictive variance so that a higher score represents lower uncertainty. Predicted trust had a higher correlation point estimate in 3 of 6 environments at 5K and 5 of 6 at 10K.

For the matched filtering comparison, we ranked the same 4,096 candidate transitions by predicted trust or by negative predictive variance and retained the top 2,048 under each rule. The two conditions used the same fixed real dataset, PPO update budget, and evaluation seeds. Trust-based filtering yielded higher mean returns in 5 of 6 environments at both 5K and 10K. The unadjusted 95% confidence intervals for the return difference were above zero for HalfCheetah at 5K and Walker2d at 10K, and included zero for the other comparisons.

5K real-interaction budget

EnvironmentPredicted trust ρNegative predictive variance ρTrust filter − low-variance filter return [95% CI]
LunarLander+0.170+0.045+12.17 [−6.22, +33.12]
MountainCarContinuous−0.484−0.427−0.44 [−1.03, +0.16]
Hopper+0.164+0.146+3.37 [−10.65, +16.49]
HalfCheetah−0.195+0.311+34.35 [+13.86, +54.75]
Walker2d+0.099+0.083+9.76 [−8.36, +28.64]
Swimmer−0.289−0.263+0.92 [−0.29, +2.51]

10K real-interaction budget

EnvironmentPredicted trust ρNegative predictive variance ρTrust filter − low-variance filter return [95% CI]
LunarLander−0.092−0.238+18.76 [−0.90, +39.76]
MountainCarContinuous−0.002−0.006+0.65 [−0.86, +2.10]
Hopper−0.186−0.229+0.02 [−1.85, +1.92]
HalfCheetah+0.050−0.090−3.47 [−11.57, +5.12]
Walker2d−0.090−0.200+26.87 [+12.33, +42.62]
Swimmer−0.242−0.230+0.13 [−4.20, +3.85]

Question 5: Reward and State Scaling

We compute the one-step prediction error in the observation coordinates returned by the environment, without per-dimension standardization. The scalar error scale used for the trust target and the division by state dimension in the WM–PWM disagreement score do not account for differences in coordinate units. We will make this explicit in the Methodology section.

In Section 3.2.2, we wrote that “CV characterizes the stability of imagined rollouts.” This wording is too broad. CV measures the dispersion of trajectory returns relative to the magnitude of their mean. Even for trajectories of equal length, the offset can change the mean return and move the CV across the gate threshold while leaving the standard deviation unchanged.

To quantify this sensitivity, we used the unified implementation to run a diagnostic at 5K across 34 environment, backbone, and world-model configurations with one seed each. We held the calibration trajectories and reliability scores fixed, added a constant to the predicted rewards, and recomputed the CV and gate decision. Per-step offsets of −0.1 and +0.1 changed the gate in four and one configurations, respectively. For TD3 with WM on HalfCheetah, a +0.1 offset raised the CV from 1.979 to 2.535, crossing the threshold of 2.0 and switching the gate from critic-only to block-all.

This experiment shows that the gate can switch when the reward zero point changes even though the predicted dynamics and trajectories remain fixed. We will revise Section 3.2.2 and Appendix A (Q2) to describe CV as a measure of relative return dispersion under the chosen reward definition and report the fixed-trajectory diagnostic.

AuthorsOpenReview ↗

Response to Reviewer qUNv (3/3)

Question 6: Implementation and Reporting

  1. Critic trust weights. We set wic=clip⁡(τi/τˉ,0.05,1)w_i^c = \operatorname{clip}(\tau_i / \bar{\tau}, 0.05, 1) for admitted synthetic critic samples when reliability is below 0.40 and the variance of their trust scores within the minibatch exceeds 0.01. Here τˉ\bar{\tau} is the mean trust of those samples. All other admitted samples receive unit weight.

  2. Rollout termination. We stop a rollout before storing the first transition whose trust falls below the calibrated cutoff saved in the trust checkpoint. The value 0.01 is the trust variance threshold for enabling loss weighting, and 0.05 is the loss-weight floor.

  3. LunarLander results. We will cross-check all reported results against the archived runs and revise the affected tables and text.

  4. Imagined rewards and termination. The selected world model predicts reward and termination through separate learned heads. A predicted termination probability above 0.5 marks the transition as terminal. The trust cutoff and rollout horizon can also end a rollout.

  5. PPO on stored real transitions. We score each stored action under the policy immediately before the update block and use that probability as the fixed denominator in the PPO-style likelihood ratio. We compute GAE separately within each trajectory using the pre-update value function. True termination masks bootstrap. Timeouts and dataset boundaries end the advantage trace while retaining the next-state bootstrap.

We take responsibility for the inconsistencies between the submitted manuscript and the implementation. Addressing them requires a major revision. We have started a comprehensive rerun under the unified implementation and will update the methodology, experimental results, and affected analyses in the revised manuscript.

Additional: 5K Budget Results from Rerunning

We have rerun RG-TAIL (PPO, SAC, TD3) and three baselines under 5K budget. As shown in the tables below, RG-TAIL with SAC achives the highest raw return, and averagely RG-TAIL achieves the highest normalized return.

Raw returns at 5K

TaskRG-TAIL (PPO)RG-TAIL (SAC)RG-TAIL (TD3)MBPOTD-MPC2DreamerV3
MountainCarContinuous−66.98 (0.05%)−1.28 (0.00%)−1.84 (0.00%)−0.10 (0.00%)−0.05 (0.00%)−47.87 (3.90%)
Hopper192.1637.2727.6326.9136.38108.32
HalfCheetah−346.18−35.20−440.66−45.28−117.29−276.23
Walker2d177.12−21.523.6649.292.55151.92
Swimmer26.5217.7728.66−8.36−4.4617.61
LunarLander−405.06 (0.00%)−164.87 (1.00%)N/A−865.00 (0.00%)−183.85 (0.00%)−161.77 (0.00%)
Average−70.40−27.97−76.51−140.42−44.45−34.67

Normalized returns at 5K

TaskRG-TAIL AverageMBPOTD-MPC2DreamerV3
MountainCarContinuous0.5090.9930.9940.000
Hopper0.1280.0000.0210.177
HalfCheetah0.0230.1250.0930.022
Walker2d0.1690.1560.0000.500
Swimmer0.9640.0000.1150.766
LunarLander0.7250.0000.8510.879
Average0.4200.2120.3460.391

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.