- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to use model-generated experience effectively when real interaction data are limited. TAIL learns transition-level trust from world-model prediction errors and uses it to weight synthetic experience. RG-TAIL adds a reliability gate that determines whether synthetic experience contributes to both actor and critic updates, critic updates only, or neither. The manuscript reports experiments across six control environments, three reinforcement learning algorithms, and multiple data budgets, together with ablations, failure cases, and statistical analyses.
Critical strengths
-
The method makes a useful distinction between actor and critic training. Synthetic data may help estimate values while giving poor directions for improving the policy. Allowing critic-only updates addresses this possibility, and using only real data when reliability is low gives the method a clear fallback.
-
The experiments cover a useful range of settings and report failures as well as gains. The study uses multiple algorithms, data budgets, and ten random seeds, with ablations and threshold-sensitivity tests. The statistical appendix reports that the 5K comparisons with TD-MPC2 and DreamerV3 are not statistically significant. This helps readers distinguish the measured gains from differences that remain uncertain.
Critical weaknesses
-
The success-rate metric is too coarse to support the headline performance claim. Section 4 counts an episode as successful whenever its return is above zero, across all six tasks. The meaning of that cutoff depends on the environment. For example, Table 18 reports a 100% success rate for every method on Hopper, although their mean returns range from 71.92 to 578.63. This metric treats substantially different levels of performance as equally successful.
The abstract emphasizes an average success-rate improvement of approximately 19 percentage points. Interpreting this number requires a task-specific justification for the zero-return cutoff, a named reference baseline, and an explicit averaging rule. The supplied scripts also use different success-rate conventions, so the authors should provide the shared postprocessing code used for the tables. These details determine what the headline improvement actually measures.
-
The paper and code use different definitions of the proposed method. Equations (5)–(6) compute reliability from disagreement between the deterministic and probabilistic world models, together with rollout length. The PPO implementation follows this definition. The SAC and TD3 implementations replace model disagreement with the mean learned transition trust. This changes the signal that decides whether synthetic data are used for actor updates, critic updates, or neither. The authors need to identify which calculation was used for the reported SAC and TD3 results.
The trust calculation also differs. Equation (2) divides prediction error by its median, while the SAC and TD3 code use the standard deviation and the PPO code uses the larger of the median and standard deviation. These choices change the resulting trust weights. In addition, Section 6.2 describes No Gate as equivalent to TAIL, but the supplied SAC No Gate and standalone TAIL implementations use different weighting choices. The authors should specify the code and settings behind these comparisons and explain how the trust-training data were kept separate from world-model fitting. Until these points are resolved, the tables cannot be tied to one clearly specified implementation.
Questions for authors
-
For the SAC and TD3 results in Tables 1–2, was reliability computed from WM–PWM disagreement as in Equations (5)–(6), or from mean transition trust as in the supplied code? Please identify the function and settings used for those runs.
-
Section 6.2 states that No Gate is equivalent to TAIL. Were their trust normalization, actor and critic weights, and other training settings identical in the reported comparison? Please explain the differences between the supplied SAC implementations.
-
What does a return above zero represent in each task, and which common postprocessing script produced the reported success rates? Please give the reference baseline and averaging rule used to obtain the approximately 19-percentage-point headline improvement.
3 comments
Thank you for clarifying how the 19-percentage-point figure was calculated and for providing the new 5K results. Given that the consistency of the gains across the remaining reruns and the gate’s contribution under a matched comparison have yet to be established, I will maintain my original rating of 2 for now.
Thank you again
We sincerely thank the reviewer for their thoughtful assessment and recognition of our responses. We completely agree that additional experiments and reruns are needed to resolve the issues. We deeply appreciate your time in this work, and we will continue to refine the study and present a stronger paper.
Response to Reviewer N5dY
We thank the reviewer for their valuable feedback.
Weakness 1 & Question 3: Success Matrix
We defined an episode as successful when its return was greater than zero because several of the six environments have no official task-completion criterion. Our intent was to provide a shared metric across all six environments. We agree that this threshold is an inappropriate proxy for task completion.
MountainCarContinuous has an explicit goal-reaching event, and the LunarLander documentation considers an episode solved when its return is at least 200. The four MuJoCo locomotion tasks have no explicit completion event. Hopper and Walker2d can terminate early when the agent enters an unhealthy state, while HalfCheetah and Swimmer run to a time limit. In all four tasks, the episode return captures forward progress and control costs.
In the revision, we will report mean episodic return with 95% confidence intervals across training seeds as the primary metric for all six environments. We will also report the MountainCarContinuous goal-reaching rate and the fraction of LunarLander episodes that meet the documented 200-point threshold. We will remove the success rates based on positive returns.
The original average +19% gain can be reproduced from the Average rows of Tables 1, 2, and 19. We treated 5K and 10K as extremely low budgets and 500K as an extremely high budget. At each of these budgets, we subtracted each reported external baseline’s positive-return rate from RG-TAIL’s rate:
The eight comparisons received equal weight, yielding 156.7 ÷ 8 = 19.5875 percentage points.
Weakness 2 & Question 1&2: Inconsistency Issues
We checked the training code and confirmed the inconsistency. We developed RG-TAIL with PPO as the primary backbone and introduced a different reliability calculation when porting it to SAC and TD3. We deeply apologize for this mistake. We have started rerunning the experiments for PPO, SAC, and TD3 with a shared implementation of the WM–PWM agreement-based reliability calculation in Eqs. (5)–(6). We will update the affected tables and analyses using the rerun results.
Additional: 5K Budget Results from Rerunning
We have rerun RG-TAIL (PPO, SAC, TD3) and three baselines under 5K budget. As shown in the tables below, RG-TAIL with SAC achieves the highest raw return, and averagely RG-TAIL achieves the highest normalized return.
Raw returns at 5K
Normalized returns at 5K