RG-TAIL: Reliability-Gated Trust-Aware Imagination Learning under Limited Budgets

Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned world models to generate synthetic experience. However, model-generated transitions can be harmful when model errors accumulate especially in data-scarce settings, where reliable synthetic experience is essential due to the high cost of real-world interaction. Our large-scale empirical study, $\text{Real-or-Imagined}$ (RoI), reveals that while synthetic data can substantially improve reinforcement learning, uncertainty estimates do not necessarily align with the long-term utility of synthetic transitions. Based on this observation, we propose $\text{Reliability-Gated Trust-Aware Imagination Learning}$ ($\text{RG-TAIL}$), a framework that combines transition-level trust estimation with a higher-level reliability gate to determine whether synthetic experience could be used in policy optimization. We perform extensive experiments across diverse environments, reinforcement learning algorithms, model configurations, and data budgets. Compared with internal baselines and representative MBRL methods, RG-TAIL achieves the best overall performance under both extremely low- and high-budget settings, improving task success rates by approximately $\text{+19}$% on average.