acceptodds
Withdrawn from ICLR 2027

TCGI: Transition-Consistency-Guided Imagination for Visual Imitation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models support visual imitation learning by generating latent trajectories for policy optimization. Expert demonstrations provide transition information that can also guide the construction of these trajectories. We introduce Transition-Consistency-Guided Imagination (TCGI), which incorporates expert transition preferences into conservative model-based imitation learning (CMIL). A learned scorer distinguishes demonstrated latent transitions from unfiltered world-model rollouts. Its scores weight the fusion of candidate successor states, augment the imitation reward, and control when an ensemble-disagreement penalty is applied. These uses connect demonstration supervision to imagined state construction and policy optimization within a fixed imagination horizon. At deployment, the actor operates directly on inferred latent states. We evaluate TCGI on manipulation and locomotion tasks, with component ablations and parameter sensitivity analyses. Reported performance gains vary across tasks and metrics, with trade-offs between late-training scores and performance over recorded evaluation intervals. TCGI provides a concrete mechanism for incorporating expert transition preferences into latent imagination and a basis for examining how this guidance affects visual imitation learning.

Primary area:
reinforcement learning
Keywords:
Imitation Learning, Model-Based Reinforcement Learning, Visual Control, World Models, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HDDyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a modification based on conservative model-based imitation learning (CMIL). The proposed method adds a scorer on top of the existing CMIL architecture, which distinguishes demonstrated latent transitions from unfiltered world-model rollouts. The method then leverages the generated score and the uncertainty of the scorer for reward augmentation to improve downstream policy and critic learning. The empirical experiments compared their proposed method with standard CMIL, ROT and BC baseline across multiple locomotion (DMControl tasks) and manipulation (Meta World, Franka Kitchen and Shadow Hand) tasks. Results demonstrate slight improvement against standard CMIL method.

Critical strengths
  • The overall presentation of the paper is generally understandable. The proposed method is simple and straight-forward (an additional reward augmentation term with the scorer's uncertainty).

  • The paper provides adequate ablation studies and multi-seed results in the appendix, which facilitates reproducibility and demonstrate the effectiveness of the proposed method.

Critical weaknesses
  • Limited novelty: The paper's proposed method is an additional scorer module on top of existing CMIL architecture for reward augmentation. This is more like a small modification improvement. Compared with standard CMIL, the proposed modification only provides a small amount of gain on cumulative rewards in a diverse set of tasks, where the importance of this proposed modification should be questioned.

  • Lack of recent baselines: The paper should compare with more recent baselines on online imitation learning with world models, such as IQ-MPC [1]. The baselines leveraged in the paper is before 2025 and seem to be quite outdated.

  • More evaluation tasks: The proposed method only evaluates on 8 manipulation and locomotion tasks, which is quite insufficient. The proposed method should evaluate its effectiveness on more complex tasks such as dog-run and humanoid-walk tasks in DMControl and some complex manipulation tasks in ManiSkill3 such as Pick YCB or Peg Insertion Side.

[1] Li, S., Huang, Z., & Su, H. (2025). Reward-free world models for online imitation learning. In Proceedings of the 42nd International Conference on Machine Learning (Vol. 267, pp. 34702–34724). Proceedings of Machine Learning Research.

Questions for authors

See weakness section.

Reviewer_65oROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Transition-Consistency-Guided Imagination (TCGI), an extension of a previous method called Conservative Model-Based Imitation Learning (CMIL). TCGI learns a discriminator to distinguish expert latent transitions from imagined transitions, which is then used downstream for a number of things including weighting and fusing candidate future states, augmenting the imitation reward, and penalizing uncertainty in world model prediction. The trained policy executes directly without imagination during inference. The paper evaluates the proposed method on eight simulated tasks, comparing against three baselines on two tasks, and comparing against CMIL on six tasks. There are some ablation experiments.

Critical strengths

The idea of using expert demonstrations to guide imagined trajectories is reasonable and provides a concrete way to learn from expert data beyond imitation learning. The proposed method is also compatible with direct policy rollouts during test time. The reported improvements over CMIL provide some evidence that the proposed method can be useful.

Critical weaknesses
  • My main concern is inadequate evaluations. The standalone BC baseline's architecture and training details are not specified in the paper. Table 1 includes BC, ROT, CMIL, whereas Table 2 compares with only CMIL, the prior art method that the proposed method develops from. Results are also presented with Last-10, Best, and AUC metrics, which, to my limited knowledge, are not quite standard metrics for reporting policy performance. These evaluations, taken together, do not form sufficient empirical evidence to prove the effectiveness of the proposed method.
  • There are quite a lot of moving pieces in this method. The ablations do not establish the contribution of the proposed mechanisms. Ablations numbers are mostly inconclusive, and also potentially confounded (e.g. "w/o filter" confounds the effects of filtering with the effects of generating single vs multiple candidates). It is unclear what makes the method work/not work.
Questions for authors
  • Could you elaborate on the training, evaluation, and model architecture details for the BC baseline?
  • What are the environment interaction budgets for each comparison? And could you elaborate on how we should interpret Table 1/2, the Last-10, and AUC metrics? What is the takeaway here?
  • Are BC, ROT or other competitive baselines available for Table 2?
Reviewer_9j2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies model bias in model-based visual imitation learning. Using CMIL as a backbone, it trains a transition discriminator to distinguish expert transitions from imagined transitions. This score has three uses: 1) to refine the predicted latent next states using the weighted score; 2) as a reward to determine whether the ensemble disagreement penalty is turned on. Experiments report improvements over CMIL on Kitchen, ShadowHand, MetaWorld, and DMC, but the gains were limited on most tasks.

Critical strengths

This paper has a clear motivation: constrain the bias of imagined transitions from a learned world model using expert transitions. The proposed method can be easily integrated into existing world-model-based imitation learning (IL) frameworks and adds no additional computational cost. The paper provides comprehensive implementation details.

Critical weaknesses

W1. State averaging and the reward choice lack sufficient justification.

  • TCGI takes a weighted average of candidate next states (Sec. 4.2). It does not distinguish differences between ensemble heads (epistemic uncertainty) from inherent environment randomness (aleatoric uncertainty). Stochastic transitions should be sampled rather than averaged. Averaging reduces variance and can produce states between modes. It may also underestimate risk.
  • The weighted average score is no lower than the unweighted average and increases with the spread of candidate scores. This makes the penalty more likely to be disabled in regions with high model uncertainty (Sec. 4.3).

W2. Empirical gains are limited, and the evidence fails to isolate the contributions of individual components.

  • In the Kitchen and ShadowHand environments, the observed improvements fall within the range of inter-seed variance (Tables 1, 6–7). The paper does not show that model bias manifests in the CMIL rollouts; for instance, it lacks an analysis of how rollout error evolves over time steps or diagnostics on model exploitation. Moreover, the paper provides no learning curves or mechanistic analysis—such as how often the penalty is triggered or the effective sample size of the weights.
  • The ablations do not isolate each component’s contribution (Sec. 5.3, Table 3). “w/o Filter” changes the model rollouts, the reward bonus, and how often the uncertainty penalty is applied. “w/o Safety” removes the entire penalty. The paper does not compare the gated penalty with the same penalty applied at every step.
Questions for authors

Q1. Where do the performance gains come from?

All three ablations perform worse than CMIL on ShadowHand Last-10 and Kitchen AUC. Do these results depend on interactions between components or on hyperparameter tuning? Please compute the reward bonus and penalty gate without softmax weights. Then repeat the ablations.

Q2. Why use DD as the imitation reward?

The reported CMIL results use DD from the inherited implementation. The original CMIL paper uses log⁡D−log⁡(1−D)\log D-\log(1-D). Please clarify this difference and evaluate both CMIL and TCGI with each reward.

Q3. How were the hyperparameters selected?

Were hyperparameters tuned on the evaluation tasks? Were the same random seeds used for tuning and final evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.