acceptodds
Withdrawn from ICLR 2027

Value-Guided Variational Latent Dynamics for Offline Skill Stitching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study building agents capable of solving long-horizon tasks in the context of offline reinforcement learning (RL). Existing RL methods effectively learn individual skills. However, seamlessly combining these skills to tackle long-horizon tasks presents a significant challenge, as the termination state of one skill may be unsuitable for initiating the next skill, leading to cumulative distribution shifts. Previous works have studied skill stitching through online RL, while such methods can be time-consuming and may raise safety concerns when learning in the real world. Some other trajectory chaining approaches in previous literature can be utilized as offline skill stitching, but they either rely on time-intensive model-based planning, or neglect the underlying dynamics system implicit in the environment. In this paper, we propose to encode the states separated in time by several steps in a self-supervised manner via a variational auto-encoder. We then align the latent representations with the real actions, modeling the latent dynamics. Given that the aggregated datasets from all skills provide diverse and exploratory data, such latent dynamics can likely be leveraged to generate necessary transitions for stitching skills. During inference, we sample a population of latent actions and make selections according to a score-based state assessment. Extensive experimental results across benchmarks of both navigation and manipulation validate the effectiveness of our approach in comparison to baseline methods under offline settings.

Primary area:
reinforcement learning
Keywords:
Skill Stitching, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x7g9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VG-VLD, which combines variational latent dynamics learned from pooled offline skill data with a value- and distance-based evaluator to connect predefined skill policies.

Critical strengths

VG-VLD combines a shared latent dynamics model with a skill-conditioned evaluator to connect predefined policies using their existing offline data. This provides a practical mechanism for reusing skill policies without online fine-tuning.

Critical weaknesses
  • Given the pooled skill dataset, the paper does not establish why separate skill policies and a stitching module are preferable to goal-conditioned or hierarchical methods such as HIQL, GAS(ICML 2025), and AQM(ICML 2026).

  • Eq. (8) has the same form as HILP’s temporal-distance representation objective, yet HILP is not cited. The additional contribution beyond existing components needs clearer articulation. [HILP] Foundation Policies with Hilbert Representations, ICML 2024

  • Appendix D.8 describes executing sampled generated trajectories, whereas BiTrajDiff originally uses augmented data to train an offline RL policy. This change leaves unclear whether the performance gap reflects algorithmic superiority or the modified baseline implementation.

  • The paper does not investigate whether poor results arise from generation errors, trajectory selection, stopping criteria, or time-budget consumption. Its ablations examine VG-VLD components but do not explain the comparative performance gaps.

  • The statement that all baselines underperform No-Stitching on "bottom burner → light switch" is inconsistent with Table 2, where BiTrajDiff achieves higher AR and CSR-2.

  • Maze experiments use only PointMaze "stitch" datasets filtered to trajectories ending at three selected goals. Retained dataset sizes, state coverage, and inter-skill overlap are not reported, and sensitivity to data quantity and goal selection is not evaluated.

  • Experiments cover only state-vector PointMaze and Kitchen settings. Neither more complex locomotion dynamics, such as AntMaze, nor visual observations are evaluated, and extension to visual environments is not substantively discussed.

  • Despite its section title, “Conclusion and Limitations” identifies no concrete limitations concerning the predefined skill sequence, data coverage, baseline adaptations, or evaluation scope.

Questions for authors

Please refer to the concerns raised in the Weaknesses section.

Reviewer_C8XwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Q1. Could the authors compare SkillDuet with a baseline using the same number of successful SFT trajectories, with comparable trajectory lengths and effective SFT weights, but collected without skill guidance? This would help clarify whether the main benefit comes from skill internalization or from having more successful training demonstrations.

Q2. How frequently does the environment harness provide feedback at different stages of a trajectory? In particular, does feedback delivered in later stages contribute more to task success than early-stage feedback? An early-only versus late-only feedback comparison would help validate the paper's central motivation.

Q3. Could the authors report success-rate improvements within each trajectory-length group in Figure 2, ideally with uncertainty estimates? The current contribution-based metric is difficult to interpret because the groups contain different numbers of tasks.

Q4. How stable are the improvements over the strongest baselines across different training seeds on Hard-ALFWorld? Given the relatively small margins in several settings, this information could materially affect the assessment.

Critical strengths

S1. An effective approach to offline skill stitching with strong empirical results.

The paper addresses a practical limitation of independently trained skill policies: successful completion of one skill does not guarantee that the resulting state is suitable for the next skill. I find the idea of learning a shared transition model from aggregated skill datasets appealing, particularly because it avoids requiring additional environment interaction for learning the stitching policy.

The reported improvements are also substantial. In Table 1, VG-VLD achieves an average return of 2.68 on OGBenchMaze, compared with 2.04 for Vanilla CEM. Similarly, Table 2 shows that the proposed method improves the average conditional success rate on two-skill Kitchen tasks from 0.342 for the strongest competing baseline to 0.625. These results suggest that the method is effective, at least for the evaluated short skill compositions.

S2. The score-based evaluator is reasonably motivated and supported by ablation results.

The proposed evaluator considers not only the value function of the next skill but also the temporal distances to states in the offline datasets. This is a reasonable design choice, since value estimates alone may be unreliable when the agent encounters states outside the training distribution of a particular skill.

The ablation in Table 8 is useful in this regard. The full evaluator achieves an average CSR-2 of 0.625 on Kitchen, compared with 0.418 for Value-Only and 0.569 for Distance-Only. Although the full method is not better for every skill combination, the results generally support the benefit of combining these two sources of information.

Critical weaknesses

W1. The comparisons with several generative stitching baselines are difficult to interpret because their original learning procedures have been substantially modified.

My main concern is the implementation of the data-augmentation-based baselines. As discussed in Section 2, methods such as DiffStitch, SCoTS, ASTRO, and BiTrajDiff are designed to generate additional trajectories and subsequently train an offline RL policy or diffusion planner using the augmented data.

However, the implementations described in Appendix D differ considerably from these original approaches. For example, SCoTS and ASTRO are adapted by generating trajectories in advance and sampling them from a buffer during the stitching phase, rather than using their complete policy-learning pipelines. The DiffStitch implementation similarly uses generated transitions and an inverse dynamics model for direct execution instead of training the final offline RL policy.

I understand that these methods require adaptation because the paper studies the composition of independently trained skills rather than learning a separate policy for each complete task. Nevertheless, the resulting comparisons do not necessarily reflect the capabilities of the original methods. Their relatively poor performance could partly result from these adaptations rather than limitations of the underlying approaches.

This matters because a major empirical claim of the paper is that VG-VLD substantially outperforms existing generative and planning-based stitching methods. I would find this claim more convincing with stronger justification for the adaptations, or additional comparisons that retain the original learning procedures under a comparable experimental setting.

W2. The results on longer skill sequences do not fully support the claimed effectiveness for long-horizon task completion.

The paper is motivated by the difficulty of composing multiple skills over long horizons. However, the strongest results are primarily observed for two-skill transitions and the three-goal navigation benchmark.

The additional Kitchen experiments in Appendix E raise some concerns. In Table 10, VG-VLD achieves a higher average return than No-Stitching for four-skill tasks (1.48 vs. 1.27), but its average CSR-4 is actually lower (0.071 vs. 0.089). Several task combinations also have extremely low conditional success rates for the third or fourth skill.

These results suggest that much of the improvement may come from successfully completing earlier skills rather than reliably completing the entire sequence. Since average return rewards partial completion, it does not necessarily reflect the ability to solve a complete long-horizon task.

I think this distinction deserves more attention. In particular, the paper would benefit from reporting the proportion of episodes that successfully complete all required skills, together with a more detailed analysis of failures at later stitching stages. The conditional success rates are informative, but their interpretation also depends on how many episodes reach each stage, especially when some entries are undefined.

Questions for authors

Q1. Could the authors clarify why the data-augmentation-based baselines are evaluated using sampled stitching trajectories rather than their original augmented-policy training procedures? Have the authors considered comparing against versions that retain the complete training pipelines of DiffStitch, SCoTS, or ASTRO? I believe this is important for establishing the fairness of the main experimental comparisons.

Q2. In the four-skill Kitchen experiments (Table 10), the average CSR-4 of VG-VLD is lower than that of No-Stitching despite the improvement in average return. Could the authors provide the complete-task success rates and the number of episodes reaching each stitching stage? This would help clarify whether the method consistently improves long-horizon task completion or primarily benefits earlier transitions.

Q3. There appears to be an inconsistency between Section 4.3 and Algorithm 2. Section 4.3 states that stitching starts after the previous skill succeeds, whereas Algorithm 2 performs the stitching phase before executing the current skill policy. It also appears to reference skill (i+1) when processing the final skill. Could the authors clarify the actual execution order and provide corrected pseudocode if necessary?

Q4. The translator is trained using latent variables inferred from observed state transitions, whereas inference samples latent variables from the prior and selects the highest-scoring candidates from a population of 100,000. How reliably do the predicted future states correspond to the actual states reached after executing the translated action chunks? An analysis of prediction errors for selected candidates, rather than only randomly sampled candidates, would help establish whether the evaluator is exploiting inaccurate model predictions.

Reviewer_4CfhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses long-horizon tasks that require stitching skills in an offline reinforcement learning setting. Although agents can learn individual skills effectively from offline data, composing them remains challenging: the terminal state of one skill may not align with the starting conditions required by the next. The authors propose Value-Guided Variational Latent Dynamics (VG-VLD), a fully offline approach for connecting independently learned skills through two main components:

  • Latent dynamics modeling: A temporally chunked Conditional Variational Autoencoder (CVAE) learns multi-step state transitions from the combined datasets of all skills. By encoding these transitions in a latent space and aligning the latent representations with corresponding action chunks, the model captures the environment’s underlying dynamics.
  • Value-guided planning at inference time: To transition between skills, the model samples candidate latent action sequences( or action chunk) and selects the best sequence using a score-based evaluator. The evaluator combines the next skill’s Q-values with a temporal distance metric to assess predicted future states, guiding the agent toward a state from which it can successfully execute the next skill.

They evaluated their method on both navigation and multi-objective manipulation benchmarks and show better performance over several baselines, including planning-based method, data augmentation methods, and others.

Critical strengths
  • Well motivation of latent dynamics modelling: The use of a temporal-chunked CVAE redues the accumulated errors.
  • Computational Efficiency: By utilizing a variational latent dynamics model and sampling multi-step "action chunks" directly from the latent space, the approach avoids the computationally expensive, multi-turn optimization required by traditional step-wise model-based planning methods.
  • Extensive experiments: The paper cover two domains, including navigation and multi-object manipulation benchmarks.
Critical weaknesses
  • Lack of novelty: The paper simply models multi-step state transition dynamics via ELBO (Eq. (4)) and obtains action chunk through these learned dynamics and value function. Ultimately, this approach can be viewed as a straghtforward multi-step extension of single-step dynamics.
  • Action chunking is already widely used: Many existing methods use action chunks to improve computational efficiency.

The primary weakness of this paper is the naive approach to learning the multi-step transition dynamics, p(st+H∣st,zt)p(s_{t+H}|s_{t}, z_t). I would expect to see deeper exploration here, specifically (1) a comparative analysis detailing the specific challenges and trade-offs of training multi-step transition models versus traditional single-step models, and (2) the introduction of a more sophisticated learning method or objective for the latent representation ztz_t to better encapsulate action chunks or skills.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.