acceptodds
Withdrawn from ICLR 2027

Explicitly capturing demonstrator multi-modality for distillation into diffusion policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The diversity of a distilled diffusion policy's outputs matters beyond how it impacts the policy's success rate. In particular, downstream applications like test-time search and policy improvement require exploration. Thus, we want to distill multi-modal demonstrators into diffusion policies which reproduce their conditional action distributions. Standard data collection for behavior cloning applies a sliding window to demonstration episodes, which pairs unique observation histories with actions. These datasets can be fit by a policy that is either multi-modal, or unimodal with high Lipschitz constant to capture a diversity of actions seen at nearby observation samples. Prior work finds that diffusion policies trained on sparse data learn the latter, memorizing one action per observation. We adapt vine sampling to behavior cloning data collection to resolve this ambiguity: at some states the demonstrator visits, query it times from exactly the same observation. On a synthetic task, we study how the number of actions per observation, , affects the learned policy, and find that larger yields a policy that better matches the demonstrator's conditional distribution, including its multi-modality and diversity. On a 2D truck unloading task, vine sampling likewise leads to a policy closer to the demonstrator and substantially improves both best-of- and expert iteration performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
diffusion policies, behavior cloning, data collection, multi-modal action distributions, policy diversity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nTPxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why diffusion policies trained on sparse demonstrations may fail to preserve the demonstrator’s multimodal action distribution. The authors adapt vine sampling to behavior cloning by querying the demonstrator multiple times at selected observations, while retaining the standard diffusion training objective. Experiments on a synthetic task and a simulated 2D truck unloading task show improved distribution matching and downstream performance in best-of-N search and expert iteration at matched training dataset sizes. The main contribution is an analysis and empirical evaluation of data collection, rather than a new policy architecture or training objective.

Critical strengths
  1. The paper clearly distinguishes a policy’s capacity to represent multimodal distributions from whether its training data support learning those distributions. The controlled synthetic experiments provide useful evidence about the tradeoff between collecting more observations and collecting more actions per observation at a fixed dataset size.

  2. The evaluation goes beyond single-rollout success rates. By examining distribution matching, action diversity, and downstream search and expert iteration, the paper connects its data collection strategy to practical uses of policy diversity.

Critical weaknesses
  1. The experimental comparison does not fully isolate the effect of collecting multiple actions per observation or establish an advantage at matched collection cost. Appendix B.1 states that the vine dataset contains planned trajectories, whereas the sliding-window dataset contains executed trajectories that include controller tracking errors. The reference actions used for distributional evaluation are also plans. Consequently, differences in training targets and their alignment with the evaluation reference may contribute to the observed gains. In addition, vine collection requires approximately 30 times the computation of sliding-window collection. Matching the number of training samples therefore does not establish which approach is preferable under a fixed collection budget.

  2. The evidence is limited relative to the incremental methodological contribution. Collecting multiple teacher actions per observation has precedents, and vine sampling itself is an existing procedure. The paper’s contribution therefore depends substantially on the depth and generality of its analysis. However, the robotics evaluation covers only one 2D simulated task, without real-world validation or a systematic study of query multiplicity and re-query probability in that task. Each synthetic configuration is trained once, and the reported robotics intervals do not establish robustness across independent training runs. The results support a useful observation in the studied settings, but do not yet establish a broadly applicable data collection recommendation.

Questions for authors
  1. Clarification of the comparison: Would the advantage remain if both collection methods used planned trajectories, or both used executed trajectories? Can the current evidence separate the effect of multiple actions per observation from differences in trajectory quality and training targets?

  2. Collection efficiency: Under the same demonstrator query budget or collection compute budget, how does vine sampling compare with collecting demonstrations from more distinct observations?

  3. Robustness: Are results available for different query multiplicities, re-query probabilities, and independent training runs in the truck unloading task? These results would help establish whether the improvement is stable and how the collection budget should be allocated.

  4. Interpretation of downstream performance: Appendix B.4 reports expert iteration performance during search-based data collection and excludes search time from throughput. Are standalone policy evaluations or throughput measurements that include search time available? These would clarify whether the gains reflect improved policies and practical execution efficiency.

Reviewer_avhJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the data collection procedure affects the ability of diffusion policies to reproduce the conditional action distributions of multimodal demonstrators. The authors argue that conventional sliding-window behavior cloning produces datasets containing only one action per observation history, making it difficult for a sufficiently expressive diffusion model to distinguish genuinely multimodal demonstrator behavior from memorized individual actions.

To address this limitation, the paper adapts vine sampling to behavior cloning, querying the demonstrator multiple times from the same observation to explicitly capture alternative actions. A controlled synthetic experiment investigates the tradeoff between the number of distinct observations and the number of actions collected per observation, under a fixed training-sample budget.

The method is further evaluated in a simulated 2D truck-unloading task, using distributional fidelity, best-of-N search, and expert iteration. The results indicate that vine sampling better preserves demonstrator multimodality and leads to improved downstream search and policy improvement compared with standard sliding-window collection and several diversity-enhancement baselines.

Critical strengths
  1. The paper identifies an important limitation of conventional behavior-cloning datasets: in sparse, high-dimensional observation spaces, each conditioning input is typically associated with only one demonstrated action. This makes it difficult for a diffusion policy to recover the demonstrator's full conditional action distribution, particularly when the model is operating in a memorization regime. The synthetic experiments are well designed to investigate this phenomenon. By varying the number of actions per observation while keeping the total number of training samples fixed, the authors isolate the effects of repeated conditional sampling. The analysis using Wasserstein distance, behavioral entropy, off-support mass, and mode recall provides complementary evidence about distributional fidelity and multimodality.
  2. Rather than evaluating diversity alone, the authors investigate whether better coverage of the demonstrator's action distribution improves test-time search and subsequent policy optimization. The truck-unloading experiments compare vine sampling against standard sliding-window collection, early stopping, additional inference-time search, SIME, and PostBC. The experiments control the number of training samples and the diffusion-policy training budget. The evaluation also considers both task success and throughput, which helps distinguish useful behavioral diversity from arbitrary stochasticity. The improvements in best-of-N search and expert iteration provide evidence that preserving demonstrator multimodality can benefit downstream optimization, not merely distribution-matching metrics.
Critical weaknesses
  1. Appendix B.1 states that the vine dataset stores plans returned by demonstrator re-queries, whereas the sliding-window dataset stores executed trajectories, which differ from the plans because of controller tracking errors. This distinction is potentially consequential. The two datasets differ not only in the number of actions associated with each observation, but also in the type of action supervision. The vine policy receives planned action sequences, whereas the sliding-window policy is trained on the executed behavior after low-level control errors. Furthermore, the distributional evaluation uses demonstrator re-queries, which are themselves planned actions. This gives the vine policy a closer match between its training targets and the reference distribution used for evaluation. Although the proposed interpretation is plausible, this discrepancy makes it difficult to attribute the observed improvements exclusively to repeated sampling and multimodality preservation. A controlled experiment matching the action-label source across collection strategies would substantially strengthen the causal evidence.
  2. Although the main experiments match the number of training samples, the data collection budgets are not matched in computational cost. The authors explicitly report that collecting the vine dataset requires approximately 30 times the demonstrator computation of standard sliding-window collection. This is a substantial practical difference, particularly when demonstrator queries require expensive motion planning or physical interaction. It remains unclear whether the same resources would be better spent collecting additional conventional demonstrations or sampling more diverse initial conditions. The truck-unloading evaluation also considers only one simplified 2D simulation environment, using a stochastic planner that can be repeatedly queried from identical states. Moreover, the best-of-N search procedure relies on simulator rollouts for candidate evaluation. These assumptions may not hold in real-world robotic applications. Finally, the practical experiment uses only one vine-sampling configuration (k = 10, p = 0.1). Without a broader cost-performance analysis, it is difficult to determine how efficiently the proposed strategy trades off state coverage, conditional diversity, and demonstrator computation.
Questions for authors
  1. Appendix B.1 indicates that vine sampling stores planned action sequences while sliding-window collection stores executed trajectories. Can the authors provide a comparison in which both methods use the same type of action supervision, for example by constructing a sliding-window dataset using the demonstrator's planned actions? This would help determine whether the improvements arise from multimodality preservation rather than differences in action-label quality or alignment with the evaluation reference.
  2. Given the approximately 30-fold difference in data collection cost, how does vine sampling compare with standard sliding-window collection when the total demonstrator computation is matched rather than the number of stored training samples? Can the authors also characterize the effect of varying k and p in the truck-unloading task? These results would be important for evaluating the method's practical data-collection efficiency.
Reviewer_BwiDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to explicitly add multimodal (in terms of strategy) demonstrations to a dataset, specifically through vine sampling. The authors compare a policy trained through this multimodal-generated dataset against the demonstrator's distribution, and find better matching.

Critical strengths

The strength of this paper is its clarity of approach and writing.

Critical weaknesses

The major weakness of this paper is that it seems to be a direct application of prior work (vine sampling) with minimal modification. But then, when positioned as an application paper, the results are only on simulation, have a rather narrow evaluation setup, and lack thorough insight into valuable hyperparameters.

The authors evaluate results only on simulation, particularly ones that can be explicitly reset to specified states. This severely limits the applicability of the approach to practical robotics. Given that the method itself is not particularly novel (a direct application of vine sampling, and training on it), readers may look more towards strength in downstream application to provide interesting insight. Particularly as the authors chose robotics, it would be beneficial to demonstrate applicability to real-world settings or arbitrary simulators.

The authors utilize a best-of-N search that simulates forward in the simulator and discards candidates with failed simulations. Towards practical robotics that cannot always operate in or access a perfect simulator that can instantiate at arbitrary states and roll out flawlessly, it may be better to either use a learned (and potentially imperfect) world model for multiple simulation steps, or just simply use one-step evaluations from Q-networks to select the best from an action candidacy set (e.g. EMaQ [2] and Q-Chunking Best of N [3]). This would be closer to practical use cases.

For comparing methods, it would be better to also provide RL-finetuning experiments as an additional practical evaluation strategy. In particular, the Best-of-N style approach naturally works with exploration-based techniques such as DF-ExpEnse [4]. This would be a valuable experiment to perform - whereas additional multimodality in a policy is nice to have from a theoretical perspective, the usual practical use case of a policy does not always justify increasing multimodality where we simply want the policy to solve the task. DF-ExpEnse can showcase the benefits of improved multimodality (produced by vine sampling) for online exploration and sample-efficient finetuning, helping to add a quantitative and practical reason why we may be interested in increasing multimodality.

The authors state that "For conciseness, we have not investigated optimal values for the re-query probability pp and the query count kk." Yet these are the most interesting and important parameters, for the method. It seems quite necessary to investigate and provide insights on these values - for example, do benefits saturate at high kk?

I currently do not recommend acceptance, due to limited novelty of approach, weak and limited evaluations, and lack of key thorough ablations. I feel like there was limited insight and impact from this paper in its current draft form.

[1] Nakamoto et al., Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance, CoRL 2024.

[2] Ghasemipour et al., EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL, ICML 2021.

[3] Li et al., Reinforcement Learning with Action Chunking, NeurIPS 2025.

[4] Luo et al., DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning, ICML 2026.

Questions for authors

Questions are tied up in the previous section - would be interested in hearing the reviewer's responses.

Comments on the paper

1 comment

We thank the reviewers for their time, effort and feedback. We have decided to withdraw the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.