- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes VIMLE, an offline reinforcement learning framework that combines conditional IMLE with value-guided action selection. Multiple IMLE generators produce diverse action candidates; ensemble agreement filters potentially unreliable actions; and independently trained IQL critics select the high-value candidates. Experiments on D4RL report competitive performance across MuJoCo and AntMaze, with additional evaluations on Kitchen and Adroit. The authors also report substantial reductions in training time and inference latency compared with diffusion-based approaches.
Critical strengths
The framework is simple and avoids extra policy optimization: decoupling IMLE generation from value learning gives one-step action generation. The evaluation covers a reasonable range of offline RL domains, and the speedup over Diffuser is consistent with the one-step design.
Critical weaknesses
The methodological novelty is incremental. VIMLE builds on conditional IMLE and standard IQL, and ensemble-based consistency filtering is the main additional component. The paper does not establish whether the filtering mechanism offers substantial advantages over simpler alternatives. A comparison against nearest-neighbor filtering with a single generator, or other lightweight candidate-selection strategies, would isolate the contribution directly.
The link between ensemble agreement and behavioral support is not established. All generators are trained on the same offline dataset and may share similar modeling errors. Appendix D connects nearest-neighbor distances to the learned conditional density, but it does not show that agreement under the learned density guarantees support under the actual behavior distribution. Quantitative measurements of OOD action rates or behavior-mode coverage would make this argument testable.
The ensemble-size ablation in Table 6 works against the filtering claim. Increasing the ensemble size improves the overall score but reduces the scores on all three HalfCheetah tasks; halfcheetah-medium falls from 73.43 with one generator to 60.19 with five. Stronger filtering removes useful candidates in this regime, and the paper does not discuss the trade-off or consider an adaptive filtering strategy.
The efficiency comparison does not separate the sources of the speedup. VIMLE is considerably faster than Diffuser (Table 3), but IQL and TD3+BC still have lower inference latency. The paper should distinguish the gain over iterative generative policies from the gain over offline RL methods more broadly, and report the computational cost of candidate generation, consistency filtering, and value ranking separately.
Questions for authors
Could you compare ensemble agreement with nearest-neighbor filtering using a single generator on a few representative D4RL tasks, matching the candidate budget, so that the contribution of the filtering mechanism is isolated? Could you report OOD action rates or behavior-mode coverage for the retained candidates, so that the connection between ensemble agreement and support under the behavior distribution can be checked empirically? How should the HalfCheetah degradation in Table 6 be interpreted? Is the trade-off between ensemble size and per-task scores expected given that all generators share the same training distribution? Could you report the computational cost of candidate generation, consistency filtering, and value ranking separately, so that the speedup over Diffuser can be attributed to one-step generation rather than to implementation differences?
Response to Reviewer 6h3i
We thank the reviewer for the constructive suggestions. We have conducted additional experiments and revisions to address the concerns below.
To evaluate ensemble-based consistency filtering, we compared VIMLE with Single Nearest-Neighbor Filtering (M=1, Table 7), Centroid Filtering (Appendix G.5) , and Min-Distance Filtering (Appendix G.5).
Across 12 MuJoCo tasks, VIMLE achieves an average score of 94.18, outperforming Centroid Filtering (69.42) and Min-Distance Filtering (71.11) on all tasks. Table 7 further shows that VIMLE-5 achieves a higher average score (90.32) than Single Nearest-Neighbor Filtering (VIMLE-1, 88.41) across 9 tasks. These results support the advantages of ensemble-based consistency filtering over simpler alternatives.
We agree that generators trained on the same offline dataset may share modeling errors. However, different stochastic latent variables z used in IMLE training help improve the independence of modeling errors across generators.
To examine the relationship between ensemble agreement and true behavior support, we added quantitative analyses in Appendix E.2. Fig. 11 shows strong Spearman correlations (0.873–0.927) between consistency scores and true behavior support distances, indicating that lower consistency scores generally correspond to smaller support distances. We further evaluated OOD action rates and behavior-mode coverage:
As M increases from 1 to 5, the OOD rate decreases from 3.359% to 0.080%, while mode coverage remains around 78% for M=3 and M=5. These results demonstrate that ensemble-based filtering substantially reduces OOD actions while retaining considerable behavior-mode coverage, providing empirical support for its effectiveness.
We agree that larger ensembles improve overall performance but may cause overly conservative filtering. To investigate the HalfCheetah degradation, we added Table 8 in Appendix G.1, analyzing top-10% high-Q candidate retention and the relative reduction in maximum predicted Q-value (Qdrop).
As M increases from 1 to 5, high-Q retention decreases on both HalfCheetah and Walker2d. However, Qdrop increases from 2.065%, 3.038%, and 0.978% to 4.168%, 5.396%, and 4.658% across the three HalfCheetah tasks, while remaining relatively stable on Walker2d. These results suggest that larger ensembles may excessively filter valuable high-Q candidates on HalfCheetah, potentially contributing to its performance degradation.
Although all IMLE generators share the same training dataset, independent initialization and stochastic training introduce diversity across generators. Appendix D further supports variance reduction through ensemble averaging under certain assumptions. However, larger ensembles also incur higher computational costs. So we selected M=5 to balance overall performance and computational cost.
We acknowledge that fixed ensemble sizes and filtering strengths may not suit all environments. Adaptive filtering is a promising direction for future work, and we have added this discussion and the corresponding limitations to Appendix G.1.
We measured the inference cost of each VIMLE component: candidate generation takes 0.391 ms, consistency filtering 0.243 ms, and value ranking 0.304 ms. VIMLE's efficiency advantage primarily lies in avoiding iterative denoising used by generative policies such as Diffuser. In contrast, IQL and TD3+BC require only a single policy-network forward pass, resulting in lower inference latency. VIMLE incurs additional costs for consistency filtering and value-guided selection.
We added this inference-time breakdown to Appendix C.
We hope these clarifications address the reviewer's concerns.