acceptodds
Under review as a conference paper at ICLR 2027

Value-guided Implicit Maximum Likelihood Estimation for Offline Reinforcement Learning

Abstract

Offline reinforcement learning aims to learn effective policies from pre-collected datasets without further environment interaction. However, offline datasets often exhibit multimodal behavior distributions, making accurate behavior modeling difficult and increasing the risk of selecting out-of-distribution actions during policy improvement. Recent diffusion- and flow-based policies improve multimodal modeling but often rely on iterative generation, distillation, or auxiliary optimization, leading to substantial computational overhead. To address these issues, we propose Value-guided Implicit Maximum Likelihood Estimation (VIMLE), an efficient offline RL framework that enables multimodal behavior modeling, behavioral consistency filtering, and value-guided action selection. Specifically, VIMLE uses multiple conditional IMLE generators for one-step parallel action generation, filters unreliable candidates through ensemble agreement, and selects high-value actions using an independently learned value function. This decoupled design enables policy improvement over behavior-consistent candidates while avoiding multi-step generation and auxiliary policy training. Extensive evaluations on D4RL demonstrate competitive performance across diverse tasks, while substantially reducing training time and inference latency compared with diffusion-based approaches. Code is provided in the supplementary material.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Implicit Maximum Likelihood Estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6h3iOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VIMLE, an offline reinforcement learning framework that combines conditional IMLE with value-guided action selection. Multiple IMLE generators produce diverse action candidates; ensemble agreement filters potentially unreliable actions; and independently trained IQL critics select the high-value candidates. Experiments on D4RL report competitive performance across MuJoCo and AntMaze, with additional evaluations on Kitchen and Adroit. The authors also report substantial reductions in training time and inference latency compared with diffusion-based approaches.

Critical strengths

The framework is simple and avoids extra policy optimization: decoupling IMLE generation from value learning gives one-step action generation. The evaluation covers a reasonable range of offline RL domains, and the speedup over Diffuser is consistent with the one-step design.

Critical weaknesses

The methodological novelty is incremental. VIMLE builds on conditional IMLE and standard IQL, and ensemble-based consistency filtering is the main additional component. The paper does not establish whether the filtering mechanism offers substantial advantages over simpler alternatives. A comparison against nearest-neighbor filtering with a single generator, or other lightweight candidate-selection strategies, would isolate the contribution directly.

The link between ensemble agreement and behavioral support is not established. All generators are trained on the same offline dataset and may share similar modeling errors. Appendix D connects nearest-neighbor distances to the learned conditional density, but it does not show that agreement under the learned density guarantees support under the actual behavior distribution. Quantitative measurements of OOD action rates or behavior-mode coverage would make this argument testable.

The ensemble-size ablation in Table 6 works against the filtering claim. Increasing the ensemble size improves the overall score but reduces the scores on all three HalfCheetah tasks; halfcheetah-medium falls from 73.43 with one generator to 60.19 with five. Stronger filtering removes useful candidates in this regime, and the paper does not discuss the trade-off or consider an adaptive filtering strategy.

The efficiency comparison does not separate the sources of the speedup. VIMLE is considerably faster than Diffuser (Table 3), but IQL and TD3+BC still have lower inference latency. The paper should distinguish the gain over iterative generative policies from the gain over offline RL methods more broadly, and report the computational cost of candidate generation, consistency filtering, and value ranking separately.

Questions for authors

Could you compare ensemble agreement with nearest-neighbor filtering using a single generator on a few representative D4RL tasks, matching the candidate budget, so that the contribution of the filtering mechanism is isolated? Could you report OOD action rates or behavior-mode coverage for the retained candidates, so that the connection between ensemble agreement and support under the behavior distribution can be checked empirically? How should the HalfCheetah degradation in Table 6 be interpreted? Is the trade-off between ensemble size and per-task scores expected given that all generators share the same training distribution? Could you report the computational cost of candidate generation, consistency filtering, and value ranking separately, so that the speedup over Diffuser can be attributed to one-step generation rather than to implementation differences?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6h3i

We thank the reviewer for the constructive suggestions. We have conducted additional experiments and revisions to address the concerns below.

Q1 & W1: Compare with Simpler Filtering Strategies

To evaluate ensemble-based consistency filtering, we compared VIMLE with Single Nearest-Neighbor Filtering (M=1M=1, Table 7), Centroid Filtering (Appendix G.5) , and Min-Distance Filtering (Appendix G.5).

Across 12 MuJoCo tasks, VIMLE achieves an average score of 94.18, outperforming Centroid Filtering (69.42) and Min-Distance Filtering (71.11) on all tasks. Table 7 further shows that VIMLE-5 achieves a higher average score (90.32) than Single Nearest-Neighbor Filtering (VIMLE-1, 88.41) across 9 tasks. These results support the advantages of ensemble-based consistency filtering over simpler alternatives.

Q2 & W2: OOD Action Rates and Behavior-Mode Coverage

We agree that generators trained on the same offline dataset may share modeling errors. However, different stochastic latent variables zz used in IMLE training help improve the independence of modeling errors across generators.

To examine the relationship between ensemble agreement and true behavior support, we added quantitative analyses in Appendix E.2. Fig. 11 shows strong Spearman correlations (0.873–0.927) between consistency scores and true behavior support distances, indicating that lower consistency scores generally correspond to smaller support distances. We further evaluated OOD action rates and behavior-mode coverage:

Ensemble SizeOOD RateMode Coverage
Before Filtering20.309%99.996%
M=1M=13.359%86.330%
M=3M=30.273%77.396%
M=5M=50.080%77.895%

As MM increases from 1 to 5, the OOD rate decreases from 3.359% to 0.080%, while mode coverage remains around 78% for M=3M=3 and M=5M=5. These results demonstrate that ensemble-based filtering substantially reduces OOD actions while retaining considerable behavior-mode coverage, providing empirical support for its effectiveness.

Q3 & W3: Performance Degradation on HalfCheetah

We agree that larger ensembles improve overall performance but may cause overly conservative filtering. To investigate the HalfCheetah degradation, we added Table 8 in Appendix G.1, analyzing top-10% high-Q candidate retention and the relative reduction in maximum predicted Q-value (QdropQ_{\mathrm{drop}}).

As MM increases from 1 to 5, high-Q retention decreases on both HalfCheetah and Walker2d. However, QdropQ_{\mathrm{drop}} increases from 2.065%, 3.038%, and 0.978% to 4.168%, 5.396%, and 4.658% across the three HalfCheetah tasks, while remaining relatively stable on Walker2d. These results suggest that larger ensembles may excessively filter valuable high-Q candidates on HalfCheetah, potentially contributing to its performance degradation.

Q4 & W3: Ensemble Size and Shared Modeling Errors

Although all IMLE generators share the same training dataset, independent initialization and stochastic training introduce diversity across generators. Appendix D further supports variance reduction through ensemble averaging under certain assumptions. However, larger ensembles also incur higher computational costs. So we selected M=5M=5 to balance overall performance and computational cost.

We acknowledge that fixed ensemble sizes and filtering strengths may not suit all environments. Adaptive filtering is a promising direction for future work, and we have added this discussion and the corresponding limitations to Appendix G.1.

Q5 & W4: Inference Latency Breakdown and Sources of Speedup

We measured the inference cost of each VIMLE component: candidate generation takes 0.391 ms, consistency filtering 0.243 ms, and value ranking 0.304 ms. VIMLE's efficiency advantage primarily lies in avoiding iterative denoising used by generative policies such as Diffuser. In contrast, IQL and TD3+BC require only a single policy-network forward pass, resulting in lower inference latency. VIMLE incurs additional costs for consistency filtering and value-guided selection.

ComponentLatency (ms)
Candidate Generation0.391
Consistency Filtering0.243
Value Ranking0.304

We added this inference-time breakdown to Appendix C.

We hope these clarifications address the reviewer's concerns.

Reviewer_cbMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Value-guided Implicit Maximum Likelihood Estimation (VIMLE), an offline RL framework that keeps behavior modeling and value learning separate.. Several IMLE generators with BC and one IQL critics are used and trained. At test time each method samples candidate actions and keeps the ones that fall in the dense region of the generator outputs, with the one with highest Q value to execute. Over D4RL benchmark the performance is competitive to the baseline methods while the training is faster than flow based diffussion models.

Critical strengths
  • This paper is well written and easy to follow. The motivation of "multimodal behavior distributions" and critics making inaccurate Q value estimation from OOD data are classic challenges in RL domain.

  • This method follows a one step generation that is simple and fast, with no distillation stage or extensive hyper parameter tuning required, a clear advancement from diffusion/FQL.

  • The performance over D4RL benchmarks are competitive with diffusion and flow based methods.The ablation design clearly demonstrates a real failure mode where argmax-Q over generative samples exploits low-density candidates, and a simple density filter fixes it on narrow datasets. It is also a practical finding regardless of the IMLE framing.

Critical weaknesses
  • This paper seems to be a combination of previous works, and the combination of them did not show evidence from the novel parts that lead to major improvements. For example, in approaches like EMaQ already showing efforts of sampling actions from a behavior model and picking the one with the highest Q, the genuine novel components of this paper seems to be the filter in IMLE; If they are compared with these baselines then a clear evidence for the improvements can be confirmed.

  • The baseline selection is not fully justified, as an example, in Table includes FQL, while in Table 2 only includes IFQL, the IDQL-style baseline from the FQL paper, but not FQL itself. (Although I expect the performance of IFQL might be higher than FQL), why are they not using the same set of baselines?

  • Some of the details of the experiment design are not explained, e.g. the versions of benchmarks and baselines, hyper parameters of Value-guided IMLE variants and sample count, etc.

  • The theoretical analysis in Appendix D is mostly decorative, e.g. it is writing the expectation or experience into equations rather than showing optimality. (minor weakness)

Questions for authors
  • How does the performance change with the number of candidates kept? this should be included as an ablation study.

  • I wonder if it is possible to show the results of same critic and max-Q selection with a non-IMLE behavior model (e.g., Gaussian or few-step diffusion) with and without your filter?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cbMd

We thank the reviewer for the constructive suggestions. We have conducted additional experiments and revisions to address the concerns below.

Q1: Sensitivity to Candidate Count

We added sensitivity experiments on the number of generated candidates NN and retained candidates HH in Appendix G.3, summarized below. VIMLE shows low sensitivity within reasonable ranges, though H has a slightly greater impact: small H may discard high-value actions, while large H may retain unreliable OOD candidates.

Candidate Count N3264128
Average Score89.2391.4190.22
Retention Size H81632
Average Score86.7091.4188.76

Q2: Comparison with Alternative Behavior Models and Filtering

We added experiments in Appendix G.4, replacing conditional IMLE with conditional VAE (CVAE), GMM, and one-step diffusion generators, while keeping the same IQL critic and max-Q selection, both with and without consistency filtering.

Without filtering, CVAE, GMM, and one-step diffusion achieve average normalized scores of 21.89, 31.19, and 29.01, respectively, which improve to 43.45, 77.82, and 66.73 with filtering, demonstrating its effectiveness across different generators. Nevertheless, VIMLE achieves a higher average score of 94.18, further highlighting the general effectiveness of consistency filtering and the additional advantage of conditional IMLE.

Behavior Modelw/o Filteringw/ Filtering
CVAE21.8943.45
GMM31.1977.82
One-step Diffusion29.0166.73
Conditional IMLE (VIMLE)—94.18

W1: Technical Novelty and Contribution of Consistency Filtering

While VIMLE shares candidate-based action selection with EMaQ, its key distinction is an ensemble-based consistency filtering mechanism tailored to IMLE's one-step generation, applied before value-guided selection. Sections 4.4–4.5 validate the effectiveness of consistency filtering, while Appendix G.4 shows the advantage of conditional IMLE over CVAE, GMM, and one-step diffusion. Together, these results support the effectiveness of consistency filtering and conditional IMLE beyond candidate-based value selection.

We added EMaQ comparisons to Tables 2 and 5. VIMLE achieves average scores of 75.5 and 55.77 on AntMaze and Adroit, respectively, outperforming EMaQ (30.8 and 38.37), providing further evidence of its effectiveness.

W2: Baseline Selection and Comparability

The baseline sets initially differed because we adopted reported results from different prior studies. We thank the reviewer for pointing out this inconsistency and have added FQL results to Table 2 for a more consistent comparison.

W3: Experimental Settings and Reproducibility

We thank the reviewer for pointing out the missing experimental details. We added dataset versions in Appendix A.1 (MuJoCo/AntMaze: v2; Kitchen: v0; Adroit: v1) and clarified baseline implementations, configurations, and sources in Appendix A.2.

Appendix E.1 details the value-guided IMLE variants: Weighted uses β=3.0\beta=3.0 and wmax⁡=100w_{\max}=100, while Regularized uses λ=0.1\lambda=0.1. Both use K=3K=3 for IMLE training and share VIMLE's architecture and hyperparameters (Appendix H). All VIMLE experiments use 5 seeds with 10 evaluations per seed. These settings are clarified in the revision.

W4: Theoretical Justification of Consistency Filtering

We agree that Appendix D does not establish policy optimality guarantees in practical environments but provides theoretical support for ensemble-based consistency filtering. Specifically, Eq. 9 in Appendix D.1 establishes an inverse relationship between nearest-neighbor distances and local generator density. Eq. 12 in Appendix D.2 further relates the ensemble consistency score to the local densities of individual generators, while Eq. 13 further provides a variance bound supporting ensemble averaging.

We acknowledge that the analysis relies on local continuity, sufficient sampling, and conditional independence. These limitations remain directions for future work.

We hope these clarifications address the reviewer's concerns.

Reviewer_FxAYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Value-guided Implicit Maximum Likelihood Estimation (VIMLE), an offline reinforcement learning framework combining multimodal behavior modeling with value-guided action selection. The method trains multiple conditional IMLE generators to produce diverse action candidates through one-step generation. It then uses ensemble-based behavioral consistency filtering to identify reliable candidates and selects the highest-value action using independently trained IQL critics. The authors evaluate VIMLE on D4RL MuJoCo and AntMaze benchmarks, with additional experiments on Adroit and Franka Kitchen. The reported results show competitive performance and reduced training and inference costs compared with diffusion-based policies.

Critical strengths
  1. An intuitive and efficient framework. The proposed framework is intuitive and relatively straightforward. Combining conditional IMLE for multimodal action generation, ensemble-based consistency filtering, and value-guided action selection provides a practical alternative to iterative diffusion- and flow-based policies. The computational efficiency results are promising: the method substantially reduces training time and inference latency compared with Diffuser while avoiding iterative generative sampling and additional policy distillation. This practicality is the main advantage the method offers over iterative generative policies.

  2. Reasonably broad evaluation with competitive results and useful ablations. The empirical evaluation covers 18 MuJoCo and AntMaze tasks in the main paper, with additional experiments on Adroit and Franka Kitchen in the appendix. The reported results are competitive across different offline dataset qualities. VIMLE achieves particularly strong results on several medium-replay and medium-expert tasks while maintaining reasonable performance on sparse-reward AntMaze environments. The paper also includes useful ablation studies examining ensemble-based filtering, IMLE-based policy learning strategies, ensemble size, and the number of reference samples. Together, these results support the method's practical viability.

Critical weaknesses
  1. The technical novelty appears somewhat incremental, and the contributions of IMLE and the consistency filtering are not sufficiently validated.

    • Novelty. The method primarily combines existing components, including IMLE-based behavior generation, IQL value learning, and candidate-based action selection. Similar ideas of generating behavior-supported action candidates and selecting high-value actions have already been explored in offline RL methods such as BCQ and IDQL. The main novelty lies in the ensemble-based consistency filtering mechanism, but its contribution beyond existing behavior-support estimation techniques is not sufficiently established.
    • Limited theoretical justification. The analysis in Appendix D connects nearest-neighbor distances to the local density of the learned generators under assumptions of sufficiently large reference sets and regular conditional densities. However, high agreement among independently trained generators does not necessarily guarantee that an action lies within the support of the true offline behavior distribution, especially when the generators share systematic modeling errors, a limitation acknowledged in Appendix D.
    • Possible over-conservatism. As shown in Table 6, increasing the ensemble size from 1 to 5 improves performance on several Hopper and Walker2d tasks but degrades performance on all three HalfCheetah tasks. For example, HalfCheetah-medium decreases from 73.43 to 60.19, suggesting that the filtering mechanism may discard useful high-value actions. This possible failure mode, also discussed in Appendix G, deserves further investigation, particularly regarding its sensitivity to multimodal behavior distributions.
    • Incomplete ablations. The ablations do not fully isolate the contribution of IMLE and the proposed consistency filtering. It would be useful to compare VIMLE against simpler behavior models, such as conditional VAEs or Gaussian mixture models, equipped with the same ensemble filtering and IQL-based action selection, to determine whether the gains come specifically from IMLE or from the general candidate-generation and value-selection framework.
    • Unanalyzed inference-time hyperparameters. The final action selection depends on the ensemble size, the number of generated candidates, the number of reference samples, and the number of retained candidates. While the paper studies ensemble size and reference sample count, the sensitivity to candidate-set size and retention ratio remains unclear. Increasing the candidate count could also amplify Q-value estimation errors during maximum-value selection.
  2. The comparative evidence is incomplete and not fully controlled.

    • Missing OFQL baseline. The empirical comparison omits an important and directly relevant baseline, One-Step Flow Q-Learning (OFQL) [1], despite citing it in the references. OFQL also enables efficient one-step action generation without additional distillation and reports strong performance on D4RL benchmarks. Since computational efficiency is a central contribution of VIMLE, comparing against OFQL in terms of policy performance, training time, and inference latency would better establish the advantages of the proposed method.
    • Uncontrolled comparisons. Appendix A.2 indicates that most baseline scores are collected from different published papers rather than reproduced under a unified implementation and evaluation protocol. Although this is common in offline RL research, differences in training budgets, hyperparameters, implementations, and evaluation protocols could affect the comparison, particularly when the performance differences between VIMLE and the strongest baselines are relatively small.
    • Non-uniform advantages. VIMLE substantially underperforms SUMO on HalfCheetah-medium and HalfCheetah-medium-replay, while its average improvement over the strongest listed baselines on AntMaze is relatively modest. More analysis of when VIMLE outperforms or underperforms alternative offline RL methods would strengthen the empirical conclusions.
    • Untested multimodality. Since the method is primarily motivated by modeling multimodal behavior distributions, the current evaluation does not sufficiently test this capability in more diverse manipulation settings. Although the D4RL medium-replay and medium-expert datasets contain heterogeneous behaviors, evaluating VIMLE on OGBench [2] single-task manipulation environments (e.g., Cube, Scene, or Puzzle), whose play datasets contain diverse manipulation behaviors, would provide stronger evidence for the proposed method's effectiveness.

[1]: Nguyen, Xuan Thanh, and Chang Yoo. "One-step flow q-learning: Addressing the diffusion policy bottleneck in offline reinforcement learning." ICLR 2026.

[2]: Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." ICLR 2025.

Questions for authors
  1. Why does increasing the ensemble size from 1 to 5 degrade performance on all three HalfCheetah tasks in Table 6? Can the authors analyze whether the consistency filtering removes useful high-value candidates, particularly those belonging to less frequent behavior modes?
  2. Are the reported baseline results directly comparable under the same evaluation protocols? Please specify the dataset versions and relevant protocol differences, and reconcile the baseline sources in Appendices A.2 and F.
  3. On one or two existing tasks, how does VIMLE compare against a simpler candidate-based approach that uses a conditional VAE or Gaussian mixture behavior model with the same consistency filtering and IQL critic? This would help isolate the specific advantage of conditional IMLE.
  4. Using the existing synthetic examples, can the authors provide a more direct empirical validation of the relationship between the ensemble consistency score and actual out-of-distribution action risk? For example, does a lower consistency score reliably correspond to higher behavior density or lower Q-value estimation error?
  5. On a few representative tasks, how sensitive is VIMLE to the number of generated action candidates N and the number of retained actions H? In particular, does increasing the candidate set amplify Q-value overestimation, and how does the consistency filter mitigate this effect?
  6. Can the authors provide training-time and inference-latency comparisons against OFQL on the hopper-medium-expert-v2 task used in Table 3?
  7. Since multimodal behavior modeling is a central motivation of VIMLE, evaluation on more diverse offline manipulation datasets, such as the OGBench [2] single-task Cube, Scene, and Puzzle environments with play datasets, would be a useful direction for future work.
4 comments
AuthorsOpenReview ↗

Response to Reviewer FxAY

We thank the reviewer for the constructive suggestions. We have conducted additional experiments and revisions to address the concerns below.

Q1 & W1.3:Impact of Ensemble Size on Performance and Candidate Filtering

We agree larger ensembles may cause overly conservative filtering. To investigate HalfCheetah's performance degradation, we added Tab.8 to Appendix G.1, comparing top-10% high-Q retention and the reduction in maximum predicted Q-value (QdropQ_{\mathrm{drop}}).

Tab.8 shows that increasing MM reduces high-Q retention on both HalfCheetah and Walker2d, but QdropQ_{\mathrm{drop}} rises on HalfCheetah, suggesting that filtering may discard useful high-value actions, particularly those from less frequent behavior modes. Nevertheless, VIMLE achieves strong performance across most evaluated tasks, demonstrating its overall effectiveness. We added the limitation in Appendix G.1.

Q2 & W2.2:Baseline Comparability and Evaluation Protocols

We use published results to preserve their original configurations and avoid reimplementation bias, following common practice in offline RL research. While implementation details may vary across studies, all methods use the same datasets and normalized score, providing a consistent basis for comparison. Baseline sources are documented in Appendix A.2.

We added dataset versions in Appendix A.1 (MuJoCo/AntMaze: v2; Kitchen: v0; Adroit: v1) and corrected citations in Appendix F to align with Appendix A.2.

Q3 & W1.4:Comparison with other Generative Behavior Models

We added experiments in Appendix G.4, replacing conditional IMLE with conditional VAE (CVAE), GMM, and one-step diffusion. Their average scores are 43.5, 77.8, and 66.7, respectively, compared with 94.2 for VIMLE, demonstrating the advantage of conditional IMLE in behavior modeling. Further analysis is provided in Appendix G.4.

Q4 & W1.2:Relationship Between Consistency Scores and OOD risk

Building on the toy experiments in Appendix E.2, we compute the Spearman correlation between consistency scores and distances to true behavior support.

As shown in Fig. 11, the correlation increases from 0.873 to 0.916 and 0.927 with ensemble size, indicating that lower scores correspond to smaller support distances. These results support the effectiveness of ensemble agreement in identifying potentially OOD actions.

Q5 & W1.5:Sensitivity to Candidate Numbers N and H

We added sensitivity experiments on N and H in Appendix G.3. For N=32,64,128, average scores are 89.2, 91.4, and 90.2; for H=8,16,32, they are 86.7, 91.4, and 88.8. VIMLE shows low sensitivity within reasonable ranges, though H has a slightly greater impact: small H may discard high-value actions, while large H may retain unreliable OOD candidates.

We further analyze maximum predicted Q-values on hopper-medium-expert-v2. As N increases, maximum Q-values rise before filtering but remain nearly stable afterward, suggesting that consistency filtering mitigates the risk of selecting overestimated actions while preserving value-guided selection.

NBefore FilteringAfter
161280.01279.5
641290.71280.9
2561295.31281.0

Q6 & W2.1:Performance and Efficiency Comparison with OFQL

As no official implementation is available, we implemented OFQL based on the original paper. Under the same setup, VIMLE achieves a higher score (112.2 vs. 110.2), with lower training time (2.02 vs. 2.52 hours) and inference latency (0.950 vs. 1.195 ms) on hopper-medium-expert-v2, demonstrating VIMLE's advantages in both performance and efficiency.

Q7 & W2.4:Evaluation on OGBench Manipulation Tasks

We evaluated VIMLE on four OGBench manipulation environments, achieving average success rates of 96.5%, 52.5%, 30.2%, and 44.0%, outperforming all baselines on three. These results further support VIMLE's effectiveness on diverse multimodal manipulation tasks. Details are provided in Appendix F.

W1.1:Technical Novelty

We agree that candidate generation and value selection have been explored in BCQ and IDQL. VIMLE differs by introducing ensemble-based consistency filtering, which uses averaged nearest-neighbor distances across IMLE generators to identify behavior-consistent candidates before value-guided selection, reducing reliance on potentially unreliable Q-values. Appendices G.4–G.5 further demonstrate its advantages over alternative behavior models and simpler filtering strategies.

W2.3:Non-uniform Performance Gains

We acknowledge that VIMLE's gains are not uniform. As analyzed in Appendix G.1, overly conservative filtering may discard useful high-Q actions on HalfCheetah, potentially contributing to its lower performance than SUMO. While gains on AntMaze are modest, VIMLE remains competitive across sparse-reward tasks and achieves strong performance across most evaluated tasks.

We hope these clarifications address the reviewer's concerns.

Thank you again for your detailed and constructive feedback. We hope our additional experiments and clarifications address your concerns, and we look forward to further discussion.

I thank the authors for their thorough rebuttal and additional experiments. The newly added ablations isolating conditional IMLE and consistency filtering, the sensitivity studies on candidate and retention counts, the OGBench results, and the comparison with OFQL resolve my primary empirical concerns. While the methodological novelty remains somewhat incremental and fixed ensemble filtering exhibits task-dependent conservatism (as seen on HalfCheetah), the paper is now much better substantiated. Accordingly, I will raise my score to 3.

We sincerely thank you for your time and thoughtful consideration of our rebuttal and additional experiments. We greatly appreciate your constructive feedback, which has helped us improve the paper.

Comments on the paper

2 comments

We sincerely thank the reviewer for their constructive feedback and continued support. We greatly appreciate your time and thoughtful consideration of our rebuttal and additional experiments.

AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their constructive feedback. We appreciate the positive aspects highlighted in their reviews. Reviewer FxAY notes VIMLE's intuitive framework, efficiency, and competitive performance. Reviewer cbMd highlights its clear presentation, one-step generation, and practical consistency filtering. Reviewer 6h3i also notes the framework's simplicity and efficiency over iterative generative policies.

We also appreciate the reviewers' critical comments, which have helped us identify areas for improvement. To address the reviewers' concerns, we conducted additional experiments and revised the manuscript. The main revisions are highlighted in blue. Specifically, we:

  1. We compared VIMLE with alternative behavior models (CVAE, GMM, and one-step diffusion) and simpler filtering strategies to validate the contributions of conditional IMLE and consistency filtering (Appendices G.4–G.5).

  2. We added consistency–support correlations, OOD rates, and mode coverage analyses (Appendix E.2), and investigated HalfCheetah degradation and filtering limitations (Appendix G.1).

  3. We added OGBench evaluations, EMaQ and FQL baselines, an OFQL comparison in our response, and candidate-count sensitivity analyses. We also clarified inference costs, theoretical assumptions, and experimental settings (Appendices A, C, F).

We hope these additions provide further empirical support for VIMLE and help address the reviewers' concerns. We thank you again for your valuable feedback.

Thank you for your hard work and support.

Best regards,

The authors of Submission 5387.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.