- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether test-time model-based planning can improve offline RL performance while reducing hyperparameter sensitivity. It proposes SHARP, which combines horizon-specific action advantages using inverse-variance weights estimated from ensemble rollouts. The paper further argues that behavior cloning is sufficient as an action proposer, leading to SHARP-BC. Experiments on D4RL locomotion and Adroit evaluate performance, sensitivity to planning and offline RL hyperparameters, and alternative policy-extraction objectives.
Critical strengths
-
The paper addresses a practically important evaluation problem. Hyperparameter selection without online interaction is a genuine difficulty in offline RL. Studying performance across configurations, rather than reporting only the best tuned result, is useful. The distinction between action proposal and test-time action evaluation also provides an informative perspective on simplifying policy extraction.
-
The empirical investigation includes substantial supporting analyses. Beyond the main comparisons, the paper examines alternative aggregation rules, computational budgets, model capacity, policy-extraction objectives, and a flow-based proposer. The appendices also report runtime and explicitly discuss failure cases on Adroit cloned datasets. These analyses help establish the scope and limitations of the approach.
Critical weaknesses
-
The performance gains over strong planning baselines are modest and inconsistent. In Table 2, SHARP-BC achieves a locomotion average of 84.0 with ReBRAC and 82.8 with IQL, compared with 83.5 for DV. Thus, the strongest variant improves the average by only 0.5 points, while the other variant performs slightly worse. There are also substantial task-level deficits: on Hopper-medium, both variants achieve approximately 68, compared with 83.6 for DV; on HalfCheetah-medium-expert, they achieve 99.4/102.3, compared with 106.2 for MOPP. The Adroit average does improve, particularly through Pen-cloned, but performance remains poor on Hammer-cloned, Door-cloned, and Relocate-cloned. Table 2 does not provide uncertainty estimates sufficient to establish the reliability of the small aggregate locomotion gain. These results support competitive performance on selected tasks, but not a clear, broadly consistent performance advantage. Reduced tuning could justify similar returns, but that makes the strength of the robustness evidence especially important.
-
The evaluation only partially supports the broad claim of requiring no planning-specific tuning. Figure 1 selects the best inverse temperature separately for each horizon, and Figure 3 reports the best planning configuration. These are useful controlled analyses, but do not themselves demonstrate deployment without online selection. Table 1 averages over a chosen hyperparameter grid, which measures sensitivity under that grid rather than the performance of a configuration selected entirely offline. Table 2 provides stronger evidence by fixing (H=8) and (\kappa=10), but Appendix E.3 explains this choice as taking the largest tested values; the paper does not establish its transfer to tasks excluded from configuration development. Moreover, the IQL robustness sweep includes the AWR coefficient, which changes the extracted actor that SHARP-BC subsequently replaces with BC. Robustness to that coefficient should be distinguished from robustness to value-learning hyperparameters. The evidence supports reduced sensitivity within the evaluated settings, but the broader tuning-free claim needs qualification.
-
The reliability interpretation of the horizon weights is insufficiently validated. Equations (4)–(5) treat low variance across models and particles as evidence that a horizon provides a reliable action estimate. However, this variance also reflects stochastic transitions and future action sampling, while agreement among models does not rule out shared prediction bias. Estimates at different horizons additionally reuse the same rollout prefixes and are correlated. The weighting rule may work as a heuristic, but the paper does not directly show that its precision estimates track actual return-estimation errors or action-ranking accuracy. Appendix B.2 compares against hard horizon selection and return-based aggregation, but a uniform average of horizon-wise advantages would be a useful control for separating the benefit of averaging from the benefit of uncertainty weighting. This leaves the central explanation for the method’s robustness incompletely supported.
Questions for authors
-
Performance clarification: Are seed-level results or uncertainty estimates available for Table 2, particularly the 84.0 versus 83.5 locomotion comparison? Is the gain distributed across tasks or concentrated in a small subset?
-
Evaluation clarification: Were (H=8) and (\kappa=10) fixed before evaluating the Table 2 tasks? Can the IQL robustness results be separated into sensitivity to the expectile parameter and sensitivity to the AWR coefficient?
-
Mechanism clarification: Do existing analyses measure the relationship between estimated precision and actual prediction or ranking error, or compare against uniform horizon averaging? Such evidence would materially affect my assessment of the proposed weighting mechanism.
3 comments
Thank you for your response. I have no further questions, and I’m happy to raise my score to 3.
We are very grateful to the reviewer for acknowledging our rebuttal and for the effort the reviewer put into helping us improve our submission.
Response to Reviewer Re52
We sincerely thank the reviewer for the thorough and constructive comments. Please find our responses to your questions below.
We thank the reviewer for suggesting seed-level uncertainty. First, we would like to correct a reporting error: the original Table 2 reported scores averaged over κ∈0.1,0.5,1.0,5.0,10.0 rather than at κ=10, as stated in the original manuscript. The revised Table 2 now reports κ=10 results with per-dataset standard deviations; the averaged scores are included in Table 6 (Appendix D.4) of the revision.
On 10 of 17 datasets, both SHARP-BC variants (IQL and ReBRAC) exceed the strongest baseline’s (DV) reported score by at least one standard deviation. Both also achieve higher average scores than the strongest baseline on locomotion (86.1/87.2 vs. 83.5) and Adroit (73.9/76.9 vs. 70.6). These results show that SHARP-BC achieves performance comparable to baselines that use task-specific tuning while using a single fixed planning configuration across all tasks.
We thank the reviewer for raising these concerns. We agree that “requiring no planning-specific tuning” overstates our claim. While SHARP-BC has tunable planning hyperparameters, we kept them fixed across all tasks in our evaluation. We have revised the paper wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483)
For our evaluations, we leverage the definition that “A method in offline RL consists of an algorithm and a fixed sampling range for each hyperparameter [1].”
In Table 1, we report the average performance over the hyperparameter grid to simulate expected return when online selection is unavailable. In Table 2, we use a heuristic to choose hyperparameters from the grid before evaluating online. Both evaluations measure the performance of a configuration selected entirely offline from each hyperparameter's sampling range.
Since SHARP mitigates model-error accumulation over longer horizons, we choose the longest horizon within our computational budget. Moreover, as SHARP achieves more accurate advantage estimates, it leads us to select a larger κ from the search space considered in prior work, placing greater emphasis on high-advantage candidates. We adopt this heuristic to fix H=8 and κ=10 for SHARP-BC on all evaluated tasks without online task-specific hyperparameter selection.
We have added the rationale for this heuristic to Appendix D (line1294) and Appendix E (line 1709) of the revision.
We thank the reviewer for this suggestion. We isolate IQL’s sensitivity to τ by varying it at fixed values of α and report the median relative returns:
These results support SHARP-BC’s reduced sensitivity to value-learning hyperparameters. Since SHARP-BC replaces the action proposer with a fixed behavior cloning policy, it is unaffected by the AWR coefficient α, which only affects the extracted policy. The full results and discussion are in Appendix D.2 of the revision.
We thank the reviewer for this suggestion. We conducted two analyses to verify whether precision reflects model reliability and improves advantage estimation.
First, we measure the Pearson correlation between SHARP’s precision and rollout return error under learned versus true dynamics:
The results show a consistently negative correlation across datasets, suggesting that precision provides a useful signal of the error induced by learned dynamics models.
Next, we examine whether precision-weighted aggregation yields more accurate advantage estimates than fixed-horizon evaluation or uniform aggregation. We compute the difference between these three advantage estimates, with the reference advantages obtained using the real dynamics. The mean advantage errors across 15 locomotion datasets are:
The results show that SHARP achieves the lowest advantage error, suggesting that its precision-weighted aggregation downweights unreliable horizon-specific estimates to better approximate true dynamics advantages. The full results and experimental details are included in Figures 11, 12, and Appendix B.7 of the revision.
Reference
[1] Jackson et al. A Clean Slate for Offline Reinforcement Learning. NeurIPS 2025.