acceptodds
Under review as a conference paper at ICLR 2027

Less Tuning, Better Planning: Simplifying Offline Reinforcement Learning via Planning

Abstract

Offline reinforcement learning (RL) aims to learn effective policies entirely from precollected data, making it attractive for domains where online interaction is expensive or unsafe. Yet, existing offline RL methods are highly sensitive to hyperparameters and design choices, often requiring online evaluation to identify a good configuration, undermining the offline setting. This work investigates whether plug-and-play model-based planning can improve both policy performance and robustness to offline RL hyperparameters. We focus on two stages of the planning pipeline: test-time planning and offline policy extraction. For test-time planning, we find that performance is highly sensitive to the planning horizon, reflecting a trade-off between incorporating long-term information and accumulating model error. To address this, we propose Soft Horizon AggRegation for Planning (SHARP), which evaluates candidate actions across multiple horizons and softly aggregates their advantages using uncertainty estimates from an ensemble of dynamics models. For policy extraction, we find that Q-function exploitation can improve zero-shot policy performance but provides little additional benefit when the policy serves as an action proposer for planning, whereas behavioral regularization remains essential. Consequently, simple behavior cloning is sufficient as an action proposer. Building on these findings, we propose SHARP-BC, a plug-and-play planning method that requires no planning-specific hyperparameter tuning. Across D4RL locomotion and Adroit dexterous manipulation tasks, SHARP-BC achieves competitive or superior performance to existing offline planning methods while substantially reducing sensitivity to both planning and underlying offline RL hyperparameters.

Primary area:
reinforcement learning
Keywords:
Offline RL, Offline Model-based Planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Re52OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether test-time model-based planning can improve offline RL performance while reducing hyperparameter sensitivity. It proposes SHARP, which combines horizon-specific action advantages using inverse-variance weights estimated from ensemble rollouts. The paper further argues that behavior cloning is sufficient as an action proposer, leading to SHARP-BC. Experiments on D4RL locomotion and Adroit evaluate performance, sensitivity to planning and offline RL hyperparameters, and alternative policy-extraction objectives.

Critical strengths
  • The paper addresses a practically important evaluation problem. Hyperparameter selection without online interaction is a genuine difficulty in offline RL. Studying performance across configurations, rather than reporting only the best tuned result, is useful. The distinction between action proposal and test-time action evaluation also provides an informative perspective on simplifying policy extraction.

  • The empirical investigation includes substantial supporting analyses. Beyond the main comparisons, the paper examines alternative aggregation rules, computational budgets, model capacity, policy-extraction objectives, and a flow-based proposer. The appendices also report runtime and explicitly discuss failure cases on Adroit cloned datasets. These analyses help establish the scope and limitations of the approach.

Critical weaknesses
  • The performance gains over strong planning baselines are modest and inconsistent. In Table 2, SHARP-BC achieves a locomotion average of 84.0 with ReBRAC and 82.8 with IQL, compared with 83.5 for DV. Thus, the strongest variant improves the average by only 0.5 points, while the other variant performs slightly worse. There are also substantial task-level deficits: on Hopper-medium, both variants achieve approximately 68, compared with 83.6 for DV; on HalfCheetah-medium-expert, they achieve 99.4/102.3, compared with 106.2 for MOPP. The Adroit average does improve, particularly through Pen-cloned, but performance remains poor on Hammer-cloned, Door-cloned, and Relocate-cloned. Table 2 does not provide uncertainty estimates sufficient to establish the reliability of the small aggregate locomotion gain. These results support competitive performance on selected tasks, but not a clear, broadly consistent performance advantage. Reduced tuning could justify similar returns, but that makes the strength of the robustness evidence especially important.

  • The evaluation only partially supports the broad claim of requiring no planning-specific tuning. Figure 1 selects the best inverse temperature separately for each horizon, and Figure 3 reports the best planning configuration. These are useful controlled analyses, but do not themselves demonstrate deployment without online selection. Table 1 averages over a chosen hyperparameter grid, which measures sensitivity under that grid rather than the performance of a configuration selected entirely offline. Table 2 provides stronger evidence by fixing (H=8) and (\kappa=10), but Appendix E.3 explains this choice as taking the largest tested values; the paper does not establish its transfer to tasks excluded from configuration development. Moreover, the IQL robustness sweep includes the AWR coefficient, which changes the extracted actor that SHARP-BC subsequently replaces with BC. Robustness to that coefficient should be distinguished from robustness to value-learning hyperparameters. The evidence supports reduced sensitivity within the evaluated settings, but the broader tuning-free claim needs qualification.

  • The reliability interpretation of the horizon weights is insufficiently validated. Equations (4)–(5) treat low variance across models and particles as evidence that a horizon provides a reliable action estimate. However, this variance also reflects stochastic transitions and future action sampling, while agreement among models does not rule out shared prediction bias. Estimates at different horizons additionally reuse the same rollout prefixes and are correlated. The weighting rule may work as a heuristic, but the paper does not directly show that its precision estimates track actual return-estimation errors or action-ranking accuracy. Appendix B.2 compares against hard horizon selection and return-based aggregation, but a uniform average of horizon-wise advantages would be a useful control for separating the benefit of averaging from the benefit of uncertainty weighting. This leaves the central explanation for the method’s robustness incompletely supported.

Questions for authors
  • Performance clarification: Are seed-level results or uncertainty estimates available for Table 2, particularly the 84.0 versus 83.5 locomotion comparison? Is the gain distributed across tasks or concentrated in a small subset?

  • Evaluation clarification: Were (H=8) and (\kappa=10) fixed before evaluating the Table 2 tasks? Can the IQL robustness results be separated into sensitivity to the expectile parameter and sensitivity to the AWR coefficient?

  • Mechanism clarification: Do existing analyses measure the relationship between estimated precision and actual prediction or ranking error, or compare against uniform horizon averaging? Such evidence would materially affect my assessment of the proposed weighting mechanism.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Re52

We sincerely thank the reviewer for the thorough and constructive comments. Please find our responses to your questions below.

Performance of SHARP-BC

We thank the reviewer for suggesting seed-level uncertainty. First, we would like to correct a reporting error: the original Table 2 reported scores averaged over κ∈0.1,0.5,1.0,5.0,10.0\kappa \in \\{0.1, 0.5, 1.0, 5.0, 10.0\\} rather than at κ=10\kappa = 10, as stated in the original manuscript. The revised Table 2 now reports κ=10\kappa = 10 results with per-dataset standard deviations; the averaged scores are included in Table 6 (Appendix D.4) of the revision.

On 10 of 17 datasets, both SHARP-BC variants (IQL and ReBRAC) exceed the strongest baseline’s (DV) reported score by at least one standard deviation. Both also achieve higher average scores than the strongest baseline on locomotion (86.1/87.2 vs. 83.5) and Adroit (73.9/76.9 vs. 70.6). These results show that SHARP-BC achieves performance comparable to baselines that use task-specific tuning while using a single fixed planning configuration across all tasks.

Tuning-free Claim & Evaluation clarification

We thank the reviewer for raising these concerns. We agree that “requiring no planning-specific tuning” overstates our claim. While SHARP-BC has tunable planning hyperparameters, we kept them fixed across all tasks in our evaluation. We have revised the paper wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483)

For our evaluations, we leverage the definition that “A method in offline RL consists of an algorithm and a fixed sampling range for each hyperparameter [1].”
In Table 1, we report the average performance over the hyperparameter grid to simulate expected return when online selection is unavailable. In Table 2, we use a heuristic to choose hyperparameters from the grid before evaluating online. Both evaluations measure the performance of a configuration selected entirely offline from each hyperparameter's sampling range.

Since SHARP mitigates model-error accumulation over longer horizons, we choose the longest horizon within our computational budget. Moreover, as SHARP achieves more accurate advantage estimates, it leads us to select a larger κ\kappa from the search space considered in prior work, placing greater emphasis on high-advantage candidates. We adopt this heuristic to fix H=8H=8 and κ=10\kappa=10 for SHARP-BC on all evaluated tasks without online task-specific hyperparameter selection.

We have added the rationale for this heuristic to Appendix D (line1294) and Appendix E (line 1709) of the revision.

Separate sensitivity of each hyperparameter of IQL

We thank the reviewer for this suggestion. We isolate IQL’s sensitivity to τ\tau by varying it at fixed values of α\alpha and report the median relative returns:

α\alphaZero-shotSHARPImprovement
0.579.0%94.7%+15.7
379.5%94.7%+15.2
1078.3%94.7%+16.3

These results support SHARP-BC’s reduced sensitivity to value-learning hyperparameters. Since SHARP-BC replaces the action proposer with a fixed behavior cloning policy, it is unaffected by the AWR coefficient α\alpha, which only affects the extracted policy. The full results and discussion are in Appendix D.2 of the revision.

Precision weights analysis

We thank the reviewer for this suggestion. We conducted two analyses to verify whether precision reflects model reliability and improves advantage estimation.

First, we measure the Pearson correlation between SHARP’s precision and rollout return error under learned versus true dynamics:

H12345678
Mean (all 15 locomotion datasets)-0.65±0.10-0.67±0.12-0.67±0.13-0.67±0.15-0.67±0.16-0.67±0.17-0.66±0.18-0.66±0.19

The results show a consistently negative correlation across datasets, suggesting that precision provides a useful signal of the error induced by learned dynamics models.

Next, we examine whether precision-weighted aggregation yields more accurate advantage estimates than fixed-horizon evaluation or uniform aggregation. We compute the difference between these three advantage estimates, with the reference advantages obtained using the real dynamics. The mean advantage errors across 15 locomotion datasets are:

Method \ H12345678
Fixed Horizon0.841.301.682.022.322.602.843.06
Uniform0.841.181.441.661.862.052.222.37
SHARP0.841.171.401.601.761.902.022.11

The results show that SHARP achieves the lowest advantage error, suggesting that its precision-weighted aggregation downweights unreliable horizon-specific estimates to better approximate true dynamics advantages. The full results and experimental details are included in Figures 11, 12, and Appendix B.7 of the revision.

Reference

[1] Jackson et al. A Clean Slate for Offline Reinforcement Learning. NeurIPS 2025.

Thank you for your response. I have no further questions, and I’m happy to raise my score to 3.

We are very grateful to the reviewer for acknowledging our rebuttal and for the effort the reviewer put into helping us improve our submission.

Reviewer_xZe5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether test-time model-based planning can reduce the hyperparameter sensitivity of offline reinforcement learning. The authors first identify planning horizon as an important source of sensitivity in existing plug-and-play planners, since longer rollouts provide more future information but also accumulate model error. They propose SHARP, which evaluates candidate actions over multiple horizons and aggregates horizon-specific advantages using precision estimated from an ensemble of dynamics models. The paper further investigates policy extraction and finds that, while Q-function exploitation is important for zero-shot policy performance, it provides little additional benefit when the policy is only used as an action proposer for planning. This motivates using a simple behavior cloning policy. Combining the two observations, the authors propose SHARP-BC. Experiments on D4RL locomotion and Adroit tasks show that SHARP-BC is competitive with tuned offline planning baselines while being substantially less sensitive to planning and underlying offline-RL hyperparameters.

Critical strengths
  1. A useful and practically relevant perspective on hyperparameter sensitivity in offline RL. Instead of only optimizing best-case return after hyperparameter tuning, the paper explicitly studies performance robustness when online model selection is unavailable. This is well aligned with the motivation of offline RL and makes the evaluation more practically meaningful.
  2. A simple but effective decomposition of proposal and optimization. The observation that behavioral regularization is important for generating plausible candidate actions, while Q-function exploitation contributes relatively little once test-time planning is used, is interesting and potentially useful beyond the specific SHARP algorithm. The experiments with BC, AWR, and DDPG+BC provide reasonable support for this claim.
Critical weaknesses
  1. The method does not fully eliminate planning hyperparameter sensitivity. Although SHARP substantially reduces sensitivity to the planning horizon, it still relies on the softmax inverse-temperature parameter κ. The authors show that replacing soft action selection with argmax hurts performance, so this remaining parameter is non-trivial. Therefore, I think the “less tuning” claim is well supported, but the method should not be interpreted as tuning-free.
  2. The robustness benefit depends on the quality of the underlying learned value function. SHARP-BC can reduce variation across reasonable offline-RL configurations, but it cannot recover when the underlying value estimates are consistently poor, as seen on several Adroit cloned datasets. This somewhat limits the generality of the robustness claim.
Questions for authors
  1. How sensitive is SHARP to the calibration quality of the ensemble variance? In particular, have the authors considered cases where ensemble disagreement is poorly correlated with actual rollout error?
  2. The paper shows that BC is sufficient as an action proposer on the evaluated benchmarks. Could the authors provide more discussion or experiments on when Q-aware proposal might still be beneficial, e.g., when the behavior dataset has broad but highly suboptimal support?
  3. Since κ remains an important parameter, could the authors clarify how much performance varies across fixed choices of κ when using the proposed SHARP-BC configuration?
4 comments
AuthorsOpenReview ↗

Response to Reviewer xZe5

We sincerely thank the reviewer for the thorough and constructive comments. Please find our responses to your questions below.

Tuning-free Claim

We thank the reviewer for raising these points and agree that “tuning-free” overstates our claim. SHARP-BC has tunable planning hyperparameters, and we kept them fixed across all evaluated tasks. We have revised the wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483)

Robustness to value function quality

We agree that SHARP-BC’s robustness depends on the quality of the underlying value function. Only when value estimates are sufficiently informative for planning can SHARP-BC reduce variation in performance across offline RL hyperparameter configurations. We discuss this limitation and clarify the scope of our claim in the Limitations and Appendix D.3.

SHARP's sensitivity to the ensemble variance quality

We thank the reviewer for this suggestion. To assess whether SHARP’s precision reflects prediction error, we analyze its correlation with the return difference between real and learned dynamics. We expect a negative correlation, as higher precision should correspond to lower error. The results are summarized below, with full results and experimental details in Appendix B.7 of the revision.

H12345678
Mean (all 15 locomotion datasets)-0.65±0.10-0.67±0.12-0.67±0.13-0.67±0.15-0.67±0.16-0.67±0.17-0.66±0.18-0.66±0.19

The results show that the correlation is consistently negative across tasks, with only HalfCheetah Random (R) and Medium (M) exhibiting moderate absolute correlation.

HalfCheetah

Dataset \ H12345678
R-0.37-0.34-0.32-0.30-0.27-0.25-0.23-0.22
M-0.52-0.51-0.48-0.46-0.43-0.41-0.39-0.39

To examine whether these weaker correlations lead to reduced SHARP performance, we compare SHARP and LOOP on these datasets across horizons 1-8.

HalfCheetah (R)

Method \ H12345678
SHARP23.224.024.324.624.824.925.025.2
LOOP23.222.822.722.422.322.121.921.7

HalfCheetah (M)

Method \ H12345678
SHARP56.759.059.759.960.160.360.460.5
LOOP56.858.457.957.456.656.154.754.1

The results show that, although the correlations are weaker on these two datasets, SHARP maintains consistently good performance as the horizon increases. Nevertheless, we agree that highly stochastic dynamics may limit SHARP's ability to accurately capture epistemic uncertainty, potentially affecting its reliability.

When Q-aware proposal might still be beneficial

We thank the reviewer for this insightful question and discuss when Q-aware proposers may benefit planning in Appendix C.2 (Figures 14 and 15). We find that Q-aware proposers offer greater benefits on random datasets, while planning performance is largely insensitive to the proposer on other dataset types. To investigate, we evaluate medium-replay datasets, where BC and Q-aware proposers perform comparably under the default planning budget. By reducing the number of sampled candidate actions K, we find that Q-aware proposers become substantially more beneficial, while their advantage diminishes as the planning budget increases.

These findings highlight a trade-off between training-time policy extraction and test-time planning. Q-aware proposers can improve sampling efficiency under limited compute, especially on more suboptimal datasets, whereas BC is simple and robust when sufficient test-time compute is available.

SHARP

DatasetAWR(0.5)AWR(0.05)BCDDPG(0.5)DDPG(0.05)
R43.137.236.239.144.8
M238.0234.3229.8230.3238.0
MR247.8246.3245.9246.9247.1
ME320.9324.7323.6323.3323.6
FR293.5292.7292.2292.6292.9

Medium-Replay

KAWR(0.5)AWR(0.05)BCDDPG(0.5)DDPG(0.05)
2211.2195.8183.7184.7222.4
8242.0240.8239.6241.2243.3
32247.8246.3245.9246.9247.1

kappa sensitivity on SHARP-BC

We thank the reviewer for the suggestions. To examine how performance varies with the choice of κ\kappa, we further analyze the sensitivity of SHARP-BC over κ∈0.1,0.5,1.0,5.0,10.0\kappa\in\\{{0.1, 0.5,1.0, 5.0, 10.0}\\}.

We report the average performance of SHARP-BC on locomotion and Adroit tasks using IQL and ReBRAC. The results suggest that, although SHARP-BC exhibits some performance drops at small κ\kappa, its performance remains generally stable at larger values.

Results and discussion appear in Appendix D.4 (Figure 19) of the revision.

MethodEnv0.10.51510
IQLLocomotion73.283.785.186.086.1
Adroit75.676.074.973.873.9
ReBRACLocomotion76.283.384.986.887.2
Adroit76.276.376.676.976.9

We would like to express our sincere gratitude to the reviewer for the thorough and constructive feedback. We are confident that our responses adequately address the concerns and questions raised by the reviewer, including the following points:

  • Revision of the tuning-free claim: We have revised the wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483).
  • Discussion of the robustness to value function quality: We have discussed this limitation and clarified the scope of our claim in the Limitations and Appendix D.3.
  • Analysis of SHARP’s sensitivity to the ensemble variance quality: We have analyzed the correlation between SHARP’s precision and the prediction error in Appendix B.7, and discussed the relationship between the ensemble variance quality and SHARP’s performance.
  • Discussion of when Q-aware proposal might still be beneficial: We examine when Q-aware action proposal might still be beneficial over using behavior cloning. We show that Q-aware proposers may be beneficial under highly suboptimal datasets and a relatively limited computational budget available for test-time planning. (Appendix C.2)
  • Analysis of the kappa sensitivity on SHARP-BC: We analyze how performance varies with the choice of the inverse temperature κ\kappa. The results show that the performance of SHARP-BC remains generally stable at larger values, supporting the heuristic of our hyperparameter selection. (Appendix D.4)

Please kindly let us know if the reviewer has any additional concerns or if further experimental results are required. We are fully committed to resolving any potential issues, should time permit. Again, we thank the reviewer for all the detailed review and the time the reviewer put into helping us to improve our submission.

Thank you for your detailed response. I would keep my positive score.

Thank you for considering our rebuttal. We hope that our responses have addressed the concerns raised in your review. If any concerns remain, or if additional experimental results would help clarify your assessment, please let us know. We would be happy to address them within the remaining time. If you find that your concerns have been resolved, we would be grateful if you would consider updating your score accordingly. We sincerely appreciate your detailed feedback and the time you have devoted to helping us improve our submission.

Reviewer_TMjXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Offline RL is highly sensitive to hyperparameters and often requires online evaluation to pick a configuration, which defeats the offline premise. The paper studies plug-and-play planning in two stages:

Test-time planning is sensitive to the horizon HH (long-term information vs. model-error accumulation). SHARP rolls out model ensembles with multiple particles, subtracts a shared cross-candidate baseline to obtain advantages, and softly aggregates them across horizons using precision (inverse variance) weights before a softmax action selection. Offline policy extraction: a (σ,η)(\sigma,\eta) intervention plus AWR/DDPG+BC checks show that Q-exploitation strongly affects zero-shot performance but barely helps when the policy is only an action proposer, whereas behavioral regularization remains essential. Hence SHARP-BC = SHARP + a behavior-cloned proposer, using one configuration (H=8,κ=10H=8,\kappa=10) for all datasets. On 15 D4RL locomotion and 8 Adroit tasks, it beats Zero-Shot/MBOP/LOOP under a protocol that averages over the full H×κH\times\kappa grid, is competitive with fully tuned baselines (locomotion 84.0 vs 83.5; Adroit 76.5 vs 73.7), and stays closer to the best attainable performance across nine IQL/ReBRAC configurations.

Critical strengths

Clean and actionable policy-extraction finding. Two independent evidence chains (the (σ,η)(\sigma,\eta) intervention and α\alpha sweeps of AWR/DDPG+BC) agree that a BC proposer suffices and Q-exploitation can be left to the planner. This removes an entire class of hyperparameters and is directly useful for future planning-based methods. Horizon sensitivity is turned into an analyzable phenomenon. Fig. 1 exposes the double-edged behavior of MBOP/LOOP as HH grows, and precision weighting addresses it mechanistically. It is a single aggregation step inside the planning loop: no retraining, runtime on par with MPPI.

Critical weaknesses

"Tuning-free" is overstated and the comparison is asymmetric. The inverse temperature κ\kappa remains; H=8,κ=10H=8,\kappa=10 are exactly the largest values in the swept grid; averaging over H,κH,\kappa is not comparable to baselines reporting tuned best configurations; and gaps such as 84.0 vs 83.5 are likely within noise at 3 seeds x 20 episodes, with no confidence intervals or significance tests. Narrow evidence and missing comparisons. Main conclusions rest on 3 environments x 3 seeds; Adroit cloned is near-degenerate, so robustness holds only on expert and Pen-cloned.

Questions for authors
  1. In Table 2, SHARP-BC is fixed to H=8,κ=10H=8,\kappa=10, which are exactly the largest values in the swept grid. How were these two values selected (were they chosen on the same evaluation tasks)?
  2. With 3 seeds x 20 episodes, are the gaps of 84.0 vs 83.5 (locomotion) and 76.5 vs 73.7 (Adroit) statistically significant? Could you report across-seed standard deviations?
  3. Do the gains come from cross-horizon aggregation, or merely from more candidates plus ensembling? Could you add a BC + mean-return-over-K-candidates baseline (no cross-horizon aggregation) and a fixed short-horizon control?
5 comments
AuthorsOpenReview ↗

Response to Reviewer TMjX

We sincerely thank the reviewer for the thorough and constructive comments. Please find our responses to your questions below.

Tuning-free Claim & Evaluation Clarification

We thank the reviewer for raising these points and agree that “tuning-free” overstates our claim. SHARP-BC has tunable planning hyperparameters, and we kept them fixed across all evaluated tasks. We have revised the wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483)

For SHARP-BC’s planning configuration, since SHARP mitigates model-error accumulation over longer horizons, we choose the longest horizon within our computational budget. Moreover, as SHARP achieves more accurate advantage estimates, it leads us to select a larger κ\kappa from the search space considered in prior work, placing greater emphasis on high-advantage candidates. We adopt this heuristic to fix H=8H=8 and κ=10\kappa=10 for SHARP-BC on all evaluated tasks without online task-specific hyperparameter selection.

We have added the rationale for this heuristic to Appendix D (line1294) and Appendix E (line 1709) of the revision.

SHARP-BC performance

We thank the reviewer for the suggestion of adding the across-seed standard deviations.

We would first like to correct a reporting error: the original Table 2 reported scores averaged over κ∈0.1,0.5,1.0,5.0,10.0\kappa \in \\{0.1, 0.5, 1.0, 5.0, 10.0\\} rather than at κ=10\kappa = 10, as stated. The revised Table 2 now reports κ=10\kappa = 10 results with per-dataset standard deviations; the averaged scores appear in Table 6 (Appendix D.4) of the revision.

On 10 of 17 datasets, both SHARP-BC variants (IQL and ReBRAC) exceed the strongest baseline’s (DV) reported score by at least one standard deviation. Both also achieve higher average scores than the strongest baseline on locomotion (86.1/87.2 vs. 83.5) and Adroit (73.9/76.9 vs. 70.6). These results show that SHARP-BC achieves performance comparable to baselines that use task-specific tuning while using a single fixed planning configuration across all tasks.

Robustness evidence on new environment

We thank the reviewer for raising this concern about the relatively narrow robustness evidence.
To broaden the robustness analysis, we further evaluate SHARP-BC’s robustness and performance on navigation (Maze2D) using ReBRAC. We summarize the experimental results below. The results show that SHARP-BC consistently improves robustness across ReBRAC hyperparameters while achieving performance comparable to DV with a fixed planning configuration (H=8, κ\kappa=10), without task-specific tuning. Full results and discussions are provided in Appendix D.3 (Figure 17 and Table 5).

Robustness to ReBRAC hyper

EnvZero-ShotSHARP-BC (Ours)
Maze2D-Medium46%61%
Maze2D-Large53%72%
Maze2D-UMaze58%64%
All Envs52%68%

Normalized return

EnvDVSHARP-BC (Ours)
Maze2D-Medium150.7166.7±1.4
Maze2D-Large203.6215.8±9.2
Maze2D-UMaze136.6143.4±9.7
Average163.6175.3

Performance gains from cross-horizon aggregation

We thank the reviewer for suggesting this ablation. We conduct an experiment to isolate the effect of using cross-horizon aggregation. Following the reviewer’s suggestion, we replace the advantage for final action selection with the mean return over all the ensembles and particles without cross-horizon aggregation (no-cross). The results are summarized below, with full details in Appendix D.5 of the revision.

IQL Locomotion

H12345678
no-cross83.8±1.286.5±0.984.9±2.382.0±2.882.8±2.083.7±1.583.1±0.983.4±1.3
SHARP-BC83.2±1.284.8±1.186.3±0.886.3±1.186.4±0.885.3±0.986.5±1.086.1±0.4

IQL Adroit

H12345678
no-cross74.6±1.174.2±1.674.0±1.573.3±1.973.3±0.973.9±0.572.1±1.170.4±1.4
SHARP-BC73.6±0.774.0±0.675.5±0.675.0±1.275.9±1.474.9±1.274.1±0.873.9±1.7

Rebrac Locomotion

H12345678
no-cross87.2±0.986.9±0.986.6±0.885.7±0.685.4±0.285.1±0.684.4±0.584.3±0.3
SHARP-BC87.2±0.987.0±0.387.7±0.387.8±0.387.4±0.887.8±0.787.1±0.687.2±0.9

Rebrac Adroit

H12345678
no-cross74.6±1.675.6±0.674.9±0.875.0±1.475.5±1.472.7±1.372.1±1.271.1±1.1
SHARP-BC74.8±1.276.9±0.675.0±0.875.0±1.175.9±0.675.5±0.976.5±0.976.9±1.4

While the variant without cross-horizon aggregation performs comparably at shorter horizons, its performance often degrades at longer horizons, requiring task-specific online evaluation for horizon selection. In contrast, SHARP-BC maintains more stable performance across tasks and horizons, highlighting the benefits of soft horizon aggregation beyond simply using a dynamics ensemble.

We would like to express our sincere gratitude to the reviewer for the thorough and constructive feedback. We are confident that our responses adequately address the concerns and questions raised by the reviewer, including the following points:

  • Revision of the tuning-free claim and clarification of the evaluation configuration: We have revised the wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483) and clarified the heuristic we use to fix the planning configuration of SHARP-BC on all evaluated tasks without online task-specific hyperparameter selection. (line 1294 and line 1709)
  • Inclusion of SHARP-BC performance std over seeds: We revise Table 2 to report per-dataset standard deviations over seeds. On 10 of 17 datasets, both SHARP-BC variants (IQL and ReBRAC) exceed the strongest baseline’s (DV) reported score by at least one standard deviation. Supporting SHARP-BC’s comparable performance to baselines.
  • Additional robustness evidence on new environment: We extend the robustness analysis of SHARP-BC to navigation (Maze2D). SHARP-BC remains robust while performing comparably to DV. (Figure 17 and Table 5)
  • Additional experiments on the performance gains from cross-horizon aggregation: We compare SHARP-BC with a variant without cross-horizon aggregation. SHARP-BC maintains more stable performance across tasks and horizons, highlighting the benefits of soft horizon aggregation. (Appendix D.5)

Please kindly let us know if the reviewer has any additional concerns or if further experimental results are required. We are fully committed to resolving any potential issues, should time permit. Again, we thank the reviewer for all the detailed review and the time the reviewer put into helping us to improve our submission.

We hope that our responses have addressed the concerns raised in your review. If any concerns remain, or if additional experimental results would help clarify your assessment, please let us know. We would be happy to address them within the remaining time. If you find that your concerns have been resolved, we would be grateful if you would consider updating your score accordingly. We sincerely appreciate your detailed feedback and the time you have devoted to helping us improve our submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for handling our submission and for the reviewers’ thoughtful feedback. Below, we summarize our responses and the corresponding revisions to the paper.

  • [Reviewers TMjX, xZe5, and Re52] Tuning-free claim & Evaluation clarification: We have revised the wording to “using a shared planning configuration across tasks without task-specific tuning.” (lines 29, 420, and 483) and clarified the heuristic we use to fix the planning configuration of SHARP-BC on all evaluated tasks without online task-specific hyperparameter selection. (line 1294 and line 1709)
  • [Reviewers TMjX and Re52] SHARP-BC performance std over seeds: We revise Table 2 to report per-dataset standard deviations over seeds. On 10 of 17 datasets, both SHARP-BC variants (IQL and ReBRAC) exceed the strongest baseline’s (DV) reported score by at least one standard deviation. Supporting SHARP-BC’s comparable performance to baselines.
  • [Reviewers xZe5 and Re52] Precision weights analysis: We analyze the correlation between SHARP’s precision and rollout error. The results show that precision highly correlates with rollout error, resulting in more accurate advantage estimates than fixed-horizon evaluation or uniform aggregation. (Appendix B.7)
  • [Reviewer TMjX] Robustness evidence on new environment: We extend the robustness analysis of SHARP-BC to navigation (Maze2D). SHARP-BC remains robust while performing comparably to DV. (Figure 17 and Table 5)
  • [Reviewer TMjX] Performance gains from cross-horizon aggregation: We compare SHARP-BC with a variant without cross-horizon aggregation. SHARP-BC maintains more stable performance across tasks and horizons, highlighting the benefits of soft horizon aggregation. (Appendix D.5)
  • [Reviewer xZe5] When Q-aware proposal might still be beneficial: We examine when Q-aware action proposal might still be beneficial over using behavior cloning. We show that Q-aware proposers may be beneficial under highly suboptimal datasets and a relatively limited computational budget available for test-time planning. (Appendix C.2)
  • [Reviewer xZe5] Kappa sensitivity on SHARP-BC: We analyze how performance varies with the choice of the inverse temperature κ\kappa. The results show that the performance of SHARP-BC remains generally stable at larger values, supporting the heuristic of our hyperparameter selection. (Appendix D.4)
  • [Reviewer Re52] Separating sensitivity of each hyperparameter of IQL: We isolate the sensitivity of each hyperparameter of IQL. The results support SHARP-BC’s reduced sensitivity to value-learning hyperparameters. (Appendix D.2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.