acceptodds
Under review as a conference paper at ICLR 2027

Selective Factorized Risk Regularization for Offline Reinforcement Learning

Abstract

Offline reinforcement learning must balance policy improvement against unreliable extrapolation beyond the available data. Local action support and marginal successor-state support provide useful evidence, but do not establish whether a specific state–action–successor relation is supported or whether subsequent policy decisions will remain supported. Moreover, insufficient data support does not necessarily imply low return, motivating the use of value information when determining how support evidence should constrain learning. We propose Selective Factorized Path Risk (SFPR), a framework that regularizes policy improvement through factorized, value-guided future support costs. SFPR separately models action support conditioned on the current state and transition support conditioned on the state–action pair. Two branch-specific cost estimators recursively connect these local assessments to subsequent decisions. Data-support and behavior-relative value assessments regulate batch-level cost strengths, while support-dependent continuation reduces recursive propagation at weakly supported transitions. SFPR integrates these future costs with behavior-distribution fitting and selective value regularization to constrain both policy updates and the value estimates guiding them. SFPR achieves the highest average performance among the compared methods on both the D4RL locomotion and Maze2D navigation benchmarks. These results demonstrate SFPR's overall performance advantages across continuous-control and navigation tasks.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, distribution shift, out-of-distribution actions, conditional transition modeling, long-horizon risk, policy regularization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_QpcsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SFPR learns separate action-support and conditional-transition costs and propagates them through two branches of cost critics. Relative-value estimates, critic disagreement, and cost magnitudes determine batch-shared weights, while transition support controls continuation. The Gaussian actor combines return maximization, behavior-distribution fitting, and a cost penalty; the reward critics use DOSER-style selective value regularization. Experiments cover nine D4RL locomotion datasets and three Maze2D tasks. SFPR has the highest listed average in both tables, at 94.62 and 172.74, respectively. Ablations examine the cost branches, weighting scheme, and continuation terms.

Critical strengths

The extra models are confined to training. Deployment requires only one sample from the Gaussian policy, with no support-model queries or action search. If the gains hold across runs, this would make SFPR practical to use. I also value the NoPath control: it retains behavior fitting and value regularization while removing the cost penalty from the actor objective. This helps isolate the contribution of that penalty.

Critical weaknesses

1. The contribution of the path costs remains unclear.

In Figure 2(c), the final five plotted evaluations average roughly 201 for NoPath and 222 for SFPR, while the local DOSER score in Table 2 is 105.75. The curves and table cannot provide a precise decomposition of the gain, but much of the gap to DOSER is already visible in NoPath. The published DOSER actor uses entropy regularization without behavior fitting (Wang et al., 2026), whereas NoPath retains energy-score fitting and omits entropy. I would therefore focus on the smaller SFPR-NoPath difference, with variability across runs. Removing the transition term from the actor objective does not test whether conditioning on the state-action pair matters. Replacing only the conditional scorer with a marginal successor-state scorer would address that question. Without evidence separating these effects, I am not convinced that conditional support accounts for the gains.

2. The reporting and experimental details need clarification.

Figure 3(b) presents an individual Walker2d-medium run whose last five plotted SFPR values average approximately 88.60. Table 1 reports a two-run mean of 88.60 with standard deviation 2.56. If the curve is one of those two runs and shows the unsmoothed evaluations used for Tail, these numbers are inconsistent. The run identities and plot processing are unspecified, so this is a discrepancy to resolve, not a demonstrated error. SFPR results are reported over only two runs, and Tables 3-4 give no variability estimates for the ablations. The 1.44-point locomotion advantage over DOSER also combines different reporting protocols. Network architectures and frozen-model pretraining details are missing, and Appendix F.2's reference presets are not mapped to the reported runs. My weak-reject recommendation rests on these reporting questions and the unclear attribution of the gains. Clear run-level results and a convincing comparison of the support scores could change my assessment.

Questions for authors
  1. Which runs produced Figure 3(b) and the corresponding Table 1 entry? Please provide each run's final five evaluation scores, explain any plot smoothing, and state how the standard deviation was calculated.

  2. Can you compare SFPR, NoPath, and local DOSER on representative tasks using the same training budget, Tail definition, and evaluation action-selection rule, with additional independent runs? Please also report variability for the existing ablations.

  3. What happens if the conditional-transition score is replaced by the existing marginal-state score, with comparable normalization and the same predicted successors, cost critics, recursion, and actor objective?

  4. Please provide the actual network and pretraining configurations, hyperparameter-selection procedure, and local DOSER settings used for the reported results, and explain their relationship to Appendix F.2's reference presets.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Qpcs

Thank you for your detailed review and clear assessment criteria.

Q1: Walker2d-M Scores, Smoothing & SD

Table 1's Walker2d-M (88.60±2.56) was misrecorded (corrected 9-task mean: 94.61 vs. 94.62); Fig. 3(b) plots the unsmoothed curve of seed 3. Unsmoothed final-5 scores (1M steps/run):

Seed920k940k960k980k1MTail-5 (TiT_i)
088.17688.97888.21387.80987.73688.182
188.57988.02187.37688.09288.76788.167
289.57289.92089.29289.82089.42989.607
389.26289.24489.05389.25989.30589.225
487.77687.80687.00687.24987.53187.474

Seeds 0–4 give 88.53±0.87 (seeds 2–4: 88.77±1.14), using unbiased sample SD (N=5N=5): 1N−1∑i(Ti−Tˉ)2\sqrt{\frac{1}{N-1}\sum_i(T_i-\bar{T})^2}.

Q2: Unified Comparison (SFPR, NoPath, Local DOSER) and Ablation Variability

All runs below use 3 independent seeds (mean±sample SD), 1M policy updates, deterministic mean actions at evaluation, and Tail-5 (mean of evaluations at 920k–1M steps; for tuned local DOSER on Maze2D, candidates were ranked by Tail-5 and evaluated at 1M under the same deterministic protocol):

1. Backbone vs. Path-Cost Gains: On Maze2D, tuned local DOSER (λ=0.002\lambda=0.002 on U-Maze, P=99.5P=99.5 on Medium, β=0.002\beta=0.002 on Large) improves substantially over default local DOSER, and Energy-Score fitting (NoPath) further strengthens the backbone. Adding SFPR's path costs yields consistent gains over NoPath across all 12 tasks (+4.39 locomotion mean, up to +11.83 on W2d-MR; +13.24 Maze2D mean, up to +16.47 on Maze-M):

NavigationTuned Local DOSERNoPath (3-seed)SFPR (3-seed)
Maze-U130.14±3.07133.94±4.68142.11±2.28
Maze-M122.08±5.79136.21±3.97152.68±4.40
Maze-L190.04±3.45205.76±5.02220.84±2.80
Mean147.42158.64171.88
LocomotionNoPath (3-seed)SFPR (3-seed)
HC-M67.37±0.4568.79±0.84
H-M98.16±1.30104.56±1.32
W2d-M86.50±0.7888.77±1.14
HC-MR60.74±1.5664.95±1.17
H-MR97.32±1.96105.32±1.17
W2d-MR84.73±1.0496.56±0.85
HC-ME94.69±1.7598.00±1.12
H-ME110.69±0.40111.45±0.97
W2d-ME110.45±0.23111.74±0.14
Mean90.0794.46

2. Tables 3 & 4 Variability (3 seeds):

Table 3Action-OnlyTransition-OnlySFPR (3-seed)
Maze-L212.67±3.80202.50±1.82220.84±2.80
H-ME110.70±1.17107.99±1.42111.45±0.97
HC-M65.38±0.6567.69±0.0768.79±0.84
W2d-M88.94±1.7688.32±1.7588.77±1.14
Table 4SAMG-zeroSFPR (3-seed)
H-ME103.18±3.04101.00±2.69111.45±0.97
W2d-ME110.54±0.21110.80±0.59111.74±0.14
Maze-L214.30±3.81211.91±2.46220.84±2.80

Branch dominance is task-dependent; full SFPR leads on 3/4 Table 3 tasks (within 1 SD on W2d-M) and beats SAM and G-zero across Table 4.

Q3: Conditional Transition vs. Marginal State Score

Keeping the same predicted successors s^′\hat{s}', comparable normalization, cost critics RTR_T, recursion κT\kappa_T, and actor objective, we replaced only the conditional transition score with the existing marginal state score (Marginal, 3 seeds, Tail-5):

DatasetNoPathAction-OnlyMarginalSFPR (3-seed)
H-ME110.69±0.40110.70±1.17107.35±5.32111.45±0.97
H-MR97.32±1.96—100.35±2.32105.32±1.17
W2d-ME110.45±0.23—111.66±0.09111.74±0.14
Maze-L205.76±5.02212.67±3.80209.06±2.99220.84±2.80

Both perform similarly on saturated W2d-ME, whereas conditioning on (s,a)(s,a) beats Marginal by +4.10 (H-ME), +4.97 (H-MR), and +11.78 (Maze-L) with lower SD. On Maze-L and H-ME, Marginal even trails Action-Only, as invalid transitions (e.g., crossing walls) can still land in high-marginal-density states.

Q4: Architectures, Pretraining, Tuning & F.2

  • ReLU MLPs: Actor has 3x256 layers (mean/log-SD, tanh). Four Q(s,a) + two V(s) heads, twin (s,a) cost heads per branch, and two separate predictors (s,a)->(s',r) each use 2x256 layers.
  • Denoisers & Updates: Three 4x256 Mish MLPs (16-D time embedding) model action given s, Δs given (s,a), and marginal s. Auxiliary models freeze; actor, Q/V, and cost critics update online (soft targets); test uses actor + norm stats.
  • Pretraining & Norm: Offline task data with saved state-norm stats (mapped to scorer space) and fixed 50th/95th-percentile score norms. H-ME/Maze-L predictor logs: 100k steps, batch 256, lr 3e-4 (other budgets unverified).
  • Presets & DOSER: Runs match Tables 5–7 (Tables 5–6 fixed across 12 tasks via H-MR/Maze-L sweeps; F.2 = fixed a priori configs, not per-seed tuning). Local DOSER used 9 locomotion presets, default Maze2D (β=λ=0.001, η=expectile=0.9, P=99), and a 1M-step search (β,λ,η,expectile,P).
AuthorsOpenReview ↗

Dear Reviewer Qpcs,

Thank you again for your thoughtful feedback. We would appreciate hearing whether our responses have addressed your concerns, and we would be happy to clarify any remaining questions.

Thank you for the additional experiments. The NoPath comparison provides clearer evidence for the contribution of the cost penalty, and the conditional-versus-marginal comparison supports the benefit of conditional scoring on Hopper-ME and Maze2D-large. Together with the additional independent runs and tuned DOSER results, these findings substantially address my main concerns. I will raise my rating to 3.

AuthorsOpenReview ↗

Dear Reviewer Qpcs,

We are very glad that our clarifications and additional results have resolved most of your concerns. Your constructive questions and suggestions helped us substantially strengthen our analysis and clarify our presentation. We will incorporate all of these results and clarifications into the revised manuscript, and we sincerely appreciate your time and recognition of these improvements.

Reviewer_xPSaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an offline RL algorithm that builds on a DOSER-style approach to OOD regularisation with a formulation that account for longer-term support costs throughout an episode. The authors provide an evaluation on locomotion and antmaze D4RL tasks, beating the reported baseline methods. Ablation results also provide evidence for the benefit of including various components into algorithm's architecture.

Critical strengths

The empirical evaluation provides evidence for the value of the proposed method. It displays strong empirical results, particularly on the antmaze baseline. Moreover, the ablation studies investigate and confirm that there is benefit in including the various components engineered into the method (branch-specific costs, support-cost and it's recursive estimation) and provide some evidence toward the necessity of the elaborate engineering scaffolding behind the algorithm.

The core idea of separating support into action and transition support is simple but principled, and the idea of considering the effect that later transitions have on the support cost of a policy is also principled and well-motivated conceptually.

Critical weaknesses

While the empirical results are strong, I have some concerns regarding the evaluation protocol. The strong performance on antmaze compared to DOSER references a local DOSER evaluation. It is unclear what degree of tuning, if any, went into the DOSER evaluation on this task as the original paper does not include it. A fair comparison would require a comparable tuning effort.

The method proposed by the authors involves a large number of components that require tuning and numerous specific hyperparameters relative to other standard offline RL implementations. In addition to the computational burden these introduce, which is not addressed, it is unclear how these hyperparameters were chosen or tuned, and whether there is a possibility that the strong results are in part due to noise from running a large number of hyperparameters configurations. This issue is further magnified by the fact that the authors present tail results over only two seeds, where the return correlations across tail samples could be significantly higher than those across a larger number of independent runs, making the reported uncertainties less reliable.

Questions for authors

Clarification requests:

  • Why were only two seeds chosen for each empirical task evaluation?
  • What is the computational cost of the proposed method compared to DOSER and simpler canonical offline RL baselines (eg TD3+BC)?
  • How were the hyperparameters chosen?
  • Why is SPFR's relative performance compared to DOSER much stronger in the antmaze task than in the locomotion tasks?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer xPSa

Thank you for your constructive comments. Below we address your concerns regarding seed count, computational cost, hyperparameter tuning, and navigation baseline fairness.

Q1: Seed Count and Empirical Robustness

We agree that robust evaluation is essential. We expanded our evaluation with three independent seeds (seeds 2–4) for SFPR across all nine locomotion and three Maze2D tasks. Tail-5 scores (mean ± sample SD):

DatasetSFPR (3-seed Tail)DatasetSFPR (3-seed Tail)
HC-M68.79 ± 0.84HC-ME98.00 ± 1.12
H-M104.56 ± 1.32H-ME111.45 ± 0.97
W2d-M88.77 ± 1.14W2d-ME111.74 ± 0.14
HC-MR64.95 ± 1.17Maze-U (U-Maze)142.11 ± 2.28
H-MR105.32 ± 1.17Maze-M (Medium)152.68 ± 4.40
W2d-MR96.56 ± 0.85Maze-L (Large)220.84 ± 2.80

These 3-seed results (locomotion mean 94.46, Maze2D mean 171.88; 5-seed W2d-M: 88.53 ± 0.87) closely match our Table 1–2 averages (94.61 after correcting a W2d-M table typo; 172.74 on Maze2D), confirming cross-seed stability.

Q2: Computational Cost Comparison

We evaluated runtime, peak memory, and latency on the same RTX 3090 (averaging 3 repetitions):

DatasetMethod1,000 policy updates (s)Peak Torch allocated (MiB)Action latency (ms, B=1)
H-MESFPR219.74 ± 15.6365.320.726 ± 0.038
H-MEDOSER145.13 ± 5.8858.820.727 ± 0.072
H-METD3+BC10.66 ± 0.3922.310.415 ± 0.033
Maze-LSFPR212.86 ± 12.6865.410.774 ± 0.037
Maze-LDOSER140.87 ± 6.6659.750.638 ± 0.023
Maze-LTD3+BC9.52 ± 0.4322.170.402 ± 0.017

Timing excludes setup, evaluation, saving, and pretraining. SFPR takes ~1.51× DOSER's update time and adds ~6 MiB in peak tensor allocation, while maintaining identical single-Gaussian deployment speed.

Q3: Hyperparameter Selection and Baseline Tuning

DOSER tuning: We performed a uniform one-factor search on U-Maze, Medium, and Large exploring β,λ∈\beta, \lambda \in {0.0005, 0.001, 0.002}, η∈\eta \in {0.7, 0.9, 1.0}, expectile ∈\in {0.7, 0.9, 0.95}, and P∈P \in {95, 99, 99.5} (defaults: β=λ=0.001,η=expectile=0.9,P=99\beta=\lambda=0.001, \eta=\text{expectile}=0.9, P=99; ranked by Tail at 920k–1M updates):

Navigation taskSelected candidateTuned candidate DOSER
U-Mazeλ=0.002\lambda = 0.002130.14 ± 3.07
MediumP=99.5P = 99.5122.08 ± 5.79
Largeβ=0.002\beta = 0.002190.04 ± 3.45

SFPR parameter selection & tuning range: We retain DOSER reference settings for Q-regularization, expectile regression, OOD detection, EDM diffusion, and base architectures, while adding Energy-Score fitting. To determine SFPR-specific parameters (Table 6), we conducted single-factor sensitivity sweeps on H-MR and Maze-L over wide ranges: gating threshold u0∈u_0 \in {0.15, 0.25, 0.35} (default 0.25), support threshold z0∈z_0 \in {0.1, 0.2, 0.3} (default 0.2), continuation upper bound γT,max⁡∈\gamma_{T,\max} \in {0.125, 0.25, 0.5} (default 0.25), support penalty weight βR∈\beta_R \in {0.3, 1.0, 3.0} (default 1.0), and scaling factor c0∈c_0 \in {0.25, 0.5, 0.75} (default 0.5). These parameters were fixed across all 12 tasks without per-task tuning. Sensitivity sweeps confirm high stability:

SettingH-MRMaze-LSettingH-MRMaze-L
u0=0.15u_0 = 0.15101.09 ± 1.22216.65 ± 1.88βR=0.3\beta_R = 0.397.09 ± 1.70202.47 ± 2.81
u0=0.35u_0 = 0.35100.47 ± 1.45214.65 ± 2.94βR=3\beta_R = 398.47 ± 1.97204.02 ± 1.13
z0=0.1z_0 = 0.1102.15 ± 2.82221.95 ± 1.19c0=0.25c_0 = 0.2592.99 ± 4.84209.95 ± 1.53
z0=0.3z_0 = 0.3104.40 ± 1.97220.06 ± 0.56c0=0.75c_0 = 0.75101.56 ± 2.33211.06 ± 2.41
γT,max⁡=0.125\gamma_{T,\max} = 0.125103.64 ± 1.78217.60 ± 4.21
γT,max⁡=0.5\gamma_{T,\max} = 0.5101.67 ± 2.05214.80 ± 4.53

Q4: Navigation Gains & Baseline Fairness

Comparing SFPR against our newly tuned DOSER baselines (Q3: U-Maze 130.14, Medium 122.08, Large 190.04) and 3-seed NoPath ablations (Maze2D NoPath: U-Maze 133.94 ± 4.68, Medium 136.21 ± 3.97, Large 205.76 ± 5.02; locomotion below):

DatasetNoPath TailSFPR (3-seed)DatasetNoPath TailSFPR (3-seed)
HC-M67.37 ± 0.4568.79 ± 0.84W2d-MR84.73 ± 1.0496.56 ± 0.85
H-M98.16 ± 1.30104.56 ± 1.32HC-ME94.69 ± 1.7598.00 ± 1.12
W2d-M86.50 ± 0.7888.77 ± 1.14H-ME110.69 ± 0.40111.45 ± 0.97
HC-MR60.74 ± 1.5664.95 ± 1.17W2d-ME110.45 ± 0.23111.74 ± 0.14
H-MR97.32 ± 1.96105.32 ± 1.17Mean90.0794.46

SFPR consistently outperforms both tuned DOSER (by +11.97 to +30.80 on Maze2D) and NoPath across all 12 tasks, confirming gains stem from our core mechanisms (p(s′∣s,a)p(s'|s,a), recursive future costs, Energy-Score fitting) rather than unoptimized baselines.

Thank you for your response. The response addressed my main concerns regarding statistical robustness. I appreciate the clarification on computation cost and hyperparameter tuning. Overall, the degree of hyperparameter tuning required seems quite significant and the increase in computational overhead (excluding pretraining) is, as expected, also substantial. My overall score review remains unchanged but I now feel slightly more confident in recommending acceptance.

Reviewer_mCJmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Factorized Path Risk (SFPR), an offline RL method that regularizes policy improvement with value-guided, multi-step support costs. The authors argue that marginal successor-state support, as used in DOSER, does not establish whether a specific (s, a, s') relation is supported, and that single-step support does not ensure that subsequent decisions remain supported. SFPR factorizes data support into action support p_D(a|s) and conditional transition support p_D(s'|s,a), each scored by a frozen conditional denoising model. Two recursive cost critics propagate these local costs to future decisions. Their influence on the actor is modulated by batch-shared coefficients derived from behavior-relative value and critic disagreement, and transition-cost propagation is attenuated at poorly supported transitions through a support-dependent continuation factor. The method is combined with energy-score behavior fitting and DOSER's selective value regularization, and is evaluated on D4RL locomotion and Maze2D.

Critical strengths
  1. The distinction between conditional transition support and marginal successor support is well motivated. A successor state can be common in the data while the transition leading to it is not, so modeling p_D(s'|s,a) is a reasonable refinement over prior consequence-aware methods.

  2. The support-dependent continuation factor is an intuitive way to handle compounding errors in multi-step cost propagation: it keeps the immediate cost but reduces reliance on bootstrapped estimates where the predicted consequence is least trustworthy.

  3. All auxiliary models are used only during training. Deployment requires a single Gaussian policy, with no diffusion sampling, candidate search, or planning at test time.

  4. Results on Maze2D are strong, especially on Maze2D-large, where SFPR outperforms both value-based methods and diffusion planners. The paper is also careful in stating what its proxy scores and theoretical results do and do not guarantee.

Critical weaknesses
  1. Incomplete ablations on the core claims and component contributions. First, while the primary motivation is that conditional transition support p_D(s'|s,a) is more informative than DOSER's marginal successor-state support p_D(s'), no ablation directly replaces p_D(s'|s,a) with p_D(s') under the same backbone. Second, SFPR inherits its behavior fitting and value regularization from DOSER, and the average gain over DOSER across the nine locomotion datasets is modest (+1.44 points). Figure 2 shows that NoPath (which removes both proposed branch-cost penalties) performs comparably on Walker2d-ME, yet Tail scores for NoPath across all nine locomotion datasets are not reported.

  2. High architectural and hyperparameter complexity. The pipeline requires five pretrained frozen models (action and increment denoisers, a marginal state scorer, and two separate dynamics predictors) plus an actor, four reward critics, two value heads, and four cost critics during online training, alongside reference-action sampling and bisection root-finding at each update. It also introduces numerous gating and continuation hyperparameters (Table 6) on top of the task-specific presets inherited from DOSER (Table 7). Despite this overhead, neither wall-clock training cost nor hyperparameter sensitivity analysis is provided.

  3. Conceptual tension in batch-shared weighting. Collapsing sample-specific relative-value and disagreement assessments into a single scalar coefficient shared across the entire minibatch weakens the "selective" premise of the method: a well-supported, high-advantage action can still receive a transition penalty driven by other samples in the batch (as noted in Appendix B.7). Moreover, the samplewise baseline (SAM) is not precisely defined in the main text, nor is there a clear analysis of why per-sample gating underperforms. (Minor note: Section 5.3 also contains several grammatical errors that should be fixed.)

Questions for authors
  1. How does SFPR perform when the conditional transition score is replaced by DOSER's marginal successor-state score under the same backbone, and what are the NoPath Tail scores across all nine locomotion datasets?

  2. How does the support-dependent continuation factor compare with a fixed non-zero continuation discount?

  3. What are the wall-clock training time and GPU memory costs compared with DOSER (including pretraining), and why are two separate dynamics predictors (f_psi and f_G) needed instead of a shared model?

  4. How is the SAM variant formulated, why does samplewise weighting degrade performance, and how sensitive is SFPR to the gating and continuation hyperparameters in Table 6?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer mCJm

Thank you for recognizing our motivation, controlled continuation, deployment efficiency, and scope statements. New controls use our 1M-step Tail protocol (mean±SD).

Q1 & W1: Marginal vs. Conditional Support and Locomotion NoPath

1. Marginal vs. Conditional Scoring: Under the matched SFPR backbone, we replaced cT(s,a,s^′)c_T(s,a,\hat{s}') with normalized marginal state scoring (Marginal):

DatasetMarginalSFPR (Paper)
Hopper-ME107.35±5.32112.72±0.72
Walker2d-ME111.66±0.09111.21±0.23
Maze2D-large209.06±2.99221.72±1.41

Effects are task-dependent: on saturated Walker2d-ME (NoPath: 110.45), Marginal and SFPR are similar (111.74±0.14 on 3-seed SFPR below); on Maze2D-large and Hopper-ME, conditional scoring adds +12.66 and +5.37 with lower SD. On Maze2D-large, Marginal (209.06) trails Action-Only (211.72, Table 3), as marginal density cannot verify (s,a,s′)(s,a,s') validity.

2. Locomotion NoPath & 3-Seed SFPR: Alongside Table 1 (correcting a W2d-M table typo to 5-seed 88.53±0.87 gives 94.61 mean), we report 3-seed NoPath and 3-seed SFPR:

MethodHC-MH-MW2d-MHC-MRH-MRW2d-MRHC-MEH-MEW2d-MEMean
NoPath (3-seed)67.37±0.4598.16±1.3086.50±0.7860.74±1.5697.32±1.9684.73±1.0494.69±1.75110.69±0.40110.45±0.2390.07
SFPR (3-seed)68.79±0.84104.56±1.3288.77±1.1464.95±1.17105.32±1.1796.56±0.8598.00±1.12111.45±0.97111.74±0.1494.46
SFPR (Table 1*)69.03±0.60104.36±1.2388.53±0.8764.32±1.25105.57±0.8497.56±0.3598.20±2.02112.72±0.72111.21±0.2394.61

SFPR (3-seed) matches Table 1 (94.46 vs. 94.61 mean), confirming stability, and both beat NoPath (90.07) on all 9 tasks (+4.39/+4.54 mean; W2d-MR +11.83/+12.83, H-MR +8.00/+8.25, H-M +6.40/+6.20).

Q2: Support-Dependent Continuation vs. Fixed Non-Zero Discounts

With other mechanisms unchanged, we tested 3-seed controls setting κT∈\kappa_T \in {0.125, 0.25}:

DatasetG-zero (3-seed)Fixed-0.125Fixed-0.25SFPR (3-seed)
Hopper-ME101.00±2.69108.53±1.05106.40±2.02111.45±0.97
Walker2d-ME110.80±0.59110.02±0.09110.56±0.68111.74±0.14
Maze2D-large211.91±2.46202.44±3.54205.31±4.00220.84±2.80

Fixed discounts trail SFPR on Hopper-ME and Maze2D-large, and fall below G-zero on Maze2D-large (214.34 in Table 4): uniform bootstrapping across weakly supported transitions compounds errors, while adaptive κT\kappa_T curbs unreliable propagation.

Q3 & W2: Computational Cost and Dual Frozen Predictors

1. Runtime, Memory & Pretraining: On one RTX 3090 (3 reps), SFPR takes ~1.51× DOSER's update time (H-ME: 20.62× vs. DOSER's 13.62× over TD3+BC) and deploys only the actor:

DatasetSFPR / DOSER: 1k updates (s)Peak Torch alloc. (MiB)Action latency (ms, B=1)
H-ME219.74±15.63 / 145.13±5.8865.32 / 58.820.726±0.038 / 0.727±0.072
Maze-L212.86±12.68 / 140.87±6.6665.41 / 59.750.774±0.037 / 0.638±0.023

Timing excludes setup, evaluation, saving, and pretraining; Torch alloc. is peak PyTorch tensor memory, not process GPU memory. Reused DOSER modules were not re-timed; historical logs for extra predictor fψf_\psi (100k steps) took 19.09 min (1,145.27 s, H-ME) and 6.05 min (363.19 s, Maze-L).

2. Dual Predictors (fψ,fGf_\psi, f_G): We keep fGf_G for DOSER's selective-regularization interface, while fψf_\psi averages 5 samples in scorer coordinates (Appendix A.3). A 3-seed Shared-G control matches Full on Maze2D-large (220.92±2.04 vs. 220.84/221.72), but on Hopper-MR (90.62±2.70 vs. 105.32/105.57) some seeds suffer late degradation and higher SD; for stability, we keep dual predictors.

Q4 & W3: Selectivity Scope, SAM, Sensitivity, and Presentation

1. Selectivity & SAM: Batch sharing grants no per-sample exemptions; samplewise support/value signals regulate aggregate minibatch constraint strength (per-sample penalties/gradients vary via RA,RTR_A, R_T). SAM uses samplewise weights in action-cost targets and actor transition terms, which is overly sensitive to local Arel,UQA_{\text{rel}}, U_Q errors and mixed data quality. We will clarify this scope.

2. Table 6 Sensitivity: Table 6 is shared across all 12 tasks (u0=0.25,z0=0.2,γT,max⁡=0.25u_0=0.25, z_0=0.2, \gamma_{T,\max}=0.25). One-factor sweeps stay well above NoPath (97.32 / 205.76):

Settingu0=0.15u_0=0.15u0=0.35u_0=0.35z0=0.1z_0=0.1z0=0.3z_0=0.3γT,max⁡=0.125\gamma_{T,\max}=0.125γT,max⁡=0.5\gamma_{T,\max}=0.5Default
H-MR101.09±1.22100.47±1.45102.15±2.82104.40±1.97103.64±1.78101.67±2.05105.57±0.84
Maze-L216.65±1.88214.65±2.94221.95±1.19220.06±0.56217.60±4.21214.80±4.53221.72±1.41

3. Writing: We will fix Section 5.3's grammar, define SAM and G-zero in the main text, and limit ablation claims to reported settings.

AuthorsOpenReview ↗

Dear Reviewer mCJm,

Thank you again for your thoughtful feedback. We would appreciate hearing whether our responses have addressed your concerns, and we would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Dear Reviewer mCJm,

Thank you again for your careful review and constructive feedback. We have carefully considered your comments and addressed them with additional experiments and clarifications in our rebuttal.

As the discussion period draws to a close, we would greatly appreciate your further feedback on whether our responses have adequately resolved your main concerns. If any questions remain, we would be happy to provide further clarification immediately.

We sincerely appreciate your valuable time, effort, and support!

Best regards, The Authors

Reviewer_t1veOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an offline RL method that addresses unreliable extrapolation in OOD actions, by regularizing policy improvement through value-guided future support costs. The value-guided future support costs are integrated with the behavior distribution to constrain policy updates and the value estimates. Their method adopts selective value-regularization from DOSER. The authors evaluate their method on Maze2D and D4RL tasks, showing better than performance than published baseline methods.

Critical strengths
  1. Proposed method shows better performance than baseline methods in Maze2D task.
  2. Well thought through idea of having separate action and transition support .
Critical weaknesses
  1. Insufficient random seeds in evaluation, may not account for training and evaluation variances.
  2. Table 1 compares SFPR's tail score which averages the last 5 evaluations over 10 episodes each at 1M updates, directly against published baseline numbers that may use different evaluation protocols from the authors tail score.
  3. The branch ablation was applied to only Maze2D large, having more experiments with other datasets will give better representation to establish if having both action and transition support is more generally beneficial.
Questions for authors
  1. Can the authors share updated benchmark results(Table 1 and 2) for more seeds?
  2. Can the authors provide more results for the branch ablation on other datasets, or justify its use on only Maze2D large?
  3. Can the authors kindly clarify/justify the evaluation of their method against the published baselines where they may have different result reporting than their tail method?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer t1ve

Q1: Multi-Seed Benchmark Results (Tables 1 & 2)

We evaluated SFPR across three independent training seeds (mean ± sample SD) under the paper's 1M-update Tail-5 protocol across all nine locomotion datasets and three Maze2D tasks:

Table 1 Task3-Seed TailTable 1 Task3-Seed Tail
HC-M68.79 ± 0.84W2d-MR96.56 ± 0.85
H-M104.56 ± 1.32HC-ME98.00 ± 1.12
W2d-M88.77 ± 1.14H-ME111.45 ± 0.97
HC-MR64.95 ± 1.17W2d-ME111.74 ± 0.14
H-MR105.32 ± 1.17Locomotion Mean94.46
Table 2 TaskMaze-UMaze-MMaze-LMaze2D Mean
3-Seed Tail142.11 ± 2.28152.68 ± 4.40220.84 ± 2.80171.88

These 3-seed means (94.46 on locomotion, 171.88 on Maze2D; 5-seed W2d-M: 88.53 ± 0.87) closely match our published Table 1–2 averages (94.61 after correcting a W2d-M table typo; 172.74 on Maze2D) with consistently small standard deviations, confirming stability across random seeds.

Q2: Multi-Dataset Branch Ablation (Action vs. Transition Support)

We expanded the branch ablation beyond Maze2D-large to three locomotion datasets (H-ME, HC-M, and W2d-M) across three independent training seeds (1M updates, Tail-5, mean ± sample SD), comparing NoPath, Action-Only, Transition-Only, and full SFPR:

DatasetNoPathAction-OnlyTransition-OnlySFPR (Full)
Maze-L205.76 ± 5.02212.67 ± 3.80202.50 ± 1.82220.84 ± 2.80
H-ME110.69 ± 0.40110.70 ± 1.17107.99 ± 1.42111.45 ± 0.97
HC-M67.37 ± 0.4565.38 ± 0.6567.69 ± 0.0768.79 ± 0.84
W2d-M86.50 ± 0.7888.94 ± 1.7688.32 ± 1.7588.77 ± 1.14

Single-branch dominance is task-dependent: Action-Only leads Transition-Only on Maze-L and H-ME, whereas Transition-Only beats Action-Only on HC-M (where Action-Only alone drops below NoPath). Combining both branches in SFPR improves over NoPath on all four tasks, achieves the highest score on 3/4 datasets (+8.17 over the stronger single branch on Maze-L and +1.10 on HC-M), and matches Action-Only within 0.17 points (well within 1 SD, with lower variance) on W2d-M.

Q3: Clarification and Justification of the Tail Evaluation Protocol

Rationale for Tail-5: In Table 1, we cite author-reported 1M-update baseline numbers following standard D4RL practice to avoid under-tuning re-implementations. For SFPR, Tail averages the final five checkpoints from 920k to 1M updates (10 episodes each, 50 episodes total) with no best-checkpoint selection. Because policies have converged in this late-training window, Tail-5 summarizes the same 1M-step performance regime as a single Final evaluation while reducing single-checkpoint rollout fluctuations (e.g., across 5 seeds on W2d-M, the single 1M checkpoint is 88.55 ± 0.88 vs. Tail-5 at 88.53 ± 0.87). We will explicitly distinguish quoted baseline protocols from our Tail-5 evaluation in Table 1's caption.

Matched-Protocol Verification: Under the exact same 3-seed, 1M-update, deterministic-action Tail-5 protocol, SFPR consistently outperforms its matched backbone (NoPath) across all 9 locomotion datasets (94.46 vs. 90.07 mean; e.g., H-M: 104.56 vs. 98.16, H-MR: 105.32 vs. 97.32, W2d-MR: 96.56 vs. 84.73) and all 3 Maze2D tasks (171.88 vs. 158.64; tuned local DOSER ranked by Tail-5 and evaluated at 1M reaches 147.42), confirming that our gains are not artifacts of the evaluation protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.