- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SFPR learns separate action-support and conditional-transition costs and propagates them through two branches of cost critics. Relative-value estimates, critic disagreement, and cost magnitudes determine batch-shared weights, while transition support controls continuation. The Gaussian actor combines return maximization, behavior-distribution fitting, and a cost penalty; the reward critics use DOSER-style selective value regularization. Experiments cover nine D4RL locomotion datasets and three Maze2D tasks. SFPR has the highest listed average in both tables, at 94.62 and 172.74, respectively. Ablations examine the cost branches, weighting scheme, and continuation terms.
Critical strengths
The extra models are confined to training. Deployment requires only one sample from the Gaussian policy, with no support-model queries or action search. If the gains hold across runs, this would make SFPR practical to use. I also value the NoPath control: it retains behavior fitting and value regularization while removing the cost penalty from the actor objective. This helps isolate the contribution of that penalty.
Critical weaknesses
1. The contribution of the path costs remains unclear.
In Figure 2(c), the final five plotted evaluations average roughly 201 for NoPath and 222 for SFPR, while the local DOSER score in Table 2 is 105.75. The curves and table cannot provide a precise decomposition of the gain, but much of the gap to DOSER is already visible in NoPath. The published DOSER actor uses entropy regularization without behavior fitting (Wang et al., 2026), whereas NoPath retains energy-score fitting and omits entropy. I would therefore focus on the smaller SFPR-NoPath difference, with variability across runs. Removing the transition term from the actor objective does not test whether conditioning on the state-action pair matters. Replacing only the conditional scorer with a marginal successor-state scorer would address that question. Without evidence separating these effects, I am not convinced that conditional support accounts for the gains.
2. The reporting and experimental details need clarification.
Figure 3(b) presents an individual Walker2d-medium run whose last five plotted SFPR values average approximately 88.60. Table 1 reports a two-run mean of 88.60 with standard deviation 2.56. If the curve is one of those two runs and shows the unsmoothed evaluations used for Tail, these numbers are inconsistent. The run identities and plot processing are unspecified, so this is a discrepancy to resolve, not a demonstrated error. SFPR results are reported over only two runs, and Tables 3-4 give no variability estimates for the ablations. The 1.44-point locomotion advantage over DOSER also combines different reporting protocols. Network architectures and frozen-model pretraining details are missing, and Appendix F.2's reference presets are not mapped to the reported runs. My weak-reject recommendation rests on these reporting questions and the unclear attribution of the gains. Clear run-level results and a convincing comparison of the support scores could change my assessment.
Questions for authors
-
Which runs produced Figure 3(b) and the corresponding Table 1 entry? Please provide each run's final five evaluation scores, explain any plot smoothing, and state how the standard deviation was calculated.
-
Can you compare SFPR, NoPath, and local DOSER on representative tasks using the same training budget, Tail definition, and evaluation action-selection rule, with additional independent runs? Please also report variability for the existing ablations.
-
What happens if the conditional-transition score is replaced by the existing marginal-state score, with comparable normalization and the same predicted successors, cost critics, recursion, and actor objective?
-
Please provide the actual network and pretraining configurations, hyperparameter-selection procedure, and local DOSER settings used for the reported results, and explain their relationship to Appendix F.2's reference presets.
4 comments
Dear Reviewer Qpcs,
Thank you again for your thoughtful feedback. We would appreciate hearing whether our responses have addressed your concerns, and we would be happy to clarify any remaining questions.
Thank you for the additional experiments. The NoPath comparison provides clearer evidence for the contribution of the cost penalty, and the conditional-versus-marginal comparison supports the benefit of conditional scoring on Hopper-ME and Maze2D-large. Together with the additional independent runs and tuned DOSER results, these findings substantially address my main concerns. I will raise my rating to 3.
Dear Reviewer Qpcs,
We are very glad that our clarifications and additional results have resolved most of your concerns. Your constructive questions and suggestions helped us substantially strengthen our analysis and clarify our presentation. We will incorporate all of these results and clarifications into the revised manuscript, and we sincerely appreciate your time and recognition of these improvements.
Author Response to Reviewer Qpcs
Thank you for your detailed review and clear assessment criteria.
Q1: Walker2d-M Scores, Smoothing & SD
Table 1's Walker2d-M (88.60±2.56) was misrecorded (corrected 9-task mean: 94.61 vs. 94.62); Fig. 3(b) plots the unsmoothed curve of seed 3. Unsmoothed final-5 scores (1M steps/run):
Seeds 0–4 give 88.53±0.87 (seeds 2–4: 88.77±1.14), using unbiased sample SD (N=5): N−11∑i(Ti−Tˉ)2.
Q2: Unified Comparison (SFPR, NoPath, Local DOSER) and Ablation Variability
All runs below use 3 independent seeds (mean±sample SD), 1M policy updates, deterministic mean actions at evaluation, and Tail-5 (mean of evaluations at 920k–1M steps; for tuned local DOSER on Maze2D, candidates were ranked by Tail-5 and evaluated at 1M under the same deterministic protocol):
1. Backbone vs. Path-Cost Gains: On Maze2D, tuned local DOSER (λ=0.002 on U-Maze, P=99.5 on Medium, β=0.002 on Large) improves substantially over default local DOSER, and Energy-Score fitting (NoPath) further strengthens the backbone. Adding SFPR's path costs yields consistent gains over NoPath across all 12 tasks (+4.39 locomotion mean, up to +11.83 on W2d-MR; +13.24 Maze2D mean, up to +16.47 on Maze-M):
2. Tables 3 & 4 Variability (3 seeds):
Branch dominance is task-dependent; full SFPR leads on 3/4 Table 3 tasks (within 1 SD on W2d-M) and beats SAM and G-zero across Table 4.
Q3: Conditional Transition vs. Marginal State Score
Keeping the same predicted successors s^′, comparable normalization, cost critics RT, recursion κT, and actor objective, we replaced only the conditional transition score with the existing marginal state score (Marginal, 3 seeds, Tail-5):
Both perform similarly on saturated W2d-ME, whereas conditioning on (s,a) beats Marginal by +4.10 (H-ME), +4.97 (H-MR), and +11.78 (Maze-L) with lower SD. On Maze-L and H-ME, Marginal even trails Action-Only, as invalid transitions (e.g., crossing walls) can still land in high-marginal-density states.
Q4: Architectures, Pretraining, Tuning & F.2