- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RDR, which scores the policy action and its perturbations by predicted first-arrival probabilities at several deadlines, normalized by the best candidate, and fits the policy to their weighted average. It reports the best average success over manipulation and maze tasks, with the largest gains on a few maze layouts.
Critical strengths
Comparing actions through arrival probabilities at several deadlines, learned without bootstrapping, is an intuitive way to credit detours around obstacles. The maze results are strong, and the ablations openly show which components matter.
Critical weaknesses
- Novelty is limited. The arrival predictor comes from SVL, the candidate weighting follows MPO and the gradient term is DDPG-style, while the relative-regret normalization amounts to a per-deadline softmax temperature. The ablations show that the adaptive deadline weights and the gradient term add little. MPO: A. Abdolmaleki et al. Maximum a Posteriori Policy Optimisation. ICLR, 2018. DDPG: T. P. Lillicrap et al. Continuous Control with Deep Reinforcement Learning. ICLR, 2016.
- The theory does not support the method. The return bound is vacuous for every discount factor, because a step-shaped deadline utility cannot approximate geometric discounting at states that cannot reach the goal immediately, and the convergence result simply assumes the contraction it needs.
- The evidence is weaker than the text suggests. Much of the aggregate gain comes from a few maze layouts, RDR trails the strongest baseline on most manipulation tasks, including an obstacle task, and the candidate-fitting ablation compares against a degenerate control. The authors also report that executing the fitting target directly does not improve arrival.
- Key details and controls are missing. Seeds, interaction budgets and most hyperparameters are not given in the main text, the claim that Bellman errors are the problem is never tested with a TD-based critic in the same pipeline, and DWSL is not compared.
Questions for authors
- How many seeds were run, how many environment steps make one epoch, and do the baselines receive the same interaction budget, implementations and tuning effort? Please give per-task means with standard errors and a significance test for the small margins, and state whether Push-Hard, PickPlace-Hard and CustomLarge come from prior work.
- What happens if F̂ is replaced by a TD-learned Q-function or distance inside exactly the same candidate fitting and normalization pipeline? Could you also add DWSL and an AWR- or MPO-style weighting baseline? AWR: X. B. Peng et al. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv:1910.00177, 2019.
- Please describe the direct target substitution experiment (L484-485) and its numbers. If executing ā does not raise arrival rates, where does the gain come from? A calibration plot of F̂ against empirical completion under the current policy, split into obstructed and clear states, would help.
- Do you agree that Δ_γ ≥ 1 - γ + γ^(max I) whenever H = 1 is not admitted, which makes Eq. (7) trivial? Could you instead bound suboptimality for the deadline objective Σ_H ω_H F_H, and check whether L_g < r holds for the hyperparameters actually used?
- Please tune λ_g for RDR_NoReg before comparing, explain why the completion gradient is kept given its effect on PickPlace-Hard, and report RDR_Return in the main text together with a sweep over γ.