- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper keeps the offline data and the goals fixed, and changes only what a multi-step backup does at a cut that is not a real ending. One version still uses the value of the next state. The other treats the cut as the end. Treating the cut as the end lowers success. A learner that backs up only one step does not read the cuts.
Critical strengths
The contribution is a comparison that changes only whether a nonterminal cut keeps the next state's value. Bootstrapping keeps more of the task success. Treating the cut as an ending loses it. The same cuts do not affect a one-step learner.
Critical weaknesses
Gymnasium, OGBench masks, and D4RL timeouts already separate a cut from true termination. Official IQL bootstraps with masks. One-step methods that follow these interfaces are unaffected. The drop appears only when a multi-step backup treats the cut as an ending.
I would raise the score if the authors explain what is new and how many current offline RL methods are actually affected.
Questions for authors
What should other papers take from this work beyond the known bootstrap rule?
Which current methods, besides the two tested here, treat a nonterminal cut as an ending? Are one-step OGBench or D4RL implementations affected?