- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies offline reinforcement learning with trajectory-level supervision instead of per-step rewards. It proposes OPAC, which jointly learns a reward model and a pessimistic critic before updating the policy. For additive returns, it establishes an upper bound under single-policy state-action coverage and a lower bound matching the explicit horizon and coverage dependence, up to function-class complexity factors. It also extends the analysis to preferences and studies nonlinear returns through an exponential hardness example and two conditioning coefficients.
Critical strengths
-
The paper clarifies trajectory-level labels can support offline policy optimization under single-policy state-action coverage.
-
The nonlinear analysis highlights that supervision granularity interacts strongly with the return objective.
Critical weaknesses
-
Theorem 9 combines reward-class closure, uniform Bellman completeness, and a product critic class. These requirements can conflict even in a two-step multiplicative setting: if the maximum first-stage critic value is () and continuation values include (), closure requires a reward (), while exact completeness requires representing (). This raises concerns about the scope of the generalized guarantee.
-
The contributions is concentrated on the theory analysis. The novelty of the proposed algorithm is limited in my opinion. Also, I think the performance of this algorithm may be not very good due to the joint optimization.
Questions for authors
-
Could the authors also make the function-class complexity dependence explicit when comparing the additive upper and lower bounds?
-
Could the authors work through one concrete scenario to explain how the formal assumptions apply and what actionable insight the theoretical results provide?
Dear Reviewer mMTB,
As the rebuttal period nears its end, we wanted to share a short update.
Following our discussion with Reviewer aJum, we added neural-network experiments in Appendix B.2.
All methods train on the same 2,000 complete, 1,000-step trajectories from simulated OGBench PointMaze-Medium, with only one outcome label per trajectory. We report cumulative 1,000-step returns: each seed's score averages five goal tasks and three fixed checkpoints, and the table gives the mean and sample SD across two training seeds. Higher is better.
OPAC has the highest aggregate mean among the tested methods, including the two-stage baseline using the same actor–critic. These results support joint learning with neural networks in this setting. Appendix B.2 describes the methods and exploratory simulator tuning.
For the concerns in your review:
Please let us know if any concerns remain. We hope you will reconsider your assessment in light of these revisions and results.
Thank you for your time and consideration.
Best regards,
The Authors