- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies policy improvement from Q-value estimates obtained by rolling out each action under a fixed rollout policy, and argues that the improvement target should depend on the old policy, since a target built from Q-values alone is identical for any two old policies with the same Q-values. It uses the standard KL-regularized target and fits it with a residual parameterization, so the network learns only a log-ratio correction to the old policy instead of a new policy from scratch. In small-scale experiments the residual parameterization moves expected Q in the right direction more reliably than fitting a softmax policy from scratch.
Critical strengths
- Separating target construction from target fitting, and varying only the reference distribution while holding everything else fixed, makes it clear that the bottleneck is fitting, not the target.
Critical weaknesses
- Hard to read. Many coined terms replace standard RL vocabulary, obscuring the actual idea. And many places of clear ai generated content. (example: "Primary comparisons share the corrected V2 protocol" in Experiment section)
- The paper introduces no new method, as the authors acknowledge (“Neither the operator nor the function family is new,” in Introduction section), so its value rests on showing that conditioning the target on the old policy matters in practice. The evidence does not establish this. Proposition 1 holds by construction: a target computed from Q alone cannot depend on the old policy, but this does not show that it should. Empirically, the residual linear policy realizes an expected-Q gain of only +0.0053 against a target gain of +0.0638.
Questions for authors
None