- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PACT, which improves critic training through Actor-then-Critic updates, importance-corrected targets, and BCE. The authors derive token-level credit under three regularity conditions and report performance gains on mathematical reasoning tasks and SWE-bench Verified.
Critical strengths
- PACT outperforms the compared baselines across four mathematical benchmarks, demonstrating significant average improvements.
- PACT without Importance Sampling (IS), while maintaining other training settings, demonstrates the independent contribution of critic-side IS to mathematical reasoning.
- The comparison between BCE and MSE utilizes identical initialization and rollout data, effectively controlling the experimental conditions for critic pre-training.
Critical weaknesses
- Appendix B.7 only proves that current-token IS corresponds to a hybrid policy; the experiments do not directly verify whether it reduces the value estimation error relative to the updated policy.
- In Appendix D.4, only PACT employs actor-side TIS; the absence of baselines using the same correction makes it difficult to attribute the full performance gains observed in the main results solely to critic training.
- Figure 1 does not explicitly utilize held-out rollouts and lacks an ablation study replacing BCE during full RL training; thus, the evidence is insufficient to confirm BCE's contribution to final performance.
- The proof in Appendix B.6 does not account for the ratio masking actually employed, and the conditional target mean after masking does not necessarily fall within the [0, 1] range; the conclusion regarding critic consistency cannot be directly applied to the actual algorithm.
Questions for authors
NaN