- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PDPO, which uses second-order information to accelerate DPO and introduces random walks to estimate the update direction. It provides convergence analysis and experiments on synthetic problems and language-model fine-tuning.
Critical strengths
- The connection between DPO, comparison graphs, and random walks is interesting. It gives a way to understand and compute the proposed update.
- The synthetic experiments are carefully designed.
Critical weaknesses
-
The convergence theory does not cover the algorithm used for language-model training. The theory updates a separate score for each response, while the actual algorithm updates the model parameters. These updates are different, and the paper does not show that the same fast convergence holds for the actual algorithm.
-
The theory assumes a finite optimum, but the consistent reward-based preferences in the LLM experiments allow the score-space loss to keep decreasing as score gaps grow. The authors should clarify how the convergence guarantee applies to this setting
Questions for authors
- Can the authors explain whether the convergence guarantees apply to the actual model-parameter updates?
- The paper reports accuracy gains, but it is unclear whether random walks make training faster. The paper does not report the sampling cost or training time compared with DPO.
- Appendix F.2 reports preference pairs drawn from responses generated by the initial policy, while Algorithm 4 requires on-policy sampling. It is unclear whether the reported online experiment follows the proposed algorithm.
There is no rebuttal, and I keep my score.