- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This position paper mainly argues that the traditional opinion that SFT does not create new capabilities while RL does is incorrect. The main criterion for differentiating whether the training elicits behaviors that already exists in pre-trained models or creating new capabilities should be based on whether the learnt behaviors are reachable by the original model, regardless of which training method is used. The authors interpret this position from free-energy view.
Critical strengths
The main strength of the paper is to challenge the standard belief that RL is creating new capabilities while SFT only mimics existing behaviors with solid justifications. The argument is very clear, pointing out the importance of differentiating behavior reweighting from capability creation regardless of which training method is used. The conclusion that SFT can also create new capabilities if the data source is new and RL may only perform behavior reweighting is correct and inspiring.
Critical weaknesses
- The usage of the thermodynamic terms seems unnecessary. The main points in this paper can be described using more straightforward mathematical or probabilistic language, instead of the free-energy terms. I did not see what additional benefits or information was added by the free-energy view.
- The positions made in this paper are straightforward and easy to understand, lacking more in-depth investigations, e.g., some case studies with concrete examples on more complex practical applications.
Questions for authors
- Can you provide more introduction of the free-energy background? What is the target from the free-energy view? To minimize the free-energy?
- What is the unique benefit of interpreting everything from free-energy view? For example, instead of using the more abstract free-energy terms, the condition for elicitation can be easily described as a non-zero sampling probability of the demonstration behavior. The free-energy interpretation seems unnecessary.