acceptodds
Withdrawn from ICLR 2027

Position: Distinguish Capability Elicitation from Capability Creation in Post-Training – A Free-Energy Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Debates about large language model post-training often treat supervised fine-tuning (SFT) as imitation and reinforcement learning (RL) as discovery. But this distinction is too coarse. What matters is whether a training procedure increases the probability of behaviors the pretrained model could already produce, or whether it changes what the model can practically reach. We argue that post-training research should distinguish between capability elicitation and capability creation. We make this distinction operational by introducing the notion of accessible support: the set of behaviors that a model can practically produce under finite budgets. Post-training that reweights behaviors within this support is capability elicitation; whereas changing the support itself corresponds to capability creation. We develop this argument through a free-energy view of post-training. SFT and RL can both be seen as reweighting a pretrained reference distribution, only with different external signals. Demonstration signals define low-energy behavior for SFT, and reward signals define low-energy behavior for RL. When the update remains close to the base model, the main effect is local reweighting, not capability creation. Within this framework, the central question is no longer whether post-training is framed as SFT or RL, but whether it reweights behaviors already within reach, or instead expands the model's reachable behavioral space through search, interaction, tool use, or the incorporation of new information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training, supervised fine-tuning, reinforcement learning, free energy perspective

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9F8UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This position paper mainly argues that the traditional opinion that SFT does not create new capabilities while RL does is incorrect. The main criterion for differentiating whether the training elicits behaviors that already exists in pre-trained models or creating new capabilities should be based on whether the learnt behaviors are reachable by the original model, regardless of which training method is used. The authors interpret this position from free-energy view.

Critical strengths

The main strength of the paper is to challenge the standard belief that RL is creating new capabilities while SFT only mimics existing behaviors with solid justifications. The argument is very clear, pointing out the importance of differentiating behavior reweighting from capability creation regardless of which training method is used. The conclusion that SFT can also create new capabilities if the data source is new and RL may only perform behavior reweighting is correct and inspiring.

Critical weaknesses
  1. The usage of the thermodynamic terms seems unnecessary. The main points in this paper can be described using more straightforward mathematical or probabilistic language, instead of the free-energy terms. I did not see what additional benefits or information was added by the free-energy view.
  2. The positions made in this paper are straightforward and easy to understand, lacking more in-depth investigations, e.g., some case studies with concrete examples on more complex practical applications.
Questions for authors
  1. Can you provide more introduction of the free-energy background? What is the target from the free-energy view? To minimize the free-energy?
  2. What is the unique benefit of interpreting everything from free-energy view? For example, instead of using the more abstract free-energy terms, the condition for elicitation can be easily described as a non-zero sampling probability of the demonstration behavior. The free-energy interpretation seems unnecessary.
Reviewer_vzKmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that post-training should be analyzed through the distinction between capability elicitation and capability creation, rather than identifying SFT with imitation and RL with discovery. The authors introduce “accessible support” to describe behaviors a model can practically produce under finite computational budgets, defining elicitation as reweighting already accessible behaviors and creation as expanding this accessible set. They use a free-energy perspective to interpret demonstration fitting and KL-regularized reward optimization as transformations of a pretrained reference distribution, and propose four regimes: demonstration-covered elicitation, tail reweighting, barrier-crossing discovery, and unsupported regimes.

Critical strengths
  1. The paper addresses an important question: whether post-training elicits existing capabilities or creates new ones.
  2. It usefully separates training objectives from data sources and exploration mechanisms, encouraging more careful comparisons between SFT and RL.
  3. The emphasis on finite-budget reachability provides a practical perspective on capability, beyond strictly nonzero output probabilities.
  4. The paper clearly states its position and discusses alternative views, helping readers understand the intended scope of its framework.
Critical weaknesses
  1. The boundary between elicitation and creation needs further clarification. Under fixed computational budgets, probability reweighting can also change practical reachability, making it unclear how the framework would classify such improvements.
  2. The additional diagnostic value of the free-energy formulation remains unclear. The paper does not sufficiently explain how this formulation helps determine whether a particular improvement reflects elicitation or creation.
  3. The distinction between tail reweighting and barrier-crossing discovery needs stronger justification. Low-probability intermediate steps can also make complete trajectories rare, so clearer criteria are needed to distinguish these regimes.
  4. A worked case study would strengthen the argument. Although new experiments are not essential for a position paper, a concrete application would help demonstrate how the framework improves the interpretation of actual post-training results.
Questions for authors
  1. How does the framework distinguish elicitation from creation when probability reweighting alone makes a behavior accessible under a fixed computational budget
  2. Could you provide a concrete example showing how the free-energy perspective helps distinguish tail reweighting from barrier-crossing discovery
Reviewer_EJYcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Questionable submission fit. This paper is primarily a conceptual position piece without novel methods, empirical results, or substantial theoretical contributions. While the topic is relevant to foundation models, its contribution seems insufficient for the ICLR main conference, and it may be better suited to a position-paper workshop.

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.