- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the learnability of reasoning by superposition, where a Transformer represents and iteratively updates a set of candidate states in a single hidden state. The main contributions are (i) sufficient architectural conditions for gradient training to recover the full target trajectory, (ii) sufficient supervision conditions ensuring that gradients correct the earliest inaccurate intermediate state, and (iii) a budget-optimal allocation rule for intermediate-state labels based on marginal error reduction per unit cost. The theory is validated on controlled graph-reachability experiments and a GPT-2-scale experiment comparing several supervision strategies.
Critical strengths
- A coherent theory and experiment connection.
- Practical supervision allocation result.
Critical weaknesses
- Strong assumptions limit the scope of the theoretical claims. The main learnability guarantees rely on substantial architectural and optimization assumptions, including the correction frame, attention margins, smoothness, and gradient-transfer conditions. It is not clear how frequently these assumptions hold in realistic models.
- Limited empirical scope. The paper primarily concerns synthetic reachability/superposition tasks. Although the GPT-2 experiment provides a useful additional validation, the results do not yet establish that the proposed principles extend to different reasoning problems or modern LLMs.
Questions for authors
- Which of the main theoretical conditions are actually satisfied by the models used in the GPT-2 experiment?
- How sensitive is the cost-aware supervision advantage to the assumed error-response model?
- Do the same supervision allocation principles hold on reasoning tasks that are not naturally formulated as cumulative set expansion?