- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper study depth compatibility in looped transformers. It separate prediction accuracy and shared readout cost. Authors propose risk decomposition and low-rank adaptive heads. Experiments on synthetic linear regression show multi-depth training better with shared head. But fixed-depth model become better with few adaptive directions.
Critical strengths
Good theoretical formulation. Paper clearly separate independently decodable prediction risk and shared-head compatibility cost. Risk decomposition is clear and useful. It explain why multi-depth training improve shared-head accuracy.
Experiment design is careful. Only four adaptive directions reverse comparison in all three pairs. This is interesting finding.
Critical weaknesses
-
Limited experimental generalization. Main experiment only use small synthetic in-context linear regression, with four-dimensional input and width-32 transformer. This task are much simpler than real language modeling or reasoning. Although paper include Huginn model audit, it does not directly validate proposed adaptive readout mechanism. So not clear whether same conclusion hold for larger models and complex tasks.
-
Limited training seeds and practical evaluation. Main comparison only have three paired initialization seeds. The confidence interval mainly resample test episodes, not independent trained models. Also adaptive heads are fitted using labeled examples from all 16 evaluated depths. This require extra supervision and depth-specific coefficients. Paper not fully show whether method can work efficiently for unseen depths or practical adaptive computation. Therefore practical significance is still uncertain.
-
it is better to have a framework figure.
Questions for authors
Can authors test on larger transformer or real language tasks? Does same reversal still happen?
How sensitive is result to model width, training depth and number of seeds?
Can adaptive heads generalize to unseen depths without new labels?
can the authors provides with one first image in the paper as a framework pipeline figure?