- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In looped language models, achieving fixed points in recurrence can improve efficiency in training and decoding by allowing the terminal end-point to be a replacement for the full trajectory. Thus it can give a number of benefits such as faster decoding by shrinking the KV cache, faster prefill in distillation and faster rollout scoring in RL. This paper proposes two techniques to shape fixed points: learning a depth prior and orthogonal input injection. The proposed approach for learning a depth prior improved upon a fixed-depth scheme and a fixed prior scheme in terms of validation perplexity by 1.1-1.9%. It speeds up scoring and backpropagation in RL by 1.5x with a slight degradation in quality. A distilled 2-block student based on the learned prior model is 1.5-1.7% faster than the teacher in terms of prefill latency while keeping most of its quality though there is quite a bit of variation across evaluation tasks. For input injection, the current baseline (Parcae's injection) uses a method whereby the portion of the input that the fixed point retains depends on the state. The paper proposes a method to remove this component so that each recurrence is passed the same input. This yields improvements by around 0.7-1.6% in terms of perplexity.
Critical strengths
- Proposes 2 well-motivated approaches for improving the efficiency of looped language models. Learning prior improves upon fixed-depth and fixed-prior in terms of perplexity/downstream accuracy, faster RL and prefill latency in distillation. Orthogonal injection improves upon Parcae's injection in terms of perplexity, though the gains are small.
- Trains models at 3 scaled: 100M, 400M and 1.6B parameters
Critical weaknesses
- The paper could benefit from additional analysis: 1) Why Fixed depth works as well as the proposed scheme for OBQA on M and L scales? 2) Why Huginn-Linear and Parcae-Decay works better than orthogonal injection for ARC-C and OBQA respectively?
- The improvements in perplexity are rather small: 1.1-1.9% for learning depth prior and 1.6% for better input injection.
Questions for authors
- L182: Shouldn't h_r be a function of t? i.e. h_r^t?
- L183: Clarify this phrase: "prefix-dependent input to its update".
3 comments
Thanks for providing the additional analyses.
Thank you for reading our response and for supporting the paper.
Response to Reviewer r4xo
Thank you for the positive assessment and the precise questions.
W1. Gaps on OBQA and ARC-C.
By BPB, our method is better in three of the four cases; in the fourth, the interval contains zero, so we count it as a tie.
W2. Size of the perplexity gains.
The gains are consistent: the learned prior lowers Val PPL relative to Huginn’s fixed prior, and OrthoInj lowers it relative to Parcae-Decay at every scale in the paper. Both changes are nearly free: the learned prior adds at most 1.6% training FLOPs, and neither adds inference cost.
The PPL gain carries over to downstream tasks. At 1.6B, with both models under terminal KV sharing (Table 2), the learned prior is at or above Huginn's fixed prior on all ten tasks:
Averaged over the seven tasks, BPB shows the same pattern:
Every interval excludes zero except the learned prior at 100M, a tie, and its gain grows with scale. Under terminal sharing, fixed-depth training drops from 59.90 to 56.90 on the 7-task average and from 50.64 to 21.23 on GSM8K; the learned prior reaches 60.00, matching the full-cache average of fixed-depth training.
Q1. L182: shouldn't hr be a function of t, i.e., hrt?
Yes. Lemma 1 fixes the current token t and drops the index; the revision writes htr in Lemma 1 and Section 3.2.
Q2. L183: clarify "prefix-dependent input to its update".
C<tr is everything that token t's update at recurrence r reads from earlier positions: the keys and values of tokens before t in each physical attention layer at that recurrence, plus any other recurrent state they carry. The revision states this definition explicitly.