PonderLM-3: Token-Adaptive Pondering with Differentiable Masking
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
Looped language models offer an effective way to improve model performance through iterative computation, motivating a natural follow-up question: when are additional loop iterations actually needed? To address this question, we introduce *PonderLM-3*, a pretraining framework for token-adaptive pondering, where easy tokens stop pondering early while difficult tokens receive additional pondering steps. This allows inference computation to be dynamically allocated at the per-token level. Under purely self-supervised training, PonderLM-3 learns how many pondering steps to allocate to each token through a differentiable attention masking mechanism, which also maintains train–inference consistency. At inference, the learned allocation is realized through hard pruning. The framework builds on the PonderLM-2 backbone. Compared with existing loop-based baselines, PonderLM-3 defines a stronger Pareto frontier, achieving lower pretraining perplexity at equal inference FLOPs. It also achieves the highest average performance on downstream benchmarks.
- Primary area:
- foundation or frontier models, including LLMs