- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LEAPS, a new benchmark for evaluating long-context LLMs beyond simple information retrieval. It focuses on two important abilities: state tracking and pattern learning, using 12 datasets covering tasks like code development, chess, and forecasting. The authors evaluate 30 models across different context lengths and find that even models performing well on traditional recall benchmarks often struggle with these tasks. They also show that performance generally drops as context length increases, and that thinking models sometimes perform worse than their base counterparts. Overall, the paper highlights important limitations of current long-context LLMs and provides a useful benchmark for studying them.
Critical strengths
- The paper addresses an important gap in long-context evaluation.
Most existing benchmarks focus heavily on information retrieval, while this paper looks at state tracking and pattern learning, which are also important for real-world agent tasks. I think this is a meaningful direction, since being able to retrieve information from a long context does not necessarily mean a model can effectively use that information.
- The benchmark covers a good variety of tasks and models.
The authors introduce 12 datasets, including both synthetic and real-world-derived tasks, and evaluate 30 models across different context lengths. I like that they compare performance on traditional recall benchmarks with the proposed tasks. The results show that models with similarly strong recall performance can behave quite differently on state tracking and pattern learning. I think this makes the benchmark useful for understanding the limitations of current long-context models.
Critical weaknesses
- It is not entirely clear whether some pattern learning tasks actually contain enough learnable information.
For example, on the Kaggle and Social Media forecasting tasks, many models perform close to the random guessing baseline. I'm not sure whether this is because the models struggle to learn from long contexts or because the underlying data is difficult to predict in the first place. I think comparing against stronger traditional machine learning or statistical baselines would help establish whether these tasks really measure the intended capability.
- The effect of context length is not fully separated from task difficulty.*
In several state tracking tasks, increasing the context length also means adding more operations that the model needs to track. This makes me wonder whether the performance drop is mainly caused by longer contexts or simply by having more steps to process. I think an additional experiment that controls the number of relevant operations while varying the context length would make the main conclusions more convincing.
- Whether benchmarking long context is still meaningful given the existing success of LLMs
Clearly LLMs has been better in dealing with long contexts than any time before, and there has been a lot of previous work on dealing with long context. However, the benchmark is using limited approaches in comparison (which methods are helping long context) to systematically evaluate these methods, and limited insight is provided into how we can solve the problem in the future.
Questions for authors
- How much of the performance drop is caused by longer contexts versus increased task difficulty?
In tasks like Cup Swapping and Coins, longer inputs also introduce more operations to track. Have you considered keeping the number of relevant operations fixed while adding irrelevant information to increase context length? This would help me better understand whether the degradation is mainly due to context length or the growing number of operations.
- Could the weaker performance of thinking models be partly explained by generation budget limitations?
I noticed that the paper discusses cases where thinking models run out of tokens, and Appendix A.2 mentions that some settings may exceed the models' native context windows. Although Figure 6 explores different thinking budgets, I'm still curious how much of the performance gap remains when comparing models under conditions where they can reliably complete their responses.