acceptodds
Withdrawn from ICLR 2027

LEAPS: Long-context Evaluation of Adaptive Pattern learning and State tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-context language models (LCLMs) power agents that act over long trajectories, from improving a codebase to running research experiments to forecasters that condition on lengthy histories. These applications draw on three distinct long-context capabilities: recall, where the answer is a fact present in the context; state tracking, where the outcome can be computed by performing a series of transformations to a specified state; and pattern learning, where the goal is to make a prediction by sampling from a distribution inferred from many observations. Existing benchmarks primarily focus on the first capability. In this work, we introduce LEAPS (Long-context Evaluation of Adaptive Pattern learning and State tracking), a collection of 12 novel datasets, spanning agentic workflows, behavioral modeling, and forecasting of market and world events. By construction, LEAPS is resistant to data contamination, extends to millions of input tokens, and uses reliable metrics. We evaluate 30 API and open models from 8K to 512K input tokens. LEAPS distinguishes models that perform near perfectly on recall tasks, finding that such models' state tracking and pattern learning abilities can vary significantly and degrade much more dramatically with increasing input lengths—even for pattern learning, where more observations should make learning a theoretically easier task. Moreover, thinking models often underperform their base counterparts, exhausting their generation budget or oversimplifying the task. We hope LEAPS can guide the development of the next generation of LCLMs.

Primary area:
datasets and benchmarks
Keywords:
Long-context, language models, benchmarks, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2ezWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LEAPS, a new benchmark for evaluating long-context LLMs beyond simple information retrieval. It focuses on two important abilities: state tracking and pattern learning, using 12 datasets covering tasks like code development, chess, and forecasting. The authors evaluate 30 models across different context lengths and find that even models performing well on traditional recall benchmarks often struggle with these tasks. They also show that performance generally drops as context length increases, and that thinking models sometimes perform worse than their base counterparts. Overall, the paper highlights important limitations of current long-context LLMs and provides a useful benchmark for studying them.

Critical strengths
  1. The paper addresses an important gap in long-context evaluation.

Most existing benchmarks focus heavily on information retrieval, while this paper looks at state tracking and pattern learning, which are also important for real-world agent tasks. I think this is a meaningful direction, since being able to retrieve information from a long context does not necessarily mean a model can effectively use that information.

  1. The benchmark covers a good variety of tasks and models.

The authors introduce 12 datasets, including both synthetic and real-world-derived tasks, and evaluate 30 models across different context lengths. I like that they compare performance on traditional recall benchmarks with the proposed tasks. The results show that models with similarly strong recall performance can behave quite differently on state tracking and pattern learning. I think this makes the benchmark useful for understanding the limitations of current long-context models.

Critical weaknesses
  1. It is not entirely clear whether some pattern learning tasks actually contain enough learnable information.

For example, on the Kaggle and Social Media forecasting tasks, many models perform close to the random guessing baseline. I'm not sure whether this is because the models struggle to learn from long contexts or because the underlying data is difficult to predict in the first place. I think comparing against stronger traditional machine learning or statistical baselines would help establish whether these tasks really measure the intended capability.

  1. The effect of context length is not fully separated from task difficulty.*

In several state tracking tasks, increasing the context length also means adding more operations that the model needs to track. This makes me wonder whether the performance drop is mainly caused by longer contexts or simply by having more steps to process. I think an additional experiment that controls the number of relevant operations while varying the context length would make the main conclusions more convincing.

  1. Whether benchmarking long context is still meaningful given the existing success of LLMs

Clearly LLMs has been better in dealing with long contexts than any time before, and there has been a lot of previous work on dealing with long context. However, the benchmark is using limited approaches in comparison (which methods are helping long context) to systematically evaluate these methods, and limited insight is provided into how we can solve the problem in the future.

Questions for authors
  1. How much of the performance drop is caused by longer contexts versus increased task difficulty?

In tasks like Cup Swapping and Coins, longer inputs also introduce more operations to track. Have you considered keeping the number of relevant operations fixed while adding irrelevant information to increase context length? This would help me better understand whether the degradation is mainly due to context length or the growing number of operations.

  1. Could the weaker performance of thinking models be partly explained by generation budget limitations?

I noticed that the paper discusses cases where thinking models run out of tokens, and Appendix A.2 mentions that some settings may exceed the models' native context windows. Although Figure 6 explores different thinking budgets, I'm still curious how much of the performance gap remains when comparing models under conditions where they can reliably complete their responses.

Reviewer_4v81OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Existing long-context benchmarks primarily focus on recall-based tasks. To evaluate broader long-context capabilities, this paper introduces LEAPS, a benchmark targeting state tracking and pattern learning, which require integrating information across long contexts rather than simply retrieving facts. LEAPS comprises 12 synthetic and real-derived datasets, and the authors evaluate 30 language models to assess performance on these capabilities.

Critical strengths
  1. Meaningful evaluation dimensions beyond recall. The paper addresses an important limitation of existing long-context benchmarks by evaluating state tracking and pattern learning, which require models to integrate and utilize information across long contexts rather than simply retrieve specific facts. These capabilities are particularly relevant to long-horizon agentic applications, making the motivation for the benchmark meaningful.

  2. Diverse benchmark construction and broad model evaluation. LEAPS covers a diverse set of 12 synthetic and real-derived datasets. The evaluation across 30 models and multiple context lengths provides a broad empirical perspective on how current language models perform on these tasks.

Critical weaknesses
  1. Insufficient validation of benchmark quality and construct validity. Although LEAPS introduces a diverse collection of tasks, the paper provides limited evidence that the benchmark reliably measures the intended long-context capabilities. In particular, the authors do not clearly describe a systematic quality-control or human-validation procedure for the evaluation datasets. This is especially concerning for pattern-learning tasks such as Chess, Kaggle, and social media market forecasting, where outcomes may be inherently stochastic and predictive performance depends on the strength of the underlying signal. The paper should clarify how the validity of the evaluation targets and the suitability of the metrics are established. Furthermore, a consolidated presentation of dataset statistics, including sample counts, representative examples, and difficulty characteristics, would help assess benchmark reliability and should be included in the main paper. Without stronger validation, it is difficult to determine to what extent low benchmark scores reflect limitations in long-context information utilization rather than task-specific difficulty or data characteristics.
  2. Insufficient analysis to substantiate the main empirical conclusions. The paper reports substantial performance degradation on state-tracking and pattern-learning tasks, as well as poorer performance of several thinking models relative to their base counterparts. However, the analysis does not sufficiently disentangle limitations in long-context utilization from other factors, such as increasing computational demands, intrinsic task difficulty, and generation-budget constraints. In particular, the qualitative analysis of thinking-model failures relies on a small number of examples, while the authors themselves observe that some models exhaust their generation budgets without producing final answers. More systematic error analyses and controlled evaluations would be needed to establish whether the reported degradation reflects the specific capabilities that LEAPS aims to measure. These limitations weaken the interpretation of the benchmark results and the broader conclusions drawn from them.
  3. The manuscript would benefit from substantial revisions to improve its clarity and logical coherence. The connections between claims and supporting evidence are often insufficiently developed, making it difficult to follow the reasoning behind some of the paper's conclusions.
Questions for authors
  1. Benchmark validation. Could the authors clarify what quality-control procedures were applied to the evaluation datasets? In particular, were any human verification or sanity checks conducted to establish the validity and difficulty of the test instances? For inherently stochastic prediction tasks, how do the authors distinguish limitations in contextual pattern learning from the intrinsic unpredictability of the target outcomes?
  2. Relationship to prior benchmarks. The distinction between LEAPS and existing long-context benchmarks that evaluate multi-evidence integration, continual information updates, and entity or state tracking is not sufficiently clear. Could the authors elaborate on the differences in the capabilities assessed and the evaluation settings, particularly relative to prior work such as:
    • Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
    • Entity Tracking in Language Models
    • Evaluating Very Long-Term Conversational Memory of LLM Agents
    • Longmemeval: Benchmarking chat assistants on long-term interactive memory
    • Storybench: A dynamic benchmark for evaluating long-term memory with multi turns
    • MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

reference

  • Kim, J., Lee, H., Zhou, D., Park, S. H., Yoon, S., Bui, T., ... & Seo, M. (2026, July). Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 42240-42272).
  • Kim, N., & Schuster, S. (2023, July). Entity tracking in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 3835-3855).
  • Maharana, A., Lee, D. H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024, August). Evaluating very long-term conversational memory of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 13851-13870).
  • Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K. W., & Yu, D. (2024). Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813.
  • Wan, L., & Ma, W. (2025). Storybench: A dynamic benchmark for evaluating long-term memory with multi turns. arXiv preprint arXiv:2506.13356.
  • Lee, H., Chen, J. C. Y., Singh, J., Khan, Z., Stengel-Eskin, E., & Bansal, M. (2026). MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems. arXiv preprint arXiv:2605.18565.
Reviewer_kHZ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that today’s long-context benchmarks are mostly retrieval evaluations, which don’t really conduct the model to reason or to learn from the context (few-shots), which is what an agent actually do in real world. The authors propose to construct a benchmark which incoperate reasoning and in-context learning. In the paper, they cast reasoning as state tracking and in-context learning as pattern learning.

There are 12 new datasets in this benchmark, half synthetic and half real data, and range up to 512k tokens and beyond. The real data include contents like following code changes across GitHub commits and market forecasting. They run about 30 models on the benchmark, including API models and open weight models.

In the experiments, the models which are perfect on the standard retrieval suites don’t generalize here. A lot of them end up near or even under baselines. State tracking gets hit worst as the context grows, falling by up to 80% between 8K and 128K tokens.

Moreover, the reasoning / thinking models do worse than common base models. The reason seems to be that they spend their whole generation budget trying to brute-force each intermediate state update and run out of budget before they get the answer.

Critical strengths

The paper addresses a gap in long-context evaluation. Benchmarks like RULER and HELMET are mostly fact retrieval, and models are already scoring close to 100% on them. LEAPS treats state tracking and pattern learning as separate capabilities that are orthogonal to retrieval.

The experiments are comprehensive. LEAPS test 30 models on 12 tasks that are hard to contaminate, up to 512k tokens. The performance of state tracking drops by 80% between 8k~128k, and the reasoning/thinking models end up behind the base models since they run out of budget. Both of those findings point to concrete failures to fix, whether on the arch side or in post-training in future work.

Critical weaknesses

The proposed benchmark motivates its tasks with autonomous software engineering, however, there are no multi-turn conversations, no tool use, and no feedback from the environment, so it can't be used as an interactive agentic benchmark reflects real world scenario.

The paper does not mention what evaluation harness it used during evaluation, which makes me wonder how many of the reported "reasoning failures" are really failures of the setup. On the other hand, scoring is exact match on the final answer. There's no check on whether the intermediate states were right.

Questions for authors
  1. Did author try other prompting setups for the thinking models, something like asking for a compact running summary instead of unrolling the whole sequence?

  2. Is there any evidence that performing well on these static tasks lines up with performing well on interactive, tool-using benchmarks that reflects real world scenario?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.