acceptodds
Under review as a conference paper at ICLR 2027

StateTracking: Benchmarking Complete Current-State Reconstruction from Evolving Histories

Abstract

Language models increasingly serve as long-running assistants in settings such as AI-assisted coding and workflow management, where users expect them to maintain an accurate account of the current state despite long histories of edits, deletions, and updates. Existing benchmarks study long-term memory, symbolic state changes, and file reconstruction from revision histories, but often rely on query-specific evaluation, synthetic data, or real-world histories without explicit factor control, leaving them unable to jointly test evolving text collection recovery and complete final values for all active items (called keys). This task probes whether models can selectively update information associated with each key while preserving valid content and excluding information invalidated by later changes. To fill this evaluation gap, we introduce StateTracking, a benchmark for complete current-state reconstruction from an initial state and an ordered history. It contains 3,420 histories adapted from five datasets spanning document editing, online discussions, software issue workflows, financial orders, and network records, with workload difficulty characterized by three factors: final active-key count , transition count , and deletion frequency . Three datasets–Wikipedia Article, Wikipedia Talk, and Jir–are grounded in real revision or workflow histories. The other two—LOBSTER and DNS—provide controlled workloads with broader coverage of these factors, enabling systematic evaluation across conditions and scalability analysis as state size and history length increase. Completed evaluations reveal a gap between recovering the final keys and their complete values. Error analysis also identifies deleted keys and superseded text in model outputs, alongside distinct text-fidelity and formatting failures.

Primary area:
datasets and benchmarks
Keywords:
State Tracking, Large Language Model Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_iYfyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StateTracking evaluates whether models can reconstruct all surviving keys and their complete final values from an initial state and an ordered update history. Its 3,420 histories cover Wikipedia Article, Wikipedia Talk, Jira, synthetic orders and DNS records, organized by final state size, transition count and deletion frequency. The evaluation excludes external retrieval, persistent memory and executable tools. It measures exact key recovery and complete state exact match, with analyses of obsolete content, deletion failures, text fidelity and output errors.

Critical strengths
  1. Requiring both the final collection’s membership and every retained item’s complete value makes updating, preservation and deletion observable in one answer. This extends designated file reconstruction, such as LoCoDiff, by requiring the model to identify which items remain active. The distinction between key-set exact match and complete state exact match also exposes cases where the model finds the right items but reconstructs their contents incorrectly.

  2. The dataset provides five domains and twenty histories per joint workload bin. Reference states are checked through replay, and LOBSTER and DNS use matched operation and key access sequences. Wikipedia additionally provides paired natural language and formal representations. These features support comparisons across workload conditions and offer a basis for testing how input representation affects performance.

Critical weaknesses
  1. The connection to practical assistant workflows is motivated but not demonstrated. The introduction discusses coding, collaborative editing and workflow management, yet Section 3.1 excludes the persistent state and executable tools that could handle explicitly specified updates. Section 4.2 clearly limits the benchmark to reconstruction from a fixed history, so the existence of a replay solution does not invalidate its model-level objective. However, the experiments do not establish whether performance under this restriction predicts reliable state management in an assistant that uses external tools. The practical interpretation of the scores therefore remains uncertain.

  2. The evaluation does not sufficiently separate instruction interpretation, state reconstruction and exact reproduction. The authors already reduce ambiguity through templates, identifiable keys and explicit operation semantics; there is no basis to assume the instructions are inherently ambiguous. Nevertheless, Section 3.1 evaluates Wikipedia using natural language descriptions, while Section 2.3 and Appendix A.2 describe paired formal inputs without reporting a same history performance comparison. It is therefore unclear how much the input representation contributes to the observed failures. Strict scoring adds another distinction: Section 3.4 attributes 299 of Kimi-K3’s 362 Wikipedia Talk answers with correct keys but incorrect values to whitespace differences. Deleted keys and obsolete passages demonstrate state errors, but the overall EM gap does not establish their prevalence relative to interpretation and copying failures.

Questions for authors
  1. Clarification: Beyond measuring toolfree reconstruction, what use do you intend for the benchmark scores in assessing practical assistants? Given the replayable update histories, which capabilities would still need to be measured when persistent state and executable tools are available, and what evidence supports that interpretation?

  2. Clarification: Were the paired natural language and formal Wikipedia inputs evaluated on the same histories? If so, how do key-set and complete-state exact match differ between representations, and how much of the remaining value-error gap disappears after whitespace normalization? If these comparisons were not performed, please clarify the limits this places on attributing failures specifically to state tracking.

3 comments
AuthorsOpenReview ↗

Response to Reviewer iYfy (Part 1 of 2)

We thank the reviewer for recognizing the complete-state reconstruction objective, reference validation, and workload coverage. We have completed the requested same-history natural/formal comparisons, including whitespace-normalized scoring, and clarified StateTracking's contribution as a controlled model-level evaluation of updating, preservation, and invalidation.

W1: model-level evaluation and practical assistants

We agree that the predictive value of these scores for tool-enabled assistant reliability remains to be established, and view StateTracking's current contribution as a controlled evaluation of model-level state reconstruction. Our tool-free protocol assesses the model's own ability to reconstruct an evolving state from its history. It provides model-level comparisons of updating, preservation, and exclusion under fixed update semantics and verifiable references.

A central feature of StateTracking is the joint requirement to identify the full active collection and recover every item's complete final value, without being given the final key set. Local updates must be integrated while unchanged content is preserved and deleted or superseded content is excluded. The common complete-state objective and N,T,ρN,T,\rho workload bins make these requirements observable across different workloads.

The same histories and references could be reused with persistent state and tools, allowing comparisons of how external state management changes these errors and how state correctness relates to downstream task success.

W2: input representation, state reconstruction, and exact reproduction

We have added Qwen's same-history natural/formal comparison (Appendix A.7, Table 10), whitespace-normalized EM (Table 9), and history-grounded error counts (Appendix A.5, Table 7).

Exact reproduction and history-grounded errors.

Complete-state EM combines collection correctness and exact value reproduction. The per-model, per-domain analysis distinguishes text fidelity, membership, obsolete content, updating, and preservation. Among 634 analyzed Qwen DNS answers, 76 returned at least one deleted key and 54 returned obsolete content; among 585 analyzed GLM LOBSTER answers, the corresponding counts were 189 and 70. Content error counts cover readable, normally completed, schema-valid answers. Generation and output-format failures are reported separately in Table 6. These history-grounded counts provide direct evidence of membership and value-update errors.

The input-representation and normalization results are detailed in Q2 below.

Q1: intended use and capabilities with persistent state and tools

For assistants with persistent state and tools, we would assess correct update interpretation, item selection, consistency of stored state, and use of current state in subsequent tasks. Given canonical operations, an external executor can apply the specified state changes. For example, a successfully executed call could target the wrong issue, or a later answer could use an obsolete value despite a correct stored state.

Section 3.4 and Appendix A.5 document missed updates, returned deleted keys, and loss of unchanged content, providing concrete outcomes to check in such workflows. These histories and references would allow us to record tool calls and maintained states, and compare those states with replay references. Reporting complete-state EM alongside success on tasks that depend on the resulting state could connect reconstruction to practical outcomes and help locate errors in update interpretation, tool coordination, or subsequent state use.

AuthorsOpenReview ↗

Response to Reviewer iYfy (Part 2 of 2)

Q2: same-history representations and whitespace normalization

Same-history input representations.

We have added Qwen's same-history natural/formal evaluations on 540 Article and 540 Talk histories, with identical reference states for each pair (Appendix A.7, Table 10).

DatasetInput representationHistoriesComplete-state EM (%)Whitespace-normalized EM (%)Key-set EM (%)
Wikipedia ArticleNatural language54057.2260.9393.15
Wikipedia ArticleFormal54065.5668.5297.59
Wikipedia TalkNatural language54010.5644.2683.70
Wikipedia TalkFormal54011.4852.7882.22

For Qwen, formal input improves whitespace-normalized complete-state EM from 60.93% to 68.52% on Article and from 44.26% to 52.78% on Talk. These paired results show that input representation affects complete-state recovery in both datasets.

Whitespace sensitivity of the current natural-input results.

Whitespace normalization substantially reduces the gap between key-set and complete-state EM on Talk for all four models evaluated on the full benchmark (Appendix A.7, Table 9). Whitespace normalization removes all Unicode whitespace from values. It retains the original keys, answer schema, and evaluation denominators.

Wikipedia Talk, natural inputs (%). The four models evaluated on the full benchmark retain all 540 histories; GPT has 108 sampled histories.

ModelHistoriesComplete-state EMWhitespace-normalized EMKey-set EM
Qwen54010.5644.2683.70
DeepSeek54018.1571.6795.93
GLM54011.6749.8177.78
Kimi-K354021.4876.8588.52
GPT-6-Astra10896.3096.3099.07

For the Kimi Talk result cited by the reviewer, whitespace normalization reduces the gap between key-set and complete-state EM from 67.04 to 11.67 percentage points. Of the 362 correct-key, inexact-value Kimi Talk answers, 299 become exact after this whitespace normalization. Table 9 also reports Article, where whitespace normalization improves EM by 2.59–6.67 points for the four models evaluated on the full benchmark. The normalization results quantify text-fidelity sensitivity across models and domains, while the history-grounded counts above identify concrete membership and value-update errors.

Thanks. I will keep the positive score.

Reviewer_VURcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StateTracking, a benchmark for reconstructing the complete current state from an initial state and an ordered history of changes. Models must identify all remaining active keys and recover their complete final values, including unchanged content. The benchmark contains 3,420 histories across Wikipedia Article, Wikipedia Talk, Jira, and synthetic LOBSTER and DNS workloads, organized by final active-key count, transition count, and deletion frequency.The authors evaluate four models on the full benchmark and GPT-6-Astra on stratified subsets using complete-state and key-set exact match. Results reveal a substantial gap between recovering the correct keys and reconstructing their complete values. Error analysis identifies retained deleted content, superseded values, omitted unchanged passages, and text-fidelity and output-format failures.

Critical strengths
  1. A clearly defined and practically relevant task. StateTracking requires models to identify all surviving keys and reconstruct their complete final values, jointly testing updating, preservation, and invalidation. This captures requirements of long-running assistants that query-specific memory evaluations may overlook.

  2. Diverse datasets with systematic workload coverage. The benchmark includes 3,420 histories spanning document editing, discussions, issue workflows, and structured records. Organizing these histories by final-state size, transition count, and deletion frequency enables comparisons across workload conditions, combining realistic histories with controlled synthetic settings.

  3. Informative evaluation supported by reference validation. Reporting both key-set and complete-state exact match exposes failures hidden by membership accuracy alone. Replay-based reference checks strengthen confidence in the dataset, while traced error examples distinguish missed updates, returned deleted content, and lost unchanged passages.

Critical weaknesses

1.The key–value accuracy gap conflates state-tracking and text-fidelity errors. Whitespace differences account for 299 of Kimi-K3’s 362 Wikipedia Talk answers with correct keys but incorrect values . Normalized metrics and substantive-error counts are needed to clarify how often models fail to update or invalidate content.

2.The workload factors are not sufficiently isolated. Deletion frequency also changes text length or operation mix, while larger states mechanically reduce joint exact match. Length-matched comparisons and per-key accuracy would strengthen conclusions about the individual difficulty factors.

Questions for authors

1.Workload factors are not sufficiently isolated. Changes in state size, history length, and deletion frequency also affect text length or operation mix, limiting attribution of performance differences.

2.State errors are conflated with text-fidelity errors. Many incorrect answers differ in whitespace, so the key–value accuracy gap does not clearly establish failures of updating or invalidation.

3.Can you separately report context-limit, generation-truncation, and formatting failure rates, together with EM on a common feasible subset? How much of the history-length degradation remains after excluding resource-related failures?

4.Are errors driven by update recency rather than history length alone? Can you report stale-value and omission rates against the distance from each key’s last update, while controlling for history and output length?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VURc (Part 1 of 2)

We thank the reviewer for recognizing the complete-state reconstruction task, systematic workload coverage, and reference validation. We have added normalization, per-key metrics, length-matched Article comparisons, last-update analyses, and resource-failure breakdowns, providing direct evidence of reconstruction errors and a clearer interpretation of workload trends across models and datasets.

W1: interpreting complete-state EM

We have added whitespace-normalized complete-state EM for all five models (Appendix A.7, Table 9), together with strict value recovery and history-grounded error counts. Across the four models evaluated on the full benchmark, whitespace-normalized Talk EM is 44.26–76.85%, while GPT's Talk EM remains 96.30% on its sample. These results quantify text-fidelity sensitivity. The history-grounded error analysis separately identifies returned deleted keys, obsolete content, and missed latest updates, supporting our findings on updating and invalidation. We provide the detailed results in Q2 below.

W2: workload coordinates and joint correctness

We have added length-matched comparisons on Wikipedia Article (Appendix A.10, Table 15) and strict value recovery for all five models and datasets (Appendix A.7, Table 11). Deletion frequency is coupled with addition/update composition. In controlled LOBSTER/DNS, deletion–insertion pairs maintain final active-key count while replacing overwrites (Section 2.3.3 and Appendix A.3). Increasing NN also increases the number of key–value pairs that must be correct simultaneously. Measuring this accumulated risk is central to complete-state reliability, even when per-key recovery remains high. We summarize the results in Q1 below.

Q1: length, operation mix, and per-key comparisons

We have added length-matched Article comparisons (Appendix A.10, Table 15) and strict value recovery alongside complete-state EM for all five models and datasets (Appendix A.7, Table 11). For example, on Qwen Talk, comparing the lowest and highest NN settings leaves strict value recovery nearly unchanged (64.77% to 63.72%), while complete-state EM falls from 23.89% to 1.11%. Recovering a similar fraction of individual values therefore does not ensure that the complete final state is correct.

To examine the contribution of text length, we compared two groups of 44 Article histories with medium versus high deletion frequency, drawn from the same state-size and history-length groups and with similar input and reference-answer lengths. The mean lengths differ by only 0.28% and 1.32%, respectively. EM decreases by 18.18, 15.91, and 13.64 percentage points for Qwen, DeepSeek, and GLM, while increasing by 15.91 points for Kimi. The same directions remain in a larger comparison. Thus model-dependent differences remain when text lengths are similar, supporting our conclusion that more frequent deletion does not uniformly make reconstruction harder. Appendix A.10, Table 15 provides the selection details.

Deletion frequency remains coupled with insertion and overwrite composition. For low versus high TT, our check found too few histories with comparable lengths to estimate a length-independent effect. We therefore interpret the reported trends as reliability across history workloads, with the completion analysis in Q3 explaining part of the variation.

Q2: normalized metrics and substantive errors

We have added whitespace-normalized complete-state EM (Appendix A.7, Table 9) and history-grounded content error analysis (Appendix A.5, Table 7) for all five models. Article EM increases by only 2.59–6.67 percentage points for the four models evaluated on the full benchmark. Their normalized Talk EM remains 44.26–76.85%, while GPT's Wiki scores are unchanged (Appendix A.7, Table 9).

The following counts summarize substantive errors across Wikipedia Article and Talk. Each count is the number of analyzed answers containing the specified error. Analyzed answers are readable, normally completed, and schema-valid. Generation, formatting, and request failures are reported separately in Table 6. Complete-state EM is computed over all evaluated histories, including these failures.

ModelAnalyzed answersMissing active keysReturned deleted keysReturned obsolete content
Qwen1,01760916
DeepSeek1,07028125
GLM93120115
Kimi1,03724132
GPT214001

These traced membership and obsolete-content errors show that the evaluation captures failures of updating and invalidation in addition to text fidelity. Appendix A.5, Table 7 reports the complete per-model, per-dataset results across all five datasets.

AuthorsOpenReview ↗

Response to Reviewer VURc (Part 2 of 2)

Q3: resource failures, feasible subsets, and the remaining T trend

We have added failure-type counts (Appendix A.5, Table 6) and low/high-TT comparisons within valid answers (Appendix A.8, Table 12). The four models evaluated on the full benchmark still show lower EM at high TT in Article, LOBSTER, DNS, and Jira. Section 3.1 also specifies the inference settings, 1M total context window, and 128k generation budget including reasoning. Appendix A.5, Table 6 separates generation-length termination, JSON parsing failures, schema failures, and request failures.

Appendix A.8, Table 12 reports the lowest/highest TT outcomes for all five models and datasets. Below is complete-state EM within normally completed, schema-valid answers. Wikipedia uses whitespace-normalized comparison, and the record datasets use the original field comparison.

Complete-state EM within valid answers: lowest → highest TT (%).

DatasetQwenDeepSeekGLMKimiGPT
Wikipedia Article76.47 → 55.9094.41 → 75.7266.07 → 49.6993.85 → 90.1794.44 → 94.29
Wikipedia Talk50.57 → 48.2184.44 → 60.8960.38 → 62.3190.96 → 74.0397.22 → 100.00
LOBSTER98.33 → 13.7999.00 → 28.7781.61 → 15.4897.64 → 71.76100.00 → 95.40
DNS87.29 → 14.9587.63 → 17.3978.45 → 3.1290.62 → 50.0095.56 → 94.12
Jira74.44 → 22.7888.33 → 35.0381.44 → 33.3395.51 → 73.60100.00 → 98.84

Recorded completion and formatting failures therefore do not explain the full observed contrasts in these domains. Talk varies by model, with increases for GLM and GPT. GPT's valid-answer EM remains at least 94.12% in every high-TT setting. Table 12 separately provides all-history EM and failure rates.

These analyses of model-specific valid-answer subsets are descriptive. They do not establish a common feasible subset selected independently of model outputs, or a length-independent TT effect.

Q4: last-update recency

We have expanded the last-update recency analysis to all five models and datasets (Appendix A.10, Table 16). We compare keys updated within the final 0–4 events with keys followed by at least 17 events, within the same history. This holds N/T/ρN/T/\rho and complete-input and reference-output lengths fixed. For each history, we compute the missing-key and old-complete-value fractions in each group, then average across histories containing both groups.

Missing active keys: recent → older updates (%).

DatasetQwenDeepSeekGLMKimiGPT
Wikipedia Article0.00 → 0.000.53 → 2.230.00 → 0.000.00 → 0.050.00 → 0.00
Wikipedia Talk2.75 → 3.132.63 → 1.372.31 → 1.331.25 → 1.040.00 → 0.00
LOBSTER8.29 → 9.463.77 → 4.321.73 → 4.651.58 → 2.020.00 → 0.01
DNS5.29 → 7.781.80 → 3.000.71 → 2.482.44 → 3.940.41 → 0.41
Jira0.78 → 0.973.25 → 3.930.63 → 0.550.08 → 0.340.00 → 0.00

For all four models evaluated on the full benchmark, older-key omission rates are higher in LOBSTER and DNS. Talk has no uniform increase across models. Returned old complete-value rates also increase with age for all four models evaluated on the full benchmark across the three record datasets, but remain below 1% in every model/dataset/group. Table 16 reports both error types and the histories used. Exact old-complete-value matches cover part of obsolete content, with partial obsolete content analyzed separately in Table 7. These within-history comparisons show that recency patterns depend on the model and dataset and do not support a uniform recency-only explanation of the history-length trends.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Response

Thank you again for your time and thoughtful review. With the rebuttal period nearing its end, we would be grateful if you could take a look at our response when your schedule permits. We would be happy to clarify any remaining questions.

Reviewer_3vEYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors tackle the problem of long context evaluation with evolving histories in language model workflow. They note how current benchmark do not enable evaluating rigorously each component of this problem and propose to close this gap by introducing a new benchmark called StateTracking. This benchmark contains 3420 histories that cover 5 datasets on software issue workflows, financial orders, document editing, online discussions, and network records. This benchmark does in particular offer an initial state which corresponds to a set of keys and an ordered history of changes which makes the evaluation more rigorously accessible. The authors define workload difficulty using final active-key count, transition count, and deletion frequency. The analysis using this benchamark identifies that revealing final keys can still conceal missing complete values and that there are several failure modes when it comes to history management.

Critical strengths
  • The motivation for this benchmark is very well explained and makes this a well needed effort and work to share with the community.
  • The benchmark achieves its intended purpose of enabling thorough evaluation by making it possible to track the keys and values clearly and evaluate the difficulty component as well.
  • The evaluation is thorough and covers multiple state of the art models.
  • The insights from the analysis are interesting and expose new failures modes for model working with long histories.
  • I like the breakdown and use of synthetic data here where it seems targeted, informative and does not take away from the applicability of the findings to real trajectories that are covered by the benchmark.
  • I also like how the design is aware of potential training-data leakage and selects the data accordingly.
  • Overall, I can see that the authors did a good job trying to bring rigor to this benchmark's design.
Critical weaknesses
  • The main issue for me is the reliance on exact match when the last sentence of the introduction is already mentioning how exact match could fail to separate state tracking from other failures such as formatting and copying issues. It makes me wonder how much of the takeaways are overshadowed by this metric not measuring exactly what we care about.
  • While I think benchmark is useful, it is a bit removed from real history usage in language models and AI assistants where the changes are more implicit and harder to put in the form of keys and values. I wonder what the authors think about this point and how this gap can be further closed.
  • The GPT-6 comparison seems a bit harder to interpret since it's on different samples. I might be leaning towards removing it or adding it as its isolated experiment rather than comparing to other models.
  • Would definitely like to see uncertainty estimates added here to make better since of what's significant and what's not.
  • Would like to see inference details as well, eg maximum output tokens (super critical for this work) and context length (super critical too). please let me know if I missed them.
Questions for authors

See weaknesses. Would love to have the authors address each.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3vEY (Part 1 of 2)

We thank the reviewer for the helpful comments. We have doubled GPT-6-Astra's evaluation and added the requested identical-history comparisons, uncertainty estimates, and complementary metrics, strengthening the evidence for the model comparisons and clarifying what complete-state EM measures.

W1: complete-state EM and failure analysis

We have added complementary text-normalization and per-key metrics (Appendix A.7, Tables 9 and 11), alongside analysis of generation, format, and history-grounded content errors (Section 3.4 and Appendix A.5, Tables 6 and 7). On the identical 108 Talk histories, whitespace normalization increased complete-state EM to 49.07–81.48% across Qwen, DeepSeek, GLM, and Kimi, while GPT remained at 96.30%. Normalization therefore narrowed the observed gaps, while complete-state correctness remained lower than GPT on these matched histories. Complete-state EM checks whether the final active-key set and every complete value are correct. For example, 76 of 634 analyzed Qwen DNS answers and 189 of 585 analyzed GLM LOBSTER answers returned deleted keys, directly revealing invalidation errors beyond formatting differences. Qwen Talk's strict value recovery remained almost unchanged between the lowest and highest NN settings (64.77% to 63.72%), while complete-state EM fell from 23.89% to 1.11%. EM quantifies complete-state task reliability, while the history-grounded error analysis separately supports our findings on updating, preservation, and invalidation errors.

W2: explicit histories versus implicit assistant updates

StateTracking isolates complete-state reconstruction under explicit update semantics, while identifying implicit changes from interactions is a complementary capability for practical assistants. For example, task completion may be inferred from tool results and file changes without explicit user confirmation. StateTracking fixes update semantics to isolate complete-state reconstruction, but does not yet evaluate the identification of such implicit changes. To narrow this gap, future extensions could combine dialogue histories, tool execution traces, and file revisions, with reference states verified against task completion criteria and environmental evidence. This would jointly evaluate implicit-update identification and state reconstruction, helping establish their relationship with practical assistant reliability.

Identifiable keys and verifiable references let us measure updating, preservation, and exclusion errors directly. Inferring operations and arguments from implicit user requests is a complementary assistant capability.

W3: comparability of the GPT sample

We have added a five-model comparison on identical histories in Appendix A.9, Table 13, and doubled GPT-6-Astra's evaluation from 513 to 1,026 distinct histories. The sample contains 108 histories each for Article and Talk and 270 each for Jira, LOBSTER, and DNS. The comparison uses the same scoring policy and retains every selected history in the denominator. GPT has the highest observed EM in every dataset on these matched histories. On Talk, GPT achieved 96.30% EM, compared with 23.15% for the best-performing other models. The advantage therefore persists when all models are compared on the same questions. This sampled comparison complements the four models' evaluation on the full benchmark of 3,420 histories.

Complete-state EM on the identical GPT-selected histories (%).

DatasetHistoriesQwenDeepSeekGLMKimi-K3GPT-6-Astra
Wikipedia Article10857.4183.3355.5683.3395.37
Wikipedia Talk10811.1123.1512.0423.1596.30
Jira27053.3366.3055.9385.9396.30
LOBSTER27059.6361.8538.8968.1596.67
DNS27048.5251.8535.9357.0488.15

Sampling was uniform without replacement within each workload bin: four histories per bin for Article and Talk, six for LOBSTER and DNS, and ten for Jira. Every population bin contains 20 histories (Section 3.3).

AuthorsOpenReview ↗

Response to Reviewer 3vEY (Part 2 of 2)

W4: uncertainty estimates

We have added paired uncertainty estimates for the expanded matched sample in Appendix A.9, Table 14, accounting for stratified sampling and weighting each of the 3,420 benchmark histories equally. GPT's overall EM advantage was 30.78–54.91 percentage points across the four model comparisons, and all simultaneous 95% confidence bounds excluded zero. Against Kimi-K3, the advantage was 30.78 points with a bound of 20.37–41.19 points. These results support GPT's overall advantage after accounting for sampling uncertainty.

Overall GPT-minus-model complete-state EM differences (percentage points).

ComparisonDifferenceApproximate 95% intervalSimultaneous 95% bound
GPT − Qwen46.4044.22–48.5936.00–56.81
GPT − DeepSeek36.9034.82–38.9826.49–47.31
GPT − GLM54.9152.64–57.1844.51–65.32
GPT − Kimi-K330.7828.57–32.9920.37–41.19

The approximate intervals account for stratified sampling without replacement. We used Hoeffding bounds with Bonferroni correction to provide simultaneous 95% coverage for the four model comparisons.

These intervals quantify sampling uncertainty on the fixed benchmark.

W5: inference budget and context

We have added the total context window, generation budget, reasoning effort, and tool/harness policy to Section 3.1 (Experimental Setup) of the revised paper. We apologize for the unclear description. All models used direct APIs without agentic harnesses or enabled tools. Each had a 1M-token total context window, including a generation budget of up to 128k tokens for reasoning and final output. Reasoning effort was the highest supported value: xhigh for Qwen3.8-Flash-Next and max for the other models.

Thanks for the clarifications and additional experiments, I think the matched-history comparisons and complementary metrics help address my concerns, although the normalization results show that formatting accounts for a substantial part of the original performance gaps. The workload comparisons also remain difficult to interpret because text length and operation mix are not fully controlled, so I would be cautious about drawing conclusions about the individual effects of history length and deletion frequency. Overall, I find the benchmark useful and the additional analysis helpful, so I keep my current positive score.

Thank you for your thoughtful feedback. Your suggestions helped us clarify the evaluation and strengthen the model comparisons. We are glad that the matched-history comparisons and complementary metrics helped address your concerns.

Reviewer_7xUkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark called StateTracking that targets a LMs ability to track an evolving state through a history of edits in five different settings: three grounded in real world data (Wikipedia articles / Wikipedia Talk discussion forums / Jira tickets) and two synthetic tasks (LOBSTER (Synthetic orders) / DNS (Synthetic network traffic)). The task is to return the final state after being provided with the initial state and a sequence of edits (each identified by unique keys); the final state is returned as a set of key-value pairs, where the keys are the edit IDs and the value is the final text after all edits. The authors source 20 histories for each combination of 3 hyperparameters (N: the number of keys to be returned during final state; T: the number of edits; rho: the proportion of text that is deleted across the edits), each of which are binned into three categories, for a total of 20 * 3^3 = 540 total data points for each setting.

The authors evaluate 4 open weight language models accessed through API and GPT-6-Astra. All models are relatively capable of returning all the keys, but show much more variance when tasked with returning the final state of text. The authors lastly provide detailed breakdowns in performance and identify the effect of varying N, T, and rho.

Critical strengths
  • I think that this is an extremely relevant benchmark given the agentic tasks that language models are being leveraged in. Many of these tasks (coding in a large repository, large scale summarization) require the faithful tracking of complex states, and this benchmark targets the LM's ability to perform complex state tracking.
  • The dataset is well constructed and is expansive over 5 separate domains involving both real-world and synthetic data. The hyperparameter bins are well motivated and provide a good characterization of the axes that govern complex state tracking tasks.
Critical weaknesses
  • The experimental setup section is not that well specified. What are the max allowed tokens for each model? Are the models given access to tools? Are the models called with barebones API or are they equipped with agentic harnesses (that might allow models to copy paste text easier)?
  • I would like to see some more analysis on the failure modes of each model. Right now, all failures are treated equally, regardless if the model missed a whitespace (failure to copy text), ran out of generation context window (could have extended max allowed tokens), produced degenerate outputs (a more serious failure case).
  • Overall, I think that the motivation and the benchmark construction are quite good, but I would like to see more details on both the experimental setups and how they differ between models and further analysis of results.
Questions for authors

rho is the hyperparameter controlling deletion frequency in your benchmark. Is there a notion of additional information being added from edits? How would you expect that your results would be affected if this is another axis in which the benchmark can vary?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7xUk

We thank the reviewer for recognizing the relevance of the task and the breadth of the benchmark. We have clarified the inference settings, expanded error analysis across all five models and datasets, and explained how addition/update frequency relates to deletion frequency and could affect performance.

W1: inference settings

We have added the context and generation budgets, reasoning settings, and tool/harness policy to Section 3.1 (Experimental Setup) of the revised paper. We apologize for not making these settings clear. All models were called directly without agentic harnesses or enabled tools. Each had a 1M-token total context window, including a generation budget of up to 128k tokens for reasoning and final output. We used the highest supported reasoning effort: xhigh for Qwen3.8-Flash-Next and max for the other models.

W2: failure modes

We have expanded the failure analysis across all five models and datasets in Section 3.4 and Appendix A.5, separating generation truncation and output-format failures from history-grounded content errors. Tables 6 and 7 report these two groups of failures, with complementary metrics in Appendix A.7, Table 9. The error analysis identified missed updates, returned deleted keys or obsolete content, and loss of content that remained valid through later edits. For example, 76 of 634 analyzed Qwen DNS answers and 189 of 585 analyzed GLM LOBSTER answers returned at least one deleted key, providing concrete evidence of invalidation errors.

Q1: additions, updates, and deletion frequency

Edits in StateTracking introduce new content, and varying addition/update frequency could affect performance differently across models and datasets (Appendix A.6). Whereas addition/update frequency can vary independently within fixed N,T,ρN,T,\rho settings, an additional axis would separate the demands of incorporating new content from preserving existing content. More frequent local additions or replacements could increase the need to integrate fragments and apply updates, potentially increasing missed updates or returned obsolete content. Complete-value overwrites could instead simplify reconstruction by exposing the final value directly. Frequency and the amount of content introduced per edit are distinct, so matching input/output lengths would help distinguish these effects from longer-output demands.

Addition and updating are also coupled to ρ\rho through the operation mix. In the current LOBSTER and DNS schedules, each deletion is paired with an insertion and the remaining changes are overwrites, so fixing TT and ρ\rho fixes both frequencies. Wikipedia permits further variation: at a fixed deletion frequency, a deletion event can remove text alone or replace it with new text. We have added this explanation to Appendix A.6, where Table 8 characterizes the existing addition/update frequencies.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer xTxP (Part 1 of 2)

We thank the reviewer for recognizing the complete-state task formulation, dataset diversity, and informative error analysis. We have added complementary metrics, history-grounded error analyses, length-matched Article comparisons, and identical-history model comparisons with uncertainty estimates, strengthening the evidence for state reconstruction errors and clarifying the interpretation of the workload results.

W1 / Q1: complementary metrics and complete-state correctness

We have added whitespace-normalized complete-state EM, normalized value accuracy, and normalized character-level F1 (Appendix A.7, Table 9). On the identical 108 Talk histories, whitespace normalization improves complete-state EM for all four models evaluated on the full benchmark, while GPT remains at 96.30%:

ModelComplete-state EM (%)Whitespace-normalized EM (%)
Qwen11.1149.07
DeepSeek23.1575.93
GLM12.0450.93
Kimi23.1581.48
GPT96.3096.30

Whitespace differences therefore account for a substantial part of the complete-state EM gaps. Normalization reduces these gaps across all four models, while complete-state correctness remains lower than GPT on the matched histories.

We have also expanded history-grounded error analysis across all five models and datasets (Section 3.4 and Appendix A.5, Tables 6 and 7). In the four full 540-history Talk evaluations, we identified 75, 36, 25, and 27 answers with missing active keys, returned deleted keys, or obsolete content for Qwen, DeepSeek, GLM, and Kimi, respectively. These traced errors provide direct evidence of membership and invalidation failures. The analysis also identifies missed latest updates and loss of still-valid content across the benchmark, supporting our findings on updating and preservation. Content analyses cover readable, normally completed, schema-valid answers. Generation and format failures are reported separately in Table 6.

For example, one Talk answer recovered all 16 normalized reference values but also returned a deleted section, yielding 100% normalized value accuracy and 98.36% normalized character-level F1, yet zero complete-state EM. The auxiliary metrics distinguish value recovery and text fidelity, while complete-state EM additionally checks that the answer contains exactly the current keys and their complete values.

AuthorsOpenReview ↗

Response to Reviewer xTxP (Part 2 of 2)

W2 / Q2: workload interpretation and controlled comparisons

We have added length-matched deletion-frequency comparisons (Appendix A.10, Table 15) and low/high-TT comparisons within valid answers (Appendix A.8, Table 12). Deletion-frequency differences remain model dependent when text lengths are similar.

Addition and updating are coupled to deletion frequency through operation composition (Section 2.3.3 and Appendix A.6, Table 8). In LOBSTER and DNS, deletion–insertion pairs replace overwrites while maintaining final active-key count. Our grid characterizes these reconstruction workloads. TT measures the supplied operation workload, and Appendix A.3.1 explicitly describes LOBSTER's order-independent solution. The evaluation objective is correct complete-state recovery.

All models were configured with a 1M-token total context window and a requested generation budget of up to 128k tokens for reasoning and final output (Section 3.1). Available API usage across all five models and datasets shows input medians of 16.4k–80.1k tokens, with every model–dataset group's 95th percentile at most 301.6k tokens and a maximum reported input in these usage records of 681.0k tokens. These inputs therefore had substantial context headroom even when reserving the full generation budget. Across all 3,420 histories, dataset medians of complete reference-answer length were 4.7k–26.5k characters.

To examine text-length effects, we compared two groups of 44 Article histories at medium versus high ρ\rho, within the same N/TN/T groups and with similar input and reference-answer lengths. Mean lengths differ by only 0.28% and 1.32%, respectively. EM changes by −18.18, −15.91, −13.64, and +15.91 percentage points for Qwen, DeepSeek, GLM, and Kimi. The same directions persist in a larger comparison (Appendix A.10, Table 15), supporting our finding that higher deletion frequency does not uniformly make reconstruction harder when text lengths are similar.

Appendix A.5, Table 6 reports generation, formatting, and request failures. Appendix A.8, Table 12 analyzes low/high TT outcomes for all five models and datasets. For valid-answer EM, Wikipedia uses whitespace normalization and records use the original field comparisons. Within normally completed, schema-valid answers, all four models evaluated on the full benchmark still have lower EM at high TT in Article, LOBSTER, DNS, and Jira. For example, LOBSTER EM falls from 98.33% to 13.79% for Qwen and 99.00% to 28.77% for DeepSeek. Talk varies by model, and GPT's high-TT EM remains at least 94.12% in every dataset. Completion and formatting failures therefore explain only part of these contrasts. These low/high-TT comparisons do not independently fix operation mix or text length, and comparable-length histories were too sparse for a length-independent TT estimate. Section 3.3 interprets the trends accordingly.

Q3: matched GPT comparison and uncertainty

We have added five-model comparisons on 1,026 identical histories and paired uncertainty estimates in Appendix A.9 (Tables 13 and 14). GPT-6-Astra's evaluation has been doubled from 513 histories, and the comparison uses the same scoring policy. GPT has the highest observed EM in every dataset. On Talk, GPT achieves 96.30%, compared with 23.15% for the best-performing other models. Table 14 reports overall paired EM differences and uncertainty estimates accounting for stratified sampling. GPT's overall advantage is 30.78–54.91 percentage points across the four model comparisons, and all simultaneous 95% confidence bounds exclude zero. These results support its overall advantage after accounting for sampling uncertainty.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Response

Thank you again for your time and thoughtful review. With the rebuttal period nearing its end, we would be grateful if you could take a look at our response when your schedule permits. We would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

General Response

We sincerely thank the PCs, SACs, ACs, and reviewers for their time, thoughtful feedback, and constructive comments.

We appreciate the reviewers' consistent recognition of StateTracking's motivation, task formulation, and design. 7xUk described it as “an extremely relevant benchmark” and noted that “the motivation and the benchmark construction are quite good”. 3vEY wrote that “The motivation for this benchmark is very well explained” and highlighted how the design makes it possible to “track the keys and values clearly and evaluate the difficulty component as well”. VURc recognized “A clearly defined and practically relevant task”, while xTxP noted that our work “addresses an important limitation of existing memory and state-tracking evaluations”. iYfy also highlighted that the design “makes updating, preservation and deletion observable in one answer”.

The reviewers' thoughtful feedback highlighted opportunities to further strengthen the paper through additional experiments and analyses. We have addressed their comments point by point with clarifications and new results, and summarize the main revisions in three areas below:

  1. History-grounded error analysis: We expanded the analysis of updating, preservation, and invalidation across all five models and datasets, adding error counts and traced examples tied directly to the given histories and reference states (Section 3.4; Appendix A.5).

  2. Additional workload comparisons: We added length-matched Article comparisons, within-history last-update analyses, and comparisons among valid answers at low and high transition counts. We also explained how addition and update frequencies relate to deletion frequency (Section 3.3; Appendices A.6, A.8, and A.10).

  3. Complete-state and per-value evaluation: We added complementary metrics and per-value recovery comparisons across all five models and datasets, examining both individual-value recovery and the correctness of the entire final state (Appendix A.7; Tables 9 and 11).

All revisions are marked in red in the revised paper, and detailed answers are provided in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.