- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
StateTracking evaluates whether models can reconstruct all surviving keys and their complete final values from an initial state and an ordered update history. Its 3,420 histories cover Wikipedia Article, Wikipedia Talk, Jira, synthetic orders and DNS records, organized by final state size, transition count and deletion frequency. The evaluation excludes external retrieval, persistent memory and executable tools. It measures exact key recovery and complete state exact match, with analyses of obsolete content, deletion failures, text fidelity and output errors.
Critical strengths
-
Requiring both the final collection’s membership and every retained item’s complete value makes updating, preservation and deletion observable in one answer. This extends designated file reconstruction, such as LoCoDiff, by requiring the model to identify which items remain active. The distinction between key-set exact match and complete state exact match also exposes cases where the model finds the right items but reconstructs their contents incorrectly.
-
The dataset provides five domains and twenty histories per joint workload bin. Reference states are checked through replay, and LOBSTER and DNS use matched operation and key access sequences. Wikipedia additionally provides paired natural language and formal representations. These features support comparisons across workload conditions and offer a basis for testing how input representation affects performance.
Critical weaknesses
-
The connection to practical assistant workflows is motivated but not demonstrated. The introduction discusses coding, collaborative editing and workflow management, yet Section 3.1 excludes the persistent state and executable tools that could handle explicitly specified updates. Section 4.2 clearly limits the benchmark to reconstruction from a fixed history, so the existence of a replay solution does not invalidate its model-level objective. However, the experiments do not establish whether performance under this restriction predicts reliable state management in an assistant that uses external tools. The practical interpretation of the scores therefore remains uncertain.
-
The evaluation does not sufficiently separate instruction interpretation, state reconstruction and exact reproduction. The authors already reduce ambiguity through templates, identifiable keys and explicit operation semantics; there is no basis to assume the instructions are inherently ambiguous. Nevertheless, Section 3.1 evaluates Wikipedia using natural language descriptions, while Section 2.3 and Appendix A.2 describe paired formal inputs without reporting a same history performance comparison. It is therefore unclear how much the input representation contributes to the observed failures. Strict scoring adds another distinction: Section 3.4 attributes 299 of Kimi-K3’s 362 Wikipedia Talk answers with correct keys but incorrect values to whitespace differences. Deleted keys and obsolete passages demonstrate state errors, but the overall EM gap does not establish their prevalence relative to interpretation and copying failures.
Questions for authors
-
Clarification: Beyond measuring toolfree reconstruction, what use do you intend for the benchmark scores in assessing practical assistants? Given the replayable update histories, which capabilities would still need to be measured when persistent state and executable tools are available, and what evidence supports that interpretation?
-
Clarification: Were the paired natural language and formal Wikipedia inputs evaluated on the same histories? If so, how do key-set and complete-state exact match differ between representations, and how much of the remaining value-error gap disappears after whitespace normalization? If these comparisons were not performed, please clarify the limits this places on attributing failures specifically to state tracking.
3 comments
Response to Reviewer iYfy (Part 2 of 2)
Q2: same-history representations and whitespace normalization
Same-history input representations.
We have added Qwen's same-history natural/formal evaluations on 540 Article and 540 Talk histories, with identical reference states for each pair (Appendix A.7, Table 10).
| Dataset | Input representation | Histories | Complete-state EM (%) | Whitespace-normalized EM (%) | Key-set EM (%) |
|---|---|---|---|---|---|
| Wikipedia Article | Natural language | 540 | 57.22 | 60.93 | 93.15 |
| Wikipedia Article | Formal | 540 | 65.56 | 68.52 | 97.59 |
| Wikipedia Talk | Natural language | 540 | 10.56 | 44.26 | 83.70 |
| Wikipedia Talk | Formal | 540 | 11.48 | 52.78 | 82.22 |
For Qwen, formal input improves whitespace-normalized complete-state EM from 60.93% to 68.52% on Article and from 44.26% to 52.78% on Talk. These paired results show that input representation affects complete-state recovery in both datasets.
Whitespace sensitivity of the current natural-input results.
Whitespace normalization substantially reduces the gap between key-set and complete-state EM on Talk for all four models evaluated on the full benchmark (Appendix A.7, Table 9). Whitespace normalization removes all Unicode whitespace from values. It retains the original keys, answer schema, and evaluation denominators.
Wikipedia Talk, natural inputs (%). The four models evaluated on the full benchmark retain all 540 histories; GPT has 108 sampled histories.
| Model | Histories | Complete-state EM | Whitespace-normalized EM | Key-set EM |
|---|---|---|---|---|
| Qwen | 540 | 10.56 | 44.26 | 83.70 |
| DeepSeek | 540 | 18.15 | 71.67 | 95.93 |
| GLM | 540 | 11.67 | 49.81 | 77.78 |
| Kimi-K3 | 540 | 21.48 | 76.85 | 88.52 |
| GPT-6-Astra | 108 | 96.30 | 96.30 | 99.07 |
For the Kimi Talk result cited by the reviewer, whitespace normalization reduces the gap between key-set and complete-state EM from 67.04 to 11.67 percentage points. Of the 362 correct-key, inexact-value Kimi Talk answers, 299 become exact after this whitespace normalization. Table 9 also reports Article, where whitespace normalization improves EM by 2.59–6.67 points for the four models evaluated on the full benchmark. The normalization results quantify text-fidelity sensitivity across models and domains, while the history-grounded counts above identify concrete membership and value-update errors.
Thanks. I will keep the positive score.
Response to Reviewer iYfy (Part 1 of 2)
We thank the reviewer for recognizing the complete-state reconstruction objective, reference validation, and workload coverage. We have completed the requested same-history natural/formal comparisons, including whitespace-normalized scoring, and clarified StateTracking's contribution as a controlled model-level evaluation of updating, preservation, and invalidation.
W1: model-level evaluation and practical assistants
We agree that the predictive value of these scores for tool-enabled assistant reliability remains to be established, and view StateTracking's current contribution as a controlled evaluation of model-level state reconstruction. Our tool-free protocol assesses the model's own ability to reconstruct an evolving state from its history. It provides model-level comparisons of updating, preservation, and exclusion under fixed update semantics and verifiable references.
A central feature of StateTracking is the joint requirement to identify the full active collection and recover every item's complete final value, without being given the final key set. Local updates must be integrated while unchanged content is preserved and deleted or superseded content is excluded. The common complete-state objective and N,T,ρ workload bins make these requirements observable across different workloads.
The same histories and references could be reused with persistent state and tools, allowing comparisons of how external state management changes these errors and how state correctness relates to downstream task success.
W2: input representation, state reconstruction, and exact reproduction
We have added Qwen's same-history natural/formal comparison (Appendix A.7, Table 10), whitespace-normalized EM (Table 9), and history-grounded error counts (Appendix A.5, Table 7).
Exact reproduction and history-grounded errors.
Complete-state EM combines collection correctness and exact value reproduction. The per-model, per-domain analysis distinguishes text fidelity, membership, obsolete content, updating, and preservation. Among 634 analyzed Qwen DNS answers, 76 returned at least one deleted key and 54 returned obsolete content; among 585 analyzed GLM LOBSTER answers, the corresponding counts were 189 and 70. Content error counts cover readable, normally completed, schema-valid answers. Generation and output-format failures are reported separately in Table 6. These history-grounded counts provide direct evidence of membership and value-update errors.
The input-representation and normalization results are detailed in Q2 below.
Q1: intended use and capabilities with persistent state and tools
For assistants with persistent state and tools, we would assess correct update interpretation, item selection, consistency of stored state, and use of current state in subsequent tasks. Given canonical operations, an external executor can apply the specified state changes. For example, a successfully executed call could target the wrong issue, or a later answer could use an obsolete value despite a correct stored state.
Section 3.4 and Appendix A.5 document missed updates, returned deleted keys, and loss of unchanged content, providing concrete outcomes to check in such workflows. These histories and references would allow us to record tool calls and maintained states, and compare those states with replay references. Reporting complete-state EM alongside success on tasks that depend on the resulting state could connect reconstruction to practical outcomes and help locate errors in update interpretation, tool coordination, or subsequent state use.