- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies persistent world modeling under partial observability: tracking the interaction-established environment states.
The authors argue that methods that retain interaction history (full context, summaries, recurrent memory) can answer single-slot queries accurately while failing to maintain a jointly consistent state.
To address this issues, the paper proposes StateCommit, which (1) maintains an explicit, validated and typed state table updated at interaction time; (2) for visual prediction, binds each slots to the frame observed at its last update.
They also introduce StateReturn, a benchmark that tests whether interaction-established state is maintained out of view and reflected in future predictions, which existing metrics (per-query accuracy, full-frame reconstruction) can miss. It contains AI2-THOR text episodes and MiniGrid visual worlds, with K unrelated interactions inserted before evaluation, and is scored by full-state exact match (FSEM) and counterfactual-paired pixel metrics. At K=16, Full-History reaches 98.61% target accuracy but only 13.89% FSEM, while StateCommit achieves 98.67–100% FSEM and reduces state-sensitive prediction error by 58.1%.
Critical strengths
-
the paper identifies a clear and practically relevant failure mode: retaining full history can achieve high accuracy on single-slot query while fail to maintain whole-state consistencies, which indicates a gap in the per-query metrics.
-
the evaluation design is detailed and thoughtful.
-
the authors release the dataset and the implementation code, and the paper is clearly written with informative figures.
Critical weaknesses
-
missing baselines. the core idea of StateCommit is not new: maintaining validated active records of relevant entities instead of reconstructing the state from full interaction history. several agent memory approaches, such as Mem0 and A-MEM, have adopted the same idea, maintaining a graph structure to store the facts about the entities. However the paper does not compare them with the proposed approach.
-
Limited benchmark headroom. the proposed approach nearly saturates the StateReturn benchmark, which limits its value for future work.
Questions for authors
-
Is there an ablation that removes runtime verification? According to Table 4, no proposal was ever rejected on Structured text or visual inputs, so verification cannot have affected those results. On Natural text, rejected proposals were all repaired. Without this ablation, the contribution of the transactional mechanism remains unclear.
-
How do the missing baselines mentioned in the weaknesses section perform in the proposed benchmark?
6 comments
Response to Reviewer AjT7 (part 2)
Q1: Is there an ablation that removes runtime verification? According to Table 4, no proposal was ever rejected on Structured text or visual inputs, so verification cannot have affected those results. On Natural text, rejected proposals were all repaired. Without this ablation, the contribution of the transactional mechanism remains unclear.
We thank the reviewer for this important question. We conduct both a Runtime-removal ablation and a controlled transaction fault-injection diagnostic to clarify the contribution of Runtime verification.
(1). Standard StateReturn evaluation does not fully expose Runtime's protective role. We replace the verified Runtime with an unverified Active-State updater and evaluate its effect on FSEM. The results are presented below:
| Input | w/ Runtime FSEM (%) | w/o Runtime FSEM (%) |
|---|---|---|
| Structured | 100.00 | 100.00 |
| Natural | 98.67 | 98.67 |
Removing Runtime leaves FSEM unchanged for both Structured and Natural inputs, suggesting that the standard evaluation does not sufficiently stress the transaction failures targeted by Runtime verification.
(2). Runtime verification prevents state corruption under injected transaction faults. We conduct a paired diagnostic with 300 invalid transactions (100 each for stale versions, expected-value mismatches, and invalid typed updates) and 300 clean controls. The results are reported below:
| Transaction Type | Count | Runtime Rejection | FSEM w/ Runtime | FSEM w/o Runtime |
|---|---|---|---|---|
| Stale Version | 100 | 100.00 | 100.00 | 0.00 |
| Expected-Value Mismatch | 100 | 100.00 | 100.00 | 0.00 |
| Invalid Typed Update | 100 | 100.00 | 100.00 | 0.00 |
| Clean Control | 300 | 0.00 | 100.00 | 100.00 |
Runtime rejects all 300 invalid transactions without rejecting any clean controls, preserving state integrity in every case. These results demonstrate the protection provided by deterministic transaction verification.
Thank you for the detailed response and the additional experiments. The new results on state-consistent prediction and degraded visual evidence address my concern about benchmark headroom, and the Mem0/A-MEM comparison and Runtime ablation are helpful. I have one follow-up question:
Q: StateCommit relies on a predefined state schema: a fixed set of attributes (place, openness, toggle state) with closed value domains. For general world models, deciding what state to track is a key part of the problem. How would the method extend to open-world settings where the relevant attributes are not known in advance? Also, were Mem0 and A-MEM given the same schema (e.g. in their extraction prompts to inform them what to track), or did they have to decide on their own what to store?
Response to Reviewer AjT7 (part 1)
We would like to thank the reviewer for the very insightful and valuable comments. Below are our responses to the comments in Weaknesses and Questions.
We thank the reviewer for highlighting these relevant approaches. We conduct additional comparisons with Mem0[1] and A-MEM[2], and clarify the distinction between existing agent memory and StateCommit below.
(1). StateCommit substantially outperforms agent-memory baselines in persistent-state maintenance. We evaluate Mem0 and A-MEM on StateReturn-Text (Natural). The results are presented below:
StateCommit achieves 98.67% FSEM, outperforming A-MEM by 63.00 percentage points, with clear advantages in target accuracy and coverage.
(2). StateCommit differs from existing agent memory in how it maintains and uses world state. Mem0 incrementally updates entity-relation memories, while A-MEM organizes and evolves interconnected memory notes for retrieval. In contrast, StateCommit maintains a query-independent Active State through versioned transactions and atomic commits, enabling direct state readout and state-consistent visual prediction.
Reference:
[1]: Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory, ECAI 2025.
[2]: A-Mem: Agentic Memory for LLM Agents, NeurIPS 2025.
We thank the reviewer for raising this important concern. We agree that StateReturn-Text leaves limited headroom. However, we would like to highlight two points below.
(1). StateReturn retains substantial headroom in State-Consistent Prediction. Although StateCommit achieves near-perfect FSEM in State Maintenance, its State-Consistent Prediction performance remains below saturation across interaction horizons. The results are summarized below:
StateCommit achieves 83.33% CFP overall, decreasing to 76.39% at K=16. This indicates that translating accurately maintained states into consistent future predictions remains an open challenge within the existing benchmark.
(2). Imperfect visual evidence reveals further benchmark headroom. We further examine StateCommit under four visual evidence conditions, with results summarized below:
StateCommit's FSEM drops from 100.00% under clean conditions to 13.66% under moderate sensor degradation. These results highlight opportunities to extend StateReturn toward more challenging visual evidence conditions.