- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks whether learned models that give identical, correct answers about a system's current state stay interchangeable after further updates. The authors build pairs of histories that reach the same true state and apply the same actions to both. Predictions often diverge later. Rebuilding the state from its current answers, a classical requirement called readout closure, restores agreement and makes the world model nearly perfectly accurate, though in memory agreement doesn't guarantee accuracy.
Critical strengths
- The phenomenon is concrete and clearly demonstrated. The paired design holds the checkpoint, true state and continuation fixed, which isolates the effect of history.
- The reporting is transparent. It includes mixed and negative results, prespecified success criteria, and a failed earlier protocol.
- Reconstruction gives large and consistent accuracy gains in C-SWM.
- NRT comes with a formal equivariance guarantee and proof.
Critical weaknesses
- Lack of clarity and readability. The abstract and introduction uses terms like “readout” and “retained state” without explaining what they mean. The NRT (Section 4 onwards) only loosely ties to the rest of the paper so it reads like two papers. There is not enough coherence to make this paper publishable.
- Limited experimental results. Small number of samples are used with only 30-53 pairs for memory experiments. Two memory checkpoints are not independent models. Arbitrary toy tasks only with a 64-slot table and 5x5 grid with five objects.
- Lack of generalizability. It is unclear how it would generalize to real world models where the full state cannot be fully read out.
- NRT is not better than Structured SIT. A simpler Structured SIT beats NRT on accuracy on all the conditions, which makes it unclear how NRT is a contribution.
- Lack of novelty and incremental contribution. The paper acknowledges that its closure criterion is the classical congruence/quotient condition from universal algebra. The contribution therefore rests mainly on the experiments, whose scope is limited.
Questions for authors
- What is the unifying claim connecting Sections 2–3 with Section 4?
- Do the results hold with independently trained memory models and more C-SWM seeds?
- How would reconstruction work when the readout is partial or the state cannot be fully decoded?
- Given that Structured SIT matches NRT’s guarantee with higher accuracy, what is the case for NRT?
Thank you for asking how the two parts connect. The revision clarifies the relation that should survive updates in each setting and adds tests across model initializations and partial readouts.
From observed failure to a design specification. In the original §§2–3, two histories can reach the same sufficient source state and give the same correct current answers, yet yield different predictions under identical actions. One C-SWM reconstruction from each state's own answers preserves those answers and improves future source correctness with the same updater (original Tables 3–4, revised Figure 3). The revised Introduction calls this an interchangeability failure. Section 4 treats the decoded-store setting, where we can specify exactly how a change of state encoding and its corresponding actions should transport the predictions.
Independently trained models. We added three C-SWM initializations on the original training data and one independent memory-model initialization. Two grid initializations diverged and benefited from reconstruction in both history families. The third was already correct at all tested endpoints. The independently initialized memory model diverged after both deletes and overwrites (§3, revised Tables 4–5, Appendices B.2 and B.5).
Partial readouts. We added a check with dialogue trackers factorized by restaurant attributes (WOZ 2.0). With fixed weights and deterministic updates, replaying the observed prefix from the fixed initial state recovers the 96 of 840 recurrent coordinates that preserve the tracker's own area and price predictions under any common continuation. This identifies a compact history-dependent state for those two predictions (revised Appendix B.9).
Purpose of NRT and Structured SIT. NRT realizes the specified correspondence by design. Transporting the decoded store and its action sequence together under identifier and slot permutations preserves corresponding predictions for finite source-legal rollouts in deterministic, exact arithmetic (Theorem 1). We revised §4.2 to separate the original slot-augmented SIT control from the direct comparison with two alternative architectures on the same interface. Structured SIT checks the same correspondence and accuracy criteria in another architecture. It leads on both accuracy measures in all four conditions. NRT comes within 1.2 percentage points of it in mean query accuracy in each condition and within 2.1 points in full-state accuracy for tiered memory with 1,024 training transitions. The full-state gap is larger for Records / 1,024 (original Table 7, revised Appendix E.5, Table 10).