acceptodds
Under review as a conference paper at ICLR 2027

WorldConsistMem: Evaluating Representation-Sensitive State-Transition Reconstruction

Abstract

Long-term memory systems for evolving worlds must reconstruct state changes, not merely retrieve related records. WorldConsistMem separates evidence availability from state-transition reconstruction on gold evolving histories: a transition query asks for the previous state, the transition event, and the resulting state. Under a fixed GPT-4.1 snapshot and byte-matched BM25 evidence packs with gold-support recall = 1, a generic response interface reconstructs only 13/150 transitions (8.7%). A preregistered bundled structured transition prompt/representation intervention—specifying the required prev/event/next representation and a canonical subtype-specific output schema—raises accuracy to 109/150 on the same evidence and snapshot. On the original explicitly/operationally supported slice (S1+S2; 93 packs), accuracy rises from 13/93 to 52/93 (paired Δ+41.9 percentage points; bundle-bootstrap 95% CI [31.2, 52.7]; McNemar p ≈ 3.8 × 10^−11). A prospective post-result factorization shows the gain is not reducible to formatting alone: schema-only reaches 90/150 and representation-only 39/150, both increments remain positive after the other, and their interaction is additive-compatible. A forensic audit finds leakage PASS, 0 format-only (M5) wins, and 0 scorer-weakness (M8) wins among 97 genuine non-format improvements. Document transitions and original-label S2 cases remain 0/34 in every arm; CEO transitions reach 23/30 under the bundled interface. We conclude that measured transition reconstruction is highly representation-sensitive under this benchmark, prompt bundle, and snapshot—with schema the larger measured lever and explicit prev/event/next independently contributing—not that either factor alone, intrinsic model incapacity, or complete resolution explains the residual floors.

Primary area:
datasets and benchmarks
Keywords:
long-term memory, state-transition reconstruction, representation sensitivity, retrieval-augmented generation, evaluation benchmark, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EdSaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WorldConsistMem, a controlled benchmark for evaluating state-transition reconstruction over evolving synthetic histories. A transition query requires the model to reconstruct the previous state, the transition event, and the resulting state, explicitly separating evidence retrieval from reconstruction.

The primary experiment fixes the GPT-4.1 snapshot and byte-identical BM25 evidence packs with gold-support recall =1, then compares a generic response interface against a preregistered bundled structured transition interface specifying prev/event/next and a canonical subtype-specific schema. Accuracy increases from 13/150 to 109/150.

A prospective post-result factorization separates schema-only and representation-only effects. Schema-only reaches 90/150, representation-only reaches 39/150, and the bundled condition reaches 109/150, with both factors contributing positively and no claimed interaction beyond additivity.

The paper also performs forensic analyses of scorer behavior, parsing, leakage, and residual failures. Document transitions and original-label S2 cases remain 0/34 under every arm, while CEO transitions improve to 23/30 under the bundled interface.

Critical strengths
  • The paper cleanly separates evidence availability from reconstruction, avoiding the common mistake of treating retrieval recall and downstream state reasoning as the same capability.

  • The primary intervention is tightly controlled: same model snapshot, same query IDs, byte-identical evidence packs, and a preregistered paired comparison.

  • The headline effect is large and statistically convincing, especially on the original S1+S2 slice, where accuracy improves by 41.941.9 percentage points.

  • The post-result factorization is reported transparently. The authors explicitly avoid presenting F1/F2 as preregistered factorial evidence after observing F3.

  • The forensic audit is a strong aspect of the work: scorer aliases and a runner-wrapper defect are disclosed without altering frozen raw model outputs.

  • Residual failures are not hidden. Persistent document and S2 floors make clear that better representation does not solve evidence insufficiency or all reconstruction errors.

Critical weaknesses
  • The benchmark is synthetic and the primary structured experiment uses a single GPT-4.1 snapshot, limiting confidence that the effect generalizes across models or realistic memory systems.

  • Schema-only reaches 90/150, suggesting that much of the gain comes from specifying the expected answer structure rather than uniquely from explicit temporal-state reasoning.

  • Exact-match scoring makes the benchmark particularly sensitive to field structure, aliases, and output contracts, so measured reconstruction accuracy may partly reflect interface compliance.

  • The original generic baseline may be substantially disadvantaged by asking for an underspecified response while evaluating against a rigid structured tuple.

  • Human validation is incomplete and small, with only n=30n=30 planned examples, so there is little independent evidence that the benchmark’s exact-match notion matches human judgments of correct reconstruction.

  • Document and S2 cases remain unresolved in all conditions, leaving a sizable portion of the benchmark outside the demonstrated benefit of the proposed representation.

Questions for authors

My main question is how much of the F0-to-F3 improvement should be interpreted as task clarification versus improved state-transition reasoning. Since F1 alone reaches 90/15090/150, could the authors provide a semantic evaluation that ignores output structure entirely and compares whether the underlying previous/event/next content was inferable from F0 responses?

Second, could the authors evaluate at least one additional model under the full F0–F3 design? The paper explicitly notes that a Gemini replication is prepared but not executed. Given the central claim of representation sensitivity, cross-model replication would materially affect my confidence.

Third, how was the generic F0 interface originally selected? Was it representative of interfaces used by existing memory systems, or mainly a deliberately generic baseline? This matters for interpreting the practical significance of the 13/150 baseline.

Finally, for the persistent S2 and document failures, can the authors quantify how many cases are impossible under the provided evidence versus merely unresolved by the tested interfaces? A sharper separation would help distinguish representation limitations from evidence insufficiency.

Reviewer_vfz3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

WorldConsistMem evaluates state-transition reconstruction in synthetic worlds, comparing prompt representations while holding the model and retrieved evidence fixed.

Critical strengths

The controlled comparison with identical evidence packs helps isolate the effects of prompt representation.

Critical weaknesses

1 The reported improvement is confounded with compliance with the answer schema. Appendix T asks when a conflict was resolved, yet rejects the correct scalar answer “21” because additional fields are missing. Unless the baseline explicitly requires those fields, this comparison does not establish improved reconstruction.

2 The diagnostic labels are unstable. Appendix P reports that a sufficiency re-audit changes 77/150 labels, while the original labels remain in use. This undermines the interpretation of failures as evidence insufficiency versus reconstruction failure, even if aggregate accuracy remains unchanged.

Questions for authors

Does the complete F0 prompt explicitly require every gold field? What conclusions survive task-aligned semantic grading and the corrected evidence-sufficiency classification?

Reviewer_RkeoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces WorldConsistMem, a synthetic benchmark for evaluating whether LLM readers can reconstruct state transitions over evolving histories when gold-supporting evidence is available. It explicitly separates evidence availability (gold-support recall = 1) from transition reconstruction, and focuses on transition queries that require a structured prev/event/next tuple. In the primary experiment, evidence packs and the GPT-4.1 snapshot are fixed; the original generic response interface (F0) achieves 13/150 transition accuracy, while a bundled structured transition prompt/representation intervention (F3) achieves 109/150. On the original S1+S2 slice, accuracy rises from 13/93 to 52/93. A post-result factorization estimates schema-only (F1, 90/150), representation-only (F2, 39/150), and approximately additive contributions. The paper’s main contributions are a controlled benchmark, an evidence-conditioned evaluation separating missing support from reconstruction failure, a confirmatory bundled intervention result, and unusually transparent scorer/parser corrections and audits. Validation is based on frozen outputs, leakage audit, independent rescoring, and secondary retrieval/oracle/joint cells, but remains limited to synthetic worlds and a single GPT-4.1 snapshot.

Critical strengths
  1. The clean separation between evidence availability and transition reconstruction is the strongest contribution. It shows that recall = 1 can coexist with systematic failure to reconstruct a resulting state, which is directly relevant to retrieval-augmented and memory-based LLM evaluation. This is consequential because many evaluations conflate retrieval success with correct use of evidence.

  2. The transparency around scorer correction, parser defects, frozen predictions, and leakage audits is a real strength. It makes the main result more trustworthy than a typical prompt-engineering gain, even though the paper’s defensive framing makes it harder to read.

Critical weaknesses
  1. The paper is scientifically dense but poorly presented. Key contributions are buried under defensive non-claims, many abbreviations, and frequent appendix cross-references. The typography/fonts in the PDF are also inconsistent or malformed, and several pages contain garbled numeric/page artifacts, which materially impairs readability. A cleaner, consistently typeset manuscript with a clearer narrative would substantially improve assessment.

  2. The central finding is useful: evidence availability is not equivalent to correct state-transition reconstruction, and the representation/schema of the required update can dramatically change measured accuracy. However, because the benchmark is synthetic and the analysis is limited to one GPT-4.1 snapshot, I did not gain a substantially deeper understanding of real-world memory systems or general LLM transition reasoning.

  3. The authors are unusually transparent about scorer v1→v2, the bare-JSON runner defect, frozen outputs, and leakage audits, which is commendable. Still, the headline result depends on a bundled intervention; F1/F2 are explicitly post-result rather than preregistered; the benchmark is synthetic; only one GPT-4.1 snapshot is used for the main structured/factorized arms; human validation is incomplete; and document/S2 floors remain unresolved. These do not invalidate the paper, but they limit the strength and generality of the conclusions.

  4. External validity is narrow. The benchmark is synthetic, the main structured/factorized experiments use a single GPT-4.1 snapshot, and there is no multi-provider or frontier replication. The large gain may be specific to this benchmark’s schema-elicitable fields, so the implications for real memory systems remain limited.

  5. The central F3 intervention is bundled, and the F1/F2 factorization is post-result, not preregistered. Combined with the difficult presentation, inconsistent typography/fonts, and many defensive non-claims, it is hard to locate the exact scientific contribution and to determine how much of the gain is semantic reconstruction versus representation, schema, parser, or scorer compatibility.

Questions for authors
  1. Can you provide a concrete audit of how subtype schemas were selected using only the public query-id slot? How do you rule out inadvertent answer leakage?

  2. Since F1/F2 were designed after observing F3, what exactly was frozen before execution, and how did you control analyst degrees of freedom?

  3. How much of the F0→F3 gain remains under the original v1 scorer and the original online parser? Please separate semantic reconstruction gains from parser/scorer compatibility gains.

  4. For document and original-label S2 cases that remain 0/34 in every arm, is the required resulting-state statement or version identifier actually present in the evidence packs? If not, should these be counted as reconstruction failures?

  5. Will the camera-ready version fix the font/typography issues and provide a clean, consistently typeset PDF?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.