acceptodds
Under review as a conference paper at ICLR 2027

The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory

Abstract

Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same p=0.03 can support confirmation when produced by a preregistered outcome under a fixed stopping rule, yet only exploration when selected from twenty inspected outcomes. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.

Primary area:
datasets and benchmarks
Keywords:
agent memory, scientific agents, scientific validity, benchmark evaluation, scientific state representation, causal diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oAcqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SciValidityBench to evaluate whether agents use remembered scientific evidence appropriately, and SciStateForge to represent experimental histories as explicit scientific states. Experiments across four model families report improved decision accuracy over baseline memory representations.

Critical strengths

The paper proposes an interesting task that distinguishes correctly recalling evidence from using it appropriately in scientific decisions, offering a useful perspective for evaluating scientific agents.

Critical weaknesses

Limited task difficulty and realism. The benchmark uses short synthetic histories, averaging only three public events per world. Qwen3-14B already achieves 99.1% accuracy with Flat SSG, leaving little headroom in this setting. The benchmark may therefore offer limited challenge to frontier models when compiled states are provided.

Insufficient evidence of external generalizability. Although the paper reports additional TaskSpecs and source-level controls, the evaluation remains based on predefined validity mechanisms and synthetic generators. The reported gains show that readers benefit from supplied decision-relevant states, but do not establish whether the state-construction rules transfer reliably to independently sourced histories within the same mechanisms. This limits the demonstrated external applicability of SciStateForge.

Questions for authors
  1. Is the benchmark sufficiently challenging for frontier models? What absolute accuracy do frontier models achieve using only raw event histories, without SciStateForge? Can they already achieve high accuracy, and how much additional benefit does Forge provide?

  2. Why are the GPT gains in Figure 3B substantially larger than those for Qwen? The reported gains of 47.0–48.6 percentage points imply Event Graph accuracies no higher than approximately 51.4%–53.0%, whereas Qwen3-14B achieves 79.4%. Please report the hosted models’ absolute accuracies under each condition and explain this discrepancy, including whether the evaluation settings are comparable.

  3. Can Forge’s rules generalize beyond the benchmark’s templates? Which rules are template-specific, and which are shared across tasks? Within the five validity mechanisms studied, is there evidence that the same core rules apply to independently sourced scientific histories?

6 comments

We thank Reviewer oAcq for highlighting the importance of absolute raw-history baselines, comparable hosted-reader results, and evidence beyond the benchmark templates. We agree that the original paper needed to state its realism and external-validity limits more directly. The responses below provide the requested rates and describe the additional controlled checks without treating them as natural-laboratory validation.

Q1. Is the benchmark challenging enough for frontier models, and what are the raw-history baselines?

“Limited task difficulty and realism. The benchmark uses short synthetic histories, averaging only three public events per world. Qwen3-14B already achieves 99.1% accuracy with Flat SSG, leaving little headroom in this setting.”

“Is the benchmark sufficiently challenging for frontier models? What absolute accuracy do frontier models achieve using only raw event histories, without SciStateForge? Can they already achieve high accuracy, and how much additional benefit does Forge provide?”

Response. We agree that the current suite is short and controlled, and we now report absolute rates rather than only gains. The open-weight matrix is:

ReaderRawBM25Event GraphFlat SSGSVL
Qwen3-14B0.7980.8020.7940.9910.981
Gemma-3-12B0.7330.7190.7190.9030.902
Mistral-7B0.5560.5860.5660.6920.678
Phi-3.5-mini0.4800.4750.4780.6090.570

Qwen’s Flat−Raw gain is +0.193 and Flat−Event gain is +0.197; the high compiled-view score does not imply that the raw-history problem is solved. The headroom analysis above also shows that the remaining strong-reader failures are concentrated in Sunk Cost, while weaker readers remain challenged across multiple mechanisms.

Two full matrices now cover 4,800 decisions over 120 worlds each. For Luna, Raw/Event/Flat/SVL validity is 0.6500/0.4917/0.9992/1.0000; for Terra it is 0.6042/0.4842/0.9858/0.9767. Pooled validity at k=0/4/16/64/256 is 0.7771/0.7875/0.7854/0.7896/0.7865 for Luna and 0.7302/0.7604/0.7698/0.7771/0.7760 for Terra. Each matrix has parse rate 1.0000 and 2,400 complete pairs; action consistency is 0.9771 for Luna and 0.9688 for Terra. Distractors surround already compiled views, with near-miss status explicitly unrelated. The smaller Sol pilot gives a compatible additional check, with validity of 0.7500/0.7708/0.7708/0.7708/0.7604 over the same five k values.

Q2. Why are the hosted GPT gains larger than the Qwen gains?

“Why are the GPT gains in Figure 3B substantially larger than those for Qwen? ... Please report the hosted models’ absolute accuracies under each condition and explain this discrepancy, including whether the evaluation settings are comparable.”

Response. The hosted gains are real and arise mainly from a lower uncompiled Event Graph baseline. The hosted absolute rates are:

Seed blockReaderRawBM25Event GraphFlat SSGSVL
Block 1GPT-5.6-Sol0.64690.59380.50630.98130.9922
Block 1GPT-5.50.72500.69690.51410.98750.9969
Block 2GPT-5.6-Sol0.64380.60470.51560.96880.9922
Block 2GPT-5.50.73910.71720.52970.98751.0000

Thus hosted Event Graph is 0.5063–0.5297, while hosted compiled views are 0.9688–1.0000. The hosted Raw rates are 0.6438–0.7391. The comparison shell, legal actions, output contract, and registered deterministic decoding protocol are matched within each evaluation, but hosted and open-weight readers differ in checkpoint, tokenizer/chat template, instruction following, and serving stack. Consequently, gain magnitudes are baseline-dependent and should be interpreted together with absolute condition rates; they are not directly comparable as a ranking of model ability.

Q3. Do Forge’s rules generalize beyond the benchmark templates?

“Insufficient evidence of external generalizability. Although the paper reports additional TaskSpecs and source-level controls, the evaluation remains based on predefined validity mechanisms and synthetic generators. The reported gains show that readers benefit from supplied decision-relevant states, but do not establish whether the state-construction rules transfer reliably to independently sourced histories within the same mechanisms. This limits the demonstrated external applicability of SciStateForge.”

“Can Forge’s rules generalize beyond the benchmark’s templates? Which rules are template-specific, and which are shared across tasks?”

Response. We agree that the current evidence does not establish random-sample external validity on natural laboratory records. It supports a narrower claim: the same five validity interfaces—search-family/stopping, disclosure timing, construct comparability, absence mechanism, and prospective relevance—can be instantiated across controlled scientific surfaces and audited with the same intervention/scoring contract. Domain-specific fields and event language vary; the validity interface and paired intervention logic are shared.

The completed controlled source-level transfer uses 10 source-style episodes, original/counterfactual pairs, two readers, and 80 rows. RAW_FULL validity is 0.750; GOLD_FLAT validity is 0.900. Pair relation accuracy is GOLD_FLAT 0.800, RAW_FULL Sol 0.600, and RAW_FULL Terra 0.500. This is evidence for controlled source-level transfer, not natural-world external validation.

The completed blind audit also shows where the natural-language history is underspecified: model–oracle agreement is 0.6438, ambiguity is 0.7375, and mechanism agreement ranges from 0.4688 (Inferential) to 0.9062 (Absence). We will not present this model audit as human expert validation.

The design-version writer scope check covers 32 extension-catalog tasks, six design families absent from the earlier 32-task base catalog, two seeds, three writer updates, and two reader sides (64 worlds / 128 side rows). These tasks also occur in the primary 64-task reader matrix; “held-out” is relative to the earlier base catalog. Parse rate was 1.0000, validity 0.5234, normalized reference-schema node recall 0.0677, relation recall 0.0000, and required-relation recall 0.0000. The writer processes the original history once; a deterministic registered patch then modifies matching targets in its inferred state for the counterfactual reader call. This is not a second learned ingestion trajectory. Strict pair accuracy was 9/64 (0.1406), requiring both valid decisions and the correct oracle flip. A patch could be applied in 53/64 states (0.8281); the flag is duplicated on both side rows. Stale-state suppression is NA, since there is no superseded-version intervention. This prompt-specific extension check leaves learned state construction open.

The two-annotator human audit is complete: over 40 blinded cases, A / B oracle agreement was 0.9500 / 1.0000, action agreement 0.9500, and ambiguity-label agreement 0.9000. All cited event IDs existed, verifying ID integrity rather than evidence-selection correctness. Both action disagreements and all four A ambiguity flags were Inferential cases, with missing authorization selected.

We appreciate the reviewer's emphasis on external applicability. The added source-level, design-version, and human-audit results provide bounded evidence about transfer and rule readability, while we explicitly acknowledge that they do not establish generalization to independently sampled natural laboratory records. We hope the absolute-rate comparisons, added checks, and revised limitations address each concern, and we welcome further discussion.

Thank you to the authors for the detailed and thoughtful rebuttal. I appreciate the effort put into addressing my questions, as well as the additional experiments and clarifications. These additions help clarify several aspects of the paper. However, I still feel that my main concerns have not been sufficiently addressed:

  1. Benchmark difficulty: The near-perfect accuracy of frontier models with compiled states raises questions about whether the benchmark is sufficiently challenging and useful. The additional experiments do not convincingly demonstrate its difficulty in more complex, realistic settings.
  2. Experimental results: GPT models perform substantially worse than Qwen3-14B on Event Graph, with a large drop from their own Raw Events performance. This discrepancy remains insufficiently explained.
  3. Generalizability: It remains unclear whether SciStateForge can generalize beyond predefined synthetic tasks to independently sourced scientific histories.

Overall, I appreciate the authors' efforts and the additional evidence provided in the rebuttal. Nevertheless, I do not think these results sufficiently resolve my original concerns. Therefore, I will maintain my original score.

Reviewer_XrKfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a stricter memory-evaluation problem for scientific agents: not just retrieving a correct fact from history, but deciding whether that fact is valid to use under the governing scientific state. It introduces SciValidityBench, an executable benchmark factory with TaskSpecs, programmatic oracles, interventions, and a compiler, SciStateForge, that turns event histories into explicit decision-governing state. The paper validates this setup across multiple domains and model families, showing higher valid-decision accuracy when models receive compiled scientific state instead of raw events or a generic event graph.

Critical strengths
  1. The paper identifies a real gap in existing memory evaluation and turns it into an executable benchmark object rather than a vague desideratum. Section 3.1-3.5, Equation 1-2, Table 1, and Figure 2 make the target of evaluation precise: a TaskSpec defines how public history is compiled into governing state and how a legal action is scored by a programmatic oracle. That is consequential because it separates “remembering the fact” from “using the fact validly,” which is the paper’s main conceptual contribution.

  2. The empirical design does a good job isolating what actually drives the gains. Table 3, Table 4, Figure 3, and Table 17 collectively show that compiled state consistently outperforms Raw/BM25/Event Graph, that this is not explained by longer context, and that the effect survives a direct-overlap audit. For a benchmark paper, this kind of control-heavy validation is the main reason I lean positive.

Critical weaknesses
  1. The paper’s framing overreaches from reader-side state presentation to “scientific agent memory” more broadly. The core system in Section 3.3 is a deterministic compiler that has access to the full public event history and emits the decision-governing state; the experiments in Section 4 then compare how frozen readers perform under different renderings of that state. This supports the claim that exposing compiled state helps downstream decision-making, but it does not directly show that end-to-end memory systems can build, maintain, or update this state during agent execution. That matters because the title, abstract, and introduction repeatedly frame the contribution as a benchmark for agent memory, while the main evidence is really about reader-side use of a hand-compiled memory view. A convincing resolution would be either to narrow the claims consistently to reader-side evaluation, or to include, in the main paper, at least one end-to-end memory policy that must write/update memory before being scored under the same TaskSpec contracts.

  2. The benchmark appears uneven in headroom once the scientific state is made explicit, which limits its long-term discriminative value. In Table 14, Qwen reaches 1.000 on Inferential, Blind, Construct, and Absence under compiled views, and Gemma also saturates Blind, Construct, and Absence; Table 18 shows near-perfect pair relation accuracy for Qwen and above 0.90 for Gemma. Even outside compiled views, some modules are already trivial for stronger readers, for example Qwen and Gemma are at 1.000 on Absence in multiple representations in Table 14. This weakens the benchmark’s significance a bit, because part of the observed gain may come from making the answer almost explicit on sizable parts of the suite, and it raises the question of how much headroom remains for future systems. A good fix would be to quantify task difficulty more prominently in the main paper, highlight which modules remain genuinely challenging after compilation, and discuss plans for harder variants in saturated modules.

Questions for authors
  1. [Clarification] The main experiments in Section 4 evaluate different model-visible representations produced from the full public history, while the paper is framed throughout as a benchmark for “scientific agent memory.” Should the main claim be understood as evaluating reader-side use of compiled state, or as evaluating end-to-end memory systems more broadly? If the intended scope is the former and the paper is revised to say so explicitly, that would address my main concern.

  2. [Clarification] In Table 14 and Table 18, several module-reader combinations are effectively saturated under Flat/SVL. How many TaskSpecs remain below 90% validity for the strongest reader under compiled views, and are those failures concentrated in specific modules? If a substantial subset remains challenging, I would view the headroom concern as less serious.

  3. [Clarification] The Phi results in Table 4 are unusual: $ \text{Flat} - \text{Matched Source} = -0.295$ while $ \text{SVL} - \text{Flat} = +0.488$. Can you clarify what properties differ between Matched Source, Flat, and SVL beyond token count and the presence of compiled relations, and whether for Phi this weakens the claim that the main benefit is compilation rather than serialization?

3 comments

We thank Reviewer XrKf for the positive assessment of the benchmark's conceptual focus and experimental controls, and for identifying where the framing, headroom analysis, and Phi interpretation needed greater precision. We agree that these points are important to the paper's contribution and respond to each below.

Q1. Is the paper evaluating reader-side compiled-state use or end-to-end memory systems?

“The paper’s framing overreaches from reader-side state presentation to ‘scientific agent memory’ more broadly.”

“This supports the claim that exposing compiled state helps downstream decision-making, but it does not directly show that end-to-end memory systems can build, maintain, or update this state during agent execution.”

“Should the main claim be understood as evaluating reader-side use of compiled state, or as evaluating end-to-end memory systems more broadly? If the intended scope is the former and the paper is revised to say so explicitly, that would address my main concern.”

Response. We agree with this clarification. The intended estimand is reader-side use of a deterministic, source-linked compiled state. The compiler is a reference construction used to define the target state and make interventions auditable; it is not presented as a learned memory-writing agent.

The end-to-end writer scope check makes this boundary empirical rather than rhetorical. On 60 worlds with three sequential writer updates per world, GPT-5.6-Sol achieved 0.5333 final valid decisions, but normalized reference-schema node and relation recall were both 0.0000. This single prompting configuration, under exact record matching, supports separating reader-side use from learned ingestion; it is not an impossibility result for learned writers. The revised abstract, Introduction, Section 3.3, and conclusion state this boundary.

Q2. How much headroom remains under compiled views, and where are failures concentrated?

“The benchmark appears uneven in headroom once the scientific state is made explicit, which limits its long-term discriminative value.”

“How many TaskSpecs remain below 90% validity for the strongest reader under compiled views, and are those failures concentrated in specific modules?”

Response. We re-aggregated the frozen primary ledger per TaskSpec over the ten original-world seeds.

Reader / viewTaskSpecs at 100%TaskSpecs below 90%Concentration
Qwen3-14B / Flat SSG62/642/64both Sunk Cost: SC-BIO-01, SC-SYS-01
Qwen3-14B / typed SVL60/644/64all Sunk Cost: SC-AB-01, SC-BIO-01, SC-ML-01, SC-SYS-01
Gemma-3-12B / Flat SSG50/6414/646 Inferential, 8 Sunk Cost
Gemma-3-12B / typed SVL50/6414/646 Inferential, 8 Sunk Cost

The reviewer’s saturation observation is correct. Inferential, Blind, Construct, and Absence often function as state-use diagnostics for strong readers, while Sunk Cost retains the most headroom because the correct action requires suppressing irrelevant past effort. Mistral and Phi remain substantially harder on paired relation tracking and several module/representation combinations. The main text and headroom appendix now report this heterogeneity; longer dependency chains, stale updates, and learned writing remain future extensions.

Q3. What explains the unusual Phi behavior, and does it weaken the compilation claim?

“Can you clarify what properties differ between Matched Source, Flat, and SVL beyond token count and the presence of compiled relations, and whether for Phi this weakens the claim that the main benefit is compilation rather than serialization?”

Response. The Phi result reveals a reader–serialization interaction, so it weakens a universal serialization-independence claim but not the narrower compiled-content claim. The exact rates and controls are:

Condition/contrastPhi result
Raw0.480
BM250.475
Event Graph0.478
Flat SSG0.609
Typed SVL0.570
Flat − Event Graph+0.189
Flat − Matched Source, token matched−0.295
SVL − Flat+0.488
Opaque SVL − Opaque Flat−0.024

Flat and SVL contain the same compiled nodes, attributes, and governing relations; they differ in reader-facing serialization. Matched Source has the same full-chat token count as Flat but omits compiler-derived relations. For Phi, Flat improves over Event but is 29.5 points worse than Matched Source; compilation does not universally beat complete source facts. SVL−Flat shows that Phi consumes the renderings differently; the opaque contrast shrinks to −2.4 points, consistent with lexical/format compatibility. The revised paper states that compiled views improve over Event Graph while the preferred content/serialization depends on the reader.

We appreciate the opportunity to clarify the estimand and report the remaining headroom directly. We hope the revised framing and the explicit Phi controls address the reviewer's concerns; the manuscript also identifies harder variants and learned state maintenance as future work rather than implying that the current benchmark resolves them.

AuthorsOpenReview ↗

A Friendly Follow-up on Our Rebuttal Responses

Dear Reviewer XrKf,

Thank you for asking us to distinguish using a compiled state from learning to build one. We revised the abstract and experimental setup to make clear that our main result concerns how readers use the supplied state. Following your suggestion, we also examined all 64 tasks individually. Every task on which Qwen fell below 90% with either compiled representation involved sunk costs. We now report where Phi performs better with the original source facts, which qualifies our claim about the benefit of compilation. We would be grateful to know whether the revised framing addresses your main concern and would welcome any further questions.

Best regards,

The Authors

Reviewer_HCKzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark (SciValidityBench) for testing whether an agent uses stored scientific facts appropriately given the current state of a task. The key idea is that a fact can be true and retrievable but still invalid to act on.

Critical strengths
  • The central premise of the paper, namely that correctness/retrievability of a fact does not necessarily imply validity of a decision made based on that fact, is useful and interesting.

  • The benchmark is carefully designed, with multiple domains, counterfactuals, and repair tests.

  • Experimental results show large gains when models are given compiled state rather than raw event histories.

Critical weaknesses
  • The benchmark's compiler constructs the decision-governing state, which is then used by the oracle to determine the valid action. My understanding is that the model is given a representation of this same compiled state, making it difficult to tell how much of the reported gain is due to general memory/state reconstruction as opposed to having a representation that already specifies the variables and relations used to determine the correct decision.

  • It is not clear if the reported results would carry over to settings where the evidence is spread across much longer sequences. The histories used in the paper are relatively short, so the benchmark looks like state-conditioned reasoning rather than realistic long-horizon memory.

  • It is unclear how well the 64 TaskSpecs and their rules reflect how domain experts would actually judge scientific validity.

  • The paper claims that the gains come from the governing state "rather than a particular graph serialization," but Table 4 shows substantial differences between flat records and a typed ledger for some models.

Questions for authors

Can the decision-governing state be constructed by a generic or learned mechanism, rather than by the TaskSpec-specific compiler?

Do the results apply to longer or more realistic histories?

6 comments

We thank Reviewer HCKz for recognizing the benchmark's focus on scientific-use validity and for raising important questions about compiler dependence, history length, expert judgment, and serialization. We agree these questions call for a precise statement of what the current evidence does and does not establish. We address each point below.

Q1. Can the decision-governing state be constructed by a generic or learned mechanism, rather than by the TaskSpec-specific compiler?

“The benchmark's compiler constructs the decision-governing state, which is then used by the oracle to determine the valid action. My understanding is that the model is given a representation of this same compiled state, making it difficult to tell how much of the reported gain is due to general memory/state reconstruction as opposed to having a representation that already specifies the variables and relations used to determine the correct decision.”

Response. We agree that the current primary experiment does not evaluate learned state construction. Its estimand is narrower: whether a reader can use a correctly compiled, source-linked decision state once that state is exposed. The revised manuscript states this explicitly and separates reader-side validity from learned ingestion and memory maintenance.

The representation controls isolate what is being measured. Within each comparison, the system instruction, request, legal choices, decision function, seed, and decoding are fixed; only the memory readout changes. Event Graph contains all public events and is the longest view for every reader, yet stays close to Raw Events. BM25 retrieval also does not reproduce the cross-family gain. Flat SSG and typed SVL expose identical compiled nodes, attributes, and governing relations. Matched Source exactly matches Flat SSG in full-chat token count, while Opaque Flat/SVL remove familiar state, relation, and action names.

The completed new-seed generic-ledger experiment addresses whether this generic structured memory is sufficient (Terra: 120 worlds / 480 decisions):

ViewAbsolute validity
Raw Events0.6167
Generic Ledger0.5167
Flat SSG1.0000
Typed SVL0.9917

TaskSpec-cluster paired effects are Flat−Generic +0.4833 [0.4167, 0.5417] and SVL−Generic +0.4750 [0.4083, 0.5333]; SVL−Flat is −0.0083 [−0.0250, 0.0000]. Thus this generic ledger of canonical events, sequence positions, and object mentions does not recover the benefit of explicit governing relations; it is not a test of every generic memory system.

We also ran a small end-to-end writer scope check: GPT-5.6-Sol updated a JSON state after three public events on 60 worlds (12 TaskSpecs × 5 seeds), followed by a final reader decision. The final valid-decision rate was 0.5333, while normalized reference-schema node and relation recall were both 0.0000. Matching requires reference IDs, types/endpoints, labels, and attributes; it does not assess semantic equivalence of arbitrarily renamed states. This prompt-specific result leaves learned ingestion open and is not merged into the primary reader estimate.

The independent GPT-5.6-Luna replication is also complete (480/480; parse rate 1.000): Raw Events 0.6667, Generic Ledger 0.6000, Flat SSG 1.0000, and typed SVL 1.0000. The paired effects are Flat−Generic +0.4000 [0.3083, 0.4917] and SVL−Generic +0.4000 [0.3083, 0.4917], with SVL−Flat 0.0000 [0.0000, 0.0000]. This reproduces the representation-control conclusion across two hosted readers.

Manuscript revision. The blue-marked manuscript now includes the Terra/Luna Generic Ledger controls, writer scope checks, complete Luna and Terra long-history matrices, human audit, headroom counts, and hosted absolute rates. Reader-side scope is explicit in the abstract, Introduction, Section 3.3, experiment setup, and conclusion.

Q2. Do the results apply to longer or more realistic histories?

“It is not clear if the reported results would carry over to settings where the evidence is spread across much longer sequences. The histories used in the paper are relatively short, so the benchmark looks like state-conditioned reasoning rather than realistic long-horizon memory.”

Response. We agree that the current primary suite is intentionally a controlled state-use diagnostic, not a naturalistic long-horizon benchmark. We completed two registered 480-row EXP-01P pilots under the same frozen 12-TaskSpec × 1-seed × 5-length × 4-view × 2-padding design:

RegistrationTargetDurable rowsStatus
EXP-01 full, GPT-5.6-Luna4,8004,800closed; complete matrix
EXP-01 full, GPT-5.6-Terra4,8004,800closed; complete matrix after resumable retries
EXP-01P, GPT-5.6-Terra480480closed; all cells complete
EXP-01P, GPT-5.6-Sol480480closed; all cells complete

The closed pilot results are:

Readerk=0k=4k=16k=64k=256OverallMean input tokens
GPT-5.6-Terra0.71880.76040.76040.76040.75000.750015,198.7
GPT-5.6-Sol0.75000.77080.77080.77080.76040.764615,468.6

By view, Terra gives Raw 0.5250, Event Graph 0.5000, Flat SSG 0.9917, and SVL 0.9833; Sol gives Raw 0.5667, Event Graph 0.4917, Flat SSG 1.0000, and SVL 1.0000. Parse rate is 1.000 for both readers. Irrelevant-versus-near-miss padding-pair consistency is 0.9833 for Terra and 0.9625 for Sol; both-padding validity is 0.7417 and 0.7458, respectively, over 240 complete pairs. The length curves are flat within sampling uncertainty: from k=0 to k=256, validity changes by +0.0312 for Terra and +0.0104 for Sol, despite input length rising to approximately 38.4k and 38.7k tokens. The frozen pilot ledger stores hashes, choices, and decisions but not rendered edge spans, so required-edge recall is not estimable and no proxy is imputed.

These pilots provide a closed stress test for reader-side use under longer serialized histories, but they do not establish learned incremental state maintenance or naturalistic long-horizon robustness.

We also completed the full 4,800-row EXP-01 matrix for GPT-5.6-Luna under the frozen 24-TaskSpec × 5-seed × 5-length × 4-view × 2-padding registration. Parse rate was 1.0000 throughout. Validity by view was Raw Events 0.6500, Event Graph 0.4917, Flat SSG 0.9992, and SVL 1.0000. By padding length k=0/4/16/64/256, validity was 0.7771/0.7875/0.7854/0.7896/0.7865, with mean input tokens increasing from 19.0k to 50.3k. Across 2,400 irrelevant-versus-near-miss pairs, selected-action consistency was 0.9771 and both-padding validity was 0.7738.

The same complete registration for GPT-5.6-Terra also has 4,800/4,800 unique, parse-valid rows. Terra's view rates are Raw Events 0.6042, Event Graph 0.4842, Flat SSG 0.9858, and SVL 0.9767. Pooled validity at k=0/4/16/64/256 is 0.7302/0.7604/0.7698/0.7771/0.7760, while mean input tokens rise from 6,957 to 38,777. Across 2,400 complete padding pairs, action consistency is 0.9688 and both-padding validity is 0.7471. Both hosted full matrices therefore show stable reader-side use across the registered distraction range.

Q3. Do the 64 TaskSpecs and their rules reflect expert scientific validity?

“It is unclear how well the 64 TaskSpecs and their rules reflect how domain experts would actually judge scientific validity.”

Response. The benchmark uses executable TaskSpec contracts so that validity rules, interventions, and scoring are auditable across five mechanisms and six scientific surfaces. This gives precise reproducibility, but we agree it does not substitute for independent expert annotation.

The completed blind audit was deliberately model-based, not human-expert evidence. It contains 160 judgments from GPT-5.6-Terra/Sol: parse rate 0.9688, model–oracle agreement 0.6438, ambiguity 0.7375, Sol agreement 0.6875, and Terra agreement 0.6000. Agreement by module was Absence 0.9062, Sunk Cost 0.7812, Blind 0.5312, Construct 0.5312, and Inferential 0.4688. The high ambiguity in inferential and construct cases is a reason to retain explicit TaskSpec contracts rather than claim that the natural-language history alone fully specifies validity.

The completed source-level transfer is also limited but positive: 80 rows over 10 independently generated source-style episodes, original/counterfactual pairs, and RAW_FULL/GOLD_FLAT views. Validity was 0.750 for RAW_FULL and 0.900 for GOLD_FLAT; pair relation accuracy was 0.600 for Sol RAW_FULL, 0.500 for Terra RAW_FULL, and 0.800 for GOLD_FLAT. We label this controlled source-level transfer, not natural-world validation.

Two human annotators completed the packet of 40 blinded cases (eight per mechanism, two domain surfaces per mechanism). A agreed with the executable oracle on 38/40 (0.9500), B on 40/40 (1.0000), and their actions agreed on 38/40 (0.9500). Ambiguity rates were 4/40 (0.1000) for A and 0/40 for B; ambiguity-label agreement was 36/40 (0.9000). All cited event IDs existed in their cases; this 1.0000 ID-validity rate is not evidence-selection correctness or recall. Absence, Blind, Construct, and Sunk Cost had complete action/oracle agreement; the two action disagreements were Inferential cases HEA-001 and HEA-005 (A 6/8, B 8/8). A flagged those and HEA-002/HEA-006 as ambiguous, citing missing authorization.

Q4. Does the Flat/SVL serialization difference weaken the compilation claim?

“The paper claims that the gains come from the governing state ‘rather than a particular graph serialization,’ but Table 4 shows substantial differences between flat records and a typed ledger for some models.”

Response. We agree that the original wording was too broad. The revised claim is that the common cross-reader gain is associated with exposing compiled scientific content, while the best reader-facing serialization is model-dependent.

For Phi-3.5-mini, the absolute rates are Raw 0.480, BM25 0.475, Event Graph 0.478, Flat 0.609, and SVL 0.570. The source/serialization controls are:

Phi contrastEffect
Flat − Event Graph+0.189
Flat − Matched Source, exact full-chat token match−0.295
SVL − Flat+0.488
Opaque SVL − Opaque Flat−0.024

Flat−Event remains positive, but Flat trails the complete length-matched source view by 29.5 points. Phi therefore does not support a universal compilation advantage over Matched Source. Its SVL−Flat interaction shrinks from +48.8 to −2.4 points under opaque labels, consistent with reader/format compatibility. We now distinguish the primary benefit over Event Graph from reader-dependent comparisons with complete source facts.

We appreciate the reviewer's careful attention to these scope and interpretation issues. We hope the additional controls, full long-history results, and revised claim address the concerns while making clear that learned ingestion and naturalistic long-horizon validation remain open research questions. We would be glad to clarify any point further during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewers feedback and authors response

We thank all reviewers for their careful reading and constructive feedback. Their comments helped us clarify our contribution, strengthen the evidence, and improve the manuscript.

Recognized strengths. We appreciate the reviewers' recognition of the distinction between recalling correct evidence and using it validly in scientific decisions. The reviews also highlighted the executable benchmark formulation, multi-domain design, counterfactual and repair diagnostics, and controlled empirical evidence for the benefit of compiled scientific state.

Responses to the main concerns. Our point-by-point responses address every item under Critical Weaknesses and Questions For Authors: compiler dependence and end-to-end scope; history length and realism; expert support for validity rules; task difficulty and hosted-model performance; and reader-dependent serialization effects. We now consistently define the primary evaluation as reader-side use of deterministic, source-linked compiled scientific state, and distinguish this contribution from the separate challenge of learning to construct and maintain that state.

Additional supporting evidence. We added targeted experiments and analyses, summarized in the main text with detailed protocols and results in the appendices:

  • Longer histories (Appendix H.4): Complete GPT-5.6-Luna and Terra matrices contain 4,800 decisions each, over 120 repeated worlds per reader and up to 256 distractor records. Pooled Flat/SVL validity is 97.67%-100%, compared with 60.42%-65.00% for Raw Events.
  • Generic event structure (Appendix G.2): A new-seed control with 480 decisions per reader yields 51.67%/60.00% validity for the generic ledger versus 99.17%-100% for Flat/SVL across Terra and Luna.
  • Validity audits and source-style evaluation (Appendices G.3, I.2-I.3): Two human annotators agree with the oracle on 38/40 and 40/40 blinded cases. We also report a separate model-based audit and a controlled evaluation on generated source-style episodes, where validity rises from 75% to 90% with gold compiled state.
  • State construction (Appendices I.1, I.4): Sequential state-construction and extension-design writer tests characterize construction difficulty, with final decision validity of 53.33% and 52.34%, respectively.
  • Difficulty and representation analysis (Appendices D.4, H.1): We quantify TaskSpec-level headroom, report absolute hosted-reader rates, and clarify reader-specific Flat/SVL behavior, including the Phi controls.

Revised manuscript. Changes are marked in blue for ease of review. We revised the abstract, Introduction, compiler description, experimental setup, results and conclusion. These revisions clarify the reader-side scope, explain saturation and hosted-model gains, refine the serialization claims, and integrate the supporting experiments into the relevant discussions and appendices.

We thank the reviewers again for helping improve the paper and welcome further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.