- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SciValidityBench to evaluate whether agents use remembered scientific evidence appropriately, and SciStateForge to represent experimental histories as explicit scientific states. Experiments across four model families report improved decision accuracy over baseline memory representations.
Critical strengths
The paper proposes an interesting task that distinguishes correctly recalling evidence from using it appropriately in scientific decisions, offering a useful perspective for evaluating scientific agents.
Critical weaknesses
Limited task difficulty and realism. The benchmark uses short synthetic histories, averaging only three public events per world. Qwen3-14B already achieves 99.1% accuracy with Flat SSG, leaving little headroom in this setting. The benchmark may therefore offer limited challenge to frontier models when compiled states are provided.
Insufficient evidence of external generalizability. Although the paper reports additional TaskSpecs and source-level controls, the evaluation remains based on predefined validity mechanisms and synthetic generators. The reported gains show that readers benefit from supplied decision-relevant states, but do not establish whether the state-construction rules transfer reliably to independently sourced histories within the same mechanisms. This limits the demonstrated external applicability of SciStateForge.
Questions for authors
-
Is the benchmark sufficiently challenging for frontier models? What absolute accuracy do frontier models achieve using only raw event histories, without SciStateForge? Can they already achieve high accuracy, and how much additional benefit does Forge provide?
-
Why are the GPT gains in Figure 3B substantially larger than those for Qwen? The reported gains of 47.0–48.6 percentage points imply Event Graph accuracies no higher than approximately 51.4%–53.0%, whereas Qwen3-14B achieves 79.4%. Please report the hosted models’ absolute accuracies under each condition and explain this discrepancy, including whether the evaluation settings are comparable.
-
Can Forge’s rules generalize beyond the benchmark’s templates? Which rules are template-specific, and which are shared across tasks? Within the five validity mechanisms studied, is there evidence that the same core rules apply to independently sourced scientific histories?
6 comments
Q3. Do Forge’s rules generalize beyond the benchmark templates?
“Insufficient evidence of external generalizability. Although the paper reports additional TaskSpecs and source-level controls, the evaluation remains based on predefined validity mechanisms and synthetic generators. The reported gains show that readers benefit from supplied decision-relevant states, but do not establish whether the state-construction rules transfer reliably to independently sourced histories within the same mechanisms. This limits the demonstrated external applicability of SciStateForge.”
“Can Forge’s rules generalize beyond the benchmark’s templates? Which rules are template-specific, and which are shared across tasks?”
Response. We agree that the current evidence does not establish random-sample external validity on natural laboratory records. It supports a narrower claim: the same five validity interfaces—search-family/stopping, disclosure timing, construct comparability, absence mechanism, and prospective relevance—can be instantiated across controlled scientific surfaces and audited with the same intervention/scoring contract. Domain-specific fields and event language vary; the validity interface and paired intervention logic are shared.
The completed controlled source-level transfer uses 10 source-style episodes, original/counterfactual pairs, two readers, and 80 rows. RAW_FULL validity is 0.750; GOLD_FLAT validity is 0.900. Pair relation accuracy is GOLD_FLAT 0.800, RAW_FULL Sol 0.600, and RAW_FULL Terra 0.500. This is evidence for controlled source-level transfer, not natural-world external validation.
The completed blind audit also shows where the natural-language history is underspecified: model–oracle agreement is 0.6438, ambiguity is 0.7375, and mechanism agreement ranges from 0.4688 (Inferential) to 0.9062 (Absence). We will not present this model audit as human expert validation.
The design-version writer scope check covers 32 extension-catalog tasks, six design families absent from the earlier 32-task base catalog, two seeds, three writer updates, and two reader sides (64 worlds / 128 side rows). These tasks also occur in the primary 64-task reader matrix; “held-out” is relative to the earlier base catalog. Parse rate was 1.0000, validity 0.5234, normalized reference-schema node recall 0.0677, relation recall 0.0000, and required-relation recall 0.0000. The writer processes the original history once; a deterministic registered patch then modifies matching targets in its inferred state for the counterfactual reader call. This is not a second learned ingestion trajectory. Strict pair accuracy was 9/64 (0.1406), requiring both valid decisions and the correct oracle flip. A patch could be applied in 53/64 states (0.8281); the flag is duplicated on both side rows. Stale-state suppression is NA, since there is no superseded-version intervention. This prompt-specific extension check leaves learned state construction open.
The two-annotator human audit is complete: over 40 blinded cases, A / B oracle agreement was 0.9500 / 1.0000, action agreement 0.9500, and ambiguity-label agreement 0.9000. All cited event IDs existed, verifying ID integrity rather than evidence-selection correctness. Both action disagreements and all four A ambiguity flags were Inferential cases, with missing authorization selected.
We appreciate the reviewer's emphasis on external applicability. The added source-level, design-version, and human-audit results provide bounded evidence about transfer and rule readability, while we explicitly acknowledge that they do not establish generalization to independently sampled natural laboratory records. We hope the absolute-rate comparisons, added checks, and revised limitations address each concern, and we welcome further discussion.
Thank you to the authors for the detailed and thoughtful rebuttal. I appreciate the effort put into addressing my questions, as well as the additional experiments and clarifications. These additions help clarify several aspects of the paper. However, I still feel that my main concerns have not been sufficiently addressed:
- Benchmark difficulty: The near-perfect accuracy of frontier models with compiled states raises questions about whether the benchmark is sufficiently challenging and useful. The additional experiments do not convincingly demonstrate its difficulty in more complex, realistic settings.
- Experimental results: GPT models perform substantially worse than Qwen3-14B on Event Graph, with a large drop from their own Raw Events performance. This discrepancy remains insufficiently explained.
- Generalizability: It remains unclear whether SciStateForge can generalize beyond predefined synthetic tasks to independently sourced scientific histories.
Overall, I appreciate the authors' efforts and the additional evidence provided in the rebuttal. Nevertheless, I do not think these results sufficiently resolve my original concerns. Therefore, I will maintain my original score.
We thank Reviewer oAcq for highlighting the importance of absolute raw-history baselines, comparable hosted-reader results, and evidence beyond the benchmark templates. We agree that the original paper needed to state its realism and external-validity limits more directly. The responses below provide the requested rates and describe the additional controlled checks without treating them as natural-laboratory validation.
Q1. Is the benchmark challenging enough for frontier models, and what are the raw-history baselines?
Response. We agree that the current suite is short and controlled, and we now report absolute rates rather than only gains. The open-weight matrix is:
Qwen’s Flat−Raw gain is +0.193 and Flat−Event gain is +0.197; the high compiled-view score does not imply that the raw-history problem is solved. The headroom analysis above also shows that the remaining strong-reader failures are concentrated in Sunk Cost, while weaker readers remain challenged across multiple mechanisms.
Two full matrices now cover 4,800 decisions over 120 worlds each. For Luna, Raw/Event/Flat/SVL validity is 0.6500/0.4917/0.9992/1.0000; for Terra it is 0.6042/0.4842/0.9858/0.9767. Pooled validity at k=0/4/16/64/256 is 0.7771/0.7875/0.7854/0.7896/0.7865 for Luna and 0.7302/0.7604/0.7698/0.7771/0.7760 for Terra. Each matrix has parse rate 1.0000 and 2,400 complete pairs; action consistency is 0.9771 for Luna and 0.9688 for Terra. Distractors surround already compiled views, with near-miss status explicitly unrelated. The smaller Sol pilot gives a compatible additional check, with validity of 0.7500/0.7708/0.7708/0.7708/0.7604 over the same five k values.
Q2. Why are the hosted GPT gains larger than the Qwen gains?
Response. The hosted gains are real and arise mainly from a lower uncompiled Event Graph baseline. The hosted absolute rates are:
Thus hosted Event Graph is 0.5063–0.5297, while hosted compiled views are 0.9688–1.0000. The hosted Raw rates are 0.6438–0.7391. The comparison shell, legal actions, output contract, and registered deterministic decoding protocol are matched within each evaluation, but hosted and open-weight readers differ in checkpoint, tokenizer/chat template, instruction following, and serving stack. Consequently, gain magnitudes are baseline-dependent and should be interpreted together with absolute condition rates; they are not directly comparable as a ranking of model ability.