- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the limitation of evaluating scientific research agents only through isolated or final outputs by testing whether their scientific commitments remain coherent as evidence is released over time.
Critical strengths
The benchmark targets a consequential and under-evaluated capability: maintaining coherent scientific commitments as evidence arrives sequentially. Representing hypotheses, methods, metrics, judgments, and plans as an explicit, evolving state makes cross-step dependencies directly inspectable.
Critical weaknesses
Lack of empirical analysis of long-horizon state evolution. Despite positioning scientific state evolution as the central contribution, Figures 4 and 5 are entirely schematic, rather than derived from actual model trajectories. The experiments primarily report aggregate success rates, without showing when errors emerge, how they propagate across dependent decisions, or whether agents correct earlier mistakes. Given the 18,900 checkpoint-level outputs, this omission substantially weakens the empirical support for the paper's central claims.
Limited benchmark scale and confounded capability comparisons. IdeaDec contains only 137 trajectories, including just 14 recovery tasks. H1, H2, and H3 also differ in task content and trajectory length, making their success rates difficult to compare directly. Consequently, the consistently lower H3 performance cannot establish that recovery is intrinsically more challenging. The small H3 sample further limits the reliability of model comparisons.
Questions for authors
Why are Figures 4 and 5 schematic rather than empirical? Can the authors provide checkpoint-level analyses from actual model trajectories, including the distribution of first failures, error propagation across dependent decisions, and subsequent self-correction? These analyses seem essential to substantiate the benchmark's motivation.
What are the inter-annotator agreement rates for the five semantic gates and H1–H3 target judgments?
Can the authors provide matched comparisons across H1, H2, and H3? How stable are the H3 results given only 14 instances?
What is the impact of explicit state tracking? How do M0–M3 compare on SC, SS, and H1–H3? Does the State-Aware setting improve semantic consistency over full-history prompting without explicit state projection?
How well does IdeaDec predict performance in realistic research workflows? Is there evidence that success on these short, predefined evidence sequences correlates with an agent's ability to maintain coherent research decisions during longer, tool-mediated scientific exploration?
We thank the reviewer for highlighting the need for empirical evidence of state evolution. We clarify the additional findings below.
1. Empirical state evolution
The additional analysis directly addresses self-correction: 27.6% of prior errors classified as eligible for correction were corrected at a later checkpoint. This is conditional on the eligible-error set, not a success rate over all trajectories. Later correction and uninterrupted consistency are different outcomes: a model can correct an error after an earlier decisive violation has already caused Semantic Success (SS) to fail.
The analysis examined 2,466 model–trajectory combinations from the original 18-configuration evaluation. Figures 4 and 5 explain the evaluation rules using schematic paths and values; the correction statistic comes from model trajectories. It establishes that later correction occurs, but does not provide the requested first-failure distribution or propagation frequencies, which are not reported here.
2. Inter-annotator agreement
Independent human annotation yields gate-level Cohen's κ of 0.76–0.86 and pooled target κ of 0.88. Two annotators independently assessed 756 model trajectories before adjudication:
Overall raw agreement was 88.9%. These are pre-adjudication statistics; benchmark scores use adjudicated labels. The pooled target κ does not establish separate agreement rates for H1, H2, and H3. The rubric accepts alternative scientific decisions that satisfy the available evidence, dependencies, and admissible transitions, without requiring one reference wording.
3. Recovery coverage and matched comparisons
We expanded H3 from 14 to 64 trajectories, increasing the benchmark to 187 trajectories and 1,500 checkpoints; H1 and H2 remain at 60 and 63. One outcome now changes the H3 rate by 1.56 percentage points, compared with 7.14 originally.
The additions retain the nine-checkpoint structure and evaluation contract. They cover resource or instrument unavailability, methodological invalidity, contradictory evidence, configuration changes, and recovery involving multiple dependencies. This improves coverage and resolution, without itself establishing ranking stability.
The empirical finding is that H3 has the lowest observed rate in all 18 original configurations under the fixed benchmark distribution. We have not established a matched H1–H3 comparison: content and horizon differ, so this finding identifies poor performance on these recovery cases without isolating recovery's intrinsic difficulty.
4. Contribution of explicit state tracking
Yes. In the six-model ablation, M3 achieves approximately 56% SS versus 45% for M2, an 11-percentage-point difference. Both supply full response history; M3 additionally supplies the latest structurally valid state. Structural Completion (SC) rises from 92% to 95%.
The ablation uses the same 137 original trajectories. All settings receive the seed, current instruction, and released evidence or configuration. M0 supplies no previous responses; M1 supplies the preceding response; M2 supplies full history; M3 adds the five-component state projection.
Aggregate percentages, rounded to whole numbers, are:
These are descriptive aggregate differences based on rounded values, not claims of statistical significance or uniform gains across models. M3 carries forward the model's own state, including semantic errors, without reference-state repair or evaluator annotations.
5. Predictive validity
Controlled capability evaluation and prediction of end-to-end research performance are distinct validation questions. Fixed evidence release lets us compare state updates given the same information, controlling differences in evidence acquisition and tool availability. This tests consistency among hypotheses, methods, metrics, judgments, and plans as information changes.
Our experiments address this controlled capability; we have not measured a correlation with longer, tool-mediated research performance. The 7–9 checkpoint tasks do not evaluate autonomous evidence acquisition, experimental execution, or tool selection. The supported contribution is a diagnostic of scientific-state consistency, without claiming validated prediction of overall research-agent success.
If these responses address your concerns, we would appreciate your reconsideration of the score.