acceptodds
Under review as a conference paper at ICLR 2027

IdeaDec: Benchmarking Long-Horizon Scientific Idea Decoding in AI Research Agents

Abstract

AI research agents increasingly participate in literature analysis, hypothesis formulation, experiment design, and proposal development, making the coherence of their evolving ideas consequential to downstream research. Yet existing evaluations largely score final outputs or isolated stages and therefore do not test whether agents sustain valid scientific commitments as evidence changes. We introduce IdeaDec, a benchmark for long-horizon scientific idea decoding under controlled evidence release. IdeaDec operationalizes this problem through 137 trajectories of 7–9 ordered checkpoints, each tracking a five-component state: hypothesis, method, metric, judgment, and plan. Three disjoint subsets evaluate dependency maintenance, conflict-aware revision, and dynamic recovery, while complementary metrics assess structural completion, success on a pre-specified target operation, and five-gate whole-trajectory semantic success. Across 18 fixed model configurations in the State-Aware setting, mean structural completion is 89.05% but mean semantic success is 51.95%, a gap of 37.10 percentage points; similar aggregate scores can also conceal distinct capability profiles. Recovery has the lowest observed subset rate for every configuration under the fixed benchmark distribution. These findings motivate evaluating scientific agents through state evolution rather than format compliance or endpoint outputs alone.

Primary area:
datasets and benchmarks
Keywords:
Long-Horizon, Scientific Reasoning, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KiP3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the limitation of evaluating scientific research agents only through isolated or final outputs by testing whether their scientific commitments remain coherent as evidence is released over time.

Critical strengths

The benchmark targets a consequential and under-evaluated capability: maintaining coherent scientific commitments as evidence arrives sequentially. Representing hypotheses, methods, metrics, judgments, and plans as an explicit, evolving state makes cross-step dependencies directly inspectable.

Critical weaknesses

Lack of empirical analysis of long-horizon state evolution. Despite positioning scientific state evolution as the central contribution, Figures 4 and 5 are entirely schematic, rather than derived from actual model trajectories. The experiments primarily report aggregate success rates, without showing when errors emerge, how they propagate across dependent decisions, or whether agents correct earlier mistakes. Given the 18,900 checkpoint-level outputs, this omission substantially weakens the empirical support for the paper's central claims.

Limited benchmark scale and confounded capability comparisons. IdeaDec contains only 137 trajectories, including just 14 recovery tasks. H1, H2, and H3 also differ in task content and trajectory length, making their success rates difficult to compare directly. Consequently, the consistently lower H3 performance cannot establish that recovery is intrinsically more challenging. The small H3 sample further limits the reliability of model comparisons.

Questions for authors

Why are Figures 4 and 5 schematic rather than empirical? Can the authors provide checkpoint-level analyses from actual model trajectories, including the distribution of first failures, error propagation across dependent decisions, and subsequent self-correction? These analyses seem essential to substantiate the benchmark's motivation.

What are the inter-annotator agreement rates for the five semantic gates and H1–H3 target judgments?

Can the authors provide matched comparisons across H1, H2, and H3? How stable are the H3 results given only 14 instances?

What is the impact of explicit state tracking? How do M0–M3 compare on SC, SS, and H1–H3? Does the State-Aware setting improve semantic consistency over full-history prompting without explicit state projection?

How well does IdeaDec predict performance in realistic research workflows? Is there evidence that success on these short, predefined evidence sequences correlates with an agent's ability to maintain coherent research decisions during longer, tool-mediated scientific exploration?

1 comment

We thank the reviewer for highlighting the need for empirical evidence of state evolution. We clarify the additional findings below.

1. Empirical state evolution

The additional analysis directly addresses self-correction: 27.6% of prior errors classified as eligible for correction were corrected at a later checkpoint. This is conditional on the eligible-error set, not a success rate over all trajectories. Later correction and uninterrupted consistency are different outcomes: a model can correct an error after an earlier decisive violation has already caused Semantic Success (SS) to fail.

The analysis examined 2,466 model–trajectory combinations from the original 18-configuration evaluation. Figures 4 and 5 explain the evaluation rules using schematic paths and values; the correction statistic comes from model trajectories. It establishes that later correction occurs, but does not provide the requested first-failure distribution or propagation frequencies, which are not reported here.

2. Inter-annotator agreement

Independent human annotation yields gate-level Cohen's κ of 0.76–0.86 and pooled target κ of 0.88. Two annotators independently assessed 756 model trajectories before adjudication:

JudgmentCohen's κ
State Validity0.84
Evidence Consistency0.78
Dependency Preservation0.86
Transition Correctness0.76
Terminal Coherence0.78
Target judgment, pooled across H1–H30.88

Overall raw agreement was 88.9%. These are pre-adjudication statistics; benchmark scores use adjudicated labels. The pooled target κ does not establish separate agreement rates for H1, H2, and H3. The rubric accepts alternative scientific decisions that satisfy the available evidence, dependencies, and admissible transitions, without requiring one reference wording.

3. Recovery coverage and matched comparisons

We expanded H3 from 14 to 64 trajectories, increasing the benchmark to 187 trajectories and 1,500 checkpoints; H1 and H2 remain at 60 and 63. One outcome now changes the H3 rate by 1.56 percentage points, compared with 7.14 originally.

The additions retain the nine-checkpoint structure and evaluation contract. They cover resource or instrument unavailability, methodological invalidity, contradictory evidence, configuration changes, and recovery involving multiple dependencies. This improves coverage and resolution, without itself establishing ranking stability.

The empirical finding is that H3 has the lowest observed rate in all 18 original configurations under the fixed benchmark distribution. We have not established a matched H1–H3 comparison: content and horizon differ, so this finding identifies poor performance on these recovery cases without isolating recovery's intrinsic difficulty.

4. Contribution of explicit state tracking

Yes. In the six-model ablation, M3 achieves approximately 56% SS versus 45% for M2, an 11-percentage-point difference. Both supply full response history; M3 additionally supplies the latest structurally valid state. Structural Completion (SC) rises from 92% to 95%.

The ablation uses the same 137 original trajectories. All settings receive the seed, current instruction, and released evidence or configuration. M0 supplies no previous responses; M1 supplies the preceding response; M2 supplies full history; M3 adds the five-component state projection.

Aggregate percentages, rounded to whole numbers, are:

SettingSC (%)SS (%)H1 (%)H2 (%)H3 (%)
M08738483821
M18842544426
M29245594930
M39556636033

These are descriptive aggregate differences based on rounded values, not claims of statistical significance or uniform gains across models. M3 carries forward the model's own state, including semantic errors, without reference-state repair or evaluator annotations.

5. Predictive validity

Controlled capability evaluation and prediction of end-to-end research performance are distinct validation questions. Fixed evidence release lets us compare state updates given the same information, controlling differences in evidence acquisition and tool availability. This tests consistency among hypotheses, methods, metrics, judgments, and plans as information changes.

Our experiments address this controlled capability; we have not measured a correlation with longer, tool-mediated research performance. The 7–9 checkpoint tasks do not evaluate autonomous evidence acquisition, experimental execution, or tool selection. The supported contribution is a diagnostic of scientific-state consistency, without claiming validated prediction of overall research-agent success.

If these responses address your concerns, we would appreciate your reconsideration of the score.

Reviewer_7zVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ideaDec asking a question "can an AI keep its research idea consistent while new facts arrive one by one"

Specifically, the AI gets a short research story as its starting point, then in later steps it gets a new piece of evidence and is asked to rewrite its research idea, method, metrics for evaluation success, judgement, and plan.

The authors examine three things during the process:

  1. whether the AI follows a strict rule set early in process
  2. whether the AI change its idea correctly when it is presented with a new evidence that is against its current idea
  3. whether the AI can build a new plan while keeping it valid

The authors graded the AI's answers by checking:

  1. whether every step's output is in the correct format
  2. is the output correct across the whole story

Their discoveries include:

  1. most of the answers from the AI passed check 1 (89%) but not check 2 (52%). They consider this behavior being 'the answer look right but not actually right'
  2. Recover, which is about whether the AI can create a new plan without diverging from the goal, is the weakest skill of the AI tested.
Critical strengths

The idea is intuitive and easy to understand. Figures and the main results are easy to follow.

Critical weaknesses
  1. the paper didn't discuss existing previous works that are similar to their set-up like Belief-R (Wilie et al., EMNLP 2024) which presents a dataset designed to test LMs' belief revision ability when presented with new evidence
  2. Several jargons are uncommon or different from its common definition in this field or are customized metrics of unclear values or not clearly defined: trajectories, scientific idea decoding, checkpoint, static control, s-avg, drop, reliability gap, etc.
  3. The scale of the paper is relatively small: 137 items, only 14 for Recover, and one run per model
  4. Whether the evaluations are human or AI (and if so which model) is not clearly defined
Questions for authors
  1. who is evaluating the models' answers against the checks and the three skills list? A human or an LLM? And if it's an LLM, which model is it?
  2. How would you distinguish your work from existing works like Belief-R (Wilie et al., EMNLP 2024)
1 comment

We thank the reviewer for the feedback on positioning, terminology, and evaluation.

1. Who performs the evaluation?

Human evaluators judge scientific correctness; a parser checks format. No LLM judge is used. Structural Completion (SC) requires every checkpoint response to be present and parseable with the prescribed fields and order. Humans assess the five gates below and the designated capability target. Semantic Success (SS) requires SC and all five gates to pass over their specified scopes.

We independently double-annotated 756 model trajectories. Agreement was measured before adjudication:

JudgmentCohen's κ
State Validity0.84
Evidence Consistency0.78
Dependency Preservation0.86
Transition Correctness0.76
Terminal Coherence0.78
Designated target operation, pooled across H1–H30.88

Raw agreement was 88.9%; disagreements were then adjudicated. This supports consistent rubric application on the annotated sample.

Alternative methods or recovery plans pass if they respect the available evidence, objective, and remaining constraints. Evaluator annotations never enter the model's context or repair its state. We will clarify these scoring roles in the main text.

2. Relationship to Belief-R

IdeaDec adds evaluation of dependent research decisions to belief revision. Belief-R (Wilie et al., EMNLP 2024) evaluates whether models update or maintain conclusions after new premises arrive, using two successive stages and separate update/maintenance accuracies. These are established evaluation targets; we will cite this directly relevant work.

The distinction is what constitutes success. Belief-R evaluates the inferred conclusion. IdeaDec requires a hypothesis, method, metric, judgment, and plan across 7–9 checkpoints, preserving their dependencies. A correct hypothesis revision can still fail if the dependent method, metric, or plan remains incompatible.

For example, after evidence invalidates a methodological assumption, changing the hypothesis while retaining an experimental plan that relies on it is an unsuccessful update. Recovery additionally requires a feasible continuation preserving the objective and unaffected constraints. Whole-trajectory scoring retains earlier decisive violations even if the final answer is plausible.

We will state the contribution at this scope: scientific-state dependencies, revision propagation, and recovery across multiple updates.

3. Terminology and metrics

The terminology describes the evaluation unit: a trajectory is a complete update sequence, and a checkpoint is one step. “Scientific idea decoding” means updating a scientific state from sequential evidence; we will prefer “state update.” The ten static controls are non-sequential auxiliary cases excluded from the main denominators.

SC asks whether the sequence is structurally complete; SS asks whether it is also semantically valid; H1–H3 assess the designated maintenance, revision, or recovery operation. The derived summaries serve narrower descriptive purposes:

  • RG = SC − SS, in percentage points: the share of all trajectories that are structurally complete but semantically unsuccessful.
  • S-Avg = (SC + SS) / 2, an auxiliary average of the two rates.
  • Drop = H1 − H3, in percentage points: a cross-subset contrast, not within-trajectory deterioration or proof of intrinsic difficulty.

Low completion can also produce a small RG. We will center interpretation on SC, SS, and capability-specific rates, define terms at first use, and treat derived summaries as descriptive.

4. Benchmark size and repeated evaluation

We expanded recovery coverage and repeated evaluation. Adding 50 H3 trajectories increases H3 from 14 to 64 and the benchmark to 187 trajectories and 1,500 checkpoints, from 137 and 1,050. H1 and H2 remain at 60 and 63. New nine-checkpoint cases cover unavailable resources or instruments, invalid methods, contradictory evidence, configuration changes, and multiple-dependency recovery.

One H3 outcome now changes its rate by 1.56 percentage points instead of 7.14, improving resolution. Original results retain the 137-trajectory denominator. Differences in content and horizon mean H1–H3 diagnose separate capabilities rather than establish intrinsic difficulty.

We completed three independent runs for each of six models on the original benchmark, extending evaluation beyond one run for those models. The H3 expansion increases scenario coverage.

If these responses address your concerns, we would appreciate your reconsideration of the score. If concerns remain, we would welcome specific feedback on the claim, experimental comparison, or methodological issue requiring further clarification.

Reviewer_TzGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IdeaDec, a benchmark for evaluating AI research agents' ability to maintain and update scientific ideas as new evidence emerges. The benchmark consists of 137 trajectories and 1,050 checkpoints, covering three capabilities: dependency maintenance, conflict-aware revision, and dynamic recovery. The authors evaluate 18 model configurations using structural completion, target-operation success, and whole-trajectory semantic success. Results reveal a substantial gap between structural completion (89.05%) and semantic success (51.95%), with dynamic recovery showing the lowest success rate. The findings highlight the importance of evaluating scientific reasoning consistency throughout the research process rather than focusing solely on final outputs.

Critical strengths
  1. Broad model coverage with appropriately qualified empirical reporting. The evaluation includes 18 model configurations and reports both aggregate outcomes and separate capability profiles.
  2. The paper addresses an important and relatively underexplored problem: evaluating whether AI research agents can maintain scientific consistency as new evidence emerges.
Critical weaknesses

1. Unclear effectiveness of whole-trajectory evaluation. The reported whole-trajectory success counts match the aggregated target-operation success counts across all models, raising questions about the additional value of this evaluation metric.

2. Limited benchmark scale. The benchmark contains only 137 trajectories, including just 14 recovery cases, which may limit the reliability and generalizability of the results.

3. Limited real-world applicability. The benchmark uses predefined trajectories and fixed evidence sequences, which may not fully capture the complexity and open-ended nature of real scientific research.

Questions for authors
  1. Why do the whole-trajectory success counts exactly match the aggregated target-operation success counts across all models? What additional information does this metric provide?

  2. How were the 137 trajectories selected, and how well do they represent diverse scientific reasoning scenarios?

  3. How well do the predefined trajectories and fixed evidence sequences reflect realistic scientific research workflows?

  4. How is the consistency of semantic evaluation ensured, especially when multiple scientifically valid decisions are possible?

2 comments

1. Why do whole-trajectory and target-operation success counts coincide?

The additional ablation shows that SS identifies unsuccessful trajectories that pass the designated operation. Target success is necessary for Semantic Success (SS); SS additionally requires structural completion and all five semantic gates across their specified scopes. A correct target response therefore does not suffice when another required response is missing or the terminal plan is invalid.

In the original Table 3, the two counts coincide, so that table does not demonstrate additional rejections by SS. We distinguish this original finding from the additional context ablation:

EvaluationPooled target success (%)SS (%)
Original 18 configurations, M351.9551.95
Additional six models, M2≈51.4≈45
Additional six models, M3≈58.6≈56

Pooled target success is (60 × H1 + 63 × H2 + 14 × H3) / 137. All evaluations use the original 137 trajectories. Ablation values derive from rounded rates; model populations differ, so comparisons concern the measures within each row.

The additional evaluations show approximately 6.4 and 2.6 percentage points of separation. Target rates diagnose the designated operation; SS checks whether the complete sequence also meets the evaluation requirements. These results establish the practical distinction between the outcomes in the added evaluation settings.

2. How were trajectories constructed, and how representative are they?

The benchmark provides structured coverage of three research-state operations. The original 60 H1, 63 H2, and 14 H3 trajectories test dependency maintenance, conflict-aware revision, and dynamic recovery over 7, 8, and 9 checkpoints respectively.

Each starts from a source-grounded seed specifying an objective, evidence, hypothesis, method, and constraints. One annotator constructs the evidence schedule, dependencies, admissible transitions, and target; a second reviews these before the contract is frozen and applied across models (Section 3.1; Appendix A).

We added 50 H3 trajectories, increasing H3 from 14 to 64 and the benchmark from 137 trajectories / 1,050 checkpoints to 187 trajectories / 1,500 checkpoints. H1 and H2 are unchanged. New cases retain nine checkpoints and cover resource or instrument unavailability, methodological invalidity, contradictory evidence, constraint or configuration changes, and recovery involving multiple dependencies.

The expansion broadens recovery coverage and reduces the change per H3 outcome from 7.14 to 1.56 percentage points. Original results retain their denominators. This improves diagnostic coverage without establishing statistical representativeness or ranking stability. Since H1–H3 differ in content and horizon, their rates are not matched estimates of intrinsic difficulty.

3. How do fixed sequences reflect real research workflows?

Fixed evidence controls the information supplied, not the scientific decision. Alternative methods and plans pass when they satisfy the evidence, objective, and remaining constraints. Models carry forward their own latest structurally valid states, including scientific errors, without reference repair. This permits inspection of whether new findings change affected commitments while valid dependencies remain enforced.

The schedule gives models comparable information, separating state updating from differences in autonomous evidence acquisition. It captures a component of research workflows. Autonomous search, tool use, and experimental execution are outside the 7–9-checkpoint setting; predictive validity for longer workflows remains unestablished.

4. How is semantic evaluation consistent when several decisions are valid?

Consistency comes from a shared rubric of admissible outcomes, not a unique reference answer. Human annotators assess the evidence scope, dependencies, and required transitions in the frozen contract. A valid alternative can pass; later evidence does not retrospectively invalidate a decision justified when made.

Two annotators independently assessed 756 model trajectories on all five gates and the designated target before adjudication:

JudgmentCohen's κ
State Validity0.84
Evidence Consistency0.78
Dependency Preservation0.86
Transition Correctness0.76
Terminal Coherence0.78
Target judgment, pooled across H1–H30.88

Overall raw agreement was 88.9%; disagreements were subsequently adjudicated. These pre-adjudication results support consistent rubric application while allowing alternative valid decisions. The target κ is pooled, not a separate estimate for each capability. We will clarify this scope and the human scoring procedure.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the clarification of the SS metric, the expansion of H3 trajectories, and the additional inter-annotator agreement analysis.

While these additions help clarify several aspects of the evaluation, I believe my main concerns regarding the benchmark's representativeness and its generalizability to realistic research workflows remain only partially addressed. Therefore, although I acknowledge the improvements, I do not find them sufficient to change my overall assessment, and I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.