acceptodds
Under review as a conference paper at ICLR 2027

ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment

Abstract

AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, while generation-time retrieval and tools are restricted to the cutoff-aligned evidence package. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Agent-based methods improve traceability over Hybrid RAG, while their gains in future-target alignment over native LLMs are modest and task dependent. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. Task-specific checks between evidence and final decision reduce drift and improve future-target alignment. As research agents move from assisting known tasks to shaping open research directions, ForeSci offers a first controlled step toward measuring whether their judgement can be trusted to guide that future.

Primary area:
datasets and benchmarks
Keywords:
Forward-looking scientific judgment, Research-agent evaluation, Temporal leakage control, Evidence-grounded forecasting, Future-target alignment, Evidence-decision drift, Scientific decision making, LLM benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MJ3TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ForeSci contains 500 tasks spanning four AI domains and four decision types: forecasting directions, identifying bottlenecks and opportunities, prioritizing research, and positioning work for venues. Systems receive historically restricted literature; later developments provide evaluation targets. The evaluation distinguishes prediction factuality, future-target alignment, evidence traceability, and reviewer persuasiveness. Five system configurations are evaluated across four backbones. Agent workflows improve traceability more consistently than future-target alignment. The authors diagnose evidence-to-decision drift and test task-specific decision checks on disjoint confirmation tasks.

Critical strengths
  • The evaluation is substantial with broad coverage, there are five system configurations across four backbones with human check; while there are also human checks on construction, human validation of the persuasiveness predictor, and held-out intervention experiments.
  • The idea is interesting to move beyond information retrieval and executable implementation to research judgment.
Critical weaknesses
  • There is some weakness in target validity, a later popular direction is not necessarily the best decision a resource-constrained team could have made at the earlier date, and there may be reasonable alternatives that don't match the trends.

  • Embedding-based FTA may reward broad or topically similar answers without establishing prediction correctness.

Questions for authors
  • How did the human audit handle answers that disagree with the eventual research trajectory but are defensible under the task's historical constraints? Were such alternatives admitted to the targets, and is there existing agreement or adjudication evidence for these cases?

  • Is there any example of existing validation that shows that Direction and Bottleneck(reference-guided) FTA distinguishes a correct prediction(or not) from a negated, overly broad, or merely topically similar answer?

4 comments
AuthorsOpenReview ↗

Response to Reviewer MJ3T (1/2)

Thank you for recognizing ForeSci's coverage and its focus on forward-looking research decisions beyond retrieval and implementation. Your two questions concern the validity of the future targets and the discrimination provided by reference-guided FTA. We address them separately with the target audit and controlled tests of the original scorers.

Q1. How are historically defensible alternatives handled, and what agreement or adjudication evidence supports this?

We would like to clarify this by checking both the construction of the frozen future targets and the audit of model-generated answers. The hidden future targets are constructed and frozen before the primary model evaluation, with acceptable alternative routes identified during task construction. Separately, generated answers are audited for historical defensibility and later validation. A generated answer that is historically defensible but not supported by the later validation corpus retains its historical-quality label, but it is not counted as future-target covered. If the audit reveals a historically defensible, later-supported route absent from the frozen target set, we test that route only in a development-subset target-completion sensitivity analysis; we do not use model outputs to rewrite the primary ground truth. The benchmark therefore separates whether a proposal was reasonable given what was known at the cutoff from whether later evidence supports its future target.

We implement this distinction in two analyses. First, in the ground-truth target construction, the hidden targets include acceptable alternatives rather than only one reference wording. Across the audited tasks, all 125 Direction tasks contain acceptable neighboring directions (mean 4.43 per task); 94/125 Bottleneck tasks contain acceptable bottleneck variants and 86/125 contain acceptable opportunity variants. Each decision slot is scored against its best matching accepted alternative, so a valid route is not rejected merely because it uses different terminology or reaches the target through a different path.

Second, for the model-generated answers, we asked three annotators to assess 160 answers from 80 development tasks. They first judged historical reasonableness, evidence support, and feasibility using only information available at the cutoff. Only after those judgments were fixed did they inspect the later validation material and judge future coverage against the frozen targets. Initial agreement before discussion or adjudication was 0.784 for historical reasonableness, 0.812 for support, 0.746 for feasibility, and 0.762 for future coverage. After the initial labels were recorded, disagreements were discussed and adjudicated; four unresolved cases were retained separately.

The audit distinguishes the alternatives relevant to your question:

Audit categoryAnswers (of 160)Treatment
Historically reasonable, future-supported, and covered60Recognized by the existing target slots
Historically reasonable and future-supported, but omitted from the targets6Tested as additions only in a development-subset target-completion sensitivity analysis; not added to the frozen primary targets
Historically reasonable, but not supported by the future validation corpus12Retained as historically reasonable; absence of future support is reported separately
Partially reasonable or unreasonable78Classified from the historical-stage assessment
Unresolved4Retained as unresolved

Adding the six supported alternatives only in the development-subset sensitivity analysis leaves the method ranking unchanged on these 160 answers. The primary target set and primary results are therefore unchanged. The audit supports the intended interpretation of the benchmark: later popularity is not treated as the definition of a good decision, while historical defensibility is reported as a separate property rather than being used to redefine future-target correctness.

Taken together, the target audit and the two-stage human assessment show that ForeSci can credit reasonable alternative routes while keeping historical quality separate from future validation. This is the distinction needed for evaluating foresight rather than hindsight.

Revision: Appendix D.1 (p. 23) specifies accepted target variants and best-variant matching. Appendix I.1 (pp. 44–45, Table A28) reports the two-stage protocol, initial agreement, and adjudication counts. Appendix I.2 (p. 45, Table A30 and accompanying text) separates the audit categories and documents the target-completion sensitivity check.

AuthorsOpenReview ↗

Response to Reviewer MJ3T (2/2)

Q2. Does reference-guided FTA distinguish correct predictions from negated, overly broad, or merely topical answers?

We test this directly by holding the task, length, format, and style fixed while changing only the decision. Each task has a correct reference, a meaning-preserving paraphrase, an explicit negation, a causal reversal, an overly broad answer, and a merely topical answer that does not make the required decision. The analysis covers 30 Direction and 30 Bottleneck tasks, or 360 texts in total.

The table reports the difference between the correct reference and each variant under the original frozen bge-m3 FTA scorer. A positive value means that the scorer gives the correct reference a higher score; the final two columns count how often that happens on the same task.

VariantDirection: ΔFTA [95% CI]Bottleneck: ΔFTA [95% CI]Direction winsBottleneck wins
Explicit negation+0.046 [+0.024, +0.068]+0.039 [+0.018, +0.060]23/3022/30
Overly broad+0.208 [+0.176, +0.240]+0.207 [+0.174, +0.240]29/3029/30
Merely topical+0.330 [+0.292, +0.368]+0.322 [+0.285, +0.359]30/3030/30
Causal reversal+0.064 [+0.041, +0.087]+0.062 [+0.039, +0.085]25/3024/30

Meaning-preserving paraphrases have much smaller average differences: +0.007 [−0.008, +0.022] in Direction and +0.005 [−0.011, +0.021] in Bottleneck. The contrast supports sensitivity to the tested decision errors beyond wording changes or topical overlap.

Complementary Claim Verification. Under explicit negation, Fact drops from 0.904 to 0.214 in Direction and from 0.936 to 0.186 in Bottleneck; under causal reversal, it falls to 0.265 and 0.228, respectively.

Across all tested error types, the scorer gives the correct reference a higher FTA on average in both task families, and every paired interval excludes zero. The separation is largest for overly broad and merely topical answers. Fact provides a sharper contradiction signal for negation and causal reversal, which is why the revised evaluation reports both measures.

The added OW-ECTA analysis extends this result by making claim support and required-decision coverage separately inspectable. The controlled test above uses the original frozen scorers; the OW-ECTA semantic challenge and human calibration are reported separately in Appendices J.2 and H.3.

Taken together, the controls show that reference-guided FTA is not simply rewarding shared topic words: it is relatively stable under meaning-preserving wording changes and decreases when the answer negates, reverses, broadens, or omits the decision.

Revision: Appendix H.2 (pp. 39–40, Table A19) reports the controlled variants, original scoring pipelines, paired intervals, and per-family discrimination results. Section 4.1 (p. 7) and Appendix H.1 (p. 39) introduce OW-ECTA; Appendices H.3 (pp. 40–41) and J.2 (p. 49) report its component validation separately.

Thank you again for your assessment. We are following up on the two specific questions about target validity and reference-guided FTA.

For defensible alternatives, our two-stage audit evaluates historical reasonableness before revealing future evidence. Reasonable proposals without observed future support retain their historical-quality judgment. For six future-supported answers omitted from the targets, admitting the missing alternatives in the audited development subset leaves the method ranking unchanged on the 160 audited answers (Q1; Appendix I.2).

For FTA discrimination, we directly test the original frozen scorer on 30 Direction and 30 Bottleneck tasks. Correct references receive higher mean FTA than negated, overly broad, merely topical, and causally reversed variants in both families, with each paired 95% interval excluding zero. Meaning-preserving paraphrases have much smaller average differences, while Fact provides a complementary contradiction signal (Q2; Appendix H.2).

Do these clarifications and tests address your two questions? In particular, we would appreciate your view on the separation of historical defensibility from future coverage and the evidence for the original FTA scorer. We would be happy to clarify any remaining concern.

Thank you for your reply. I have updated my rating.

Reviewer_QGCzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ForeSci (bench) introduces 500 temporally controlled tasks evaluating research agents ability to make forward looking decisions from historical evidence in four families of tasks: Direction Forecasting, Bottleneck Opportunity Discovery, Strategic Research Planning, and Venue Conditioned Positioning. It also assesses factuality, future-target alignment, evidence traceability, and persuasiveness, and identifies evidence<->decision drift as a recurring failure mode. Some frontier LLMs (Qwen, gpt, GLM, Gemini) with time cut off are evaluated, with evaluation harnesses of RAG and research agent.

Critical strengths

S1: Forecasting AI research is an interesting direction to go with. The ability to identify next-step AI research directions could support progress toward recursive AI self improvement, yet high quality benchmarks for this capability remain limited. I encourage the authors to continue developing this line of research.

S2: The paper clearly describes the benchmark construction, evaluation protocol, and combined human LLM auditing, as good starting points, and with clear time cutoff control.

Critical weaknesses

W1: Quality Issues: The dataset appears to only emphasize basic, insufficiently grounded question answering rather than substantive research judgment. Even the (cherry picked) illustrative examples in Figure 1 show generic LLM generated questions that could be answered largely by retrieving historical information. More importantly, the tasks are vaguely specified and lack clear answer rubrics, making it difficult to verify response quality or distinguish real research foresight of the model. Not to mention the community is expecting to see some long horizon traces more than simple QA.

W2: Reliability of human evaluation issues. Assessing AI research judgment is (from my opinion) inherently challenging and subjective. It is unclear how agreement across annotators is measured, how disagreements are resolved, and what criteria experts use to assess the novelty, feasibility, and scientific value of generated research directions from models.

W3: Evaluation harness design. The evaluated harnesses appear relatively basic, and their implementation is insufficiently specified. Manually designed RAG is not a good way to interact with frontier models. More advanced designed harnesses, or even less advanced fair harnesses (just give simple instructions) are the much better way to benchmark models.

Questions for authors

Q1 (regarding W1): Given that this is a benchmark paper, how can the authors demonstrate or improve the dataset’s quality and value to the community? I am currently worried about the quality of the data.

Q2 (regarding W2): How can the evaluation distinguish meaningful research quality from differences in individual experts’ preferences or backgrounds? What justifies the chosen criteria, and how do the authors establish that judgments of model generated research directions are sufficiently consistent and reliable for benchmarking?

Q3 (regarding W3): How can the authors establish a fair and sound evaluation playground for frontier models’ research judgment? How sensitive are the findings to harness design? Why manually designed RAG is still in use when most of frontier model has shifted to agentic.

These are hard questions, i do not expect authors to address all of them in the rebuttal.

7 comments
AuthorsOpenReview ↗

Response to Reviewer QGCz (1/3)

Thank you for highlighting the importance of benchmarking forward-looking research decisions and recognizing the temporal controls and construction protocol. We address task quality, human reliability, and harness sensitivity separately below.

W1/Q1. Substantive research decisions, task quality, and community value

We would like to clarify that ForeSci tasks require agents to gather historical evidence, organize the research trajectory, and make a research decision; they are designed to go beyond simple question answering or historical restatement. Each task asks for an explicit forward-looking decision: a next-step direction and trajectory, a root bottleneck and the opportunity it enables, a comparative research priority, or a conditional venue strategy. Direction tasks allow the answer to be generated within the stated scope, while planning tasks provide options whose priority must be decided.

Every task specifies the decision to be made, the evidence boundary, the required output, and the rubric used to evaluate it. The revised Figure 1 now shows these elements directly, and the anonymous benchmark repository makes the public task specifications, evaluator-only target slots, code, and judge prompts inspectable. Thus, the benchmark's research-decision requirements are explicit even though the future targets remain hidden for evaluation.

Furthermore, we test whether the benchmark can be solved by historical restatement or simple trend continuation. On the same 120-task confirmation set, Qwen3-235B receives identical frozen historical evidence under three instructions: extract the historical conclusion, continue the historical trend, or make the required research decision. Target-F1 measures how precisely and completely the answer covers the required decision slots.

StrategyTarget-F1Direct-decision advantage, paired 95% CI
Historical-conclusion extraction (H1)0.395+0.080 [0.055, 0.105]
Historical-trend continuation (H2)0.428+0.047 [0.025, 0.069]
Direct research-decision instructions (H3)0.475Reference

The paired intervals resample tasks while keeping the compared conditions together. Direct research-decision instructions reach 0.475 Target-F1, compared with 0.395 for historical-conclusion extraction and 0.428 for trend continuation. The same pattern appears in the earlier 80-task development control. Because the evidence is identical, the difference shows that the task requires a decision beyond historical restatement.

Separately, an independent audit provides a second check on task quality. Among the 80 development tasks, 78 had clear decision requirements, 77/78 evaluable premises were historically supported, and 77/79 evaluable rubrics were executable. None of the 80 tasks had its complete final answer directly supplied by the historical evidence. The public wording and required outputs have since been clarified, while the original audit counts remain reported.

Taken together, the task contracts, audit, and matched strategy comparison show that ForeSci evaluates an evidence-grounded research choice before research execution, with later developments used only to assess the quality of that choice. It therefore tests forward-looking research judgment beyond historical extraction or generic question answering.

Revision: Section 3.4 (pp. 5–6) clarifies task requirements. Appendices D.1–D.3 (pp. 23–26) and D.5 (pp. 29–30) provide scoring criteria, rating anchors, a worked example, and evaluator prompts. Appendix I.1 (p. 44, Table A27) reports the task audit; Appendix J.1 (pp. 48–49, Tables A37–A38) reports the 120-task confirmation. The revised Figure 1 (p. 2) illustrates the task-to-evaluation structure.

AuthorsOpenReview ↗

Response to Reviewer QGCz (2/3)

W2/Q2. Human reliability, criteria, and expert background

We evaluate four aspects of research judgment: whether a proposal was reasonable given the evidence available at the cutoff, whether its claims were supported, whether it was feasible under the stated constraints, and whether it covered the later target. Persuasiveness is measured separately. This separation lets us ask whether a proposal was a sensible decision at the time even when later events do not confirm it.

To test whether experts apply these criteria consistently, three annotators independently assessed 160 answers. They judged reasonableness, support, and feasibility before seeing future targets, then assessed future coverage after those initial judgments were fixed. Agreement is computed from the initial labels, before discussion or adjudication.

CriterionInitial Krippendorff's α [95% CI]
Historical reasonableness0.784 [0.721, 0.842]
Historical support0.812 [0.756, 0.864]
Resource/time feasibility0.746 [0.678, 0.809]
Future-target coverage0.762 [0.695, 0.825]

The initial agreement is 0.784 for historical reasonableness, 0.812 for support, 0.746 for resource/time feasibility, and 0.762 for future-target coverage, with the corresponding intervals shown in the table. The first three measures use ordinal agreement; future coverage uses nominal agreement.

We also test whether the judgments depend on one person or a narrow expertise profile. Two closely matched-domain raters and two related-domain raters evaluated the same 40 answers. They agreed on historical reasonableness for 33/40 answers and on within-task preferences for 16/20 pairs; leaving out each original rater preserves 86.9–89.4% of reasonableness judgments.

Finally, we compare the automatic scores with independent judgments on the same answers. Under OW-ECTA, Target-F1 measures required-decision coverage and Support measures support for the claims actually made. A05 and A06 assessed 300 answers from 60 original AI tasks without seeing machine scores. Support correlates with their support judgments at ρ=0.669 and 0.644, compared with 0.532 and 0.474 for Fact. For A06's coverage judgments, Target-F1 gives ρ=0.522, compared with 0.404 for FTA. The separate persuasiveness study provides an additional reliability check: 1,200 answers rated by three annotators yield mean pairwise Spearman ρ=0.818 and quadratic-weighted κ=0.821.

Together, these results show that the criteria are anchored in the task's evidence and decision requirements, produce substantial initial agreement, remain reasonably stable across expert backgrounds, and correspond better to human judgments when support and coverage are evaluated separately.

Revision: Section 4.1 (p. 7) introduces OW-ECTA and its two reported components. Appendices H.1 (p. 39) and H.3 (pp. 40–41) define the protocol and report expert calibration (Table A20). Appendix I.1 (pp. 44–45, Tables A28–A29) documents independent agreement, adjudication, and background sensitivity; Appendices D.2 (pp. 25–26) and D.4.1 (p. 26) document the original persuasiveness protocol and agreement.

AuthorsOpenReview ↗

Response to Reviewer QGCz (3/3)

W3/Q3. Fair harness comparison and modern autonomous agents

We test harness sensitivity by comparing three ways of using the same cutoff-aligned evidence: a retrieval baseline, a concise autonomous agent, and a modern agent with planning, delegation, and context compression. The comparison is repeated with Qwen3-235B and GPT-5.2 on the same 120-task confirmation set, so differences can be attributed to the workflow and its budget as well as to the model backbone.

The three original agent systems—CoI, ResearchAgent, and ARIS—are deliberately retained as fully specified research-agent harnesses, including their planning or review stages, state handling, stopping and recovery rules, and native answer assembly. Only their network search is replaced by access to the same historical paper library. Hybrid RAG is included deliberately as a retrieval baseline: it measures what evidence retrieval alone provides, rather than serving as a competing agent harness. To test sensitivity to harness choice, we additionally introduce two autonomous controls: Simple Agent as a lightweight autonomous sensitivity baseline, and Hermes as a contemporary agent harness with planning, delegation, and context compression. All systems start each task with a fresh session and the same historical evidence boundary.

Under the shared budget, Simple and Hermes each have up to eight model calls, six evidence accesses, 16,000 cumulative input tokens, and 2,500 cumulative output tokens. We also evaluate a higher-budget Hermes condition with up to 14 calls, 10 evidence accesses, 32,000 input tokens, and 4,000 output tokens. The final answer is limited to 1,024 tokens in both conditions.

ModelPaired comparison on 120 tasksΔFTA [95% CI]ΔTarget-F1 [95% CI]ΔTrace [95% CI]
Qwen3-235BHermes − Hybrid RAG+0.020 [0.004, 0.036]+0.021 [0.003, 0.039]+0.061 [0.040, 0.082]
GPT-5.2Hermes − Hybrid RAG+0.025 [0.009, 0.041]+0.025 [0.007, 0.043]+0.101 [0.079, 0.123]
Qwen3-235BHermes − Simple+0.006 [−0.009, 0.021]+0.005 [−0.012, 0.022]+0.028 [0.009, 0.047]
GPT-5.2Hermes − Simple+0.007 [−0.008, 0.022]+0.009 [−0.008, 0.026]+0.031 [0.012, 0.050]
Qwen3-235BHermes: higher − shared budget+0.003 [−0.008, 0.014]+0.003 [−0.011, 0.017]+0.021 [0.004, 0.038]
GPT-5.2Hermes: higher − shared budget+0.003 [−0.008, 0.014]+0.004 [−0.010, 0.018]+0.019 [0.003, 0.035]

Hermes−baseline comparisons use the shared-budget Hermes configuration. Intervals use paired task-level bootstrap resampling, and all 120 planned tasks remain in the metric denominators.

Hermes improves both FTA and Target-F1 over Hybrid RAG for both backbones. Compared with Simple Agent, it consistently improves evidence traceability, while the additional Target-F1 gain remains uncertain. Against H3, GPT-5.2 also has a positive FTA contrast (+0.017 [0.001, 0.033]), but neither backbone establishes an additional Target-F1 gain over H3.

The higher Hermes budget reduces forced stops from 13/120 to 2/120 for Qwen and from 10/120 to 1/120 for GPT, while increasing mean total tokens from 13,280 to 23,020 and from 13,270 to 22,490. It improves Trace, but the additional Target-F1 gains remain uncertain. Under the shared budget, delegation occurs in 51/120 Qwen and 57/120 GPT runs, and compression in 22/120 and 19/120.

Taken together, these comparisons show that ForeSci is sensitive to how an agent searches and organizes evidence: a modern agent improves future alignment over retrieval alone, while richer organization or a larger budget does not reliably produce an additional gain in the final research decision. This separates improvements in evidence handling from improvements in forward-looking judgment.

Revision: Section 4.2 (p. 7) and Appendix E (pp. 30–31, Table A10) explain the retained original agent mechanisms and offline literature interface. Section 5.1 (p. 8) summarizes the modern-harness result. Appendix J.5 (pp. 50–52, Tables A42–A45) reports the two-backbone Hermes comparison, shared resource ceilings, paired contrasts, execution outcomes, and resource use. Appendices I.4 (pp. 46–48) and J.1 (pp. 48–49) retain the earlier Simple Agent controls.

Reviewer_fNgNOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ForeSci, a temporally controlled benchmark designed to evaluate the forward-looking research judgment of Large Language Model (LLM) agents, specifically their ability to make decisions like bottleneck discovery, direction forecasting, strategic planning, and venue positioning based strictly on historical evidence. To prevent hindsight bias, the benchmark enforces a strict temporal cutoff, hiding post-cutoff literature during the generation phase and using it exclusively for validation. The authors evaluate various systems, including native LLMs, Hybrid RAG, and agentic workflows, revealing that while agents improve evidence traceability and reviewer persuasiveness, their actual alignment with future research trajectories remains modest. Furthermore, the study identifies a critical failure mode termed "evidence-decision decoupling," where agents correctly cite historical evidence but misalign it with the wrong research objects or causal roles, and demonstrates that targeted decision checks can help mitigate this drift.

Critical strengths

The primary strength of this work lies in its rigorous temporal control and the highly relevant, forward-looking problem it addresses, which is crucial as autonomous research agents are increasingly deployed for scientific ideation and planning. By enforcing strict evidence boundaries and utilizing a hidden future validation set, ForeSci effectively eliminates the hindsight bias and data leakage that plague many existing forecasting benchmarks. Additionally, the multi-signal evaluation protocol is exceptionally well-designed, decoupling surface-level persuasiveness and evidence traceability from actual future-target alignment, which provides deep insights into agent behavior. The diagnostic analysis of "evidence-decision decoupling" is particularly insightful, offering a nuanced understanding of why agents fail to translate retrieved evidence into correct prospective decisions, and the proposed task-specific interventions provide a practical, measurable pathway for improving agentic reasoning.

Critical weaknesses

Despite its innovative design, the benchmark's scope is somewhat limited, as it focuses exclusively on four fast-moving AI subfields and relatively short forecast horizons (three to six months), leaving it unclear whether the findings and the identified "decoupling" failure modes generalize to slower-evolving scientific domains or longer-term strategic planning. Furthermore, the evaluation relies heavily on semantic matching and LLM-based proxies for metrics like Future-Target Alignment and Factuality, which may struggle to capture genuine paradigm shifts or novel terminologies that characterize true scientific breakthroughs. Finally, while the proposed task-specific interventions statistically reduce drift, the absolute gains in future-target alignment are relatively modest, indicating that current LLM agents still face fundamental limitations in genuine foresight that simple prompting or decision-checking strategies cannot fully resolve.

Questions for authors

please see weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fNgN (1/3)

Thank you for recognizing ForeSci's temporal controls, multi-signal evaluation, and evidence–decision decoupling analysis. A central finding is that better organization and traceability of historical evidence do not consistently translate into better forward-looking research decisions in the evaluated agents. ForeSci makes this distinction measurable, identifies failures in the transition from evidence to decision, and tests mechanism-guided improvements. The additional evidence below addresses the forecast horizon and domain, semantic validity, and interpretation of the intervention separately.

W1. Longer forecast horizons and mechanism transfer across domains

We focus on fast-moving AI subfields because rapid change is precisely where forward-looking judgment is most necessary: pre-cutoff evidence can become obsolete quickly, so an agent must decide which direction, bottleneck, or opportunity is worth pursuing before later evidence becomes available. To examine whether our findings nevertheless depend on short AI forecast windows or the original domain coverage, we evaluate a one-year forecast horizon and a separate AI-assisted protein-design domain. We also test whether the diagnosed evidence–decision errors and the targeted-check effect recur in these settings.

One-Year and Protein Evaluations. The one-year AI subset contains 100 tasks across the same four AI domains, evaluated with four backbones and five methods (2,000 answers); it forms part of the revised 600-task AI benchmark. The protein-design cohort contains 60 tasks, evaluated with two backbones and five methods (600 answers). Both use a September 30, 2025 historical cutoff and an October 1, 2025–September 30, 2026 validation window. Appendix H.4 reports construction, all-method results, and an independent domain expert's review of the protein tasks and reference slots, including the full cohort and the expert-approved subset.

In the one-year AI evaluation, AgentAvg–Hybrid differences in Target-F1 range from −0.025 to +0.012 across the four backbones. On the 52 expert-approved protein tasks, GPT-5 agents improve Trace over Hybrid by +0.0299 (95% CI [0.0098, 0.0537]), while the Target-F1 difference is −0.0204 [−0.0564, 0.0119]. This comparison resamples the 17 shared-topic/source groups while retaining the methods within each task. These results extend the observation that stronger evidence traceability does not consistently establish stronger future-target coverage.

The original intervention study developed diagnosis-guided checks on 80 tasks and evaluated them on 420 disjoint confirmation tasks. To test whether the same evidence–decision failure appears beyond the original short-horizon AI tasks, we ran a separate audit on the 100 one-year AI tasks and 60 protein-design tasks. After each agent produced its initial answer from the cutoff evidence, we gave it one additional revision turn under one of two conditions. In the first, the agent was simply asked to reconsider and improve its answer using the same evidence. In the second, it was asked to revisit the link between the evidence and the required decision, using the diagnosis from the original intervention. Thus, both conditions allow the agent to revise once, while only the second directs attention to the evidence–decision errors identified by ForeSci.

The generic revision left most error rates close to those of the original answers. The diagnosis-guided revision reduced every listed error category—wrong object, wrong scope, wrong causal role, wrong dependency, unsupported claims, and omitted decisions—and improved Target-F1 over generic revision in both cohorts. Error categories are non-exclusive.

Mechanism-audit cohortTasksAnswers per condition (3 agents)Target-F1: targeted − general [95% CI]
One-year AI100300+0.052 [0.031, 0.073]
Protein design60180+0.048 [0.022, 0.074]

The intervals resample tasks, retaining all agents and compared conditions within each task. Both Target-F1 intervals exclude zero; the category-level error reductions are reported descriptively. These results address the scope concern with both outcome-level comparisons and a mechanism test: evidence–decision errors recur in the one-year and protein settings, and targeted checks improve future-target coverage over general reflection in both cohorts.

Taken together, the results indicate that the foresight gap is not an artifact of the original short-horizon AI setting: it persists across a longer forecast horizon and a different scientific domain, while diagnosis-guided checks provide a transferable way to improve the translation from evidence to research decisions.

Revision: Appendix H.4 (pp. 41–43, Tables A23–A26) reports the one-year and protein-design evaluations and domain review. Appendix J.4 (p. 50, Table A41 and accompanying text) reports the cross-cohort mechanism audit and paired Target-F1 effects.

AuthorsOpenReview ↗

Response to Reviewer fNgN (2/3)

W2. Semantic matching, new terminology, and LLM-based proxies

To test whether our metrics reward topical overlap or penalize unfamiliar wording, we separate support for an answer's claims from coverage of the required research decision.

Our added protocol, Open-World Evidence-Conditioned Target Alignment (OW-ECTA), complements FTA by separating reference support from decision coverage. Fact checks atomic claims; Support assesses substantive claims against supplied reference units; Target-F1 combines claim-match precision with required-slot coverage. The frozen semantic judge accepts equivalent wording and within-slot alternatives, with model identities and human scores hidden. The protocol adapts claim-level support and entailment checks from FActScore (Min et al., 2023), RAGChecker (Ru et al., 2024), RefChecker (Hu et al., 2024), and VeriScore (Song et al., 2024), while ForeSci defines the research-decision slots, accepted alternatives, and temporal evidence boundary.

We first test the metric on 60 tasks with five controlled answers per task: a correct reference, a low-overlap paraphrase, an overly broad answer, a causal reversal, and a topical answer that does not make the required decision.

Answer variantFactFTATarget-F1
Concrete correct reference0.9200.7800.880
Meaning-preserving reformulation0.9070.7740.865
Overly broad0.3960.5730.280
Causal reversal0.2470.7170.180
Merely topical, no decision0.1120.4540.050

Meaning-preserving reformulations remain close to the reference on average: the FTA change is −0.006, with a paired 95% CI of [−0.022, +0.010]. By contrast, broadness, causal reversal, and omission of the decision produce substantial drops in Fact and Target-F1. The smaller FTA change under causal reversal is why we report claim support and target coverage together rather than relying on either one alone. Appendix J.2 (Table A39) reports these results; Appendix H.2 (Table A19) separately reports the original-scorer analysis with explicit negation (60 tasks, six variants, 360 texts).

We then compare the automatic scores with independent expert judgments. A05 and A06 assessed 300 answers from 60 original AI tasks without seeing machine scores. Support correlates with their reference-support judgments at ρ=0.669 and 0.644, compared with 0.532 and 0.474 for Fact. For A06's future-coverage judgments, Target-F1 gives ρ=0.522, compared with 0.404 for FTA. These comparisons use the same answers, annotators, and criteria for each metric.

A concrete example illustrates how this accommodates different terminology. In RFDIR-0032, ResearchAgent expresses the reference's counterfactual-evaluation requirement through removing video frames and perturbing evidence. Target-F1 credits these concrete operations (1.000), whereas Hybrid's answer, which omits that requirement and extension beyond static images, scores 0.750. Both experts give the same coverage ordering, and both answers receive support grade 2. The example distinguishes support for the claims actually made from coverage of the required research decision.

These results show little average penalty for the tested meaning-preserving reformulations and substantially lower Fact/Target-F1 for decision errors. The expert comparisons and paired example support crediting semantic equivalents while identifying missing decision requirements.

ForeSci evaluates future developments grounded in pre-cutoff signals. Targets must have an identifiable historical basis and be verifiable against later evidence, making model comparisons reproducible. Breakthroughs driven by contingent events without such a basis do not supply a fair forecasting target. This criterion concerns forecastability from the available evidence, rather than scientific importance.

Together, Fact and the added Support/Target-F1 components assess claim correctness and decision completeness alongside FTA. The semantic controls and expert calibration test these judgments directly, while the temporal protocol anchors forecasts to historical signals and validates them against later evidence.

Revision: Section 4.1 (p. 7) introduces OW-ECTA, Target-F1, and answer-level Support. Appendices H.1–H.3 (pp. 39–41, Tables A19–A20) define the protocol and report the original-scorer tests, human calibration, and paired example. Appendix J.2 (p. 49, Table A39) reports the five-variant semantic challenge.

AuthorsOpenReview ↗

Response to Reviewer fNgN (3/3)

W3. What the foresight gap and intervention contribute

We agree that the absolute gains from the intervention are modest, and we do not present the checks as a complete solution to the broader difficulty of forward-looking research judgment. We use this experiment primarily as a mechanism diagnosis rather than as a new agent algorithm. The question is narrower: when an agent has already organized the available evidence, does the remaining error lie in how that evidence is connected to the research decision, and can that error be identified by a targeted probe?

The original intervention study developed task-specific checks on 80 tasks and evaluated them on 420 disjoint confirmation tasks across three agents. The checks direct attention to the connection between evidence and the decision—for example, distinguishing an observed failure from its upstream cause before naming a bottleneck, or comparing candidate studies at the same decision scale before prioritizing them.

We then ran a separate matched control on 105 tasks × three agents = 315 paired task-agent runs. Each agent received the same original answer and frozen evidence and one additional revision turn. In one condition it was asked to reconsider and improve the answer generally; in the other it was asked to revisit the evidence–decision connection using the diagnosis-guided checks. Neither condition could retrieve new evidence, and the additional token use was similar—approximately 1,850 and 1,920 tokens.

MetricTargeted Checks − General Reflection95% task-level paired bootstrap CI
Drift−0.146[−0.185, −0.107]
Fact+0.045[+0.030, +0.060]
FTA+0.024[+0.010, +0.038]
Target-F1+0.057[+0.038, +0.076]
Support+0.050[+0.032, +0.068]

All five intervals exclude zero. The gains remain modest, but under the same evidence and a nearly matched revision budget, targeted checks consistently outperform a generic revision on drift, Fact, FTA, Target-F1, and Support. This pattern shows that the diagnosed evidence–decision errors are measurable and contribute to the observed failures.

Taken together, this experiment supports a narrower contribution: ForeSci does not claim to introduce a complete agent solution, but it identifies a specific failure mechanism that can be tested independently of retrieval and writing quality. The result provides a basis for future agent designs that explicitly connect research objects, causal roles, dependencies, and forecast horizons to the final decision.

Revision: The contribution statement in Section 1 (p. 2) identifies the evidence-to-decision gap and the tested improvement direction. Section 5.2 (pp. 9–10) and Appendix G.3 (pp. 35–38) describe the diagnosis and task-specific interventions. Appendix J.3 (pp. 49–50, Table A40) reports the separate matched revision control. The incremental conclusion update (p. 10) connects these findings to agent design while retaining the paper's forward-looking research-judgment focus.

Thank you again for recognizing the temporal controls and evidence–decision analysis. We would value your assessment of the additional evidence addressing your three concerns.

To examine scope, we added a one-year AI evaluation with 100 tasks and a protein-design evaluation with 60 tasks. The separate mechanism audit finds the same evidence–decision error categories in both settings, with positive paired Target-F1 effects for targeted checks over general reflection (W1; Appendices H.4 and J.4).

For metric validity, the 60-task semantic challenge shows little average FTA change under meaning-preserving reformulation (−0.006; 95% CI [−0.022, +0.010]), while broad, causally reversed, and merely topical answers receive much lower Fact and Target-F1 scores. We also introduce OW-ECTA’s separate coverage and support judgments and report calibration against expert ratings (W2).

To test the intervention’s practical value, the matched control on 105 tasks and three agents disables new retrieval and uses nearly matched revision budgets. Targeted checks improve Target-F1 over general reflection by +0.057 [0.038, 0.076] (W3; Appendix J.3). This connects the diagnosed foresight gap to a tested direction for agent improvement.

Could you let us know whether these results address the scope and metric concerns, and clarify the contribution of the diagnosis and intervention? We would be happy to discuss any remaining issue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.