- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ForeSci contains 500 tasks spanning four AI domains and four decision types: forecasting directions, identifying bottlenecks and opportunities, prioritizing research, and positioning work for venues. Systems receive historically restricted literature; later developments provide evaluation targets. The evaluation distinguishes prediction factuality, future-target alignment, evidence traceability, and reviewer persuasiveness. Five system configurations are evaluated across four backbones. Agent workflows improve traceability more consistently than future-target alignment. The authors diagnose evidence-to-decision drift and test task-specific decision checks on disjoint confirmation tasks.
Critical strengths
- The evaluation is substantial with broad coverage, there are five system configurations across four backbones with human check; while there are also human checks on construction, human validation of the persuasiveness predictor, and held-out intervention experiments.
- The idea is interesting to move beyond information retrieval and executable implementation to research judgment.
Critical weaknesses
-
There is some weakness in target validity, a later popular direction is not necessarily the best decision a resource-constrained team could have made at the earlier date, and there may be reasonable alternatives that don't match the trends.
-
Embedding-based FTA may reward broad or topically similar answers without establishing prediction correctness.
Questions for authors
-
How did the human audit handle answers that disagree with the eventual research trajectory but are defensible under the task's historical constraints? Were such alternatives admitted to the targets, and is there existing agreement or adjudication evidence for these cases?
-
Is there any example of existing validation that shows that Direction and Bottleneck(reference-guided) FTA distinguishes a correct prediction(or not) from a negated, overly broad, or merely topically similar answer?
4 comments
Response to Reviewer MJ3T (2/2)
Q2. Does reference-guided FTA distinguish correct predictions from negated, overly broad, or merely topical answers?
We test this directly by holding the task, length, format, and style fixed while changing only the decision. Each task has a correct reference, a meaning-preserving paraphrase, an explicit negation, a causal reversal, an overly broad answer, and a merely topical answer that does not make the required decision. The analysis covers 30 Direction and 30 Bottleneck tasks, or 360 texts in total.
The table reports the difference between the correct reference and each variant under the original frozen bge-m3 FTA scorer. A positive value means that the scorer gives the correct reference a higher score; the final two columns count how often that happens on the same task.
| Variant | Direction: ΔFTA [95% CI] | Bottleneck: ΔFTA [95% CI] | Direction wins | Bottleneck wins |
|---|---|---|---|---|
| Explicit negation | +0.046 [+0.024, +0.068] | +0.039 [+0.018, +0.060] | 23/30 | 22/30 |
| Overly broad | +0.208 [+0.176, +0.240] | +0.207 [+0.174, +0.240] | 29/30 | 29/30 |
| Merely topical | +0.330 [+0.292, +0.368] | +0.322 [+0.285, +0.359] | 30/30 | 30/30 |
| Causal reversal | +0.064 [+0.041, +0.087] | +0.062 [+0.039, +0.085] | 25/30 | 24/30 |
Meaning-preserving paraphrases have much smaller average differences: +0.007 [−0.008, +0.022] in Direction and +0.005 [−0.011, +0.021] in Bottleneck. The contrast supports sensitivity to the tested decision errors beyond wording changes or topical overlap.
Complementary Claim Verification. Under explicit negation, Fact drops from 0.904 to 0.214 in Direction and from 0.936 to 0.186 in Bottleneck; under causal reversal, it falls to 0.265 and 0.228, respectively.
Across all tested error types, the scorer gives the correct reference a higher FTA on average in both task families, and every paired interval excludes zero. The separation is largest for overly broad and merely topical answers. Fact provides a sharper contradiction signal for negation and causal reversal, which is why the revised evaluation reports both measures.
The added OW-ECTA analysis extends this result by making claim support and required-decision coverage separately inspectable. The controlled test above uses the original frozen scorers; the OW-ECTA semantic challenge and human calibration are reported separately in Appendices J.2 and H.3.
Taken together, the controls show that reference-guided FTA is not simply rewarding shared topic words: it is relatively stable under meaning-preserving wording changes and decreases when the answer negates, reverses, broadens, or omits the decision.
Revision: Appendix H.2 (pp. 39–40, Table A19) reports the controlled variants, original scoring pipelines, paired intervals, and per-family discrimination results. Section 4.1 (p. 7) and Appendix H.1 (p. 39) introduce OW-ECTA; Appendices H.3 (pp. 40–41) and J.2 (p. 49) report its component validation separately.
Thank you again for your assessment. We are following up on the two specific questions about target validity and reference-guided FTA.
For defensible alternatives, our two-stage audit evaluates historical reasonableness before revealing future evidence. Reasonable proposals without observed future support retain their historical-quality judgment. For six future-supported answers omitted from the targets, admitting the missing alternatives in the audited development subset leaves the method ranking unchanged on the 160 audited answers (Q1; Appendix I.2).
For FTA discrimination, we directly test the original frozen scorer on 30 Direction and 30 Bottleneck tasks. Correct references receive higher mean FTA than negated, overly broad, merely topical, and causally reversed variants in both families, with each paired 95% interval excluding zero. Meaning-preserving paraphrases have much smaller average differences, while Fact provides a complementary contradiction signal (Q2; Appendix H.2).
Do these clarifications and tests address your two questions? In particular, we would appreciate your view on the separation of historical defensibility from future coverage and the evidence for the original FTA scorer. We would be happy to clarify any remaining concern.
Thank you for your reply. I have updated my rating.
Response to Reviewer MJ3T (1/2)
Thank you for recognizing ForeSci's coverage and its focus on forward-looking research decisions beyond retrieval and implementation. Your two questions concern the validity of the future targets and the discrimination provided by reference-guided FTA. We address them separately with the target audit and controlled tests of the original scorers.
Q1. How are historically defensible alternatives handled, and what agreement or adjudication evidence supports this?
We would like to clarify this by checking both the construction of the frozen future targets and the audit of model-generated answers. The hidden future targets are constructed and frozen before the primary model evaluation, with acceptable alternative routes identified during task construction. Separately, generated answers are audited for historical defensibility and later validation. A generated answer that is historically defensible but not supported by the later validation corpus retains its historical-quality label, but it is not counted as future-target covered. If the audit reveals a historically defensible, later-supported route absent from the frozen target set, we test that route only in a development-subset target-completion sensitivity analysis; we do not use model outputs to rewrite the primary ground truth. The benchmark therefore separates whether a proposal was reasonable given what was known at the cutoff from whether later evidence supports its future target.
We implement this distinction in two analyses. First, in the ground-truth target construction, the hidden targets include acceptable alternatives rather than only one reference wording. Across the audited tasks, all 125 Direction tasks contain acceptable neighboring directions (mean 4.43 per task); 94/125 Bottleneck tasks contain acceptable bottleneck variants and 86/125 contain acceptable opportunity variants. Each decision slot is scored against its best matching accepted alternative, so a valid route is not rejected merely because it uses different terminology or reaches the target through a different path.
Second, for the model-generated answers, we asked three annotators to assess 160 answers from 80 development tasks. They first judged historical reasonableness, evidence support, and feasibility using only information available at the cutoff. Only after those judgments were fixed did they inspect the later validation material and judge future coverage against the frozen targets. Initial agreement before discussion or adjudication was 0.784 for historical reasonableness, 0.812 for support, 0.746 for feasibility, and 0.762 for future coverage. After the initial labels were recorded, disagreements were discussed and adjudicated; four unresolved cases were retained separately.
The audit distinguishes the alternatives relevant to your question:
Adding the six supported alternatives only in the development-subset sensitivity analysis leaves the method ranking unchanged on these 160 answers. The primary target set and primary results are therefore unchanged. The audit supports the intended interpretation of the benchmark: later popularity is not treated as the definition of a good decision, while historical defensibility is reported as a separate property rather than being used to redefine future-target correctness.
Taken together, the target audit and the two-stage human assessment show that ForeSci can credit reasonable alternative routes while keeping historical quality separate from future validation. This is the distinction needed for evaluating foresight rather than hindsight.
Revision: Appendix D.1 (p. 23) specifies accepted target variants and best-variant matching. Appendix I.1 (pp. 44–45, Table A28) reports the two-stage protocol, initial agreement, and adjudication counts. Appendix I.2 (p. 45, Table A30 and accompanying text) separates the audit categories and documents the target-completion sensitivity check.