- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PreSage-Bench, a dataset of 1,012 public-figure statements from 2015–2026 annotated with a affected-entity × impact-domain space, together with a risk-evidence inventory. It further proposes PreSage, a multi-agent system consisting of nine role agents, an Evidence Judge, and an Impact Judge. A self-evolution procedure uses training-case prediction errors to propose prompt edits that are accepted or rejected using a fixed development-set gate. The evaluation uses a temporal case split at 1 January 2025 and reports results on post-2025 cases. The full PreSage configuration increases recall substantially relative to direct prompting, while its advantage over the dual-layer ablation in micro-F1 is small. The authors also evaluate prompt evolution and several API backends.
Critical strengths
Representing adverse impact as an affected entity × impact domain is more informative than a binary toxicity or controversy label, and the benchmark covers a relatively large set of real public-figure cases across multiple domains. Separating heterogeneous role agents, evidence adjudication, impact prediction, and prompt evolution is sensible. The development-set acceptance gate for prompt updates is also considerably more disciplined than unconstrained prompt self-modification.
Critical weaknesses
Restricting the prompt to information available before the statement’s release is not sufficient when the underlying LLM may already know the statement and its aftermath. The inputs contain the public figure’s identity, exact statement, release date, scene, and platform, making retrieval from parametric memory particularly plausible. A case-level temporal split at January 2025 separates prompt-evolution examples from test examples, but it does not establish that Qwen-plus, Gemini-2.5-flash-lite, or GPT-4o-mini were trained only on information preceding each test event. Consequently, the current results cannot distinguish genuine anticipation from memorized retrospective knowledge. This directly challenges the paper’s central “genuine foresight” claim. The abstract makes that claim explicitly.
Questions for authors
What are the exact training/knowledge cutoffs or immutable model snapshot dates for Qwen-plus, Gemini-2.5-flash-lite, and GPT-4o-mini? Can the authors demonstrate that each evaluated event occurred strictly after the corresponding model’s possible training-data cutoff?
3 comments
Thank you for your response. I acknowledge the clarification that the 759 training cases all occurred before January 2025, while the 253 test cases occurred after this point; I also note the additional information regarding the knowledge cutoff dates for the models used. This helps clarify the temporal partitioning strategy employed between the prompt self-evolution phase and the final evaluation phase. However, my primary concern—the potential for "retrospective knowledge contamination"—remains unaddressed.
First, the temporal split between the training and test sets does not guarantee that the underlying large language model (LLM) was trained exclusively on information available prior to the occurrence of each test event. Specifically, the authors have not provided the exact, immutable version (snapshot) of the Qwen-Plus model used. Official Alibaba Cloud documentation lists multiple Qwen-Plus snapshots released throughout 2025, including one from December 2025. Therefore, the authors' claim that Qwen-Plus has a knowledge cutoff date of January 2025 requires supporting evidence linked to the specific model version actually employed.
Second, the knowledge cutoff date reported by the authors for GPT-4o-mini appears to be factually incorrect. According to official OpenAI documentation, the knowledge cutoff for this model is October 2023, not July 2024 as claimed; July 2024 actually corresponds to the release date of GPT-4o-mini.
Third, while it is true that Gemini-2.5-Flash-Lite has a knowledge cutoff date of January 2025, this does not automatically ensure that all test events occurred strictly after that date. Given that the test set begins on January 1, 2025, the authors must compare the model's precise knowledge cutoff date against the actual occurrence date of each test event to rule out any potential temporal overlap. More importantly, Appendix C.2 of the paper explicitly acknowledges that "exact runtime and model-snapshot equivalence are not established" across different execution environments and model snapshots. This statement creates a clear inconsistency between the authors' current response—which claims that model knowledge cut-off dates were strictly controlled—and the experimental conditions as documented in the paper.
Consequently, I request that the authors provide the following evidence: (1) the precise model snapshot IDs and API call dates used in all experiments, along with authoritative sources for the corresponding knowledge cut-off dates; (2) a case-by-case temporal audit demonstrating that each test event occurred strictly after the verified knowledge boundary of the model used; and (3) targeted experiments addressing the risk of "memorization contamination," specifically by testing whether the base models can directly recall the specific events and their subsequent outcomes from the test set, and by re-evaluating model performance on a subset of cases where no clear evidence of post-event knowledge is detected. It would be even more compelling if the authors could conduct a truly prospective validation—recording and freezing predictions before the consequences of an event have unfolded, and subsequently verifying their accuracy against actual outcomes.
In the absence of such validation, the current experimental results may, in part, reflect the models' existing knowledge of historical events rather than a genuine capacity for ex ante (pre-event) forecasting. Therefore, I believe the authors' current response is insufficient to support the paper's core claim of "genuine foresight," and my initial concerns regarding the soundness of the paper remain unresolved.
We thank you for the detailed follow-up. The knowledge cutoff for GPT-4o-mini is indeed October 2023, earlier than the July 2024 date we had listed, which further rules out leakage of the test events into the backbone.
On Appendix C.2, that note concerns runtime reproducibility across execution environments, not which events a model has seen. Our memorization probe in (3) settles the latter directly, since the models recall none of the test events. We will revise the wording of C.2 so it is not read as a statement about temporal control.
As requested, we provide the following evidence and experiments.
(1) Model snapshots and cutoff sources. We checked the knowledge cutoff dates of our models more carefully and compared them against the dates of our test cases. The results are below.
| Model | Snapshot | Knowledge cutoff | Source |
|---|---|---|---|
| Qwen-plus | qwen-plus-2025-01-25 | On or before 25 Jan 2025 (snapshot release date; Alibaba publishes no separate cutoff) | https://www.alibabacloud.com/help/en/model-studio/qwen-plus |
| Gemini-2.5-flash-lite | gemini-2.5-flash-lite-preview-09-2025 | January 2025 (exact day not disclosed) | https://aistudio.google.com/docs/models/gemini-2.5-flash-lite-preview-09-2025?hl=en |
| GPT-4o-mini | gpt-4o-mini-2024-07-18 | 1 October 2023 | https://developers.openai.com/docs/models/gpt-4o-mini |
(2) Case-by-case temporal audit. We compared every test event date against each model's knowledge boundary. Because Gemini-2.5-flash-lite's exact cutoff day is not disclosed, we remove the five test cases dated in January 2025 and re-run the full evaluation. As the table shows, after removal the scores are almost unchanged, and the main result still holds.
| Test set | cases | Micro-Recall (Ours) | Micro-F1 (Ours) |
|---|---|---|---|
| Full | 253 | 0.5853 | 0.6275 |
| After removing the 5 January 2025 cases | 248 | 0.5736 | 0.6254 |
(3) Memorization probe and re-evaluation on the uncontaminated subset. We probe whether each backbone recalls the test events. For every case we ask the base model whether a named person posted a statement on a given platform on a given date. It may answer only remember, do not remember, or unsure, and we allow no search and no guessing.
- On the 253 test events, all three models answer do not remember every time.
- We also build 45 fabricated person, platform, and date combinations and test whether the models have seen them. The models recall none, so the probe raises no false recall.
- We select 15 well-known events from before each model's cutoff and test whether the models remember them. The results are in the table below.
| Probe set | cases | Recalled |
|---|---|---|
| Test events, each of Qwen-plus, Gemini-2.5-flash-lite, GPT-4o-mini | 253 | 0 |
| Fabricated combinations (decoys) | 45 | 0 |
| Known pre-cutoff events, Qwen-plus | 15 | 5 |
| Known pre-cutoff events, Gemini-2.5-flash-lite | 15 | 5 |
| Known pre-cutoff events, GPT-4o-mini | 15 | 10 |
The experiments above show that the base model has not seen the test events or their outcomes.
We re-evaluate our method on the subset where the probe finds no sign of post-event knowledge. Since the probe recalls no case, this subset is the full test set, and the scores stay at 0.5853 recall and 0.6275 F1.
| Evaluation subset | cases | Micro-Recall (Ours) | Micro-F1 (Ours) |
|---|---|---|---|
| Full test set | 253 | 0.5853 | 0.6275 |
| Cases with no recalled post-event knowledge | 253 | 0.5853 | 0.6275 |
Since Direct LLM, D2D, the comparable-budget single agent, and ours all run on this backbone, the memory available to one is available to all. This shows that the improvement of ours over the baselines does not come from the model memorizing these events and their outcomes, because that memory is equally available to every baseline. We will add the snapshot table, the audit, and the probe to the revised manuscript.
Response to Reviewer 2i4B
Response to Reviewer 2i4B
Dear Reviewer 2i4B,
Thank you for your constructive feedback. We appreciate your recognition of the 6×6 Impact Space and the design of our multi-agent system.
Regarding your concern about potential data leakage, we have implemented a strict temporal split that accounts for both the models' training/knowledge cutoff dates and the dates of the public-figure statements.
All 759 training cases occurred before January 2025, while all 253 test cases occurred after January 2025. This temporal separation ensures that the test events and their consequences do not leak into the training process.
The training/knowledge cutoff dates of the models used in our experiments are listed below. None of these dates is later than January 2025.