acceptodds
Under review as a conference paper at ICLR 2027

PreSage: Pre-release anticipation of public-figure Statement impact by multi-agents

Abstract

Pre-release anticipation of adverse impacts from public-figure statements supports proactive risk governance, yet prior work focuses on post-circulation intervention with limited consequence types and lacks multi-dimensional impact datasets. Via LLM-assisted annotation and human adjudication on decade-long real-world cases, we construct a 6×6 impact space and PreSage-Bench for public-figure statement risk assessment. PreSage-Bench contains 1,012 real cases covering 943 public figures (2015–2026) across six social scenarios and enables genuine foresight risk evaluation instead of retrospective post-hoc reasoning. Furthermore, we propose PreSage, a self-improving multi-agent system for pre-assessing statement-induced risks. It integrates nine role agents, an Evidence Judge, and an Impact Judge for joint prediction. The system distills recurring prediction errors from finished cases into generalized rules to refine agents’ skills and prompts. Critically, all predictions rely only on information available before statement release. Evaluated on 250 post-2025 test cases with valid outputs from all baselines, PreSage achieves a micro precision of 0.677 and micro recall of 0.586. Compared with the recall (0.318) of directly using LLM prompting, our PreSage can support genuine foresight risk evaluation of public-figure statements rather than retrospective reasoningThe dataset and code will be released after the paper is accepted..

Primary area:
datasets and benchmarks
Keywords:
Impact prediction, Multi-agent simulation, Speech-risk benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2i4BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PreSage-Bench, a dataset of 1,012 public-figure statements from 2015–2026 annotated with a 6×66\times6 affected-entity × impact-domain space, together with a risk-evidence inventory. It further proposes PreSage, a multi-agent system consisting of nine role agents, an Evidence Judge, and an Impact Judge. A self-evolution procedure uses training-case prediction errors to propose prompt edits that are accepted or rejected using a fixed development-set gate. The evaluation uses a temporal case split at 1 January 2025 and reports results on post-2025 cases. The full PreSage configuration increases recall substantially relative to direct prompting, while its advantage over the dual-layer ablation in micro-F1 is small. The authors also evaluate prompt evolution and several API backends.

Critical strengths

Representing adverse impact as an affected entity × impact domain is more informative than a binary toxicity or controversy label, and the benchmark covers a relatively large set of real public-figure cases across multiple domains. Separating heterogeneous role agents, evidence adjudication, impact prediction, and prompt evolution is sensible. The development-set acceptance gate for prompt updates is also considerably more disciplined than unconstrained prompt self-modification.

Critical weaknesses

Restricting the prompt to information available before the statement’s release is not sufficient when the underlying LLM may already know the statement and its aftermath. The inputs contain the public figure’s identity, exact statement, release date, scene, and platform, making retrieval from parametric memory particularly plausible. A case-level temporal split at January 2025 separates prompt-evolution examples from test examples, but it does not establish that Qwen-plus, Gemini-2.5-flash-lite, or GPT-4o-mini were trained only on information preceding each test event. Consequently, the current results cannot distinguish genuine anticipation from memorized retrospective knowledge. This directly challenges the paper’s central “genuine foresight” claim. The abstract makes that claim explicitly.

Questions for authors

What are the exact training/knowledge cutoffs or immutable model snapshot dates for Qwen-plus, Gemini-2.5-flash-lite, and GPT-4o-mini? Can the authors demonstrate that each evaluated event occurred strictly after the corresponding model’s possible training-data cutoff?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2i4B

Response to Reviewer 2i4B

Dear Reviewer 2i4B,

Thank you for your constructive feedback. We appreciate your recognition of the 6×6 Impact Space and the design of our multi-agent system.

Regarding your concern about potential data leakage, we have implemented a strict temporal split that accounts for both the models' training/knowledge cutoff dates and the dates of the public-figure statements.

All 759 training cases occurred before January 2025, while all 253 test cases occurred after January 2025. This temporal separation ensures that the test events and their consequences do not leak into the training process.

The training/knowledge cutoff dates of the models used in our experiments are listed below. None of these dates is later than January 2025.

ModelKnowledge Cutoff Date
Qwen-PlusJanuary 2025
Gemini-2.5-flash-liteJanuary 2025
GPT-4o-miniJuly 2024

Thank you for your response. I acknowledge the clarification that the 759 training cases all occurred before January 2025, while the 253 test cases occurred after this point; I also note the additional information regarding the knowledge cutoff dates for the models used. This helps clarify the temporal partitioning strategy employed between the prompt self-evolution phase and the final evaluation phase. However, my primary concern—the potential for "retrospective knowledge contamination"—remains unaddressed.

First, the temporal split between the training and test sets does not guarantee that the underlying large language model (LLM) was trained exclusively on information available prior to the occurrence of each test event. Specifically, the authors have not provided the exact, immutable version (snapshot) of the Qwen-Plus model used. Official Alibaba Cloud documentation lists multiple Qwen-Plus snapshots released throughout 2025, including one from December 2025. Therefore, the authors' claim that Qwen-Plus has a knowledge cutoff date of January 2025 requires supporting evidence linked to the specific model version actually employed.

Second, the knowledge cutoff date reported by the authors for GPT-4o-mini appears to be factually incorrect. According to official OpenAI documentation, the knowledge cutoff for this model is October 2023, not July 2024 as claimed; July 2024 actually corresponds to the release date of GPT-4o-mini.

Third, while it is true that Gemini-2.5-Flash-Lite has a knowledge cutoff date of January 2025, this does not automatically ensure that all test events occurred strictly after that date. Given that the test set begins on January 1, 2025, the authors must compare the model's precise knowledge cutoff date against the actual occurrence date of each test event to rule out any potential temporal overlap. More importantly, Appendix C.2 of the paper explicitly acknowledges that "exact runtime and model-snapshot equivalence are not established" across different execution environments and model snapshots. This statement creates a clear inconsistency between the authors' current response—which claims that model knowledge cut-off dates were strictly controlled—and the experimental conditions as documented in the paper.

Consequently, I request that the authors provide the following evidence: (1) the precise model snapshot IDs and API call dates used in all experiments, along with authoritative sources for the corresponding knowledge cut-off dates; (2) a case-by-case temporal audit demonstrating that each test event occurred strictly after the verified knowledge boundary of the model used; and (3) targeted experiments addressing the risk of "memorization contamination," specifically by testing whether the base models can directly recall the specific events and their subsequent outcomes from the test set, and by re-evaluating model performance on a subset of cases where no clear evidence of post-event knowledge is detected. It would be even more compelling if the authors could conduct a truly prospective validation—recording and freezing predictions before the consequences of an event have unfolded, and subsequently verifying their accuracy against actual outcomes.

In the absence of such validation, the current experimental results may, in part, reflect the models' existing knowledge of historical events rather than a genuine capacity for ex ante (pre-event) forecasting. Therefore, I believe the authors' current response is insufficient to support the paper's core claim of "genuine foresight," and my initial concerns regarding the soundness of the paper remain unresolved.

We thank you for the detailed follow-up. The knowledge cutoff for GPT-4o-mini is indeed October 2023, earlier than the July 2024 date we had listed, which further rules out leakage of the test events into the backbone.

On Appendix C.2, that note concerns runtime reproducibility across execution environments, not which events a model has seen. Our memorization probe in (3) settles the latter directly, since the models recall none of the test events. We will revise the wording of C.2 so it is not read as a statement about temporal control.

As requested, we provide the following evidence and experiments.

(1) Model snapshots and cutoff sources. We checked the knowledge cutoff dates of our models more carefully and compared them against the dates of our test cases. The results are below.

ModelSnapshotKnowledge cutoffSource
Qwen-plusqwen-plus-2025-01-25On or before 25 Jan 2025 (snapshot release date; Alibaba publishes no separate cutoff)https://www.alibabacloud.com/help/en/model-studio/qwen-plus
Gemini-2.5-flash-litegemini-2.5-flash-lite-preview-09-2025January 2025 (exact day not disclosed)https://aistudio.google.com/docs/models/gemini-2.5-flash-lite-preview-09-2025?hl=en
GPT-4o-minigpt-4o-mini-2024-07-181 October 2023https://developers.openai.com/docs/models/gpt-4o-mini

(2) Case-by-case temporal audit. We compared every test event date against each model's knowledge boundary. Because Gemini-2.5-flash-lite's exact cutoff day is not disclosed, we remove the five test cases dated in January 2025 and re-run the full evaluation. As the table shows, after removal the scores are almost unchanged, and the main result still holds.

Test setcasesMicro-Recall (Ours)Micro-F1 (Ours)
Full2530.58530.6275
After removing the 5 January 2025 cases2480.57360.6254

(3) Memorization probe and re-evaluation on the uncontaminated subset. We probe whether each backbone recalls the test events. For every case we ask the base model whether a named person posted a statement on a given platform on a given date. It may answer only remember, do not remember, or unsure, and we allow no search and no guessing.

  • On the 253 test events, all three models answer do not remember every time.
  • We also build 45 fabricated person, platform, and date combinations and test whether the models have seen them. The models recall none, so the probe raises no false recall.
  • We select 15 well-known events from before each model's cutoff and test whether the models remember them. The results are in the table below.
Probe setcasesRecalled
Test events, each of Qwen-plus, Gemini-2.5-flash-lite, GPT-4o-mini2530
Fabricated combinations (decoys)450
Known pre-cutoff events, Qwen-plus155
Known pre-cutoff events, Gemini-2.5-flash-lite155
Known pre-cutoff events, GPT-4o-mini1510

The experiments above show that the base model has not seen the test events or their outcomes.

We re-evaluate our method on the subset where the probe finds no sign of post-event knowledge. Since the probe recalls no case, this subset is the full test set, and the scores stay at 0.5853 recall and 0.6275 F1.

Evaluation subsetcasesMicro-Recall (Ours)Micro-F1 (Ours)
Full test set2530.58530.6275
Cases with no recalled post-event knowledge2530.58530.6275

Since Direct LLM, D2D, the comparable-budget single agent, and ours all run on this backbone, the memory available to one is available to all. This shows that the improvement of ours over the baselines does not come from the model memorizing these events and their outcomes, because that memory is equally available to every baseline. We will add the snapshot table, the audit, and the probe to the revised manuscript.

Reviewer_tVqPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PreSage, a multi-agent system that predicts the negative consequences of public figures’ statements before release. It combines role-based discussion, two judges, and feedback-driven prompt updates. The authors also introduce a dataset of public statements and associated impact labels.

Critical strengths

The paper introduces a dataset of public figures’ statements and associated impacts, which could be useful for future research on statement-related risks.

Critical weaknesses

I think the paper offers limited novelty. The method mainly combines existing multi-agent discussion and reflection-based prompt optimization techniques. It is not clear what new technical contribution this combination provides.

The ablation studies and baseline comparisons are insufficient. In particular, the paper does not show whether the method outperforms a single agent with carefully designed prompts or a single agent given a comparable budget for iterative prompt optimization. These comparisons are needed to establish whether the gains come from the proposed design rather than additional model calls and prompt tuning. The reported improvements also lack statistical significance tests, making the small gains difficult to assess.

Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer tVqP

Response to Reviewer tVqP

Dear Reviewer tVqP,

Thank you for your constructive feedback. We appreciate your recognition of the value of our dataset for future research. We have addressed your concerns about experimental comparisons through additional experiments. We also provide detailed explanations below to clarify the contributions of our work.

W1. Novelty and Key Contributions

We respectfully argue that our work makes distinct and novel contributions. Specifically, our contributions are twofold:

  • We construct the first dataset of inappropriate public-figure statements and their consequences across multiple industries. Our dataset contains over 1,000 real-world cases spanning 2015–2026. To the best of our knowledge, it is the largest and most comprehensive dataset of its kind to date.

  • We introduce a structured framework for evaluating the consequences of public-figure statements. Our proposed 6×6 Impact Space characterizes these consequences along six categories of affected entities and six impact domains, enabling a more comprehensive assessment of adverse impacts than conventional single-category risk detection tasks.

Building on these contributions, we develop PreSage, a multi-agent system for impact prediction and consequence classification. A key innovation lies in the design of task-specific Role Agents and the empirical justification of our role selection. Each Role Agent can independently identify correct risk evidence that other roles do not identify, as shown below.

Role AgentUnique Correct Evidence Count
critic42
legal_regulatory_specialist71
neutral_observer86
partner_decision_maker18
public_figure_pr_team85
same_field_competitor62
same_field_peer55
supporter87
targeted_party52

As you noted, multi-agent system builds on established techniques. Instead, our task-specific Role Agent design enables PreSage to effectively leverage our dataset and impact evaluation framework to improve the prediction of consequences arising from public-figure statements.

W2. Computational Budget Fairness

To address your suggestion, we have added a single-agent baseline with carefully designed prompts and a comparable budget for iterative prompt optimization. The results are presented below.

MetricSingle AgentOursImprovement 95% Confidence Interval
Precision0.77800.6763[−0.1281, −0.0749]
Recall0.28020.5854[+0.2815, +0.3297]
Micro-F10.41200.6276[+0.1912, +0.2412]
FPR0.03480.1220[+0.0753, +0.0991]

W3. Statistical Significance of Experimental Results

We agree that statistical significance is important for assessing the reliability of experimental results. Accordingly, we have supplemented all experimental results with 95% confidence intervals and p-values.

The statistical significance results are summarized below.

ComparisonMetricBaselineOursImprovementBootstrap 95% CI
Before vs. After EvolutionPrecision0.63820.6763+0.0382**[+0.0189, +0.0583]
Recall0.57490.5854+0.0105[−0.0132, +0.0353]
Micro-F10.60490.6276+0.0227*[+0.0057, +0.0406]
FPR0.14190.1220−0.0199**[−0.0324, −0.0083]
Comparable Agent vs. PreSagePrecision0.77800.6763−0.1017***[−0.1281, −0.0749]
Recall0.28020.5854+0.3052***[+0.2815, +0.3297]
Micro-F10.41200.6276+0.2156***[+0.1912, +0.2412]
FPR0.03480.1220+0.0872***[+0.0753, +0.0991]
Direct Qwen vs. PreSagePrecision0.73930.6773−0.0620***[−0.0882, −0.0358]
Recall0.31770.5858+0.2681***[+0.2453, +0.2907]
Micro-F10.44440.6282+0.1838***[+0.1655, +0.2020]
FPR0.04860.1211+0.0725***[+0.0603, +0.0849]
D2D vs. PreSagePrecision0.71110.6773−0.0338[−0.0924, +0.0259]
Recall0.05880.5858+0.5270***[+0.5009, +0.5538]
Micro-F10.10850.6282+0.5197***[+0.4978, +0.5412]
FPR0.01040.1211+0.1107***[+0.0973, +0.1244]

Note: Improvements are calculated as Ours minus Baseline. Statistical significance is determined using Holm-adjusted permutation p-values (p < 0.05: *, p < 0.01: **, p < 0.001: ***). Results without asterisks are not statistically significant. CI means confidence interval. The reported metrics may vary slightly across comparisons due to differences in the number of cases successfully processed by each method. These minor variations do not affect our overall conclusions.

Reviewer_UPudOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PreSage-Bench: 1,012 public-figure statements from 11 years. Each one is labelled with risk evidence (67 types used) and with a set of adverse impacts. An adverse impact is one of 36 cells: 6 affected parties (self, targeted party, organization, partner, audience, field) × 6 domains (reputation, career, safety, economic, legal, value). Labels come from an LLM first pass followed by human verification. PreSage works in three steps: Nine role agents (PR team, targeted party, peers, supporters, detractors and so on) debate for three rounds using only pre-release information. An Evidence Judge settles the risk evidence and an Impact Judge predicts the set of impacts. They evaluate the method on Qwen-plus.

Critical strengths
  • task well motivated: predicting which parties are harmed and how, instead of whether a statement is toxic. Separating "who" from "what domain" is a sensible way to structure the labels.
  • The temporal split for pre-release information is well designed and a real-case scenario
  • detailed reporting across amny dimensions is helpful to understand harm in more detail
Critical weaknesses
  • the significance of the results lack a decent reference. a constant predictor that ignores the input beats PreSage. how can this be? Also what about other frontier model, its unclear what the max would be.
  • while multi-agent setting is nice and relevant its unclear how single agents would perform and whichb agent is actually the most crucial part.
  • the annotation is not entirely clear to me, how were the LLM annotations obtained?
Questions for authors

please see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer UPud

Response to Reviewer UPud

Dear Reviewer UPud,

Thank you for your constructive feedback. We appreciate your recognition of the motivation and significance of our task, the soundness and informativeness of distinguishing "who is affected" from "in what domain," and the soundness of our temporal dataset split. We would like to clarify several points that may have led to misunderstandings.

W1. Constant Predictor and Evaluation Metrics

Our task is not conventional single-label binary classification, but multi-label prediction across 36 impact cells. As shown in the table below, the constant predictor, which relies on the most frequent labels, does not outperform our method.

MetricConstant PredictorOurs
Micro-Precision0.61320.6763
Micro-Recall0.55270.5854
Micro-F10.58140.6276
Micro-FPR0.15160.1220

We share your concern regarding the constant predictor, as it underscores the importance of evaluating performance across multiple metrics rather than relying on a single measure. In our task, Recall and F1-score are particularly important. Recall helps assess the system's ability to identify actual adverse impacts, while F1-score balances Precision and Recall. Our method achieves a Recall of 0.5858 and an F1-score of 0.6282, ranking first among all compared methods.

W2. Evaluation with Newer Frontier Models

To prevent models from having prior knowledge of test events and their consequences, we require the prediction models to have been released before the public-figure controversies in the test set occurred.

If a model has already encountered these events during training, it becomes impossible to determine whether improved prediction performance stems from our method or from the model's memorization of the correct outcomes. We will make this distinction explicit in the revised manuscript.

W3. Single-Agent Comparisons and Budget Fairness

Section 5.2 of our paper already provides a comparison between PreSage and Direct LLM. Across the same 250 test cases, our multi-agent approach achieves improvements of 0.5854 − 0.3177 in Micro Recall and 0.6276 − 0.4444 in Micro F1 over the single-agent baseline. These results demonstrate that our approach identifies more actual adverse impacts and delivers better overall prediction performance.

To further address the concern about computational budget, we have added comparable-budget single-agent experiments to the main text. These experiments use the same backbone model and task inputs, while equipping the single agent with prompt designs and a self-evolution process comparable to those of PreSage. This setup enables a fairer comparison of single-agent and multi-agent approaches on the same task. The results are presented below.

MetricComparable BudgetOursImprovement 95% Confidence Interval
Precision0.77800.6763[−0.1281, −0.0749]
Recall0.28020.5854[+0.2815, +0.3297]
Micro-F10.41200.6276[+0.1912, +0.2412]
FPR0.03480.1220[+0.0753, +0.0991]

W4. LLM-Assisted Data Collection and Annotation

Our dataset construction follows a pipeline that combines LLM-based data collection with human expert verification.

Data Collection

  • LLM-Based Collection: We first identify the social scenarios to be covered and then use ChatGPT-5.6 Medium to search for inappropriate public-figure statements and their actual consequences across these scenarios. All collected cases must be authentic and verifiable. Specifically, we require accurate identification of public figures, preservation of the original statements, verification of actual consequences against reliable sources, and documentation of release dates, social scenarios, original publication platforms, and source links.

  • Human Verification: We use a questionnaire to verify the existence of the public figures, the accuracy of their identities, the completeness of their statements, and other relevant details. We exclude inaccurate or incomplete cases based on the questionnaire responses.

Annotation

  • LLM-Based Annotation: For each event, we instruct the LLM to select the most appropriate impact cells from the 6×6 impact matrix. We require multiple rounds of verification, along with explicit factual evidence and annotation rationales, to ensure that the results are traceable and verifiable.

  • Human Verification: The questionnaire also examines whether the reported consequences can be verified and whether any consequences have been omitted. We use the responses to identify missing or incorrectly assigned impact cells.

thanks for the reply, really appreicated.

W1: while agreed that F1 is important and your method is ahead this constant predictor, the gap is still small. The results stays unconvincing this way

W2: this is a pretty valid argument. At the same time, its hard to disentangle simply knowledge cutoff and capability. I would suggest thinking about a setting where frontier models have no access to and still can make use of their capabilities

W3,W4: thanks for adding those results/clarifications, they look convincing to me.

Overall, I tend to keep my score, but am also willing to increase it, in case the remaining points are addressed.

Thank you for the follow-up. On W1 and W2:

W1. The constant predictor is strong because the label distribution is head-heavy. 9 of the 36 elements hold most positives, and it outputs those 9 for every case and nothing else, so it scores 0 on all 27 long-tail elements. Micro-F1 is dominated by the head, which is why the gap looks small. However, the per element gap is large. Our method recovers the long tail and reaches non-zero F1 on 16 of the 27 elements where the constant predictor is 0.

Long-tail element (constant predictor = 0)Gold positives (of 249 cases)Ours RecallOurs F1
targeted party × legal470.5960.459
self × legal930.4950.495
audience × value640.5160.400
field × legal760.3420.380

On the 9 head elements our method also cuts the false-positive rate from 1.000 to 0.383. The constant predictor already beats every other baseline in micro-F1 (D2D 0.109, Direct LLM 0.444, the comparable-budget single agent 0.412, constant 0.581). Ours is the only system above it, at 0.628.

W2. During the response period we added a memorization probe. We ask each base model directly whether it recalls a given test event and its outcome, with no search allowed. The models recall none of the test events, and they do recall well-known events from before their cutoff, so the probe detects recall when it exists.

Probe setCasesRecalled
Test events (each model)2530
Fabricated person/platform/date decoys450
Known events before cutoff, Qwen-plus155
Known events before cutoff, Gemini-2.5-flash-lite155
Known events before cutoff, GPT-4o-mini1510

The reported numbers therefore already come from the setting you suggest, where the model has no memorized access to the event and predicts from capability alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.