- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies self-evolving LLM agents for disaster response. The authors argue that for harness evolvement, forecast-verification feedback records are often correlated, because forecasts may share inputs, revise one another, or be verified against the same observations; treating them as independent can lead to harmful edits. They introduce Streaming², a benchmark of 1,000 historical disaster events with 10,000 action-selection and spatial-analysis tasks, in which evidence is released over time and forecast lineage is annotated. They propose FLARE, which encodes lineage and content residuals with a graph encoder, generates complete harness edits with a fine-tuned LLM, and retains an edit only if a learned predictor, trained on measured future gains, estimates a positive gain. On the held-out split, FLARE outperforms other baselines.
Critical strengths
- The comparison against other baselines is comprehensive. The paper compares against twelve offline and online evolution baselines and includes ablations of representation, generation and selection.
Critical weaknesses
-
The benchmark documentation is not clear. The paper does not explain how the 1000 events were selected or how the 10000 tasks were generated. Reference answers are produced with LLM assistance and approved by human experts, but the number of experts, their qualifications, and inter-annotator agreement are not reported.
-
There is possible knowledge leakage. The events are historical, while the executor is a recent LLM likely trained on their outcomes. The release protocol controls which documents the agent sees, but not what the model already knows, which weakens the benchmark's core premise of deciding with only currently available evidence.
Questions for authors
- How were the 1,000 events selected and the 10,000 tasks generated?
- To build the benchmark, how many domain experts were involved, and with what expertise? How about the inter-annotator agreement?
- What were the total GPU-hours for FLARE training, and what was each baseline's budget? How does Text-Gain perform when given the same RL stage?
- How does FLARE behave on the 40% of records without lineage links? Does performance degrade on events with low lineage coverage?
- The improvements against some strong baselines may not be statistically significant, could the authors report the mean ± std over the five test streams for Table 1?
- In Table 1, FLARE exactly matches EvoSkill on Init., Plan, Ext., and Obj. Could the authors explain these ties?
- The events are historical, and the executor was likely trained on their outcomes. How do the authors rule out that the model answers from memorized knowledge rather than from the evidence available at each decision point?