acceptodds
Under review as a conference paper at ICLR 2027

: Acting on Evidence Streams, Evolving through Disaster Event Streams

Abstract

Disaster-response agents revise their harnesses to improve later decisions by adapting programs and configurations governing evidence and tool use. These updates use feedback from comparing forecasts with later observations. However, forecasts may share inputs or be checked against the same observations. Treating their feedback as independent can overstate support for retaining updates. Existing benchmarks do not trace how these dependencies affect updates and later performance. We introduce Streaming², a benchmark annotating forecast lineage and measuring how updates affect later tasks. Its 1,000 events contain 10,000 action-selection and spatial-analysis tasks, with evidence arriving within events and harnesses evolving across them. Validation experiments show that a baseline agent makes fewer harmful updates when given these annotations. However, some updates still improve performance on past tasks but not on future events. These findings motivate FLARE, which compares content along forecast lineage to generate complete updates and predict their future gains. It learns from measured performance differences between updated and unchanged harnesses on the same later training events. At test time, learned parameters remain frozen while predicted gains guide retention. On Streaming²-Bench, FLARE raises the overall task score by 21.0% relative to the fixed harness and 4.4% relative to MemEvolve, the strongest self-evolving baseline.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, self-evolving agents, continual learning, harness adaptation, streaming benchmarks, disaster response

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tPmeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-evolving LLM agents for disaster response. The authors argue that for harness evolvement, forecast-verification feedback records are often correlated, because forecasts may share inputs, revise one another, or be verified against the same observations; treating them as independent can lead to harmful edits. They introduce Streaming², a benchmark of 1,000 historical disaster events with 10,000 action-selection and spatial-analysis tasks, in which evidence is released over time and forecast lineage is annotated. They propose FLARE, which encodes lineage and content residuals with a graph encoder, generates complete harness edits with a fine-tuned LLM, and retains an edit only if a learned predictor, trained on measured future gains, estimates a positive gain. On the held-out split, FLARE outperforms other baselines.

Critical strengths
  1. The comparison against other baselines is comprehensive. The paper compares against twelve offline and online evolution baselines and includes ablations of representation, generation and selection.
Critical weaknesses
  1. The benchmark documentation is not clear. The paper does not explain how the 1000 events were selected or how the 10000 tasks were generated. Reference answers are produced with LLM assistance and approved by human experts, but the number of experts, their qualifications, and inter-annotator agreement are not reported.

  2. There is possible knowledge leakage. The events are historical, while the executor is a recent LLM likely trained on their outcomes. The release protocol controls which documents the agent sees, but not what the model already knows, which weakens the benchmark's core premise of deciding with only currently available evidence.

Questions for authors
  1. How were the 1,000 events selected and the 10,000 tasks generated?
  2. To build the benchmark, how many domain experts were involved, and with what expertise? How about the inter-annotator agreement?
  3. What were the total GPU-hours for FLARE training, and what was each baseline's budget? How does Text-Gain perform when given the same RL stage?
  4. How does FLARE behave on the 40% of records without lineage links? Does performance degrade on events with low lineage coverage?
  5. The improvements against some strong baselines may not be statistically significant, could the authors report the mean ± std over the five test streams for Table 1?
  6. In Table 1, FLARE exactly matches EvoSkill on Init., Plan, Ext., and Obj. Could the authors explain these ties?
  7. The events are historical, and the executor was likely trained on their outcomes. How do the authors rule out that the model answers from memorized knowledge rather than from the evidence available at each decision point?
Reviewer_QnGUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Streaming$^2$ is a benchmark for evaluating disaster-response agents that make decisions as evidence arrives and evolve their execution harnesses across sequential events. It contains 1,000 disaster events and 10,000 task instances covering action selection and spatial analysis. Based on this benchmark, the authors propose FLARE, which combines a lineage-residual encoder, a language-model-based harness editor, and an edit-gain predictor. The predictor is trained on performance differences between updated and unchanged harnesses over subsequent training events. At test time, learned parameters stay frozen, and the predicted gain determines whether a candidate harness update is kept.

Critical strengths
  1. The paper does more than improve how an agent responds to a single task; it asks whether a proposed update should persist into future tasks. Distinguishing between fixing past tasks and improving performance on later events matters for agents that operate in changing environments.
  2. The proposed approach links evidence availability, forecast lineage, task execution, released feedback, and subsequent harness updates. Comparing updated and unchanged harnesses pairwise on later events is a useful design choice for measuring the consequences of individual updates.
Critical weaknesses
  1. The main novelty of Streaming² lies in combining existing task types with forecast lineage and cross-event harness updates, rather than introducing new tasks for evaluating disaster-response capabilities. Although the benchmark provides a valuable framework for studying dependencies among forecast evidence and persistent agent evolution across events, its underlying tasks, such as action selection, spatial analysis, and resource allocation, overlap with those in existing disaster-response and geospatial agent benchmarks. Moreover, only 60% of the forecast records have verified lineage links, which raises concerns about the completeness of the dependency annotations and their impact on the benchmark's evaluation coverage and broader utility.

  2. FLARE's main contribution is more an integration of existing methods for handling forecast dependencies than a clearly distinct learning mechanism. FLARE combines several established techniques, including heterogeneous graph representation learning, soft-prompt conditioning, executable harness editing, and performance-based update selection. However, related works such as Heterogeneous Graph Transformer, G-Retriever, and Harness-R1 have already explored these components individually.

Questions for authors
  1. Beyond the streaming evaluation protocol and harness evolution, what new agent capabilities or failure modes does Streaming² evaluate that are not covered by existing benchmarks such as DORA, GeoDisaster, and LifelongAgentBench? Are these differences sufficient to establish the benchmark's independent research value?

  2. Only 60% of the forecast records have verified lineage links. Does this incomplete coverage affect the reliability of dependency evaluation? Are the annotated and unannotated records comparable in terms of disaster types and data sources?

  3. To what extent do FLARE's performance gains come specifically from modeling shared evidence dependencies, rather than from additional graph features or model capacity? Is there evidence that the conditional residuals capture information relevant to harmful harness updates?

  4. Is FLARE's 2.9-point improvement over MemEvolve consistent across the five independent test streams? Is this performance difference statistically significant?

Reviewer_QYRCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark Streaming^2. It evaluates whether harness changes improve the performance of disaster-response AI agents on later events. The paper also proposes FLARE, a method for generating and selecting candidate harness updates.

Critical strengths

The benchmark compares updated and unchanged harnesses on the same later events to measure the effect of each update. It also reruns the original tasks. These comparisons help distinguish improvements on past tasks from improvements on later events. This is an important contribution.

Critical weaknesses

Tables 1-3 report point estimates without confidence intervals. This makes it difficult to assess how stable the reported improvements are. Appendix D states that five streams were used for testing. However, these runs only change the random seeds at runtime. They do not show whether the results are robust across independently trained models. The paper describes how experts checked the reference answers, but provides limited quantitative results from these checks. The number of experts and their expertise are not reported. Disagreement rates, correction rates, and exclusion rates are also not reported.

Questions for authors

For Tables 1-3, could you report the results for each test stream and confidence intervals? If possible, could you also repeat training independently several times and check whether the improvements still hold? Could you report the disagreement rates, correction rates, etc. from the experts’ checks of the reference answers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.