acceptodds
Under review as a conference paper at ICLR 2027

DynaWeave: Learning to Adapt through Dynamic Scenario Synthesis

Abstract

Large language model (LLM) agents increasingly use tools to complete tasks in stateful environments. However, many training environments remain largely static apart from changes caused by the agent’s own actions, whereas external events introduced by other users, services, or sudden contingencies in real environments can invalidate ongoing plans. To address this gap, we introduce DynaWeave, an automated synthesis framework that transforms existing static tasks into verifiable dynamic scenarios. The framework generates exogenous events relevant to each task, covering changes in task requirements, environment state, and external information. It uses directed acyclic graphs (DAGs) to represent dependencies between events and an execution engine to activate them during agent interaction. A dynamic verifier revises only the evaluation checks affected by events that actually occur and retains all other checks. Using DynaWeave, we collect 2,095 verified interaction trajectories across 36 environments and use them for supervised fine-tuning of Qwen3.5 models. Experiments on Gaia2, CostBench, and ToolMaze show that DynaWeave achieves consistent improvements in overall performance across all three evaluated model sizes, increasing the task completion rate on CostBench by 67.34 percentage points and even exceeding the performance of Gemini-3.1-Pro-Preview on ToolMaze. These results support dynamic scenario synthesis as a practical approach to training agents to adapt to changing environments. Our code is available at https://anonymous.4open.science/r/Anonymous-DynaWeave-81B9.

Primary area:
datasets and benchmarks
Keywords:
LLM agent, Dynamic environment, Tool-use benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gfeAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DynaWeave, a framework for automatically transforming static LLM agent tasks into verifiable dynamic scenarios. It generates task-relevant exogenous events, models their dependencies using DAGs, controls their execution during agent interactions, and dynamically adapts task verifiers to reflect environmental changes. Using 2,095 verified trajectories collected across 36 environments, the authors fine-tune Qwen3.5 models of different sizes. Experiments on Gaia2, CostBench, and ToolMaze demonstrate consistent improvements in overall performance.

Critical strengths
  1. The research problem is practically relevant and clearly distinguished from existing work on agent environment synthesis.
  2. The paper proposes a relatively comprehensive and reusable framework for dynamic scenario generation.
  3. The experiments cover multiple model scales and different types of dynamic benchmarks, demonstrating promising generalization potential.
Critical weaknesses
  1. Lack of a direct comparison with static-task SFT. The paper does not sufficiently demonstrate the unique contribution of training on dynamic scenarios. In particular, the substantial improvement of the 2B model on CostBench, from 3.00% to 70.34%, may partly result from the base model's limited tool-use capabilities rather than improved adaptation to dynamic environments. Therefore, the absence of a static SFT baseline significantly weakens the evidence supporting the paper's central claim.

  2. Insufficient direct evidence of improved dynamic adaptation capabilities. Although the paper aims to improve agents' ability to adapt to environmental changes, its evaluation primarily relies on overall task success rates and composite metrics. Notably, on Gaia2's Adaptability metric, both the 2B and 4B models remain at 0 before and after training, while the 9B model improves only from 0 to 2.1. Most of the gains in the overall score come from the Execution and Search metrics, raising concerns about whether the reported improvements truly reflect enhanced dynamic adaptation.

  3. Insufficient quantitative validation of the correctness of generated dynamic scenarios and verifiers. For example, the paper does not clearly report the validity rate of generated events, the false positive and false negative rates of dynamic verifiers, or the consistency of verification rules when multiple events occur sequentially. Furthermore, it does not quantify how many successful training trajectories actually involve events that invalidate the original plan. These issues directly affect the quality of the synthetic data and whether the training trajectories genuinely demonstrate meaningful adaptation to dynamic changes.

  4. Lack of targeted ablation studies on key design components. The paper does not sufficiently establish the necessity of its core components. In particular, it lacks a comparison with event generation strategies that do not explicitly model event dependencies, as well as analyses of the individual contributions of the three trigger mechanisms and three observability modes to the agents' capabilities.

Questions for authors

See Weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer gfeA (Part 1)

We extend our sincere gratitude for your meticulous and constructive feedback. Your insightful observations and valuable recommendations have greatly contributed to improving the rigor and clarity of our work!

W1: Lack of a direct comparison with static-task SFT.

Thank you for this insightful suggestion! To distinguish general SFT gains from the contribution of dynamic training, we have added static-task SFT baselines, using the same source environments and tasks with dynamic events removed. The results are shown in the table below, where G/C/T denote Gaia2/CostBench/ToolMaze.

ModelG‑Exec.G‑SearchG‑Ambig.G‑Adapt.G‑TimeG‑OverallC‑Gap ↓C‑TCRC‑ITUR ↓T‑Avg
Qwen3.5-4B6.310.40.00.00.03.30.65072.4113.9826.26
+ static-task SFT7.613.10.80.00.04.30.75580.096.2822.50
+ DynaWeave8.314.62.10.00.05.00.33190.812.4729.99
Qwen3.5-9B6.316.70.00.00.04.60.75586.767.0830.59
+ static-task SFT8.723.21.30.00.06.70.28288.292.0625.96
+ DynaWeave10.426.03.12.11.08.50.14092.390.0032.28

Static-task SFT improves CostBench task completion over the base models, demonstrating the benefits of ordinary task training. DynaWeave provides further gains: compared with static-task SFT, it increases TCR by 10.72 and 4.10 percentage points for 4B and 9B, respectively, while also reducing Cost Gap and invalid tool use. These results suggest that dynamic training helps models complete tasks more reliably and efficiently under changing conditions. On ToolMaze, static-task SFT lowers Avg relative to the base models, whereas DynaWeave outperforms both baselines. This contrast suggests that improvements in general task execution do not necessarily translate into better recovery from perturbations. Together, these findings demonstrate that DynaWeave improves models’ ability to adapt to dynamic environments beyond the general benefits of ordinary SFT.

W2: Insufficient direct evidence of improved dynamic adaptation capabilities.

Thank you for raising this important concern! To better understand the low scores, we analyzed DynaWeave-4B and DynaWeave-9B trajectories on Gaia2’s Adaptability and Time tasks. Among failed rollouts on tasks with external events, 79.4% ended before the first event occurred. These early failures suggest that end-to-end scores may not fully reflect models’ ability to handle events once they encounter them. These tasks are also challenging for larger models: Llama-3.3-70B-Instruct achieves 1.9% on Adaptability and 0.4% on Time, while Grok-4 and GPT-5 (high) both score 0% on Time.

To evaluate behavior after an external event more directly, we conduct Oracle-Prefix Evaluation. We execute the ground-truth actions up to the first external event, then let the model continue autonomously using the executed history and observations, without revealing future reference actions. To control for general tool-calling improvements from SFT, we also include a static-task SFT baseline trained on trajectories collected from the same source environments and tasks with dynamic events disabled. The results are shown in the table below.

ModelAdapt.Adapt. (Oracle)TimeTime (Oracle)
Llama3.3-70B-Instruct1.90—0.40—
Grok-44.40—0.00—
GPT-5 (high)40.40—0.00—
Qwen3.5-4B0.006.880.000.00
+ static-task SFT0.0013.120.002.14
+ DynaWeave0.0026.870.007.86
Qwen3.5-9B0.0012.500.007.14
+ static-task SFT0.0021.880.007.14
+ DynaWeave2.1036.251.0011.43

Under this evaluation, DynaWeave outperforms the base models and static-task SFT on both Adaptability and Time, indicating that it helps models adapt more effectively to external events. Besides, the gains from CostBench and ToolMaze in Table 1 also support this claim, since CostBench evaluates the ability to replan under changing conditions, while ToolMaze’s PRR measures recovery from perturbations. DynaWeave outperforms both the base model and static-task SFT on these evaluations. We also present two case studies in Appendix E showing how DynaWeave responds to external events and adjusts its actions. The corresponding figure is also available at https://anonymous.4open.science/r/Anonymous-DynaWeave-81B9/examples/case_study.pdf.

AuthorsOpenReview ↗

Response to Reviewer gfeA (Part 2)

W3: Insufficient quantitative validation of the correctness of generated dynamic scenarios and verifiers.

Thank you for highlighting this issue!

(1) Event validity. We generated 4,928 event candidates, of which 3,414 passed verification, yielding an acceptance rate of 69.28%. The remaining 1,514 candidates were excluded because they violated dependency constraints, were implausible, or had insufficient impact on the original plan.

(2) Verifier correctness. Before verifier corrections, the false positive rate (FPR) was 11.11% and the false negative rate (FNR) was 21.95%, measuring incorrect acceptance and incorrect rejection, respectively.

(3) Sequential consistency. All multi-event trajectories passed checks of event ordering, dependencies, and verifier activation. These checks confirm that events follow their specified dependencies and that the corresponding verification rules are activated consistently across sequential events.

(4) Plan-invalidation criterion. During generation, we explicitly filter out event candidates judged to leave the original plan unaffected. This criterion aims to retain scenarios that require agents to adjust their actions in response to dynamic events.

W4: Lack of targeted ablation studies on key design components.

Thank you for the valuable question! We have added ablations of event dependencies, trigger mechanisms, and observability modes, using equally sized training sets of 1,048 examples. The results are shown in the tables below, where CB/TM denote CostBench/ToolMaze.

Training configurationCB‑Gap ↓CB‑TCR ↑CB‑ITUR ↓TM‑NP ↑TM‑TSR ↑TM‑RC ↓TM‑Avg ↑
Full (DynaWeave)0.28687.081.2761.7524.1373.2928.91
Event dependencies
Independent Events1.67368.241.3256.2519.1277.9524.17
Trigger mechanisms
w/o Action Completion0.71975.853.9054.5015.8179.2022.57
w/o Virtual Time0.28677.693.2257.2517.6978.9723.37
w/o Random Activation0.29281.601.9758.0019.2577.3024.70
Observability modes
w/o Loud1.98379.794.7657.5016.4480.1821.91
w/o Silent0.71981.081.5958.7518.3177.2024.69
w/o Partial0.39488.451.9557.0018.1278.9923.23

These results suggest that all three components support complementary aspects of adaptation. Explicitly modeling event dependencies helps models learn to revise plans when changes affect subsequent actions. Diverse triggers expose agents to different event timings, with the largest drop from removing action-completion triggers highlighting the value of changes tied to ongoing execution. Loud notifications provide explicit signals for replanning, while the benefits of retaining Silent and Partial observations suggest that learning to inspect the environment and reason under incomplete information also improves adaptation. Together, these findings suggest that effective adaptation benefits from learning how changes propagate, when they occur, and how they can be detected.

AuthorsOpenReview ↗

Kind Reminder of Approaching Rebuttal Deadline

We greatly appreciate the time and effort you've invested in reviewing our work! With the rebuttal deadline nearing, we would truly value any additional feedback you could provide. We are keen to resolve any remaining issues you might identify to improve our submission further, your suggestions are crucial to refining our work!

Reviewer_CdT2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DynaWeave, a framework that converts existing static tool-use tasks into dynamic scenarios with exogenous events. It generates events and their dependencies, triggers them during agent execution, and adapts the task verifier. The authors use 2,095 verified interaction trajectories from 36 environments to supervise fine-tuning of Qwen3.5 models with 2B, 4B, and 9B parameters. All three model sizes improve on the overall metrics of Gaia2, CostBench, and ToolMaze after training.

Critical strengths
  1. The paper integrates event generation, dynamic execution, and verifier adaptation into a reusable workflow while preserving the original tool interface. Overall scores improve across three model sizes and different evaluation environments, suggesting that the generated training data are useful.
Critical weaknesses
  1. The central experiments do not directly show that the models learned to adapt to changes. Although the overall scores in Table 1 improve, the Gaia2 Adaptability score, the metric most directly related to adaptation, remains at zero before and after training for the 2B and 4B models, and reaches only 2.1 for the trained 9B model. On ToolMaze, the 4B model's task success rate under one implicit output corruption condition decreases from 9.00% to 8.50%.

  2. Several key methodological and evaluation details are insufficiently specified. The paper reports the teacher model and fine-tuning parameters, but does not clearly specify the model configurations used to generate events and verifiers, the exact rules for programmatic checks, or the number of evaluation tasks and repeated runs. Table 1 also states that the definition of ToolMaze's composite score appears in Appendix A, but the appendix does not provide an explicit formula.

Questions for authors
  1. How do the authors explain the improvement in overall scores alongside the near-zero Gaia2 Adaptability scores and the decline under some implicit-error conditions in ToolMaze? Can they provide an analysis that directly measures whether the models revise their plans after an event occurs?

  2. Can the authors provide the model configurations used for event and verifier generation, the rules for programmatic checks, the number of evaluation tasks and repeated runs, and the formula for ToolMaze's composite score?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CdT2 (Part 1)

We are deeply grateful for your thorough and insightful feedback. Your expertise and dedicated time have significantly contributed to improving the quality of this work! We have revised the paper based on your feedback and updated the PDF accordingly.

W1.1: Marginal improvements on Gaia2 Adaptability score.

Thank you for raising this important concern! To better understand the low scores, we analyzed DynaWeave-4B and DynaWeave-9B trajectories on Gaia2’s Adaptability and Time tasks. Among failed rollouts on tasks with external events, 79.4% ended before the first event occurred. These early failures suggest that end-to-end scores may not fully reflect models’ ability to handle events once they encounter them. These tasks are also challenging for larger models: Llama-3.3-70B-Instruct achieves 1.9% on Adaptability and 0.4% on Time, while Grok-4 and GPT-5 (high) both score 0% on Time.

To evaluate behavior after an external event more directly, we conduct Oracle-Prefix Evaluation. We execute the ground-truth actions up to the first external event, then let the model continue autonomously using the executed history and observations, without revealing future reference actions. To control for general tool-calling improvements from SFT, we also include a static-task SFT baseline trained on trajectories collected from the same source environments and tasks with dynamic events disabled. The results are shown in the table below.

ModelAdapt.Adapt. (Oracle)TimeTime (Oracle)
Llama3.3-70B-Instruct1.90—0.40—
Grok-44.40—0.00—
GPT-5 (high)40.40—0.00—
Qwen3.5-4B0.006.880.000.00
+ static-task SFT0.0013.120.002.14
+ DynaWeave0.0026.870.007.86
Qwen3.5-9B0.0012.500.007.14
+ static-task SFT0.0021.880.007.14
+ DynaWeave2.1036.251.0011.43

Under this evaluation, DynaWeave outperforms the base models and static-task SFT on both Adaptability and Time, indicating that it helps models adapt more effectively to external events. Besides, the gains from CostBench and ToolMaze in Table 1 also support this claim, since CostBench evaluates the ability to replan under changing conditions, while ToolMaze’s PRR measures recovery from perturbations. DynaWeave outperforms both the base model and static-task SFT on these evaluations. We also present two case studies in Appendix E showing how DynaWeave responds to external events and adjusts its actions. The corresponding figure is also available at https://anonymous.4open.science/r/Anonymous-DynaWeave-81B9/examples/case_study.pdf.

W1.2: Decline in the 4B Model’s Task Success Rate on ToolMaze P3.

Thank you for the insightful question! ToolMaze P3 requires both detecting corrupted outputs actively and taking effective corrective actions. The 4B traces show that the model sometimes recognizes an error but continues without verification or skips the affected step. For example, it may directly skip this task, or just state there is something wrong without addressing it. This explains the slight decline. In comparison, 2B makes substantially more tool calls (7.37→18.98 on average), suggesting greater reliance on trial-and-error recovery, while 9B identifies errors and successfully selects alternative tools. These differences may reflect the training coverage. Silent events in training data encourage modes to actively inspect tool results for unannounced changes, which may help them recognize tool errors. However, the training data do not explicitly include tool failures or output corruption and therefore do not directly teach the corresponding recovery strategies. This may explain why error recognition transfers to these settings while effective correction remains inconsistent across models.

AuthorsOpenReview ↗

Response to Reviewer CdT2 (Part 2)

W2 & Q2: Insufficient configuration and evaluation details.

Sorry for these omissions. We have added the following clarifications to the revised PDF manuscript.

(1) Generation configuration. We use Qwen3.5-Plus to generate dynamic event candidates, specify their execution conditions, and adapt the corresponding verifiers. GPT-6 Luna independently reviews the plausibility and task relevance of the events, as well as the consistency of the associated verifier adaptations.

(2) Programmatic checks. We use Python code to validate required fields, data types, and allowed values, and to confirm that referenced tools, state paths, and original verifier checks exist. These checks also reject incompatible or ineffective state modifications and verify that goal changes are authorized. For event DAGs, we check reference uniqueness, acyclicity, and compatibility between event effects and preconditions.

(3) Evaluation tasks and repeated runs. Gaia2 contains 800 evaluation tasks, with 160 tasks in each of its five categories. CostBench contains 381 queries per evaluation condition. ToolMaze contains 400 underlying tasks, with 100 tasks at each of four complexity levels. We conduct three evaluation runs and report the mean and standard deviation.

(4) ToolMaze composite score. We compute the Avg as:

Avg=TSR‾+PRR‾+(100−RC‾)3\mathrm{Avg} = \frac{\overline{\mathrm{TSR}} + \overline{\mathrm{PRR}} + (100 - \overline{\mathrm{RC}})}{3}

where TSR denotes task success rate, PRR perturbation recovery rate, and RC normalized recovery cost. TSR‾\overline{\mathrm{TSR}} averages over all four complexity levels and the unperturbed condition. PRR‾\overline{\mathrm{PRR}} and RC‾\overline{\mathrm{RC}} average over the four complexity levels only. The explicit formula and averaging conventions are provided in Appendix B.1.

AuthorsOpenReview ↗

Kind Reminder of Approaching Rebuttal Deadline

We greatly appreciate the time and effort you've invested in reviewing our work! With the rebuttal deadline nearing, we would truly value any additional feedback you could provide. We are keen to resolve any remaining issues you might identify to improve our submission further, your suggestions are crucial to refining our work!

Reviewer_RAKGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose DynaWeave, a general framework for automatically transforming existing static, executable tool-use tasks into verifiable dynamic variants with exogenous changes occurring during task execution. Experiments on three dynamic benchmarks show consistent improvements across three model sizes, demonstrating the value of DynaWeave for training agents to adapt to changes in dynamic environments.

Critical strengths
  1. The paper is well-presented with critical findings and contributions clearly highlighted.
  2. Authors take great care to present background and settings for readers new to the area.
  3. The methodology and experimental settings are sound.
Critical weaknesses
  1. The paper lacks enough qualitative examples to show the impact of CTA on the tasks when compared to baselines.
  2. The authors are encouraged to discuss valid limitations of their methods and future work.
  3. Authors need to add statistical significance for the results. Current results do not show if the improvements due to DynaWeave is significant.
Questions for authors
  1. The paper lacks enough qualitative examples to show the impact of CTA on the tasks when compared to baselines.
  2. The authors are encouraged to discuss valid limitations of their methods and future work.
  3. Authors need to add statistical significance for the results. Current results do not show if the improvements due to DynaWeave is significant.
2 comments
AuthorsOpenReview ↗

Response to Reviewer RAKG

We are deeply grateful for your insightful and thorough feedback, and we appreciate the recognition of our work's contribution! The suggestions and comments made for our work have significantly helped to improve its quality. We have revised the manuscript in response to your feedback and updated the PDF accordingly.

Q1 & W1: Lacking qualitative examples.

Thank you for your valuable feedback! We have added qualitative case studies to Appendix E to illustrate how DynaWeave improves models’ ability to adapt to dynamic environments. The corresponding figure is also available at https://anonymous.4open.science/r/Anonymous-DynaWeave-81B9/examples/case_study.pdf. In ToolMaze, both models succeed without perturbations; after a permanent tool failure, the baseline skips the required email delivery, whereas our model finds an alternative workflow and completes the task. In Gaia2, the task requires waiting four minutes, replying to all emails received during that interval, and buying one gift per sender 30 seconds later. The baseline responds prematurely and buys only one gift, whereas our model waits for the full interval, replies to all three emails, and buys three gifts after the required delay. These cases illustrate that DynaWeave improves models' adaptation ability in dynamic environments.

Q2 & W2: Limitations and future work discussion.

Thank you for the helpful suggestion! We have expanded the discussion of limitations and future work in Appendix A. First, detecting unreliable tool outputs and responding within precise time windows remain challenging for the trained agents. Future work will expand scenarios requiring information verification and timely replanning, with curricula that progressively increase event complexity. Second, our current framework uses finite event graphs in which each event node is activated at most once. Representing recurring changes therefore requires additional event nodes. We will explore reusable event templates and incremental event composition to support longer interactions with recurring changes while preserving reproducibility and verifier consistency.

Q3 & W3: Statistical significance.

Thank you for pointing this out! We have repeated each evaluation in Table 1 and reported the mean ± standard deviation. The results are shown in the table below, where G/C/T denote Gaia 2/CostBench/ToolMaze:

ModelG‑Exec.G‑SearchG‑Ambig.G‑Adapt.G‑TimeG‑OverallC‑Gap ↓C‑TCRC‑ITUR ↓T‑Avg
Qwen3.5-2B0.0 ± 0.02.1 ± 0.60.0 ± 0.00.0 ± 0.00.0 ± 0.00.4 ± 0.11.998 ± 0.2113.0 ± 0.6319.81 ± 2.8318.52 ± 0.56
+DynaWeave1.0 ± 0.35.2 ± 0.80.0 ± 0.00.0 ± 0.00.0 ± 0.01.3 ± 0.21.242 ± 0.10770.34 ± 2.950.16 ± 0.0323.49 ± 0.54
Qwen3.5-4B6.3 ± 1.010.4 ± 1.30.0 ± 0.00.0 ± 0.00.0 ± 0.03.3 ± 0.30.650 ± 0.14972.41 ± 3.0613.98 ± 1.9226.26 ± 0.45
+DynaWeave8.3 ± 1.114.6 ± 1.62.1 ± 0.30.0 ± 0.00.0 ± 0.05.0 ± 0.40.331 ± 0.05790.81 ± 1.572.47 ± 0.9429.99 ± 0.85
Qwen3.5-9B6.3 ± 1.516.7 ± 1.80.0 ± 0.00.0 ± 0.00.0 ± 0.04.6 ± 0.40.755 ± 0.21686.76 ± 1.537.08 ± 1.6030.59 ± 0.57
+DynaWeave10.4 ± 1.226.0 ± 1.93.1 ± 0.52.1 ± 0.31.0 ± 0.38.5 ± 0.50.140 ± 0.01292.39 ± 1.620.00 ± 0.0032.28 ± 0.33

The two-sided Welch’s t-tests yielded p-values below 0.05 for three benchmarks across all three model sizes, which indicates statistically significant improvements on these metrics, demonstrating DynaWeave’s effectiveness in improving agent performance in dynamic environments.

AuthorsOpenReview ↗

Kind Reminder of Approaching Rebuttal Deadline

We greatly appreciate the time and effort you've invested in reviewing our work! With the rebuttal deadline nearing, we would truly value any additional feedback you could provide. We are keen to resolve any remaining issues you might identify to improve our submission further, your suggestions are crucial to refining our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.