- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DynaWeave, a framework for automatically transforming static LLM agent tasks into verifiable dynamic scenarios. It generates task-relevant exogenous events, models their dependencies using DAGs, controls their execution during agent interactions, and dynamically adapts task verifiers to reflect environmental changes. Using 2,095 verified trajectories collected across 36 environments, the authors fine-tune Qwen3.5 models of different sizes. Experiments on Gaia2, CostBench, and ToolMaze demonstrate consistent improvements in overall performance.
Critical strengths
- The research problem is practically relevant and clearly distinguished from existing work on agent environment synthesis.
- The paper proposes a relatively comprehensive and reusable framework for dynamic scenario generation.
- The experiments cover multiple model scales and different types of dynamic benchmarks, demonstrating promising generalization potential.
Critical weaknesses
-
Lack of a direct comparison with static-task SFT. The paper does not sufficiently demonstrate the unique contribution of training on dynamic scenarios. In particular, the substantial improvement of the 2B model on CostBench, from 3.00% to 70.34%, may partly result from the base model's limited tool-use capabilities rather than improved adaptation to dynamic environments. Therefore, the absence of a static SFT baseline significantly weakens the evidence supporting the paper's central claim.
-
Insufficient direct evidence of improved dynamic adaptation capabilities. Although the paper aims to improve agents' ability to adapt to environmental changes, its evaluation primarily relies on overall task success rates and composite metrics. Notably, on Gaia2's Adaptability metric, both the 2B and 4B models remain at 0 before and after training, while the 9B model improves only from 0 to 2.1. Most of the gains in the overall score come from the Execution and Search metrics, raising concerns about whether the reported improvements truly reflect enhanced dynamic adaptation.
-
Insufficient quantitative validation of the correctness of generated dynamic scenarios and verifiers. For example, the paper does not clearly report the validity rate of generated events, the false positive and false negative rates of dynamic verifiers, or the consistency of verification rules when multiple events occur sequentially. Furthermore, it does not quantify how many successful training trajectories actually involve events that invalidate the original plan. These issues directly affect the quality of the synthetic data and whether the training trajectories genuinely demonstrate meaningful adaptation to dynamic changes.
-
Lack of targeted ablation studies on key design components. The paper does not sufficiently establish the necessity of its core components. In particular, it lacks a comparison with event generation strategies that do not explicitly model event dependencies, as well as analyses of the individual contributions of the three trigger mechanisms and three observability modes to the agents' capabilities.
Questions for authors
See Weakness
5 comments
Response to Reviewer gfeA (Part 2)
W3: Insufficient quantitative validation of the correctness of generated dynamic scenarios and verifiers.
Thank you for highlighting this issue!
(1) Event validity. We generated 4,928 event candidates, of which 3,414 passed verification, yielding an acceptance rate of 69.28%. The remaining 1,514 candidates were excluded because they violated dependency constraints, were implausible, or had insufficient impact on the original plan.
(2) Verifier correctness. Before verifier corrections, the false positive rate (FPR) was 11.11% and the false negative rate (FNR) was 21.95%, measuring incorrect acceptance and incorrect rejection, respectively.
(3) Sequential consistency. All multi-event trajectories passed checks of event ordering, dependencies, and verifier activation. These checks confirm that events follow their specified dependencies and that the corresponding verification rules are activated consistently across sequential events.
(4) Plan-invalidation criterion. During generation, we explicitly filter out event candidates judged to leave the original plan unaffected. This criterion aims to retain scenarios that require agents to adjust their actions in response to dynamic events.
W4: Lack of targeted ablation studies on key design components.
Thank you for the valuable question! We have added ablations of event dependencies, trigger mechanisms, and observability modes, using equally sized training sets of 1,048 examples. The results are shown in the tables below, where CB/TM denote CostBench/ToolMaze.
| Training configuration | CB‑Gap ↓ | CB‑TCR ↑ | CB‑ITUR ↓ | TM‑NP ↑ | TM‑TSR ↑ | TM‑RC ↓ | TM‑Avg ↑ |
|---|---|---|---|---|---|---|---|
| Full (DynaWeave) | 0.286 | 87.08 | 1.27 | 61.75 | 24.13 | 73.29 | 28.91 |
| Event dependencies | |||||||
| Independent Events | 1.673 | 68.24 | 1.32 | 56.25 | 19.12 | 77.95 | 24.17 |
| Trigger mechanisms | |||||||
| w/o Action Completion | 0.719 | 75.85 | 3.90 | 54.50 | 15.81 | 79.20 | 22.57 |
| w/o Virtual Time | 0.286 | 77.69 | 3.22 | 57.25 | 17.69 | 78.97 | 23.37 |
| w/o Random Activation | 0.292 | 81.60 | 1.97 | 58.00 | 19.25 | 77.30 | 24.70 |
| Observability modes | |||||||
| w/o Loud | 1.983 | 79.79 | 4.76 | 57.50 | 16.44 | 80.18 | 21.91 |
| w/o Silent | 0.719 | 81.08 | 1.59 | 58.75 | 18.31 | 77.20 | 24.69 |
| w/o Partial | 0.394 | 88.45 | 1.95 | 57.00 | 18.12 | 78.99 | 23.23 |
These results suggest that all three components support complementary aspects of adaptation. Explicitly modeling event dependencies helps models learn to revise plans when changes affect subsequent actions. Diverse triggers expose agents to different event timings, with the largest drop from removing action-completion triggers highlighting the value of changes tied to ongoing execution. Loud notifications provide explicit signals for replanning, while the benefits of retaining Silent and Partial observations suggest that learning to inspect the environment and reason under incomplete information also improves adaptation. Together, these findings suggest that effective adaptation benefits from learning how changes propagate, when they occur, and how they can be detected.
Kind Reminder of Approaching Rebuttal Deadline
We greatly appreciate the time and effort you've invested in reviewing our work! With the rebuttal deadline nearing, we would truly value any additional feedback you could provide. We are keen to resolve any remaining issues you might identify to improve our submission further, your suggestions are crucial to refining our work!
Response to Reviewer gfeA (Part 1)
We extend our sincere gratitude for your meticulous and constructive feedback. Your insightful observations and valuable recommendations have greatly contributed to improving the rigor and clarity of our work!
Thank you for this insightful suggestion! To distinguish general SFT gains from the contribution of dynamic training, we have added static-task SFT baselines, using the same source environments and tasks with dynamic events removed. The results are shown in the table below, where G/C/T denote Gaia2/CostBench/ToolMaze.
Static-task SFT improves CostBench task completion over the base models, demonstrating the benefits of ordinary task training. DynaWeave provides further gains: compared with static-task SFT, it increases TCR by 10.72 and 4.10 percentage points for 4B and 9B, respectively, while also reducing Cost Gap and invalid tool use. These results suggest that dynamic training helps models complete tasks more reliably and efficiently under changing conditions. On ToolMaze, static-task SFT lowers Avg relative to the base models, whereas DynaWeave outperforms both baselines. This contrast suggests that improvements in general task execution do not necessarily translate into better recovery from perturbations. Together, these findings demonstrate that DynaWeave improves models’ ability to adapt to dynamic environments beyond the general benefits of ordinary SFT.
Thank you for raising this important concern! To better understand the low scores, we analyzed DynaWeave-4B and DynaWeave-9B trajectories on Gaia2’s Adaptability and Time tasks. Among failed rollouts on tasks with external events, 79.4% ended before the first event occurred. These early failures suggest that end-to-end scores may not fully reflect models’ ability to handle events once they encounter them. These tasks are also challenging for larger models: Llama-3.3-70B-Instruct achieves 1.9% on Adaptability and 0.4% on Time, while Grok-4 and GPT-5 (high) both score 0% on Time.
To evaluate behavior after an external event more directly, we conduct Oracle-Prefix Evaluation. We execute the ground-truth actions up to the first external event, then let the model continue autonomously using the executed history and observations, without revealing future reference actions. To control for general tool-calling improvements from SFT, we also include a static-task SFT baseline trained on trajectories collected from the same source environments and tasks with dynamic events disabled. The results are shown in the table below.
Under this evaluation, DynaWeave outperforms the base models and static-task SFT on both Adaptability and Time, indicating that it helps models adapt more effectively to external events. Besides, the gains from CostBench and ToolMaze in Table 1 also support this claim, since CostBench evaluates the ability to replan under changing conditions, while ToolMaze’s PRR measures recovery from perturbations. DynaWeave outperforms both the base model and static-task SFT on these evaluations. We also present two case studies in Appendix E showing how DynaWeave responds to external events and adjusts its actions. The corresponding figure is also available at https://anonymous.4open.science/r/Anonymous-DynaWeave-81B9/examples/case_study.pdf.