acceptodds
Under review as a conference paper at ICLR 2027

DWELL: Persist or Wrap Up? Measuring and Governing Dynamical Instability in LLM Multi-Agent Orchestration

Abstract

When tools repeatedly fail, should a large language model (LLM) orchestrator persist or wrap up? Stopping early can lose recoverable successes, while continued retries can exhaust the budget on unsolvable tasks. We study this decision by separating observed progress from the allocation of further retries. We introduce a trajectory-level stability margin to measure progress and a budget-aware dwell-time controller, DWELL, that determines how long to keep retrying from the failure history and remaining budget under a simple recovery model. Experiments show that task success declines gradually as tool failures become more persistent and drops to zero when the last source of required information is lost. Comparisons with fixed stopping rules quantify how earlier stopping trades recoverable successes for resource savings. Within this trade-off, exploratory analysis on the primary frozen faulted workload finds that DWELL reduces tokens per successful task by 27.5% relative to ungoverned orchestration. Tests on new tasks with equal approach show that models stop when instructed and succeed when tools recover before the retry limit. Whether stopping helps depends on the task and how long the model would otherwise keep trying.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM multi-agent systems, agentic orchestration, runtime governance, progress monitoring, tool failures, fault tolerance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_itgqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when an LLM multi-agent orchestrator should continue retrying failed tools versus terminate execution. It introduces a trajectory-level stability margin to characterize task progress and a budget-aware controller, DWELL, that determines stopping behavior based on consecutive failures and remaining execution budget. The authors evaluate the approach through controlled fault injection and comparisons across several LLM deployments.

Critical strengths

The paper addresses a practical resource-management problem in LLM agent systems. Its controlled experiments systematically examine the trade-off between reducing wasted retries and preserving recoverable task success.

Critical weaknesses

1. Limited methodological novelty and contribution to machine learning.

The proposed DWELL controller is essentially a heuristic stopping policy based on the number of consecutive failures and the remaining execution budget. The method does not involve learning a policy from data, nor does it demonstrate a substantive new optimization or adaptive decision-making capability.

Although the paper introduces concepts from dynamical systems and Bayesian modeling, the actual intervention rule reduces to a simple budget-dependent threshold. It is unclear what methodological contribution this provides beyond conventional retry management and resource-allocation heuristics.

2. The proposed stability margin does not meaningfully contribute to the deployed controller.

A particularly concerning result is the authors' own ablation showing that removing the runtime stability margin while retaining the failure-count trigger does not change intervention timing on any evaluated faulted trajectory.

This substantially undermines the claimed integration of dynamical stability measurement and runtime governance. The elaborate stability analysis appears largely disconnected from the actual behavior of the proposed controller.

3. No convincing improvement over simple stopping baselines.

The experiments do not establish that DWELL outperforms straightforward fixed-threshold or budget-based stopping rules.

In the primary four-arm evaluation, DWELL succeeds on only 21 of 40 recoverable tasks, compared with 25 for Fixed-7 and 40 without intervention. Although DWELL reduces resource consumption, the improvement is primarily attributable to earlier termination, which necessarily sacrifices recoverable successes.

Furthermore, the reported token-efficiency advantages over simple stopping baselines are not statistically established. Consequently, the results do not demonstrate a compelling benefit from the proposed controller's additional complexity.

Questions for authors
  1. What is the substantive algorithmic contribution of DWELL beyond a budget-scaled retry threshold?

  2. Given that removing the stability margin does not change intervention timing on the evaluated faulted trajectories, what measurable benefit does the proposed dynamical stability analysis provide to the deployed controller?

  3. Can the authors demonstrate that DWELL outperforms carefully tuned fixed or budget-scaled stopping baselines under matched success-rate constraints and computational budgets?

  4. How does the proposed framework adapt to task-specific recovery probabilities or non-stationary tool failures, given that the implemented stopping rule depends only on failure counts and remaining budget?

Reviewer_iL5tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an LLM orchestrator should continue retrying or wrap up a task under repeated tool failures. The authors separate progress measurement from retry budget allocation. For progress measurement, they construct a task potential from unresolved work, failures, repeated calls, and context load, and define a trajectory-level stability margin and its windowed counterpart. For retry allocation, they assume that calls are conditionally independent given the failure probability and use a uniform Beta prior to estimate the probability of recovery within the remaining call budget. This yields the DWELL rule, which depends on the failure count and remaining budget. Through controlled fault injection, evaluations across multiple model deployments, and fresh-task transfer experiments, the paper examines the resource savings and success losses associated with stopping rules. The main empirical finding is that these rules reduce expenditure on unsolvable tasks but sacrifice some recoverable successes. Their practical value depends on the task mix, available budget, and the model's native persistence. The paper also notes that strong discrimination on completed trajectories does not by itself establish the ability to predict recovery online.

Critical strengths
  1. The research question is well defined, and the overall approach, text, and figures are reasonably clear.
  2. The experiments demonstrate token savings while also showing that early stopping can cause otherwise solvable tasks to fail. The authors openly report this cost, providing a useful basis for future work on balancing resource expenditure against task success.
Critical weaknesses
  1. The practical benefit of the margin over simple rules remains unclear. Although the paper compares the margin with rules based on consecutive steps without progress or consecutive failures, it does not demonstrate an advantage in both earlier detection and fewer false alarms. A high AUC on completed trajectories also does not directly establish greater usefulness during task execution. The authors' replay analysis further shows that removing the margin leaves intervention escalation timing unchanged on faulted trajectories. The current evidence therefore does not clearly establish what the margin adds beyond simple counters when deciding when to intervene.
  2. The appendix's post-hoc analysis suggests that fixed thresholds of 11 or 12 failures could match DWELL's 27/30 successes, but their token costs are not reported. Comparing token usage under the same workload and call budget would clarify whether DWELL provides additional savings.
  3. The experimental settings remain limited. In the main experiments, executors largely return information according to predefined protocols, and the runtime governance experiments focus primarily on repeated call failures. Although the paper considers other fault types, further evaluation is needed to determine whether the method remains effective when progress across multiple branches is interleaved or when failures and successes alternate.
Questions for authors

See weaknesses.

Reviewer_oAemOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper separates observed progress from the decision to continue retrying after tool failures. It introduces a potential-based trajectory margin and an in-band observer, then derives DWELL's budget-indexed retry boundary from a simple Bayesian recovery model. Controlled fault-injection studies and several deployment replications compare ungoverned execution, fixed stopping rules and DWELL, measuring both resource savings and recoverable successes lost. The paper also reports monitor ablations, counterexamples to observability, token-efficiency analyses and a fresh-task transfer experiment with a shared executor-call cap.

Critical strengths
  1. The evaluation directly measures the cost of premature stopping. The four-arm design in Section 5.2 separates different stopping rules and reports success alongside expenditure. For example, Table 1 records 21/40 fragile-task successes for DWELL versus 40/40 for OFF, rather than presenting resource reduction alone as success. Appendix C.7 checks the original call-budget asymmetry, and Appendix C.14 adds a fresh experiment with a common executor-call cap. This is a useful empirical treatment of a practically important trade-off.
  2. The reporting makes the limitations unusually inspectable. The manuscript separates deployment epochs, reports attrition and sensitivity analyses, distinguishes retrospective margin discrimination from decision-time recovery prediction, and labels exploratory efficiency analyses. It also reports that removing the margin does not change faulted-run escalation timing in fixed-tape replay and that the fixed-rule comparisons do not establish a consistent efficiency advantage. These disclosures make the scoped empirical conclusions more credible.
Critical weaknesses
  1. The additional contribution of the proposed monitoring/control machinery is limited by the paper's own ablations. Under consistent counting, the margin reduces to a net log-potential drop, and the alternative executor weighting adds no measurable separation on the tested data (Appendices A.2 and C.2). In replay, removing the margin leaves faulted-run L2/L3 escalation timing unchanged. The independent-failure model also yields a deterministic budget-scaled threshold; Appendix C.14 reports that fixed thresholds of 11 or 12 match DWELL's 27/30 recoverable successes on the shifted-budget streams in a post-hoc calculation. These are not proof that the monitor is universally useless or that fixed policies are equivalent, but they substantially weaken the case for the specific proposed machinery. The remaining contribution is mainly an empirical characterization of stopping trade-offs, whose novelty and actionable consequences need sharper articulation.
  2. The controlled workload leaves limited evidence for broader orchestration use. The in-band observer tracks explicitly requested keys and key–value assertions, and the main fault studies use constructed information-holder tasks (Sections 2.3 and 3.1). The fresh-task and multi-deployment experiments strengthen transfer within that setting, but do not establish comparable behavior when progress is semantically ambiguous or recovery depends on changing tools and actions. The external IWG audit illustrates the boundary: seven of nine failures end within two steps, before the margin can activate (Appendix C.1). Given this scope, it remains unclear how much guidance the method provides beyond choosing a stopping threshold for the specified retry process.
Questions for authors
  1. What decision-relevant information does the proposed margin provide beyond the net potential drop and failure counter on the evaluated workloads? Is there an existing example where its earlier warning changes a beneficial intervention? Please distinguish a descriptive diagnostic contribution from a controller improvement.
  2. The common-cap transfer experiment is welcome. Is there a prospective common-cap comparison with a budget-scaled fixed threshold under the same advisory channel and a predeclared success-versus-cost objective? The post-hoc 11/12-threshold calculation is informative, but does not replace that closed-loop comparison.
  3. Is there evidence on tasks whose progress or recovery opportunities are not captured by explicit key completion and a target's failure streak? If not, what concrete deployment decision should a reader take from the current controlled findings, beyond the already-established dependence on persistence, budget and the cost of failure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.