- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how LLM-based multi-agent systems can maintain performance when agents become unavailable and their communication links are removed. It proposes RobMAS, a two-stage reinforcement learning framework that adds edges between surviving agents. The first stage uses graph-based rewards to encourage connectivity while limiting communication bottlenecks and unnecessary edges. The second stage combines the task representation with recurrent states from previous recovery decisions to refine the topology. The proposed contribution is to adapt communication to the observed dropout pattern while retaining recovery history across rounds. The authors evaluate static and dynamic dropout on MMLU-Pro, MATH, and Chess, with additional transfer, backbone, efficiency, and ablation experiments. They report that, at a 40% dropout probability, accuracy falls by approximately 0.8–3.0 percentage points relative to the no-dropout setting, alongside improvements over the evaluated baselines and higher accuracy per token than ResMAS.
Critical strengths
The paper considers a practically relevant failure mode: agents becoming unavailable, rather than remaining online and producing incorrect answers. Losing an agent also removes its communication links, so methods designed to suppress unreliable messages may not address the resulting disruption. Studying both persistent and round-dependent unavailability is a useful aspect of the experimental setup. This gives the work a relevant reliability question to investigate, even though the current evidence does not establish the necessity or effectiveness of the proposed solution convincingly.
Critical weaknesses
There are several critical weaknesses I want to mention in the paper:
-
The motivation does not clearly establish the problem the method needs to solve. The paper motivates dropout through network failures, timeouts, and API limits, but evaluates independent agent unavailability sampled before communication rounds. It does not explain how this simplified model captures those failures, particularly when failures are correlated or agents lose execution state. The temporal motivation is also indirect: showing that historical messages affect accuracy and token consumption does not establish that remembering previous graph-editing decisions improves recovery. These gaps weaken the connection between the motivating observations and the proposed framework.
-
The related-work discussion does not establish a convincing contribution beyond existing adaptive methods. Differences in implementation are described, but the paper does not show why existing topology generators or runtime adaptation mechanisms cannot address the surviving-agent setting. Methods such as DyTopo and MAS² deserve a more substantive comparison. More concerningly, ResMAS is described as explicitly addressing node dropout, although its original perturbation model involves agents producing random responses. Without explaining how this baseline was adapted and trained, the comparison leaves both the research gap and the claimed advantage unclear.
-
Key methodological choices are insufficiently justified and specified. The structural reward combines spectral, betweenness, and reachability terms, but the paper provides little evidence that these particular objectives predict successful task recovery. It also omits the experimental values of their weights, the spatial/task reward weights, and the prior-scaling parameter (), alongside essential details of Muse and TSRA. According to the equations and pseudocode, TSRA receives topology features and the original query, rather than agents’ actual responses, which limits its ability to adapt to the evolving reasoning process. The conditional graph-theoretic results do not establish that the learned policy achieves their assumptions or restores task performance.
-
The experimental evidence contains unresolved inconsistencies and does not isolate the central mechanism. Section 4.4 reports 400 Chess and 132 MMLU questions, whereas total tokens divided by tokens per question in Table 2 imply approximately 132 and 400, respectively. ResMAS’s Chess dropout accuracies also differ between Tables 1 and 2 without an explanation. Figure 6 is more troubling: its first-round graph contains reciprocal edges despite the DAG constraint, and an edge shown in Round 1 disappears from the purported cumulative frequencies. Finally, the temporal ablation does not clearly separate recurrent memory from task conditioning, residual policy learning, and additional edge budget. These issues prevent a reliable interpretation of the reported gains.
-
The paper does not distinguish successful recovery from unnecessary initial redundancy. Maintaining nearly the same accuracy after dropout raises the question of whether a smaller, appropriately selected team could achieve comparable performance from the outset. Dynamic dropout is not equivalent to permanently using fewer agents, so this is not itself a logical contradiction. Nevertheless, the paper lacks controlled comparisons with smaller teams, proactive agent pruning, or replacement agents under matched computation budgets. Without these comparisons, it remains unclear whether the proposed approach offers a useful accuracy–cost–reliability trade-off or mainly repairs an unnecessarily large initial system.
Questions for authors
The following are my questions for the authors:
-
Can you reconcile the inconsistencies in the reported experiments? Section 4.4 specifies 400 Chess and 132 MMLU questions, while the token totals and per-question counts in Table 2 imply the reverse. ResMAS’s Chess dropout accuracies also differ between Tables 1 and 2. Were these evaluations conducted on different samples or runs? Please clarify the actual sample sizes, seeds, and aggregation procedure. In Figure 6, how can reciprocal edges occur in Round 1 under the DAG constraint, and why does an observed edge subsequently disappear from the cumulative frequencies?
-
What information does TSRA retain, and which experiment isolates the benefit of that memory? The equations and pseudocode appear to feed TSRA topology features and the original query, without agents’ responses or execution feedback. Is this the implemented design? Please specify the recurrent architecture, how states pass between rounds, and the values of the structural reward weights, spatial/task reward weights, and (). Does “w/o Temporal” remove only recurrent memory while preserving task conditioning, training, and edge budget, or does it remove other components of Stage 2?
-
What evidence distinguishes recovery from redundancy in the initial team? In particular, under static dropout, how does the recovered system compare with a system using the same surviving agents from the beginning, with a suitable initial topology and matched computation budget? Do the existing experiments include this control or a comparison with proactive agent elimination? I recognize that rotating availability under dynamic dropout differs from a fixed smaller team, but the current results do not establish why starting with ten agents and repairing their topology is preferable.
5 comments
Response: Methodology and TSRA Memory
Regarding Weakness 3 and Question 2: Methodological Justification, TSRA Memory, and Implementation Details.
We thank the reviewers for highlighting the need to better justify our methodological choices and clarify the information retained by TSRA. We address these concerns from four perspectives.
1. Structural reward design. The three structural objectives serve complementary purposes in communication recovery. The spectral ratio encourages weak connectivity while discouraging excessive degree concentration; maximum betweenness penalizes reliance on critical relay agents; and reachability encourages information propagation among surviving agents.
To further examine this relationship, we compare topology quality scores , which underpin the structural reward, and task accuracy on MMLU-Pro and GSM8K (Appendix D, Table 5). Overall, higher structural scores are generally associated with higher task accuracy, indicating the effectiveness of structural optimization in communication recovery. However, structural quality alone does not fully determine task performance. On MMLU-Pro, RobMAS achieves higher accuracy than Greedy Recovery despite comparable or slightly lower structural scores. On GSM8K, RobMAS achieves the highest structural scores and accuracy under both dropout settings. These findings suggest that structural optimization provides a strong foundation for recovery, while task-adaptive refinement in Stage 2 further improves accuracy beyond structural quality alone.
2. Information retained by TSRA. TSRA is designed to retain compact, task-conditioned topology recovery history rather than agents' intermediate responses or reasoning traces. Its inputs consist of the current topology representation, the task query embedding, and the recurrent state from preceding recovery decisions. This design focuses temporal modeling on communication recovery while avoiding the additional complexity of explicitly encoding intermediate agent responses. Consequently, TSRA adapts to evolving topology and availability patterns, but does not directly model changes in agents' intermediate reasoning content.
3. Temporal memory ablation and implementation. We agree that the existing w/o Temporal ablation does not, by itself, isolate the contribution of recurrent memory from other Stage 2 components. To address this, we have introduced a history-reset comparison w/o State History that preserves task conditioning, residual policy optimization, and the edge budget, while resetting recurrent states at communication-round boundaries. This comparison will more directly evaluate the contribution of cross-round recovery history. We have specified the Muse and TSRA architectures, recurrent-state initialization and propagation, structural reward coefficients, spatial/task reward weights, and prior-scaling parameter in Appendix H.
4. Scope of theoretical guarantees. We agree that Theorems 1–3 provide conditional structural characterizations rather than guarantees of learned-policy performance. Specifically, they establish properties of graphs satisfying the stated spectral or centrality conditions and characterize when the residual policy preserves structural action preferences. We have clarified this distinction and strengthen the empirical evidence connecting structural in Appendix B.
Response: Experimental Consistency
Regarding Weakness 4 and Question 1: Experimental Consistency, Figure 6 Validity, and Temporal Mechanism Isolation
We thank the reviewers for identifying these important issues concerning experimental consistency, visualization validity, and the isolation of temporal contributions. We agree that these aspects require clarification to support a reliable interpretation of our results. We address them jointly below.
1. Experimental consistency and evaluation protocol.
We clarify that both issues arise from reporting errors rather than differences in experimental settings. The sample sizes in Section 4.4 were inadvertently reversed. The correct evaluation sizes are 132 Chess questions and 400 MMLU questions, consistent with the token statistics in Table 2. The discrepancies in ResMAS's Chess dropout accuracies between Tables 1 and 2 resulted from an error during result compilation. Both tables use the same 132 Chess questions, with consistent random seeds and identical dropout masks across methods to ensure fair comparisons. We have corrected the inconsistent entries and update any affected derived metrics accordingly.
2. DAG validity and cumulative edge frequencies in Figure 6.
DAG validity. Figure 6 visualizes the cumulative occurrence frequencies of directed communication edges across three rounds of the same task, rather than the adjacency matrix of an individual communication graph. For example, in Round 2, both (2,8) and (8,2) have a cumulative frequency of 0.5. This reflects that the edge (8,2) was established in Round 1, whereas (2,8) was established in Round 2. These oppositely directed edges therefore belong to different communication rounds and never coexist within the same recovered graph. Consequently, their simultaneous appearance in the cumulative visualization does not violate the DAG constraint.
Dynamic topology evolution. Under dynamic agent dropout, agent availability is resampled at each communication round, and RobMAS reconstructs the communication topology according to the currently available agents. As a result, an edge established in an earlier round may be absent from the current communication graph due to agent unavailability or subsequent topology adaptation. Figure 6 is intended to illustrate this evolving connectivity while reflecting the persistence of historical communication links. We have clarified the distinction between cumulative historical edge frequencies and per-round active communication graphs in the figure caption and Appendix I to avoid ambiguity.
3. Isolating the contribution of temporal memory.
We agree that the existing w/o Temporal ablation evaluates the broader contribution of Stage 2 rather than recurrent memory alone. We have clarified the original ablation configuration and introduce a controlled history-reset comparison w/o State History that preserves task conditioning, residual policy optimization, and the same edge budget while removing cross-round recurrent-state propagation. This will allow us to more directly assess the contribution of temporal recovery history.
Response: Motivation and Related Work
Regarding Weakness 1: Motivation and Dropout Modeling Concerns
We thank the reviewer for pointing out the need to better connect our motivation with the proposed recovery framework.
1. Dropout Modeling. Our independent dropout model provides a controlled abstraction of agent unavailability rather than a complete simulation of real world failures. Similar to the independent agent-failure assumption adopted in ResMAS, this formulation allows us to systematically evaluate robustness under stochastic disruptions. Importantly, RobMAS conditions its recovery decisions on the observed availability mask and does not require failure events to be independent at inference time. Although the current experiments focus on independently sampled dropout patterns, the recovery mechanism can also operate on correlated failure configurations.
2. Temporal motivation. We agree that the temporal communication dependence illustrated in Figure 1 does not directly establish the benefit of remembering previous graph-editing decisions. Rather, it motivates the broader need to selectively utilize cross-round information. TSRA addresses a more specific problem by conditioning subsequent graph-editing decisions on recurrent recovery states. The ablation results in Table 3 show that removing the temporal adaptation stage substantially reduces accuracy, supporting its contribution to recovery effectiveness.
Regarding Weakness 2: Related-Work Positioning and ResMAS Baseline Evaluation.
We thank the reviewer for highlighting the need for a more substantive comparison with existing adaptive methods and clearer baseline evaluation protocols.
1. Distinction from Existing Adaptive Methods. We acknowledge that DyTopo and MAS² can potentially reconstruct communication structures over surviving agents. In contrast, RobMAS directly addresses communication recovery under agent dropout by reconstructing the surviving communication topology while preserving the original agent roles. This enables targeted topology adaptation without agent replacement or additional LLM-based reconfiguration.
DyTopo constructs communication edges based on semantic similarity between agents' Need and Offer descriptions, using a fixed threshold to determine connectivity. However, agent dropout changes the set of available roles, requiring semantic descriptions and matching relationships to be adjusted accordingly. A fixed threshold may not effectively accommodate such dynamic changes in the surviving agent set. In contrast, MAS² triggers corrective actions through its Rectifier when execution failures occur or resource consumption exceeds predefined thresholds. Handling agent dropout would require additional mechanisms to monitor agent availability; otherwise, recovery may incur substantial latency. Moreover, both methods involve LLM-based adaptive processes, which may incur additional inference overhead when agent availability fluctuates frequently.
2. ResMAS Baseline and Evaluation Protocol. We clarify that the original ResMAS primarily addresses response perturbations, where agents remain available but may produce unreliable outputs, whereas RobMAS considers complete agent unavailability. In our evaluation, we use the communication topology produced by the original ResMAS method without retraining it for the agent-unavailability setting. We apply the same evaluation dropout masks across methods and remove unavailable agents and their incident edges before communication. Accordingly, this comparison evaluates the transfer robustness of a pre-optimized ResMAS topology under a different failure model.
We have further clarified the research motivation and distinctions from existing methods in Sections 1 and 2.2, and documented the baseline adaptation protocol in Appendix G.