- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MASFACT, which adds a library of historical structures to existing methods for generating multi-agent graphs. It retrieves and aligns historical structures based on agent attributes and connections, then uses a small number of samples from the new task to adjust the connections while limiting the magnitude of the changes. The algorithm updates the structure library with effective new structures.
Critical strengths
-
It improves performance across multiple graph-construction methods. Table 4 shows that after integrating MASFACT, AGP, GDesigner, and ARGDesigner all achieve higher average accuracy and less forgetting in experiments involving successive changes in question categories, datasets, and task types.
-
Preserving historical collaboration structures and continuing to adapt and reuse them as new tasks arrive is a practically meaningful direction. This paper offers an approach to accumulating and using experience from past collaborations in continually learning multi-agent systems.
Critical weaknesses
-
The additional benefits of the structure library over the original graph-construction policy remain unclear
Methods such as GDesigner already learn graph-construction policies from training tasks, retaining training experience in the controller parameters. The authors adapt these baselines into versions that update stage by stage as tasks arrive, but do not report their performance when they stop updating after base-stage training and directly handle subsequent tasks. Fixing the network parameters does not mean using the same graph for every question, so comparisons with fixed chains, trees, and historical graph reuse cannot replace this result. Without this comparison, it remains difficult to determine how much continual adaptation adds over directly using the existing policy.
The authors also need to explain whether the original graph constructor continues to be updated after integrating MASFACT, and whether its update procedure remains consistent with that of the baseline algorithm. Although Table 5 reports a performance drop after removing the historical prior, it does not explain which graph-construction and training steps this version retains. If different versions update different parameters, the differences in forgetting may not come solely from the structure library.
-
Key steps in training and old-task testing are not clearly described
The paper does not explain how feedback from actual problem solving is used to update the connection parameters. The differentiable sampling in the paper handles discrete edge selection, but feedback such as answer correctness and code test results cannot be differentiated directly. The authors need to explain how the parameters are updated using this feedback. The paper states that parameters are not updated during old-task testing, but does not specify which set of connection-adjustment parameters is used. Does testing directly use the latest parameters after learning new tasks, or does it also use parameters saved separately for old tasks? These choices affect old-task performance and the assessment of forgetting.
Questions for authors
-
Does the original graph constructor continue to be trained after integrating MASFACT? Which parameters are updated in the full method and in the version without the historical prior in Table 5? Apart from the historical prior, what other differences are there in the graph-construction and training steps of these two versions?
-
How is actual problem-solving feedback, such as answer correctness and code test results, used to update the connection parameters? Beyond differentiable edge sampling, what parameter-update method is used?
-
After learning new tasks, which set of connection-adjustment parameters is used when testing old tasks? Are parameters saved separately for each old task?
5 comments
(This comment continues our previous response to W1-1.)
Table D. MMLU-Pro new-stage accuracy (%).
Base stage: Physics.
| Method | Chemistry | Biology | Mathematics | Computer Science | Health | Engineering | Business | Law | Economics | Psychology | History | Philosophy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GDesigner, frozen | 42.23 ± 0.72 | 54.18 ± 0.39 | 30.14 ± 0.26 | 42.81 ± 0.75 | 48.25 ± 0.19 | 30.87 ± 0.48 | 45.06 ± 0.62 | 28.37 ± 0.36 | 42.42 ± 0.74 | 55.15 ± 0.93 | 47.12 ± 0.85 | 45.20 ± 0.51 |
| GDesigner, 10-shot | 42.93 ± 0.36 | 54.60 ± 0.83 | 31.31 ± 0.74 | 43.82 ± 0.81 | 49.22 ± 0.46 | 32.08 ± 0.53 | 46.15 ± 0.26 | 28.64 ± 0.41 | 42.57 ± 0.56 | 56.25 ± 0.17 | 48.17 ± 0.49 | 45.27 ± 0.32 |
| GDesigner, standard | 45.13 ± 0.76 | 56.38 ± 0.64 | 35.09 ± 0.95 | 46.96 ± 0.82 | 50.40 ± 0.59 | 34.96 ± 0.74 | 48.29 ± 0.50 | 31.24 ± 0.27 | 44.08 ± 0.45 | 57.27 ± 0.31 | 50.42 ± 0.54 | 49.35 ± 0.49 |
| MASFACT, 10-shot | 45.15 ± 0.12 | 56.28 ± 0.35 | 34.88 ± 0.23 | 47.19 ± 0.79 | 50.83 ± 0.53 | 36.16 ± 0.41 | 48.31 ± 0.48 | 29.81 ± 0.32 | 44.97 ± 0.44 | 57.88 ± 0.56 | 51.06 ± 0.37 | 49.04 ± 0.50 |
| MASFACT, standard | 48.27 ± 0.47 | 59.16 ± 0.58 | 38.44 ± 0.36 | 50.11 ± 0.52 | 53.37 ± 0.43 | 38.58 ± 0.61 | 51.35 ± 0.40 | 33.53 ± 0.55 | 47.47 ± 0.49 | 60.46 ± 0.34 | 53.41 ± 0.57 | 52.43 ± 0.45 |
The experimental findings are as follows:
-
The frozen dynamic policy preserves old-task performance but provides insufficient adaptation to new tasks. Its mean new-stage accuracy falls below few-shot MASFACT by 11.24%, 4.11%, and 3.31% at the task, domain, and class levels, respectively. Although the frozen constructor generates input-dependent graphs, its construction rules receive no subsequent task feedback.
-
Continual constructor updates improve new-task learning but introduce forgetting. On the task stream, mean new-stage accuracy reaches 45.20% with few-shot updates and 47.67% with standard updates. The latter incurs 7.12 AF, illustrating the retention cost of adapting the constructor to subsequent tasks.
-
MASFACT achieves a better adaptation–retention balance than continual constructor training. Relative to few-shot GDesigner, it improves task-level AA by 9.86 percentage points and reduces AF by 4.88 percentage points, with higher AA and lower AF across all three levels. Aligned historical references guide adaptation, while constrained residual updates adjust communication paths using current feedback.
-
Historical reuse enables effective continual adaptation with limited new-task feedback. Few-shot MASFACT exceeds standard GDesigner’s final AA by 8.04, 6.91, and 3.70 percentage points across the three levels. By reusing collaboration experience and updating connection residuals, MASFACT supports adaptation after deployment without repeatedly retraining the original constructor.
W1-2 & Q1: Constructor Updates and Baseline Adaptation Protocols
-
After integrating MASFACT, the original graph constructor is frozen after base-stage training. Incremental stages optimize only connection residuals and selectively update the prior bank. The frozen constructor generates a task-dependent scaffold for historical-prior retrieval and alignment to the current agent space. Residual editing around the aligned prior center then produces the executable communication graph.
-
All MASFACT-integrated optimizers in Table 4 (AGP, GDesigner, and ARGDesigner) follow this same freezing and adaptation protocol. Each method first learns or optimizes its topology-construction components on sufficient base-stage data according to its original protocol. These components are then fixed, while the MASFACT's historical-prior retrieval, alignment, and residual-adaptation pipeline handles subsequent tasks. Table 4 shows that structural prior guided residual adaptation enhances continual learning across different topology-construction methods. (Table 4, Appendix G.2)
-
The standalone baselines in Tables 1–3 retain their original optimization targets and objectives, using current-stage support under a shared continual-learning protocol. Variants marked with † are replay-based baselines. For each old task, replay variants adapt the model using historical training examples from that domain, then evaluate its performance on that domain. MASFACT uses structural memory without replaying raw historical examples, and the results support a better overall accuracy–forgetting trade-off. (Tables 1–3, Appendices F.2–F.3)
MASFACT-integrated methods therefore share a frozen-constructor protocol, while standalone baselines retain their respective adaptation mechanisms, with continual-learning benefits assessed against both standalone and replay-enhanced alternatives.
Thank you! We would be glad to provide further details upon request.
To assess whether the graph constructor’s base-stage experience is sufficient for subsequent tasks, we compare frozen graph construction, continual constructor updates, and MASFACT, evaluating both new-task learning and old-task retention.
Table A. AA and AF across three continual-learning levels
Table B. Task-level new-stage accuracy (%).
Sequence: KQA → Math → Code → MQA. Base stage: KQA.
Table C. Code domain-level new-stage accuracy (%).
Sequence: APPS → HumanEval → LiveCodeBench → MBPP.
Base stage: APPS.
(Due to the character limit, our response to W1-1 continues in the next comment.)