- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
CollabFlow trains a director to assemble teams of agents, assign skills, and choose communication paths and protocols. A frozen executor runs each team, and an evidence gate determines whether a receiver keeps, adopts, or revises an answer. The director is trained with Collaborative Trajectory Balance (CTB), while a skill library evolves across rounds. The experiments cover QA, reasoning, coding, and interactive tasks. CollabFlow reports the best average result for each row in its five-run main comparison and evaluates six additional frozen executors.
Critical strengths
S1. The communication design is worth studying. The director chooses who communicates and how, while the evidence gate determines the receiver's response. This gives the paper a framework for examining communication alongside team structure.
S2. The evaluation spans different tasks and includes useful controls. The main results report five runs. Table 2 compares CollabFlow with a single agent, voting, a fixed team, and simpler communication rules. The fixed-team control is especially helpful for judging what the learned director adds.
Critical weaknesses
W1. The benefit of learning the team policy is smaller than the headline gain. Section 5 emphasizes a 33.3-point IID gain over direct inference, but the Fixed team control is much stronger than direct inference. In Table 2, which reports a single run, CollabFlow exceeds Fixed team by 2.68 points on average across the IID tasks and 3.39 points on OOD tasks. Some gaps amount to only one or two correct items. Please run the Fixed team under the same five-run protocol as the main comparison and report uncertainty for the difference. Its skills, tools, and budget also need to be clear.
W2. The CTB comparison is too short to establish its contribution. Figure 5b compares training methods for only eight steps, without run-to-run uncertainty. The diversity plot uses other systems and an untrained director rather than the same director trained with a different objective. A full training comparison of held-out accuracy and diversity would show whether CTB's early advantage persists. Please also define what replaces CTB in the “w/o CTB” ablation.
W3. The QA gate ablation is hard to square with the stated rule. With κ = 1, Eq. 7 adopts a differing answer only when the evidence-score difference exceeds 1 and keeps the current answer only when it is below −1. If QA tasks have no executable checks, scores can only be 0 or 1, so every disagreement appears to trigger revision. Yet Table 2 shows large gaps between CollabFlow and “Always revise”: 92.97 versus 68.75 on NQ-Open and 46.88 versus 24.22 on MuSiQue. Which checks are available, and how often does each gate branch run?
W4. Key settings are still hard to recover. The paper needs a compact account of the tools and retrieval available to each agent, the training and execution budgets, and the settings used to compare executors. These choices may affect the gains as much as the team policy does, especially since QA performance varies markedly across executors (Table H.1b). A single setup table would make the main comparisons easier to interpret and reproduce.
W5. The skill-evolution evidence is selective. The method describes binding and editing skills, but it is unclear how changes to skills affect team identity or past success records. The persistence plots also show the twenty steps with the largest immediate gains rather than the full sequence. Please explain how edited skills enter later training rounds and show the complete evolution trace. That would help assess whether the library improves reliably over time.
W6. The paper is hard to follow. Details about skill binding, library edits, and training are scattered, and the relationship among evolution steps, training rounds, and the steps in Figures 4c and 5b is unclear. Please define SR in Figure 5a and better justify the reward in Eq. 13. A simpler overview and one worked example would make the method easier to assess.
Questions for authors
- How does the Fixed team compare under the main five-run protocol with matched tools, skills, and budgets?
- Does CTB still outperform a matched alternative after the full training budget? What exactly is “w/o CTB”?
- Which executable checks and retrieval sources are available on the QA tasks? How frequently does each gate branch occur, and what does “Always revise” change when agents agree?
- Can you summarize the tools, budgets, and executor settings in one place?
- How do skill edits affect team identity and records, and what does the full evolution trace show?