- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a failure mode of multi-agent LLM systems that coordinate through persistent shared memory over several rounds. The observation is that collaboration does not always help: once a locally unproductive update lands in shared memory, it persists across later roles and rounds, and the system can keep "collaborating" without making any real progress. The authors name this stalled regime Phantom Collaboration.
The contribution is in two parts. First, a theoretical analysis. They define a system-level potential that combines a memory-level and per-agent log-loss confidence in the ground truth, characterize a "collaborative stagnation region" (CSR) as a high-potential region that is absorbing and makes non-positive expected system-level progress, and then show the training consequence: when full-horizon GAE propagates the TD residuals of a stalled tail backward, the credit assigned to genuinely useful earlier coordination is reduced, and under sufficiently adverse conditions the expected advantage of an earlier good action can flip negative (Theorem 4.2 and its appendix version). The cleanest object here is the decomposition of full-horizon GAE into a pre-entry contribution, a non-terminal stalled tail, and a terminal residual, which makes the source of the distortion explicit.
Second, the method. PHANTOM-CONTROL detects persistent low observable progress — an observable proxy must stay below a threshold for k consecutive system transitions — and places a common optimization boundary at the end of the first confirmed low-progress window. Policy optimization then uses a prefix-supported GAE that excludes post-boundary residuals from the credit of retained actions, while policy updates stay role-specific. For the practical detector the authors prove an exponential bound on the fixed-window false-trigger probability.
Evaluation is on five interactive active-reasoning tasks (Circuit Decoding, Situation Puzzles, Guess Numbers, Preference Estimation, Movie Recommendation), all on a three-role sequential system with a frozen Qwen2.5-7B-Instruct backbone plus LoRA adapters, trained with PPO. The headline claims are late-stage reward gains of 35–50% in representative regimes and a 20–35% reduction in mean generated response length. Two controls carry the empirical argument: Random-Matched (same masking budget placed at random) does not reproduce the gains, and a Role-Local variant underperforms the shared-trajectory system-level intervention.
Critical strengths
A useful and well-formalized conceptual contribution. The paper does more than name a phenomenon — it ties "the system keeps talking but stops improving" to a concrete training pathology (stalled-tail TD residuals dragging down, and potentially inverting, the advantage of earlier useful coordination) and makes the mechanism explicit through the GAE decomposition. This is the part I'd cite and build on, and it is what tips me toward acceptance.
The right control experiments. The obvious skeptical reaction to "mask the stalled tail" is "you've just reinvented trajectory shortening / early stopping." Random-Matched directly rebuts this by holding the masking budget and the per-round masked-length distribution fixed while destroying the correspondence between the boundary and the observed per-rollout progress — and it underperforms. The Role-Local comparison similarly isolates the value of acting on the shared trajectory rather than one estimator update. Taken together these make the central empirical claim ("placement, conditioned on observed progress, is what matters") credible rather than asserted.
Critical weaknesses
The theory-to-practice distance is large, and the paper is candid that it is. Everything clean is proved at the ideal boundary under assumptions that are neither checked nor easy to check in an LLM system, and the detector that is actually deployed carries no guarantee of landing near the regime the theorems describe. This leaves the formal apparatus in an uncomfortable spot: impressive, but more motivational than load-bearing. I don't think the theorems are wrong; I think the paper slightly oversells how much they say about the method that was run.
Loose empirical reporting on a narrow setup. With one backbone, three seeds, five short-horizon tasks (capped at 5–15 rounds), and gains quoted as ranges, it is difficult to assess how large and how robust the improvement really is. The response-length reduction is framed as a pure efficiency win, but on these short horizons it also leaves open the opposite worry — that boundary-based truncation could penalize tasks where progress is genuinely slow-but-real and gets misclassified as stagnation. A per-task results table and at least one additional backbone would settle most of this.
Questions for authors
- The credit-distortion half of the argument — a long uninformative tail dragging down earlier credit through full-horizon GAE — seems like it would apply to any long-horizon RL problem, not specifically to shared-memory multi-agent systems. What is distinctively multi-agent about it, beyond the mechanism by which the stall arises and persists through shared memory? This is a clarification, not a demand for experiments; it would help scope the contribution.
- Could you report a per-task main table (final task metric for Vanilla PPO, Phantom-Control, and Random-Matched, with seed variance) rather than aggregate ranges? I'm asking for a clearer view of the runs you already have, not new ones.
- Is there a regime where the intervention hurts — e.g., a task with legitimately slow early progress that the detector cuts off prematurely? A short negative result or a discussion of this failure direction would strengthen the paper.
- How was k (and Δ_min) chosen per task, and does final task performance track the boundary-activation behavior shown in Table 1? Some sense of sensitivity here would help readers who want to deploy the detector.
3 comments
Thank you for your response. I will keep my positive score.
Author Response to Reviewer 3h7W
Thank you for taking the time to read our response. We appreciate your feedback and the care you took in reviewing our work.
Author Response to Reviewer 3h7W
Thank you for the careful questions. We clarify system-level credit, task-wise results, and detector sensitivity.
Q1 — What is distinctively multi-agent? We agree that full-horizon GAE propagates later TD residuals into earlier advantages in general long-horizon RL. Our contribution is to characterize how persistent shared memory turns local low progress into a system-level credit problem. Sequential role proposals and environment feedback enter persistent memory, shaping later decisions; local improvement need not imply collaborative progress. Section 4.1 models this through the joint role–memory potential
Ψtsys=αΨtmem+(1−α)∑iwiΨi,t
and the collaborative stagnation region (CSR). Proposition 4.1 compares actual collaborative belief updates with Bayesian-oracle updates under the same action and observation. Conditional on CSR entry and its assumptions, Theorem 4.2 bounds degradation of earlier common system-level GAE; Lemma C.5 connects that common credit to independent role-wise, token-wise PPO gradients. The GAE identity is general; shared-memory stagnation and cross-role credit are our distinctive objects. PHANTOM-CONTROL sets a common boundary after complete shared-memory transitions without combining role policies.
Q2 — Per-task final metrics. We report the task-specific final evaluation scores (mean ± SD across runs, percentage points), with the metrics of Table 2:
For PE, success means cosine similarity ≥0.88; MR evaluates Top-1 recommendation. PC has the highest mean final score in all five tasks. These complement Figures 2 and 4.
Q3 — When can intervention hurt? Low observable progress may precede useful delayed information; early masking then removes useful residuals and updates. Our original Table 1 quantifies this risk: in GN the Positive-Gain Rate among activated trajectories is 49.0% for k=1 versus 6.4% for k=3 (different activated subsets). Additional Figure 7(a) shows that k=1 suffers a mid-training held-out accuracy drop before recovering, whereas k=3 is steadier. Additional Table 4 shows that k=1 confirms before round 10 on 94.67% of training trajectories, omitting 67.00% on average, versus 19.00% and 5.20% for k=3. The corresponding final accuracies are 96.48±1.02% and 97.50±1.41%. Persistence makes intervention more selective.
Q4 — How are k and Δmin selected, and does activation predict performance? We use task-level observable progress proxies after complete system transitions (Table 3), with k=6,5,3,2,2 for CD, SP, GN, PE, MR respectively. Discrete proxies detect no candidate refinement; continuous proxies use calibrated low-progress thresholds (Appendix F). The latent CSR and practical trigger differ: the theorems characterize credit degradation after CSR entry and preservation at the ideal boundary; deployment uses a measurable progress proxy. Our original Figure 1 shows reduced CD progress around the detected boundary and higher retained-prefix raw advantage when later residuals are excluded. Original Random-Matched (Figure 4) holds activation count and the masked-tail-length multiset fixed but randomly reassigns boundaries; its weaker results support progress-aligned placement over masking quantity.
Sensitivity evidence further guides calibration. In the new GN sweep (Table 4), k=4 triggers less but ends at 91.25±1.77%, versus 97.50±1.41% for k=3; activation is not a monotone predictor of accuracy. New Figure 7(b) replays the same 40 PE trajectories at early/late checkpoints: increasing Δmin/Δref raises trigger frequency and generally advances the trigger. New Figure 10 compares PC against Vanilla PPO on CD with Qwen2.5 3B/7B/14B backbones. Mean held-out Exact Match gains over PPO steps 40, 44, 48, and 49 are +2.50/+26.88/+3.13 percentage points. The gains vary with scale: 3B may lack stable useful coordination, limiting credit to preserve; 7B benefits as Vanilla deteriorates late; 14B Vanilla reaches 95.63% late-stage Exact Match, leaving little headroom. During the first ten training steps, 27/30 3B rollouts trigger versus 17/30 for both 7B and 14B. Equal trigger counts at 7B/14B but different gains show that intervention value also depends on learned coordination and baseline headroom.