acceptodds
Under review as a conference paper at ICLR 2027

When Collaboration Backfires: Phantom-Control for Credit Preservation in Multi-Agent LLMs

Abstract

Multi-agent LLM systems use persistent shared memory to coordinate decisions across multiple rounds. However, continued collaboration can become harmful when locally unproductive updates persist across roles and rounds. We identify this failure mode as Phantom Collaboration, where a stalled collaborative tail can distort the credit assigned to earlier useful coordination. Our analysis shows that a sufficiently strong stalled-tail contribution can overwhelm the contribution of the pre-entry prefix and make the expected advantage of an earlier useful action negative. To address this problem, we propose Phantom-Control, which detects persistent low observable progress and sets a common optimization boundary to restrict policy optimization to the retained collaborative prefix. At the ideal boundary, removing the stalled continuation preserves higher expected prefix credit. For the practical detector, we derive an exponential bound on the fixed-window false-trigger probability. Across five multi-agent active-reasoning tasks, Phantom-Control achieves late-stage reward gains reaching 35–50% and reduces mean total generated response length by 20–35% in representative regimes. Additional controls indicate that the gains depend on targeted system-level boundary placement rather than trajectory shortening alone.

Primary area:
reinforcement learning
Keywords:
Multi-agent, LLM, Collaboration, Reinforcement Learning, Trajectory-Level Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3h7WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure mode of multi-agent LLM systems that coordinate through persistent shared memory over several rounds. The observation is that collaboration does not always help: once a locally unproductive update lands in shared memory, it persists across later roles and rounds, and the system can keep "collaborating" without making any real progress. The authors name this stalled regime Phantom Collaboration.

The contribution is in two parts. First, a theoretical analysis. They define a system-level potential that combines a memory-level and per-agent log-loss confidence in the ground truth, characterize a "collaborative stagnation region" (CSR) as a high-potential region that is absorbing and makes non-positive expected system-level progress, and then show the training consequence: when full-horizon GAE propagates the TD residuals of a stalled tail backward, the credit assigned to genuinely useful earlier coordination is reduced, and under sufficiently adverse conditions the expected advantage of an earlier good action can flip negative (Theorem 4.2 and its appendix version). The cleanest object here is the decomposition of full-horizon GAE into a pre-entry contribution, a non-terminal stalled tail, and a terminal residual, which makes the source of the distortion explicit.

Second, the method. PHANTOM-CONTROL detects persistent low observable progress — an observable proxy must stay below a threshold for k consecutive system transitions — and places a common optimization boundary at the end of the first confirmed low-progress window. Policy optimization then uses a prefix-supported GAE that excludes post-boundary residuals from the credit of retained actions, while policy updates stay role-specific. For the practical detector the authors prove an exponential bound on the fixed-window false-trigger probability.

Evaluation is on five interactive active-reasoning tasks (Circuit Decoding, Situation Puzzles, Guess Numbers, Preference Estimation, Movie Recommendation), all on a three-role sequential system with a frozen Qwen2.5-7B-Instruct backbone plus LoRA adapters, trained with PPO. The headline claims are late-stage reward gains of 35–50% in representative regimes and a 20–35% reduction in mean generated response length. Two controls carry the empirical argument: Random-Matched (same masking budget placed at random) does not reproduce the gains, and a Role-Local variant underperforms the shared-trajectory system-level intervention.

Critical strengths

A useful and well-formalized conceptual contribution. The paper does more than name a phenomenon — it ties "the system keeps talking but stops improving" to a concrete training pathology (stalled-tail TD residuals dragging down, and potentially inverting, the advantage of earlier useful coordination) and makes the mechanism explicit through the GAE decomposition. This is the part I'd cite and build on, and it is what tips me toward acceptance.

The right control experiments. The obvious skeptical reaction to "mask the stalled tail" is "you've just reinvented trajectory shortening / early stopping." Random-Matched directly rebuts this by holding the masking budget and the per-round masked-length distribution fixed while destroying the correspondence between the boundary and the observed per-rollout progress — and it underperforms. The Role-Local comparison similarly isolates the value of acting on the shared trajectory rather than one estimator update. Taken together these make the central empirical claim ("placement, conditioned on observed progress, is what matters") credible rather than asserted.

Critical weaknesses

The theory-to-practice distance is large, and the paper is candid that it is. Everything clean is proved at the ideal boundary under assumptions that are neither checked nor easy to check in an LLM system, and the detector that is actually deployed carries no guarantee of landing near the regime the theorems describe. This leaves the formal apparatus in an uncomfortable spot: impressive, but more motivational than load-bearing. I don't think the theorems are wrong; I think the paper slightly oversells how much they say about the method that was run.

Loose empirical reporting on a narrow setup. With one backbone, three seeds, five short-horizon tasks (capped at 5–15 rounds), and gains quoted as ranges, it is difficult to assess how large and how robust the improvement really is. The response-length reduction is framed as a pure efficiency win, but on these short horizons it also leaves open the opposite worry — that boundary-based truncation could penalize tasks where progress is genuinely slow-but-real and gets misclassified as stagnation. A per-task results table and at least one additional backbone would settle most of this.

Questions for authors
  1. The credit-distortion half of the argument — a long uninformative tail dragging down earlier credit through full-horizon GAE — seems like it would apply to any long-horizon RL problem, not specifically to shared-memory multi-agent systems. What is distinctively multi-agent about it, beyond the mechanism by which the stall arises and persists through shared memory? This is a clarification, not a demand for experiments; it would help scope the contribution.
  2. Could you report a per-task main table (final task metric for Vanilla PPO, Phantom-Control, and Random-Matched, with seed variance) rather than aggregate ranges? I'm asking for a clearer view of the runs you already have, not new ones.
  3. Is there a regime where the intervention hurts — e.g., a task with legitimately slow early progress that the detector cuts off prematurely? A short negative result or a discussion of this failure direction would strengthen the paper.
  4. How was k (and Δ_min) chosen per task, and does final task performance track the boundary-activation behavior shown in Table 1? Some sense of sensitivity here would help readers who want to deploy the detector.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3h7W

Thank you for the careful questions. We clarify system-level credit, task-wise results, and detector sensitivity.

Q1 — What is distinctively multi-agent? We agree that full-horizon GAE propagates later TD residuals into earlier advantages in general long-horizon RL. Our contribution is to characterize how persistent shared memory turns local low progress into a system-level credit problem. Sequential role proposals and environment feedback enter persistent memory, shaping later decisions; local improvement need not imply collaborative progress. Section 4.1 models this through the joint role–memory potential

Ψtsys=αΨtmem+(1−α)∑iwiΨi,t\Psi_t^{\mathrm{sys}}=\alpha\Psi_t^{\mathrm{mem}}+(1-\alpha)\sum_iw_i\Psi_{i,t}

and the collaborative stagnation region (CSR). Proposition 4.1 compares actual collaborative belief updates with Bayesian-oracle updates under the same action and observation. Conditional on CSR entry and its assumptions, Theorem 4.2 bounds degradation of earlier common system-level GAE; Lemma C.5 connects that common credit to independent role-wise, token-wise PPO gradients. The GAE identity is general; shared-memory stagnation and cross-role credit are our distinctive objects. PHANTOM-CONTROL sets a common boundary after complete shared-memory transitions without combining role policies.

Q2 — Per-task final metrics. We report the task-specific final evaluation scores (mean ±\pm SD across runs, percentage points), with the metrics of Table 2:

TaskMetricPCRandom-MatchedVanilla
CDEM95.83 ± 2.3366.67 ± 19.2562.50 ± 10.76
SPWord F127.16 ± 2.3024.65 ± 2.1322.18 ± 1.94
GNEM97.50 ± 1.4193.53 ± 2.0194.98 ± 2.97
PEBinary Sim.35.83 ± 4.8131.58 ± 4.1729.42 ± 12.50
MRTop-1 EM41.67 ± 1.4633.33 ± 2.0734.55 ± 4.10

For PE, success means cosine similarity ≥0.88\geq 0.88; MR evaluates Top-1 recommendation. PC has the highest mean final score in all five tasks. These complement Figures 2 and 4.

Q3 — When can intervention hurt? Low observable progress may precede useful delayed information; early masking then removes useful residuals and updates. Our original Table 1 quantifies this risk: in GN the Positive-Gain Rate among activated trajectories is 49.0% for k=1k=1 versus 6.4% for k=3k=3 (different activated subsets). Additional Figure 7(a) shows that k=1k=1 suffers a mid-training held-out accuracy drop before recovering, whereas k=3k=3 is steadier. Additional Table 4 shows that k=1k=1 confirms before round 10 on 94.67% of training trajectories, omitting 67.00% on average, versus 19.00% and 5.20% for k=3k=3. The corresponding final accuracies are 96.48±1.02%96.48\pm1.02\% and 97.50±1.41%97.50\pm1.41\%. Persistence makes intervention more selective.

Q4 — How are kk and Δmin⁡\Delta_{\min} selected, and does activation predict performance? We use task-level observable progress proxies after complete system transitions (Table 3), with k=6,5,3,2,2k=6,5,3,2,2 for CD, SP, GN, PE, MR respectively. Discrete proxies detect no candidate refinement; continuous proxies use calibrated low-progress thresholds (Appendix F). The latent CSR and practical trigger differ: the theorems characterize credit degradation after CSR entry and preservation at the ideal boundary; deployment uses a measurable progress proxy. Our original Figure 1 shows reduced CD progress around the detected boundary and higher retained-prefix raw advantage when later residuals are excluded. Original Random-Matched (Figure 4) holds activation count and the masked-tail-length multiset fixed but randomly reassigns boundaries; its weaker results support progress-aligned placement over masking quantity.

Sensitivity evidence further guides calibration. In the new GN sweep (Table 4), k=4k=4 triggers less but ends at 91.25±1.77%91.25\pm1.77\%, versus 97.50±1.41%97.50\pm1.41\% for k=3k=3; activation is not a monotone predictor of accuracy. New Figure 7(b) replays the same 40 PE trajectories at early/late checkpoints: increasing Δmin⁡/Δref\Delta_{\min}/\Delta_{\mathrm{ref}} raises trigger frequency and generally advances the trigger. New Figure 10 compares PC against Vanilla PPO on CD with Qwen2.5 3B/7B/14B backbones. Mean held-out Exact Match gains over PPO steps 40, 44, 48, and 49 are +2.50/+26.88/+3.13 percentage points. The gains vary with scale: 3B may lack stable useful coordination, limiting credit to preserve; 7B benefits as Vanilla deteriorates late; 14B Vanilla reaches 95.63% late-stage Exact Match, leaving little headroom. During the first ten training steps, 27/30 3B rollouts trigger versus 17/30 for both 7B and 14B. Equal trigger counts at 7B/14B but different gains show that intervention value also depends on learned coordination and baseline headroom.

Thank you for your response. I will keep my positive score.

AuthorsOpenReview ↗

Author Response to Reviewer 3h7W

Thank you for taking the time to read our response. We appreciate your feedback and the care you took in reviewing our work.

Reviewer_zoBDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies stalled collaboration progress in multi-agent active reasoning with persistent shared memory. It argues that a stalled collaborative tail can distort the credit assigned to earlier useful coordination. Motivated by this, the paper proposes PHANTOM-CONTROL which detects persistent low observable progress, sets a common optimization boundary, and retains collaborative prefix. Experiments compare the method with vanilla PPO, random-matched boundaries, and a role-local intervention, indicating that Phantom-Control achieves late-stage reward gains and reduces mean total generated response length.

Critical strengths
  1. The Random-Matched control preserves the intervention frequency and the complete masked-tail-length at each training step while randomizing their assignment across rollouts. This is a meaningful control for determining whether rollout-specific boundary placement matters beyond the amount of masking.
Critical weaknesses
  1. Insufficient delineation of the contribution relative to T3 [1]. Although T3 is cited in the related work, the paper does not sufficiently explain the close correspondence between the two theoretical frameworks and progress-based intervention rules. Most of the theoretical and methodological frameworks are directly inherited from T3 [1]. In particular, the core presented sections, including Proposition 4.1, Theorem 4.2, Proposition 5.1, Corollary 5.2, share almost the same math structure (changes happen in notations and potentially some adaptations in multi-llm settings) but the paper does not claim this. Explicit attribution and a result-by-result comparison are needed to distinguish adaptations of the earlier framework from additional arguments specific to multi-agent shared-memory dynamics.

  2. The experiments do not sufficiently identify the proposed shared-memory-specific mechanism. The adverse effect on early prefixes of stalled tails has been developed by T3. Given the acknowledged prior work on belief deviation and trajectory-tail contamination, the unique challenges from Multi-Agent LLMs settings beyond what T3 has established should be stated more clearly. In other words, a more specific and clearer comparison between T3 and this paper on the mechanism-level is needed.

[1] Zou, Deyu, et al. "Reducing belief deviation in reinforcement learning for active reasoning of llm agents." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  • Could the authors identify T3 as a direct theoretical and methodological precursor and delineate the additional contribution of the system-level formulation, common optimization boundary, boundary handling, and empirical controls? Making this relationship explicit would allow the incremental novelty to be assessed fairly.

  • Could the authors clearly specify the unique challenges from Multi-Agent LLMs settings beyond what T3 has established?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer zoBD

Thank you for the constructive review and for asking us to clarify our relationship to T3T^3. We recognize T3T^3 as a direct theoretical and methodological precursor. While our analysis and optimization share its foundations, our work arises from a different research question and develops a system-level formulation for persistent shared-memory collaboration.

Q1 — Relationship to T3T^3 and additional contributions. Our starting point is the structure of multi-agent active reasoning: role-specific agents repeatedly exchange intermediate results through persistent shared memory. Such persistence enables coordination, but also allows a locally unproductive update to influence other roles and later rounds. We seek to formalize when this propagation becomes system-level collaborative stagnation and how it distorts credit for earlier useful coordination. T3T^3 instead examines belief deviation and stalled tails along a single-agent trajectory. Our question concerns a failure mechanism induced by collaboration, not simply the use of single-agent truncation for more policies.

(1) System-level formulation. We explicitly model sequential role proposals, their common environment-facing action, and shared memory persistent across rounds. A role may improve while the collaborative system does not. We therefore define (Section 4.1) Ψtsys=αΨtmem+(1−α)∑iwiΨi,t,\Psi_t^{\mathrm{sys}}=\alpha\Psi_t^{\mathrm{mem}}+(1-\alpha)\sum_iw_i\Psi_{i,t}, combining role-specific and memory-induced beliefs to characterize the collaborative stagnation region (CSR). Theorem 4.2 establishes conditions under which post-entry TD residuals reduce earlier expected system-level advantage and can reverse its sign. Lemma C.5 further shows how this common advantage factors into independent role-wise, token-wise policy gradients. The result is a collaborative trajectory with a common credit signal, not isolated role trajectories.

(2) Common optimization boundary. PHANTOM-CONTROL evaluates persistent low observable progress after each complete system transition and sets one boundary H^\widehat H for the collaborative trajectory. It recomputes prefix-supported system-level GAE and applies the common prefix mask to all roles while retaining role-specific PPO updates (Section 5.3).

(3) Boundary handling. The full confirmation window is retained. At an artificial boundary, the final retained TD residual still bootstraps from Vsys(XH^)V^{\mathrm{sys}}(X_{\widehat H}), and subsequent residuals do not enter earlier credit. The environment reward is unchanged.

(4) Empirical controls. Our Random-Matched control (Section 6.3, Figure 4) preserves, at each PPO outer step, both the number of activated rollouts and the multiset of masked-tail lengths, but reassigns the boundaries randomly. Its weaker training dynamics support the importance of progress-aligned boundary placement beyond trajectory shortening. Our original Role-Local control (Figure 6) compares an estimator-local intervention with System-Level control. The new CD credit-cut experiment (Figure 8(a)) holds the system-level detector and boundary fixed, applying the prefix-supported intervention either only to the Estimator or to all roles; the latter shows more sustained late-stage Exact Match. Figure 8(b) additionally documents raw advantage sign reversals under both intervention scopes.

Q2 — Challenges specific to multi-agent LLMs. The challenge is cross-role propagation and shared credit. An unproductive output persists across roles and rounds through shared memory; local progress need not imply system progress. Section 4.1 compares actual collaborative updates with Bayesian-oracle updates under the same environment action and observation. The resulting gap combines role and memory beliefs and underlies Proposition 4.1's CSR conditions.

Theorem 4.2 and Lemma C.5 connect stagnation to learning: post-CSR TD residuals can degrade the common system-level advantage, which enters separate token-wise policy gradients across roles. This motivates complete-system progress detection and shared prefix GAE with independent role updates.

Our controls test these choices. The original Figure 6 shows more persistent System-Level gains than Estimator-local intervention, especially on CD and PE. New Figure 8 fixes the detector and boundary: applying the credit cut across all roles sustains stronger late-stage CD Exact Match than Estimator-only truncation. Both variants exhibit raw advantage sign reversals, showing that even local truncation changes credit. The original Random-Matched control (Figure 4) supports progress-aligned boundary placement rather than masking amount alone. New Figure 9 illustrates shared-memory propagation: masking a judgment increases candidate reduction by 24.0–32.6 percentage points and reduces repeated queries by 36.0–41.9 points. Together, they support system-level progress detection and cross-role credit control.

Thank you for the additional clarification. My concern remains the extent of the incremental contribution. Proposition 4.1, Theorem 4.2, Proposition 5.1, and Corollary 5.2 closely parallel T3’s stagnation analysis, advantage-inversion theorem, false-trigger bound, and credit-preservation result, respectively. The persistent low-progress detector also follows the same principle. Also, I recommend revising the manuscript to give T3 more explicit credit, clearly presenting the relevant theoretical framework and intervention principle as developments built on T3, and distinguishing inherited foundations from new contributions. I remain unconvinced that the theoretical and methodological advances extend sufficiently beyond adapting T3 to a shared-memory multi-agent formulation. I therefore maintain my original score.

AuthorsOpenReview ↗

Author Response to Reviewer zoBD

Thank you for clarifying your concerns about the connections to T3T^3. We acknowledge that parts of our analysis draw on T3T^3, and appreciate your suggestion to distinguish the inherited foundations from our contributions more explicitly. We will strengthen the attribution alongside the corresponding results and clarify the following three aspects.

First, our analysis concerns collaborative stagnation under shared memory and its effects on training credit across roles. Our system state includes both individual agents' beliefs and the belief induced by shared memory, and our system-level potential measures their joint progress. An improvement within one role can therefore be offset by deterioration in other roles or in shared memory, while a low-progress update can influence later roles and rounds through memory. Within this formulation, Proposition 4.1 gives sufficient conditions for system-level stagnation, and Theorem 4.2 characterizes how, under the stated assumptions, a stalled continuation reduces the expected credit assigned to earlier coordination. The role-wise factorization in Lemma C.5 further shows how the common system-level advantage weights each role's policy gradient. Thus, the same stalled tail can affect credit for earlier decisions across multiple roles. These dependencies motivate our system-level analysis and common credit boundary. We will make clearer what this formulation adds and how the individual results relate to T3T^3.

Second, complementary experiments examine the limitations of local credit cuts in this collaborative setting. The shared-memory masking experiment in Figure 9 shows that changing the shared judgment visible to downstream roles affects repeated queries and candidate elimination. Figure 6 compares Estimator-local and system-level intervention, while Figure 8 fixes the system-level detector and boundary and compares applying the credit cut to the Estimator alone versus all roles. Figures 10(b) and 10(c) extend this examination to a role-wise adaptation of T3T^3, in which every role independently monitors progress and sets its credit boundary after six consecutive local stagnation events.

Figure 10(c) records 27 local-progress resets across two seeds after at least six consecutive role transitions with an unchanged candidate set. Correcting a report or diagnosis can reset a role's stagnation counter without narrowing the candidate set. Such resets can delay local credit boundaries and retain more of the stalled collaborative continuation in that role's credit estimate. Because roles communicate through shared memory and their updates use a common system-level advantage, the consequences extend across the collaborative process. Consistent with this account, PHANTOM-CONTROL achieves higher two-seed mean Exact Match at all six evaluation checkpoints from updates 21 through 41 in Figure 10(b). Together, these results show that divergence between local and collaborative progress can affect training even when every role has its own detector and credit boundary. We interpret this comparison in light of differences in initial performance and comparison protocols.

Finally, role-wise progress detection requires criteria appropriate to each role's responsibilities. In the comparison in Figures 10(b) and 10(c), the Estimator uses candidate-report accuracy, the Finalizer uses diagnostic correctness, and the Planner uses whether query feedback reduces the candidate set. These distinct criteria reflect the division of responsibilities and require examining how each role's progress relates to progress on the shared task. As agent count, specialization, communication structure, and domain knowledge vary, selecting and coordinating these criteria may become more challenging. Understanding when local corrections translate into collaborative gains therefore warrants finer-grained study. We hope to investigate these factors in future work and examine the conditions under which role-level and system-level credit control are most effective.

Thank you again for encouraging us to clarify these distinctions. Your feedback has helped us connect the theoretical formulation to more direct experimental comparisons and explain our contribution in relation to T3T^3 more precisely.

Reviewer_DobXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the phenomenon of collaborative stagnation that emerges during sustained interaction among multi-agent LLMs with shared memory, and examines its impact on credit assignment. The authors identify Phantom Collaboration—a failure mode where a stalled collaborative tail undermines the advantage estimates of effective actions taken earlier, potentially even making their expected advantage negative.

Critical strengths
  1. The author introduces the issue of stagnation in active reasoning to the context of multi-agent collaboration with shared memory, and demonstrates the effectiveness of the proposed approach through rigorous theoretical derivation. For instance, the author provides an explicit decomposition of the GAE into prefix contributions and the contributions from the stagnant tail.
  2. To mitigate the distortion of credit assignment for earlier effective coordination caused by persistently low-progress collaborative trajectories, this paper designs a common optimization boundary that enables all agents to share a unified prefix mask at the system level to exclude tail TD residuals, while still preserving disjoint policy parameters and role-specific token-wise PPO updates at the optimization level.
Critical weaknesses
  1. Although this paper focuses on multi-agent collaboration, its methodological framework bears substantial similarities to the single-agent active-reasoning work T³. Specifically, T³ has already introduced the Belief-Trap Region (BTR), analyzed GAE advantage inversion, and proposed persistent low-progress detection and stalled-tail truncation to preserve credit assigned to earlier actions. The CSR formulation, advantage inversion analysis (Theorem 4.2), observable progress detector (Proposition 5.1), and prefix-supported GAE (Corollary 5.2) follow a broadly similar framework. Therefore, the methodological novelty appears somewhat limited, and it remains unclear what fundamentally new insights or mechanisms are introduced beyond extending T³ to the multi-agent setting.
  2. The experiments lack a direct comparison with T³, despite both works being evaluated on the same five types of tasks. I suggest comparing PHANTOM-CONTROL with both the original single-agent T³ and a multi-agent adaptation of T³ under comparable experimental conditions. Such comparisons would help assess the benefits of multi-agent collaboration and, more importantly, whether the proposed system-level intervention offers improvements beyond a straightforward adaptation of the existing T³ approach.
Questions for authors
  1. What are the substantive novel contributions of PHANTOM-CONTROL compared to T³? I would like the authors to provide a direct, point-by-point comparison of the corresponding contributions of the two papers, clearly highlighting the key methodological and theoretical differences.
  2. A more direct comparison with T³ would help clarify the distinct contributions of this work. I would be willing to raise my rating if these concerns are adequately addressed.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer DobX

Thank you for the thoughtful review and for suggesting a direct comparison with T3T^3. We clarify the shared-memory mechanism underlying Phantom Collaboration, its implications for temporal credit assignment across roles, and the distinction in our optimization boundary. We also add a CD comparison between System-Credit-Cut and Local-Credit-Cut (Figure 8), complementing our original Random-Matched results.

Q1 — Relation to T3T^3 and our contributions. Work on long-horizon multi-agent systems often seeks to improve collaboration through role specialization, communication, or memory design. We ask a complementary question: How can we preserve credit for earlier useful coordination once collaboration has stalled? T3T^3 connects stalled tails to belief deviation within a single-agent trajectory. Our work considers a distinct mechanism: a locally unproductive update can persist through shared memory, affect later roles and rounds, and allow subsequent TD residuals to distort credit assigned to earlier coordination. A related temporal credit-assignment issue may arise in bilevel decision-making or hierarchical RL, where unproductive lower-level execution affects credit for earlier upper-level decisions through the return. We regard these as directions for future investigation.

First, stagnation is characterized at the system level. One role may improve while other roles or shared memory offset that progress. Our system-level potential (Section 4.1),

Ψtsys=αΨtmem+(1−α)∑iwiΨi,t,\Psi_t^{\mathrm{sys}}=\alpha\Psi_t^{\mathrm{mem}}+(1-\alpha)\sum_i w_i\Psi_{i,t},

combines role-specific and memory-induced beliefs to characterize a collaborative stagnation region (CSR), rather than judging progress from any one role alone.

Second, the credit consequences reach multiple role policies. Each role has distinct trainable policy parameters, but uses a common system-level advantage. Conditional on CSR entry Eτ\mathsf E_\tau, Theorem 4.2 establishes under its assumptions that

E[A^t,presys−A^tsys∣Eτ]≥γκVρχsysΓtailsys(t,τ).\mathbb E[\widehat A_{t,\mathrm{pre}}^{\mathrm{sys}}-\widehat A_t^{\mathrm{sys}}\mid\mathsf E_\tau]\geq\gamma\kappa_V\rho_\chi^{\mathrm{sys}}\Gamma_{\mathrm{tail}}^{\mathrm{sys}}(t,\tau).

When this bound exceeds positive expected ideal-prefix credit, the expected full-horizon advantage becomes negative. Lemma C.5 shows how the common advantage enters independent role-wise, token-wise policy gradients. Thus, stalled continuation can affect credit for earlier decisions across roles, not only within one agent's trajectory.

Third, control is defined on the shared trajectory. PHANTOM-CONTROL detects persistent low observable progress after complete system transitions, sets a common optimization boundary, and recomputes prefix-supported system-level GAE. Post-boundary TD residuals no longer affect retained decisions; the boundary-state value bootstrap, original environment reward, and role-specific PPO updates are preserved (Section 5.3). The contribution is not a joint actor or merely applying separate truncations to more agents, but a shared boundary and credit signal defined by collaborative progress.

Q2 — Why apply the boundary across roles? Our controls examine credit scope and boundary placement.

First, the original Role-Local comparison (Figure 6) intervenes on the Estimator while system-level credit remains full-horizon. PHANTOM-CONTROL recomputes shared-trajectory prefix GAE for all roles. System-Level shows more persistent gains over Vanilla, especially on CD and PE, supporting shared-trajectory credit control.

Second, our new CD comparison (Figure 8) holds the system-level detector and boundary fixed. Local-Credit-Cut applies prefix GAE and post-boundary masking only to the Estimator; System-Credit-Cut applies them across all roles, retaining independent PPO updates. Figure 8(a) shows stronger sustained late-stage Exact Match with the system-wide cut. Figure 8(b) confirms local truncation changes credit: raw advantage sign reversals occur in both variants (Local: 32.5%/54.0%; System: 44.3%/56.4%, seeds 44/45). The rates diagnose credit changes, not performance. Thus, local credit-cut has an effect, but consistent cross-role application achieves stronger learning dynamics.

Third, the original Random-Matched control (Figure 4) preserves activated-rollout counts and masked-tail-length multisets while randomly reassigning boundaries. Its weaker training dynamics support progress-aligned placement, not merely the quantity truncated. The new shared-memory illustration (Figure 9) shows why collaborative state matters: masking a shared judgment raises candidate reduction by 24.0–32.6 points and reduces repeated queries by 36.0–41.9 points. This illustrates propagation, not credit truncation.

Together, these controls support detecting stagnation on the shared trajectory and applying its boundary consistently across independently optimized role policies (Lemma C.5).

Thank you for your detailed response and for addressing my concerns. Your clarifications and additional experiments have partially alleviated my concerns, and I am willing to raise my rating accordingly. However, I would be more convinced of the paper's contributions if you could further include the direct experimental comparisons with T3T^3 and its multi-agent adaptation, as suggested in Q2 of my original review.

AuthorsOpenReview ↗

Author Response to Reviewer DobX

Thank you for reviewing our response and acknowledging the value of the additional clarifications and experiments. Following your suggestion to include a direct comparison with a multi-agent adaptation of T3T^3, we have added a role-wise T3T^3 comparison on Circuit Decoding (CD) in Figures 10(b) and 10(c). Within the same three-role shared-memory architecture, each role monitors progress according to its function. The Estimator uses the accuracy of its candidate report, the Finalizer uses the correctness of its diagnosis, and the Planner uses whether query feedback reduces the candidate set. Each role independently maintains a stagnation counter and sets its credit boundary after six consecutive local stagnation events. This comparison examines whether applying a credit cut independently to every role can achieve the benefits of a common system-level boundary.

This comparison highlights a difficulty in multi-agent collaboration: progress within an individual role can diverge from progress on the overall task. Figure 10(c) records 27 local-progress resets across the two seeds after at least six consecutive role transitions with an unchanged candidate set. For example, correcting an earlier report or diagnosis can reset a role's stagnation counter without narrowing the candidate set. Such resets can delay local credit boundaries, allowing more of a stalled collaborative continuation to enter a role's credit estimate. Meanwhile, roles influence one another through shared memory, and subsequent interactions affect credit for earlier decisions through the shared system-level generalized advantage estimation (GAE). Consequently, the effects of locally determined boundaries extend into an interdependent collaborative process.

These observations complement the existing controls. Figure 6 compares Estimator-local and system-level intervention, while Figure 8 holds the system-level detector and boundary fixed and varies whether the credit cut applies to the Estimator alone or to all roles. The additional comparison in Figure 10(b) shows that independently monitoring progress and setting credit boundaries for every role can still yield different training outcomes from a common system-level boundary. PHANTOM-CONTROL achieves higher two-seed mean Exact Match at all six evaluation checkpoints from updates 21 through 41. Together, these results support measuring progress along the complete collaborative trajectory and using a common credit boundary, while differences in initial performance and comparison protocols should also be considered when interpreting the curves.

Furthermore, monitoring each role requires a progress criterion appropriate to its responsibilities, as illustrated by the three distinct criteria above. As role count, specialization, and communication structure vary, understanding when local corrections translate into overall collaborative gains warrants further investigation. In future work, we hope to study finer-grained progress detection across domains with different knowledge requirements and examine when role-level and system-level credit control are most effective. Thank you again for suggesting this comparison, which helped us clarify the issue through a more direct experiment.

Comments on the paper

1 comment

Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.