acceptodds
Under review as a conference paper at ICLR 2027

Coordination Scaffolds: Task-Dependent Withdrawal of Privileged Observations in Cooperative Multi-Agent Reinforcement Learning

Abstract

Training-only privileged observations can simplify reinforcement learning, but a deployable policy must eventually act without them. Existing guided-observability methods typically use one withdrawal recipe across tasks. We show that this assumption breaks in cooperative multi-agent reinforcement learning (MARL), where privileged state can scaffold the discovery of coordination itself. Across SMAC and MPE, the same withdrawal schedule can improve one task and degrade another: augmentation prevents bootstrap failure on the hardest map, while linear decay loses up to 0.205 win rate on another and fast decay is consistently poor on low-demand MPE tasks. Schedule-mismatch interventions, dose-matched timing controls, and withdrawal-recovery analyses show that these effects arise along the optimization path; the final removal produces little immediate disruption. We introduce Observation-Control Irreducibility (), a probe-conditioned diagnostic based on the predictive gain from modeling agents' local observations jointly rather than additively. An OCI-guided schedule uses this demand estimate to choose earlier or later withdrawal. It avoids the large fixed-schedule failures on the development tasks and, when frozen before training, is oracle-best on one informative held-out task and within 0.078 of the oracle on another. A QMIX replication preserves the task-dependent diagnosis but requires a different demand-to-schedule calibration. These results identify privileged observations as coordination scaffolds whose withdrawal should be conditioned on the task and learner.

Primary area:
reinforcement learning
Keywords:
cooperative multi-agent reinforcement learning, privileged information, centralized training decentralized execution, coordination, curriculum learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QacwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when training-only privileged observations should be withdrawn from cooperative MARL policies. The results show that a shared schedule can help or hurt depending on the task. This paper introduces Observation-Control Irreducibility (O1), a probe-conditioned predictive gap between joint and additive models of agents' local observations, to select a withdrawal schedule. Experiments are conducted on MAPPO on SMAC and MPE, schedule-mismatch and dose-matched controls, prospective held-out tasks, and a replication with QMIX, validating the effectiveness of the proposed method.

Critical strengths
  • This paper studies a practically important but underexplored problem.

  • The authors report paired statistical tests and include negative findings for discussion.

Critical weaknesses
  • Limited generalization to unseen tasks. The prospective evaluation includes only two informative held-out tasks, while the third remains unsolved under all schedules. OCI-guided scheduling achieves the best mean on one task but ranks second on the other, with one seed collapsing to 0.049. More held-out tasks are needed to establish its generalization ability.

  • The diagnostic depends on the probe distribution, target selection, and predictor capacity. Although the paper acknowledges the importance of probe quality, it lacks systematic sensitivity analyses and uncertainty estimates. It is therefore unclear whether O_1 provides a stable and cost-effective basis for schedule selection.

  • Limited transfer across learners. The MAPPO-calibrated OCI-guided schedule performs poorly on QMIX, with some seeds collapsing. This suggests that O_1 alone is insufficient to determine an effective schedule and that learner-specific calibration is necessary. The evidence is also limited to one additional learner and three seeds per condition.

  • Although OCI-guided scheduling achieves a higher observed worst-seed SMAC score than exponential decay, the difference is statistically unresolved, and exponential decay has a slightly higher mean SMAC performance. Given the heterogeneous results across tasks and the small number of seeds in several conditions, the claimed robustness advantage remains inconclusive.

Questions for authors

Please see weakness.

Reviewer_1X2nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates a focused but interesting question in MARL: when should privileged information be withdrawn during training? The authors propose conditioning the withdrawal schedule on task-level coordination demand, estimated using their proposed Observation–Control Irreducibility (OCI) metric. Experiments on SMAC and MPE show that this approach mitigates performance drops caused by fixed withdrawal schedules and remains competitive on held-out tasks.

Critical strengths
  • The paper studies an interesting and practically relevant question: when to withdraw training-only privileged information in cooperative MARL.
  • The empirical investigation includes timing interventions, exposure-matched controls, held-out tasks, and a second learner. The authors also disclose negative results and a failed replication.
Critical weaknesses
  • Limited empirical gains. The proposed scheduler does not demonstrate a clear and consistent performance advantage. In Table 1, it improves over standard training on only one of the three SMAC tasks, where simple exponential decay performs better. The MPE results are largely comparable to or worse than standard training, and OCI-guided scheduling underperforms standard training on all three QMIX tasks. These results support the importance of schedule choice more strongly than the effectiveness of the proposed rule.
  • Unclear interpretation of information withdrawal. The main implementation scales privileged state by α\alpha, which mathematically preserves its information whenever α>0\alpha>0. The observed effects could therefore partly reflect input scaling and optimization dynamics rather than reduced information availability. This distinction weakens the interpretation of the method as an information curriculum.
  • Insufficient justification and practical validation of OCI. OCI measures the predictive advantage of jointly modeling local observations over an additive predictor, rather than directly measuring the benefit of privileged information. Its connection to the appropriate withdrawal time remains indirect. Furthermore, obtaining reliable estimates requires competent probe trajectories on SMAC, while the scheduling rule requires learner-specific calibration. The practical advantage over a small schedule search is therefore unclear.
  • Limited statistical support and generalization evidence. Appendix H reports no statistically significant task–schedule interactions in the discovery grid, and the selected Bernoulli-mask interaction fails to replicate on independent seeds. The held-out evaluation contains only two informative tasks, with a collapsed OCI-guided seed on one of them. Although these limitations are disclosed, they leave substantial uncertainty about the reliability of the claimed benefits.
  • Repetitive and insufficiently focused presentation. The coordination-scaffold argument is repeated across several sections, while the distinctions between task-dependent schedule effects, the validity of OCI, and the effectiveness of OCI-guided scheduling are not sufficiently clear. A more concise presentation separating these claims and their supporting evidence would make the contribution easier to assess.
Questions for authors
  1. After accounting for probe acquisition and calibration costs, does OCI-guided scheduling outperform standard training or a small search over simple withdrawal schedules under the same total computational budget?
  2. Since scaling privileged state by a nonzero α\alpha preserves its information, how do the authors distinguish the effects of information withdrawal from those of input rescaling?
  3. What evidence establishes that OCI predicts the appropriate withdrawal time, beyond measuring cross-agent predictive interactions? Would simpler diagnostics provide comparable schedule selection?
  4. How sensitive are the results to the probe policy and scheduling thresholds, and what must be recalibrated for a new learner, particularly given the poor transfer to QMIX?
Reviewer_hj2ROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study task-dependent removal of training-time global-state inputs from decentralized actors. It compares withdrawal schedules on SMAC/MPE, investigates timing and recovery, and proposes OCI: a joint-versus-additive observation-prediction gap mapped to a sigmoid withdrawal schedule. Two informative prospective tasks and a QMIX replication test the selector's limits.

Critical strengths
  1. The research problem is meaningful and interesting.

  2. Meaningful controls rather than only a winning mean. Dose-matched timing, always-privileged policies, distillation, and alternative removal interfaces examine different explanations. Disclosure of independent-mask non-replication and QMIX failure improves interpretability.

  3. A prospective task split. Freezing the selector before testing is preferable to reporting only development-grid tuning. It is a useful empirical extension of guided observability and privileged-information dropout, although those works already establish the underlying training interface.

Critical weaknesses
  1. OCI is not validated as the information deficit that privileged state resolves (§3.2, Appendix B). Both predictors use the union of local observations; their gap measures nonadditivity under a probe distribution, not information present only in privileged state or the need for a particular exposure duration. The main text/appendix does not sufficiently specify the predictor capacities, fitting details, target horizons, and uncertainty of the maximum-over-target statistic. SMAC probes rely on competent checkpoints from earlier projects; random/scripted and 10%-budget probes fail. This makes the diagnostic dependent on already having useful policies and leaves its acquisition cost unaccounted for. MPE's headline selected runs partly use a different estimator. A capacity-controlled nonadditivity test and a complete checkpoint/probe provenance and cost specification are necessary to support the claimed diagnostic contribution.

  2. The automatic schedule and mechanism claims exceed the evidence (§§4.3–4.6). For any α>0, multiplying state by α does not remove its information; it changes input scale and optimization. The stricter availability-masking interaction fails independent replication. Timing/recovery controls therefore localize an effect to training, but do not identify coordination discovery as its cause. OCI-guided has lower development SMAC mean than exponential, a statistically unresolved worst-seed advantage, a 0.049 held-out collapse, and is worst under QMIX. Only two informative held-out tasks remain; the SMACv2 task is unlearned and truncated. There is no constant-sigmoid midpoint control that separates OCI selection from the chosen schedule family. The defensible result is interface/task/learner sensitivity, not a demonstrated reliable scheduler.

Questions for authors
  1. Specify the probe predictors, comparable capacity/regularization, exact targets and horizons, data sizes, and OCI uncertainty. What evidence connects joint-local nonadditivity to missing privileged information?

  2. What did acquiring the competent SMAC checkpoints cost, and what behavior/provenance did held-out-task probes use? Report full selection cost rather than assuming those checkpoints are free.

  3. Are fixed-midpoint sigmoid results already available? How does the unified-OCI MPE reference run change the headline grid, and which mechanism conclusion survives the independent Bernoulli-mask non-replication?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.