acceptodds
Under review as a conference paper at ICLR 2027

Team Calibration for Multi-Agent Imitation under Decentralized Information

Abstract

Multi-agent imitation systems must turn centralized demonstrations into high-value decentralized behavior, because teacher information and team objectives are not preserved by separate local decisions. Overcoming this mismatch requires calibration to executable team utility, rather than accurate reproduction of agentwise action labels. Local imitation is well suited to decentralized deployment, yet its reliance on teacher marginals leaves the utility-optimal team protocol unidentified when value depends on joint behavior, especially in symmetric teams with private signals. Within that setting, joint-occupancy imitation captures demonstrated coordination through distribution matching, but its target remains tied to demonstrated behavior and cannot resolve utility interventions that preserve the demonstrations. We therefore propose Structured Team-Calibrated Projection (\stcp), which constructs a fixed executable portfolio by applying public-context routing and shared-clock switching to theory-identified decentralized atoms and selects its deployed member by held-out team value; our analysis provides an exact phase characterization, a heterogeneous convex-closure result, and a finite-portfolio selection bound. Experiments on controlled utility interventions, synchronization tasks, and graph consensus show that recovers the predicted protocol transitions and improves team value over representative decentralized baselines.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z3BaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the gap between local imitation accuracy and team utility under decentralized information. It shows when local teacher statistics fail to identify a good team policy. For symmetric binary teams, the analysis characterizes when agents should follow private signals or use a common convention. It also studies heterogeneous rules, private randomization, and time-dependent protocols. The proposed method, STCP, constructs a finite set of executable policies. It combines local learners, utility-aware learners, and predefined protocol components. The validation team's return selects the deployed policy.

Critical strengths

The reviewer thinks experimental evaluation provides stronger evidence within those environments than sampled average returns alone. In addition, selected benchmarks are reasonable.

Critical weaknesses

Although the reviewer finds good strengths in experimental results, critical weaknesses are present. The reviewre’s biggest concern is that the COGNAC comparison does not establish comparable search and evaluation costs. The validation grid contains 9216×10249216 \times 1024 candidate-state pairs. Anonymous IQL trains for 100,000 environment steps. The reviewer acknowledges that these quantities measure different operations, so they do not establish a direct cost ratio. However, there are no information about the total environment steps, runtime, or evaluation cost for each method. Next, the reviewer thinks that the paper do not provide enough ablation study, which isolate the contribution of the proposed protocol structure.

Questions for authors
  1. What information does each method receive about team utility? Which experiment isolates the benefit beyond access to the target utility?
  2. The reviewer wonders how the method controls candidate approximation error outside the exact binary setting. Can the method detect that all available candidates are poor?
  3. Following the paper, the authors exclude deterministic stationary rules that use only the current position. However, a recurrent policy can infer elapsed time from its local history. How does the synchronization result extend to policies with memory?

  4. COGNAC supports probabilistic influence from neighboring agents. What adjacency matrix does the experiment use? Are its transitions deterministic?
  5. The static phase result assumes a shared rule without agent identifiers. How do the main comparisons change when agents have identifiers or separate parameters?
Reviewer_PHfWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows, through proof and simple setup, the gap that can happen between optimizing local imitation objectives and optimizing team utility under decentralized execution. The authors provide possible policy atoms that can be optimal in different cases, with proof explaining these conditions. Finally, they construct a portfolio of these policies, together with routing and switching protocol, and select the best-performing policy from this portfolio based on validation team reward. They evaluate the method on controlled utility interventions, synchronization tasks, and graph consensus.

Critical strengths

Perhaps the most interesting part of the paper for me is the setup and examples in which they prove the mentioned gap, and also the proof showing why certain policies can be optimal in the studied setting. The examples are well motivated, and the proofs, especially the private-evidence and convention policies, are interesting. The experiments also support their argument in different setups.

Critical weaknesses

What I don't understand in this paper is the significance of Theorem 3. I believe the bound on selecting the best policy among the candidate in the portfolio is really straightforward, as it follows from standard concentration argument. It would be a lot more desirable to bound the gap between the best policy in the portfolio and the optimal executable decentralized team policy. The paper introduces this approximation gap, but does not bound it beyond some restricted theoretical setting in Theorem 1. And it is unclear how good the constructed portfolio approximates the optimal policy in a more general environment.

The other thing was that the paper was really hard to follow, and some relatively straightforward arguments were turned into confusing sentences with unnecessary explained wordings and terminology. In particular, the connection between the calibration setup, the binary-team analysis, and the portfolio construction was difficult to follow. This made it harder to understand which part of the proposed method directly followed from the theoretical results.

Also, I believe that related work should help motivate the method, but in the case of this paper, it mostly introduces relevant papers from different fields without sufficiently explaining how they connect to the core of the proposed method. Although the paper discusses these connections, it is still unclear when the proposed portfolio construction offers an advantage over existing methods that already optimize team reward under decentralized execution.

Another concern is understanding where the performance improvement comes from. The paper includes a validation-selected IQL/VDN/QMIX baseline and a switching ablation in COGNAC, which are useful comparisons. However, in the action-dependent experiments, it is still unclear how much of the improvement comes from the coordination structures and how much comes from having a broader collection of policies trained with different objectives. In the action-dependent experiments, a comparison keeping the candidate-generation objectives and training budgets fixed, while varying whether public-context routing is allowed, would help clarify this.

Questions for authors

Is it possible to bound the gap between the best policy in the constructed portfolio and the optimal executable decentralized team policy beyond the restricted setting already covered? If additional assumptions are needed, what would those assumptions be?

From what I understand, portfolio construction first needs to decide which type of policies and composition operators to allow. Based on what are these choices made in a new environment? The theoretical results motivate particular choices in specific settings, but is there a more general principle for choosing the atoms, public-context routing variable, and switching times?

For example, when switching is allowed, the method considers policies constructed from different atoms and predefined switching times, and compares them with the other candidates using validation team reward. Since the selection criterion is the same, selecting the best switching policy first and then comparing it with the other atoms appears equivalent to selecting over the whole portfolio, provided the same validation scores are used. Is this understanding correct? If so, could the paper clarify their main contribution about candidate construction beyond the final validation-based selection?

Reviewer_bA3DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The problem is whether demonstrations from a centralized expert, who conditions on the global state, determine the joint behavior a decentralized team should execute for its team utility when each agent acts only from its local history. Matching each agent's teacher marginal does not identify that behavior when the utility depends on the joint action. STCP places decentralized candidates, including ones trained with team value, into a finite portfolio fixed before validation, combines them by public-context routing and shared-clock switching, and deploys the candidate with the highest held-out team value. The main limitation is that this does not close the gap. When the validation set can score the team, deployment only selects inside that fixed portfolio. When it cannot, and the two utilities have incompatible optimal behaviors while the method does not receive the utility, identical demonstrations do not separate those behaviors. The exact comparison between following the private signal and everyone playing one common action applies to anonymous agents, each with one binary signal and one binary action, and a score monotone in the number correct. With one shared rule the optimum is the better of the two; under discrete convexity, letting each agent use its own rule does not improve that value. The reported experiments include paired utility interventions, ten-step sequences, action-dependent binary tasks, COGNAC, and CASEC Hallway.

Critical strengths

The first strength is a clear cutoff among the rules. There are four deterministic rules: copy the signal, flip it, or play one of the two constants. When the signal is at least as good as a coin flip, monotonicity removes the flip, so a shared rule chooses between copying the signal and everyone taking the same action. When the team score is discrete convex, one shared rule is enough, and giving each agent a different rule does not help. The convexity condition is sharp: the quorum example shows a role split can beat every shared rule.

The second strength is that some checks hold. The paired setup keeps the demonstrations fixed and changes only the team objective, so the chosen behavior follows the objective. On Hallway, the short argument rules out deterministic stationary local rules, and it holds; randomized stationary rules are outside it. On COGNAC, all 1,022 non-consensus states are enumerated only on a ring of ten agents. Through size 20, every observed non-consensus case is explained by the period-3 family, the period-4 family, or both.

Critical weaknesses

W1. The method does not close the gap it states. Theorem 3 only bounds selection inside a finite portfolio fixed before validation. When the validation set can score the team, STCP picks the highest score in that set. When it cannot, and the two utilities satisfy the incompatibility condition in Eq. 3 while the method does not receive the utility, identical demonstrations do not separate those behaviors.

W2. Table 1 does not show a newly learned behavior. The paper enumerates all 16 shared deterministic policies. For 4, 6, and 8 agents at signal accuracies 0.70, 0.70, and 0.75, Local BC, Agreement, and Global branch are 0.6161±0.00050.6161 \pm 0.0005, 0.5586±0.00080.5586 \pm 0.0008, and 0.5502±0.00100.5502 \pm 0.0010. STCP is 0.7466±0.00120.7466 \pm 0.0012, 0.7487±0.00090.7487 \pm 0.0009, and 0.7508±0.00090.7508 \pm 0.0009. Table 8 prints the paired differences versus local BC as 0.130480.13048, 0.190150.19015, and 0.200600.20060. The text says STCP adopts a convention when everyone must be correct and follows private information when one correct agent is enough. Local imitation, agreement, and the global branch recover one shared rule. Figure 1 reports agreement with the predicted branch in 40 of 40 runs. The ±\pm values are the sample standard deviations the paper reports across five seeds.

W3. Some reported gains recover a rule the paper already states, or a protocol found by search. On Hallway, the successful rule is the one already stated: wait, then move toward the goal. On COGNAC, the final STCP candidate uses no learned update and is selected from 9,2169{,}216 two-phase protocols, while anonymous IQL is a network trained for 100,000100{,}000 environment steps. These counts use different units. The paper says the baselines share STCP's information structure and observations, but the listed IQL inputs do not include the shared clock used by the selected STCP action. On the action-dependent task, IQL, VDN, and QMIX already use the same validation selection, and STCP also includes candidates trained with the team utility.

W4. The paper says STCP keeps the higher mean on all three action-dependent transitions, and reports positive paired 95% intervals on the wrong-majority and odd-parity transitions. The higher disagreement mean is a point estimate. In Table 8, that comparison against the selected IQL, VDN, or QMIX run has mean difference 0.027330.02733, a paired 95% interval from −0.01942-0.01942 to 0.074070.07407, and 3 wins out of 5 runs. In the ten-step sequence, actions do not change the target, and the paper treats that task as a contextual bandit with discount γ=0\gamma = 0.

W5. The negative statements are immediate once their conditions hold. The paper says the proof of Proposition 1 is immediate: equal local observation-and-label laws give separable surrogates the same risk. Corollary 1 says that if the method does not receive the utility and the optimal behaviors are incompatible as in Eq. 3, identical demonstrations do not separate them. When the team loss is a positively weighted sum of the individual losses and each local surrogate is calibrated for its own loss, the paper already notes that the product of the local Bayes rules minimizes team risk, so there is no gap in that case.

W6. Theorem 1 scores how many actions equal a latent bit ZZ. Theorem 2 does not count that number directly: its value is 12(Bg(r0)+Bg(r1))\frac{1}{2}\left(B_g(r_0)+B_g(r_1)\right), with r0,r1r_0,r_1 defined from Pr⁡(Xi=Z∣Z)=p\Pr(X_i=Z\mid Z)=p. Section 3 defines the team risk against the teacher-action vector YY. Neither theorem mentions the teacher or demonstrations, and the paper never states that each teacher label YiY_i equals ZZ.

W7. Table 5 gives Hallway results with no seed dispersion. It prints Reactive BC 0.06250.0625 and History BC 0.43750.4375. The main instance enumerates 64 initial states, and these two printed values are 4/644/64 and 28/6428/64. The text says MIFQ succeeds from every initial state in four of five runs, while the table lists 0.95000.9500. The baselines Hand boundary sweep and Stationary-93 are never defined.

Questions for authors

Q1. In Table 1, are the reported scores the average of the two team objectives? For 4, 6, and 8 agents, Local BC, Agreement, and Global branch are tied at 0.6161±0.00050.6161 \pm 0.0005, 0.5586±0.00080.5586 \pm 0.0008, and 0.5502±0.00100.5502 \pm 0.0010, while Conditional occupancy is 0.5014±0.00220.5014 \pm 0.0022, 0.4981±0.00170.4981 \pm 0.0017, and 0.5016±0.00180.5016 \pm 0.0018. The text says the first three recover one shared rule, and the regime-conditioned joint-marginal objective selects a constant rule. Are the tied columns that shared rule, and is Conditional occupancy the constant rule?

Q2. When the validation set cannot score the team, which candidate does STCP deploy, and what in the identical demonstrations identifies it when the two utilities satisfy Eq. 3?

Q3. On COGNAC, does anonymous IQL receive the shared clock? The paper says the baselines share STCP's information structure and observations. The listed IQL inputs are the local ring ordering, a shared recurrent network without agent identifiers, and the previous action. The selected STCP action also uses the shared clock.

Q4. Does each teacher label YiY_i equal the latent bit ZZ? Section 3 uses the teacher-action vector Y=(Y1,…,Yn)Y=(Y_1,\ldots,Y_n). Theorem 1 counts actions equal to ZZ. Theorem 2 instead uses 12(Bg(r0)+Bg(r1))\frac{1}{2}\left(B_g(r_0)+B_g(r_1)\right), with r0,r1r_0,r_1 defined from Pr⁡(Xi=Z∣Z)=p\Pr(X_i=Z\mid Z)=p. Neither theorem mentions the teacher.

Q5. In Table 5, how is MIFQ's 0.95000.9500 obtained from the five runs? The text says MIFQ succeeds from every initial state in four of five runs. What are Hand boundary sweep and Stationary-93?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.