acceptodds
Under review as a conference paper at ICLR 2027

SRC: SINKHORN RESPONSIBILITY CREDIT FOR SPATIAL MULTI-AGENT REINFORCEMENT LEARNING

Abstract

Credit assignment in cooperative multi-agent reinforcement learning (MARL) is commonly formulated as attributing a shared team return to individual agents through value estimates, counterfactual baselines, or decomposed utilities. In spatial and object-centric tasks, however, coordination also depends on which agents should cover which landmarks, goals, resources, or targets. Shared rewards and global shaping signals can provide useful team-level feedback while leaving this assignment pattern unresolved. We propose Sinkhorn Responsibility Credit (SRC), a transport-based credit module that models this pattern as an entropic optimal-transport plan between agents and task objects. SRC retains the plan as a soft responsibility matrix and converts it into dense agent-specific credit. This differs from global optimal-transport shaping, which uses the same geometric objective but broadcasts a single team-level scalar. Under a fixed policy backbone, marginal SRC improves MPE simple spread returns relative to shared rewards by 5.07 to 20.03 points and relative to global optimal-transport shaping by 2.71 to 16.50 points, with larger gains as the number of agents grows. This isolates the central mechanism: the same spatial objective is more informative when its transport assignment is returned to individual agents than when it is collapsed into a global potential. On VMAS navigation, static and temporal SRC scale more reliably than leave-one-out marginal responsibility: across N=4,6,8 and 20 matched seeds, SRC-S wins 60/60 and SRC-T wins 59/60 seed-wise comparisons against shared rewards with paired p<7×10^-8 at every team size. Difference rewards remain the strongest MPE baseline because they evaluate task-utility counterfactuals directly; SRC instead provides an assignment signal from observed agent-object geometry. Auxiliary learner checks, hybrid experiments, BenchMARL/TorchRL MAPPO runs, and stress tests further identify the regime in which SRC is most effective: tasks where observable objects and meaningful costs expose decision-relevant assignment structure.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Credit Assignment, Optimal Transport, Sinkhorn Algorithm, Spatial Coordination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MH2JOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focus on the credit assignment problem in spatial multi-agent reinforcement learning. It uses entropy-regularized optimal transport to derive assignments from geometric costs and generate individual reward signals. Experiments show improvements on coverage and navigation tasks, while benefits are limited in more complex settings.

Critical strengths
  1. It provides a simple reward-shaping mechanism that derives agent-specific rewards from optimal transport over geometric costs.

  2. It shows performance improvements in the main MPE and VMAS experiments, supporting its effectiveness in spatial coverage and navigation tasks.

Critical weaknesses
  1. The scope is relatively narrow, and the overall novelty appears limited. Basically SRC formulates geometric costs(reward) as an entropy-regularized optimal transport problem, solves it using Sinkhorn iterations to construct several types of rewards. The contribution beyond combining these established components is not sufficiently demonstrated. Moreover, its effectiveness depends on the task property (geometric proximity), which may not hold in all tasks. This is also reflected in the experiments, where clear gains are concentrated in coverage and navigation tasks.

  2. The use of Sinkhorn is not well justified. Why ot is needed? It is unclear why Sinkhorn-based soft assignment is preferable to simpler alternatives, such as nearest-target heuristics or Hungarian-based hard assignment. I would not be surprised if that these alternatives would perform worse, but evaluating them would strengthen the rationale for the design and validate the proposed mechanism.

  3. The positive evidence mainly comes from spatial coverage and navigation tasks. The lack of consistent gains on more complex tasks, like GRF or SMAC. The broader use of the method is uncertain.

Questions for authors

See above

1 comment

Thank you for the detailed response and additional experiments. The new comparisons with simpler assignment methods and the credit delivery controls address much of my concern about the experimental validation. In particular, the results provide useful evidence that agent-specific credit delivery is beneficial.

However, my main concern about the limited novelty remains. The new results do not establish a clear advantage of Sinkhorn over simpler alternatives such as Hungarian matching, and the effectiveness is still largely limited to spatial tasks. While I appreciate the more careful positioning of the contribution, I will maintain my original score.

Reviewer_UNk3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sinkhorn Responsibility Credit (SRC), a reward-shaping method for cooperative multi-agent reinforcement learning (MARL). SRC constructs an entropic optimal-transport plan between agents and task objects, interprets this plan as a soft assignment of responsibility, and derives agent-specific learning signals. Experiments primarily demonstrate improvements on MPE coverage and VMAS navigation under a fixed learning backbone. Additional learner comparisons, hybrid methods, and interaction-heavy environments investigate where geometric responsibility provides useful information and where it becomes insufficient.

Critical strengths
  • This method is simple and interpretable for structural assignment structure. Assigning credit to which agent should cover which object is relevant to cooperative navigation and coordination.
  • The main MPE and VMAS evaluations use 20 seeds, with confidence intervals and paired comparisons. I appreciate the inclusion of Difference rewards, alternative learning backbones, and negative results on more interaction-heavy tasks.
Critical weaknesses
  • The abstract does not follow the required format.
  • I would like to see learning curves with original evaluation return, coverage, and success plotted over policy updates for MPE and VMAS, with uncertainty across the matched seeds.
  • The necessity of the Sinkhorn construction is not wrt simpler assignment methods. The experiments compare against shared rewards, global OT, and task-utility Difference rewards, but do not show whether globally balanced soft assignment is necessary. Nearest-object shaping, a row-wise softmax over object costs, and hard minimum-cost matching are needed. This matters because the strongest results occur in tasks whose structure is already closely aligned with the chosen distance cost.
Questions for authors
  • Can the authors provide learning curves over policy updates and environment interactions for the main MPE and VMAS experiments, using original task metrics and uncertainty across the 20 seeds?
  • Does SRC still outperform a baseline that broadcasts the mean of the same per-agent SRC credits to every agent?
  • Would randomly permuting the credits across agents destroy performance?
Reviewer_XPTQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Sinkhorn Responsibility Credit (SRC) for spatial MARL. It converts spatial agent-object costs into a soft Sinkhorn responsibility plan and uses it to produce agent specific credit.

Critical strengths
  1. The idea of reward shaping based on agent's spatial information sounds to be simple and useful as agent's feedback.

  2. The experimental verification is done under 20 seeds which demonstrate its consistency over environment randomness.

Critical weaknesses
  1. Lack of comparison to the currently available algorithms or SOTA as baselines (e.g. STAS: Spatial-temporal return decomposition for solving sparse rewards problems in multi-agent reinforcement learning). SRC has been compared against shared reward, global OT, and difference without any references as described in Line 302-310. Thus, I assume these baselines are developed by the authors.

  2. The definition of Spatial Multi-agent Reinforcement Learning is not clearly defined and explained. As the authors motivated and stated the problem specifically in spatial and object-centric domains, it is expected to have a formal definition of this type of MARL problem.

  3. As an impact of the weakness point number 2, the results on "Stress-test environment" are remote to the specific problem as mentioned in the introduction part (Line 43-45). Despite the results are contra productive, it perhaps not necessary to have an experiments on this type of environments. It also appears that this SRC method is an environment or problem specific approach (or cannot be applied in general credit assignment MARL problem).

  4. In Eq. 10, the author define a coefficient (\lambda) to control the shaping strength, however I cannot find the sensitivity analysis of this coefficient.

  5. The main objective of ablation studies are rather unclear. It should be decomposed from the SRC features by removing a single or some part of them. However, it compares against SRC variants.

Questions for authors
  1. What is the main reason of using the distance only as a responsibility feedback? As spatial MARL tasks may have different objectives and there always is a possibility where distance alone cannot be used (as stated in line 244-245).

  2. What is the reason to use sinkhorn iteration algorithm in this method?

  3. Do SRC variants only work for specific environments? As shown in Table 1 and Table 2, MPE and VMAS have different SRC variant results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.