acceptodds
Under review as a conference paper at ICLR 2027

LOD: Latent Objective Discovery in Heterogeneous Multi-Agent Reinforcement Learning

Abstract

In multi-agent reinforcement learning (MARL) settings, agents are often heterogeneous with diverse intrinsic utility functions. In this study, we introduce **L**atent **O**bjective **D**iscovery (**LOD**) strategy among agents to decompose their distinct utilities into mixed underlying objectives in a low-dimensional latent space. Latent objective discovery aims to capture flexible and abstract coordination patterns among agents and each abstract objective is associated with a shared value function. During learning, all the learned values are aggregated by each agent according to their personalized weights learned in an adaptive manner. Therefore, agents inherit objective-level estimates for policy updates and value learning, enabling structured information sharing without requiring access to other agents’ policies. Algorithmically, we incorporate our approach into Multi-Agent Proximal Policy Optimization (MAPPO) to exploit this structure. Theoretically, we establish convergence guarantees under linear function approximation within the actor-critic framework. Empirically, we extensively validate the advantage of introducing latent objective discovery in popular multi-agent testbeds with heterogeneous agents.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Heterogeneous Agents, Latent Objective

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GBtPOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Critical strengths

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Critical weaknesses

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Questions for authors

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

1 comment

We thank the reviewer for the positive assessment of our work. Please let us know if you have any further comments or feedback. We appreciate your suggestions.

Reviewer_ihrEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Latent Objective Discovery (LOD), a framework for exploiting shared structure among agents with heterogeneous rewards. LOD decomposes agents’ reward signals into shared latent factors and agent-specific mixing weights, learns critics for the latent objectives, and combines their estimates to guide individual policy updates. The framework is instantiated with MAPPO and extended to shared-reward settings using counterfactual local advantages as surrogate signals. The paper provides convergence analysis for a generic average-reward actor–critic algorithm under linear function approximation and additional assumptions. Experiments on MPE, LBF, RWARE, and LLM-grounded ChartQA and MuSiQue strategy-selection workflows show improvements over MAPPO in several settings, with sensitivity analyses of the latent dimension and mixing-weight update rate.

Critical strengths
  1. Structured value sharing with agent-specific preferences. LOD uses shared latent reward factors and personalized mixing weights to enable value sharing while retaining differences among agents’ objectives. This is a useful framework design for balancing shared learning with individual preferences in heterogeneous MARL.
  2. Empirical evidence across several cooperative settings. The evaluation covers navigation, foraging,warehouse coordination, and LLM-grounded trategy-selection workflows. Improvements over MAPPO in several settings, particularly the warehouse tasks, provide encouraging evidence of the framework’s practical usefulness beyond a single environment.
Critical weaknesses

1 The experiments do not sufficiently establish the meaning of the learned factors or the source of the performance gains. Comparisons primarily against MAPPO and different choices of K show that the LOD module can be useful, but do not isolate the contributions of low-rank reward approximation, multi-head value learning, and personalized mixing weights. Parameter-matched baselines and controlled experiments with known latent reward factors would help substantiate the central claim of latent objective discovery. 2 The theoretical guarantees need further clarification. Appendix B.1 assumes a zero conditional mean for the residual without sufficient justification, while Appendix B.4 contains inconsistencies in the critic stability argument. These issues weaken confidence in the stated guarantees.

Questions for authors

1 What evidence distinguishes the benefits of discovering latent objectives from those of low-rank reward denoising or a multi-head critic architecture? Do you have parameter-matched comparisons or controlled experiments showing recovery of known latent reward factors? 2 How were K and the mixing-weight update rate selected without using the final evaluation results? What additional training time and hyperparameter-search budget does LOD require relative to MAPPO? 3 How do you maintain alignment between the latent factors and their corresponding critic heads as the factorization is updated online? Have you observed factor permutations or representation drift, and how do these affect training stability?

2 comments

We would like to sincerely express our gratitude to the reviewer for their time and effort in reviewing our paper. We address each concern you raised and are happy to answer any further questions.

Response to Weaknesses

1.The experiments do not sufficiently establish the meaning of the learned factors or the source of the performance gains. Comparisons primarily against MAPPO and different choices of K show that the LOD module can be useful, but do not isolate the contributions of low-rank reward approximation, multi-head value learning, and personalized mixing weights. Parameter-matched baselines and controlled experiments with known latent reward factors would help substantiate the central claim of latent objective discovery.

Response: Thank you for raising this point and we have performed ablation studies to isoloate the contributions of each component in LOD-MAPPO.

(1) To examine whether the performance gain is explained by multi-head critic, i.e., the increased critic model size, we compare our method with a parameter-matched MAPPO with KK multi-head critic without reward factorization. The comparison results have been updated in the revised paper. As shown in the right panel of Figure 4 in Spread (continuous) and left two panels of Figure 7 in LBF and RWARE-tiny, MAPPO with KK multi-head critic only performs similarly to standard MAPPO, substantiating that the performance gain of LOD-MAPPO is not related to introducing additional parameters in the critic's output layer.

(2) To investigate whether the performance gain is explained by learned personalized mixing weights, we compare LOD-MAPPO with a variant that keeps Γ\Gamma fixed at uniform weights throughout training across Spread(continuous) in Figure 4 and Figure 8 (ChartQA) in the revised paper. The results suggests that the uniform weight cannot explain the performance gain of LOD-MAPPO over MAPPO and the learned weights in LOD is key for the performance gain.

Finally, for the meaning of the learned factors, please refer to our response to Question 3 in the following.

2.The theoretical guarantees need further clarification. Appendix B.1 assumes a zero conditional mean for the residual without sufficient justification, while Appendix B.4 contains inconsistencies in the critic stability argument. These issues weaken confidence in the stated guarantees.

Response: We woud like to clarify that the condition of zero conditional mean for the residual detailed in Appendix B.1 is very common and widely adopted in linear factor model [1] in classical statistical method. Here, the residual ϵt\epsilon_t represents the idiosyncratic component of the reward that is not explained by the latent factors. Following the standard innovation interpretation of white noise in factor models, it is zero-mean and exogenous to the initial state-action condition. We also added this justification and revised Appendix B.1 accordingly in blue. As for Appendix B.4. we thank you for pointing the inconsistency in the proof here and have revised the critic stability proof by replacing the Perron–Frobenius eigenvalue argument with a more direct quadratic-form analysis. The changes are in blue in the revised paper.

Response to Questions

1.What evidence distinguishes the benefits of discovering latent objectives from those of low-rank reward denoising or a multi-head critic architecture? Do you have parameter-matched comparisons or controlled experiments showing recovery of known latent reward factors?

Response: Yes, we have added ablation study to substantiate benefits of discovering latent objectives rather than other factors. Please refer to our response to Weakness 1.

2.How were K and the mixing-weight update rate selected without using the final evaluation results? What additional training time and hyperparameter-search budget does LOD require relative to MAPPO?

Response: (1) The optimal choice of KK is task-dependent and may require the prior knowledge of the environment (for example, one can perform a pre-screening of the rank or PCA of the histortically collected reward matrix). A rule of thrumb in all our tasks is that LOD-MAPPO(K>1K>1) often outperforms LOD-MAPPO(K=1K=1) in the hetergerous reward setting.

(2) We include a computational cost comparison on RWARE-Tiny in the right panel of Figure 7, showing that introducing LOD incurs only a small increase in training time in this setting. This low overhead is mainly due to the leverage of the shared critic backbone: increasing KK only expands the output layer, resulting in a small increase in model size for the values of KK considered.

3.How do you maintain alignment between the latent factors and their corresponding critic heads as the factorization is updated online? Have you observed factor permutations or representation drift, and how do these affect training stability?

Response: We thank the reviewer for these important questions and we also added Appendix D.3.3 in blue in the revisied paper for the interpretation of the factor learned, since this is one advantage of our LOD method.

1. Factor–critic alignment. Our online factorization inherits a common baseline established by an initial loading matrix Γ0\Gamma_0, estimated from historical counterfactual signals. Subsequent online refits are warm-started from the latest loading matrix rather than independently initializing and relabeling the factors. Consequently, factor kk is consistently associated with critic head kk. Thus, no separate permutation-matching procedure is required at every update.

2. Factor permutations and representation drift. We conducted an additional interpretation run using our submitted ChartQA implementation. To interpret each learned factor, we examine the two largest loading values within its corresponding column of Γ\Gamma, identifying the two workflow roles most strongly associated with that factor. The initial and final loading matrices are:

Initial loading matrix

Workflow roleFactor 1Factor 2Factor 3
ChartAnalyzer0.7448810.2551190.000000
DataExtractor0.1024000.6422100.255390
AnswerCalculator0.3574410.2384050.404154
AnswerChecker0.3399720.2579110.402116
AnswerRefiner0.4302290.1404130.429358

Final loading matrix

Workflow roleFactor 1Factor 2Factor 3
ChartAnalyzer0.7768100.2097410.013450
DataExtractor0.1675990.5099850.322415
AnswerCalculator0.4174310.0503960.532173
AnswerChecker0.3711270.2259560.402918
AnswerRefiner0.2268450.5240690.249086

Here, in the final factorization, Factor 1 is primarily associated with chart analysis, with a secondary calculation association (0.7768 vs. 0.4174); Factor 2 captures a combined extraction-and-refinement functionality (0.5241 vs. 0.5100); and Factor 3 is mainly calculation-oriented, with a secondary checking association (0.5322 vs. 0.4029). These interpretations are post-hoc and do not impose predefined semantic meanings during learning.

Importantly, the observed changes reflect adaptive evolution of factor meanings as different workflow roles contribute to different questions, rather than permutation-induced misalignment. Since the factor coordinates remain consistently indexed with their critic heads, we observe no evidence of permutation-induced training instability. Warm-starting encourages temporal correspondence, although it does not theoretically preclude representation drift.

Thanks again for the reviewer's consistent dedication to reviewing our work and we are happy to answer any further questions or concerns. Should this rebuttal addresses your concerns, we would be grateful for a revised score.

Reference

[1] Derrick Norman Lawley and Albert Ernest Maxwell. Factor analysis as a statistical method. Journal of the Royal Statistical Society. Series D (The Statistician), 12(3):209–229, 1962.

Reviewer_WwmNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Objective Discovery (LOD), a framework for heterogeneous multi-agent reinforcement learning that decomposes agents' reward signals into shared latent factors and learns agent-specific mixing weights. The resulting latent factors are used to train multiple centralized critics, whose value estimates are combined for personalized policy optimization. The framework is integrated into MAPPO and evaluated across several MARL benchmarks, with additional convergence analysis under linear function approximation.

Critical strengths

The paper investigates objective heterogeneity in MARL and proposes a framework that integrates low-rank reward factorization with personalized critic aggregation. The method can be incorporated into existing actor-critic algorithms and is evaluated across several multi-agent environments. The paper also provides convergence analysis under restrictive assumptions.

Critical weaknesses
  1. The experimental comparison is too narrow to support the paper's central claim.
  2. The ablation study is insufficient.
Questions for authors
  1. The ablation studies mainly examine (K) and (\tau), without sufficiently isolating the contributions of the proposed components. In particular, comparisons with multi-head critics without reward factorization and alternative low-rank decomposition methods are missing, making it difficult to attribute the improvements specifically to latent objective discovery.
  2. In the Speaker–Listener task, both agents receive identical rewards, implying that the observed reward matrix has rank at most one. However, the authors set (K=2), raising concerns about the identifiability of two distinct latent objectives from these reward signals.
  3. The experimental evaluation primarily compares LOD-MAPPO against MAPPO, without sufficient comparisons to representative heterogeneous MARL methods or alternative critic-sharing approaches. This limits the assessment of the proposed method's advantages over existing techniques.
2 comments

We would like to sincerely express our gratitude to the reviewer for their time and effort in reviewing our paper. We address each concern you raised and are happy to answer any further questions.

1.The ablation studies mainly examine KK and τ\tau, without sufficiently isolating the contributions of the proposed components. In particular, comparisons with multi-head critics without reward factorization and alternative low-rank decomposition methods are missing, making it difficult to attribute the improvements specifically to latent objective discovery.

Response: We have examined whether the performance gain is explained by multi-head critic, i.e., the increased critic model size. In particular, we compare our method with a parameter-matched MAPPO with KK multi-head critic without reward factorization. The comparison results have been updated in the revised paper. As shown in the right panel of Figure 4 in Spread (continuous) and left two panels of Figure 7 in LBF and RWARE-tiny, MAPPO with KK multi-head critic only performs similarly to standard MAPPO, substantiating that the performance gain of LOD-MAPPO is not related to introducing additional parameters in the critic's output layer. Table 4 belows suggests the final average returns when compared with MAPPO-Multihead across three environments.

Table 4: Ablation study of multi-head critic and results are averaged over five seeds

EnvironmentMAPPOMAPPO-Multihead (K=2K=2)LOD-MAPPO (K=2K=2)
Spread Continuous-78.64-81.29-71.63
RWARE-Tiny4.11404.31005.9150

2.In the Speaker–Listener task, both agents receive identical rewards, implying that the observed reward matrix has rank at most one. However, the authors set K=2K=2, raising concerns about the identifiability of two distinct latent objectives from these reward signals.

Response: Thank you for raising this point. When the agent has only access to the shared identical reward with 1Nr⊤\mathbf{1}_N r^\top, the learned factor when K=2K=2 in Speaker-Listener would degenerate to f=(Γ⊤Γ+λI)−1Γ⊤1Nr⊤:=cr⊤f=\left(\Gamma^{\top} \Gamma+\lambda I\right)^{-1} \Gamma^{\top} \mathbf{1}_N r^{\top}:=c r^{\top}, where c=(c1,c2)⊤c=(c_1, c_2)^\top, similar to an overparameterized ridge regression. This implies that the identical raw reward can still be decomposed to two parts, but they only have a constant factor difference. That's explain why LOD-MAPPO (K=2K=2) and LOD-MAPPO (K=1K=1) achieve similar performance in Speaker-Listening in Figure 3. This observation also motivates us to introduce the agent-specific counterfactual advantages as the surrogate to perform LOD in practice, with the empirical demonstration provided in Agentic RL setting in Section 4.3. We have also revised the explanation and added the results of LOD-MAPPO(K=1K=1) in Speak-Listener in Figure 3 of the revised paper.

3.The experimental evaluation primarily compares LOD-MAPPO against MAPPO, without sufficient comparisons to representative heterogeneous MARL methods or alternative critic-sharing approaches. This limits the assessment of the proposed method's advantages over existing techniques.

Response: Thank you for this great suggestion. Following your adivce, we have added the comparison with the typical MARL baselines: COMA [2] and QMIX [3]. These two baselines have been officially implemented in the widely adopted EpyMARL framework [1] across LBF, LBF-Coop, RWARE-Tiny, and RWARE-Small enviroments, ensuring a fair comparison with our method. The learning curves are reported in Figure 5 of the revised paper, which suggest that LOD-MAPPO still outperforms the new baselines. The results of final average returns are summrized in Table 2 below.

Table 2. Final performance in LBF and RWARE environments (mean ± SD).

AlgorithmLBFLBF-CoopRWARE-TinyRWARE-Small
MAPPO0.933 ± 0.0230.002 ± 0.0024.114 ± 4.9550.320 ± 0.411
LOD-MAPPO (K=2)0.871 ± 0.0650.006 ± 0.0055.915 ± 2.4970.512 ± 0.781
COMA0.044 ± 0.0250.001 ± 0.0020.006 ± 0.0120.018 ± 0.026
QMIX0.021 ± 0.0070.000 ± 0.0000.040 ± 0.0800.008 ± 0.016

Reference

[1] Georgios Papoudakis, Filippos Christianos, Lukas Sch¨afer, and Stefano V. Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (NeurIPS 2021 Dataset and Benchmark)

[2] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. AAAI 2018.

[3] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research. 2020

Thanks again for the reviewer's consistent dedication to reviewing our work and we are happy to answer any further questions or concerns. Should this rebuttal addresses your concerns, we would be grateful for a revised score.

Reviewer_HKuUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Latent Objective Discovery (LOD), which factorizes agents' rewards into shared latent signals and uses learned mixing weights to construct personalized advantages within MAPPO. It presents linear actor-critic convergence analysis and reports improvements on several multi-agent environments and ChartQA/MuSiQue strategy-selection workflows.

Critical strengths
  • Structured value sharing Latent critics and agent-specific weights provide a practical way to share value estimates while retaining individual differences. The MAPPO integration makes this mechanism accessible within an established actor-critic framework.

  • Varied empirical evaluation The experiments cover several coordination settings and include five-seed results and sensitivity analyses. The improvements provide initial evidence that structured value sharing can benefit learning.

Critical weaknesses
  • Unvalidated objective interpretation Reward reconstruction does not establish that the latent factors represent meaningful objectives such as progress and safety. The existing weight ablations do not test recovery of known preferences or deliberately mismatched weights, leaving the objective-discovery claim insufficiently supported.

  • Insufficient experimental attribution Comparisons against MAPPO lack relevant credit-assignment baselines and matched controls for critic architecture and counterfactual advantages. The agentic experiments also leave unclear whether rewards come from actual LLM workflows or a surrogate environment, and provide no separate held-out evaluation.

Questions for authors

1.Factor interpretation What evidence connects the learned factors to identifiable reward components? Please clarify whether Speaker–Listener factorizes identical raw rewards or agent-specific counterfactual advantages.

2.Weight ablations Can LOD recover contrasting preferences such as (0.8, 0.2) and (0.2, 0.8)? Please compare learned weights with uniform weights and agent-permuted weights while holding the latent critics fixed.

3.Related baselines How does LOD compare with COMA and LIIR on compatible shared-reward tasks? Please match observation access, interaction budgets, and tuning protocols.

4.Matched controls Do the gains persist with parameter-matched agent-specific critics and the same counterfactual advantages without factorization? Please quantify improvements over K = 1, including uncertainty and additional training cost.

5.Agentic evaluation Do rollouts execute complete LLM workflows or use precomputed strategy-effectiveness estimates, and which parameters are trained? Please clarify the grounding/training/test split and report held-out performance from actual workflow executions.

2 comments

We would like to sincerely express our gratitude to the reviewer for their time and effort in reviewing our paper. We address each concern you raised and are happy to answer any further questions.

1.Factor interpretation What evidence connects the learned factors to identifiable reward components? Please clarify whether Speaker–Listener factorizes identical raw rewards or agent-specific counterfactual advantages.

Response: (1) This connection is fundamnetally built on the well established linear factor model Rt=ΓFt+ϵt\boldsymbol{R}_t = \Gamma \boldsymbol{F}_t + \boldsymbol{\epsilon}_t in Eq.(3), which decomposes NN-dimensional reward signals Rt\boldsymbol{R}_t into KK-dimensional latent factor component Ft\boldsymbol{F}_t. The learned loading matrix Γ\Gamma is used in principle to explain the semantic meaning of each learned factor across all heterogenous reward signals in each environment. (2) When the agent has only access to the shared identical reward with 1Nr⊤\mathbf{1}_N r^\top, the learned factor when K=2K=2 in Speaker-Listener would degenerate to f=(Γ⊤Γ+λI)−1Γ⊤1Nr⊤:=cr⊤f=\left(\Gamma^{\top} \Gamma+\lambda I\right)^{-1} \Gamma^{\top} \mathbf{1}_N r^{\top}:=c r^{\top}, where c=(c1,c2)⊤c=(c_1, c_2)^\top. This implies that the shared raw reward can still be decomposed into two components that differ only by a constant scaling factor. This also explains why LOD-MAPPO (K=2K=2) and LOD-MAPPO (K=1K=1) achieve similar performance in Speaker-Listening in Figure 3. The degenerated decomposition in the shared reward setting also motivates us to introduce the agent-specific counterfactual advantages as the surrogate to perform LOD in practice, with the empirical demonstration provided in Agentic RL setting in Section 4.3.

2.Weight ablations Can LOD recover contrasting preferences such as (0.8, 0.2) and (0.2, 0.8)? Please compare learned weights with uniform weights and agent-permuted weights while holding the latent critics fixed.

Response: A linear factor model is invariant to a joint permutation of the factors and the corresponding columns of the loading matrix. When the loading weight changes from (0.8, 0.2) to (0.2, 0.8), the learned factors remain aligned with their associated loading weights, although their ordering is not uniquely determined.

Following your suggestion, we compare LOD-MAPPO with a variant that keeps Γ\Gamma fixed at uniform weights throughout training across Spread(continuous) in Figure 4 and agentic RL settings in Figure 8 in the revised paper. The results of final average returns are summarized below, which suggests that the uniform weight cannot explain the performance gain of LOD-MAPPO over MAPPO.

Table 1. Final performance on Spread (Continuous) and ChartQA with learned and uniform weights.

AlgorithmSpread (Continuous): Final Return (Mean ± SD)ChartQA: Final Success Rate(%) (Mean ± SD)
MAPPO−78.64 ± 4.5049.90 ± 0.85
LOD-MAPPO (learned weight)−71.63 ± 8.0152.83 ± 0.56
LOD-MAPPO (uniform weight)−79.40 ± 5.6051.80 ± 0.80

3.Related baselines How does LOD compare with COMA and LIIR on compatible shared-reward tasks? Please match observation access, interaction budgets, and tuning protocols.

Response: We have added the comparison with COMA and QMIX that are officially implemented in the widely adopted EpyMARL framework [1] across LBF, LBF-Coop, RWARE-Tiny, and RWARE-Small enviroments. We keep hyper-parameters and settings of two baselines same as in EpyMARL framework [1]. The learning curves are reported in Figure 5 of the revised paper, which suggest that LOD-MAPPO still outperforms the new baselines. The results of final average returns are summrized in Table 2 below.

Table 2. Final performance in LBF and RWARE environments (mean ± SD).

AlgorithmLBFLBF-CoopRWARE-TinyRWARE-Small
MAPPO0.933 ± 0.0230.002 ± 0.0024.114 ± 4.9550.320 ± 0.411
LOD-MAPPO (K=2)0.871 ± 0.0650.006 ± 0.0055.915 ± 2.4970.512 ± 0.781
COMA0.044 ± 0.0250.001 ± 0.0020.006 ± 0.0120.018 ± 0.026
QMIX0.021 ± 0.0070.000 ± 0.0000.040 ± 0.0800.008 ± 0.016

4.Matched controls Do the gains persist with parameter-matched agent-specific critics and the same counterfactual advantages without factorization? Please quantify improvements over K = 1, including uncertainty and additional training cost.

Response: (1) Ablation study of parameter-matched agent-specific critics. To examine whether the performance gain can be explained by the multi-head critic, i.e., the increased critic model size, we compare with a parameter-matched MAPPO with the same KK multi-head critic. As shown in the right panel of Figure 4 in Spread (continuous) and left two panels of Figure 7 in LBF and RWARE-tiny, MAPPO with the same KK multi-head critic only performs similarly to standard MAPPO, substantiating that the performance gain of LOD-MAPPO is not related to introducing additional parameters in the critic's output layer.

(2) We compare the performance of LOD-MAPPO (K>1K>1) with that of LOD-MAPPO (K=1K=1) across two environments in Table 3. We also include a computational cost comparison on RWARE-Tiny in the right panel of Figure 7, showing that introducing LOD incurs only a small increase in training time in this setting. This low overhead is mainly due to the leverage of the shared critic backbone: increasing KK only expands the output layer, resulting in a small increase in model size for the values of KK considered.

Table 3: Comparison among LOD-MAPPO with different KK and the performance improvement ratio.

EnvironmentLOD-MAPPOK=1K=1LOD-MAPPOK=2K=2LOD-MAPPOK=3K=3
LBF15-Coop0.0018670.005600(+200.00%)0.003067(+64.29%)
RWARE-Tiny3.47675.9150(+70.13%)3.4500(−0.77%)

5.Agentic evaluation Do rollouts execute complete LLM workflows or use precomputed strategy-effectiveness estimates, and which parameters are trained? Please clarify the grounding/training/test split and report held-out performance from actual workflow executions.

Response: We use a frozen pretrained Qwen2.5 model to estimate strategy effectiveness, which is reasonable for the structured ChartQA task and enables efficient rollouts without repeated LLM inference. Importantly, our goal is to test our LOD algorithm in an agentic RL environment. The agent policies, latent critics, and LOD mixing weights are ALL trained online. For data separation, 8 questions per question type are sampled for LLM grounding from a pool of 200 ChartQA tasks while RL training samples from the full pool. The reported success rates measure simulated workflow performance beyond the grounding examples rather than independent held-out full-LLM execution accuracy. We would be happy to shortly clarify this protocol in the revision.

Thanks again for the reviewer's consistent dedication to reviewing our work and we are happy to answer any further questions or concerns. Should this rebuttal addresses your concerns, we would be grateful for a revised score.

Reference

[1] Georgios Papoudakis, Filippos Christianos, Lukas Sch¨afer, and Stefano V. Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (NeurIPS 2021 Dataset and Benchmark)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.