acceptodds
Under review as a conference paper at ICLR 2027

ICMAPE: In-Context Multiagent Pure Exploration

Abstract

In some multi-agent systems, the quantity to be optimized is not an externally specified reward but the information acquired about unknown properties of the environment as done in active sequential hypothesis testing (ASHT) problems. However, the ASHT literature tends to focus on finite single-agent problems with well-specified models, while there is currently a gap for practical multi-agent methods that can perform active sequential testing. We fill this gap with ICMAPE, a Bayesian learning-based framework for decentralized multi-agent pure-exploration driven by inference objectives. ICMAPE converts the fixed-confidence identification objective into a reward derived from inference confidence, so that standard reinforcement learning machinery can be applied to decentralized pure exploration. It jointly learns a centralized neural inference network that estimates a posterior distribution over hypotheses from global trajectory data, and decentralized policies that select actions from local observation histories and learn when to stop collecting data once the target confidence is reached. On two synthetic benchmarks and a Maryland nitrate concentration monitoring task based on real-world data, ICMAPE-TD3 achieves target accuracy with fewer exploration steps.

Primary area:
reinforcement learning
Keywords:
MARL, reinforcement learning, decentralized multi-agent learning, fixed-confidence identification, cooperative exploration, centralized training decentralized execution, multi-agent reinforcement learning, hypothesis testing, pure exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rdKEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies decentralized multi-agent pure exploration: agents collect observations using local histories, while a centralized inference network identifies a latent hypothesis from their combined trajectories. The method converts identification confidence into a terminal reward, introduces a sampling cost and stopping actions, and trains decentralized actors with centralized MATD3 critics. Experiments cover capacity identification, source localization, and GP-derived nitrate-monitoring environments.

Critical strengths
  1. The problem is well motivated and separates information acquisition from environmental reward maximization.
  2. Jointly learning inference, sampling, and stopping is a useful practical design. In A3P5, the full method reports 0.91 accuracy and 13.5 stopping steps, whereas removing sampling cost increases the stopping time to 50 at similar accuracy (Table 1).
  3. The paper includes three training seeds, extensive evaluation instances, coordination diagnostics, and an explicit calibration analysis. The held-out-year evaluation is also useful within the GP-derived setting.
Critical weaknesses

My main concern lies with the experimental evaluation, which, in my view, lacks sufficient rigor to fully support the paper's claims. Moreover, the reported performance gains are relatively modest and do not seem particularly compelling. Specifically,

  1. The variance of the stopping step results in Fig. 1(a) seems too high compared to baselines, which limits the statistical significance of the results.
  2. The baselines choice should more consistent across the three benchmarks. If using the same baselines across all benchmarks is not feasible, a clear explanation for the differences and justify the selection of baselines for each benchmark may be required.
  3. In Fig. 1, the proposed ICMAPE uses VDN and TD3 as its backbone algorithms, whereas the baseline methods use DQN. This difference in backbone algorithms makes it difficult to determine whether the observed performance improvements come from the proposed modules or from the choice of RL algorithms themselves.
  4. The performance gain of ICMAPE over baselines in Fig. 3 is not clear, limiting its practical contribution.

I would encourage the authors to strengthen the experimental evaluation and provide more convincing evidence of the method's effectiveness and its advantages over existing approaches.

Questions for authors
  1. Can the authors explain why I-VDN and I-INDEP show much higher variance? Can increasing the number of seeds resolve this? If so, experiments with more seeds may be required.
  2. Why choosing different baselines for bandit capacity identification, signal source localization and Maryland nitrate monitoring, while the proposed ICMAPE uses the same backbones across the three benchmarks?
Reviewer_sKHnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ICMAPE by extending the ICPE mechanism to multi-agent active exploration. Built upon the CTDE framework, the method employs decentralized actors for exploration and a centralized inference network to provide intrinsic rewards and guide agents in deciding when to terminate exploration. The authors evaluate the proposed method in bandit environments and other tasks, achieving good results.

Critical strengths

Major Strengths

The method could provide useful support for multi-agent applications, such as future multi-UAV target search scenarios.

Minor Strengths

The paper is well organized, making it easy for readers to understand the proposed method.

Critical weaknesses
  1. The proposed method is built upon the existing ICPE framework, with essentially all components remaining the same. Simply extending the method to the multi-agent setting is insufficient in terms of novelty. Furthermore, most of the theoretical results are derived from ICPE, leaving the paper with very limited theoretical contributions.

  2. The experimental comparisons are insufficient. The paper mainly compares relatively simple plug-and-play methods, without including more advanced multi-agent reinforcement learning methods such as MAPPO, HAPPO, HASAC, QMIX, and QTRAN, or closely related methods such as decentralized multi-agent active search. In addition, using only three random seeds is insufficient to establish the reliability of the experimental results.

Questions for authors

Can the authors further clarify the novelty of the proposed method compared with ICPE, particularly in the context of CTDE, and provide corresponding theoretical analysis or derivations?

Can the authors provide additional experimental comparisons with more multi-agent reinforcement learning methods, such as MAPPO, HAPPO, HASAC, QMIX, and QTRAN, as well as related methods such as decentralized multi-agent active search [1]?

[1] Ghods, Ramina, Arundhati Banerjee, and Jeff Schneider. "Decentralized Multi-Agent Active Search for Sparse Signals." Uncertainty in Artificial Intelligence. PMLR, 2021.

Can the authors increase the number of random seeds to at least five to further demonstrate the reliability of the experimental results?

Reviewer_b5AqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Prior literature on Active Sequential Hypothesis Testing focuses on a single-agent problem. In this work, the authors introduced ICMAPE, a Bayesian learning-based framework for decentralized multi-agent pure exploration driven by inference objectives. The core idea is to convert the fixed-confidence identification objective into a reward derived from inference confidence, so that standard reinforcement learning machinery can be applied to decentralized pure exploration. The proposed method achieves target identification accuracy with significantly fewer exploration steps and lower sample complexity compared to standard multi-agent RL baselines, demonstrating strong transferability and robustness.

Critical strengths

The proposed method is intuitive and reasonable for the Active Sequential Hypothesis Testing task.

Converting the fixed-confidence identification objective into a reward derived from inference confidence allows RL machinery to apply to decentralized pure exploration.

Experiments on two synthetic benchmarks and a Maryland nitrate concentration monitoring task based on real-world data show the proposed method achieves target accuracy with fewer exploration steps.

Critical weaknesses

The proposed framework lacks novelty, while the problem formulation and the application of MARL to tackle the Active Sequential Hypothesis Testing task are interesting.

The paper lacks analysis of the method's generalization and robustness.

Questions for authors

Could you explain how the posterior kernel is independent of decentralized policies?

In practice, how is the common stopping time handled?

What are the benefits of inference-driven rewards compared to environment rewards?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.