acceptodds
Withdrawn from ICLR 2027

SOLVE: Adaptive Evidence Acquisition for Verifiable Multi-Agent Decision Making

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents increasingly integrate tools and multi-step reasoning, but consequential decisions may often require selective evidence acquisition, non-compensatory constraints, and auditable termination. We formulate this setting as evidence-constrained decision making and introduce MarsSite-Bench, a 600-task benchmark spanning six task types with explicit evidence budgets, structured outputs, and method-independent constraint-gated evaluation. We further propose Sequential Orchestration with Logical Verification and Evidence Acquisition (SOLVE), which coordinates adaptive evidence acquisition and role-specialized reasoning through a shared decision state. On 120 hidden-test tasks, SOLVE achieves a Primary CGDQ of 0.6488, compared to 0.1795 for ReAct. Matching ReAct’s evidence access raises its score to 0.3976 but leaves a substantial gap. A real-Mars case study further demonstrates capabilities in regional ranking and footprint-level abstention under heterogeneous geospatial evidence. These results support adaptive acquisition and constraint-aware orchestration for verifiable LLM-agent decisions.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
large language model agents, multi-agent systems, adaptive evidence acquisition, constraint-aware reasoning, verifiable decision making, agent benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hNf7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MarsSite-Bench and SOLVE, a multi-agent framework for adaptive evidence acquisition and constraint-aware decision making. SOLVE outperforms existing baselines on 120 hidden-test tasks and is demonstrated through a real-Mars case study.

Critical strengths
  1. The problem formulation meaningfully integrates evidence budgets, missing information, and hard constraints.
  2. The paper provides extensive baseline comparisons, matched-access controls, and ablation studies.
Critical weaknesses
  1. Questionable evaluation validity. Offline tests of the submitted evaluator show that T5 outputs citing unacquired evidence and T4 outputs listing all candidates in an incorrect order can both receive scores of 0.99. Such permissive scoring raises concerns about whether the reported gains reflect genuine improvements in decision-making ability.

  2. Insufficient validation of the proposed mechanisms. In the submitted implementation, Planner and Critic outputs do not appear to affect subsequent decisions. Moreover, removing verification does not significantly reduce CGDQ, leaving the contributions of the proposed mechanisms unclear.

Questions for authors
  1. Can the authors explain these scoring behaviors and demonstrate that SOLVE retains its advantage under stricter evidence-validity and ranking metrics?
  2. Which exact implementation produced the reported results? Can the authors demonstrate how Planner/Critic feedback changes subsequent decisions and how invalid decisions are blocked before submission?
  3. Figures 2–3 appear rasterized, with some text difficult to read. Could the authors provide vector-based or higher-resolution versions?
Reviewer_smKmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SOLVE, a multi-agent framework for decision-making under evidence budgets and hard constraints, together with MarsSite-Bench and a real-Mars case study. The framework coordinates planning, evidence acquisition, candidate analysis, criticism, and deterministic verification through a shared decision state.

Critical strengths
  1. The problem setup and formulation are interesting. In particular, explicitly separating evidence availability, hard-constraint satisfaction, and candidate preference provides a useful perspective on reliable agent decision-making.
  2. The paper is also generally readable, with a clear high-level presentation of the framework.
Critical weaknesses
  1. The evaluation does not sufficiently establish the framework’s added value. The experiments use a narrow synthetic benchmark, a single backbone, and mostly older agent designs such as ReAct and Self-Refine. A strong contemporary general-purpose agent, evaluated with the same model, tools, and comparable budget, is an important missing comparison. Without it, it remains unclear whether SOLVE’s elaborate architecture offers a meaningful advantage over a simpler, well-equipped agent.

  2. The ablations are insufficient for such a complex design. SOLVE combines several specialized roles, shared state, adaptive acquisition, and verification, but their contributions are not adequately isolated. The main ablation removes criticism and verifier feedback together without significantly reducing performance. More targeted ablations are needed to identify which components matter and whether the added complexity is justified.

The main text should also describe the dataset more concretely, including task examples, construction procedures, and scoring criteria, so readers can assess its difficulty and relevance.

Questions for authors

Questions

  1. How does the method perform in public benchmarks? It will be helpful if the user could revise the existing open-domain decision-making benchmarks into evidence-constraint form.
  2. How does the method work with larger models? Is the improvement still salient?
  3. How does the method compare to more recent agents, like the coding agents?
  4. How does each component contribute to the performance improvements? A more detailed ablation study is necessary for such a complicated workflow.

Suggestions

  1. Some key details, like the construction of the benchmark, could be moved to the main text; they are necessary for readers to understand this paper.
Reviewer_1it7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses evidence-constrained decision making for LLM agents, where the agent needs to choose which evidence to acquire under a finite budget, satisfy hard constraints, and terminate with an auditable decision of SELECT or ABSTAIN. The paper introduces MarsSite-Bench, and proposes SOLVE, which coordinates multiple components (Planner, Evidence Scout, Candidate Analyst, and Critic) over a shared decision state and terminates with a deterministic Constraint Verifier. On this dataset, SOLVE reaches 0.6488 Primary CGDQ versus at most 0.1795 for all baselines, achieving a large margin.

Critical strengths
  1. The method is clearly described. The clear presentation and delivery of equations and figures make the method very easy to follow (eq. 5 - 7, fig 1 - 3).
  2. The benchmark is valuable. It isolates an explicit acquisition budget, an UNRESOLVED constraint state, and evidence references as part of the scoring contract (sec. 3.3).
Critical weaknesses
  1. The source of the headline gap may be unresolved. All baselines score exactly 0 on T2 and T5 (table A1), and the paper does not explain whether this reflects missing evidence access, inability to emit the required status field or evidence references, or systematically wrong predictions. A closer examination of the gap would provide a clearer picture of why SOLVE is effective.
  2. Component-wise ablations are missing. The only ablation removes Critic revision and verifier feedback together (sec. 5.4). The contributions of the Planner, Scout, Critic, and shared state are not isolated.
  3. Evaluation is narrow. All results use GPT-5.4-mini on only MarsSite-Bench. While the conclusion acknowledges this limitation, for a the general-framework claim, it feels overstated without a second backbone or a second domain.
Questions for authors
  1. Why do all baselines score 0 on T2 and T5?
  2. What is the role of the train and dev splits of MarsSite-Bench? Nothing is trained, and Sec. 5.1 only mentions Development for configuration selection. Were Train tasks used for prompts, few-shot examples, or normalizer construction?
  3. What are CGDQ when the Planner, Scout, or Critic is removed individually, and on at least one other backbone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.