- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MarsSite-Bench and SOLVE, a multi-agent framework for adaptive evidence acquisition and constraint-aware decision making. SOLVE outperforms existing baselines on 120 hidden-test tasks and is demonstrated through a real-Mars case study.
Critical strengths
- The problem formulation meaningfully integrates evidence budgets, missing information, and hard constraints.
- The paper provides extensive baseline comparisons, matched-access controls, and ablation studies.
Critical weaknesses
-
Questionable evaluation validity. Offline tests of the submitted evaluator show that T5 outputs citing unacquired evidence and T4 outputs listing all candidates in an incorrect order can both receive scores of 0.99. Such permissive scoring raises concerns about whether the reported gains reflect genuine improvements in decision-making ability.
-
Insufficient validation of the proposed mechanisms. In the submitted implementation, Planner and Critic outputs do not appear to affect subsequent decisions. Moreover, removing verification does not significantly reduce CGDQ, leaving the contributions of the proposed mechanisms unclear.
Questions for authors
- Can the authors explain these scoring behaviors and demonstrate that SOLVE retains its advantage under stricter evidence-validity and ranking metrics?
- Which exact implementation produced the reported results? Can the authors demonstrate how Planner/Critic feedback changes subsequent decisions and how invalid decisions are blocked before submission?
- Figures 2–3 appear rasterized, with some text difficult to read. Could the authors provide vector-based or higher-resolution versions?