- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes MedSpaceAgent, a training-free agent for spatial reasoning in medical VQA. A frozen LLM converts the question into an executable spatial program and calls frozen perception tools; the outputs are stored in a spatial evidence graph, a deterministic executor computes geometric relations, and a verifier decides whether to answer, replan or abstain.
The validation cannot be assessed. The caption of Table 1 states: "All numerical results are hypothetical and must be replaced with actual experimental results." The text of Sections 4.2–4.3 and Appendix D describes outcomes as "expected" or "projected" (e.g., "MedSpaceAgent is expected to achieve the best overall performance"; Tables 8 and 9 are titled "Projected ..."). Tables 2–7 report numbers in the same style without such a label. As far as I can tell, the submission contains no measured results, and the abstract's claim of evaluation on compositional, counterfactual and out-of-distribution settings is not supported by any experiment in the PDF.
Critical strengths
The design principle of separating linguistic planning from deterministic geometric computation, with an explicit option to abstain, is reasonable. However, the submission provides no measured evidence that the system works, so there is no strength that can support acceptance.
Critical weaknesses
-
The paper contains no actual experimental results. The caption of Table 1 states that "All numerical results are hypothetical and must be replaced with actual experimental results." Sections 4.2–4.3 and Appendix D describe results as "expected" or "projected", and Tables 8 and 9 are titled "Projected". Tables 2–7 report specific numbers in the same style without any such label. None of the claims about accuracy, evidence consistency, robustness or efficiency is therefore supported, and the abstract's statement that the method is evaluated on compositional, counterfactual and out-of-distribution settings has no corresponding experiment.
-
The system is not specified at a level that could be implemented or checked. The perception tools are never named. The thresholds and weights in Eqs. 5, 11–15, 18, 20 and 21 are not given, nor is the procedure for choosing them without target data. The "manually verified spatial subsets" are not described (size, construction, annotators), the appendix promised for complete spatial results is absent, and no code is provided. It is also unclear how a geometry-based pipeline applies to PathVQA and PMC-VQA, where most questions are not spatial.
Questions for authors
Clarification requests:
- Has any number in Tables 1–9 been measured? If so, which ones, and under what setup (tools, backbone, test sets, number of runs)?
- Which perception tools, thresholds and confidence weights were used, and how were they selected?
- How were the spatial subsets constructed, and how many questions does each contain?
I have no suggestions for future work at this stage, since the current version cannot be evaluated empirically.