- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the distinction between predictive information in visual representations, generated answers, and the ability to acquire and use quantitative neuroimaging evidence. A controlled 279-subject experiment finds that ventricular-volume categories are linearly decodable from frozen VLM representations despite chance-level zero-shot generation; language-side fine-tuning partly closes this gap. The authors then compare nine models with and without a shared quantitative measurement interface and introduce EVA-Reason, a Qwen2.5-VL-3B model trained on teacher-generated, judge-filtered tool-use trajectories augmented with student-state and targeted corrective supervision. At inference, the student autonomously requests measurements and decides when to answer, without frontier-model calls. On 939 held-out closed-form NeuroQA questions, it achieves 96.4% overall accuracy and 89.3% on 225 L3 questions. Validation includes matched supervised/static-evidence controls, subject-cluster statistical analyses, evidence-feedback interventions, visual-input sensitivity, and a held-out-cohort experiment. The results support compact quantitative-evidence acquisition, although the L3 advantage over static relevant evidence is not significant and several frontier-model gaps disappear under alternative scoring or few-shot prompting.
Critical strengths
-
Useful diagnostic controls beyond aggregate accuracy. The paper includes Direct-SFT and static-evidence comparisons, image-removal tests, counter-majority analysis, and evidence-feedback interventions. Replacing returned measurements with same-cohort donor values reduces L3 accuracy from 89.3% to 69.8%, and withholding intermediate values reduces it to 76.9% (Appendix D.6). These findings support dependence on quantitative evidence and feedback, while the authors appropriately acknowledge the limits of the interventions. They constitute the strongest empirical aspect of the work.
-
A compact locally deployed model achieves promising task-specific performance. EVA-Reason reaches 96.4% overall and 89.3% L3 accuracy without frontier-model calls at inference. Subject-level separation, paired cluster-aware analyses, and the ABCD cohort-holdout experiment strengthen the evaluation. The paper also transparently reports that several frontier-model gaps disappear under alternative scoring or few-shot prompting. These are meaningful positive aspects, although they do not fully resolve the concerns about the contribution's scope.
Critical weaknesses
-
The central adaptive-acquisition contribution is not sufficiently distinguished from simpler alternatives or validated across student backbones. Static Relevant-Evidence achieves 88.9% L3 accuracy versus 89.3% for EVA-Reason, with no significant paired difference, while using a similar number of scalar values (3.4 versus 3.5; Appendix D.5). The feedback-withholding experiment supports sensitivity to feedback, but also introduces an input-distribution change that causes repeated requests and unproductive stops; it is not a matched trained nonadaptive baseline. Moreover, the trained acquisition model is evaluated only with Qwen2.5-VL-3B. The paper does not sufficiently justify this backbone choice or test whether the proposed training recipe remains beneficial with a newer compact backbone, such as a Qwen3-VL variant under a comparable resource budget. A newer model is not a prerequisite for a valid contribution, but this missing comparison leaves open whether the approach addresses a general acquisition problem or compensates primarily for limitations of the selected student. The introduction should make the distinction between repairing visual readout and learning external measurement acquisition more explicit.
-
The scope and inspectability of the evidence are insufficient for the broader recommendation. The L3 evaluation contains only twelve recurring question forms with fixed option sets; a TRAIN-template majority baseline already reaches 74.7%, and removing rendered MRI panels leaves EVA-Reason's L3 accuracy unchanged. Cohort holdout and counter-majority analyses help, but do not establish generalization to unseen question forms or measurement combinations. Frontier comparisons are also sensitive to answer formatting and demonstrations, as the authors acknowledge. These limitations make the claim primarily one of task-specific quantitative-evidence use. Reproducibility remains a further concern: I could not find a source-code repository link or an explicit implementation-release commitment in the provided manuscript. The reproducibility statement says evaluation trajectories are released as supplementary material, which is useful but does not establish availability of the training, routing, audit, and scoring implementations; I have not inspected supplementary files. Given the custom filtering and corrective-supervision pipeline, access to these components would materially improve independent verification. Dense statistical prose and small, crowded figure annotations further obscure the distinction between the robust findings and their limitations.
Questions for authors
-
Backbone choice and robustness: Why was Qwen2.5-VL-3B selected as the sole trained student? Is there evidence that the acquisition-training benefit persists on a newer compact backbone, such as an appropriately sized Qwen3-VL model, relative to its own Direct-SFT and static-evidence controls? A comparable-resource experiment would clarify the generality of the contribution; simply replacing the model without matched controls would not resolve this concern.
-
Added value of sequential adaptation: Can a trained one-shot measurement selector, using the same training items and a matched measurement budget, approach EVA-Reason without intermediate feedback? How much do student-state, paired acquisition-to-answer, and None-repair supervision contribute beyond judge-selected trajectories? These comparisons would help isolate the method's specific contribution.
-
Generalization beyond benchmark structure: Do you have results on held-out question forms or previously unseen combinations of measurable concepts? The existing ABCD holdout tests cohort transfer, but does not establish transfer beyond the recurring question and option structure. A focused evaluation could materially affect the assessment of significance.
-
Implementation availability: Is source code included in the supplementary material, or is an anonymous repository available? Please identify the available training scripts, trajectory-construction code, deterministic router, evidence audit, scoring implementation, and model adapters. The manuscript explicitly mentions released evaluation trajectories, but the availability of executable implementation artifacts is unclear.
-
Comparative robustness: What are the paired L3 results when the three-shot frontier runs are also evaluated with the option-prefix rule? Since each change separately narrows headline gaps, their combination would clarify whether the remaining advantage reflects evidence reasoning or interface and answer-format compliance.