acceptodds
Under review as a conference paper at ICLR 2027

EVA-Reason: Learning Adaptive Evidence Acquisition for Neuroimaging Reasoning

Abstract

Strong multimodal models often struggle to answer difficult neuroimaging questions directly from MRI. We show that these failures are not simply explained by a lack of predictive signal. On the same controlled task, an end-to-end CNN reaches 0.962 balanced accuracy and a Qwen2.5-VL post-projector linear probe reaches 0.897, while direct generation is at chance (0.333). Language-side fine-tuning with frozen vision encoder/projector raises direct generation accuracy to 0.743, narrowing but not closing the representation-to-generation gap. We examine whether explicit quantitative evidence can improve this reasoning bottleneck. We evaluate nine frontier and open-weight multimodal models on the same questions with and without access to quantitative measurement tools. On the hardest questions (L3, the most challenging reasoning level), Tool-OFF accuracy ranges from 36.4% to 56.4%. Four frontier models improve significantly with tools, reaching 75.6% to 80.0% L3 accuracy. Other models improve little or degrade, showing that tool access alone is insufficient. Motivated by these findings, we introduce EVA-Reason, a framework for learning adaptive evidence acquisition in neuroimaging. We construct multi-step evidence-seeking trajectories using Claude Sonnet 4.6 and Gemini 2.5 Pro, and use them to fine-tune a 3B VLM; frontier comparators use the same interface zero-shot. We evaluate EVA-Reason on NeuroQA questions organized into three reasoning levels. On the held-out test set, under normalized exact-match scoring, EVA-Reason achieves 96.4% overall accuracy and 89.3% L3 accuracy. Matched Qwen-3B controls reach 64.5/82.2% overall/L3 without quantitative evidence and 94.6/88.9% when a prespecified relevant-evidence set is supplied upfront; EVA-Reason selects evidence autonomously at inference, without a prespecified per-item evidence set. Under zero-shot exact match, EVA-Reason reaches 89.3% L3 versus 80.0/79.6/79.1% for Sonnet 5/Gemini 2.5 Pro/GPT-5.5; several gaps narrow to non-significance under few-shot prompting or option-prefix scoring. EVA-Reason requires no frontier-model calls at inference and runs locally on a single GPU. These results show that a small open-weight model can learn to actively acquire and use quantitative neuroimaging evidence across multiple decision steps.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, neuroimaging, adaptive evidence acquisition, tool-augmented vision-language models, active feature acquisition, medical AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rC6ZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the distinction between predictive information in visual representations, generated answers, and the ability to acquire and use quantitative neuroimaging evidence. A controlled 279-subject experiment finds that ventricular-volume categories are linearly decodable from frozen VLM representations despite chance-level zero-shot generation; language-side fine-tuning partly closes this gap. The authors then compare nine models with and without a shared quantitative measurement interface and introduce EVA-Reason, a Qwen2.5-VL-3B model trained on teacher-generated, judge-filtered tool-use trajectories augmented with student-state and targeted corrective supervision. At inference, the student autonomously requests measurements and decides when to answer, without frontier-model calls. On 939 held-out closed-form NeuroQA questions, it achieves 96.4% overall accuracy and 89.3% on 225 L3 questions. Validation includes matched supervised/static-evidence controls, subject-cluster statistical analyses, evidence-feedback interventions, visual-input sensitivity, and a held-out-cohort experiment. The results support compact quantitative-evidence acquisition, although the L3 advantage over static relevant evidence is not significant and several frontier-model gaps disappear under alternative scoring or few-shot prompting.

Critical strengths
  1. Useful diagnostic controls beyond aggregate accuracy. The paper includes Direct-SFT and static-evidence comparisons, image-removal tests, counter-majority analysis, and evidence-feedback interventions. Replacing returned measurements with same-cohort donor values reduces L3 accuracy from 89.3% to 69.8%, and withholding intermediate values reduces it to 76.9% (Appendix D.6). These findings support dependence on quantitative evidence and feedback, while the authors appropriately acknowledge the limits of the interventions. They constitute the strongest empirical aspect of the work.

  2. A compact locally deployed model achieves promising task-specific performance. EVA-Reason reaches 96.4% overall and 89.3% L3 accuracy without frontier-model calls at inference. Subject-level separation, paired cluster-aware analyses, and the ABCD cohort-holdout experiment strengthen the evaluation. The paper also transparently reports that several frontier-model gaps disappear under alternative scoring or few-shot prompting. These are meaningful positive aspects, although they do not fully resolve the concerns about the contribution's scope.

Critical weaknesses
  1. The central adaptive-acquisition contribution is not sufficiently distinguished from simpler alternatives or validated across student backbones. Static Relevant-Evidence achieves 88.9% L3 accuracy versus 89.3% for EVA-Reason, with no significant paired difference, while using a similar number of scalar values (3.4 versus 3.5; Appendix D.5). The feedback-withholding experiment supports sensitivity to feedback, but also introduces an input-distribution change that causes repeated requests and unproductive stops; it is not a matched trained nonadaptive baseline. Moreover, the trained acquisition model is evaluated only with Qwen2.5-VL-3B. The paper does not sufficiently justify this backbone choice or test whether the proposed training recipe remains beneficial with a newer compact backbone, such as a Qwen3-VL variant under a comparable resource budget. A newer model is not a prerequisite for a valid contribution, but this missing comparison leaves open whether the approach addresses a general acquisition problem or compensates primarily for limitations of the selected student. The introduction should make the distinction between repairing visual readout and learning external measurement acquisition more explicit.

  2. The scope and inspectability of the evidence are insufficient for the broader recommendation. The L3 evaluation contains only twelve recurring question forms with fixed option sets; a TRAIN-template majority baseline already reaches 74.7%, and removing rendered MRI panels leaves EVA-Reason's L3 accuracy unchanged. Cohort holdout and counter-majority analyses help, but do not establish generalization to unseen question forms or measurement combinations. Frontier comparisons are also sensitive to answer formatting and demonstrations, as the authors acknowledge. These limitations make the claim primarily one of task-specific quantitative-evidence use. Reproducibility remains a further concern: I could not find a source-code repository link or an explicit implementation-release commitment in the provided manuscript. The reproducibility statement says evaluation trajectories are released as supplementary material, which is useful but does not establish availability of the training, routing, audit, and scoring implementations; I have not inspected supplementary files. Given the custom filtering and corrective-supervision pipeline, access to these components would materially improve independent verification. Dense statistical prose and small, crowded figure annotations further obscure the distinction between the robust findings and their limitations.

Questions for authors
  1. Backbone choice and robustness: Why was Qwen2.5-VL-3B selected as the sole trained student? Is there evidence that the acquisition-training benefit persists on a newer compact backbone, such as an appropriately sized Qwen3-VL model, relative to its own Direct-SFT and static-evidence controls? A comparable-resource experiment would clarify the generality of the contribution; simply replacing the model without matched controls would not resolve this concern.

  2. Added value of sequential adaptation: Can a trained one-shot measurement selector, using the same training items and a matched measurement budget, approach EVA-Reason without intermediate feedback? How much do student-state, paired acquisition-to-answer, and None-repair supervision contribute beyond judge-selected trajectories? These comparisons would help isolate the method's specific contribution.

  3. Generalization beyond benchmark structure: Do you have results on held-out question forms or previously unseen combinations of measurable concepts? The existing ABCD holdout tests cohort transfer, but does not establish transfer beyond the recurring question and option structure. A focused evaluation could materially affect the assessment of significance.

  4. Implementation availability: Is source code included in the supplementary material, or is an anonymous repository available? Please identify the available training scripts, trajectory-construction code, deterministic router, evidence audit, scoring implementation, and model adapters. The manuscript explicitly mentions released evaluation trajectories, but the availability of executable implementation artifacts is unclear.

  5. Comparative robustness: What are the paired L3 results when the three-shot frontier runs are also evaluated with the option-prefix rule? Since each change separately narrows headline gaps, their combination would clarify whether the remaining advantage reflects evidence reasoning or interface and answer-format compliance.

Reviewer_Y596OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes EVA-Reason, a framework for training a compact vision-language model to adaptively acquire quantitative neuroimaging evidence through sequential tool interactions. The authors first identify a representation-to-generation gap in existing VLMs, showing that clinically relevant MRI information can remain linearly decodable even when direct generation performs poorly.

EVA-Reason uses frontier models to generate and select evidence-seeking trajectories, which are distilled into a 3B VLM through supervised fine-tuning. The resulting model autonomously requests quantitative measurements and integrates them into its decisions without frontier-model calls at inference.

On NeuroQA, EVA-Reason achieves 96.4% overall accuracy and 89.3% accuracy on challenging L3 questions, outperforming several frontier models under the primary exact-match evaluation protocol.

Critical strengths

Well-motivated problem formulation. The representation-probing experiments (Figure 2 and Appendix A) provide compelling evidence that information accessibility and successful answer generation are distinct capabilities. This offers a meaningful motivation for studying adaptive quantitative evidence acquisition.

Strong experimental design and analysis. The paper evaluates nine comparison models under shared tool interfaces and reports matched supervised controls, subject-aware statistical tests, evidence-feedback interventions, and held-out cohort generalization. The results support the effectiveness of learning a sequential evidence-acquisition policy.

Critical weaknesses

Limited novelty in the underlying learning methodology. EVA-Reason primarily combines established techniques, including teacher-generated trajectory distillation, supervised fine-tuning, and tool-augmented sequential reasoning. Its main contribution is the systematic application and evaluation of these techniques in neuroimaging, rather than a fundamentally new learning algorithm.

The advantage of adaptive acquisition over strong static evidence selection is modest. In Table 1 and Appendix D.5, EVA-Reason achieves 89.3% L3 accuracy compared with 88.9% for Static Relevant-Evidence, a statistically insignificant difference (p=1.00). Although the overall improvement is significant, the results do not establish a substantial accuracy advantage from adaptive evidence acquisition on the most challenging questions. Furthermore, the reported gaps over frontier models become smaller or statistically insignificant under alternative scoring and few-shot prompting conditions.

Questions for authors

Can the authors clarify which improvements are specifically attributable to sequential evidence selection rather than trajectory distillation and stronger supervised adaptation? In particular, how should the nonsignificant L3 difference against Static Relevant-Evidence be interpreted?

How would the authors characterize EVA-Reason's advantages over frontier models under option-prefix scoring and few-shot tool-use prompting, where several reported differences become statistically insignificant?

To what extent does the learned acquisition policy generalize to new question structures or measurement types beyond the twelve recurring L3 question forms and the fixed quantitative tool interface?

Reviewer_pk7MOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EVA-Reason trains a 3B VLM to request and integrate quantitative neuroimaging evidence through multi-turn tool use, using teacher-generated trajectories and additional action supervision. On the frozen NeuroQA test split, it reports 96.4% overall and 89.3% L3 normalized exact-match accuracy on closed-form questions. Controlled representation probes, matched supervised/static-evidence controls, and evidence-feedback interventions examine why quantitative evidence helps and what the learned acquisition policy contributes.

Critical strengths
  1. The controlled representation study makes a useful distinction between accessible information and generated answers. On the same 279 subjects and evaluation folds, the Qwen post-projector probe reaches 0.897 balanced accuracy, frozen direct generation is at chance, and language-side fine-tuning raises generation to 0.743 (Appendix A, Table 2). This is a stronger motivation than simply observing poor zero-shot VQA performance, while remaining an analysis of a specific controlled task.

  2. The evidence-use controls are substantive. Replacing measurement values while fixing the call sequence lowers L3 accuracy from 89.3% to 69.8%; withholding intermediate values lowers it to 76.9% despite exposing more values overall (Appendix D.6). These interventions support actual dependence on returned measurements and feedback-sensitive selection/stopping, rather than only a memorized tool-call pattern. Subject-cluster inference also accounts for multiple questions per participant.

The matched Static Relevant-Evidence control and the counter-majority analysis make the interpretation more credible. EVA-Reason reaches 68.4% versus Direct-SFT's 38.6% on the 57 L3 items opposing TRAIN-template majority answers (D.5), and cohort-holdout training retains 90.3% L3 on ABCD (D.10). Local inference without teacher calls is practical; Table 20 reports 12.1 seconds of observed model latency per L3 item on one L40S, with appropriate cautions about cross-provider timing.

Critical weaknesses
  1. Generalization is still demonstrated within a restricted question/evidence structure. The 225 L3 items use twelve recurring forms with fixed option sets, and 344 open-ended questions are excluded from primary accuracy (Tables 10-11). The counter-majority and ABCD-holdout results address important alternative explanations, but do not establish transfer to unseen question forms or new option compositions. Removing rendered MRI panels leaves EVA-Reason's L3 accuracy at 89.3% (Table 17). I therefore interpret the main achievement as acquiring and using MRI-derived measurements, rather than establishing broad direct visual reasoning.

  2. The advantage over simpler evidence selection remains incompletely isolated. Static Relevant-Evidence reaches 88.9% L3 versus 89.3% for EVA-Reason, with a paired CI spanning zero, and uses 3.4 versus 3.5 scalar values per L3 item (D.5). EVA-Reason does improve overall accuracy by 1.8 points and avoids a prespecified relevance mapping. Full-Evidence reaches 94.7% L3 with 138.6 values, demonstrating a compactness tradeoff rather than an accuracy advantage. The open-loop intervention is informative, but its repeated calls and unproductive stops make it different from a selector trained to request one evidence batch, as D.6 explicitly recognizes. Thus feedback use is supported, while an advantage over a trained nonsequential selector remains unresolved.

The comparative advantage over frontier models also depends on protocol. Option-prefix scoring brings Opus 5 to 87.1% L3, and three tool-use demonstrations bring Sonnet 5/GPT-5.5 to 86.7/86.2%, leaving nonsignificant gaps to EVA-Reason (D.3/D.13). These are meaningful sensitivities, although nonsignificance does not establish equivalence. I would emphasize the compact learned policy over a general superiority claim.

I lean toward acceptance because the controlled analyses and feedback interventions provide useful evidence for learned acquisition and integration.

Questions for authors
  1. Beyond the counter-majority and cohort-holdout analyses, is there evidence for transfer to question forms or option compositions absent from training? Please distinguish this from subject/cohort generalization.

  2. Could a Qwen-3B selector trained to request a single evidence batch, with comparable supervision and evidence budget, clarify the benefit of sequential adaptation? This would complement the existing D.6 intervention.

  3. Could you add same-hardware inference timings for Static Relevant-Evidence and Full-Evidence alongside EVA-Reason, distinguishing model time from backend/preprocessing costs? Table 20 compares frontier providers, while the most relevant remaining efficiency comparison is with these local evidence controls.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.