acceptodds
Withdrawn from ICLR 2027

QGR: Query Grounding and Refinement for Medical Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual grounding links textual expressions to relevant image regions, while grounded reasoning requires model predictions and explanations to be supported by visual evidence. This remains challenging for medical vision-language models (VLMs), which can produce plausible explanations that are poorly aligned with disease relevant regions, leading to hallucinated reasoning and unreliable interpre- tations. Existing medical visual question answering (VQA) approaches mainly rely on supervised fine-tuning (SFT) or reinforcement learning (RL) to improve answer generation and reasoning, but provide limited direct supervision for question aware spatial grounding. As a result, spatial representations are often learned indirectly through the language-modeling objective, making it difficult to align reasoning with relevant visual evidence. We propose QGR (Query Grounding and Refinement), a multimodal grounding framework that connects language reasoning with object level visual evidence. QGR has two key components: Question Conditioned Query Refinement, which adapts object queries to the question context to capture relevant evidence, and Query Level Grounding Supervision, which directly guides their semantic assignment and localization quality. Together, they establish question aware semantic spatial correspondence at the query level. QGR jointly learns an- swer generation, reasoning, semantic region prediction, bounding-box localization, and query level grounding. Experiments show consistent improvements over the corresponding SFT baselines. On Med4VQA, QGR improves Closed accuracy by +3.20–17.64 points, Box localization by +7.64–52.15 points, and Final scores by +4.18–10.11 points across four imaging modalities. The gains also transfer out of domain, reaching +14.49 Final points on SLAKE MRI and up to +35.00 Box points on GEMeX. These results highlight the benefit of query level supervision for spatially grounded medical reasoning. Full training details and anonymized code and dataset are provided for reproducibility.

Primary area:
generative models
Keywords:
Grounded Reasoning, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rvhxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QGR combines medical visual question answering with object detection and query grounding. Its architecture uses shared representations and auxiliary localization supervision. The authors report improved answer and localization metrics on Med4VQA and additional evaluations, suggesting benefits from integrating spatial supervision, although question-specific grounding and generalization remain incompletely established.

Critical strengths
  1. Table 3 includes a detector-fusion comparison rather than only a language-model baseline. QGR improves localization while recovering answer performance degraded by simpler fusion. This supports a meaningful integration contribution, even though substantial localization capability already comes from the detector.
  2. Med4VQA provides a structured setting for jointly studying medical answers and localization. The architecture, auxiliary supervision, and reported component ablations make this a useful empirical direction. Evaluations across imaging modalities and additional datasets also expose both benefits and limitations, rather than relying on one aggregate result.
Critical weaknesses
  1. Equations 3–4 and Algorithm 1 use a single context token for cross-attention, making the attention output a shared context broadcast. Residual pathways can still support question dependence, but the manuscript insufficiently explains how question-specific targets, ranking, and final answers are connected. This ambiguity limits assessment of the central grounding claim.
  2. The evaluation does not fully establish question-specific grounding or robust transfer. Box measures top-ranked overlap at IoU ≥ 0.10, rather than mean localization quality; Table 12 shows mixed answer generalization. Generated target attributes and discrepancies between Tables 3 and 12 further require clarification before interpreting the broader claims.
Questions for authors
  1. Please trace the question-conditioned path from input through grounding queries, box ranking, and answer generation. Are ground-truth objects filtered by the question, and what score selects the reported box? A precise account would clarify whether the localization metric evaluates question-specific selection.
  2. What processing produces the single context token in Equations 3–4? How does auxiliary grounding supervision influence the answer pathway? Please distinguish direct inference connections from training effects through shared parameters; the latter can improve answers without directly exposing grounding outputs to the decoder.
  3. Please reconcile the Region results in Tables 3 and 12 and describe validation of generated question–attribute targets. Existing protocol details and annotations would help determine whether the reported gains reflect the intended task rather than differences in evaluation versions or target construction.
Reviewer_Thx2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission introduces QGR, which adds a Deformable DETR branch to a VLM backbone like Qwen3-VL-4B. Specifically, the detector object queries are refined by cross-attention to a VLM "context" vector, hq = Fv[:,0,:]. And then the refined queries are supervised by an IoU-matched assignment head and a quality head. The training objective is joint with the LM, detection, region and box losses. The authors also build Med4VQA from VGMED images covering 4 modalities. They report large Box and Final gains over backbone-matched SFT, plus zero-shot transfer to datasets like SLAKE.

Critical strengths
  1. Coupling a VLM with an explicit detector for grounded medical VQA is well motivated, although not novel anymore.
  2. The experimental design is backbone-matched (Qwen3-VL, MedGemma) and covers four modalities, with SFT and GRPO variants.
  3. The authors commit to releasing code and data, and Appendices B and C give detailed training and metric specifications.
Critical weaknesses
  1. The central design is problematic: Question conditioning is likely degenerate. (1) The keys and values come from a single vector from VLM. With one key, the softmax in Eq. 4 is identically 1. The attention output is then hq·W_V for every query, so "refinement" adds the same vector to all N_o queries and cannot re-rank or select queries by question relevance. (2) F_v[:,0,:] is the first token of a causal VLM sequence, typically a system/BOS. It cannot attend to the question tokens that come later, so hq basically carry no question information at all. (3) Refinement does not modify boxes (Sec. 3.2). For QGR, the top-1 box is the "highest-ranked detector prediction", and it is unclear whether ranking uses the quality head or the detector score. If the latter, the question has no path to the Box metric.
  2. The evaluation compares different grounding sources, and the comparison is not coherent and fair enough. (1) QGR's regions and boxes come from a supervised detector, while the baselines must generate coordinates as text. The Box gains therefore largely measure "trained detector vs. VLM coordinate decoding," not QGR. Table 3 supports this. Qwen3-VL+DETR without QGR already improves Box substantially: MRI 18→49, US 47→80, X-ray 24→52. (2) The Box score is top-1 accuracy at IoU ≥ 0.10, which is very permissive. The paper should report mean IoU, [email protected], and AP. In addition, Fig. 1 shows continuous Box scores (0.90, 0.16), which contradicts the binary definition in Eq. 20.
  3. Main experiments are not well designed: (1) The zero-shot baselines cannot emit the output format and score 0 Box: BLIP-2, LLaVA-1.5, LLaVA-Med. These comparisons are uninformative. (2) Grounded medical or grounded-reasoning baselines trained on Med4VQA are missing, e.g., Med-R1 and grounded RL variants.
  4. Missing baselines: (1) Deformable DETR trained alone; (2) fine-tuned Grounding DINO on the region labels; (3) VLM SFT with a detector-predicted box injected. (4) Many existing VLMs already integrate grounding functions, which are neither referred nor compared in the experiments. Representatives models include LISA, OMG-LLaVA, MedPLIB, MedSIGHT.
  5. The reasoning is not actually grounded, which makes the design not coherent at all. (1) In Fig. 1, QGR's <reason> box [62,69,111,134] differs from its <visual_loca> box [68,62,107,125]. Text and detector outputs are produced by separate paths, which undercuts the "reasoning tied to evidence" claim. (2) It is unclear how detector features reach the LM's text generation. The shared decoder in Eq. 2 feeds auxiliary heads, while L_LM is Qwen's own head. The +17.6 Closed gain on CT is therefore unexplained.
  6. Dataset concerns. (1) Questions are 10 templates per modality, and many leak the label, e.g., "Is there a <LABEL> in the image?" X-ray has a single label (pneumothorax) and US has two, so Region = 100% is trivial. (2) The split is at image level. VGMED's CT and MRI sources contain many slices per patient, so patient/volume-level leakage is likely and must be ruled out. (3) The GPT-generated answers and reasoning have no quality audit or inter-rater check.
  7. Tables are inconsistent. (1) Open-answer scores drop sharply with QGR: US 93.98 → 78.32 and X-ray 98.12 → 87.80 (Table 12). CT Region also drops (66.8 → 57.47). (2) The Qwen3-VL SFT numbers differ between Table 3 and Table 12 (e.g., CT Final 51.53 vs 52.11; Reason 50.42 vs 52.59). (3) The Final score averages heterogeneous metrics, one of which is the Box score. This inflates the headline gains.
  8. Novelty and positioning. Language-conditioned query refinement closely resembles MDETR and Grounding DINO's language-guided query selection and cross-modality decoder. And as I mentioned before, this LLM+grounding domain are already filled with many papers.

Some minor typo or inconsistent presentations: naming and metric inconsistencies: <visual_loca> vs <visual_locations>; "GPT-4 as judge" (Fig. 1) vs GPT-4o-mini (Sec. 4.1).

Questions for authors
  1. What is token 0 of Fv, and does it carry question information? What happens with hq set to zeros or with shuffled questions?
  2. How is QGR's top-1 box ranked: by detector score or by the quality head?
  3. How do detector features influence the generated text?
  4. Are the Med4VQA splits disjoint at the patient/volume level?
Reviewer_RTaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QGR, a query grounding and refinement framework for medical visual reasoning. QGR conditions detector object queries on the question context and applies query-level semantic assignment and localization-quality supervision. Experiments on Med4VQA, SLAKE, and GEMeX show improvements over SFT baselines, particularly in Box localization. The problem is important and the results are promising, but the current evaluation is not yet fully convincing.

Critical strengths

1.The paper addresses an important problem: medical VLMs may generate correct answers while grounding their reasoning in irrelevant image regions. 2.The proposed framework is intuitive and combines language-conditioned query refinement with explicit grounding supervision.

Critical weaknesses

1.The experiments do not directly establish that the model’s reasoning depends on the localized visual evidence. 2.The Box metric is relatively lenient, since a top-1 IoU of only 0.1 is counted as successful localization. The reported gains therefore demonstrate improved localization success at a low overlap threshold, but do not establish more precise localization.

Questions for authors

1.Can the authors provide component-wise ablations for query refinement, assignment loss, and quality loss while keeping the detector and decoder configuration fixed? 2.How do the results change under stricter IoU thresholds, such as 0.3 and 0.5, and what are the mean or median IoU? 3.Does masking or replacing the predicted region affect the model’s answer and reasoning? This would help test whether the reasoning is genuinely grounded. 4.Why was the 80-question GEMeX subset used, and are confidence intervals or results on a larger evaluation set available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.