acceptodds
Under review as a conference paper at ICLR 2027

SemAlign-VQA: Semantic Alignment for Medical Visual Question Answering

Abstract

Medical visual question answering requires correct answers to persist across changes in question formulation. We study how paired training changes this outcome, separating correct-answer retention from answer agreement. SemAlign-VQA combines original-question answer supervision with SemLoss, a pull–push regularizer on projected pre-answer states, using LoRA and a projector bypassed at inference. Evaluation uses an original question and five fixed context wrappers on PathVQA or automatically screened rewrites on SLAKE-English and VQA-RAD. At seed 42, Moondream2 improves all-six correctness by 1.85 and 2.17 percentage points on PathVQA and SLAKE, with nearly unchanged original accuracy. Matched-question analysis locates these gains in retention, while outcome transitions distinguish sustained correct answers from stabilized errors. The training controls reveal a second distinction: direct rewrite supervision reaches 65.22% all-six correctness on SLAKE versus 57.30% with original-only alignment, and the summary-level answer-aware NCE control nearly matches the margin objective on PathVQA despite lower agreement. Effects also vary across backbones and training seeds. Together, the paired comparisons show where alignment changes correctness, how those changes differ from consistency gains, and why supervision-matched controls matter when choosing a training objective.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical visual question answering, vision-language models, semantic alignment, question reformulation, answer consistency, robustness evaluation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jh5EOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

SemAlign-VQA studies whether medical VQA answers remain correct across changes in question formulation. It combines original-question answer cross-entropy with SemLoss, a pull-push regularizer on projected pre-answer representations of paired question views, and adapts the model using LoRA. The projection head is bypassed at inference. Evaluation groups each original question with five fixed context wrappers on PathVQA or automatically screened rewrites on SLAKE-English and VQA-RAD. The paper distinguishes answer agreement from all-six correctness, analyzes matched outcome transitions, and compares alternative losses and rewrite supervision. Moondream2 gains 1.85 and 2.17 percentage points in all-six correctness on PathVQA and SLAKE at seed 42, but the advantage depends on the training objective, backbone, and seed.

Critical strengths
  1. The paper separates agreement from correctness clearly: of the 217 PathVQA groups that stopped being inconsistent, 124 became all-correct and 93 became identically wrong, and the image-cluster bootstrap intervals and repeated seeds back this up.

  2. The release (code, configurations, fixed question views, metrics and raw Moondream predictions) lets the reported accuracy, agreement and all-six correctness be re-scored, which checks the arithmetic but is not an independent training reproduction.

Critical weaknesses
  1. The method is incremental, since reformulated-question consistency training already appears in ConClaT, CCL, SEQA and Mechanistically Guided LoRA, and the paper’s own controls show no clear advantage for its margin design (answer-aware NCE 35.28% versus SemLoss 35.30% on PathVQA, and direct rewrite supervision 65.22% versus SemLoss 57.30% on SLAKE).
  2. The evidence for semantic robustness is too narrow to offset this, because the PathVQA variants are fixed context wrappers, the SLAKE and VQA-RAD rewrites have no human equivalence check, VQA-RAD shares images across splits, and all-six correctness improves in only 5 of 12 backbone and seed pairs, so I recommend 1: clear rejection.
Questions for authors
  1. What is the essential methodological or analytical advance beyond the already-cited question-augmentation, consistency, and contrastive VQA studies? In particular, what evidence favors the margin objective over answer-aware NCE or direct rewrite supervision when all-six correctness is the target? Please distinguish a new empirical finding from a new learning mechanism and explain why the observed differences are consequential.

  2. Is there any human audit of semantic equivalence and answer preservation for SLAKE/VQA-RAD rewrites, or an existing PathVQA evaluation using genuine paraphrases rather than fixed wrappers? Please quantify any audited error rates and explain how questionable groups affect the main findings. Which conclusions remain supported when the VQA-RAD shared-image split is taken into account?

  3. Can the alternative-loss and augmentation controls be tied to the exact evaluated question identities, checkpoints, and per-question predictions? In particular, please resolve or explicitly exclude the unverified VQA-RAD augmentation comparisons. Any already available repeated-seed SLAKE/VQA-RAD results would help assess whether their gains are stable; summary metrics alone cannot support the same paired transition analysis as the primary runs.

  4. Please clarify the impact of the implementation differences disclosed in the supplementary notes: SmolVLM LoRA also updates visual attention adapters, question/answer length caps are not uniformly enforced across adapters, and image-feature reuse and negative-pool sizes differ. What existing evidence separates these factors from pre-answer state choice? Also, how do the margin and loss weight respond to the fact that instance-based negatives can share an answer or clinical concept?

Reviewer_ARUrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SemAlignVQA and introduces a supervision signal and a semantic loss (SemLoss) based on original question-answer pairs, enabling the model to maintain correct answers even when question phrasing changes. Using a random seed of 42, Moondream2 achieved improvements of 1.85 and 2.17 percentage points in "all-six correctness" on the PathVQA and SLAKE datasets, respectively, while maintaining essentially unchanged accuracy for the original questions.

Critical strengths
  1. This paper reveals how alignment training alters accuracy and how these changes differ from improvements in consistency;
  2. The experiments are extensive, exploring how the model responds to variations in question phrasing.
Critical weaknesses
  1. The paper suffers from poor formatting, illogical charts and diagrams, and an unsound experimental design;
  2. I do not consider the loss function, which treats perturbed versions of the same problem as positive samples and other problems within the same batch as negative samples to be particularly innovative;
  3. The average overall accuracy of SemLoss is nearly identical to that of answer-aware NCE;
  4. The baseline methods used for comparison are outdated.
Questions for authors
  1. The technical contributions seem unclear to me: is it the pre-answer hidden state, the projection head, the relative distance margin, or simply applying answer supervision to the original question? How do the experimental results disentangle the effects of these different changes?

  2. In Table 6, the average overall accuracy for SemLoss and answer-aware NCE is nearly identical. So, where does the advantage lie?

  3. PathVQA uses a fixed context, whereas the other two datasets use automatically generated rewrites. How does this variation support the same conclusion regarding "semantic alignment"?

1 comment
Reviewer_ARUrOpenReview ↗

Guidelines on the Use of AI Tools for Manuscript Review

This review process utilized AI to translate and organize the core content of the manuscript. Subsequently, human reviewers conducted a thorough assessment and finalized the review results. Human oversight ensured the overall quality and formatting of the manuscript, and the final conclusions were determined by human judgment rather than by AI.

Reviewer_ddEsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SemAlign-VQA investigates whether medical VQA models can retain correct answers when questions are reformulated. It combines answer supervision on original questions with a pull–push loss that aligns representations of original and rewritten questions. The additional projection head is removed from the inference path. The evaluation measures both answer agreement and correctness across six question views. Matched analyses show that improved agreement can come from retaining correct answers or repeating incorrect ones. Direct rewrite supervision outperforms SemLoss on SLAKE, while answer-aware NCE achieves similar mean all-six correctness on PathVQA.

Critical strengths
  1. The evaluation distinguishes agreement from correctness and examines matched outcome transitions. On PathVQA, the reduction in inconsistent groups includes 124 additional all-correct groups and 93 additional identically wrong groups. This directly shows why agreement alone can misrepresent an intervention’s benefit.
  2. The paper includes relevant supervision and loss controls and reports unfavorable results openly. These comparisons help separate the effects of representation alignment from those of training on rewritten questions.
Critical weaknesses
  1. The added value and applicability of SemLoss remain unclear. On SLAKE, adding SemLoss to AugCE improves all-six correctness by only 0.28 percentage points in a single seed (Table 5). The matched-transition analysis excludes AugCE and answer-aware NCE, leaving unclear whether the observed retention and error-stabilization patterns are specific to SemLoss. SemLoss also reduces all-six correctness for BLIP in all three PathVQA seeds and yields a negative mean change for InternVL2. The study does not isolate the effects of representation readout and negative-pool size, limiting guidance on when the method helps.
  2. The test rewrites lack a measured answer-preservation rate. Automatic screening checks negation, laterality, numbers, and other features, but no human audit is reported. Given the small performance differences, evaluation on a verified equivalent subset is needed to assess whether the gains persist. PathVQA’s fixed wrappers reduce this uncertainty but test a narrower range of question changes.
Questions for authors
  1. Can the matched transition analysis be extended to AugCE and answer-aware NCE? Do these methods show the same balance between retaining correct answers and stabilizing errors?
  2. What proportion of generated rewrites preserve both meaning and the reference answer in a blinded human audit? Do the main comparisons remain similar on verified question groups?
  3. Are repeated-seed results available for AugCE versus AugCE+SemLoss on SLAKE? If so, do the paired differences support a reproducible benefit from adding alignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.