- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
SemAlign-VQA studies whether medical VQA answers remain correct across changes in question formulation. It combines original-question answer cross-entropy with SemLoss, a pull-push regularizer on projected pre-answer representations of paired question views, and adapts the model using LoRA. The projection head is bypassed at inference. Evaluation groups each original question with five fixed context wrappers on PathVQA or automatically screened rewrites on SLAKE-English and VQA-RAD. The paper distinguishes answer agreement from all-six correctness, analyzes matched outcome transitions, and compares alternative losses and rewrite supervision. Moondream2 gains 1.85 and 2.17 percentage points in all-six correctness on PathVQA and SLAKE at seed 42, but the advantage depends on the training objective, backbone, and seed.
Critical strengths
-
The paper separates agreement from correctness clearly: of the 217 PathVQA groups that stopped being inconsistent, 124 became all-correct and 93 became identically wrong, and the image-cluster bootstrap intervals and repeated seeds back this up.
-
The release (code, configurations, fixed question views, metrics and raw Moondream predictions) lets the reported accuracy, agreement and all-six correctness be re-scored, which checks the arithmetic but is not an independent training reproduction.
Critical weaknesses
- The method is incremental, since reformulated-question consistency training already appears in ConClaT, CCL, SEQA and Mechanistically Guided LoRA, and the paper’s own controls show no clear advantage for its margin design (answer-aware NCE 35.28% versus SemLoss 35.30% on PathVQA, and direct rewrite supervision 65.22% versus SemLoss 57.30% on SLAKE).
- The evidence for semantic robustness is too narrow to offset this, because the PathVQA variants are fixed context wrappers, the SLAKE and VQA-RAD rewrites have no human equivalence check, VQA-RAD shares images across splits, and all-six correctness improves in only 5 of 12 backbone and seed pairs, so I recommend 1: clear rejection.
Questions for authors
-
What is the essential methodological or analytical advance beyond the already-cited question-augmentation, consistency, and contrastive VQA studies? In particular, what evidence favors the margin objective over answer-aware NCE or direct rewrite supervision when all-six correctness is the target? Please distinguish a new empirical finding from a new learning mechanism and explain why the observed differences are consequential.
-
Is there any human audit of semantic equivalence and answer preservation for SLAKE/VQA-RAD rewrites, or an existing PathVQA evaluation using genuine paraphrases rather than fixed wrappers? Please quantify any audited error rates and explain how questionable groups affect the main findings. Which conclusions remain supported when the VQA-RAD shared-image split is taken into account?
-
Can the alternative-loss and augmentation controls be tied to the exact evaluated question identities, checkpoints, and per-question predictions? In particular, please resolve or explicitly exclude the unverified VQA-RAD augmentation comparisons. Any already available repeated-seed SLAKE/VQA-RAD results would help assess whether their gains are stable; summary metrics alone cannot support the same paired transition analysis as the primary runs.
-
Please clarify the impact of the implementation differences disclosed in the supplementary notes: SmolVLM LoRA also updates visual attention adapters, question/answer length caps are not uniformly enforced across adapters, and image-feature reuse and negative-pool sizes differ. What existing evidence separates these factors from pre-answer state choice? Also, how do the margin and loss weight respond to the fact that instance-based negatives can share an answer or clinical concept?
Guidelines on the Use of AI Tools for Manuscript Review
This review process utilized AI to translate and organize the core content of the manuscript. Subsequently, human reviewers conducted a thorough assessment and finalized the review results. Human oversight ensured the overall quality and formatting of the manuscript, and the final conclusions were determined by human judgment rather than by AI.