- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BIRD, an offline rationale construction framework designed for domain adaptation of multimodal large language models (MLLMs). For each training instance, BIRD utilizes the target MLLM’s visual representation space to retrieve question-compatible neighbors with distinct answers, and selects the most confusable alternative based on answer likelihood. Under an answer-blind setting, the target model generates paired visual observations for these samples. Candidate evidence is then functionally scored by considering both its own-side margin gain (whether it increases the preference for the correct answer over the confusable alternative on the corresponding image) and its cross-boundary transfer (whether it inappropriately promotes the wrong answer when swapped to the paired image). The selected evidence is distilled into single-sample rationales, which are subsequently used for standard supervised fine-tuning (SFT). The experimental evaluation spans two target models, two adaptation domains, and six evaluation benchmarks. On average, BIRD outperforms both original SFT and the competing baseline VC-STaR. Supplementary analyses further investigate held-out answer separation, cross-model supervision transfer, evidence selection strategies, component ablations, rationale length, and hyperparameter sensitivity.
Critical strengths
Concrete and functionally verifiable criteria for evidence selection. Instead of treating all paired visual differences as equally useful, BIRD jointly evaluates the margin gain when evidence is correctly matched and its potential for inappropriate transfer when swapped across images. Table 3 is particularly convincing, as it fixes both the candidate evidence pool and the number of refined rationales. Compared to selecting factors based solely on own-side gain, incorporating the full utility score yields average gains of 0.89 and 0.92 points across the two target models, respectively. This validates a key design choice of the proposed method rather than merely showing that rationale augmentation is beneficial.
Consistent downstream gains with a simple training and inference workflow. BIRD outperforms VC-STaR in 10 out of 12 model-benchmark combinations, achieving average improvements of 1.24 and 1.14 points across the six evaluation benchmarks. The final adaptation relies solely on standard supervised fine-tuning (SFT), requiring no retrieval or paired images at inference time. Furthermore, the rationale length analysis demonstrates that these gains cannot be attributed simply to longer rationales. These results make the proposed approach practical and effective within the evaluated settings.
Critical weaknesses
Incremental methodological novelty. Overall, BIRD closely follows the established paradigm of augmenting reasoning data for SFT via candidate generation, model-based scoring, and filtering: generating candidate evidence, filtering and refining rationales, and conducting standard SFT. The main incremental designs lie in the model-specific retrieval of confusable neighbors and the dual-aspect evidence scoring that combines own-side gain with cross-boundary transfer penalty. While these designs hold merit, the manuscript does not sufficiently demonstrate whether these additions constitute a major methodological contribution compared to existing rationale-augmentation and visual-contrast self-training frameworks.
Potential confounding from GPT-5 assistance on performance gains and fairness of comparison. The offline data construction pipeline relies on GPT-5 to verify task compatibility across questions and perform answer mapping. These operations directly affect the quality of contrastive sample pairs and confusable alternatives, which may in turn contribute to downstream performance gains. The paper does not clarify whether competing baselines received equivalent external model assistance, nor does it quantify GPT-5's impact through an ablation study that removes or replaces this module. Consequently, it is difficult to discern how much of the reported improvement stems from BIRD's core algorithmic design versus the high-quality dataset construction enabled by a strong external LLM.
Strong dependence on dataset structure and sample pairability. The proposed framework assumes that training samples contain original rationales and relies on the presence of sufficient question-compatible, distinct-answer, and visually similar neighbors to execute its core augmentation process. Although unpaired instances retain their original rationales, the framework's augmentation coverage and downstream gains remain sensitive to these dataset conditions. The evaluation is restricted to datasets where initial rationales are available and candidate pair coverage is relatively high, leaving the method's effectiveness unverified in settings with missing rationales or sparse comparable samples, thereby limiting its general applicability.
Questions for authors
Clarification on Novelty and Controlled Baselines: Compared to existing visual-contrast self-training methods (e.g., VC-STaR) and standard rationale-augmentation pipelines that rely on candidate generation and filtering, what is the most fundamental distinction of BIRD? Is there a controlled experiment—holding the rationale generation/refinement process and the number of updated samples constant—that isolates the specific performance gains contributed by model-specific pair selection and cross-boundary transfer scoring?
Impact of External Model Assistance (GPT-5): Were GPT-5’s task compatibility checks and answer mapping procedures also applied to the competing baselines? Do you have ablation results where the target MLLM itself replaces GPT-5, or where GPT-5 is omitted for subsets with directly matchable question templates? How do these variations affect candidate pair quality, rationale refinement coverage, and downstream performance?
Data Dependencies and Robustness to Coverage: Are pre-existing original rationales an indispensable input for the proposed framework? How do the downstream performance gains change as the density of comparable samples decreases and pair coverage drops? Do you have additional empirical analyses demonstrating whether the conclusions drawn from high-coverage datasets generalize to broader data conditions?
2 comments
Response
Thank you for the additional experiments. The GPT-5 replacement and low-coverage results alleviate some concerns, but the novelty over VC-STaR remains limited, and dependence on pre-existing rationales is still insufficiently addressed. My main concerns therefore remain, and I maintain my rating of 2 (weak rejection).
Reply to Reviewer XCYo
We thank the reviewer for the thoughtful comments. We address each concern below.
Q1. Methodological novelty.
Thank you for encouraging us to clarify the methodological distinction. VC-STaR and BIRD share the broader goal of improving rationale supervision through inter-sample contrast, but differ in the problem they explicitly target and how they construct supervision. VC-STaR addresses visual hallucinations in generated rationales by constructing pairs based on image and question similarity, then using contrastive analysis to refine the rationales. BIRD targets a more specific problem: which evidence resolves the target model’s own confusion between competing answers? This distinction is reflected in three components of our method:
Q2. Isolating pair selection and cross-boundary scoring.
Section 5.7, Table 4 separately ablates these components. Replacing model-guided confuser selection with the nearest valid visual neighbor reduces average performance by 1.12 and 1.09 points; removing the cross-boundary penalty reduces performance by 0.32 points for both backbones. Additionally, Section 5.5, Table 3 compares evidence-selection strategies while keeping the candidate pool and number of updated rationales fixed. Using the full utility instead of maximizing own-side gain alone improves average performance by 0.89 and 0.92 points, supporting the additional value of controlling cross-boundary transfer.
Q3. Replacing GPT-5 with the target MLLM.
We have added this ablation in Appendix C.6. Replacing GPT-5 with the corresponding base MLLM for task-compatibility checking and answer mapping, while keeping the remaining pipeline unchanged, yields identical downstream scores at the reported precision. These checks have a narrow scope, and GPT-5 was initially chosen as a conservative implementation choice. The results indicate that the target MLLM is sufficient for these operations and that the reported performance does not depend on GPT-5 assistance.
Q4. Generalization beyond high-coverage datasets.
We have added experiments on PMC-VQA-CoT (Appendix C.3, Tables 7–8). Valid-pair coverage falls to 28.6% and 25.9%, and only 17.8% and 16.2% of rationales are updated. Nevertheless, BIRD exceeds VC-STaR by 0.91 and 0.73 points in Medical Avg. These results extend the evidence to substantially lower pair coverage.