acceptodds
Under review as a conference paper at ICLR 2027

BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation

Abstract

Adapting general-purpose multimodal large language models (MLLMs) to specialized domains requires learning domain-specific decision criteria, which often hinge on subtle visual distinctions between otherwise plausible answers. Rationale augmentation aims to expose such evidence through additional observations or inter-sample comparisons, yet a visually valid cue is not necessarily decision-relevant: it may describe how samples differ without changing the model's relative preference between competing answers. We therefore introduce **BIRD**, a self-improving **B**oundary-**I**nformed **R**ationale **D**istillation framework that uses model-specific confusions to locate unresolved local decision boundaries and distills the evidence that resolves these confusions into rationales. For each sample, BIRD retrieves candidate neighbors from the target MLLM's own representation space and selects the most confusable one according to its answer preferences. It then generates answer-blind candidate evidence from their visual differences and functionally verifies which evidence most effectively strengthens the model's preference for the correct answer while avoiding inappropriate transfer across the pair. The verified evidence is then distilled into a single-sample rationale for standard supervised fine-tuning. Experiments on medical and chart VQA show that BIRD outperforms competing rationale-augmentation methods across two target MLLMs, while further analyses demonstrate clearer separation of confusable answers and stronger gains from model-matched supervision. Anonymous code: link.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Self-Improving, Decision Boundary

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_XCYoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BIRD, an offline rationale construction framework designed for domain adaptation of multimodal large language models (MLLMs). For each training instance, BIRD utilizes the target MLLM’s visual representation space to retrieve question-compatible neighbors with distinct answers, and selects the most confusable alternative based on answer likelihood. Under an answer-blind setting, the target model generates paired visual observations for these samples. Candidate evidence is then functionally scored by considering both its own-side margin gain (whether it increases the preference for the correct answer over the confusable alternative on the corresponding image) and its cross-boundary transfer (whether it inappropriately promotes the wrong answer when swapped to the paired image). The selected evidence is distilled into single-sample rationales, which are subsequently used for standard supervised fine-tuning (SFT). The experimental evaluation spans two target models, two adaptation domains, and six evaluation benchmarks. On average, BIRD outperforms both original SFT and the competing baseline VC-STaR. Supplementary analyses further investigate held-out answer separation, cross-model supervision transfer, evidence selection strategies, component ablations, rationale length, and hyperparameter sensitivity.

Critical strengths

Concrete and functionally verifiable criteria for evidence selection. Instead of treating all paired visual differences as equally useful, BIRD jointly evaluates the margin gain when evidence is correctly matched and its potential for inappropriate transfer when swapped across images. Table 3 is particularly convincing, as it fixes both the candidate evidence pool and the number of refined rationales. Compared to selecting factors based solely on own-side gain, incorporating the full utility score yields average gains of 0.89 and 0.92 points across the two target models, respectively. This validates a key design choice of the proposed method rather than merely showing that rationale augmentation is beneficial.

Consistent downstream gains with a simple training and inference workflow. BIRD outperforms VC-STaR in 10 out of 12 model-benchmark combinations, achieving average improvements of 1.24 and 1.14 points across the six evaluation benchmarks. The final adaptation relies solely on standard supervised fine-tuning (SFT), requiring no retrieval or paired images at inference time. Furthermore, the rationale length analysis demonstrates that these gains cannot be attributed simply to longer rationales. These results make the proposed approach practical and effective within the evaluated settings.

Critical weaknesses

Incremental methodological novelty. Overall, BIRD closely follows the established paradigm of augmenting reasoning data for SFT via candidate generation, model-based scoring, and filtering: generating candidate evidence, filtering and refining rationales, and conducting standard SFT. The main incremental designs lie in the model-specific retrieval of confusable neighbors and the dual-aspect evidence scoring that combines own-side gain with cross-boundary transfer penalty. While these designs hold merit, the manuscript does not sufficiently demonstrate whether these additions constitute a major methodological contribution compared to existing rationale-augmentation and visual-contrast self-training frameworks.

Potential confounding from GPT-5 assistance on performance gains and fairness of comparison. The offline data construction pipeline relies on GPT-5 to verify task compatibility across questions and perform answer mapping. These operations directly affect the quality of contrastive sample pairs and confusable alternatives, which may in turn contribute to downstream performance gains. The paper does not clarify whether competing baselines received equivalent external model assistance, nor does it quantify GPT-5's impact through an ablation study that removes or replaces this module. Consequently, it is difficult to discern how much of the reported improvement stems from BIRD's core algorithmic design versus the high-quality dataset construction enabled by a strong external LLM.

Strong dependence on dataset structure and sample pairability. The proposed framework assumes that training samples contain original rationales and relies on the presence of sufficient question-compatible, distinct-answer, and visually similar neighbors to execute its core augmentation process. Although unpaired instances retain their original rationales, the framework's augmentation coverage and downstream gains remain sensitive to these dataset conditions. The evaluation is restricted to datasets where initial rationales are available and candidate pair coverage is relatively high, leaving the method's effectiveness unverified in settings with missing rationales or sparse comparable samples, thereby limiting its general applicability.

Questions for authors

Clarification on Novelty and Controlled Baselines: Compared to existing visual-contrast self-training methods (e.g., VC-STaR) and standard rationale-augmentation pipelines that rely on candidate generation and filtering, what is the most fundamental distinction of BIRD? Is there a controlled experiment—holding the rationale generation/refinement process and the number of updated samples constant—that isolates the specific performance gains contributed by model-specific pair selection and cross-boundary transfer scoring?

Impact of External Model Assistance (GPT-5): Were GPT-5’s task compatibility checks and answer mapping procedures also applied to the competing baselines? Do you have ablation results where the target MLLM itself replaces GPT-5, or where GPT-5 is omitted for subsets with directly matchable question templates? How do these variations affect candidate pair quality, rationale refinement coverage, and downstream performance?

Data Dependencies and Robustness to Coverage: Are pre-existing original rationales an indispensable input for the proposed framework? How do the downstream performance gains change as the density of comparable samples decreases and pair coverage drops? Do you have additional empirical analyses demonstrating whether the conclusions drawn from high-coverage datasets generalize to broader data conditions?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer XCYo

We thank the reviewer for the thoughtful comments. We address each concern below.

Q1. Methodological novelty.

Thank you for encouraging us to clarify the methodological distinction. VC-STaR and BIRD share the broader goal of improving rationale supervision through inter-sample contrast, but differ in the problem they explicitly target and how they construct supervision. VC-STaR addresses visual hallucinations in generated rationales by constructing pairs based on image and question similarity, then using contrastive analysis to refine the rationales. BIRD targets a more specific problem: which evidence resolves the target model’s own confusion between competing answers? This distinction is reflected in three components of our method:

  1. Confusion-driven pair selection. BIRD first retrieves neighbors using the target model’s visual representations, then selects the competing answer according to that model’s answer preferences. Thus, visual similarity defines the search neighborhood, while the model’s actual confusion determines the alternative to be resolved.
  2. Decision-based evidence selection. Rather than directly incorporating the differences identified through comparison, BIRD treats them as candidates and evaluates how each changes the target model’s preference between the correct and competing answers.
  3. Cross-boundary verification. BIRD evaluates each candidate on its corresponding image and after swapping evidence across the pair. Its utility rewards evidence that strengthens the appropriate decision while penalizing evidence that also pushes the opposite image toward an inappropriate answer. Consequently, BIRD’s contribution is a model-conditioned criterion for deciding which pair to contrast and which resulting evidence to distill, as detailed in Sections 4.2–4.5, particularly Eqs. (8) and (11)–(17).

Q2. Isolating pair selection and cross-boundary scoring.

Section 5.7, Table 4 separately ablates these components. Replacing model-guided confuser selection with the nearest valid visual neighbor reduces average performance by 1.12 and 1.09 points; removing the cross-boundary penalty reduces performance by 0.32 points for both backbones. Additionally, Section 5.5, Table 3 compares evidence-selection strategies while keeping the candidate pool and number of updated rationales fixed. Using the full utility instead of maximizing own-side gain alone improves average performance by 0.89 and 0.92 points, supporting the additional value of controlling cross-boundary transfer.

Q3. Replacing GPT-5 with the target MLLM.

We have added this ablation in Appendix C.6. Replacing GPT-5 with the corresponding base MLLM for task-compatibility checking and answer mapping, while keeping the remaining pipeline unchanged, yields identical downstream scores at the reported precision. These checks have a narrow scope, and GPT-5 was initially chosen as a conservative implementation choice. The results indicate that the target MLLM is sufficient for these operations and that the reported performance does not depend on GPT-5 assistance.

Q4. Generalization beyond high-coverage datasets.

We have added experiments on PMC-VQA-CoT (Appendix C.3, Tables 7–8). Valid-pair coverage falls to 28.6% and 25.9%, and only 17.8% and 16.2% of rationales are updated. Nevertheless, BIRD exceeds VC-STaR by 0.91 and 0.73 points in Medical Avg. These results extend the evidence to substantially lower pair coverage.

Reviewer_XCYoOpenReview ↗

Response

Thank you for the additional experiments. The GPT-5 replacement and low-coverage results alleviate some concerns, but the novelty over VC-STaR remains limited, and dependence on pre-existing rationales is still insufficiently addressed. My main concerns therefore remain, and I maintain my rating of 2 (weak rejection).

Reviewer_QvENOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts general-purpose MLLMs to specialized domains by improving rationale supervision. Existing image-comparison methods can identify visual differences, but those differences may not help the model distinguish the correct answer from a confusable alternative. BIRD uses the target model to select comparison samples, evaluates how candidate evidence changes answer preferences on its corresponding image and on the other image, and incorporates the selected evidence into single-sample rationales for standard SFT. Across separate chart and medical adaptation settings, BIRD improves the six-benchmark average over the strongest baseline, VC-STaR, by 1.24 points for Qwen3.5-9B and 1.14 points for InternVL3-8B.

Critical strengths
  1. The proposed selection rule has relatively direct experimental support. Table 3 shows that it outperforms simpler alternatives under matched conditions.

  2. The downstream gains are fairly consistent. BIRD outperforms VC-STaR in 10 of 12 model-benchmark combinations, with no extra retrieval or comparison images at inference.

Critical weaknesses
  1. The selection rule may not be robust. In Eqs. (13) and (17), U=(Δt+Δn)/2−λGcrossU=(\Delta_t+\Delta_n)/2-\lambda G_{\mathrm{cross}} can exceed the threshold even with negative target-side gain (Δt<0\Delta_t<0). For example, Δt=−0.2\Delta_t=-0.2 and Δn=1.4\Delta_n=1.4 give U=0.6>0.2U=0.6>0.2 when the cross-boundary penalty is zero. Equation (18) then uses the selected evidence to rewrite the target rationale, potentially introducing harmful supervision.

  2. The selected confuser may not reflect genuine confusion. Equation (8) selects the wrong answer with the highest length-normalized log-likelihood, but this score may still be far below that of the correct answer. In that case, the model already strongly favors the correct answer, so BIRD may reinforce an existing distinction rather than resolve confusion.

Questions for authors
  1. BIRD relies on the target model to construct its own supervision. Does this mean that the entire data-construction process must be repeated for every new model or adaptation task? If so, the approach appears to offer limited reuse across models and tasks.

  2. The component gains in Table 4 are small, particularly for rationale distillation, which adds only 0.38 and 0.36 points for the two models. These results do not yet convince me that the full method offers a clear advantage over simpler variants with some components removed.

  3. BIRD requires visually similar examples with compatible questions but different answers, which are not straightforward to find in practice. Yet Table 6 reports valid pairs for 87.6%-93.8% of the source samples. How is such high coverage achieved, and does it mainly rely on recurring question templates in these datasets?

  4. Table 6 reports that roughly two-thirds of the samples have their rationales rewritten, but passing the automatic filters does not establish that the rewritten content is reliable. Have the authors checked the actual quality of these rewrites?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer QvEN

We thank the reviewer for the thoughtful comments. We address each concern below.

Q1. Reusing the constructed data across models and tasks.

Building on self-improving rationale methods such as VC-STaR, BIRD focuses on offline adaptation of a selected MLLM to a defined domain. For this use case, rationale construction is performed once, and the resulting dataset can be reused across fine-tuning runs without additional inference-time requirements. For a new backbone on the same task, source examples and question-compatibility judgments can be reused, while model-dependent stages are rerun to tailor the supervision. A new task requires reassessing compatibility and constructing task-appropriate evidence. Direct reuse of another model’s rationales is also possible: Section 5.4, Table 2 shows average decreases of 1.66 and 1.73 percentage points, quantifying the trade-off between reuse and model-specific construction.

Q2. Negative target-side gain masked by neighbor-side gain.

We agree that the pair-averaged utility does not guarantee positive target-side gain. We therefore measured how frequently this issue occurs among selected evidence actually used for rationale refinement under the original rule:

Target modelDomainOccurrence rate
Qwen3.5-9BChart / Plot7.4%
Qwen3.5-9BMedical11.3%
InternVL3-8BChart / Plot8.6%
InternVL3-8BMedical9.9%

These cases constitute a minority of updated samples, although their frequency alone does not establish that they are harmless.

Q3. Strongest alternative versus genuine uncertainty.

We agree that the selected alternative may still score well below the gold answer. Since the gold-answer energy is fixed, Eq. (8) selects the smallest gold-versus-alternative margin among retrieved candidates; it does not guarantee uncertainty. The resulting supervision may therefore reinforce an existing distinction. Nevertheless, replacing this selection with the nearest valid visual neighbor reduces average accuracy by 1.12 and 1.09 points (Section 5.7, Table 4). We have clarified in Section 4.3 that “confuser” denotes the strongest retrieved alternative, rather than necessarily an unresolved confusion.

Q4. Modest gains from rationale distillation.

Rationale distillation is a practical and engineering step for integrating selected evidence into the original reasoning, and we expect an incremental benefit from this refinement. The simpler append-only variant already receives the same verified evidence. Rewriting improves average performance by 0.41 and 0.39 points over that variant (Section 5.7, Table 4), supporting its usefulness without suggesting that it accounts for most of BIRD’s gains. The human evaluation in Section 5.8 additionally assesses the quality of this integration.

Q5. High valid-pair coverage.

Yes, recurring question templates in our source datasets facilitate compatible-pair construction, so the high coverage in Table 6 should not be assumed for arbitrary datasets. We have added experiments on PMC-VQA-CoT, where pair coverage drops to 28.6% and 25.9% for the two backbones. BIRD still outperforms VC-STaR by 0.91 and 0.73 points in Medical Avg., showing that it remains useful when valid pairs are substantially less common (Appendix C.3, Tables 7–8).

Q6. Human assessment of rewritten rationales.

We have added a blinded human evaluation of 500 randomly sampled updated examples, assessed independently by three reviewers (Section 5.8). Reviewers compare BIRD’s rewrite with an append-only version containing the same evidence, presented in random order without method labels. They assess visual support, evidence integration, and preservation of correct original content, with majority voting and discussion when needed. BIRD’s evidence integration is preferred in 97.4% of cases, with 1.2% judged equivalent and 1.4% favoring append-only. The results indicate that BIRD-optimized rationales are high-quality data.

Thank you for the additional effort, but most of the responses confirm or quantify the identified limitations rather than resolve them. The reuse discussion confirms that model-dependent construction must be repeated for each new backbone to achieve the best performance, while cross-model reuse causes an approximately 1.7-point drop. The 7.4%-11.3% occurrence of negative target-side gain shows that the selection-rule issue is not merely hypothetical. The nearest-neighbor comparison supports model-based alternative selection but does not demonstrate genuine uncertainty or an unresolved decision boundary. The approximately 0.4-point gain from rationale rewriting remains modest. Overall, the main methodological concerns remain, so I maintain my original assessment.

Reviewer_kMjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BIRD, a framework for constructing model-specific rationales for domain adaptation of multimodal large language models. For each training sample, BIRD retrieves question-compatible samples with different answers using the target model’s visual representations. It selects a confusable alternative based on answer energy, generates answer-blind candidate visual differences from the target–witness pair, and evaluates these candidates using own-side preference gain and cross-boundary transfer. The highest-utility factor is incorporated into the original rationale, after which standard rationale-supervised fine-tuning is performed. The method is evaluated using Qwen3.5-9B and InternVL3-8B on chart and medical VQA tasks across six benchmarks. The paper reports improved average performance over existing rationale-augmentation methods and includes cross-model experiments, ablations, sensitivity analyses, and case studies.

Critical strengths
  1. The model-specific perspective is well motivated. The paper makes the useful observation that a visually valid difference between two samples is not necessarily relevant to resolving a model’s confusion. Conditioning rationale construction on the target model’s own retrieval geometry and answer preferences is a sensible direction. The distinction between own-side gain and cross-boundary transfer also provides an intuitive way to reject generic differences that do not separate the competing answers.

  2. The empirical study is broad and the pipeline is evaluated from several angles. The evaluation covers two model families, two distinct domains, and six benchmarks. The paper also includes cross-model evidence and supervision-transfer experiments, ablations, sensitivity analyses, rationale-length analysis, and case studies. These experiments provide reasonable evidence that the complete pipeline can be useful in practice, even though they do not fully resolve the novelty and grounding concerns below.

Critical weaknesses

1. The methodological contribution appears incremental relative to existing ideas.
BIRD combines several established components: representation-based neighbor retrieval, hard-negative or confuser selection, pairwise comparison of similar samples with different answers, answer-blind candidate generation, generate-then-verify rationale construction, and rationale distillation. The main distinguishing element appears to be the heuristic utility function U=Gown−λGcrossU=G_{\mathrm{own}}-\lambda G_{\mathrm{cross}}. While intuitive, this criterion is relatively simple and is not theoretically derived or shown to identify an actual decision boundary. The ablations demonstrate that components of the pipeline contribute to performance, but they do not by themselves establish a substantial conceptual advance over the closest inter-sample and self-improving rationale methods. Thus, the current contribution feels closer to a well-executed combination of existing techniques than to a sufficiently novel method.

2. The empirical evidence does not fully substantiate the stronger decision-boundary interpretation.
Candidate evidence is validated by inserting a textual description and measuring changes in the target model’s answer energy. This demonstrates a behavioral effect, but not that the model has visually grounded the corresponding feature or that the feature causally defines a decision boundary. Since the same target model participates in candidate generation and evaluation, the selection process may also favor descriptions that exploit its existing textual biases. This concern is particularly relevant to the cross-model evidence-utility analysis. Moreover, the average gains over VC-STaR are approximately small (i.e., 1.24% and 1.14%) for the two target models, and no multiple-seed results, variance estimates, or significance tests are reported. Finally, the sensitivity analysis shows that the default settings perform best across the reported benchmarks, but the paper does not clarify whether a separate validation split was used for hyperparameter selection. These issues weaken confidence that the reported improvements represent a robust and general methodological advance.

Questions for authors
  1. What is the precise methodological advance over the closest inter-sample rationale methods, particularly VC-STaR, beyond the proposed boundary-utility score? Is there a stronger justification for Gown−λGcrossG_{\mathrm{own}}-\lambda G_{\mathrm{cross}} as a boundary-specific criterion, rather than simply a useful heuristic for selecting model-compatible text?

  2. Were any of the reported benchmark test sets used to select PP, λ\lambda, τU\tau_U, or KK? Please clarify the hyperparameter-selection protocol and, if possible, provide results over multiple independently trained seeds with variance estimates.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer kMjm

We thank the reviewer for the thoughtful comments. We address each concern below.

Q1. Methodological novelty and the utility function.

Thank you for encouraging us to clarify the methodological distinction. VC-STaR and BIRD share the broader goal of improving rationale supervision through inter-sample contrast, but differ in the problem they explicitly target and how they construct supervision. VC-STaR addresses visual hallucinations in generated rationales by constructing pairs based on image and question similarity, then using contrastive analysis to refine the rationales. BIRD targets a more specific problem: which evidence resolves the target model’s own confusion between competing answers? This distinction is reflected in three components of our method:

  1. Confusion-driven pair selection. BIRD first retrieves neighbors using the target model’s visual representations, then selects the competing answer according to that model’s answer preferences. Thus, visual similarity defines the search neighborhood, while the model’s actual confusion determines the alternative to be resolved.
  2. Decision-based evidence selection. Rather than directly incorporating the differences identified through comparison, BIRD treats them as candidates and evaluates how each changes the target model’s preference between the correct and competing answers.
  3. Cross-boundary verification. BIRD evaluates each candidate on its corresponding image and after swapping evidence across the pair. Its utility rewards evidence that strengthens the appropriate decision while penalizing evidence that also pushes the opposite image toward an inappropriate answer. Consequently, BIRD’s contribution is a model-conditioned criterion for deciding which pair to contrast and which resulting evidence to distill, as detailed in Sections 4.2–4.5, particularly Eqs. (8) and (11)–(17).

The utility operationalizes this principle: useful evidence should strengthen the appropriate decision on its own side while avoiding inappropriate transfer to the opposite side. Its linear form is an empirically motivated selection criterion, rather than a theoretically derived estimator of the decision boundary. Its justification comes from the controlled comparison in Section 5.5, Table 3: with the candidate pool and number of updated rationales fixed, the full utility outperforms both random selection and maximizing own-side gain alone. Section 5.3, Figure 4 further shows improved separation of competing answers on held-out pairs. We distinguish this functional interpretation from causal identification of visual evidence in Appendix F.1.

Q2. Hyperparameter selection.

The hyperparameters were selected using a held-out validation set, with the adaptation data divided into training and validation subsets at an 8:2 ratio. None of the benchmark test sets were used for hyperparameter selection. The sensitivity analysis is reported in Appendix D.

Q3. Multi-seed reliability.

We have added results across three random seeds for BIRD and VC-STaR (Section 5.1, Table 1). BIRD retains higher average performance and shows lower across-seed variability for both backbones, supporting its greater training stability.

Reviewer_CMoeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BIRD, a framework for improving rationale supervision in multimodal language models. It identifies model-specific confusable examples, generates candidate visual evidence, and selects evidence based on its effect on the model's answer preferences. The selected evidence is then incorporated into rationales for supervised fine-tuning. Experiments on medical and chart VQA with two MLLMs show improvements over existing rationale-augmentation methods.

Critical strengths
  1. The paper addresses an interesting problem by distinguishing visually meaningful evidence from evidence that is actually useful for resolving model confusion.
  2. The method is clearly motivated, and the ablation studies provide useful evidence for the effectiveness of the proposed evidence selection strategy.
Critical weaknesses
  1. Limited novelty over existing methods. BIRD shares much of its overall pipeline with VC-STaR, including contrastive sample selection and rationale refinement. The main additional contribution is a likelihood-based evidence selection mechanism, while the improvements over VC-STaR remain relatively modest.
  2. Insufficient validation of the central claim. The proposed evidence verification measures how textual evidence changes answer probabilities, but this does not necessarily demonstrate improved visual grounding or genuine resolution of decision boundaries. The current experiments do not fully establish this distinction.
Questions for authors
  1. In Section 4.4, the proposed evidence verification relies on changes in answer likelihood after adding textual evidence. How can the authors demonstrate that the improvement comes from better visual grounding rather than relying on additional textual cues? Could a text-only or visual perturbation experiment help clarify this?
  2. The improvements over VC-STaR in Table 1 are relatively modest. Are these improvements consistent across multiple training seeds? Could the authors provide variance or confidence intervals to establish their reliability?
  3. Given the similarity between BIRD and VC-STaR, could the authors clarify the additional benefits of the proposed boundary-based selection mechanism? In particular, how does BIRD compare with VC-STaR under matched data construction and computational budgets?
  4. Table 2 suggests that model-specific rationales are more effective. However, this difference could also arise from variations in rationale quality or generation style. Could the authors further isolate the contribution of model-specific boundary scoring in downstream training while keeping candidate pairs and evidence fixed? I currently lean toward weak rejection. However, I am open to reconsidering my rating if the authors can adequately address these concerns, particularly by providing stronger evidence for visual grounding and more convincing comparisons with existing methods.
2 comments
AuthorsOpenReview ↗

Reply to Reviewer CMoe

We thank the reviewer for the thoughtful comments. We address each concern below.

Q1. Methodological differences from VC-STaR.

Thank you for encouraging us to clarify the methodological distinction. VC-STaR and BIRD share the broader goal of improving rationale supervision through inter-sample contrast, but differ in the problem they explicitly target and how they construct supervision. VC-STaR addresses visual hallucinations in generated rationales by constructing pairs based on image and question similarity, then using contrastive analysis to refine the rationales. BIRD targets a more specific problem: which evidence resolves the target model’s own confusion between competing answers? This distinction is reflected in three components of our method:

  1. Confusion-driven pair selection. BIRD first retrieves neighbors using the target model’s visual representations, then selects the competing answer according to that model’s answer preferences. Thus, visual similarity defines the search neighborhood, while the model’s actual confusion determines the alternative to be resolved.
  2. Decision-based evidence selection. Rather than directly incorporating the differences identified through comparison, BIRD treats them as candidates and evaluates how each changes the target model’s preference between the correct and competing answers.
  3. Cross-boundary verification. BIRD evaluates each candidate on its corresponding image and after swapping evidence across the pair. Its utility rewards evidence that strengthens the appropriate decision while penalizing evidence that also pushes the opposite image toward an inappropriate answer. Consequently, BIRD’s contribution is a model-conditioned criterion for deciding which pair to contrast and which resulting evidence to distill, as detailed in Sections 4.2–4.5, particularly Eqs. (8) and (11)–(17).

Q2. Insufficient validation of the central claim.

Regarding the modest performance gap, our quantitative evaluation compares the methods on a finite set of datasets under a fixed data and training budget. The achievable performance is constrained by the available supervision and the capabilities of the target backbone, which can limit the gains attainable through rationale refinement. These experiments establish the relative effectiveness of the methods within the evaluated settings, but the observed score differences should not be interpreted as a universal measure of their methodological advantages. In particular, aggregate accuracy alone does not reveal how the methods differ in their use of visual evidence. To better examine this behavioral distinction, we have added Section 5.6 and Figure 7, comparing the attention maps of BIRD and VC-STaR at epochs 2, 6, and 10. In the presented medical and chart examples, BIRD maintains more consistent attention to answer-relevant regions, whereas VC-STaR exhibits greater distraction toward irrelevant or competing regions. These observations complement the quantitative results by providing qualitative evidence that BIRD’s boundary-informed supervision better sustains visual grounding throughout training. We interpret the attention visualizations as supporting case studies rather than a standalone causal demonstration of visual grounding. The distinction between functional boundary verification and causal grounding is explicitly discussed in Appendix F.1.

Q3. Additional textual cues.

We have added an analysis of evidence length versus diagnostic utility across all candidates, highlighting those selected by BIRD (Appendix C.5, Figure 10). Candidates of similar length exhibit substantially different utilities, and selected evidence is not systematically longer. This suggests that higher utility is not simply attributable to additional text length.

Q4. Model-specific boundary scoring.

BIRD forms an integrated self-improvement mechanism: the target model proposes candidate factors and selects those that best resolve its own decision ambiguities. Replacing only the scorer removes this alignment between evidence generation and evaluation. Nevertheless, to isolate the scorer’s contribution, we added an ablation that keeps candidate pairs and evidence fixed and changes only the scoring model (Section 5.7, Table 4). Cross-model scoring reduces average performance by 0.26 and 0.25 points for the two backbones, indicating a modest but consistent benefit from using the target model throughout the process.

Q5. Reliability across multiple random seeds.

We have added results across three random seeds for both BIRD and VC-STaR (Section 5.1, Table 1). BIRD maintains higher average performance and exhibits lower across-seed variability for both backbones, supporting its reliability and greater training stability.

Thank you for the detailed response and the extra experiments. The multi-seed runs and controlled ablations resolve my concerns about training stability and component attribution. I also appreciate that you clearly stated the method's limitations in Appendix F.1. Even though causal checks for visual grounding and compute-matched comparisons with VC-STaR are still missing, I think the paper is solid enough now, so I will raise my score to 3.

Comments on the paper

1 comment

Dear Reviewers,

We’re now past the halfway point of the ICLR discussion period. Please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.