- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the impact of textual reasoning distillation on visual language models, and uses forced termination of answers, short reasoning teachers, and weighted interpolation to mitigate the effects.
Critical strengths
A thorough audit was conducted on evaluation factors such as answer extraction, object matching, and output length.
Weighted interpolation requires no additional training and restores some accuracy while preserving longer outputs.
Critical weaknesses
-
This paper demonstrates the problem of rule extractors missing answers, but the original VOLD paper explicitly uses VLMEvalKit and GPT-4o-mini for answer extraction. It needs to be proven that these errors also exist in the original evaluation process.
-
Teachers answering incorrectly without looking at the image does not necessarily mean the question must rely on the image; it could also stem from reasoning failure or output truncation.
Questions for authors
After using VOLD's original answer extraction process, does the spurious performance gain reported in the text still exist
After using an independent model to determine image necessity, are the inter-group differences valid