- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose FinVQA-Chart that separates these three abilities via modality-controlled question triplets. It includes 149400 five-option multiple-choice questions stratified by reasoning category, modality, and difficulty. The paper evaluates 45 VLMs. showing the strongest model achieves 72.3% overall, 19.5 points below human experts.
Critical strengths
1 A comprehensive evaluation includes 45 VLMs, which is rare to see in other benchmark papers.
2 The benchmark size is quite large with 149400 questions.
Critical weaknesses
- All figures are vague in the paper. More importantly, Figure 1 shows one specific case. But the bars in the two charts look different to me, although they are from the same case and should be exactly the same. Could the authors explain why this difference happens?
- The authors use deterministic algorithms to generate ground-truth answers, but it is unclear how they check whether these answers are correct. For example, if one candlestick satisfies the conditions of multiple patterns, how do the authors ensure that only one answer is correct in the MCQ?
- The difficulty design is unclear to me. How do the authors classify questions into different difficulty levels, and what domain knowledge or criteria are used to determine these levels?
Questions for authors
Please check the weakness.