- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BlameBench, which contains 2500 questions derived from six previous benchmarks using LLMs and visual details from problem solution trajectories. They verified and filtered the questions using source images and created seven categories. They tested 11 models on both BlameBench and original source datasets. They also used InternVL3.5 model checkpoints to compare scores before and after fine-tuning to provide pretraining-stage diagnosis.
Critical strengths
- The benchmark construction steps are explained in detail.
- The qualitative examples of the task are provided in the Appendix.
Critical weaknesses
- The aim of creating perception questions from the solution evidence is not clearly stated in the paper. Why is it important? What does it reveal that is different from the source benchmark?
- The problem with the results is that they lack explanation and diagnosis. Table 2, Table 3, and Table 4 provide numerical experimental results but do not explain what they mean, which makes the purpose of the study not clear.
- In general, the purpose of the study and contributions are not clear and are not seen in the paper.
Questions for authors
- The BlameBench contains 2500 questions from six previous benchmarks, which can cause the data leakage problem for current models. The authors checked for answer leakage for models used in question generation. How do the authors validate that data contamination does not affect the evaluated model performance results rather than question generation?
- The column names in Table 1 are not clear. For example, what are dual profiles? The detailed description should be provided in the caption, same as other column names like reasoning link.
- The aim of creating perception questions from the solution evidence is not clearly stated in the paper. Why is it important? What does it reveal that is different?
- More details in reference-answer validation are required, such as what the differences are between GPT-5.6 Sol and human experts' reviews. Did domain experts review every generated question and trajectory in the benchmark?
- As I understand it, the reference trajectories are selected from the model responses that have a correct final answer. How did you validate the correctness of trajectories since the model can make mistakes in multiple steps but reach the correct final answer?
- In Table 2, some models perform better on BB score than DS score at the same benchmark, and some models vice versa. How do you explain this? What does it mean when a model achieves a higher score on DS than BB, or BB than DS?
- In Table 3, we saw the benchmark score for perception tasks; however, these scores, without connection to the source benchmark, do not provide meaningful inference, as they directly derived from other benchmarks. I expected that this method could find the underlying reason of the failure cases, but it can't provide any conclusion like that. So, the aim of the benchmark becomes unclear.
- In Table 4, for some benchmarks, the scores from Post-SFT are lower than Base DS; what does it mean, and why are some of them higher? What should we infer from these numerical results in terms of benchmarks and downstream tasks?