acceptodds
Withdrawn from ICLR 2027

BlameBench: Diagnosing Atomic Perception in Multimodal Reasoning Benchmarks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models ground reasoning in fine-grained visual evidence, but final-answer accuracy conflates errors in perception and subsequent reasoning. We introduce **BlameBench**, a task-grounded benchmark that uses visual evidence from solution trajectories with correct final answers to decide what to test. We extract visual observations, verify them against source images, and convert the resulting facts into atomic perception questions that can be answered without solving the full source problem. This construction yields a fixed 2,500-question benchmark spanning six source benchmarks and seven reporting categories, including 2,058 direct-perception questions and 442 supplementary concept probes. Across 11 frontier models, source and category profiles reveal differences concealed by overall accuracy: Qwen 3.8 Max and Claude Opus 5 are separated by only 0.64 percentage points overall but lead on different sources and categories. The same questions also provide an upstream readout: across four InternVL3.5-Pretrained checkpoints, source-centered scores before fine-tuning correlate with downstream accuracy after fine-tuning (). **BlameBench** thus supports final-model perceptual diagnosis and pretraining-stage diagnosis.

Primary area:
datasets and benchmarks
Keywords:
Multimodal, Perception, Atomic Diagnose

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LPqsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BlameBench, which contains 2500 questions derived from six previous benchmarks using LLMs and visual details from problem solution trajectories. They verified and filtered the questions using source images and created seven categories. They tested 11 models on both BlameBench and original source datasets. They also used InternVL3.5 model checkpoints to compare scores before and after fine-tuning to provide pretraining-stage diagnosis.

Critical strengths
  • The benchmark construction steps are explained in detail.
  • The qualitative examples of the task are provided in the Appendix.
Critical weaknesses
  • The aim of creating perception questions from the solution evidence is not clearly stated in the paper. Why is it important? What does it reveal that is different from the source benchmark?
  • The problem with the results is that they lack explanation and diagnosis. Table 2, Table 3, and Table 4 provide numerical experimental results but do not explain what they mean, which makes the purpose of the study not clear.
  • In general, the purpose of the study and contributions are not clear and are not seen in the paper.
Questions for authors
  • The BlameBench contains 2500 questions from six previous benchmarks, which can cause the data leakage problem for current models. The authors checked for answer leakage for models used in question generation. How do the authors validate that data contamination does not affect the evaluated model performance results rather than question generation?
  • The column names in Table 1 are not clear. For example, what are dual profiles? The detailed description should be provided in the caption, same as other column names like reasoning link.
  • The aim of creating perception questions from the solution evidence is not clearly stated in the paper. Why is it important? What does it reveal that is different?
  • More details in reference-answer validation are required, such as what the differences are between GPT-5.6 Sol and human experts' reviews. Did domain experts review every generated question and trajectory in the benchmark?
  • As I understand it, the reference trajectories are selected from the model responses that have a correct final answer. How did you validate the correctness of trajectories since the model can make mistakes in multiple steps but reach the correct final answer?
  • In Table 2, some models perform better on BB score than DS score at the same benchmark, and some models vice versa. How do you explain this? What does it mean when a model achieves a higher score on DS than BB, or BB than DS?
  • In Table 3, we saw the benchmark score for perception tasks; however, these scores, without connection to the source benchmark, do not provide meaningful inference, as they directly derived from other benchmarks. I expected that this method could find the underlying reason of the failure cases, but it can't provide any conclusion like that. So, the aim of the benchmark becomes unclear.
  • In Table 4, for some benchmarks, the scores from Post-SFT are lower than Base DS; what does it mean, and why are some of them higher? What should we infer from these numerical results in terms of benchmarks and downstream tasks?
Reviewer_G8rAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BlameBench, a benchmark that constructs atomic perception questions from visual observations in successful multimodal reasoning trajectories. The benchmark contains 2,500 questions from six source benchmarks, covering seven reporting categories. Experiments on 11 frontier models reveal performance differences across sources and categories. The authors further investigate the relationship between BlameBench scores of pretrained models and their downstream performance after fine-tuning.

Critical strengths
  1. The paper addresses the important problem of distinguishing visual perception from downstream reasoning. Constructing atomic perception questions from successful solution trajectories provides a distinctive approach while preserving traceability to the original tasks.

  2. The multi-stage screening and human validation procedures contribute to benchmark quality and reliability. The extensive experiments across diverse benchmarks and models provide a solid basis for analyzing fine-grained perceptual capabilities.

Critical weaknesses
  1. The experiments demonstrate differences in atomic perception accuracy but do not establish whether these differences identify the actual causes of errors in downstream reasoning tasks. Additional experiments linking perceptual failures to errors in the original tasks would strengthen the central diagnostic claim.

  2. The paper lacks comparisons with simpler alternatives, such as generating perception questions directly from images. It remains unclear whether extracting observations from successful trajectories provides a substantial advantage. An ablation study comparing different question construction strategies would help establish the necessity and effectiveness of the proposed approach.

Questions for authors
  1. Could the reported pretraining-stage correlations be largely explained by model scale rather than the independent diagnostic value of BlameBench?

  2. Given the uneven category distributions across source benchmarks, how independent are the source-level and category-level diagnostic results?

  3. How does screening questions using a specific model affect the representativeness and generalizability of the benchmark?

  4. Could the authors provide additional evidence supporting the reliability of image-dependence screening and automated answer evaluation?

Reviewer_rvxsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BlameBench, which is built from existing benchmarks to evaluate atomic visual perception capabilities of MLLMs. It is constructed by leveraging multiple frontier MLLMs to extract visual observations from solution trajectories, verify their consistency with the original images, and transform individual visual facts into open-ended, short-answer questions. This paper evaluates 11 frontier MLLMs on BlameBench and conducts a pretraining-stage diagnosis experiments, demonstrating the better correlation between BlameBench performance and downstream performance after fine-tuning.

Critical strengths
  1. This research is well-motivated. It focuses on the diagnosis of atomic visual perception of MLLMs, which I think is still a major bottleneck in current MLLM reasoning.

  2. The method of decomposing coarse-grained problem-solving accuracy into independently testable atomic perceptual correctness facilitates a more granular diagnosis of the capabilities of MLLMs.

  3. The pretraining-stage diagnosis experiment is interesting and demonstrates the benchmark’s potential for diagnosing the visual perception capabilities of MLLMs during pretraining.

Critical weaknesses
  1. The rationale for using different MLLMs at different stages of benchmark construction remains unclear. The paper does not explain the advantages of a multi-MLLM pipeline over a single-MLLM pipeline, nor whether these choices are motivated by cost constraints or by stage-specific performance.

  2. The quality control process is insufficiently explained. The paper only reports the number of retained examples at each stage but does not illustrate the filtering criteria or error rates, especially for Question Construction and Knowledge Construction.

  3. The pretraining-stage diagnosis experiment is limited to InternVL3.5 and a single SFT dataset, which limits the strength and generalizability of the result.

  4. How should we understand the fact that in Table 1, BB performs lower than DS on some benchmarks and higher on others? This seems to indicate that BB cannot reliably reveal the performance relationship between atomic visual perception and downstream complete tasks. Why is ZeroBench’s BB score currently above 90% while its DS score is only about 20%? Is this because there are higher-order visual capabilities not covered by BlameBench that the model has not mastered, or are all the errors in the logical reasoning part after multimodal perception?

Questions for authors

See Weaknesses 1 & 2 & 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.