- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark for evaluating multimodal AI on imaging problems, covering 22 subtasks, two of which produce text outputs. It is the first benchmark to jointly evaluate physical image formation, inverse reconstruction, calibration and design, and planner–executor behavior within a unified framework. The benchmark employs three evaluation protocols and evaluates six image-editing models.
Critical strengths
-
The benchmark covers 22 imaging tasks, including inverse reconstruction, providing comprehensive coverage of computational imaging problems.
-
The paper is clearly written and well-organized.
Critical weaknesses
-
Lack of explicit evaluation of physics reasoning capabilities. The paper emphasizes evaluating whether multimodal models can reason about imaging physics and inverse problems beyond visual appearance alone. However, the proposed Planner and Forward protocols do not explicitly verify these capabilities. It remains unclear whether the models actually understand and utilize the underlying physical principles.
-
Limited evaluation of planning capabilities. The comparison between the Planner and Expert settings is relatively simplistic. The Planner is essentially a single VLM call without iterative refinement or self-correction, rather than an agent capable of adapting its strategy based on feedback. Therefore, the results mainly reflect the overall benefit of adding a Planner to the pipeline, rather than directly measuring the Planner's physics reasoning capabilities. The limited improvement of the Planner setting may not be sufficient to draw broader conclusions about the effectiveness of reasoning or planning.
-
Lack of physics-based evaluation metrics. The evaluation mainly relies on image quality metrics and a VLM-based judge. Although the authors also conduct a human evaluation study, these metrics do not directly measure whether the reconstructed images are consistent with the underlying physical imaging models. This makes the evaluation less convincing in terms of assessing physics understanding.
-
Unclear evaluation of forward noise simulation. The paper does not clearly explain the noise model used in the Forward simulation experiments. If the noise is stochastic, even a model that correctly understands the noise formation process may not reproduce the exact noise realization in the reference image. Therefore, directly comparing the generated and reference images using full-reference metrics may not accurately reflect the model's understanding of the forward imaging process.
-
Insufficient control of variables in the Planner–Editor setup. The paper uses different combinations of Planner and Editor models without fully explaining the rationale behind these choices. This makes it difficult to distinguish the contribution of each component. For example, if Gemini paired with Gemini performs better than GPT-5 paired with Hunyuan, it is unclear whether the improvement comes from a stronger Planner, a stronger Editor, or better prompt compatibility between the two. A more controlled comparison would help clarify these factors.
-
Unclear definitions of Physics and Reasoning in Table 1. The paper does not provide sufficiently clear or operational definitions of the "Physics" and "Reasoning" dimensions used to compare existing benchmarks. It would be helpful to clarify the criteria used to determine whether a benchmark covers these capabilities.
-
Potential data contamination. Some benchmark tasks are constructed using publicly available datasets that may have been included in the training data of the evaluated models. The paper does not sufficiently discuss this potential overlap and its implications for the reliability of the evaluation.
-
Minor presentation issue. The red and green cell backgrounds in Table 2 are not explained in the caption. It would be helpful to clarify the meaning of the heatmap colors and how they are normalized, rather than leaving readers to infer their meaning.
Questions for authors
See weakness.