acceptodds
Withdrawn from ICLR 2027

Can AI Solve Imaging Problems? Benchmarking Multimodal AI for Image Formation and Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) and image-generation and editing models have demonstrated strong semantic visual capabilities, but whether they can reason about the physics, sensing mechanisms, and inverse problems underlying imaging remains largely unexplored. We introduce ImagingBench, a benchmark of 22 imaging tasks across five categories: ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration. ImagingBench evaluates multimodal models in three complementary settings: Expert, inverse reconstruction with fixed expert guidance; Planner, inverse reconstruction with model-generated planning; and Forward, forward image-formation simulation. Given a physics-based image-formation model, input measurements, and task context, models directly generate the target image without external code execution or task-specific optimization, probing their native multimodal imaging capabilities. We evaluate leading proprietary and open-source models against representative task-specific baselines. Across tasks, general-purpose multimodal models lag specialized methods, especially on physics-intensive problems such as lensless imaging, event reconstruction, and holography. Planning provides only modest, inconsistent gains, while visually plausible outputs often remain poorly faithful to the reference, revealing a gap between semantic competence and physically grounded imaging. ImagingBench provides a unified framework for quantifying this gap and tracking progress toward multimodal models that reason about imaging physics and inverse problems beyond visual appearance alone. The dataset, code and leaderboard will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
Computational Imaging, Benchmark, Inverse Problems, Physics-Based Forward Modeling, Image Editing Models, Vision-Language Models, Image Restoration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YVJ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for evaluating multimodal AI on imaging problems, covering 22 subtasks, two of which produce text outputs. It is the first benchmark to jointly evaluate physical image formation, inverse reconstruction, calibration and design, and planner–executor behavior within a unified framework. The benchmark employs three evaluation protocols and evaluates six image-editing models.

Critical strengths
  1. The benchmark covers 22 imaging tasks, including inverse reconstruction, providing comprehensive coverage of computational imaging problems.

  2. The paper is clearly written and well-organized.

Critical weaknesses
  1. Lack of explicit evaluation of physics reasoning capabilities. The paper emphasizes evaluating whether multimodal models can reason about imaging physics and inverse problems beyond visual appearance alone. However, the proposed Planner and Forward protocols do not explicitly verify these capabilities. It remains unclear whether the models actually understand and utilize the underlying physical principles.

  2. Limited evaluation of planning capabilities. The comparison between the Planner and Expert settings is relatively simplistic. The Planner is essentially a single VLM call without iterative refinement or self-correction, rather than an agent capable of adapting its strategy based on feedback. Therefore, the results mainly reflect the overall benefit of adding a Planner to the pipeline, rather than directly measuring the Planner's physics reasoning capabilities. The limited improvement of the Planner setting may not be sufficient to draw broader conclusions about the effectiveness of reasoning or planning.

  3. Lack of physics-based evaluation metrics. The evaluation mainly relies on image quality metrics and a VLM-based judge. Although the authors also conduct a human evaluation study, these metrics do not directly measure whether the reconstructed images are consistent with the underlying physical imaging models. This makes the evaluation less convincing in terms of assessing physics understanding.

  4. Unclear evaluation of forward noise simulation. The paper does not clearly explain the noise model used in the Forward simulation experiments. If the noise is stochastic, even a model that correctly understands the noise formation process may not reproduce the exact noise realization in the reference image. Therefore, directly comparing the generated and reference images using full-reference metrics may not accurately reflect the model's understanding of the forward imaging process.

  5. Insufficient control of variables in the Planner–Editor setup. The paper uses different combinations of Planner and Editor models without fully explaining the rationale behind these choices. This makes it difficult to distinguish the contribution of each component. For example, if Gemini paired with Gemini performs better than GPT-5 paired with Hunyuan, it is unclear whether the improvement comes from a stronger Planner, a stronger Editor, or better prompt compatibility between the two. A more controlled comparison would help clarify these factors.

  6. Unclear definitions of Physics and Reasoning in Table 1. The paper does not provide sufficiently clear or operational definitions of the "Physics" and "Reasoning" dimensions used to compare existing benchmarks. It would be helpful to clarify the criteria used to determine whether a benchmark covers these capabilities.

  7. Potential data contamination. Some benchmark tasks are constructed using publicly available datasets that may have been included in the training data of the evaluated models. The paper does not sufficiently discuss this potential overlap and its implications for the reliability of the evaluation.

  8. Minor presentation issue. The red and green cell backgrounds in Table 2 are not explained in the caption. It would be helpful to clarify the meaning of the heatmap colors and how they are normalized, rather than leaving readers to infer their meaning.

Questions for authors

See weakness.

Reviewer_seA6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ImagingBench, a benchmark to assess vision-language models on computational imaging tasks instead of only semantic vision problems, motivated by the physical inadequacy of existing benchmarks. It covers 22 subtasks across five imaging categories and three evaluation protocols, with multiple quantitative metrics and a VLM judge. The core contribution is a unified zero-shot diagnostic testbed for physical imaging, inverse reconstruction and planner-executor behavior. The authors test six image models and corresponding planners. Results show current models can produce plausible images yet lack fundamental low-level imaging ability, and this weakness stems not from prompting but insufficient imaging knowledge.

Critical strengths

1.ImagingBench covers 22 subtasks across 5 categories: ray/wave optics, ISP, inverse reconstruction, computational sensing, and calibration. It looks substantially broader than most task-specific imaging benchmarks and spans both reconstruction and non-reconstruction tasks such as lens design and camera calibration. 2.The paper benchmarks proprietary and open-source image-centric multimodal systems, including Gemini, GPT and Qwen, and compares them with task-specific non-agentic baselines where available. This is likely to be of interest to the community.

Critical weaknesses

1.Since the paper tries to answer a fairly specific question "Do current agentic VLMs truly understand imaging", the proposed diagnostic benchmark provides 22 subtasks in 5 categories. Ideally, such a benchmark should bring values to follow-up research: localize bottlenecks and guide future method development. The problem is that the paper mainly stops at model rankings with coarse conclusions. It does not really deliver these values or show how to obtain such values with the designed subtasks. Without mechanistic analysis, the conclusions listed in the paper read more like an experimental report and it looks insufficient to support a top-venue scientific contribution. 2.The paper uses general image-editing models as executors, with outputs mostly as 1024×1024 images. But depth, phase, raw ToF measurements, and lens prescriptions in computational imaging are not ordinary RGB image-editing problems. Model failures may come from output-interface mismatch, resolution constraints, image-prior bias, or optimization difficulty, rather than from “not understanding the forward operator”. The paper does not rule out these alternative explanations, so the causal attribution to “lack of physical understanding” is too strong. 3.As claimed in the paper, computational imaging is harder, but it does not systematically analyze whether this is due to operator non-invertibility, noise models, phase recovery, sparse sampling, or output representation. Without a failure taxonomy, the benchmark only tells “it is worse” but not “why it is worse or how to fix it”. This greatly limits the guidance value for follow-up development.

Questions for authors

See the weakness part.

Reviewer_erGyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ImagingBench, a benchmark designed to evaluate general-purpose VLMs and multimodal image editors on 22 computational imaging tasks. The benchmark evaluates six proprietary and open-weight image-editing models without external code execution across three protocols: inverse reconstruction with a fixed expert prompt (Expert), inverse reconstruction with an instance-specific VLM-generated prompt (Planner), and forward degradation synthesis (Forward). Performance is aggregated into a unified score combining four full-reference metrics, four no-reference metrics, and a VLM judge. The empirical results demonstrate that while frontier image editors achieve reasonable perceptual quality on common photographic restorations (such as denoising and inpainting), they fail on physics-intensive inverse problems (such as lensless imaging, holography, and event reconstruction) and gain little to no benefit from VLM-generated planning prompts.

Critical strengths
  • The benchmark spans a comprehensive set of 22 tasks across the computational imaging pipeline, thoughtfully combining established real-world scientific datasets, newly captured camera calibration and lens distortion data, and controlled synthetic degradation sweeps.
  • The qualitative and quantitative failure analyses are useful, particularly in exposing how VLM planners degrade downstream execution (e.g., causing GPT-Image-2 and FLUX2 to output grayscale images on demosaicking tasks because the generated plans focus on algorithmic jargon while omitting basic color requirements) and how generative priors hallucinate unrelated scenes/alter legible text.
Critical weaknesses
  • Asking general image-editing models to directly solve complex physics problems (e.g., generating holograms or estimating depth from raw sensor phases) purely by generating pixels (i.e., without letting them write code or use tools) is an unrealistic test. These models are trained to edit everyday photos, not to act as physics calculators, so it is unsurprising that they fail on raw sensor data.
  • The formula used to calculate the unified score breaks down on the hardest tasks. Specifically, the authors rescale PSNR using (PSNR−15)/25 and cut off anything below zero, which means any output with a PSNR under 15 dB gets a flat 0 for that metric. Yet Tab. 2 shows that on the hardest tasks, almost every model scores between 4 dB and 14 dB, so the PSNR metric is completely ignored on the exact tasks where measuring accuracy matters most.
  • The benchmark relies heavily on a single closed-source model (Claude Sonnet 5) as a judge, giving it the largest weight (35.3%) in the final score. However, the authors' own sanity checks (Tab. 4) show that if you simply feed the unedited, degraded input image to the judge on lens distortion or super-resolution, the judge gives it a high score that often beats the actual restored outputs. Relying on a paid, closed-source API could also hurt long-term reproducibility.
  • The baseline comparisons and model setups are uneven. Only 9 of the 20 image tasks include a specialized baseline model, and some of those baselines are too outdated (e.g., ESRGAN was published at ECCV 2018) to support the claim that general AI beats specialized methods. In addition, HunyuanImage-3.0 is tested using a heavily compressed (INT8, 8-step distilled) third-party version rather than the full official model, which could cause failures on multi-image tasks and produce broken images.
Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.