- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VDiff-Bench, a benchmark for fine-grained visual difference understanding, comprising 1,756 multiple-choice questions and 300 open-ended image pairs across ten change categories. The authors evaluate 12 MLLMs and identify substantial weaknesses in detecting subtle visual changes. They further propose Expert-guided Harness Evolution (EHE), which improves a frozen 8B model through tool and inference-harness optimization, and VDiff-Judge, an evidence-grounded evaluator for generated differences. The results are promising, but several methodological and evaluation concerns remain.
Critical strengths
S1. Meaningful and well-motivated task.
S2. Broad benchmark coverage.
S3. Promising improvements without model training.
Critical weaknesses
W1. Insufficient attribution of EHE's performance gains.
The initial harness already includes visual measurement tools and specialized prompts, achieving 80.0% development accuracy before evolution and 87.5% afterward (Appendix C.1). Only two modifications are accepted: measurement fallback and increased response budget. Without comparisons among direct prompting, the initial harness, and the evolved harness on the same held-out set, the contribution of expert-guided evolution remains unclear. The claimed ablation study primarily evaluates transferability rather than individual components.
W2. Unclear evaluation independence and potential overfitting.
EHE uses 160 training and 200 development questions, but Table 2 does not clearly distinguish held-out results from the full benchmark. Repeated development feedback may also introduce adaptive overfitting. Additionally, several low-level categories share the same original FFHQ images, raising concerns about source-image overlap across splits.
W3. Limited validation of VDiff-Judge.
The reported 97% human agreement is based on only 100 individual claims evaluated against one human annotator. This does not establish the reliability of complete-set precision, recall, F1, or model rankings. Moreover, VDiff-Judge uses a Qwen-family model to evaluate Qwen-generated outputs, introducing potential evaluator bias. The matching threshold of 0.72 also lacks sensitivity analysis.
W4. Transferability and comparison fairness are not sufficiently established.
The OmniDiff adaptation includes domain-specific prompts, deterministic lighting rules, and answer recovery, with blank outputs decreasing from 136 to 2. Their individual contributions remain unclear. Furthermore, comparing tool-augmented Qwen against directly prompted proprietary models does not isolate model capability from additional inference resources. Inference cost and statistical uncertainty are also insufficiently reported.
Questions for authors
Q1. How much of EHE's improvement comes from automated evolution rather than the initial tool-augmented harness? Could the authors provide controlled ablations on the same held-out set?
Q2. Does Table 2 include optimization examples, and are splits disjoint at the original-image level?
Q3. Can the authors validate VDiff-Judge using complete generated outputs, multiple human annotators, and different evaluator backbones?
6 comments
Thanks and Initial Response to Reviewer itz4 (2/3)
[W1 & Q1 — How much does EHE contribute? We add same-set comparisons and ablations of the initial harness, final EHE harness, and each accepted harness change to show that EHE's performance gain over initial harness is meaningful.]
- We thank the reviewer for this suggestion. We compare direct Qwen, the initial harness, and the final EHE harness on the same 1,396 evaluation questions, excluding all training and development questions. In the table below, we also present results of evaluating each accepted change separately.
- Results show that EHE improves accuracy from 79.87% to 82.45%, adding 36 correct answers (+2.58 percentage points) beyond H0. We have added these comparisons to Appendix C.1.5, Table 9 (https://openreview.net/pdf?id=PgwqK0EXNl#page=23).
- To further isolate the two accepted harness changes, Change 1 alone improves H0 by +2.44 points, while Change 2 alone improves H0 by +1.00 point. The final harness combines both changes and achieves 82.45%. These results directly quantify the measured contribution of each accepted change, and show the valid performance gain of EHE through the harness evolution process.
| Configuration | Correct / 1,396 | Accuracy (%) |
|---|---|---|
| Direct Qwen | 922 | 66.05 |
| Initial harness H0 | 1,115 | 79.87 |
| H0 + Change 1 | 1,149 | 82.31 |
| H0 + Change 2 | 1,129 | 80.87 |
| Final EHE | 1,151 | 82.45 |
- We would also like to clarify that the value of evolution is measured by improvement from accepted changes, rather than the number of accepted changes alone.
- Small numbers of retained changes also appear in previous work (e.g. Self-Harness [1], Section 4.3, Figure 5 shows 3 retained changes for MiniMax model). Therefore, we do not believe that this should be a weakness point of our approach.
[W2 & Q2 — MCQ results on held-out test set? We provide results on held-out evaluation data, showing consistent performance gain of EHE on data unused during optimization.]
- To address your concern, we additional report results on the 1,396 evaluation data excluding all 160 training and 200 development data in Appendix C.1.5, Table 9 (https://openreview.net/pdf?id=PgwqK0EXNl#page=23).
- On these evaluation questions, EHE improves over direct Qwen from 66.05% to 82.45% (+16.40 points). This shows that the observed gains extend beyond the questions used during optimization.
[W2 — Could source-image overlap explain EHE’s gains? We additionally exclude shared-source images and observe consistent improvements.]
- To directly address your concern about shared FFHQ originals, we additionally report results excluding evaluation questions whose images share a source with the train / dev data used in optimization. This removes 240 questions from the evaluation set, leaving 1,156 evaluation questions:
- As shown in the table below, after excluding shared-source questions, EHE still improves over direct Qwen by +15.31 points and over H0 by +2.85 points. Thus, the observed gains persist beyond both the optimization questions and their shared source images; source-image overlap does not explain away the improvement.
| Method | Correct / Questions | Accuracy (%) |
|---|---|---|
| Direct Qwen | 796 / 1,156 | 68.86 |
| Initial harness H0 | 940 / 1,156 | 81.31 |
| Final EHE | 973 / 1,156 | 84.17 |
- We also confirm that for the Gen task, none of the 300 evaluation pairs overlaps with the 32 training or 24 development pairs under the exact-image and documented-source-lineage checks. Therefore, observed EHE performance gain on Gen holds and already excluding potential influence of overlapping source images*.
Thanks and Initial Response to Reviewer itz4 (3/3)
[W3 & Q3 — Can we further validate VDiff-Judge? We extend our human study to using 3 annotators on 200 examples, confirming 95.5% human-judge alignment on majority-voted labels. Agreement remains 95.0% on non-Qwen-family model outputs, showing no evidence of evaluator bias.]
- We extend our human annotation study to 3 annotators on 200 generated claims, with balanced coverage across 7 category groups and 10 generator models.
- VDiff-Judge agrees with majority labels on 95.5% of examples (191/200; Cohen’s Kappa = 0.906), versus 85.0% for direct Qwen and 88.5% for GPT-5.6-Sol on the same sample. Additionally, inter-annotator agreement is high: Fleiss’ Kappa = 0.910 among the three annotators.
- We have updated Section 4.1 (https://openreview.net/pdf?id=PgwqK0EXNl#page=8) and Appendix D.3, Tables 10–11 (https://openreview.net/pdf?id=PgwqK0EXNl#page=26) to reflect these results.
- To address the reviewer's concern about bias towards Qwen-generated outputs, we additionally exclude claims from both Qwen3-VL-8B and LLaVA-OneVision-Qwen2-7B.
- Human-VDiff-Judge agreement remains 95.0% (152/160) across the other eight model outputs, showing no evidence for model family biases and constent high agreement extends beyond Qwen-generated claims.
[W4.1 — Which component contribute to OmniDiff transfer? Ablation results show gains from all components: answer recovery, specialized prompts, and lighting handling.]
- We evaluate the contributions of the different components of OmniDiff adaptation. Results in the table below show that all components contribute to the aggregated performance gain:
- Answer recovery contributes to the improvement, but does not explain the entire gain. Removing recovery reduces CIDEr/SPICE to 19.29/15.14, yet both remain above direct Qwen’s 12.54/12.94.
- Specialized prompts improve caption quality beyond producing nonempty answers. Generic prompts produce only one blank output, but achieve substantially lower CIDEr/SPICE than the full harness.
- Lighting handling provides a smaller additional contribution. Removing it lowers CIDEr from 23.83 to 23.06 and SPICE from 19.29 to 18.34, despite eliminating blank outputs.
| Configuration | CIDEr | SPICE | Blank outputs |
|---|---|---|---|
| Direct Qwen | 12.54 | 12.94 | 136 |
| Full harness | 23.83 | 19.29 | 2 |
| Without answer recovery | 19.29 | 15.14 | 230 |
| Generic prompts | 13.74 | 16.25 | 1 |
| Without lighting handling | 23.06 | 18.34 | 0 |
[W4.2 — Inference costs and statistical uncertainty of EHE? We report token usage and paired confidence intervals.]
- We have added deployment token usage to Appendix F.2, Table 13 (https://openreview.net/pdf?id=PgwqK0EXNl#page=32). The expert agent is used during optimization; the selected harness requires zero expert-agent calls at deployment.
| Configuration | Input tokens/example | Output tokens/example |
|---|---|---|
| MCQ: direct Qwen | 1,868 | 715 |
| MCQ: final EHE | 2,001 | 575 |
| Gen: direct Qwen | 2,014 | 5,109 |
| Gen: adapted EHE | 4,817 | 3,077 |
- We also added paired 95% confidence intervals from 10,000 bootstrap replicates in Appendix F.2, Table 14 (https://openreview.net/pdf?id=PgwqK0EXNl#page=32):
- As shown in the summary table below, all three confidence intervals exclude zero, supporting consistent improvements over direct prompting under evaluation-sample uncertainty.
| EHE improvement over direct Qwen | Gain (pp) | 95% CI (pp) |
|---|---|---|
| MCQ accuracy | +16.40 | [13.70, 19.15] |
| Gen micro-F1 | +26.91 | [22.12, 31.58] |
| Gen pair-macro F1 | +33.48 | [28.74, 38.10] |
Thanks and Initial Response to Reviewer itz4 (1/3)
We want to first thank the reviewer for recognizing our task’s meaningful motivation, our benchmark’s coverage, and the promising training-free improvements by EHE. To begin with, we summarize your main concerns and our additional evidence; changes in the updated paper PDF are highlighted in blue:
Below, we respond to each point and identify the corresponding paper updates.