acceptodds
Under review as a conference paper at ICLR 2027

VDiff-Bench: Diagnosing and Improving Fine-Grained Visual Difference Understanding

Abstract

Multimodal large language models (MLLMs) achieve strong performance on general visual understanding tasks, yet reliably identifying differences between similar images remains challenging. We introduce VDiff-Bench, a diagnostic benchmark that connects controlled difference identification with open-ended difference discovery through two tracks: VDiff-Bench-MCQ, which consists of 1,756 multiple-choice questions across 10 semantic, textual, and low-level change categories; and VDiff-Bench-Gen, with 300 challenging image pairs for open-ended image difference generation. Evaluation of 12 MLLMs on the MCQ track reveals pronounced category-specific weaknesses, with small open-weight models particularly struggling on low-level changes. To address these failures without updating model weights, we propose Expert-guided Harness Evolution (EHE): an expert coding agent uses traces and evaluation feedback to revise the tools, routing, and response handling around a small base agent, successfully improving its MCQ performance by 26.9%, comparable with the GPT-5.4 model. To better evaluate open-ended image difference captioning, we further introduce VDiff-Judge, a tool-augmented agentic evaluator that judges generated differences using targeted visual evidence, forming verifiable traces for assessing correctness and completeness. Even the strongest reported proprietary model achieves only 46.8% micro-F1 on the Gen track. EHE improves Qwen3-VL-8B-Thinking from 65.7% to 83.4% MCQ accuracy. Notably, EHE's performance gain is transferrable to different data and different task: adapting EHE to the Gen track improves micro-F1 by 155.5%, approaching proprietary performance; adapting EHE to a different benchmark also yields over 90% improvement.

Primary area:
datasets and benchmarks
Keywords:
harness, multimodal large language models, image difference captioning, training-free agent improvement, tool-augmented reasoning, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_itz4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VDiff-Bench, a benchmark for fine-grained visual difference understanding, comprising 1,756 multiple-choice questions and 300 open-ended image pairs across ten change categories. The authors evaluate 12 MLLMs and identify substantial weaknesses in detecting subtle visual changes. They further propose Expert-guided Harness Evolution (EHE), which improves a frozen 8B model through tool and inference-harness optimization, and VDiff-Judge, an evidence-grounded evaluator for generated differences. The results are promising, but several methodological and evaluation concerns remain.

Critical strengths

S1. Meaningful and well-motivated task.

S2. Broad benchmark coverage.

S3. Promising improvements without model training.

Critical weaknesses

W1. Insufficient attribution of EHE's performance gains.

The initial harness already includes visual measurement tools and specialized prompts, achieving 80.0% development accuracy before evolution and 87.5% afterward (Appendix C.1). Only two modifications are accepted: measurement fallback and increased response budget. Without comparisons among direct prompting, the initial harness, and the evolved harness on the same held-out set, the contribution of expert-guided evolution remains unclear. The claimed ablation study primarily evaluates transferability rather than individual components.

W2. Unclear evaluation independence and potential overfitting.

EHE uses 160 training and 200 development questions, but Table 2 does not clearly distinguish held-out results from the full benchmark. Repeated development feedback may also introduce adaptive overfitting. Additionally, several low-level categories share the same original FFHQ images, raising concerns about source-image overlap across splits.

W3. Limited validation of VDiff-Judge.

The reported 97% human agreement is based on only 100 individual claims evaluated against one human annotator. This does not establish the reliability of complete-set precision, recall, F1, or model rankings. Moreover, VDiff-Judge uses a Qwen-family model to evaluate Qwen-generated outputs, introducing potential evaluator bias. The matching threshold of 0.72 also lacks sensitivity analysis.

W4. Transferability and comparison fairness are not sufficiently established.

The OmniDiff adaptation includes domain-specific prompts, deterministic lighting rules, and answer recovery, with blank outputs decreasing from 136 to 2. Their individual contributions remain unclear. Furthermore, comparing tool-augmented Qwen against directly prompted proprietary models does not isolate model capability from additional inference resources. Inference cost and statistical uncertainty are also insufficiently reported.

Questions for authors

Q1. How much of EHE's improvement comes from automated evolution rather than the initial tool-augmented harness? Could the authors provide controlled ablations on the same held-out set?

Q2. Does Table 2 include optimization examples, and are splits disjoint at the original-image level?

Q3. Can the authors validate VDiff-Judge using complete generated outputs, multiple human annotators, and different evaluator backbones?

6 comments
AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer itz4 (1/3)

We want to first thank the reviewer for recognizing our task’s meaningful motivation, our benchmark’s coverage, and the promising training-free improvements by EHE. To begin with, we summarize your main concerns and our additional evidence; changes in the updated paper PDF are highlighted in blue:

  1. Question on EHE’s contribution beyond the initial harness.
  • We report additional results comparing direct prompting, the initial harness, EHE, and component ablations on evaluation data (excluding the training and development data). Results show that EHE selects helpful harness changes and improves accuracy beyond H0. We added these comparisons in Appendix C.1.5, Table 9.
  1. Questions on evaluation data independence and the MCQ reporting population.
  • We clarify that the main MCQ table reports the full benchmark. As we mentioned in the previous point, we provide additional results excluding all optimization questions in Appendix C.1.5, Table 9. EHE improves direct Qwen from 66.05% to 82.45% on these questions.
  1. VDiff-Judge reliability and possible bias toward Qwen-generated outputs.
  • We extended the human study to 3 annotators on 200 claims, further showing high human–VDiff-Judge agreement (95.5% alignment with majority votes). We updated Section 4.1 and Appendix D.3, Tables 10–11.
  • We additionally exclude Qwen-family-generated claims and find 95.0% agreement across the remaining eight generators, showing no model-induced preference biases.
  1. Individual contributions to OmniDiff transfer, inference costs, and statistical uncertainty.
  • We provide additional OmniDiff ablation experiment results of answer recovery, specialized prompts, and lighting handling to examine their individual contributions, showing that while each components bring positive metric gains, the full EHE harness with all components achieves best results.
  • We added inference token usage and paired confidence intervals in Appendix F.2, Tables 13–14.

Below, we respond to each point and identify the corresponding paper updates.

AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer itz4 (2/3)

[W1 & Q1 — How much does EHE contribute? We add same-set comparisons and ablations of the initial harness, final EHE harness, and each accepted harness change to show that EHE's performance gain over initial harness is meaningful.]

  • We thank the reviewer for this suggestion. We compare direct Qwen, the initial harness, and the final EHE harness on the same 1,396 evaluation questions, excluding all training and development questions. In the table below, we also present results of evaluating each accepted change separately.
    • Results show that EHE improves accuracy from 79.87% to 82.45%, adding 36 correct answers (+2.58 percentage points) beyond H0. We have added these comparisons to Appendix C.1.5, Table 9 (https://openreview.net/pdf?id=PgwqK0EXNl#page=23).
    • To further isolate the two accepted harness changes, Change 1 alone improves H0 by +2.44 points, while Change 2 alone improves H0 by +1.00 point. The final harness combines both changes and achieves 82.45%. These results directly quantify the measured contribution of each accepted change, and show the valid performance gain of EHE through the harness evolution process.
ConfigurationCorrect / 1,396Accuracy (%)
Direct Qwen92266.05
Initial harness H01,11579.87
H0 + Change 11,14982.31
H0 + Change 21,12980.87
Final EHE1,15182.45
  • We would also like to clarify that the value of evolution is measured by improvement from accepted changes, rather than the number of accepted changes alone.
    • Small numbers of retained changes also appear in previous work (e.g. Self-Harness [1], Section 4.3, Figure 5 shows 3 retained changes for MiniMax model). Therefore, we do not believe that this should be a weakness point of our approach.

[W2 & Q2 — MCQ results on held-out test set? We provide results on held-out evaluation data, showing consistent performance gain of EHE on data unused during optimization.]

  • To address your concern, we additional report results on the 1,396 evaluation data excluding all 160 training and 200 development data in Appendix C.1.5, Table 9 (https://openreview.net/pdf?id=PgwqK0EXNl#page=23).
    • On these evaluation questions, EHE improves over direct Qwen from 66.05% to 82.45% (+16.40 points). This shows that the observed gains extend beyond the questions used during optimization.

[W2 — Could source-image overlap explain EHE’s gains? We additionally exclude shared-source images and observe consistent improvements.]

  • To directly address your concern about shared FFHQ originals, we additionally report results excluding evaluation questions whose images share a source with the train / dev data used in optimization. This removes 240 questions from the evaluation set, leaving 1,156 evaluation questions:
    • As shown in the table below, after excluding shared-source questions, EHE still improves over direct Qwen by +15.31 points and over H0 by +2.85 points. Thus, the observed gains persist beyond both the optimization questions and their shared source images; source-image overlap does not explain away the improvement.
MethodCorrect / QuestionsAccuracy (%)
Direct Qwen796 / 1,15668.86
Initial harness H0940 / 1,15681.31
Final EHE973 / 1,15684.17
  • We also confirm that for the Gen task, none of the 300 evaluation pairs overlaps with the 32 training or 24 development pairs under the exact-image and documented-source-lineage checks. Therefore, observed EHE performance gain on Gen holds and already excluding potential influence of overlapping source images*.
AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer itz4 (3/3)

[W3 & Q3 — Can we further validate VDiff-Judge? We extend our human study to using 3 annotators on 200 examples, confirming 95.5% human-judge alignment on majority-voted labels. Agreement remains 95.0% on non-Qwen-family model outputs, showing no evidence of evaluator bias.]

  • We extend our human annotation study to 3 annotators on 200 generated claims, with balanced coverage across 7 category groups and 10 generator models.
  • To address the reviewer's concern about bias towards Qwen-generated outputs, we additionally exclude claims from both Qwen3-VL-8B and LLaVA-OneVision-Qwen2-7B.
    • Human-VDiff-Judge agreement remains 95.0% (152/160) across the other eight model outputs, showing no evidence for model family biases and constent high agreement extends beyond Qwen-generated claims.

[W4.1 — Which component contribute to OmniDiff transfer? Ablation results show gains from all components: answer recovery, specialized prompts, and lighting handling.]

  • We evaluate the contributions of the different components of OmniDiff adaptation. Results in the table below show that all components contribute to the aggregated performance gain:
    • Answer recovery contributes to the improvement, but does not explain the entire gain. Removing recovery reduces CIDEr/SPICE to 19.29/15.14, yet both remain above direct Qwen’s 12.54/12.94.
    • Specialized prompts improve caption quality beyond producing nonempty answers. Generic prompts produce only one blank output, but achieve substantially lower CIDEr/SPICE than the full harness.
    • Lighting handling provides a smaller additional contribution. Removing it lowers CIDEr from 23.83 to 23.06 and SPICE from 19.29 to 18.34, despite eliminating blank outputs.
ConfigurationCIDErSPICEBlank outputs
Direct Qwen12.5412.94136
Full harness23.8319.292
Without answer recovery19.2915.14230
Generic prompts13.7416.251
Without lighting handling23.0618.340

[W4.2 — Inference costs and statistical uncertainty of EHE? We report token usage and paired confidence intervals.]

ConfigurationInput tokens/exampleOutput tokens/example
MCQ: direct Qwen1,868715
MCQ: final EHE2,001575
Gen: direct Qwen2,0145,109
Gen: adapted EHE4,8173,077
  • We also added paired 95% confidence intervals from 10,000 bootstrap replicates in Appendix F.2, Table 14 (https://openreview.net/pdf?id=PgwqK0EXNl#page=32):
    • As shown in the summary table below, all three confidence intervals exclude zero, supporting consistent improvements over direct prompting under evaluation-sample uncertainty.
EHE improvement over direct QwenGain (pp)95% CI (pp)
MCQ accuracy+16.40[13.70, 19.15]
Gen micro-F1+26.91[22.12, 31.58]
Gen pair-macro F1+33.48[28.74, 38.10]

We believe the contributions of VDiff-Bench and VDiff-Judge were established in our submission, and sincerely hope the additional comparisons, ablations, and human validation address your remaining concerns. We would appreciate it if the reviewer could adjust the rating for our paper in light of this evidence. Thank you!

Reviewer_MBC9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VDiff-Bench, a diagnostic benchmark tailored for visual difference understanding, comprising two primary tasks: VDiff-Bench-MCQ (multiple-choice questions across 10 change categories) and VDiff-Bench-Gen (open-ended difference caption generation). Motivated by the noticeable struggle of existing MLLMs on the MCQ benchmark, the authors propose Expert-guided Harness Evolution (EHE), a framework designed to boost the performance of smaller MLLMs on difference reasoning. Furthermore, to reliably assess open-ended difference captions, the paper presents VDiff-Judge, a tool-augmented agentic evaluator built upon Qwen3-VL-8B, which demonstrates high agreement with human evaluations on a 100-sample test suite.

Critical strengths
  1. The dataset and evaluation protocols are well-organized, offering a clear taxonomy of change types that allows for a thorough and diagnostic assessment of a model's visual difference comprehension.
  2. The proposed VDiff-Judge is well-motivated, effectively targeting the underlying visual and semantic logic of difference captioning rather than relying on conventional, superficial lexical similarity metrics.
Critical weaknesses
  1. Effectiveness and contribution of EHE are unclear: As shown in Table 7, the initial harness has already yielded 80% accuracy on development data, and only two of the expert's proposed candidates are accepted for harness improvement. This is unable to show the effectiveness of the expert-guided evolution. Moreover, the novelty of EHE is unclear; the evaluation loop closely follows Meta-Harness[1] and Self-Harness[2], both of which the paper cites, but none of them are compared experimentally.
  2. Concern about the reliability of VDiff-Judge: Although the evaluation process of VDiff-Judge is reasonable, it is only validated on a 100-sample test set with 1 human annotator. It would be better to provide more analysis to validate the reliability of the proposed evaluation process.
  3. Concern about leakage of the development set: The result for EHE in Table 2 does not clarify whether the test set is excluded from the development set, diminishing the persuasiveness of the performance result of EHE.

[1] Lee, Yoonho, et al. "Meta-harness: End-to-end optimization of model harnesses." arXiv preprint arXiv:2603.28052 (2026). [2] Zhang, Hangfan, et al. "Self-harness: Harnesses that improve themselves." arXiv preprint arXiv:2606.09498 (2026).

Questions for authors
  1. Please clarify and compare the differences between EHE and the cited harness methods.
  2. Please provide the MCQ performance on Qwen-3-VL-8B without EHE and with H0H_0 on the test set.
  3. Please clarify the dataset setting of MCQ in Table 2.
3 comments
AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer MBC9 (1/3)

We want to first thank the reviewer for recognizing our benchmark’s diagnostic coverage and VDiff-Judge’s meaningful motivation. To begin with, we summarize your main concerns and how we addressed them; we have also updated the paper PDF with changes highlighted in blue:

  1. Question on EHE’s contribution beyond the initial harness.
  • We compare results of direct prompting, initial harness, final EHE-optimized harness, and component ablations on evaluation data that are not used during optimization (excluding train and dev splits). Results show that EHE achieves meaningful performance gain through selecting helpful harness changes. We add this in Appendix C.1.5, Table 9.
  1. Comparison with Meta-Harness and Self-Harness.
  • We added Meta-Harness results in Table 3, showing that EHE outperforms Meta-Harness.
  • We also tried to implement Self-Harness, but the Qwen model is not strong enough to modify its own harness, causing persistent syntax errors and didn't yield outputs.
  1. VDiff-Judge reliability through more human study.
  • We extended the human annotation study to 3 human annotators on 200 claims, further showing high human–VDiff-Judge agreement (95.5% alignment with majority votes). We updated results in Section 4.1.
  1. Could there be any development / test data leak?
  • No, we confirm that test / evaluation set is excluded from the development set and the train set for both MCQ and Gen data. We added further clarification in Section 4.2 and Appendix C.2.

Below, we provide detailed responses and point you to the corresponding changes in the paper.

AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer MBC9 (2/3)

[W1 & Q2 — What improvement does EHE bring beyond H0? We provide comparisons and component ablations, showing that EHE chose harness change candidates that resulted in performance improvement beyond H0.]

  • We thank the reviewer for this suggestion. We compare direct Qwen, the initial harness, and the final EHE harness on the same evaluation questions, excluding all training and development data. In the table below, we also present results of evaluating each accepted change separately.

    • Results show that EHE improves accuracy from 79.87% to 82.45%, adding 36 correct answers (+2.58 percentage points) beyond H0. We have added these comparisons to Appendix C.1.5, Table 9 (https://openreview.net/pdf?id=PgwqK0EXNl#page=23). | Configuration | Correct / 1,396 | Accuracy (%) | | --- | ---: | ---: | | Direct Qwen | 922 | 66.05 | | Initial harness H | 1,115 | 79.87 | | H0 + Change 1 | 1,149 | 82.31 | | H0 + Change 2 | 1,129 | 80.87 | | Final EHE | 1,151 | 82.45 |
  • To further isolate the two accepted harness changes, change 1 alone improves H0 by +2.44 points, while the change 2 alone improves H0 by +1.00 point. The final harness combines both changes and achieves 82.45%. This shows that both accepted changes bring meaningful performance gains toward the final EHE-optimized harness.

  • We would also like to clarify that the value of evolution is measured by the improvement from accepted changes, rather than the number of accepted changes alone.

    • Small numbers of retained changes also appear in the previous work Self-Harness [1] (Section 4.3, Figure 5: only 3 for MiniMax). As our additional ablations directly quantify how much the two accepted changes contribute, we believe that this should not be a weakness point of our approach.

[Q1 — How is EHE different from Meta-Harness and Self-Harness? We clarify our difference to the two previous methods below.]

  • Summary of differences between EHE vs. Meta-Harness and Self-Harness:
    • Meta-Harness [2] searches over all harness code using prior traces and scores and directly propose new ones.
      • EHE do this step-by-step, proposing one amendment to the previous best harness code at a step. Each newly proposed change undergo overall and category regression checks on dev, and changes that caused regression are rejected, reverting back to the previous best harness.
      • EHE's design is specifically curated for building harness surrounding smaller models: radical harness replacement and directly aggregating all previous harness codes might cause performance fluctuations in small models like the 8B Qwen model, as the model is less capable. Nevertheless, we believe that Meta-Harness's design is most effective on bigger and stronger target models.
    • Self-Harness [1] uses the target model itself to propose changes.
      • As inferred by the method name, Expert-guided Harness Evolution, EHE uses an expert agent to revise the harness around a small target model.

[W1 — How does EHE compare to Meta-Harness and Self-Harness? We add comparison experiments to show that (1) EHE outperforms Meta-Harness on our task, and (2) the 8B Qwen model fails to self-harness.]

  • We implemented Meta-Harness to optimize the harness on the same target model on our MCQ questions, and allow Meta-Harness to propose the same number of harness candidates as EHE during experiment.
    • We chose MCQ because its question format (answer choices) maps naturally to Meta-Harness's released text classification workflow example, allowing for direct transfer of Meta-Harness's design to our data.
    • Experiment results show that EHE out-performs Meta-Harness on our data: Meta-Harness achieves 76.9% MCQ accuracy, compared with 83.4% for EHE. We have added Meta-Harness' results to Table 3 (https://openreview.net/pdf?id=PgwqK0EXNl#page=5).
  • We also experimented with Self-Harness using the target model Qwen3-VL-8B Thinking as the proposer. However, none of the attempted harness proposals by the model are executable: response either contain repetitive reasonings and therefore truncated, or contain invalid string quoting persisted after syntax-error feedback.
    • The Self-Harness run therefore did not reach performance evaluation. We observe that the reason behind Self-Harness' failure on our task setting is that the small Qwen model itself is not strong enough to analyze training observations and propose valid new harness. We believe that Self-Harness will work significantly better if applied on a stronger and more capable target mode.
AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer MBC9 (3/3)

[W2 — Is VDiff-Judge sufficiently validated? We expanded our human study to 200 claims with 3 annotators, proving VDiff-Judge's validity through achieving 95.5% human-judge alignment on majority-voted labels.]

  • We extend our human annotation study to 3 annotators on 200 generated claims, with balanced coverage across 7 data category and 10 generator models.
  • VDiff-Judge agrees with majority labels on 95.5% of examples (191/200; Cohen’s Kappa = 0.906), versus 85.0% for direct Qwen and 88.5% for GPT-5.6-Sol on the same sample. Additionally, Inter-annotator agreement is high: Fleiss’ Kappa = 0.910 among the three annotators.**

[W3 & Q3 — Does the main MCQ table include optimization examples? We clarify the data population and provide additional results to show that EHE also achieves performance gain on held-out evaluation data unused during optimization.]

We believe the contributions of VDiff-Bench and VDiff-Judge were established in our submission, and sincerely hope the additional comparisons, ablations, and human validation address your remaining concerns. We would sincerely appreciate it if the reviewer could consider adjust the rating for our paper accordingly. Thank you!

Reviewer_26frOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VDiff-Bench for image difference identification (IDI), with an MCQ track (1,756 questions, 10 change categories) and an open-ended generation track (300 pairs). Evaluating 12 MLLMs reveals category-specific weaknesses: open-weight 7–8B models and Grok are strong on semantic changes but collapse on low-level changes (noise, texture). The authors propose EHE (Expert-guided Harness Evolution), where an expert coding agent iteratively rewrites the tools/routing around a frozen Qwen3-VL-8B model using trace feedback, raising MCQ accuracy from 65.7% to 83.4%. A tool-augmented evaluator, VDIFF-JUDGE, scores the generation track (97% agreement with human judgments, κ=0.937); the best proprietary model reaches only 46.8% micro-F1.

Critical strengths
  1. The two-track design is well judged: deterministic MCQ scoring for diagnosis, and an evidence-grounded agentic judge with retained verification traces for open-ended output — a more defensible evaluation stack than n-gram or ungrounded LLM judging, and the 97%/κ=0.937 human agreement is convincing on the sampled claims.
  2. The failure analysis is concrete and actionable: the “no difference” distractor capture on 48.1–80.9% of low-level questions, and the recall collapse on multi-change pairs (Gemini 3.5 Flash 77.3%→49.9%), give the community specific targets.
Critical weaknesses
  1. Attribution of the EHE gain. The largest EHE improvements are exactly on categories constructed programmatically (noise 8.0→95.3, illumination 71.8→95.9), where harness-side measurement can substitute for model perception. The paper’s claim of “strengthening a small model’s fine-grained visual understanding” is stronger than the evidence; “a tool stack around a frozen 8B model solves low-level IDI” is what the tables show.
  2. Benchmark composition transparency: the split between programmatic transformations, model-edited pairs, and human-annotated pairs per category is relegated to appendices, so the reader cannot judge weakness #1 from the main text.
  3. EHE’s optimization data (160 train/200 dev) and the reported test split are described as disjoint, but for the Gen-track EHE adaptation the isolation is less clear; VDiff-Judge’s reliability check covers only 100 claims with no per-category breakdown.
Questions for authors
  1. Please report the fraction of MCQ questions built from programmatic transformations per category, and EHE’s before/after accuracy on the human-annotated subset only. This decides how the headline gain should be interpreted.
  2. For the Gen-track EHE adaptation, confirm the optimization pairs are disjoint from the 300 evaluation pairs, and describe how the judge used during optimization differs from VDiff-JUDGE (circularity risk).
  3. Can you report VDiff-Judge error asymmetry (supported claims rejected vs. unsupported claims accepted) on a larger, category-stratified sample?
6 comments
AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer 26fr (1/3)

We want to first thank the reviewer for recognizing our benchmark’s diagnostic value and insightful failure analysis. To begin with, we hope to summarize your main concerns and how we addressed them; we have also updated the paper PDF with changes highlighted in blue:

  1. EHE’s gains on non-programmatic data
  • We present EHE results on non-programmatic data subset across MCQ and Gen, showing performance gains on both.
  1. Details on benchmark construction source composition was only in the appendix
  • We added Table 1 and a short paragraph in Section 2.2 to report % of data from programmatic transformations by category.
  1. Need more details on data splits during Gen EHE optimization, need more verification and category breakdown for VDiff-Judge.
  • We added further clarification in Section 4.2 and Appendix C.2 to confirm separate partitions of train, dev, and test / evaluation data without overlap
  • We extend our VDiff-Judge verification to 3 human annotators on 200 sampled claims, and provide category breakdown of these claims in Table 11 of Appendix D.3 to show balanced distribution.

Below, we provide more detailed responses to your concerns and questions, as well as point you to changes in the updated paper PDF.

AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer 26fr (2/3)

[W1 & Q1 - EHE Gains on non-programmatic changes? We provide additional result to show EHE gains extend beyond deterministic image perturbations]

  • We re-calculate EHE performance on the subsets of MCQ and Gen data with non-programmatic changes (excluding noise, illumination, etc..). Table below show that EHE boosts both MCQ and Gen performance on non-programmatic data.

    • For example, on the challenging MCQ subset sourced from kids' puzzle games (similar to the example in Figure 1), accuracy improves from 51.56% to 57.29% across 192 questions. | Subset | Metric (%) | Direct Qwen | + EHE | Gain (pp) | | --- | --- | ---: | ---: | ---: | | MCQ: 1,265 questions | Accuracy | 77.15 | 81.11 | +3.95 | | Gen: 174 pairs | Micro-F1 | 23.22 | 41.69 | +18.47 |
  • Nevertheless, we are happy to accept the author's advice and modify our claim, clarifying that EHE builds effective tool-augmented harness around a small model to achieve higher visual difference identification performance..

[W2 & Q1 - Construction composition of MCQ questions is only in the appendix. We updated Section 2.2 and add Table 1 in the main text to clarify MCQ question composition by programmatic / non-programmatic sources.]

  • We thank the reviewer for this suggestion, and we didn't include this information in the main text only because of the 9-page limit for the initial submission.

[W3 & Q2 - Confirm separate optimization / evaluation data split for Gen optimization. Yes, we ammend Section 4.2 and Appendix C.2 to confirm that train, development, and evaluation data for Gen experiments are separate with no overlap.]

  • We confirm that experiments on Gen uses 32 training data for failure diagnosis/revision, 24 development data for candidate selection, and 300 evaluation data for final reporting (not used during optimization), with no overlap across the three splits.
  • We would also like to clarify that the judge remains frozen, is never modified by the expert, and only serves as a performance scorer during optimization. We argue that using a fixed evaluator is a standard practice in previous work (e.g. Self-Harness [1], in which the selection and final-reporting populations are separate even though they use the same fixed metric).
    • Therefore, since the setup of our method aligns with prior work, we do not believe that Q2 should be one of our method's weaknesses.
AuthorsOpenReview ↗

Thanks and Initial Response to Reviewer 26fr (3/3)

[W3 / Q3 - Additional validation and error asymmetry on VDiff-Judge? We expanded our human study to 200 VDiff-Judge decisions with 3 annotators, still showing high human-judge agreement. We also provide details on the error asymmetry.]

  • We extend our human annotation study to 3 human annotators on 200 VDiff-Judge decisions with balanced coverage across question category and generator models.
  • We here provide additional VDiff-Judge error asymmetry analysis: of 81 human-supported claims, 6 are incorrectly rejected (7.4%); of 119 human-unsupported claims, 3 are incorrectly accepted (2.5%). The observed errors therefore lean toward withholding credit rather than accepting unsupported claims.
    • We provide detailed results are in the table below, note that False rejection = rejected / human-supported; false acceptance = accepted / human-unsupported.
CategoryAgreement (%)False rejectionFalse acceptance
Appearance/disappearance27/28 (96.4)1/150/13
Low-level visual changes25/27 (92.6)2/60/21
Motion28/28 (100.0)0/160/12
OCR/text30/30 (100.0)0/120/18
Position28/29 (96.6)0/61/23
Regional color28/29 (96.6)0/101/19
Substitution/size25/29 (86.2)3/161/13
Overall191/200 (95.5)6/81 (7.4%)3/119 (2.5%)

We believe that the contribution of our VDiff-Bench and VDiff-Judge has been established in our submission, and we sincerely hope the additional clarifications, ablations, and human validation address your remaining concerns. We would appreciate it if you could adjust your ratings accordingly, thanks!

Comments on the paper

3 comments

I thank the authors for the detailed response and the revised paper. The added comparison with Meta-Harness, the VDiff-Judge reliability study, and the held-out evaluation address most of my concerns.

However, the new experiments on EHE's effectiveness confirm my concern in W1. Table 9 shows that the initial harness H0 already accounts for most of the gain over direct prompting (+13.8 of +16.4 points), while the selected changes add only 2.6 points; and the gain over Meta-Harness in Table 3 comes almost entirely from the whole-color and noise categories, where H0's global image measurements help most. Therefore, I decide to maintain my original score.

AuthorsOpenReview ↗

Thanks and Follow-Up Response to Reviewer MBC9

We thank the reviewer for acknowledging that the additional results address most of your concerns on our proposed benchmark and VDiff-Judge! Below, we hope to provide some additional information for the reviewer to consider:

[Results prove that our EHE method provides a meaningful gain over the strong initial harness baseline, while we acknowledge there's room for further improvement.]

We agree that the initial harness (which was also proposed by us to provide a strong baseline over direct prompting) achieves remarkable performance gain. However, as Reviewer itz4 also acknowledged in their follow-up to our rebuttal, EHE still provides a meaningful additional benefit over this strong baseline: +2.58 percentage points on held-out questions (represent 36 more correct answers) beyond an already strong tool-equipped baseline, and the improvement persists after excluding shared-source images (+2.85 points over the strong baseline).

  • We agree that there is still room for future improvement on top of the proposed EHE approach on the proposed visual difference identification task. We will revise our paper to include a discussion of EHE's limitations and elaborate on future works.
    • We will also open-source our code and outputs to encourage future research efforts on this direction.

[Our main contributions as a datasets and benchmark track paper is VDiff-Bench and VDiff-Judge, which we have shown to be valid and meaningful.]

Additionally, we respectfully ask that our method performance be weighed alongside our primary contributions as a datasets and benchmark track submission:

  • We believe that we have shown value of our work by proposing meaningful benchmark and evaluation resources: VDiff-Bench’s diagnostic benchmark and VDiff-Judge’s evidence-grounded evaluation with validated human agreement.

Given the reviewer's positive assessment of our main contributions on benchmark and evaluation resources, we would sincerely appreciate it if the reviewer could potentially reconsider our final rating.

Thank you again for your time and constructive feedback!

Thank you for the follow-up clarification. I acknowledge the contributions of VDiff-Bench and VDiff-Judge. In particular, I find VDiff-Judge well-motivated and effective for evaluating open-ended visual difference descriptions, compared to the widely used language-comparison metrics.

However, my concern about EHE remains. I agree with Reviewer itz4 that EHE brings an additional gain over H0, but, as Reviewer itz4 also noted, this gain is relatively modest. On the held-out questions in Table 9, the selected changes add 2.58 points, about 15.7% of the total gain over direct prompting, while the hand-designed H0 accounts for the rest. In Table 3, the advantage of EHE over Meta-Harness also comes mainly from the whole-color and noise categories. Therefore, the claim in the contribution list that EHE enables small models to demonstrate proprietary-level IDI performance without training is not supported, since most of this gain comes from H0 rather than from the evolution process.

In addition, at ICLR, "datasets and benchmarks" is a primary area rather than a separate track, and submissions in this area are evaluated under the same criteria as other papers. Since EHE is presented as a main contribution in both the abstract and the contribution list, it should be assessed as one. Therefore, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.