acceptodds
Under review as a conference paper at ICLR 2027

ChainDiff: Controllable Fine-Grained Image-Pair Synthesis and Frequency-Domain Difference Attention for Image Difference Captioning

Abstract

Fine-grained Image Difference Captioning (IDC) requires precisely describing sub-part-level differences between two highly similar images. Existing work is constrained by two bottlenecks: fine-grained image pairs are scarce and their differences are uncontrollable, and mainstream Vision-Language Models (VLMs) rely on implicit alignment to discover weak difference signals hidden inside overwhelming commonality. We propose ChainDiff, a controllable image-pair synthesis pipeline that combines dual-channel similarity filtering, a hard upper bound on the changed-region ratio, and chained editing with segment sampling to make sub-part-level fine-grainedness and per-pair change counts explicitly controllable. We further propose the Frequency-Domain Difference Attention (FDDA) module, which decouples multi-scale difference signals via a feature-domain discrete cosine transform (DCT) decomposition and models differences explicitly via dual-branch image-pair cross-attention, integrated into VLMs in a plug-in manner during supervised fine-tuning (SFT). We term the resulting models, trained on ChainDiff data with FDDA integrated, HarmonicDiff. Across three base VLMs and eight test sets, HarmonicDiff attains the best CIDEr among Multimodal LLM (MLLM) methods on five test sets; specialized non-MLLM baselines remain stronger in CIDEr only on synthetic CLEVR-Change.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Difference Captioning, Vision-Language Models, Fine-Grained Visual Understanding, Controllable Data Synthesis, Frequency-Domain Attention

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_k6TZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets fine-grained image difference captioning, describing small, part-level changes between two nearly identical images. It introduces ChainDiff, a pipeline that applies local edits one after another with an image-editing model, keeps only small, single-region changes, gets bounding boxes from pixel differences, and has a vision-language model caption each change, so the number of changes per sample can be controlled. It also proposes FDDA, a small plug-in module between the vision encoder and the language model that splits image features into low- and high-frequency parts and runs cross-attention between the two images within each part. The authors evaluate on several base models across the standard public benchmarks and their own test set, and report the SOTA results among multimodal LLM methods on most test sets, consistent gains from FDDA, and a large jump on their own test set.

Critical strengths

A practical, transparent and controllable data pipeline.

Broad and consistently designed experiments.

Critical weaknesses

FDDA's contribution is not isolated.

  • No control that removes the frequency split. The FDDA ablation doesn't remove the frequency split while keeping everything else. It also never compares against the obvious simple options: subtracting one image's features from the other's, or giving the model the pixel difference map as an extra input. And it never checks the normalization layer the module adds. So the gains may come from extra trainable parameters, generic cross-image attention, or normalization, not from the frequency decomposition.
  • Not explicit difference modeling. Cross-attention from one image to the other copies information across images. That is not the "explicit difference modeling" the paper claims.
  • Many of FDDA's gains are a fraction of a CIDEr point. The paper's own FDDA ablation shows swings about as large as FDDA's total gain on the authors' test set. When the regenerated reference captions are used, FDDA's gain on one base model disappears. The claim that FDDA is "broadly effective" is not yet supported.

There is little evidence that ChainDiff's specific properties help fine-grained captioning, and the headline comparisons are mixed up with the choice of base model.

  • The "best among MLLMs" claim mostly reflects newer base models. On several benchmarks, the newer base models already beat earlier MLLM methods before either contribution is added. On the one base model shared with the strongest earlier method, the full model is:

    • behind that method on some benchmarks;
    • only slightly ahead on others;
    • behind on BLEU and METEOR on both OmniDiff subsets.

    The summary table also takes the best result across base models.

  • Gains on public benchmarks are small and inconsistent. Adding ChainDiff helps little and unevenly, and sometimes hurts. The biggest public gain is on Image-Edit-Request. That benchmark is the least fine-grained one, but like ChainDiff it is made of edited images. The benefit may come from matching the editing domain rather than from fine-grainedness.

  • The jump on the authors' test set is in-distribution, and the checks meant to address this are weak.

    • Training and test data come from the same pipeline and the same captioning model.
    • When the references are regenerated by another model, every score barely moves, even for the model never trained on ChainDiff. CIDEr is sensitive to wording, so truly independent references should shift scores noticeably. This looks as if the new references were close rewrites of the old ones, and the procedure is not described.
    • The test references were themselves made by subtracting the images, drawing boxes, and asking a strong vision-language model to describe them. A training-free version of that same procedure might do very well on this test set. Without that baseline, the jump says little about what the trained models actually learned.
Questions for authors

Clarifications:

  • Regenerated references. What was the second model given: only the images, or also the original captions? Which model version and prompt were used? Why do scores also stay flat for the model never trained on ChainDiff? How much word overlap is there between the two sets of references?

  • Isolating FDDA. Can you add these versions?

    • FDDA without the frequency split, but with the same size, gate and normalization;
    • a version that simply subtracts one image's features from the other's;
    • the added normalization layer on its own.
  • Simple difference baselines. How does a training-free version of your own labeling procedure (subtract the images, draw boxes, prompt a strong vision-language model) perform on your test set and on the public benchmarks where the images are pixel-aligned? How does feeding the difference map as an extra input compare to FDDA? If these simple baselines come close, the case for FDDA and for your test set is much weaker.

  • Comparison with the strongest earlier method. Your public-data-only baseline on the shared base model scores below that method's published result. Was it retrained under your recipe? Can you report that method plus ChainDiff, to separate the effect of the data from the effect of the module?

  • Where exactly is FDDA inserted?

    • For each base model, which vision-encoder layer does FDDA read from, and what does "patch merger" correspond to? In LLaVA-OneVision it would be the MLP projector; in InternVL, is it before or after pixel shuffle?
    • With tiled or native-resolution inputs, is the frequency transform applied per tile or to the stitched full-image grid?
    • How often does the module turn itself off because tile counts or grid sizes don't match?
    • Pairing comes from the image order in the batch rather than from the prompt. How does this work when an input has more than two images, or when the before and after images differ in size?
  • Data. Which filter removes the most samples in the last stage? Was the training set ever checked by hand for wrong captions or unintended edits, or only the test set?

  • Possible shortcut. For samples that start from the original image, the "before" image is a real photo and the "after" image has been fully regenerated by the editing model. Did you check whether models learn to spot generation artifacts, or small global color drift that falls below your pixel threshold?

Format:

  • The AI use statement sits at the very end of the appendix rather than before the references as in the official template. It also does not cover the paper's own use of generative models to create and filter data.
  • Several reference entries have wrong author names and should be checked against the original papers.
  • One equation in the method section and several appendix tables run past the right margin.
  • The template's default font has been changed.
Reviewer_sjhrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained image difference captioning through a synthetic dataset and a model component. ChainDiff generates image pairs through sequential local edits, filters unintended changes, and automatically produces bounding boxes and region-level captions. The dataset contains 29,043 image pairs, including 27,344 training pairs. The proposed Frequency-Domain Difference Attention (FDDA) module uses a discrete cosine transform to separate visual features into low- and high-frequency components, applies bidirectional cross-image attention within each band, and combines the outputs through a learned gate. The resulting HarmonicDiff models are evaluated with three VLM backbones on eight test sets. The paper reports the best CIDEr scores among the compared MLLM methods on five public test sets.

Critical strengths
  1. Local attribute changes are easily obscured by the extensive visual content shared between two images. Improving sensitivity to such changes is a meaningful direction for IDC.

  2. The authors combine chained local edits, DINOv2/HSV-based filtering, pixel-difference connected components, region marking, and VLM-generated descriptions. The explicit filtering criteria and reported data statistics make the procedure relatively concrete.

  3. Results are reported for LLaVA-OneVision, InternVL3, and Qwen3.5, including several external IDC test sets. The progressive ablations also provide initial evidence that both additional data and the FDDA component may be beneficial.

Critical weaknesses

1. The central contribution of frequency decomposition has not been isolated from ordinary cross-image attention.

The claimed model-side novelty is that DCT-based low/high-frequency processing better exposes subtle image-pair differences. Yet FDDA changes both the feature representation and the cross-image interaction capacity. Table A5 varies the cutoff, head layout, and transform (DCT versus FFT), but does not include the most diagnostic control: a parameter- and compute-matched bidirectional cross-attention block applied directly to the original features without frequency decomposition. Nor does it isolate the low- and high-frequency branches individually.

This omission matters because the measured gains can be modest. On the authors' ChainDiff test set, FDDA raises CIDEr from 58.1 to 59.3 with LLaVA-OneVision, 61.7 to 62.9 with InternVL3, and 60.1 to 62.3 with Qwen3.5 (Table A3). Those gains establish an improvement over the reported no-FDDA configuration, but do not show that frequency separation, rather than additional learnable cross-attention, is responsible. Moreover, cross-attention is an interaction operator; by itself it does not guarantee selective emphasis on changed rather than shared content. A direct attention-only control and localized change-response analysis are needed to substantiate the proposed mechanism.

2. The evaluation does not directly test whether the difference-enhancement mechanism suppresses pseudo-changes.

Focusing on fine-grained IDC is legitimate; the concern is not that the paper must solve every form of IDC. Rather, the paper motivates FDDA as a mechanism for extracting weak differences from overwhelmingly shared visual information, while the ChainDiff generation policy deliberately excludes spatial displacement, pose changes, and global scene modifications (Appendix A.1). In real image pairs, however, slight viewpoint shifts, object misalignment, illumination variation, and rendering noise can create strong feature differences without constituting the target semantic change.

This is particularly relevant to the high-frequency branch, which is intended to emphasize local discontinuities and boundaries. Such features might help detect subtle true edits, but could also respond to harmless boundary shifts. The latter is a plausible failure mode, not an observed failure demonstrated by the present results. The CLEVR-DC benchmark provides a useful stress test with viewpoint changes, but aggregate captioning scores do not indicate whether the model is robust to these pseudo-changes or whether it makes more false change claims. A targeted matched-pair evaluation with controlled nuisance transformations, no-change cases, and change-localization/false-positive metrics is needed to support the intended difference-selectivity argument.

3. The claimed control over semantic change counts is stronger than the automatic verification establishes.

The paper equates the intended number of changes with the length of an editing-chain segment, after requiring one sufficiently large connected component per adjacent edit and an upper bound on the changed-area ratio (Section 3.3; Appendix A.2). These are helpful pixel-level proxies, but not a semantic guarantee: one connected component can include edits to multiple adjacent objects, and a single semantic edit can produce several disconnected pixel regions. The 10% restriction applies to individual edited regions, not automatically to the combined change area of every longer segment.

The authors manually cross-check the 1,699 test pairs (Appendix B), which strengthens confidence in that test set, but does not quantify semantic change-count accuracy, region grounding accuracy, or caption correctness across the 27,344 automatically annotated training pairs. Given that precise edit-count control and region-aligned labeling are presented as central contributions, a stratified independent audit for 1/2/3-change samples and localization precision/recall would be considerably more informative than filtering statistics alone.

4. Several comparisons do not fully disentangle the source of performance gains.

The paper reports three progressive conditions: public data only; public data plus ChainDiff; and public data plus ChainDiff plus FDDA. However, FDDA without ChainDiff is not evaluated, so the interaction between the two proposed contributions cannot be assessed as a complete factorial study. Adding 27,344 ChainDiff training pairs also changes dataset size—and, under one-epoch training, the number/composition of optimization steps—without a matched-data-volume or matched-update-budget control. Thus the data-specific benefit is not fully separated from the effect of additional supervision.

Cross-backbone comparisons also require careful attribution. For example, on OmniDiff-Real, M3 Diff achieves 31.3 CIDEr using LLaVA-OneVision, whereas HarmonicDiff-L reaches 31.8 (Table 1). The highest reported HarmonicDiff result, 46.9, comes from the different Qwen3.5-4B backbone, whose public-data-only configuration already scores 46.0. The improvement of 46.9 over 31.3 therefore cannot be interpreted as the gain caused by FDDA or ChainDiff. The manuscript also notes that its OmniDiff sampling differs from M3 Diff's 2× upsampling practice (Section 5.1). These facts do not invalidate the results, but the narrative should foreground controlled within-backbone improvements, and ideally report matched-budget controls and multi-seed variation for small gains.

5. Core visual explanations are difficult to read.

Figures 2 and 3 (pp. 4 and 6) contain extremely small text and dense annotations. These are the two figures most critical for understanding the data pipeline and FDDA, yet a reader cannot easily inspect their details at ordinary PDF scale. Splitting the figures into higher-level diagrams and moving parameters or explanations into captions/appendix would substantially improve clarity. Table A2 is similarly cramped.

Questions for authors
  1. How does FDDA compare with a parameter- and compute-matched bidirectional cross-attention module without DCT? What are the results for low-frequency-only and high-frequency-only branches, preferably on external benchmarks as well as ChainDiff?

  2. Under controlled viewpoint, alignment, illumination, and rendering perturbations that do not change the underlying scene semantics, does FDDA improve or worsen false-positive change descriptions? Can the authors report change/no-change breakdowns or localized change-response analyses?

  3. On an independently audited sample of the automatically generated training set, what fraction of the nominal 1-, 2-, and 3-change pairs contain exactly that many semantic changes? How accurate are the auto-derived bounding boxes and region captions?

  4. What are the results for FDDA-only (without ChainDiff), an equally sized alternative-data condition, and matched optimization budgets? How stable are the modest external-benchmark gains across random seeds?

  5. Could Figures 2 and 3 be simplified or split, with substantially larger labels and the secondary thresholds moved to the appendix?

Reviewer_2hcmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles fine-grained image difference captioning (IDC) from both data and model perspectives. On the data side, the paper introduces ChainDiff, an automated synthesis pipeline that applies chained FLUX.2 edits to 300k web images. On the model side, the paper proposes Frequency-Domain Difference Attention (FDDA), a plug-and-play module inserted between the vision encoder and the patch merger. Integrating FDDA into base VLMs and training on ChainDiff yields HarmonicDiff. Experiments on three base VLMs demonstrate that HarmonicDiff achieves state-of-the-art CIDEr scores among MLLM-based methods on five out of eight benchmark test sets.

Critical strengths
  1. The pipeline has explicit control over change magnitude and change count.
  2. The data and module contributions show adaptability across three different base VLMs, rather than on a single model.
Critical weaknesses
  1. Many references have wrong author lists. At least 12 entries do not match the published papers; for example, Kim et al. (2021) and Huang et al. (2022), where every author name is wrong. Several entries also carry free-text notes such as "Proposes the ... model" or "No formal paper; cited as a technical release". This pattern is not consistent with the LLM-use statement, which says LLMs were only used for literature search, coding, and minor polishing. The base models (LLaVA-OneVision, InternVL3, Qwen3.5) are not cited at all.
  2. The gains on public benchmarks are small and not well-supported. Tables A6–A10 reveal that adding ChainDiff yields nearly identical performance across most public datasets, contrasting with the highlighted claim around Line 430. Substantial improvements appear confined to the authors' own test set. Since this test set shares the exact synthesis pipeline with the training set, the model may simply be exploiting distribution-specific shortcuts rather than learning true visual differences.
  3. Accuracy of the automatically generated annotations. Both the generated difference captions and their subsequent quality evaluations depend entirely on the same model (Qwen3.5-Plus), lacking any independent noise-filtering mechanism or human validation. Furthermore, there is an explicit discrepancy in the data curation pipeline: Sec. 3.5 states that the 62k samples originate from Qwen3.5-Plus generation before the 5-rule fallback filter, whereas Figure 2 illustrates them as coming from the 5-rule fallback filter.
  4. Several claims are not evaluated. The paper says the bounding boxes "endow downstream models with grounding capability", but no grounding experiment is reported. Controllable change count is claimed as a main insight, but no experiment uses it (e.g., performance as a function of change count, or training with different count distributions).
  • Kim, Hoeseong, et al. "Agnostic change captioning with cycle consistency." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.
  • Q. Huang et al., "Image Difference Captioning With Instance-Level Fine-Grained Feature Representation," in IEEE Transactions on Multimedia, vol. 24, pp. 2004-2017, 2022, doi: 10.1109/TMM.2021.3074803.
Questions for authors
  1. Please clarify the use of LLMs in paper writing.
  2. Please provide the zero-shot transfer experiments that train on ChainDiff and test on other datasets.
  3. Please provide the noise analysis of the automatically generated annotations, and clarify the exact process of data filtering and sampling.
  4. Please provide a grounding evaluation and an analysis by change count to support the corresponding claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.