- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper targets fine-grained image difference captioning, describing small, part-level changes between two nearly identical images. It introduces ChainDiff, a pipeline that applies local edits one after another with an image-editing model, keeps only small, single-region changes, gets bounding boxes from pixel differences, and has a vision-language model caption each change, so the number of changes per sample can be controlled. It also proposes FDDA, a small plug-in module between the vision encoder and the language model that splits image features into low- and high-frequency parts and runs cross-attention between the two images within each part. The authors evaluate on several base models across the standard public benchmarks and their own test set, and report the SOTA results among multimodal LLM methods on most test sets, consistent gains from FDDA, and a large jump on their own test set.
Critical strengths
A practical, transparent and controllable data pipeline.
Broad and consistently designed experiments.
Critical weaknesses
FDDA's contribution is not isolated.
- No control that removes the frequency split. The FDDA ablation doesn't remove the frequency split while keeping everything else. It also never compares against the obvious simple options: subtracting one image's features from the other's, or giving the model the pixel difference map as an extra input. And it never checks the normalization layer the module adds. So the gains may come from extra trainable parameters, generic cross-image attention, or normalization, not from the frequency decomposition.
- Not explicit difference modeling. Cross-attention from one image to the other copies information across images. That is not the "explicit difference modeling" the paper claims.
- Many of FDDA's gains are a fraction of a CIDEr point. The paper's own FDDA ablation shows swings about as large as FDDA's total gain on the authors' test set. When the regenerated reference captions are used, FDDA's gain on one base model disappears. The claim that FDDA is "broadly effective" is not yet supported.
There is little evidence that ChainDiff's specific properties help fine-grained captioning, and the headline comparisons are mixed up with the choice of base model.
-
The "best among MLLMs" claim mostly reflects newer base models. On several benchmarks, the newer base models already beat earlier MLLM methods before either contribution is added. On the one base model shared with the strongest earlier method, the full model is:
- behind that method on some benchmarks;
- only slightly ahead on others;
- behind on BLEU and METEOR on both OmniDiff subsets.
The summary table also takes the best result across base models.
-
Gains on public benchmarks are small and inconsistent. Adding ChainDiff helps little and unevenly, and sometimes hurts. The biggest public gain is on Image-Edit-Request. That benchmark is the least fine-grained one, but like ChainDiff it is made of edited images. The benefit may come from matching the editing domain rather than from fine-grainedness.
-
The jump on the authors' test set is in-distribution, and the checks meant to address this are weak.
- Training and test data come from the same pipeline and the same captioning model.
- When the references are regenerated by another model, every score barely moves, even for the model never trained on ChainDiff. CIDEr is sensitive to wording, so truly independent references should shift scores noticeably. This looks as if the new references were close rewrites of the old ones, and the procedure is not described.
- The test references were themselves made by subtracting the images, drawing boxes, and asking a strong vision-language model to describe them. A training-free version of that same procedure might do very well on this test set. Without that baseline, the jump says little about what the trained models actually learned.
Questions for authors
Clarifications:
-
Regenerated references. What was the second model given: only the images, or also the original captions? Which model version and prompt were used? Why do scores also stay flat for the model never trained on ChainDiff? How much word overlap is there between the two sets of references?
-
Isolating FDDA. Can you add these versions?
- FDDA without the frequency split, but with the same size, gate and normalization;
- a version that simply subtracts one image's features from the other's;
- the added normalization layer on its own.
-
Simple difference baselines. How does a training-free version of your own labeling procedure (subtract the images, draw boxes, prompt a strong vision-language model) perform on your test set and on the public benchmarks where the images are pixel-aligned? How does feeding the difference map as an extra input compare to FDDA? If these simple baselines come close, the case for FDDA and for your test set is much weaker.
-
Comparison with the strongest earlier method. Your public-data-only baseline on the shared base model scores below that method's published result. Was it retrained under your recipe? Can you report that method plus ChainDiff, to separate the effect of the data from the effect of the module?
-
Where exactly is FDDA inserted?
- For each base model, which vision-encoder layer does FDDA read from, and what does "patch merger" correspond to? In LLaVA-OneVision it would be the MLP projector; in InternVL, is it before or after pixel shuffle?
- With tiled or native-resolution inputs, is the frequency transform applied per tile or to the stitched full-image grid?
- How often does the module turn itself off because tile counts or grid sizes don't match?
- Pairing comes from the image order in the batch rather than from the prompt. How does this work when an input has more than two images, or when the before and after images differ in size?
-
Data. Which filter removes the most samples in the last stage? Was the training set ever checked by hand for wrong captions or unintended edits, or only the test set?
-
Possible shortcut. For samples that start from the original image, the "before" image is a real photo and the "after" image has been fully regenerated by the editing model. Did you check whether models learn to spot generation artifacts, or small global color drift that falls below your pixel threshold?
Format:
- The AI use statement sits at the very end of the appendix rather than before the references as in the official template. It also does not cover the paper's own use of generative models to create and filter data.
- Several reference entries have wrong author names and should be checked against the original papers.
- One equation in the method section and several appendix tables run past the right margin.
- The template's default font has been changed.