- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The authors look at the task of text/image alignment. They break the text into entities and claims, and measure the existence/alignment of each in a the given image. They then provide an aggregate score over all claim evaluations. The authors train a model to perform claim evaluation based on the image, the claim and a bounding box. The authors evaluate their approach on several benchmarks, including a new one introduced in the paper, AGITA-BENCH.
-
Note: I commented on several baselines, ablations and analyses that I think are important for the paper to pass the quality bar. If the authors provide them and they still indicate at the novelty and quality of VisExMEM, I would raise my score.
-
Update: the authors have fully answered all my questions and comments. With the additional experiments and info I think the paper is well rounded and I am raising my score accordingly.
Critical strengths
-
The problem of image/text alignment eval is important to keep improving image generation and caption generation with increasingly complex inputs.
-
The proposed approach is novel and interesting
-
The training process is interesting and well written at the high level, though some fine details are missing, around training data generation for example, that are needed to reproduce the approach.
-
The evaluation against existing benchmarks and prior eval works is rigorous
Critical weaknesses
-
I am missing VNLI-like (see Yarom, 2023) baseline, a model trained on similar data as used for VisExMEM, but only to predict VNLI(image, caption) -> yes/no, without breaking into claims and aggregating.
-
VQ^2 also splits the text into claims and match each claim, but in Yarom, 2023, they did not train a model. Training VisExMEM without explanations and bounding boxes would be exactly like that. This would be a useful baseline/set of ablations to understand the impact of each of these inputs/output on the alignment decision quality.
-
I am missing error analyses on VisExMEM outputs, both the alignment decision at each claim type and overall, as well as the explanation quality.
-
I am missing an analysis of the quality of VisExMEM w.r.t caption complexity, e.g. the number of generated claims, their types etc. More specifically, is there a range of captions on which VisExMEM improves the most compared to baselines, or is it improving both on easy and difficult prompts alike.
-
AGITA-BENCH looks like an interesting benchmark, but it is impossible to evaluate its usefulness and quality based on half a page. Given that it is only one of many benchmarks evaluated in the paper, I would consider a separate paper about it, to measure its novelty, quality etc. I prefer more focus and space for ablations and analyses of VisExMEM than adding a benchmark whose quality is hard to evaluate from the paper. Overall, I see AGITA-BENCH as a weakness of the paper, not its strength.
Questions for authors
-
Please see my comments above
-
[1] also proposes to generate explanations, both textual and visual (bounding box) instead of getting the bounding box candidate as input. Could you please explain why not use this fully generative approach (part of y's tokens are the bounding box coordinates) instead of getting a bounding box as input? (a disclaimer - perhaps I didn't understand the paper and that's what you are doing)
[1] Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment