- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors attempt to strengthen VLMs ability to do nonlocal visual reasoning--- the iterative process of chaining evidence or information across different parts of an image--- by finetuning on synthetic data. They introduce a natural dataset, and find that finetuning in a variety of different ways on synthetic data does not fix the problem on natural data or even other types of synthetic data.
Critical strengths
Their benchmark-- Because the original nonlocal visual reasoning benchmark is synthetic, so establishing the same deficiencies on natural tasks (RealNLR) gives more credence to the idea that this is an important deficiency, and also not one that is isolated to synthetic data.
Critical weaknesses
- Strength of results/lack of novelty: While nonlocal visual reasoning is an important deficiency in VLMs and they establish that fine-tuning does not fully solve it, I feel this is insufficient novelty for a paper. Ideally they would either show the degree of transfer that does work, or else give suggested reasons or deeper insight into why this ability cannot be recreated on arbitrary natural data. I do not see this analysis in the paper.
- Writing is poor and unclear. I would recommend a substantial rewrite and reorganization of the paper.
- E.g.: "To interpret the results, we split nonlocal reasoning into three stages: construct the relevant state from the image, operate on that state, and convert the result into an answer" (L54).
- Lots of analysis simply restate results without analysis (e.g., L224-221) Graphs are difficult to understand.(e.g., Figure 6,)
- Lots of jargon which makes the paper difficult to read "repeats the turnover pattern" (L303), "the match relation is already in the base model" (L337), . To my eye, these terms are not necessary and serve to obfuscate rather than explain their results and methodology.
Questions for authors
- How do you guarantee that the steps are valid for the process sft (L167-169)? Do you supervise on only the ground truth or also the steps?
- How did you decide the exact parameters to train, and have you established that other subsets (for instance, full SFT instead of LORA, and training language and vision separately), or other training procedures (for example, RL) would or wouldn't work?
- It seems like you have only tested minor stylistic differences in Sec 4.3, and that it does not rule out the possibility that a larger class of synthetic data could be used to train a model that can do well on natural images (e.g., a procedural scene generator). Did you try this?
- How much of the performance loss on RealNLR after training is because of formatting differences between it and the synthetic data? Do the models attempt to use a different answering format?