acceptodds
Withdrawn from ICLR 2027

RealNLR: Do VLMs Learn Nonlocal Visual Algorithms from Synthetic Fine-Tuning?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) do well on standard benchmarks but fail simple tasks that require chaining visual evidence across an image, called nonlocal visual reasoning (NLR): re-identifying a transformed object, following a chain of cues (saccadic search), and tracing a wire through clutter (contour tracing). We ask whether fine-tuning can teach these visual algorithms, and what the model learns if it does. We fine-tune five open VLMs (4B to 32B) with QLoRA on synthetic versions of the three tasks. We vary the adapted side (language or vision), the supervision (answer only, or intermediate steps too), the amount of data, and the rendering style (default or randomized). To measure transfer, we introduce RealNLR, a benchmark that poses the same tasks on realistic images. We find that (i) fine-tuning raises in-distribution accuracy by 13 points on average, and the best runs go from near zero to 98%, but transfer to RealNLR is zero; (ii) process supervision teaches the steps, with InternVL3-14B writing 96% of out-of-distribution search trajectories exactly, yet its answer still stops at the chain lengths seen in training; (iii) internal analyses show that fine-tuning improves one stage of the procedure on synthetic images, but not how the model builds visual state from a natural image; and (iv) randomized-style training does not improve transfer (mean change −0.6 points against a matched control). The models learn the dataset, not the algorithm.

Primary area:
datasets and benchmarks
Keywords:
Nonlocal visual reasoning (NLR), RealNLR, Object re-identification, Saccadic search, Contour tracing, Synthetic fine-tuning, Post-training, QLoRA, Process supervision, Visual algorithms, Transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_MFLkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors attempt to strengthen VLMs ability to do nonlocal visual reasoning--- the iterative process of chaining evidence or information across different parts of an image--- by finetuning on synthetic data. They introduce a natural dataset, and find that finetuning in a variety of different ways on synthetic data does not fix the problem on natural data or even other types of synthetic data.

Critical strengths

Their benchmark-- Because the original nonlocal visual reasoning benchmark is synthetic, so establishing the same deficiencies on natural tasks (RealNLR) gives more credence to the idea that this is an important deficiency, and also not one that is isolated to synthetic data.

Critical weaknesses
  • Strength of results/lack of novelty: While nonlocal visual reasoning is an important deficiency in VLMs and they establish that fine-tuning does not fully solve it, I feel this is insufficient novelty for a paper. Ideally they would either show the degree of transfer that does work, or else give suggested reasons or deeper insight into why this ability cannot be recreated on arbitrary natural data. I do not see this analysis in the paper.
  • Writing is poor and unclear. I would recommend a substantial rewrite and reorganization of the paper.
    • E.g.: "To interpret the results, we split nonlocal reasoning into three stages: construct the relevant state from the image, operate on that state, and convert the result into an answer" (L54).
    • Lots of analysis simply restate results without analysis (e.g., L224-221) Graphs are difficult to understand.(e.g., Figure 6,)
    • Lots of jargon which makes the paper difficult to read "repeats the turnover pattern" (L303), "the match relation is already in the base model" (L337), . To my eye, these terms are not necessary and serve to obfuscate rather than explain their results and methodology.
Questions for authors
  • How do you guarantee that the steps are valid for the process sft (L167-169)? Do you supervise on only the ground truth or also the steps?
  • How did you decide the exact parameters to train, and have you established that other subsets (for instance, full SFT instead of LORA, and training language and vision separately), or other training procedures (for example, RL) would or wouldn't work?
  • It seems like you have only tested minor stylistic differences in Sec 4.3, and that it does not rule out the possibility that a larger class of synthetic data could be used to train a model that can do well on natural images (e.g., a procedural scene generator). Did you try this?
  • How much of the performance loss on RealNLR after training is because of formatting differences between it and the synthetic data? Do the models attempt to use a different answering format?
Reviewer_mDYTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the author focus on the nonlocal visual reasoning task, and try to use fine-tuning to resolve the challenges.

Critical strengths

The author studies whether fine-tuning could close the gap of the model's inability in non-local reasoning.

Critical weaknesses
  1. The paper is more like a technical study of the existing models and methods. First of all, there are multiple fine-tuning algorithms, from data construction to various optimization algorithms. The methods tested and validated in the paper is only a small set of the fine-tuning domain thus the results may not indicate meaningful enough understanding or experience.

  2. The writing itself needs improvement, and the author tries to directly copy-paste plots from the berman & deng's paper without proper illustration or even handling the crop correct. (for example, figure 1)

  3. A lot of conclusions are not well-grounded, for example, the findings style randomization does not change transfer, the conclusion is based on comparing randomized run and default run based on the same seed, which is very limited.

Questions for authors

My main questions is that how generating a synthetic data and task from the Berman's work could provide enough evidence on whether fine-tuning works or not on non-local reasoning.

Also, how can the knowledge and conclusions transfers to other non-local reasoning tasks or domains.

Reviewer_TxWLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether VLMs can acquire transferable nonlocal visual reasoning through fine tuning on synthetic data. It studies three tasks, including object reidentification, saccadic search, and contour tracing, across five open source VLMs using QLoRA. The authors systematically vary fine tuning configurations, supervision strategies, data scales, and rendering styles, and introduce RealNLR to evaluate generalization to real images. While fine tuned models achieve up to 98% accuracy on synthetic tasks, improvements on real images are negligible, suggesting that models learn dataset specific patterns rather than generalizable reasoning algorithms.

Critical strengths
  1. RealNLR addresses the gap in prior work by evaluating whether training can remedy known NLR limitations, rather than merely diagnosing them.

  2. The systematic experimental design, including controlled interventions and ablations, provides useful insights.

Critical weaknesses
  1. Several experimental details and result interpretations are insufficiently explained, reducing the paper's clarity and readability.

  2. The main experiments use only one random seed, while results from a second seed show deviations exceeding 46 percentage points in some settings, raising concerns about the reliability and reproducibility of the quantitative findings.

  3. The evaluation sets are relatively small, with only 150 ID/OOD samples per task and 300 RealNLR samples per task in the main experiments.

  4. The human baseline is evaluated by the authors themselves rather than independent participants, potentially introducing bia.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.