acceptodds
Under review as a conference paper at ICLR 2027

AdaLoop: Thinking with the Vision Encoder

Abstract

Most vision-language models (VLMs) treat the vision encoder as a static feature extractor: it encodes an image once, independently of the question, and leaves subsequent reasoning to the language model. Yet different questions may require different visual operations and processing depths. We present AdaLoop, a framework for question-conditioned visual recurrence in VLMs. AdaLoop adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen. We also introduce VisRO, a procedural dataset and generation pipeline for training and evaluating vision-centric reasoning. On six VisRO tasks, AdaLoop outperforms standard fine-tuning and question-conditioned single-pass baselines with matched training data and optimizer updates. These gains hold across three VLM families on both in-distribution and out-of-distribution examples. Our findings suggest that pretrained vision encoders can contribute to reasoning through task-directed, iterative visual computation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, visual reasoning, looped models, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XfdhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AdaLoop, which employs a looped transformer for the vision encoder, which incorporates information from the natural language input through a small learnable network that produces hyperparameters for a new layer normalization step. The authors argue that these two work together organically so that the visual encoder now considers the natural language input when producing the visual representations, as opposed to default setups where the visual representations are produced once without conditioning on the natural language input. Empirical experiments show that the proposed method, AdaLoop, demonstrates a pretty significant gain over vanilla VLMs (three tested) on visual reasoning benchmarks, and the ablation studies clearly separate and discuss where the gains come from.

Critical strengths

The proposed AdaLoop method is fairly straightforward, well motivated, and demonstrates clear benefits on benchmarks. It can also be applied fairly easily to existing models without having to retrain with a huge amount of data, providing real usability benefits.

Critical weaknesses

The proposed loop method is not extremely well coupled with the motivation of using questions to condition the visual encoder, as the empirical evidence shows, the model gains without the loops.

Questions for authors

Could you justify a bit more on why the loop has to co-live with the current question intervention in this paper? I can see the long term benefit, but the current method of adding the layernorm does not seem to well motivate a loop transformer.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Xfdh

W1 & Q1

The model with question conditioning gains without the loops. Could you justify a bit more on why the loop has to co-live with the current question intervention in this paper?

We thank the reviewer for the question. Our goal is to make better use of a vision encoder’s learned representations and transformations by adapting visual processing to each question and its difficulty. This motivates two complementary components: question-conditioned adaptive layer normalization to guide visual processing, and iterative computation / looping to refine visual representations as needed. Our experiments demonstrate the benefits of combining these components, while our mathematical analysis offers intuition for why they work together.

  1. Empirically, the ablations show that each component contributes beyond the other. Table 3(b) provides two controlled comparisons. With question conditioning held fixed, adding recurrence improves accuracy by 4.6 ID and 2.2 OOD percentage points over single-pass AdaLN. Conversely, with recurrence and parameter count held fixed, replacing question-independent modulation with question-conditioned modulation improves accuracy by 3.7 ID and 2.4 OOD points. Simply repeating the encoder without modulation yields only a small ID improvement. These results support the combination: conditioning makes repeated use of the pretrained blocks more effective, and recurrence provides additional gains beyond conditioning alone.

  2. We also provide mathematical intuition for this design in Appendix A, which explains why modulation and recurrence are complementary. Under a first-order approximation, question-conditioned modulation can be interpreted as adjusting the “effective depth” of an individual transformer block. However, visual reasoning may also benefit from later computations informing earlier ones, allowing bottom-up and top-down processing to interact across iterations. Looping enables this feedback by feeding the output of later blocks back into earlier blocks within the repeated encoder span. Modulating individual blocks during a single forward pass does not provide this feedback path. Thus, question conditioning adjusts how the pretrained transformations contribute, while recurrence allows those transformations to be recomputed using updated visual information.

  3. The loop’s contribution persists with more capacity and on newer backbones. When the vision tower is fully fine-tuned, the looped model's OOD advantage over single-pass AdaLN grows to 9.6 points at T_train = 4 (50.7 vs 41.1), and on Qwen3-VL-4B and Qwen3.5-4B the looped model again exceeds AdaLN by about 3 ID / 2.7 OOD points (see global response G2).

I thank the authors for the rebuttal. I will maintain my positive rating, which reflects the contribution of this paper.

Reviewer_uuSGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors introduce AdaLoop, a framework to enable question-conditioned visual recurrence using the vision encoders of pretrained VLMs. AdaLoop adapts the layer normalisation parameters of the middle layers of the vision encoder conditioned on the textual question. This generates a recurrent state of latent variables representing a joint understanding of the image-question pair that can be iteratively refined. The paper studies how to best use these latents to inform the LM layers in order to solve vision-centric reasoning tasks. The paper also introduces VisRO, a procedurally generated dataset and generation pipeline comprising 6 tasks that measure fine-grained visual perception and reasoning. The authors train and evaluate their proposed method on VisRO tasks by benchmarking three different-family small models (3-7B) across both in-domain and out-of-domain tasks.

Critical strengths
  • The paper is very well designed and presented, with clear, informative ablations.
  • The procedurally generated dataset is a valuable contribution to the community. A code release, along with supplementary documentation, would be appreciated.
Critical weaknesses
  • The following design choices seem to have been tuned on the test set: a) method variant (A, B, C), b) modulation type, c) single or multiple passes. Table 3’s Qwen2.5-VL numbers for the best-performing configuration match the numbers reported in the main table, which reports test set performance. Since the performance differences between some of these configurations are small (see Table 3a), it raises concerns about whether the same conclusions could be drawn given a clear train/validation/test split separation.
  • Qwen2.5-VL’s base performance in all of the external datasets reported in the paper is higher than its trained counterpart. The model seems to get weaker in all tasks other than VisRO ones after training with AdaLoop on VisRO.
Questions for authors

This is the base model’s performance on the reported external datasets:

ModelMMStarMMBench 1.1GQATextVQA
Qwen2.5-VL-3B-Instruct55.881.561.079.3
  • Figure 7b reports lower numbers for both AdaLN and AdaLoop (sampled or deeply supervised). Does training a model on AdaLoop harm pre-existing knowledge and performance? Could you add more OOD comparisons using external datasets and include the base model’s performance? PuzzleVQA [1] and AlgoPuzzeVQA [2] which seem close in distribution to VisRO. MathVision [3], MathVista [4], and LogicVista [5] are also relevant benchmarks that come to mind.
  • Did you use a validation set? How did you tune the method configurations themselves, and how did you do hyperparameter selection for each configuration? Did the baselines receive the same treatment?
  • Why did you assign larger weights to later depths (Eq (3))? Was this choice ablated?
  • Why did the SFT baseline’s adaptation exclude the vision encoder?
  • It is surprising to me that GRPO has so little effect in a task that’s clearly verifiable. Did you tune the hyperparameters for the GRPO baseline?

[1] PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns (Chia et al., Findings 2024)

[2] AlgoPuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Algorithmic Multimodal Puzzles (Ghosal et al., NAACL 2025)

[3] Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (Wang et al., NeurIPS Datasets and Benchmarks 2024)

[4] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (Lu et al., ICLR 2024)

[5] LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts (Xiao et al., arXiv 2024)

5 comments
AuthorsOpenReview ↗

Response to reviewer uuSG (1/3)

We thank the reviewer for helpful comments and suggestions. We will release the full VisRO data generation pipeline for the community.

W1

The following design choices seem to have been tuned on the test set: a) method variant, b) modulation type, c) single or multiple passes. Table 3’s Qwen2.5-VL numbers for the best-performing configuration match the numbers reported in the main table, which reports test set performance. Performance differences between some of these configurations are small (Table 3a).

We thank the reviewer for raising this. All design choices and hyperparameters were selected on a held-out validation split: 100 examples per task, disjoint from training and from the ID and OOD test sets (see Appendix B.2). The test sets were used only for final reporting. We will make this protocol more explicit in the writing.

How design choices (a) (b) (c) are chosen. For each design choice the reviewer lists, validation and test favor the same option: This validation set performance leads to the exact design choices we have in the paper and mentioned in your list.

Design choiceValidation setTest (shown in paper)
(a) Variant: AdaLoop A is bestA 63.0 vs ≤ 59.0A 64.6 vs ≤ 63.0
(b) Modulation (looped): question > constant > none63.0 > 56.7 > 47.364.6 > 60.9 > 50.9
(c) Multiple passes > single pass (AdaLoop A vs AdaLN)63.0 > 57.264.6 > 60.0

Why Table 3 matches Table 1. Table 3 ablates the default AdaLoop-A model. Its reference row is the same checkpoint as Table 1, evaluated once on test, and every other row changes one component. Identical numbers in the shared row are therefore expected and reflect a consistent evaluation, not selection on the test set.

Difference between Table 3(a) - AdaLoop A/B/C configurations. In Table 3(a), AdaLoop-A’s performance gain is statistically significant: it outperforms C (p = 0.03) and B (p < 10⁻⁴) on the 3,600-item ID test set, using paired exact McNemar tests.

W2

Qwen2.5-VL’s base performance in all of the external datasets reported in the paper is higher than its trained counterpart. The model seems to get weaker in all tasks other than VisRO ones after training with AdaLoop on VisRO.

Thanks to the reviewer for this question and please refer to global response G1. For detailed analysis:

    1. Fine-tuning creates a trade-off between learning new tasks and retaining existing capabilities. Overall, AdaLoop gains substantially on the target task (VisRO), transfers to related visual reasoning, and retains general VQA well. On VisRO, AdaLoop achieves 64.6% ID and 32.4% OOD accuracy, compared with 49.9% and 23.9% for SFT (Table 1). These gains also transfer to vision-centric OOD benchmarks, including V*Bench and BabyVision: AdaLoop at T=3 outperforms pretrained base, while the single-pass AdaLN control does not. On general VQA benchmarks, all trained models (plain SFT, single-pass AdaLN and AdaLoop) perform slightly below the pretrained base. This pattern is consistent with forgetting under limited public replay data. As the comparison below shows, AdaLoop preserves these capabilities better than single-pass training at matched training compute.
    1. At approximately matched training FLOPs, AdaLoop better preserves general VQA performance. At the lower budget, depth-sampled AdaLoop matches the 50k-update single-pass model on VisRO and improves three of four VQA benchmarks. At the higher budget, deeply supervised AdaLoop achieves competitive VisRO performance and exceeds the 120k-update single-pass model on all four VQA benchmarks.
    1. These results demonstrate AdaLoop’s data and update efficiency. With the training mixture held fixed, AdaLoop achieves strong visual reasoning and better VQA retention with fewer example presentations at comparable compute. Expanding the mixture to more diverse tasks is a complementary direction.
AuthorsOpenReview ↗

Response to reviewer uuSG (2/3)

Q1

Figure 7b reports lower numbers for both AdaLN and AdaLoop (sampled or deeply supervised). Does training a model on AdaLoop harm pre-existing knowledge and performance? Could you add more OOD comparisons using external datasets and include the base model’s performance?

Pre-existing knowledge. After continual training on our mixture, all variants (SFT, the single-pass AdaLN control and AdaLoop) score below the pretrained base on general benchmarks. The declines in Fig. 7b are due to forgetting caused by limited access to replay data rather than an effect of AdaLoop (see W2 and the global response). Among the models we trained on the same mixture, AdaLoop retains general VQA comparably or better.

External OOD benchmarks. We evaluated on the suggested benchmarks, plus V*Bench and BabyVision, which test fine-grained search, discrimination, and visual tracking similar to the visual reasoning capabilities that VisRO targets. All those are evaluated with our VisRO-trained model checkpoints, without benchmark-specific training.

Table 1. Zero-shot visual reasoning transfer (accuracy, %)

BenchmarkBaseSFTAdaLNAdaLoop-A T=1AdaLoop-A T=2AdaLoop-A T=3
V*Bench58.159.258.160.261.862.3
BabyVision†28.928.924.427.428.930.4

†Multiple-choice subset.

On vision-centric V* and BabyVision, accuracy increases with depth (V*: 60.2 → 61.8 → 62.3; BabyVision: 27.4 → 28.9 → 30.4), and T=3 exceeds the base, SFT and the single-pass control.


The suggested math and puzzle benchmarks emphasize skills outside our training mixture. PuzzleVQA needs abstract pattern induction, AlgoPuzzleVQA adds algorithmic problem solving, and MathVista, MathVision and LogicVista add mathematical and logical reasoning. These benchmarks evaluate retention rather than gains. We evaluated all five as suggested.

Table 2: Math and logic retention (accuracy, %):

BenchmarkBaseSFTAdaLNAdaLoop-A T=1AdaLoop-A T=2AdaLoop-A T=3
MathVista testmini64.455.255.353.853.851.7
MathVision17.217.517.418.117.817.9
LogicVista35.736.832.433.533.033.3
PuzzleVQA41.532.533.134.033.533.9
AlgoPuzzleVQA28.531.931.331.030.830.9
Mean37.534.833.934.133.833.5

On MathVista and PuzzleVQA, models SFT, AdaLN and AdaLoop all fall below the base. On MathVision and AlgoPuzzleVQA, all three are at or above the base. On average, AdaLoop is within 0.7–1.3 points of SFT and on par with the single-pass AdaLN control. This is consistent with these skills lying outside our training mixture, and it contrasts with the depth gains observed on V*Bench and BabyVision above. Extending our AdaLoop model to train on a richer data mixture is a natural future direction.

Q2

Did you use a validation set? How did you tune the method configurations themselves, and how did you do hyperparameter selection for each configuration? Did the baselines receive the same treatment?

Please see W1. Yes, we used a validation set, and no design choice was tuned on the test set. All hyperparameters and design choices were selected on a held-out validation split (100 examples per task; see Appendix B.2), which is disjoint from the training data and from the ID and OOD test splits. The test splits were used only to report final results. Baselines also receive the same treatment.

What was selected on validation. Within each backbone, the following were selected on the validation split: the learning rates of the LoRA adapters, connector, and conditioning modules; the depth weighting; the loop span. All methods, including the SFT and single-pass AdaLN baselines, were tuned with the same protocol (Appendix B.1). Within each comparison, models share the training mixture, the number of optimizer updates, and seed-paired initialization and data order.

AuthorsOpenReview ↗

Response to reviewer uuSG (3/3)

Q3

Why did you assign larger weights to later depths (Eq (3))? Was this choice ablated?

Yes. Deep supervision trains every iteration to produce the final answer, which is what makes adaptive stopping possible. We weight the deepest iteration most because it is the only point where the full loop computation is available. We use Geometric weights: wt∝rTtrain−tw_t \propto r^{T_{\mathrm{train}} - t}. With T_train = 3, r = 0.6 gives (0.18, 0.31, 0.51), used for Qwen2.5-VL and LLaVA, and r = 0.8 gives (0.26, 0.33, 0.41), used for PaliGemma 2.

Our intuition is that the first iteration is closest to the pretrained single-pass computation. If it receives a large share of the objective, training can settle on a solution that relies on that pass and makes little use of later iterations. Recurrent optical-flow models make a similar choice: RAFT (Teed & Deng, 2020) supervises its sequence of updates with exponentially increasing weights.

We ablated r separately on two backbones (validation accuracy, %). On PaliGemma 2, r = 0.8 is best and on Qwen2.5-VL, r = 0.6 works better than r = 0.8 and uniform weighting. We have added these ablation results to the appendix.

Table 3. Depth weighting ablations.

Qwen2.5-VL-3B

Ratio rWeights (w₁, w₂, w₃)Validation accuracy (%)
0.6 (used in the paper)(0.18, 0.31, 0.51)63.0
0.8(0.26, 0.33, 0.41)61.0
Uniform(0.33, 0.33, 0.33)60.7

PaliGemma-2 3B

Ratio rWeights (w₁, w₂, w₃)Validation accuracy (%)
0.8 (used in the paper)(0.26, 0.33, 0.41)73.7
0.5(0.14, 0.29, 0.57)72.2
0.4(0.10, 0.26, 0.64)71.3

Q4

Why did the SFT baseline’s adaptation exclude the vision encoder?

All methods in the paper (SFT, single-pass AdaLN, and AdaLoop) keep the pretrained vision weights frozen. This isolates the question we study: whether the encoder's existing transformations can be redirected by the question and reused across iterations. Keeping the tower frozen is a strength of AdaLoop rather than a limitation: every pretrained vision weight stays untouched, and all gains in the paper come from re-using these weights recurrently under question-conditioned modulation, with only a small trainable part.

Fine-tuning the encoder makes recurrence even more important for OOD generalization. We fine-tuned the vision encoder in all three methods with the same recipe and 40k updates. Every method improves, and AdaLoop remains the best. (see global response G3)

Table. G3 Qwen2.5-VL-3B Test accuracy on VisRO ID/OOD

Arm (full vision FT)IDOOD
SFT67.540.2
AdaLN70.841.1
AdaLoop A, T_train=3 (p=1)73.748.7
AdaLoop A, T_train=4 (p=1)74.450.7

Q5

It is surprising to me that GRPO has so little effect in a task that’s clearly verifiable. Did you tune the hyperparameters for the GRPO baseline?

1. GRPO is not under-tuned; its test gains are small across the whole sweep.

In the paper (Table 2), GRPO changes test accuracy by −0.4 ID / +0.3 OOD relative to SFT. For the rebuttal, starting from the SFT checkpoint, we swept the following:

  • Learning rate {3×10−63 \times 10^{-6}, 10−510^{-5}, 3×10−53 \times 10^{-5}} × KL coefficient {10−310^{-3}, 10−210^{-2}}
  • Responses per prompt {8, 16, 32}
  • Epochs {1, 2, 4}
  • Rollout batch {64, 256} prompts per step

This gives 11 configurations and 14 checkpoints, including the best-validation ones, all evaluated on the full test sets.

Relative to the SFT start (49.9 ID / 23.9 OOD):

  • ID: Mean change +0.2 (SD 0.3, best +0.8)
  • OOD: Mean change −0.5 (SD 0.5, best +0.3)

Validation accuracy moves only within its noise (n=600n=600, SE ≈ 2 points; the best of the 14 checkpoints is +2.2), and test accuracy stays flat: the best-validation checkpoint scores 50.0 ID / 23.9 OOD.

2. A possible bottleneck is the information available in the visual embeddings.

Verifiable answers provide reliable feedback about output correctness, but successful reasoning also depends on extracting the necessary visual information. Our GRPO baseline uses the same configuration as all other baselines: it updates language adapters while keeping the vision encoder frozen. It can therefore learn to use the existing visual features better, but cannot modify how those features are extracted.

If the single-pass representation misses fine-grained details needed for a question, further policy optimization may have limited benefit. Consistent with this, changing the visual computation (AdaLoop-A) gains +14.7 ID over SFT, while GRPO on top of either model adds less than one point (see i2hm Q3).

Reviewer_i2hmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an iterative computation technique ("AdaLoop") for current VLMs wherein question-conditioned recurrence is integrated into the frozen visual encoder to selectively retrieve reasoning relevant operations, and thereby enable it to "look repeatedly" in contrast to a single pass. The authors also introduce "VisRO", a procedural dataset specifically designed to evaluate iterative visual reasoning capabilities including 6 in-distribution task families such as "path tracing", "mazes", etc and also 4 out-of-distribution task families. Experimental results indicate AdaLoop improves performance over zero-shot, SFT and Single pass AdaLN over multiple open-weight VLMs including Qwen2.5-VL-3b, PaliGemma2-3B and LLaVa-1.5-7B. Ablations are also provided to study variants of the language interface and modulation and recurrence; along with further comparisons with GRPO and LIVR and analysis of loop placement as well as inference-time scaling beyond the trained depths.

Critical strengths
  1. The idea and method are relatively straightforward and can be applied to multiple VLMs. The experiments show the method has useful improvements over parameter-matched AdaLN and regular SFT by 2.8-4.6% accuracy across three VLM families, with improvements generalizing to out-of-distribution setup as well. The gains are also seen at fixed depth (as indicated in Table 5), indicating that they do not have strong dependence on adaptive answer selection.

  2. The ablations and supporting analysis is useful and comprehensive in isolating the contributions of modulation, question conditioning and recurrence, besides determining the optimal loop placement and inference-time scaling tradeoffs. The performance benefits over traditional SFT->GRPO pipeline are also encouraging to the method. Further, the introduced evaluation dataset VisRO is also useful and apt to study iterative reasoning capabilities of VLMs with its controlled difficulty and programmatically verifiable answers.

Critical weaknesses
  1. The proposed method's benefits over single-pass models is not clear when training compute is matched. Specifically, in fig. 7, AdaLoop achieves roughly similar average ID/OOD accuracy at the lower compute budget (46.51 vs. 46.40) and lower accuracy at the higher compute budget (48.47 vs. 49.85). Thus, it is unclear whether the method provides consistent accuracy improvements over single-pass models at larger training compute budgets.
  • Separately, adaptive inference also costs 2.16x the single-pass baseline compared with 1.33x for fixed T=3 (as shown in Table 7), despite only marginal accuracy gains over fixed depth. A direct accuracy-cost comparison against different fixed depths (T=1,2,3) would better establish whether the proposed adaptive stopping policy is actually beneficial.
  1. The recurrence benefits are only evaluated on the authors' introduced VisRO dataset and there is no indication of generalization of the method to further real-world visual reasoning benchmarks or datasets. Currently, external VQA evaluations only use T=1, and it would be useful to evaluate the same at (T=1,2,3) on an independent visual reasoning benchmark such as MMStar [1] and V*Bench [2].

  2. (Relatively less critical) Authors can also consider more recent VLMs/MLLMs such as Gemma-4-E4B and Qwen3.5 to show applicability of their method wrt to more recent and stronger models. Further, there are multiple works that introduce recurrence/iterative compute for multi-pass visual reasoning such as Multi-Step Visual Reasoning [3], IPRM [4], Visual Sketchpad [5] and that should either be evaluated as baselines or differentiated against in terms of method novelty.

References [1] Are we on the right way for evaluating large vision-language models? (NeurIPS 2024)

[2] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (CVPR 2024)

[3] Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025)

[4] Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios (NeurIPS 2024)

[5] Visual SKETCHPAD: Sketching as a Visual Chain of Thought for Multimodal Language Models (NeurIPS 2024)

Questions for authors

Please see critical weaknesses, particularly:

  1. Do additional visual iterations improve performance on an independent benchmark, such as MMStar and V*Bench?
  2. How does adaptive stopping compare with fixed (T=2,3,4) in accuracy and total inference cost?
  3. How does AdaLoop perform in combination with GRPO?
  4. (Relatively minor) How does AdaLoop perform with more recent models such as Gemma4 and Qwen3.5/6? And how does the method differ from existing iterative visual reasoning methods such as Multi-Step Visual Reasoning, IPRM and V* visual search?
4 comments
AuthorsOpenReview ↗

Response to reviewer i2hm (1/2)

W1.1

The proposed method's benefits over single-pass models is not clear when training compute is matched......it is unclear whether the method provides consistent accuracy improvements over single-pass models at larger training compute budgets.

Our matched-compute evaluation measures both held-out VisRO accuracy and general VQA performance. The goal is to improve visual reasoning while preserving broader capabilities. All methods train on the same mixture of 600k VisRO and 100k LLaVA-NeXT examples, allowing us to compare how they allocate additional compute across these objectives.

Figure 7 shows a favorable balance: at the lower budget, AdaLoop matches VisRO accuracy and improves three of four general-VQA benchmarks. At the higher budget, it trails the single-pass control by 1.38 points on VisRO but exceeds it by 1–4 points on all four general-VQA benchmarks and 3.7 points on BabyVision, an external visual reasoning benchmark. Our additional matched-training-FLOPs experiment also demonstrates a VisRO ID gain: depth-sampled AdaLoop trained for 90k updates achieves 66.50%, compared with 65.53% for AdaLN trained for 120k updates.

We also test whether longer training closes the gap. On an extended 24-task VisRO benchmark, we train every arm for 120k–160k updates—3–4× the paper’s schedule. Accuracy plateaus for all arms over these schedules, while AdaLoop remains ahead of both single-pass baselines on ID and OOD across three backbones:

Table 1. Model accuracy on VisRO-24 with extended training

Backbone (updates)SFTAdaLNAdaLoop
Qwen2.5-VL-3B (160k)57.6 / 17.967.1 / 23.069.5 / 25.2
PaliGemma (120k)62.3 / 18.866.7 / 23.869.9 / 24.5
LLaVA-1.5-7B (160k)52.2 / 17.358.7 / 19.160.6 / 21.1

These extended experiments match optimizer updates across methods; AdaLoop uses p=1 stopping. The 24-task OOD split is harder than the paper’s, accounting for the lower absolute OOD scores.

We will clarify that Table 1 establishes accuracy gains at matched updates, while Figure 7 evaluates target-task accuracy, capability retention, and external transfer at approximately matched training compute.

W1.2

A direct accuracy-cost comparison against different fixed depths (T=1,2,3) would better establish whether the proposed adaptive stopping policy is actually beneficial.

Please see our response to Q2.

W2

The recurrence benefits are only evaluated on the authors' introduced VisRO dataset. Currently, external VQA evaluations only use T=1, and it would be useful to evaluate the same at (T=1,2,3) on an independent visual reasoning benchmark such as MMStar and V*Bench.

Please see our response to Q1 and global response G1.

W3

Authors can also consider more recent VLMs/MLLMs such as Gemma-4-E4B and Qwen3.5 to show applicability of their method wrt to more recent and stronger models. Discuss related work on recurrence / iterative compute for multi-pass visual reasoning.

Please see our response to Q4 and global response G2.


Q1

Do additional visual iterations improve performance on an independent benchmark, such as MMStar and V*Bench?

We evaluated the same deeply supervised Qwen2.5-VL AdaLoop-A checkpoint at fixed inference depths T=1, 2, and 3 on MMStar and V*Bench. We also added evaluation on BabyVision (multiple-choice questions subset). All fine-tuned models use the same VisRO training mixture and 40k updates and are evaluated zero-shot on independent benchmarks.

Table 2. External benchmark results

ModelV*Bench (%)BabyVision (%)MMStar (%)
Pretrained base58.128.954.7
SFT59.228.951.7
Single-pass AdaLN58.124.451.1
AdaLoop-A, T=160.227.451.1
AdaLoop-A, T=261.828.950.5
AdaLoop-A, T=362.330.449.9

Additional iterations help on vision-centric benchmarks. On V*Bench and BabyVision, accuracy increases monotonically with depth. At T=3, AdaLoop exceeds the single-pass AdaLN control by 4.2 points on V*Bench and 6.0 on BabyVision, and is above the pre-trained base and SFT on both. MMStar shows a different pattern, and all depths stay within 1.2 points of the AdaLN control. The gap to the base model is shared by SFT and reflects the broader capability-retention gap (see the global response G1).

We have included these external depth comparisons in the PDF revision, distinguishing general-capability retention after training from the benefit of additional iterations for OOD visual reasoning transfer (V*Bench and BabyVision).

AuthorsOpenReview ↗

Response to reviewer i2hm (2/2)

Q2

How does adaptive stopping compare with fixed (T=2,3,4) in accuracy and total inference cost?

AdaLoop’s benefits do not depend on adaptive stopping: the fixed-depth results in Appendix C.1 already demonstrate improvements over single-pass controls. For Qwen2.5-VL, adaptive stopping achieves the highest accuracy with an additional inference cost: 2.16× the single-pass baseline, compared with 1.33× for fixed T=3. We therefore view adaptive stopping as an optional setting for maximizing accuracy, while fixed-depth inference provides a lower-cost option.

The table below compares every fixed depth with the adaptive policy. The model is trained with T_train = 3 and deep supervision at every depth, so T = 1, 2 and 3 are all depths seen during training. T = 4 runs the loop one step beyond any training depth and is therefore extrapolation.

Table 3. AdaLoop (Qwen2.5-VL-3B) fixed depth accuracy with T_train=3

Inference policyVisRO ID (%)VisRO OOD (%)Relative inference FLOPs
Fixed T=161.8330.441.00×
Fixed T=264.0832.361.16×
Fixed T=363.6732.361.33×
Fixed T=4* (extrapolation)62.2831.221.48×
Adaptive (p=1, max T=3)64.5632.392.16×

Adaptive stopping gives the highest ID accuracy and matches the best fixed-depth OOD accuracy. Its higher cost arises from generating intermediate answers to check agreement, whereas fixed-depth inference generates an answer only at the selected depth. We have included the full comparison in the Appendix C.1 (Table 11), so readers can judge whether the incremental gain is worth the additional computation for their application.

Q3

How does AdaLoop perform in combination with GRPO?

AdaLoop and GRPO act on different parts of the model, so they can be combined. GRPO updates the language model’s output distribution, while AdaLoop changes the visual computation that feeds the language model.

We ran GRPO from two starting points (1) SFT checkpoint (2) the AdaLoop-A checkpoint, with rollouts through the looped encoder at T=3. GRPO improves either starting point by less than one point (SFT: +0.2 ID on average; AdaLoop-A: +0.8 ID / +0.5 OOD with p=1), about one standard error at n=3,600. AdaLoop’s gain over SFT (+14.7 ID / +8.5 OOD) is more than ten times larger, and AdaLoop + GRPO gives the highest overall performance. GRPO therefore combines with AdaLoop without conflict, but does not substitute for changing the visual computation.

Q4

How does AdaLoop perform with more recent models such as Gemma4 and Qwen3.5/6? And how does the method differ from existing iterative visual reasoning methods such as Multi-Step Visual Reasoning, IPRM and V* visual search?

Please see global response G2.

Newer backbones. We applied AdaLoop to Qwen3-VL-4B, Qwen3.5-4B and Gemma-4-E4B and evaluated on VisRO-Core6 with adaptive stopping (p = 1). Results are ID / OOD accuracy (%). The gain of AdaLoop-A carries over to all three newer backbones.

MethodQwen3-VL-4BQwen3.5-4BGemma-4-E4B
Zero-shot12.3 / 9.410.1 / 7.811.1 / 9.3
SFT62.4 / 32.260.8 / 30.850.6 / 25.3
Single-pass AdaLN74.1 / 44.577.8 / 47.051.3 / 26.9
AdaLoop-A77.2 / 47.280.5 / 49.770.0 / 39.1

Relation to iterative visual reasoning methods. AdaLoop differs from these methods on two axes:

  1. Where the iteration happens. AdaLoop re-applies a band of the vision encoder's own pretrained blocks, refining the visual representation in latent space, with the depth chosen adaptively at test time. Multi-Step Visual Reasoning and V* iterate at the level of language, tool calls or image regions, and Visual Sketchpad has the language model draw on the image with code tools and re-read the edited image. IPRM iterates in a separate reasoning module on top of fixed features. These approaches are complementary to ours: an agent that searches, sketches or calls tools could use an AdaLoop encoder as its visual backbone.

  2. Supervision. AdaLoop is trained with final-answer supervision alone and needs no reasoning traces. Multi-Step Visual Reasoning trains on large amounts of tool-use and visual-reasoning traces over a predefined action space (Sec. 4.1 of their paper). V* relies on an LLM's world knowledge to guide its visual search. IPRM trains a small task-specific reasoner separately on each dataset, rather than adding general reasoning capacity to a pre-trained VLM.

We have added these discussions to the related work section of the revised PDF.

Thank you for the comprehensive rebuttal; my concerns are largely addressed and I trust authors will include these updates in updated version. I will maintain my score for accept as I believe the method and results are good contributions.

Thank you for your thoughtful feedback and positive assessment. We have incorporated the additional results and discussions from our rebuttal into the revised PDF. These updates appear in the main paper and appendix and are highlighted in blue for reference.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global response to reviewers

We thank all reviewers for their thoughtful feedback and positive assessment of the paper’s clarity. Reviewers highlighted AdaLoop’s straightforward design and applicability to existing VLMs, its empirical improvements across model families, and the informative ablations isolating its key components. We also appreciate the recognition of VisRO’s value for controlled studies of visual reasoning.

Below, we address the main concerns, the evidence supporting our conclusions, and new experimental results.

G1 - Evaluation on external benchmarks

Reviewers (i2hm, uuSG) asked how AdaLoop affects performance outside of VisRO, so we tested several external benchmarks. The benchmarks measure different things and so we see distinct behaviors. There are three categories:

  • (1) Vision-centric reasoning: V*Bench and BabyVision. These test operations that are similar to VisRO: fine-grained search, discrimination, and visual tracking, with little reliance on external world knowledge. They measure zero-shot transfer of our trained model to novel visual reasoning tasks. Here, additional visual iterations in AdaLoop help. On V*Bench, AdaLoop-A improves from 60.2% at T=1 to 62.3% at T=3, above the pretrained base (58.1%), SFT (59.2%) and the single-pass AdaLN control (58.1%). On BabyVision (135 multiple-choice questions), accuracy rises from 27.4% to 30.4% across T=1–3, exceeding all baselines at T=3.

  • (2) General VQA and perception: MMStar, MMBench and the benchmarks in Fig. 7. These test capabilities that come from the base model's original training. Fine-tuning on new tasks tends to erode them, a well-documented forgetting effect in continual and multimodal fine-tuning [Rolnick et al., 2019; He et al., 2023]. The standard mitigation is replaying data from the original distribution. Qwen2.5-VL's and PaliGemma 2's instruction-tuning mixtures are not public, so all of our models replay a fixed public 100k subset of LLaVA-NeXT, which is almost certainly less diverse than the original recipe. As a result, after continual training on our mixture, every model variant (plain SFT, the single-pass AdaLN control and AdaLoop) scores below the pretrained base on some of these benchmarks. The decline comes from the shared training mixture, not from recurrence. These benchmarks therefore measure retention, and the relevant comparison is against controls trained on the same mixture, where AdaLoop is comparable or better (Fig. 7).

  • (3) Math- and logic-heavy visual reasoning: MathVista, MathVision, LogicVista, PuzzleVQA, AlgoPuzzleVQA. These test mathematical, algorithmic or abstract-pattern reasoning that appears in neither VisRO-Core6 nor the replay data. We do not expect gains from our mixture on these. The results bear this out: averaged over the five benchmarks, the base model scores 37.5%, SFT 34.8%, the single-pass AdaLN control 33.9%, and AdaLoop 33.5–34.1% across T=1–3. The drop from the base model is shared by all fine-tuned variants, and AdaLoop stays within 0.7-1.3 points of SFT on average. Extending the training mixture with math and puzzle-style data is a natural, complementary direction.

G2 - Newer VLM backbones (i2hm W3 & Q4)

We extended our experiments to three newer VLM backbones: Qwen3-VL-4B, Qwen3.5-4B and Gemma-4-E4B. On all of them, AdaLoop outperforms SFT and the single-pass AdaLN control on VisRO ID and OOD, demonstrating that its gains extend to newer architectures.

Table G2. Test accuracy (%) on VisRO ID / OOD for newer VLM backbones

MethodQwen3-VL-4BQwen3.5-4BGemma-4-E4B
Zero-shot12.3 / 9.410.1 / 7.811.1 / 9.3
SFT62.4 / 32.260.8 / 30.850.6 / 25.3
Single-pass AdaLN74.1 / 44.577.8 / 47.051.3 / 26.9
AdaLoop-A77.2 / 47.280.5 / 49.770.0 / 39.1

G3 - Full fine-tuning of the vision tower (uuSG Q4)

AdaLoop’s gains persist with both frozen (main paper) and unfrozen vision encoders. With the vision tower unfrozen, AdaLoop’s OOD advantage grows: AdaLoop-A (T_train = 3) reaches 73.7 / 48.7 vs 70.8 / 41.1 for single-pass AdaLN and 67.5 / 40.2 for SFT.

Table. G3 Qwen2.5-VL-3B Test accuracy on VisRO ID/OOD

Arm (full vision FT)IDOOD
SFT67.540.2
AdaLN70.841.1
AdaLoop A, T_train=3 (p=1)73.748.7
AdaLoop A, T_train=4 (p=1)74.450.7

G4 - RL / GRPO with SFT base model and AdaLoop (i2hm Q3, uuSG Q5)

GRPO from the SFT checkpoint gives small test gains, and this is not a tuning issue. Across 11 hyperparameter configurations, the mean change is +0.2 ID / −0.5 OOD, with validation changes within noise. GRPO started from AdaLoop-A adds +0.8 ID / +0.5 OOD. AdaLoop’s own gain over SFT (+14.7 ID / +8.5 OOD) is more than ten times larger. Post-training with GRPO therefore complements AdaLoop but does not replace it.

AuthorsOpenReview ↗

PDF revision

We have uploaded the revised PDF, incorporating the additional experiments and discussions from our rebuttal into the main paper and appendix. All changes are highlighted in blue.

We thank the reviewers again for their helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.