- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AdaLoop, which employs a looped transformer for the vision encoder, which incorporates information from the natural language input through a small learnable network that produces hyperparameters for a new layer normalization step. The authors argue that these two work together organically so that the visual encoder now considers the natural language input when producing the visual representations, as opposed to default setups where the visual representations are produced once without conditioning on the natural language input. Empirical experiments show that the proposed method, AdaLoop, demonstrates a pretty significant gain over vanilla VLMs (three tested) on visual reasoning benchmarks, and the ablation studies clearly separate and discuss where the gains come from.
Critical strengths
The proposed AdaLoop method is fairly straightforward, well motivated, and demonstrates clear benefits on benchmarks. It can also be applied fairly easily to existing models without having to retrain with a huge amount of data, providing real usability benefits.
Critical weaknesses
The proposed loop method is not extremely well coupled with the motivation of using questions to condition the visual encoder, as the empirical evidence shows, the model gains without the loops.
Questions for authors
Could you justify a bit more on why the loop has to co-live with the current question intervention in this paper? I can see the long term benefit, but the current method of adding the layernorm does not seem to well motivate a loop transformer.
2 comments
I thank the authors for the rebuttal. I will maintain my positive rating, which reflects the contribution of this paper.
Response to Reviewer Xfdh
W1 & Q1
We thank the reviewer for the question. Our goal is to make better use of a vision encoder’s learned representations and transformations by adapting visual processing to each question and its difficulty. This motivates two complementary components: question-conditioned adaptive layer normalization to guide visual processing, and iterative computation / looping to refine visual representations as needed. Our experiments demonstrate the benefits of combining these components, while our mathematical analysis offers intuition for why they work together.
Empirically, the ablations show that each component contributes beyond the other. Table 3(b) provides two controlled comparisons. With question conditioning held fixed, adding recurrence improves accuracy by 4.6 ID and 2.2 OOD percentage points over single-pass AdaLN. Conversely, with recurrence and parameter count held fixed, replacing question-independent modulation with question-conditioned modulation improves accuracy by 3.7 ID and 2.4 OOD points. Simply repeating the encoder without modulation yields only a small ID improvement. These results support the combination: conditioning makes repeated use of the pretrained blocks more effective, and recurrence provides additional gains beyond conditioning alone.
We also provide mathematical intuition for this design in Appendix A, which explains why modulation and recurrence are complementary. Under a first-order approximation, question-conditioned modulation can be interpreted as adjusting the “effective depth” of an individual transformer block. However, visual reasoning may also benefit from later computations informing earlier ones, allowing bottom-up and top-down processing to interact across iterations. Looping enables this feedback by feeding the output of later blocks back into earlier blocks within the repeated encoder span. Modulating individual blocks during a single forward pass does not provide this feedback path. Thus, question conditioning adjusts how the pretrained transformations contribute, while recurrence allows those transformations to be recomputed using updated visual information.
The loop’s contribution persists with more capacity and on newer backbones. When the vision tower is fully fine-tuned, the looped model's OOD advantage over single-pass AdaLN grows to 9.6 points at T_train = 4 (50.7 vs 41.1), and on Qwen3-VL-4B and Qwen3.5-4B the looped model again exceeds AdaLN by about 3 ID / 2.7 OOD points (see global response G2).