acceptodds
Under review as a conference paper at ICLR 2027

Retrofitting Recurrent Vision–Language Models from Pretrained Backbones

Abstract

Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Depth Recurrent Transformers, Recurrent VLMs, Retrofitting Recurrence, Looped Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BXJbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates using depth recurrence to reduce the parameters of VLMs. It proposes RR-VLM, which prunes some layers from the language backbone, repeatedly applies the retained middle block, and uses a linear adapter to fuse representations. The authors also propose a three-stage curriculum to recover model performance. The method is evaluated on TinyLLaVA.

Critical strengths
  1. The method has a relatively intuitive design and a clear overall pipeline. The three training stages respectively address language backbone recovery, vision-language realignment, and the recovery of instruction-following capabilities.
  2. The experimental design is relatively comprehensive, including comparisons among models trained with different combinations of training stages, which help analyze the role of recovery training. The ablation studies also thoroughly examine design choices such as layer allocation, adapters, and KV-cache sharing.
Critical weaknesses
  1. The method’s incremental contribution is limited. Looping over the middle block after pruning, the layer allocation, and fusion through a linear adapter all follow the existing design of McLeish et al. Furthermore, the experiments are conducted only on TinyLLaVA-1.4B, leaving the applicability to other pretrained VLMs unclear.
  2. The paper lacks a comprehensive comparison of actual inference latency, throughput, and training costs. In the main model, each token traverses an equivalent depth of 56 layers (mr=4+6×8+4=56m_r = 4 + 6 \times 8 + 4 = 56), compared with 22 layers in the original model. It remains unclear what practical deployment benefits arise from the reduction in parameter count.
Questions for authors
  1. Compared with the prune-then-loop scheme of McLeish et al., does this work introduce any other key changes to the architecture or training mechanisms beyond vision-language realignment and visual instruction tuning?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BXJb (part 1 of 2)

We thank the reviewer for the careful review and are glad that the reviewer finds our pipeline clear and our experiments comprehensive.

W1 and Q1: Differences to McLeish et al.

Beyond the two vision-language stages, we introduce three changes on top of the method of McLeish et al., and we evaluate each of them separately (Tables 1, 2 and 4). We reuse their recurrent architecture, but (1) we retrofit the recurrence into the language model of a pretrained VLM instead of a standalone language model, and we find that the resulting model outperforms their recurrent model extended by a vision encoder. (2) We distill from the unpruned model, which improves the looped model more than the unpruned model. (3) We repair the language backbone with a text-only stage before the projector alignment and the visual instruction tuning, as without this stage the looped model remains below the unpruned model. (4) In addition, we study the KV cache of the retrofitted model and evaluate QRC, which reduces it from 2.55× to 1.01× that of the base model without retraining (Table 6).

  • Starting point: McLeish et al. retrofit a standalone language model, whereas we retrofit the language model inside a pretrained VLM. To compare the two, we attach the vision encoder to their recurrent model and train it with our stages S₁ and S₂. With the released weights, trained on 52B tokens of Nemotron-CC-Math-v1 without distillation, this model achieves 55.59, compared with 58.13 for RR-VLM. When we repeat their conversion on our 60B text tokens with distillation, so that both use the same text data and budget, it achieves 56.05, still 2.08 pp below RR-VLM, and RR-VLM is higher than this version on every benchmark (table below).
ModelText stageGQAPOPETVQASQAMMBRWQASEEDAI2DCV2DAvg
Recurrent LM (released) + ViT52B Nemotron-CC-Math-v1, no distillation58.886.341.552.552.750.660.743.653.755.59
Recurrent LM (our S₀) + ViT60B (ours), distillation59.186.446.257.655.546.162.143.248.156.05
RR-VLM (ours)60B (ours), distillation59.286.747.759.158.950.562.644.653.958.13
  • Distillation: McLeish et al. train without a teacher. We distill from the unpruned model to limit the drift of the pruned and looped language model away from the representations to which the projector was aligned (Sec. 3.2). We apply it on text in S₀ and on both image-text and text data in S₂. In S₂, it improves RR-VLM from 56.65 to 58.13 (+1.48 pp), whereas the same distillation improves the unpruned baseline by only 0.25 pp (Table 4). We attribute this to distillation acting as a regularizer that keeps the looped model close to the pretrained one (Sec. 5.2).
  • Order of the recovery: Within a VLM, pruning also disrupts the alignment between the language backbone and the projector. We therefore first repair the language backbone with text-only training, then realign the projector, and finally fine-tune on visual instructions. We find that with only the stages S₁ and S₂, the resulting model reaches 54.45 and is 1.35 pp below the unpruned baseline (55.80), whereas with the text stage it reaches 58.13 (Table 2). Removing the projector stage reduces the performance by 1.28 pp.
  • KV cache: Since every pass through the recurrent block keeps its own keys and values, our model holds 2.55× the KV cache of the base model. We evaluate the cache sharing of Geiping et al., which was proposed for a model pretrained with recurrence, and find that it reduces the performance of our retrofitted model by 2.28 pp with two copies of the cache (1.09×) and by 5.43 pp with a single copy (Table 6). We therefore further evaluate QRC (App. A.7), which follows FlashLoop (Yang & Liu, 2026) with the quantizer of KIVI (Liu et al., 2024) and stores the keys and values of all recurrences after the first as differences to the preceding recurrence at 2 bits. We find that QRC reduces the KV cache to 1.01× that of the base model, while the performance remains on par with 58.05 against 58.09 (Table 6).
Training stages (Table 2)Avg
Unpruned baseline, S₀, S₁, S₂55.80
RR-VLM, S₁, S₂54.45
RR-VLM, S₀, S₂56.85
RR-VLM, S₀, S₁, S₂ (ours)58.13
KV cache (Table 6)Cache (× base model)Avg
One cache shared by all recurrences0.727×52.70
Two copies of the cache1.09×55.85
Three copies of the cache1.45×57.38
No sharing (ours)2.55×58.13
No sharing (SDPA attention)2.55×58.09
QRC (SDPA attention)1.01×58.05
AuthorsOpenReview ↗

Response to Reviewer BXJb (part 2 of 2)

W1: Other pretrained VLMs

Applying our pipeline to a different model requires that the pretraining data of its language model and its vision-language training data are public, as we repair the language model after pruning on text close to its pretraining data and retrain the VLM on its original vision-language data. TinyLLaVA meets both conditions and can be trained within a reasonable training budget, which is why we chose it for our experiments. For most larger VLMs, the pretraining data of the language model is not released, and in several cases neither is the vision-language data. Apart from this requirement on the data, our proposed method is not specific to TinyLLaVA, since the layer allocation, the adapter and the three training stages only assume a language model and a projector. We agree with the reviewer that a second model would strengthen our claims, and we will report results on a second model family whose language model was pretrained on public data in the camera-ready version.

W2: Latency, throughput and training cost

Following the reviewer's request, Table 7 now reports these costs for RR-VLM, both baselines and RR-VLM with QRC (table below). RR-VLM requires 2.1× the training compute and 2.47× the latency per token of the unpruned baseline (24.0 against 9.7 ms).

Training and inference cost (Table 7)RR-VLM (ours)Unpruned baselinePruned baselineRR-VLM, SDPARR-VLM + QRC, SDPA
Average accuracy, 9 benchmarks58.1355.8052.4158.0958.05
Stored weights (B parameters)1.151.411.151.151.15
Training compute incl. teacher (10¹⁸ FLOPs)1,1765544471,1761,176
Inference compute (GFLOPs per token)5.372.191.635.375.37
Latency, batch size 1 (ms per token)24.09.77.717.828.8
Throughput, batch size 16 (tokens/s)7381,8002,326918547
Peak memory, batch size 1 / 16 (GiB)2.24 / 3.452.69 / 3.582.21 / 3.052.24 / 3.452.22 / 3.11
KV cache (KiB per token)56.022.016.056.022.3

The last two columns use SDPA attention, as our implementation of QRC does. All other columns use FlashAttention-2. QRC is applied without retraining, and the inference compute does not include decoding its low-bit cache.

W2: Deployment benefit

We see the main benefit of RR-VLM in its accuracy, and the lower parameter count as an additional benefit. Looping the pruned model with 1.15B parameters raises its average accuracy from 52.41 to 58.13, which is 2.33 pp higher than the unpruned model with 1.41B parameters (Tables 1 and 7). Compared to the unpruned model, the gains are largest on CV-Bench-2D (+11.7 pp), RealWorldQA (+4.1 pp) and MMBench (+3.4 pp). With fewer stored parameters, RR-VLM also needs less memory than the unpruned model (2.24 against 2.69 GiB at batch size 1 and 3.45 against 3.58 GiB at batch size 16, Table 7). QRC reduces its KV cache from 2.55× to 1.01× that of the base model at a cost of 0.04 pp (Table 6), but increases the latency per token from 17.8 to 28.8 ms in our current implementation, both with SDPA attention (Table 7).

We thank the reviewer for these questions, which helped us to state more clearly how our method differs from McLeish et al. and what its costs and benefits are. We believe that our work is of value to the community in three respects. (1) To the best of our knowledge, it presents the first prune-then-loop retrofit of a pretrained VLM for VQA, with a recovery that uses no new vision-language data. (2) It shows that retrofitting the VLM directly outperforms adding a vision encoder to a recurrent language model, even when both use the same text data and distillation. (3) It is the first study to compare looping in a pretrained VLM against both the pruned and the unpruned model trained on the same data, and it shows that looping improves accuracy with fewer stored parameters, in particular on reasoning-oriented benchmarks (Sec. 1, Table 1). As our revisions state the differences to McLeish et al. and add the training and inference costs requested by the reviewer, we kindly ask the reviewer to consider raising the soundness and overall ratings, and we would be happy to discuss any further points.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer BXJb

We thank the reviewer again for their time and their review. As the discussion period ends on November 18 (AoE), we would like to ask whether our response addresses the reviewer's concerns. In our reply, we state the differences to McLeish et al. and compare against their recurrent language model extended by a vision encoder and trained on the same text data with distillation (56.05 against 58.13 for RR-VLM), we report the training and inference costs in Table 7, and we explain why we chose TinyLLaVA. We would be happy to answer any further questions, and if the concerns are resolved, we kindly ask the reviewer to consider updating their score.

Reviewer_DbYpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper converts a pretrained VLM into a recurrent model by pruning language layers and looping a remaining block. A three-stage recovery procedure with distillation improves the nine-benchmark average while reducing stored language-model parameters.

Critical strengths
  • The retrofit gives a useful way to reuse an existing VLM with fewer stored parameters.
  • Retrained unpruned and prune-only baselines help separate recurrence from recovery training.
  • The ablations explain the roles of the training stages, adapters, distillation, and cache sharing.
Critical weaknesses
  • The weight savings come with more computation and cache. The main model traverses 56 rather than 22 layers and uses 2.55 times the baseline KV cache, without an accuracy comparison at matched inference cost.
  • The evidence is limited to TinyLLaVA-1.4B and one training seed. Several architecture choices are tested only with the smaller 7.5B-token recovery budget.
  • The comparison with a converted recurrent LM matches vision-language data, but does not clearly establish matched total text-training and distillation budgets. This limits the claim that one conversion route is better.
Questions for authors
  • Where does the main model fall on an accuracy, latency, and total-memory comparison with the unpruned and prune-only baselines?
  • What are the total text-token and teacher-compute budgets for the recurrent-LM baseline, and are they comparable to the 60B-token retrofit?
  • Do the main architecture and adapter conclusions hold with the final 60B-token recipe, rather than only the smaller ablation setup?
2 comments
AuthorsOpenReview ↗

Response to Reviewer DbYp (part 1 of 2)

We thank the reviewer for their review and their precise questions.

W1 and Q1: Accuracy, latency and memory against the baselines

Following the reviewer's request, Table 7 now reports all three models and RR-VLM with QRC (table below). RR-VLM has the highest accuracy (58.13, against 55.80 for the unpruned and 52.41 for the pruned baseline) and the highest latency (24.0 against 9.7 ms per token for the unpruned baseline at batch size 1). Its peak memory is lower than that of the unpruned baseline at batch size 1 (2.24 against 2.69 GiB) and at batch size 16 (3.45 against 3.58 GiB), but its KV cache is 2.55× that of the base model, because each recurrence adds its own keys and values. QRC (App. A.7) reduces this cache to 1.01× without retraining, at a cost of 0.04 pp (58.05 against 58.09 for the full-precision cache, both with SDPA attention, Table 6).

Training and inference cost (Table 7)RR-VLM (ours)Unpruned baselinePruned baselineRR-VLM, SDPARR-VLM + QRC, SDPA
Average accuracy, 9 benchmarks58.1355.8052.4158.0958.05
Stored weights (B parameters)1.151.411.151.151.15
Training compute incl. teacher (10¹⁸ FLOPs)1,1765544471,1761,176
Inference compute (GFLOPs per token)5.372.191.635.375.37
Latency, batch size 1 (ms per token)24.09.77.717.828.8
Throughput, batch size 16 (tokens/s)7381,8002,326918547
Peak memory, batch size 1 / 16 (GiB)2.24 / 3.452.69 / 3.582.21 / 3.052.24 / 3.452.22 / 3.11
KV cache (KiB per token)56.022.016.056.022.3

The last two columns use SDPA attention, as our implementation of QRC does. All other columns use FlashAttention-2. The inference compute does not include decoding the low-bit cache of QRC.

We agree that a comparison at matched inference compute is informative. With r recurrences, RR-VLM traverses 4 + 8r + 4 layers per token, so matching the 22 layers of the unpruned model yields r = (22 − 8)/8 ≈ 2, i.e., 24 layers. We trained the 4/8/4 model with a fixed number of recurrences from r = 1 to r = 8. We ran this study in the less expensive setting of our other ablations in Table 3 (7.5B-token text stage, no text distillation) and report the average over the nine benchmarks.

r12468
Layers per token1624405672
Average51.9954.6856.2055.7556.27

The accuracy increases up to r = 4 and then plateaus, staying within about 0.5 pp between r = 4 and r = 8. Our choice of r = 6 lies on this plateau with 55.75 (Table 3). At matched inference compute (r = 2), the model reaches 54.68, which is 1.07 pp below r = 6 in the same setting and also below the unpruned model without the text stage (55.27, Table 1), so in this setting the accuracy gain requires the additional recurrences. This is in line with the literature on language models trained from scratch, where Schwethelm et al. (2026) find that replacing unique layers with recurrences increases the validation loss at matched training compute. We further note that the number of recurrences cannot be chosen independently of the size of the recurrent block. Table 3(a) compares three allocations in the same setting, all at 56 traversed layers. There, 6/4/6 with r = 11 reaches 52.56, 4/6/4 with r = 8 reaches 54.15 and 4/8/4 with r = 6 reaches 55.75, so at a fixed depth the larger block with fewer recurrences performs best.

Due to our compute budget, we could not repeat this study with the final 60B-token text stage. For the camera-ready version, we will train the model with r = 2, which matches the inference compute of the unpruned model, on our final recipe. We hope that this addresses the reviewer's question, and we would be glad to run a different experiment if the reviewer could specify the setting they have in mind.

W2: Evaluation on one model

Applying our pipeline to a different model requires that the pretraining data of its language model and its vision-language training data are public, as we repair the language model after pruning on text close to its pretraining data and retrain the VLM on its original vision-language data. TinyLLaVA meets both conditions and can be trained within a reasonable training budget, which is why we chose it for our experiments. For most larger VLMs, the pretraining data of the language model is not released, and in several cases neither is the vision-language data. Apart from this requirement on the data, our proposed method is not specific to TinyLLaVA, since the layer allocation, the adapter and the three training stages only assume a language model and a projector. We agree with the reviewer that a second model would strengthen our claims, and we will report results on a second model family whose language model was pretrained on public data in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer DbYp (part 2 of 2)

W2 and Q3: Ablations at the 7.5B-token budget

We apologize that our compute budget did not allow us to repeat every ablation at 60B tokens, since one 60B-token text stage requires 228,874 steps on 16 GPUs (App. A.1). We therefore selected the layer allocation and the adapter at the 7.5B-token budget (Table 3), where the 4/8/4 allocation is 1.60 pp above the next allocation and the linear adapter 0.73 pp above the next adapter. The ablations of the training stages, the distillation and the text interleaving, as well as the KV-cache results, use the 60B-token text stage (Tables 2, 4 and 6, App. A.4). Upon the reviewer's request, we will provide the layer-allocation and adapter comparisons at 60B tokens in the camera-ready version.

W3 and Q2: Budgets of the recurrent-LM baseline

We thank the reviewer for raising this point. The public checkpoint of McLeish et al. was trained on about 52B tokens of Nemotron-CC-Math-v1 without a teacher. Our text stage uses 60B tokens (51B from SlimPajama and 9B from FineWeb-Edu) with a teacher, which accounts for 135×10¹⁸ of the 1,131×10¹⁸ FLOPs of this stage (App. A.1). The token budgets are comparable, but the data and the distillation differ, as Sec. 5.1 now specifies for both versions.

For a fair comparison, we repeated the conversion of McLeish et al. with six recurrences and trained the recurrent TinyLlama on our 60B text tokens with distillation from the unpruned TinyLlama, as in our approach. We then attached the vision encoder and trained S₁ and S₂ as for RR-VLM. This baseline achieves 56.05 on average, 0.46 pp above the released weights and 2.08 pp below RR-VLM, and RR-VLM is higher than this baseline on every benchmark (Table 1, Sec. 5.1):

ModelText stageGQAPOPETVQASQAMMBRWQASEEDAI2DCV2DAvg
Recurrent LM (released) + ViT52B Nemotron-CC-Math-v1, no distillation58.886.341.552.552.750.660.743.653.755.59
Recurrent LM (our S₀) + ViT60B (ours), distillation59.186.446.257.655.546.162.143.248.156.05
RR-VLM (ours)60B (ours), distillation59.286.747.759.158.950.562.644.653.958.13

We thank the reviewer for these questions, which led to the cost comparison, the memory analysis and the matched recurrent-LM experiment. In light of these additions, we kindly ask the reviewer to consider raising the soundness and overall ratings, and we would be happy to discuss any further points.

Reviewer_YX8wOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper retrofits a pretrained VLM into a recurrent model through layer pruning, block looping, and a three-stage recovery curriculum with distillation. On TinyLLaVA-1.4B, it improves average VQA accuracy over a recovery-trained unpruned baseline while reducing total stored parameters by 18.1%.

Critical strengths
  1. Strong results. The main experiments demonstrate better downstream performance while compacting the headline model from 1.41B to 1.15B.
  2. The paper presents ablation studies examine training stages, adapters, layer allocation, recurrence depth, and cache sharing. The authors also include useful evidence about which choices contribute to performance and where the approach fails.
Critical weaknesses
  1. Limited evidence of generalizability. The evaluation is restricted to TinyLLaVA-1.4B. It is therefore unclear whether the method and findings extend to other model families and substantially larger backbones. Evaluating another architecture and a larger model would strengthen the contribution, particularly by testing whether the layer allocation and recovery curriculum transfer without substantial retuning.
  2. Limited method novelty. The prelude/recurrent-block/coda architecture, linear adapter, and default layer allocation closely follow existing LLM retrofitting work. The multimodal-specific contribution appears to lie primarily in the recovery, which is highly similar to VLM pretraining and instruction SFT. Although the paper addresses a relevant adaptation problem, a stronger case is needed for the methodological advance beyond combining established components.
  3. Unclear cost-effectiveness of recovery training. The method requires substantial additional training, including 60B text tokens and teacher distillation. The recovery-trained unpruned baseline is a useful control for additional data, but matching data does not match training computation. The authors should report recovery FLOPs or GPU-hours, including teacher overhead, and comparing performance under matched training-compute budgets, to better justify if the gains are worth the adaptation cost.
  4. Limited evidence of overall inference efficiency. The headline configuration reduces total parameters by 18.1%, but increases transformer layer traversals per token from 22 to 56 and uses 2.55× the baseline KV cache. Parameter savings do not translate to lower latency or total inference memory. The early-exit results are useful, but comparisons against the original model using measured latency, throughput, and peak memory are needed. Furthermore, the fixed-depth model is unstable at other depths (Table 5), while the sampled-depth model saturates, leaving the test-time-scaling motivation unsupported by the authors own results.
Questions for authors

Please see weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YX8w (part 1 of 2)

We thank the reviewer for their time and are glad that they find our results strong and our ablations useful. We would also like to note that the listed weaknesses do not question the correctness of our method or results.

W1: Generalization to other models

Applying our pipeline to a different model requires that the pretraining data of its language model and its vision-language training data are public, as we repair the language model after pruning on text close to its pretraining data and retrain the VLM on its original vision-language data. TinyLLaVA meets both conditions and can be trained within a reasonable training budget, which is why we chose it for our experiments. For most larger VLMs, the pretraining data of the language model is not released, and in several cases neither is the vision-language data. Apart from this requirement on the data, our proposed method is not specific to TinyLLaVA, since the layer allocation, the adapter and the three training stages only assume a language model and a projector. We agree with the reviewer that a second model would strengthen our claims, and we will report results on a second model family whose language model was pretrained on public data in the camera-ready version.

W2: Novelty of the method

We agree that our architecture follows McLeish et al. However, we find that combining it with the LLaVA recipe, which TinyLLaVA follows, does not outperform the unpruned model. With only the two vision-language stages S₁ and S₂, the resulting model reaches 54.45 and is 1.35 pp below the unpruned baseline (55.80), even with our distillation in S₂ (Table 2). Adding the text-only stage S₀, which repairs the pruned language backbone, raises RR-VLM to 58.13, which is 2.33 pp above the baseline. Our instruction tuning also differs from that of LLaVA, as we interleave batches of pretraining text and distill from the unpruned model. This distillation improves RR-VLM by 1.48 pp, of which 1.25 pp come from the distillation on the text batches, but improves the unpruned model by only 0.25 pp (Table 4). Overall, it shows that the gain of our pipeline over the unpruned model requires the text-only stage and is further increased by the distillation, neither of which is part of the LLaVA recipe.

Training stages (Table 2)Avg
Unpruned baseline, S₀, S₁, S₂55.80
RR-VLM, S₁, S₂54.45
RR-VLM, S₀, S₂56.85
RR-VLM, S₀, S₁, S₂ (ours)58.13
Knowledge distillation (Table 4)Avg
Unpruned baseline, with distillation in S₂55.80
Unpruned baseline, without distillation in S₂55.55
RR-VLM, without distillation in S₂56.65
RR-VLM, distillation on image-text batches56.88
+ distillation on text batches, τ = 257.24
+ distillation on text batches, τ = 1 (ours)58.13

We now address the reviewer's point that our method combines established components. Our contribution is to establish whether looping benefits a pretrained VLM. To the best of our knowledge, RR-VLM is the first prune-then-loop retrofit of a pretrained VLM for VQA, and our study is the first to compare looping against both the pruned model without looping and the unpruned model trained on the same data, which quantifies the benefit of looping. As the table below shows (from Table 1), RR-VLM outperforms all baselines on average with 23.3% fewer language-model parameters than the unpruned model, and the largest gains over the unpruned model appear on the reasoning-oriented benchmarks CV-Bench-2D (+11.7 pp), RealWorldQA (+4.1 pp) and MMBench (+3.4 pp). We also show that retrofitting a pretrained VLM directly works better than adding a vision encoder to a pretrained recurrent language model. This holds both for the released weights of McLeish et al. and when we train their model on our 60B text tokens with distillation.

ModelLM params (B)GQAPOPETVQASQAMMBRWQASEEDAI2DCV2DAvg
Unpruned baseline1.1060.486.747.360.255.546.460.443.242.255.80
Pruned baseline0.8458.086.137.750.348.647.658.338.147.152.41
Recurrent LM (released) + ViT0.8458.886.341.552.552.750.660.743.653.755.59
Recurrent LM (our S₀) + ViT0.8459.186.446.257.655.546.162.143.248.156.05
RR-VLM (ours)0.8459.286.747.759.158.950.562.644.653.958.13
AuthorsOpenReview ↗

Response to Reviewer YX8w (part 2 of 2)

W3: Cost of the recovery training

The original submission stated the training setup (60B distilled text tokens and 228,874 steps on 16 GPUs, Sec. 4 and App. A.1), but not the resulting FLOPs. Following the reviewer's request, Table 7 now reports the training FLOPs of all three stages including the teacher (App. A.1): 1,176×10¹⁸ for RR-VLM against 554×10¹⁸ for the unpruned model, i.e., 2.1× the training compute for a gain of 2.33 pp. As every token traverses more layers, looping increases the compute per token, and we therefore compare models trained on the same data, i.e., at equal training tokens, following Geiping et al. and McLeish et al. At matched training compute, the unpruned model would have to be trained on about twice as many tokens, so the two models would no longer see the same data. We also find that more training helps the unpruned model only slightly: the text stage and the distillation raise its training compute from 10×10¹⁸ to 554×10¹⁸ FLOPs (App. A.1, Table 7), but its accuracy only from 55.27 to 55.80 (Table 1), while RR-VLM reaches 58.13.

Training and inference cost (Table 7)RR-VLM (ours)Unpruned baselinePruned baselineRR-VLM, SDPARR-VLM + QRC, SDPA
Average accuracy, 9 benchmarks58.1355.8052.4158.0958.05
Stored weights (B parameters)1.151.411.151.151.15
Training compute incl. teacher (10¹⁸ FLOPs)1,1765544471,1761,176
Inference compute (GFLOPs per token)5.372.191.635.375.37
Latency, batch size 1 (ms per token)24.09.77.717.828.8
Throughput, batch size 16 (tokens/s)7381,8002,326918547
Peak memory, batch size 1 / 16 (GiB)2.24 / 3.452.69 / 3.582.21 / 3.052.24 / 3.452.22 / 3.11
KV cache (KiB per token)56.022.016.056.022.3

The last two columns use SDPA attention, as our implementation of QRC does. All other columns use FlashAttention-2. QRC is applied without retraining, and the inference compute does not include decoding its low-bit cache.

W4: Inference efficiency and test-time scaling

As requested, Table 7 now reports the measured latency, throughput and peak memory of RR-VLM, both baselines and RR-VLM with QRC (see the table under W3). RR-VLM needs less memory than the unpruned model at batch size 1 (2.24 against 2.69 GiB) and at batch size 16 (3.45 against 3.58 GiB), so the parameter savings lower the inference memory at these batch sizes. Because each recurrence adds its own keys and values to the KV cache, the gap shrinks with the batch size, and from batch size 32 RR-VLM needs more memory than the unpruned model (table below). We therefore evaluate a Quantized Recurrence Cache (QRC, App. A.7), which stores the keys and values of all recurrences after the first as 2-bit differences to the preceding recurrence, following FlashLoop (Yang & Liu, 2026) with the quantizer of KIVI (Liu et al., 2024). Without retraining, QRC reduces the KV cache from 2.55× to 1.01× that of the base model, and the average accuracy is 58.05 against 58.09 for the full-precision cache, both with SDPA attention (Table 6). With QRC, RR-VLM needs less memory than the unpruned model at all four batch sizes. We agree that the parameter savings do not lower the latency, as RR-VLM requires 24.0 ms per token against 9.7 ms for the unpruned model at batch size 1. With SDPA attention, which QRC uses, the latency of RR-VLM is 17.8 ms with the full-precision cache and 28.8 ms with QRC in our current implementation (Table 7).

Peak memory (GiB) by batch size1163264
Unpruned baseline2.693.584.536.42
RR-VLM (ours)2.243.454.747.31
RR-VLM + QRC2.223.114.075.96

We agree that the sampled-depth model does not improve beyond the trained depth (Table 5, Sec. 5.2). In the introduction, we refer to test-time scaling only to describe prior work on depth recurrence in language models, and we do not claim test-time scaling for our model. Additionally, we would like to clarify that all our reported gains are obtained at a fixed depth of six recurrences and therefore do not rely on test-time scaling.

Our revisions add the training-compute and inference-cost measurements requested by the reviewer, as well as an evaluation of QRC, which reduces the KV cache of RR-VLM. We therefore respectfully ask the reviewer to reconsider the soundness and overall ratings, and we would be happy to answer any further questions.

I thank the authors for the detailed response. However, I believe most of my problems are left unaddressed.

W1: I understand and agree with the issue of cost. I retract my statement with "larger backbones". However, there are multiple 1B scale LVLMs, such as Gemma 3 and Qwen3.5-VL. Without evaluating on multiple LVLMs, it cannot be established that the proposed method generalizes. The authors argument that the architecture only assumes an LM and a projector does not establish that the recovery procedure, optimal layer allocation, or resulting accuracy gains transfer to other model families

W2: I appreciate the authors efforts at the experiments. I believe this establishes that (i) recovery stages matter, and (ii) the recurrent-LM baseline is valuable. However, I would distinguish empirical contribution to methodology novelty. The authors have definitely demonstrated the first, but not the second.

W3: I thank the authors for this detailed breakdown, and it addresses my concern with the reporting part of W3. The equal data comparison is isolates the effect of training strategy. However, I believe the authors' justification for not reporting results on equal compute is weak. A compute efficiency comparison is exactly what this should test on: given fixed budget, we want to know which approach yields better accuracy, regardless of whether the additional compute is spent on architecture, training iterations, or more data.

W4: I appreciate the authors for reporting the inference efficiency metrics transparently. First, I acknowledge the accuracy improvement and lowered memory (-18% in model weights and -17% in peak memory) brought about by the method. However, if we cross reference the table in the response to W3, the new model has increased both inference latency and FLOPs by more than 2.4x. I would argue this is a substantial increase in latency for a marginal reduction in memory saved.

I would also like the authors to discuss (if possible), an interesting phenomenon in the response to DbYp's W1/Q1. TinyLLaVA contains 22 layers, achieving an avg. ACC of 55.27. In the table to DbYp, the authors report at r=2 (which is 24 layers traversed), the Avg. Acc is 54.68. So, at roughly the same inference depth, RR-VLM performs worse than the original model, though achieving better results at higher values of r. This suggests that the accuracy improvements might also be highly dependent on increased computation, rather than an intrinsically more compute-efficient architecture. I acknowledge that this experiment uses a smaller training recipe, and a corresponding comparison using the full 60B token recipe would be necessary for a definitive conclusion.

In all, although the experiments demonstrate a memory advantage, I do not believe that they do not establish a favourable accuracy-efficiency trade-off.

AuthorsOpenReview ↗

Response to Reviewer YX8w's follow-up

Thanks for getting back.

W1: Generalization to other model families

We want to point out that the original request was about bigger models, and there was no architecture specified. We thank you for the clarification and we are happy to provide a retrofitted version of Gemma 3 (https://huggingface.co/google/gemma-3-1b-it) and/or Qwen3.5-VL (https://huggingface.co/Qwen/Qwen3.5-2B) for the camera-ready version of the paper. However, the training data for those models is not public, and we will most likely not be able to train the S0, S1, or S2 stages on a comparable amount of training data. We therefore offer to use the same training data (TinyLLaVA) for this experiment and to compare the retrofitted version with the pruned version.

W2: Methodological novelty

Thanks for recognizing the findings of this paper as well as for clarifying that the new baseline is valuable. We agree that the qualities of this paper are more on the practical side, but would argue that ICLR, as a conference on learning representations, supports both theoretical and empirical novelty. We therefore do not see this as a reason to reject the paper.

W3: Accuracy at a fixed training budget

The original review requested that we "report recovery FLOPs or GPU-hours, including teacher overhead". We provided these in the first answer, which the reviewer confirms. We're happy to provide an ablation with a fixed budget for the camera-ready version, but the fact that we followed this recommendation and reported the full cost should not be used as a reason to reject the paper.

W4: Latency

We agree that looped models in general do trade latency and longer compute for accuracy. Schwethelm et al. (https://arxiv.org/abs/2604.21106) proposed a way to quantify the "worth" of each recurrence: a block applied rr times is worth rφr^{\varphi} unique blocks. Based on a 7.5B S0 testbed and the metric proposed, the block in our model, applied six times, is worth about 2.6 unique blocks. We get φ≈0.53\varphi \approx 0.53 by fitting the accuracies in the table below against the log of their worth in unique layers (Schwethelm et al. report φ=0.46\varphi = 0.46 for language models).

Table: rr is the number of times the block is applied. Worth in unique layers is 8+8⋅r0.538 + 8 \cdot r^{0.53}, as 8 layers run once and the 8-layer block runs rr times. Reasoning avg is the average over MMBench, RealWorldQA, SEED-Bench, AI2D and CV-Bench-2D.

ModelrLayers run per tokenWorth in unique layersReasoning avgDifference to unpruned
Unpruned TinyLLaVA–222249.48–
RR-VLM11616.046.59−2.89
RR-VLM22419.649.37−0.11
RR-VLM44024.751.72+2.24
RR-VLM (our model)65628.750.63+1.15
RR-VLM87232.151.88+2.40

This also answers the question on r=2r = 2. With r=2r = 2, RR-VLM runs 24 layers per token, but these are only worth 19.6 unique layers, which is less than the 22 layers of the unpruned model. With r=4r = 4 or more, RR-VLM is worth more than 22 unique layers and outperforms the unpruned model.

As this estimate is based on single runs, we offer to add a respective study on the full training data for the camera-ready version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank all reviewers for their time and their careful reviews. We are glad that they found our results strong, our baselines and ablations informative and our proposed pipeline clear. We have uploaded a revised version of the paper that addresses the points raised by the reviewers, and we summarize the main points below.

Contribution

To the best of our knowledge, RR-VLM is the first prune-then-loop retrofit of a pretrained VLM for VQA. Our study is also the first to compare looping against both the pruned model without looping and the unpruned model trained on the same data.

RR-VLM achieves an average accuracy of 58.13 on nine benchmarks, compared to 55.80 for the unpruned and 52.41 for the pruned model, while storing 23.3% fewer language-model parameters than the unpruned model (Table 1). We also observe that the largest gains over the unpruned model appear on reasoning-oriented benchmarks such as CV-Bench-2D (+11.7 pp), RealWorldQA (+4.1 pp) and MMBench (+3.4 pp).

Additional experiments

We repeated the conversion of McLeish et al. on TinyLlama with the same 4/8/4 allocation and six recurrences as RR-VLM, and trained it on our 60B text tokens with distillation, using the unpruned TinyLlama as the teacher. We then attached the vision encoder and trained S₁ and S₂ with the same pipeline as RR-VLM. This model achieves 56.05 on average, compared with 55.59 for the released weights and 58.13 for RR-VLM. Table 1 now reports the results of both approaches.

Training and inference cost

Following the reviewers' requests, Table 7 now reports the training and inference cost of all three models and of RR-VLM with QRC. As every token traverses 56 instead of 22 layers, RR-VLM requires 2.1× the training compute and 2.47× the latency of the unpruned baseline, and outperforms it by 2.33 pp. We compare the models at equal training tokens, following Geiping et al. and McLeish et al.

Training and inference cost (Table 7)RR-VLM (ours)Unpruned baselinePruned baselineRR-VLM, SDPARR-VLM + QRC, SDPA
Average accuracy, 9 benchmarks58.1355.8052.4158.0958.05
Stored weights (B parameters)1.151.411.151.151.15
Training compute incl. teacher (10¹⁸ FLOPs)1,1765544471,1761,176
Inference compute (GFLOPs per token)5.372.191.635.375.37
Latency, batch size 1 (ms per token)24.09.77.717.828.8
Throughput, batch size 16 (tokens/s)7381,8002,326918547
Peak memory, batch size 1 / 16 (GiB)2.24 / 3.452.69 / 3.582.21 / 3.052.24 / 3.452.22 / 3.11
KV cache (KiB per token)56.022.016.056.022.3

The last two columns use SDPA attention, as our implementation of QRC does. All other columns use FlashAttention-2. QRC is applied without retraining, and the inference compute does not include decoding its low-bit cache.

KV cache

We additionally evaluate a Quantized Recurrence Cache (QRC, App. A.7), which stores the keys and values of all recurrences after the first as 2-bit differences to the preceding recurrence, following FlashLoop (Yang & Liu, 2026) with the quantizer of KIVI (Liu et al., 2024). Without retraining, QRC reduces the KV cache of RR-VLM from 2.55× to 1.01× that of the base model, at a cost of 0.04 pp (58.05 against 58.09 for the full-precision cache, both with SDPA attention, Table 6).

We hope that the revision resolves the reviewers' concerns, and we would be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.