- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates using depth recurrence to reduce the parameters of VLMs. It proposes RR-VLM, which prunes some layers from the language backbone, repeatedly applies the retained middle block, and uses a linear adapter to fuse representations. The authors also propose a three-stage curriculum to recover model performance. The method is evaluated on TinyLLaVA.
Critical strengths
- The method has a relatively intuitive design and a clear overall pipeline. The three training stages respectively address language backbone recovery, vision-language realignment, and the recovery of instruction-following capabilities.
- The experimental design is relatively comprehensive, including comparisons among models trained with different combinations of training stages, which help analyze the role of recovery training. The ablation studies also thoroughly examine design choices such as layer allocation, adapters, and KV-cache sharing.
Critical weaknesses
- The method’s incremental contribution is limited. Looping over the middle block after pruning, the layer allocation, and fusion through a linear adapter all follow the existing design of McLeish et al. Furthermore, the experiments are conducted only on TinyLLaVA-1.4B, leaving the applicability to other pretrained VLMs unclear.
- The paper lacks a comprehensive comparison of actual inference latency, throughput, and training costs. In the main model, each token traverses an equivalent depth of 56 layers (), compared with 22 layers in the original model. It remains unclear what practical deployment benefits arise from the reduction in parameter count.
Questions for authors
- Compared with the prune-then-loop scheme of McLeish et al., does this work introduce any other key changes to the architecture or training mechanisms beyond vision-language realignment and visual instruction tuning?
3 comments
Response to Reviewer BXJb (part 2 of 2)
W1: Other pretrained VLMs
Applying our pipeline to a different model requires that the pretraining data of its language model and its vision-language training data are public, as we repair the language model after pruning on text close to its pretraining data and retrain the VLM on its original vision-language data. TinyLLaVA meets both conditions and can be trained within a reasonable training budget, which is why we chose it for our experiments. For most larger VLMs, the pretraining data of the language model is not released, and in several cases neither is the vision-language data. Apart from this requirement on the data, our proposed method is not specific to TinyLLaVA, since the layer allocation, the adapter and the three training stages only assume a language model and a projector. We agree with the reviewer that a second model would strengthen our claims, and we will report results on a second model family whose language model was pretrained on public data in the camera-ready version.
W2: Latency, throughput and training cost
Following the reviewer's request, Table 7 now reports these costs for RR-VLM, both baselines and RR-VLM with QRC (table below). RR-VLM requires 2.1× the training compute and 2.47× the latency per token of the unpruned baseline (24.0 against 9.7 ms).
| Training and inference cost (Table 7) | RR-VLM (ours) | Unpruned baseline | Pruned baseline | RR-VLM, SDPA | RR-VLM + QRC, SDPA |
|---|---|---|---|---|---|
| Average accuracy, 9 benchmarks | 58.13 | 55.80 | 52.41 | 58.09 | 58.05 |
| Stored weights (B parameters) | 1.15 | 1.41 | 1.15 | 1.15 | 1.15 |
| Training compute incl. teacher (10¹⁸ FLOPs) | 1,176 | 554 | 447 | 1,176 | 1,176 |
| Inference compute (GFLOPs per token) | 5.37 | 2.19 | 1.63 | 5.37 | 5.37 |
| Latency, batch size 1 (ms per token) | 24.0 | 9.7 | 7.7 | 17.8 | 28.8 |
| Throughput, batch size 16 (tokens/s) | 738 | 1,800 | 2,326 | 918 | 547 |
| Peak memory, batch size 1 / 16 (GiB) | 2.24 / 3.45 | 2.69 / 3.58 | 2.21 / 3.05 | 2.24 / 3.45 | 2.22 / 3.11 |
| KV cache (KiB per token) | 56.0 | 22.0 | 16.0 | 56.0 | 22.3 |
The last two columns use SDPA attention, as our implementation of QRC does. All other columns use FlashAttention-2. QRC is applied without retraining, and the inference compute does not include decoding its low-bit cache.
W2: Deployment benefit
We see the main benefit of RR-VLM in its accuracy, and the lower parameter count as an additional benefit. Looping the pruned model with 1.15B parameters raises its average accuracy from 52.41 to 58.13, which is 2.33 pp higher than the unpruned model with 1.41B parameters (Tables 1 and 7). Compared to the unpruned model, the gains are largest on CV-Bench-2D (+11.7 pp), RealWorldQA (+4.1 pp) and MMBench (+3.4 pp). With fewer stored parameters, RR-VLM also needs less memory than the unpruned model (2.24 against 2.69 GiB at batch size 1 and 3.45 against 3.58 GiB at batch size 16, Table 7). QRC reduces its KV cache from 2.55× to 1.01× that of the base model at a cost of 0.04 pp (Table 6), but increases the latency per token from 17.8 to 28.8 ms in our current implementation, both with SDPA attention (Table 7).
We thank the reviewer for these questions, which helped us to state more clearly how our method differs from McLeish et al. and what its costs and benefits are. We believe that our work is of value to the community in three respects. (1) To the best of our knowledge, it presents the first prune-then-loop retrofit of a pretrained VLM for VQA, with a recovery that uses no new vision-language data. (2) It shows that retrofitting the VLM directly outperforms adding a vision encoder to a recurrent language model, even when both use the same text data and distillation. (3) It is the first study to compare looping in a pretrained VLM against both the pruned and the unpruned model trained on the same data, and it shows that looping improves accuracy with fewer stored parameters, in particular on reasoning-oriented benchmarks (Sec. 1, Table 1). As our revisions state the differences to McLeish et al. and add the training and inference costs requested by the reviewer, we kindly ask the reviewer to consider raising the soundness and overall ratings, and we would be happy to discuss any further points.
Follow-up on our response to Reviewer BXJb
We thank the reviewer again for their time and their review. As the discussion period ends on November 18 (AoE), we would like to ask whether our response addresses the reviewer's concerns. In our reply, we state the differences to McLeish et al. and compare against their recurrent language model extended by a vision encoder and trained on the same text data with distillation (56.05 against 58.13 for RR-VLM), we report the training and inference costs in Table 7, and we explain why we chose TinyLLaVA. We would be happy to answer any further questions, and if the concerns are resolved, we kindly ask the reviewer to consider updating their score.
Response to Reviewer BXJb (part 1 of 2)
We thank the reviewer for the careful review and are glad that the reviewer finds our pipeline clear and our experiments comprehensive.
W1 and Q1: Differences to McLeish et al.
Beyond the two vision-language stages, we introduce three changes on top of the method of McLeish et al., and we evaluate each of them separately (Tables 1, 2 and 4). We reuse their recurrent architecture, but (1) we retrofit the recurrence into the language model of a pretrained VLM instead of a standalone language model, and we find that the resulting model outperforms their recurrent model extended by a vision encoder. (2) We distill from the unpruned model, which improves the looped model more than the unpruned model. (3) We repair the language backbone with a text-only stage before the projector alignment and the visual instruction tuning, as without this stage the looped model remains below the unpruned model. (4) In addition, we study the KV cache of the retrofitted model and evaluate QRC, which reduces it from 2.55× to 1.01× that of the base model without retraining (Table 6).