- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces two-phase caching and activation (TCA), a training strategy for input-efficient fine-tuning of vision Transformers. TCA lowers token-level activation and backward computation while keeping the full-sequence forward context. In the first phase it runs a full forward pass without gradients and caches the key and value representations of each layer. In the second phase it picks a subset of tokens from CLS-token attention computed in that completed pass, then replays only those tokens against the cached full-sequence context with gradients enabled, so back-propagation is confined to the selected token paths. Experiments cover 13 image classification datasets and ViT-S/16, ViT-B/16, and ViT-L/16; relative to full fine-tuning, peak memory and training-step time go down with a small drop in accuracy. The authors also test combinations with LoRA and extend the approach to Qwen3-VL-8B for multimodal understanding, FLUX.2-4B for image generation, and Wan2.2-14B for video generation.
Critical strengths
- The complexity analysis matches the method design. Section 3 distinguishes parameter-related overhead from the cost of token-level activation and backward computation, discusses the activation-storage bottleneck that PEFT still faces, and explains clearly how two-phase KV caching and selective recomputation bring training cost down.
- Resource savings are substantial and accuracy loss is small. Across three ViT scales and three task categories, Table 1 shows only a small average accuracy drop in each category relative to full fine-tuning, together with large reductions in peak memory and training-step time. Table 2 shows that TCA can also be combined with LoRA, which speaks to its practical value.
Critical weaknesses
- The comparison with prior work lacks enough implementation detail. Fig. 4 shows that the accuracy gap between TCA and TokenSeek changes with the dataset and the backbone. On ViT-S/16 and ViT-B/16 the advantage is driven mostly by Cars, while the gaps on the other four datasets are marginal; this raises the question of whether the gains are general or largely dataset-specific. The advantage is broader on ViT-L/16, with visible gaps on Cars, Dogs, and NABirds. The paper also does not say how scores are computed for the ViT adaptation, how attention and gradient scores are fused, or which tuning settings were used, so it is hard to judge whether these dataset-specific advantages are robust. The paper should report these settings clearly so that the generality of the claimed advantage can be substantiated.
- The specific gain from dynamic token selection remains unclear. TCA differs from prior work in reselecting tokens from the current forward pass, while [1] relies on fixed scores computed in advance. The paper does not report how much the token set selected for a given sample changes over training, and it does not compare fixed and dynamic selection within the same framework. Controlled experiments would show what dynamic selection actually adds.
Questions for authors
- Baseline adaptation and tuning: How was TokenSeek [1] adapted to ViT for Fig. 4? In particular, how were the attention and gradient scores computed and combined, and how were hyperparameters chosen? Were its training and tuning budgets comparable to those of TCA?
- Exact results for Fig. 4: Precise comparison is difficult with a radar chart. Could you give a table with exact accuracy values for each dataset?
- Contribution of dynamic selection: Under the same data augmentation, how much does the token set chosen for one sample actually change during training? Under the same framework, scoring rule, and token budget, does dynamic selection do better than a fixed selection made before fine-tuning?
Dear reviewers,
A reminder that the discussion period will end on November 18 at 11:59pm AoE. I encourage you to interact with the authors' rebuttal to clarify any remaining issue (and how well the rebuttal addressed your concerns) before the next phase.
Thanks, The AC