acceptodds
Under review as a conference paper at ICLR 2027

Two-Phase Caching and Activation for Input-Efficient Fine-Tuning of Vision Transformers

Abstract

Following the success of the ViT, transformers have been widely adopted for image recognition, multimodal understanding, and visual generation, increasing the need for efficient fine-tuning. Although parameter-efficient fine-tuning (PEFT) reduces trainable parameters, it can still require back-propagation through the full token sequence, retaining substantial activation and computation costs. Token pruning and merging address this overhead by shortening the sequence, but also modify the context available during forward propagation. Building on selective token back-propagation, we formulate input-efficient fine-tuning (IEFT) for vision transformers to reduce token-level backward computation while retaining full-sequence forward context. To enable token selection informed by the completed forward pass, we propose a two-phase caching-activation (TCA) strategy that first caches global keys and values during a gradient-free full forward pass, then recomputes selected tokens against this cached context. With fixed parameters and consistent deterministic computation, this replay reproduces the corresponding full-sequence forward outputs while restricting gradient propagation to selected token paths. Experiments across 13 recognition tasks demonstrate over 60% peak-memory reduction and approximately 45% training-time reduction with minimal accuracy degradation on ViT fine-tuning. Combining TCA with PEFT yields additional efficiency gains. Extensions to Qwen3-VL-8B, FLUX.2-4B, and Wan2.2-14B cover multimodal understanding and visual generation, yielding 3.4× average micro-batch capacity for the first two and approximately 2×video generation training throughput on the same hardware. The code is available in the supplementary materials and will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Input-efficient fine-tuning, Vision Transformers, Selective back-propagation, Memory-efficient training, Multimodal large language models, Diffusion Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mYdSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces two-phase caching and activation (TCA), a training strategy for input-efficient fine-tuning of vision Transformers. TCA lowers token-level activation and backward computation while keeping the full-sequence forward context. In the first phase it runs a full forward pass without gradients and caches the key and value representations of each layer. In the second phase it picks a subset of tokens from CLS-token attention computed in that completed pass, then replays only those tokens against the cached full-sequence context with gradients enabled, so back-propagation is confined to the selected token paths. Experiments cover 13 image classification datasets and ViT-S/16, ViT-B/16, and ViT-L/16; relative to full fine-tuning, peak memory and training-step time go down with a small drop in accuracy. The authors also test combinations with LoRA and extend the approach to Qwen3-VL-8B for multimodal understanding, FLUX.2-4B for image generation, and Wan2.2-14B for video generation.

Critical strengths
  1. The complexity analysis matches the method design. Section 3 distinguishes parameter-related overhead from the cost of token-level activation and backward computation, discusses the activation-storage bottleneck that PEFT still faces, and explains clearly how two-phase KV caching and selective recomputation bring training cost down.
  2. Resource savings are substantial and accuracy loss is small. Across three ViT scales and three task categories, Table 1 shows only a small average accuracy drop in each category relative to full fine-tuning, together with large reductions in peak memory and training-step time. Table 2 shows that TCA can also be combined with LoRA, which speaks to its practical value.
Critical weaknesses
  1. The comparison with prior work lacks enough implementation detail. Fig. 4 shows that the accuracy gap between TCA and TokenSeek changes with the dataset and the backbone. On ViT-S/16 and ViT-B/16 the advantage is driven mostly by Cars, while the gaps on the other four datasets are marginal; this raises the question of whether the gains are general or largely dataset-specific. The advantage is broader on ViT-L/16, with visible gaps on Cars, Dogs, and NABirds. The paper also does not say how scores are computed for the ViT adaptation, how attention and gradient scores are fused, or which tuning settings were used, so it is hard to judge whether these dataset-specific advantages are robust. The paper should report these settings clearly so that the generality of the claimed advantage can be substantiated.
  2. The specific gain from dynamic token selection remains unclear. TCA differs from prior work in reselecting tokens from the current forward pass, while [1] relies on fixed scores computed in advance. The paper does not report how much the token set selected for a given sample changes over training, and it does not compare fixed and dynamic selection within the same framework. Controlled experiments would show what dynamic selection actually adds.
Questions for authors
  1. Baseline adaptation and tuning: How was TokenSeek [1] adapted to ViT for Fig. 4? In particular, how were the attention and gradient scores computed and combined, and how were hyperparameters chosen? Were its training and tuning budgets comparable to those of TCA?
  2. Exact results for Fig. 4: Precise comparison is difficult with a radar chart. Could you give a table with exact accuracy values for each dataset?
  3. Contribution of dynamic selection: Under the same data augmentation, how much does the token set chosen for one sample actually change during training? Under the same framework, scoring rule, and token budget, does dynamic selection do better than a fixed selection made before fine-tuning?
Reviewer_TqvGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Input-Efficient Fine-Tuning (IEFT), a training paradigm that reduces the token-dependent computational cost of back-propagation while preserving the full-sequence context during the forward pass. To operationalize this idea, the authors introduce a Two-phase Caching-Activation (TCA) strategy: a gradient-free forward pass first caches the key/value (KV) representations at every layer, and a selected subset of tokens is then recomputed under gradient tracking, attending to the cached global context. Across classification benchmarks, the method yields substantial reductions in both peak memory and training latency while incurring only marginal accuracy degradation.

Critical strengths
  • The core idea is elegant, intuitive, and empirically effective, achieving a principled balance between dense-context forward computation and sparse back-propagation. It is both conceptually well-motivated and practically valuable.

  • The method is orthogonal to and composable with established PEFT techniques such as LoRA, and can therefore be deployed in conjunction with them rather than as a replacement.

Critical weaknesses
  • The token-ranking criterion is not analyzed with sufficient rigor beyond the ViT classification setting. Whereas the ViT experiments rely on CLS-attention for selection, the ranking mechanisms adopted for Qwen3-VL, FLUX, and Wan are not clearly defined.

  • The paper does not rigorously investigate whether CLS-token attention constitutes an optimal or robust selection criterion across diverse architectures and tasks.

  • The evidence for generalization to generative and multimodal settings remains limited. Although the FLUX and Wan extensions are promising, the corresponding quality assessment is superficial, and several configurations warrant more complete comparisons — for instance, reporting LoRA+TCA results alongside the primary baselines.

  • Given that the contribution is fundamentally a systems-level optimization, the accompanying systems analysis lacks depth. In particular, the paper offers limited insight into the overhead of the replay phase and the cost of cache management; a finer-grained breakdown would help clarify the source of the observed speedups and their robustness.

Questions for authors

Please check the weakness section.

Reviewer_Jv7POpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Two-Phase Caching and Activation (TCA) for input-efficient fine-tuning of vision transformers. TCA first performs a gradient-free forward pass to cache full-sequence keys and values, then recomputes selected tokens against this cached context, restricting backpropagation to active token paths. This preserves full-sequence forward context while reducing activation memory and backward computation. Across 13 classification datasets, the authors report over 60% lower peak memory usage and approximately 45% shorter training-step time, with small accuracy losses. Additional experiments demonstrate compatibility with LoRA and applicability to multimodal understanding, image generation, and video generation.

Critical strengths

1.The paper presents a clear approach to reducing token-level training costs while preserving full-sequence context. TCA caches keys and values for all tokens during a gradient-free pass, then backpropagates through selected tokens using that cached context. This targets a major source of fine-tuning overhead while avoiding the context changes caused by token pruning or merging, making the method’s central idea consequential to its contribution.

2.The experiments show a useful efficiency–accuracy trade-off across a broad set of settings. On 13 classification benchmarks, the authors report over 60% lower peak memory use and about 45% shorter training time, with less than a one-point average accuracy drop. Results combining TCA with LoRA and extending it to multimodal and generative models also suggest the approach may be useful beyond standard ViT classification.

Critical weaknesses
  1. The evidence that TCA’s token-selection strategy improves on prior selective-backpropagation methods is not conclusive. The paper describes its TokenTune and TokenSeek comparison as preliminary and cross-protocol, and reports best-test accuracy for those baselines. This makes it difficult to tell whether TCA’s results follow from its current-pass attention-based selection or from differences in training and evaluation protocols. Since this selection strategy is a central part of the paper’s contribution, the lack of a controlled comparison weakens the evidence for its claimed advantage.

  2. The generative-model scaling results provide limited evidence for the paper’s broad efficiency claims. The video quality evaluation uses 300 held-out clips, and the reported throughput comparison uses different settings: full-token LoRA uses activation checkpointing with micro-batch 4, while TCA uses no checkpointing with micro-batch 2. The resulting speedup is therefore difficult to attribute to TCA alone. This limits how strongly the experiments support the claim that TCA’s efficiency gains generalize to video generation.

Questions for authors

Since TCA blocks gradient flow through unselected tokens and selects active tokens using final-layer CLS attention, what evidence shows that this attention ranking reliably identifies the tokens whose gradients matter for fine-tuning across the evaluated tasks? This bears directly on whether the method’s central selection assumption is supported.

Comments on the paper

1 comment

Dear reviewers,

A reminder that the discussion period will end on November 18 at 11:59pm AoE. I encourage you to interact with the authors' rebuttal to clarify any remaining issue (and how well the rebuttal addressed your concerns) before the next phase.

Thanks, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.