acceptodds
Withdrawn from ICLR 2027

ResFuser: Decomposing Residual Fusion in Aggressively Token-Dropped Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision Transformers operating on dense scientific and medical inputs face a sequence-length bottleneck, motivating aggressive token dropping while preserving the information needed for restoration. We introduce ResFuser, a unified investigation framework that compares two existing residual topologies and four token-sampling rules with a shared backbone and controlled token and training budgets. Across five empirical cells spanning climate, medical, and natural-image restoration in deterministic and diffusion regimes, both full-resolution residual routes substantially improve over the no-residual ablation, while their relative ranking changes across cells. Within the evaluated 7.4M-parameter backbone, the better topology in each cell gives validation RMSE within % of a dense ViT/DiT baseline at 42-59% fewer analytical forward FLOPs. The tested sampling rules produce smaller changes than residual design; masked-region analysis also shows that input salience need not identify the regions where restoration quality improves. These findings motivate a staged design procedure: establish a residual route, compare topologies under the target budget, and then evaluate task-aligned sampling. ResFuser decomposes these choices for dense restoration.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Weather Forecasting, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KLnLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce ResFuser which is a more efficient version of the Vision Transformer operating on less tokens, where they utilize the residual structure and token sampling for improved performance over conventional efficient Transformer architectures. They formulate two residual routes inspired from other Transformer architectures, and try four token sampler schemes for training and validation. The experimental results show mixed results on various domains and tasks.

Critical strengths
  • Experiments across multiple domains: The authors perform controlled experiments across climate, medical, and natural image domains using the same settings.
  • Explicit FLOPs presentation and calculation: The authors present the FLOPs counts for verifiable and reproducible research.
  • Paper is written in a straightforward manner and easy to follow.
Critical weaknesses
  • The model is tiny (7.4M parameters) and trained for only 5,000 steps, where the authors also admitted that this could not ensure convergence. Thus, the "within 7% performance" claim could be an optimistically selected estimate.
  • The encoder, the decoder and the residual route all still run at full resolution, and no wall-clock or memory numbers are reported.
  • Other conventional token reduction alternatives are not compared and reported. Alternatives such as U-ViT, HDiT, and Swin/SwinIR would all reduce cost, and these baselines should also be compared under controlled settings. Without these baselines at equal FLOPs, the claim that the proposed method reduces cost while retaining the performance could not be verified.
  • Compared to previous works such as SPRINT, ORBIT-2, MAE, AdaFormer, SparseViT and UViT, the proposed work lacks novelty since every technical component already exists, and the paper acknowledges this by citing most of them. Their proposed sequential skip is highly similar to SPRINT with the identical sampler used. Also, lack of comparison with other Transformer variants is a critical weakness.
Questions for authors

Please refer to the weaknesses section.

Reviewer_N7x4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ResFuser compares two full-resolution residual paths for dense restoration with aggressive token dropping: a parallel CNN processing the raw input and a skip connection from encoder features. The study covers five settings across climate, medical, and natural-image data, and also examines four token-sampling rules with partial experimental coverage. Both residual paths outperform the no-residual model, although their ranking varies by task.

Critical strengths
  1. Comparing the two routes within the same backbone is useful. Token counts and training steps are matched within each task, though total computation differs. The skip wins on ERA5 and DIV2K, whereas the CNN wins on both MRI tasks. The fastMRI ranking holds across all three seeds (Figure 5).

  2. The fastMRI regional analysis is informative. Compared with structured-uniform sampling, gradient sampling slightly lowers RMSE on the high-Laplacian mask but raises it on the whole image, brain, and high-variance masks (Figure 7). This gives a concrete example of why selecting salient input patches may not improve restoration where it matters.

Critical weaknesses
  1. Since both residual designs come from prior work, I would expect the study to establish a clearer design rule. The mechanism diagnostics narrow the possible explanations but do not explain the topology reversals. Most sampler ablations use the CNN route, and ERA5-diff lacks sampler and drop-rate sweeps for its better-performing skip route. The proposed order of choosing topology before sampling therefore remains tentative.

  2. The evidence for generalization and efficiency is limited. All experiments use one small backbone, and most comparisons have only one seed. Both topology and checkpoint selection use validation RMSE; independent test results are not reported. Appendix A.3 also leaves dataset sizes, split definitions, and normalization unclear. There is no matched comparison with existing sparsification methods or measured latency and memory, so the practical advantage beyond the internal dense reference is still uncertain.

Questions for authors
  1. How much does the sparse middle stack contribute once a dense residual route is present? A model bypassing that stack, together with a parameter-matched no-residual control and learning curves, would help interpret the large gaps in Table 2.

  2. Do the rankings hold on an independent test set across seeds? For ERA5-diff, could the authors compare samplers under both topologies and sweep the skip route's drop rate? These comparisons would test whether the proposed design order survives interactions between the choices.

  3. How does the adapted skip compare with SPRINT's direct encoder-feature skip under the same restoration setup? A matched token-merging baseline and end-to-end latency and peak-memory measurements would also help assess the efficiency claim, including complete 20-step Heun sampling for diffusion.

Reviewer_ZCzKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares two residual routes and several token-selection rules for vision transformers under aggressive token dropping. In a comparison between five restoration configurations, retaining a dense residual route substantially improves validation RMSE, while the preferred topology varies by setting. Selected configurations achieve RMSE close to a dense transformer at substantially lower analytical forward FLOPs.

Critical strengths

The shared backbone provides a useful basis for comparing residual designs. The reversal in topology preference is interesting, and the fastMRI ranking is consistent across three seeds. The authors also distinguish observations from proposed explanations and acknowledge the limits of their efficiency measurements.

Critical weaknesses

Important evaluation details are missing. Appendix A.3 does not specify dataset sizes, train/validation partitions, normalisation or several preprocessing choices. In particular, the ERA5 variables, temporal split and downsampling procedure should be given, alongside MRI slice selection and whether partitions are disjoint by patient. These details are necessary to assess and reproduce the results.

The dense comparison does not isolate sparsification. Table 3 compares a plain dense transformer with a sparse transformer augmented by a residual route. This establishes a comparison between complete configurations, but changes both token density and architecture. Dense controls retaining the same residual routes would clarify the contribution of sparsification. An ablation removing the sparse middle stack would also help establish its contribution beyond the dense route and decoder.

The proposed staged selection procedure has limited support. Most sampler comparisons use the parallel CNN, including ERA5-diff, where it is substantially worse than the skip. The diffusion drop-rate sweep likewise omits the stronger topology. The available experiments therefore leave important interactions between topology, sampling and drop ratio unresolved.

Questions for authors
  1. How is spatial identity preserved at dropped positions in the no-residual control? Are positional embeddings reintroduced after mask-token restoration? If these positions receive identical tokens and no subsequent position-dependent information, the control may be unable to distinguish them.
  2. Can the dataset partitions, preprocessing and normalisation be specified, including patient-level separation for MRI and temporal separation for ERA5?
  3. Do the principal rankings persist with longer training and repeated seeds? In particular, does the ERA5-diff skip retain its advantage across samplers and drop ratios?
  4. What happens when the dense baseline retains the same residual route, and when the sparse middle stack is removed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.