- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces TRAPE, a relative positional bias for vision transformers. It combines continuous axial functions, local correction tables, and a shared bounded output function. TRAPE supports different input resolutions and improves performance on several benchmarks.
Critical strengths
The experiments cover a broad range of settings, including resolution extrapolation without fine-tuning, position-only adaptation, rectangular inputs, semantic segmentation, and inference efficiency. Some experiments use multiple random seeds. The paper also provides detailed experimental settings and clearly states several limitations.
Critical weaknesses
1.The paper does not discuss or compare with LookHere [1]. LookHere directly studies resolution extrapolation in plain ViTs, including training at and testing at . Its directional attention masks and distance biases are closely related to TRAPE’s evaluation without fine-tuning. A comparison with this method would help establish TRAPE’s contribution.
2.The method uses established ideas, including log-spaced coordinates and continuous bias functions [2], as well as axial factorization [3]. These ideas have already been explored in prior work. TRAPE therefore appears to combine existing techniques, and its main methodological novelty remains unclear.
[1] LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate. NIPS2024
[2] Swin Transformer V2: Scaling Up Capacity and Resolution. CVPR2022
[3] Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. NIPS2023
Questions for authors
Please see weakness. Since I am not an expert on positional encoding or Transformer architectures, I will consult other reviewers' comments.
2 comments
Thank you for the opportunity to clarify TRAPE's novelty. Our contribution is a relative positional bias designed jointly with a resolution tape and position-only adaptation of frozen ViTs.
We start from Swin V2-style Log-CPB [2, Sec. 3.3], with log-spaced coordinates, joint 2D continuous bias and its official output (Table 1(a)); Swin V2 permits precomputation. Row (b) applies established axial factorization, used by NaViT [3, Sec. 2.1] for absolute/fractional input embeddings, to relative bias responses. We add a shared output and central tables. Beyond these components, the resolution tape enables compact table-only deployment and independent height/width reuse; position-only adaptation grafts this bias onto frozen ViTs, retaining native encoding.
1. Relative positional bias and component ablations.
In our original main-text Sec. 3.1, Eq. (3) defines the raw axis response and Eq. (4) the attention bias (block/head indices omitted):
Here is the fixed-reference log coordinate in our main-text Eq. (2), and the local correction. Starting from (b), row (c) adds the shared output, and row (d) adds the central table, completing TRAPE. The learned is shared across blocks and heads; local corrections merge into exported tables.
Main-text Table 1 uses the same DeiT-Ti architecture and recipe, trained at and evaluated unchanged on ImageNet-1k validation (top-1, %):
| Positional encoding | |||
|---|---|---|---|
| (a) Log-CPB, | 72.2 | 52.6 | 64.3 |
| (b) (a) + axial factorization | 72.1 | 52.0 | 61.9 |
| (c) (b) + shared output | 72.8 | 62.9 | 66.0 |
| (d) (c) + central table (TRAPE) | 72.8 | 63.4 | 68.1 |
Relative to (a)/(b), full TRAPE gains 0.6/0.7 percentage point (pp) at and 10.8/11.4 pp at . Factorization alone does not yield these gains. Adding the shared output raises by 10.9 pp; the central table adds 0.5 pp there and 2.1 pp at . Thus both training-resolution accuracy and generalization improve. Row (c) changes amplitude and input scaling jointly (main-text Sec. 3.2).
2. Resolution tape and table-only deployment.
Building on this bias, the resolution tape stores raw axis responses before the nonlinearity for compact deployment and reuse across resolutions and aspect ratios. Our original main-text Sec. 3.3 gives the exact center-slice identity for fixed parameters in Eq. (5) and the empirical cross-anchor interpolation rule in Eq. (6). During position-only adaptation, is frozen. Inference slices and blends each axis independently and applies the shared nonlinearity after composition. Mixing final squashed biases generally gives a different function (App. A.2).
For , height uses anchors 224/336 and width 704/1024, without rectangular fitting or positional MLP evaluation within the exported range. Fixed-reference slicing also applies to a joint 2D bias; factorization enables independent anchor pairs and rather than stored responses per anchor/block/head. Plain-ViT attention remains full.
Appendix F/Table 16 compares fp32 positional storage for ViT-S, four anchors , and image sides 224-704:
| Stored representation | Size |
|---|---|
| TRAPE tape | 154 KiB |
| Log-CPB MLP weights | 864 KiB |
| Joint 2D anchor tables | 5.6 MiB |
Training checkpoints and expanded attention matrices are excluded. Joint 2D biases can also be cached; Table 16 notes that joint table blending need not reproduce MLP-weight blending. TRAPE retains independent axis routing.
3. Position-only adaptation of frozen ViTs.
Position-only adaptation makes this bias usable with existing frozen ViTs (main-text Sec. 3.4).
Original-submission results use position-only fine-tuning: 160 steps/anchor, global batch 128. Six square anchors 144/224/336/480/704/1024 start independently from zero branches on the public 224 checkpoint; Table 13 refits only 704. Appendix E.1/Table 13 tests the same RoPE-Mixed ViT-S with matched data and steps at fitted , where each method uses its own fitted state without interpolation:
| Method | Top-1, mean sample SD (%) |
|---|---|
| TRAPE | |
| Log-CPB | |
| 2D-Tanh |
TRAPE's 4.6/2.1 pp gains over Log-CPB/2D-Tanh persist without cross-anchor interpolation. These are three-seed comparisons under matched data/step budgets; optimization is parameterization-specific (App. B.5). We will clarify the joint design and this controlled evidence in the revision.
[2] Liu et al. Swin Transformer V2: Scaling Up Capacity and Resolution. CVPR 2022.
[3] Dehghani et al. Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. NeurIPS 2023.
Thank you for these comments and for highlighting LookHere [1]. We agree that it directly addresses resolution extrapolation in plain ViTs and should be included. We add the comparison and distinguish the mechanisms and evaluated adaptation settings below. Our table and section numbers refer to the original submission.
1. Resolution extrapolation without fine-tuning.
We compare LookHere's ViT-B/16 variants (Fuller et al., NeurIPS 2024) with TRAPE-DeiT-B/16. All are trained at 2242 and evaluated on the original 50,000-image ImageNet-1k validation set without high-resolution gradient fine-tuning. Values are top-1 (%).
LookHere values are from its Table 1 (main text; Best, 2242) and Table 6 (Appendix A.2; larger sizes), using the Val top-1 column. TRAPE-B uses one unchanged checkpoint at every size: 10242 quotes Table 1 of our main text; the remaining values are reported from evaluations of that same checkpoint under App. B.1, including additional test resolutions.
The training and evaluation protocols differ. LookHere trains for 150 epochs on 99% of ImageNet and uses aspect-preserving resizing followed by centre cropping. TRAPE uses the 300-epoch DeiT recipe and direct resizing above 2242 (App. B.1). LookHere's reported extrapolation results select a global distance slope on training-held-out minival at each target resolution. This is parameter selection rather than gradient fine-tuning; LookHere also reports effective extrapolation without slope tuning. TRAPE-B uses no target-resolution parameter selection in this comparison.
At 10242, TRAPE-B and LH-45 report 73.30% and 73.15%, respectively. These results under their respective protocols show competitive reported accuracy at the same model scale and patch size. For controlled evidence, Table 1 in our main text holds the DeiT-Ti recipe fixed: at 10242, axial Log-CPB, its variant with the shared stanh output, and full TRAPE reach 52.0%, 62.9%, and 63.4%, respectively.
2. Attention masks, distance biases, and position-only adaptation.
LookHere assigns fixed directional fields of view to selected heads, with Euclidean-distance penalties and additional undirected heads. TRAPE learns block/head-specific relative-offset responses, permits asymmetric and non-monotonic patterns, and adds finite bounded biases without directional exclusion in plain ViTs.
LookHere uses fixed masks and distance penalties without learned positional embeddings or an online positional MLP. TRAPE's compactness concerns the storage and reuse of its learned relative-bias responses.
TRAPE also provides an evaluated position-only adaptation route for existing checkpoints. A zero-output branch retains native APE/RoPE and preserves initial predictions, while content weights remain frozen. On five public classification ViTs, it updates under 0.13% of parameters using 0.048-0.096 ImageNet epochs, measured by training image presentations (Sec. 4.4; Appendix B, Table 6). The fitted responses export to tables reused across resolutions and aspect ratios. This adaptation setting is separate from the unchanged-checkpoint comparison above.
We will add LookHere to the related work and comparison, and clarify the differences in spatial inductive bias and adaptation mechanisms.
[1] Fuller et al. LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate. NeurIPS 2024.