acceptodds
Under review as a conference paper at ICLR 2027

GTR: Gated Token Recurrence for Efficient Dense Prediction

Abstract

Self-attention-based vision backbones perform well on dense prediction, but the quadratic computational cost of global softmax attention limits their efficiency as image resolution increases. We introduce **Gated Token Recurrence (GTR), a softmax-free recurrent vision backbone that combines gated linear attention**, alternating spatial scan directions, and spatially enhanced SwiGLU blocks. GTR is distilled from a detection-specialized DINOv3 teacher using only final-layer patch-token alignment through a linear projection and squared loss, without masked-token prediction or intermediate-layer supervision. With Objects365 detector pre-training, GTR-L achieves 58.9 box AP on COCO val2017 with 1.908 ms median batch-one latency under compiled FP16 execution on an RTX4090. The same backbone also transfers to instance segmentation, pose estimation, oriented detection, semantic segmentation, and monocular depth estimation. For efficient inference, we develop a specialized chunkwise CUDA operator that is faster than FLA v0.5.0 at 1.6K tokens on RTX4090. TensorRT deployment on DRIVE AGX Thor achieves 2.282–8.769 ms median batch-one latency across the evaluated models. These results show that recurrent token mixing can provide an efficient alternative to global softmax attention for high-resolution dense prediction and edge deployment. Code and pretrained models will be released upon publication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
gated linear attention, dense prediction, CUDA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pg2POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Gated Token Recurrence (GTR), a softmax-free recurrent vision backbone for efficient dense prediction. GTR combines gated linear attention, alternating spatial scan directions, and spatially enhanced SwiGLU blocks, and is distilled from a detection-specialized DINOv3 teacher using final-layer patch-token alignment. The authors also develop a specialized chunkwise CUDA operator and integrate GTR into TensorRT for edge deployment. Experiments across six dense prediction tasks and four model scales demonstrate competitive accuracy and practical inference efficiency, including favorable scaling at higher input resolutions.

Critical strengths
  1. The paper is clearly written, with detailed and easy-to-follow descriptions of the method and experiments.
  2. The work addresses both accuracy and practical efficiency, with a custom CUDA implementation and edge deployment demonstrating latency improvements.
  3. The experiments cover six dense prediction tasks and four model scales, including an analysis of performance at higher resolutions.
  4. The appendix provides detailed training, deployment, and timing protocols, along with ablations of the main design choices.
Critical weaknesses
  1. Limited architectural novelty. GTR largely combines established components, including GLA, directional scanning, local convolutions, and feature distillation. The contribution is primarily engineering-oriented, and controlled comparisons with ViG and EdgeCrafter would help establish the value of the proposed design choices.
  2. The baseline comparisons could be more complete. The Objects365-pretrained ECDet models were released one month before the submission deadline. Including these results would help clarify the accuracy–latency trade-off under more comparable pretraining settings.
  3. Accuracy and latency are measured under different inference settings for some tasks. Segmentation and depth evaluation use sliding-window or multi-view inference, whereas latency measures only a single forward pass. Reporting the runtime of the full evaluation pipeline would better reflect the practical accuracy–efficiency trade-off.
Questions for authors
  1. Could you provide accuracy and latency comparisons between GTR and RF-DETR at the same input resolution? This would help distinguish the effects of architectural choices from those of input resolution.
  2. Could you report the training time or GPU-hours for each training stage, and compare the training throughput with a comparable softmax-attention baseline such as EdgeCrafter under matched settings? Since the custom CUDA kernel is used only for inference, it remains unclear how GLA affects training efficiency.
6 comments
AuthorsOpenReview ↗

Response to Reviewer pg2P (1)

Weakness 1: Limited architectural novelty. GTR largely combines established components, including GLA, directional scanning, local convolutions, and feature distillation. The contribution is primarily engineering-oriented, and controlled comparisons with ViG and EdgeCrafter would help establish the value of the proposed design choices.

Thank you for raising this important concern! We respectfully clarify that GTR is not an incremental combination of existing components, nor is its novelty merely a different scan configuration obtained through hyperparameter tuning. Its design is motivated by a fundamentally different computational paradigm and three key insights.

1. GTR adopts a different computational paradigm from ViG and other existing architectures.

ViG integrates 2D depthwise convolution into the input of bidirectional GLA, allowing local features to participate directly in bidirectional global context modeling:

O_{\mathrm{local}} &= \mathrm{DWConv}(X), \\\\ O_{\mathrm{global}} &= \mathrm{BiGLA}(O_{\mathrm{local}}), \\\\ O &= G_{2D} \odot O_{\mathrm{local}} + (1 - G_{2D}) \odot O_{\mathrm{global}}. \end{aligned}$$ In contrast, GTR places DWConv in the value branch of an independent Spatial SwiGLU block: $$\begin{aligned} {[G, V]} &= \mathrm{Linear}_{g,v}(X), \\\\ V_{\mathrm{local}} &= \mathrm{DWConv}_{2D}(V), \\\\ Y_{\mathrm{spatial}} &= \mathrm{Linear}\big(\mathrm{SiLU}(G) \odot V_{\mathrm{local}}\big). \end{aligned}$$ By separating local spatial interaction from recurrent token mixing, GTR eliminates the need for bidirectional recurrence within each block, while retaining a fixed-size recurrent state for efficient inference. **2. GTR is motivated by three complementary insights.** **```Insight 1: Long-range dependency modeling and local spatial modeling should be decoupled.```** We argue that long-range propagation and local spatial interaction are distinct problems best addressed by complementary operators: recurrence captures long-range dependencies, while DWConv models local interactions. By explicitly handling locality, GTR eliminates the need for additional scan directions, enabling a single causal recurrence per block without sacrificing performance [Table 4]. Thus, our contribution goes beyond simply adding DWConv; we demonstrate that decoupling locality from recurrence enables an effective purely causal architecture. **```Insight 2: GTR demonstrates an effective knowledge transfer strategy across different computational paradigms.```** Transferring representations from non-causal Transformers to causal recurrent models is challenging due to their different computational structures. Existing approaches often require multi-stage alignment or multiple distillation objectives [1–3]. In contrast, we demonstrate that a single L2 loss is sufficient to transfer knowledge [Table 3]. The contribution is not a new loss function, but the finding that such a simple feature-level supervision strategy effectively bridges this architectural gap. **```Insight 3: GTR validates the effectiveness of purely linear-attention modeling for modern real-time detection.```** GTR is a purely linear-attention architecture. Most linear-attention backbones [4–8] are validated only on legacy frameworks (e.g., Mask R-CNN, Faster R-CNN), which cannot reveal their upper bound, and recent LLMs often adopt 3:1 linear–softmax hybrids [9,10]. In contrast, GTR achieves strong performance in a modern real-time detection setting without any softmax attention. As shown in [Table 5], the 3:1 hybrid changes AP by only ±0.1 while increasing latency by ~10%.
AuthorsOpenReview ↗

Response to Reviewer pg2P (2)

3. Additional controlled backbone comparisons directly demonstrate the architectural advantage of GTR.

To further isolate the contribution of the GTR architecture, we conducted additional experiments by replacing only the GTR backbone with ViG and EdgeCrafter, while keeping the training configurations and Python environment identical. The results are as follows:

ModelMedian Latency (ms)Params (M)FLOPs (G)APAP$_{50}$AP$_{75}$AP$_S$AP$_M$AP$_L$
GTR1.21512.0733.8744.960.648.227.648.360.7
ViG1.39512.3232.9242.257.945.724.345.357.7
EdgeCrafter1.40411.9757.1543.459.147.126.146.259.7

Under the same training configuration, GTR achieves +2.7 AP over ViG and +1.5 AP over EdgeCrafter, while simultaneously reducing latency by 12.9% and 13.5%, respectively. Notably, GTR also achieves lower latency with fewer FLOPs than EdgeCrafter. These results provide direct evidence that the performance is not merely inherited from the training recipe or distillation, but also benefits from the GTR architectural design.

4. The single-scan design is principle-driven rather than search-driven.

Unlike architecture search approaches that explore multiple configurations to identify favorable architectures, GTR's single-scan design follows directly from our architectural principle of decoupling local spatial interaction from long-range recurrent modeling. It is therefore not an isolated hyperparameter choice, but a consequence of the proposed computational decomposition.

[1] Guoyizhe Wei and Rama Chellappa. ViT-Linearizer: Distilling quadratic knowledge into linear-time vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20737–20747, 2025.

[2] Yifan Li, Seunghyun Yoon, Viet Dac Lai, Franck Dernoncourt, Jason Kuen, Yu Kong, and Trung Bui. ViT-AdaLA: Adapting vision transformers with linear attention. To appear in Advances in Neural Information Processing Systems (NeurIPS), 2026.

[3] Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Shiqi Huang, Min Kass Chong, Wahyu Wiratama, Peng Hu, Chen Gong, Wu Liu, Xi Peng, Chun Jian Ho, and Hongyuan Zhu. Copy the same, distill the difference: Initializing linear vision transformers. arXiv preprint arXiv:2609.35745, 2026.

[4] Dongchen Han et al. FLatten Transformer: Vision Transformer using Focused Linear Attention. ICCV, 2023.

[5] Dongchen Han et al. Agent Attention: On the Integration of Softmax and Linear Attention. ECCV, 2024.

[6] Dongchen Han et al. Demystify Mamba in Vision: A Linear Attention Perspective. NeurIPS, 2024.

[7] Bencheng Liao et al. ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention. AAAI, 2025.

[8] Weikang Meng et al. PolaFormer: Polarity-aware Linear Attention for Vision Transformers. ICLR, 2025.

[9] Qwen Team. Qwen3-Next, 2025.

[10] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture, 2025.

AuthorsOpenReview ↗

Response to Reviewer pg2P (3)

Weakness 2: The baseline comparisons could be more complete. The Objects365-pretrained ECDet models were released one month before the submission deadline. Including these results would help clarify the accuracy–latency trade-off under more comparable pretraining settings.

Thank you for your advice. To better address this concern, we provide additional comparisons with Objects365-pretrained ECDet models across different model scales.

ModelParams (M)GFLOPsLatency (ms) ↓AP ↑AP$_{50}$ ↑AP$_{75}$ ↑AP$_S$ ↑AP$_M$ ↑AP$_L$ ↑
ECDet-S10.050.01.65253.671.058.035.158.371.8
GTR-S12.133.81.22553.671.158.336.458.370.2
ECDet-M19.085.02.09556.874.361.838.861.674.0
GTR-M22.762.61.46257.375.062.441.162.074.1
ECDet-L33.0147.02.73059.077.064.442.164.276.0
GTR-L37.2106.01.90858.976.864.342.464.076.0
ECDet-X49.0204.03.02159.977.565.242.464.877.0
GTR-X46.5130.22.11559.477.364.742.164.676.4

We have updated Table 1 on page 6 to include the Objects365-pretrained ECDet results.

Weakness 3: Accuracy and latency are measured under different inference settings for some tasks. Segmentation and depth evaluation use sliding-window or multi-view inference, whereas latency measures only a single forward pass. Reporting the runtime of the full evaluation pipeline would better reflect the practical accuracy–efficiency trade-off.

Thank you for your careful reading! We clarify that our reported latency measures a single forward pass, following the same evaluation convention as YOLO26, ensuring a consistent comparison across methods. Nevertheless, to further address the concern, we additionally evaluate the full inference runtime (TinferT_{\mathrm{infer}}). The measurements use the same hardware, software environment, and timing protocol as in the original paper.

DatasetModelSingle-forward (ms)TinferT_{\mathrm{infer}} (ms)FPS
CityscapesGTR-S1.49513.0876.4
CityscapesGTR-M1.97015.3565.1
CityscapesGTR-L2.95819.3451.7
CityscapesGTR-X3.48221.4746.6
NYUGTR-S1.29615.7863.4
NYUGTR-M1.50819.2052.1
NYUGTR-L1.94825.6139.1
NYUGTR-X2.15827.7936.0
Reviewer_4bRTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GTR is a vision encoder, and the paper's change is architectural: softmax attention is replaced with gated linear attention (GLA). Each of its 12 blocks performs one directional scan over the patch grid, with four scan directions alternating across depth, and the SwiGLU channel mixer is replaced with a variant that applies a 3×3 depthwise convolution on the value branch. The backbone is trained by regressing its final patch features onto those of a frozen, detection-fine-tuned DINOv3 teacher through a linear projection with an L2 loss, then pre-trained on Objects365 and fine-tuned per task. An inference-only chunkwise CUDA kernel and a TensorRT deployment on DRIVE AGX Thor are provided. Validation covers COCO detection at four scales, five further dense prediction tasks at the S and X scales, ablations of the distillation recipe, spatial mixing, scan directions, hybrid softmax layers, and input resolution, and latency measurements on RTX 4090 and Thor.

Critical strengths

The engineering is complete and the measurements are credible. The kernel, the TensorRT deployment, and the heads and recipes for six tasks are all delivered. Every model is timed under one protocol (FP16, batch one, torch.compile, CUDA Graphs), and the Thor deployment reports cosine similarity to the FP32 reference across 48 task, scale, and batch configurations. The latency numbers can be trusted, and that is the precondition for assessing the efficiency side of the paper at all.

GTR has a clear advantage under resolution scaling. In Table 6, a 4× increase in tokens raises latency by 2.06× and leaves peak memory flat, and the gap to softmax baselines opens above 1024² (Figure 1, right). This is the one result in the paper that depends on GLA rather than on the teacher or the pre-training data.

Critical weaknesses

1. Novelty

The paper is technically sound: the kernel, the TensorRT deployment, and the heads and recipes for six tasks are complete. Composition can be a contribution, for example when it redefines a problem or adapts its parts to a new setting. This paper offers nothing of that kind. The problem statement is the one already posed by ViG and EfficientViT, both cited in the introduction, and every part is used as is: GLA and its chunkwise algorithm come from Yang et al., the layer-wise alternating scan from ZigMa, the four directions from VMamba, Spatial SwiGLU is a mirrored variant of ConvGLU, and the distillation is plain final-feature L2. The one part that would need adaptation to images, the behavior of the gate on a two-dimensional scan, is neither modified nor analyzed. What remains is an inference-only kernel and a TensorRT deployment. That is an engineering contribution.

The composition is not forced, and the workload is real. The main contribution is that workload. The difference from ViG is small, and what has been added on top of ViG, the scan order, the channel mixer, the distillation, has each been done before; the backbone in Sec 2, taken part by part, exists in prior work. A paper of this kind neither leads the field nor poses a new problem, and whether it meets the bar of ICLR has to be weighed.

2. Source of the accuracy

The paper's accuracy comes mostly from distillation, and the distillation is not the paper's contribution. Table 3 shows GTR-S at 31.1 AP from scratch and 50.7 AP after distillation, a gap of 19.6, while no architectural ablation in Table 4 moves AP by more than 1.3. The teacher and the distillation data follow EdgeCrafter, and the objective is plain final-feature L2. ECDet, a plain ViT student distilled from the same teacher family, reaches higher AP than GTR at all four scales without Objects365 (the ‡ rows of Table 1). Whether the accuracy comes from the architectural design or from the training design, the teacher, the distillation objective, and the Objects365 pre-training, is not verified by any experiment in the paper.

3. Evidence versus claim

The paper's experimental data does not support its conclusion. The claim of "best AP across all four scales" compares Objects365-pre-trained GTR against ECDet without Objects365, while Objects365 alone is worth 2.9 AP (Table 15). With Objects365 removed, ECDet, a softmax-attention student distilled from the same teacher family, outperforms GTR at every scale (the ‡ rows of Table 1). Tables 4 and 5 compare GTR variants against each other only, and the paper contains no controlled GLA-versus-softmax comparison under the same recipe and schedule, so the evidence supports a latency advantage but not the claimed accuracy–efficiency advantage of the architecture. The core conclusion, that linear attention can effectively replace softmax attention, is at present supported by a systems result held up jointly by a strong teacher, strong pre-training, and kernel engineering, and by nothing that isolates the attention itself.

4. Presentation

The text in the figures is too small to read: in Figure 1 the axis labels, legends, and bar values are illegible at print size, and the result figures that follow are the same. The submission is 42 pages and 23.6 MB; about 22 MB of that is embedded images, 18.5 MB of them in the qualitative visualizations on pages 32 to 42. Sec 2, the entire method, runs to about two pages with Figures 2 to 4 inside it; for a paper whose claim is an architecture, that is the whole description of the architecture.

Questions for authors

1. The accuracy–efficiency trade-off has to be shown to exist.

The paper claims an accuracy–efficiency trade-off for the GTR architecture. The evidence does not attribute the accuracy to the architecture rather than to distillation (Weaknesses 2 and 3), so the trade-off itself has to be shown. The natural control is already in the paper: ECDet is a softmax-attention student of the same teacher family, and its published numbers exceed GTR's at every scale without Objects365. A comparison between GTR and ECDet under one protocol, with the data, the teacher, the distillation objective, and the downstream schedule held fixed, and with the configurations and checkpoints of the control released as the paper already promises for its own models, would settle this. The same comparison at the resolutions where GTR's latency advantage appears would also settle Weakness 3.

2. What the gate does has to be explained.

The gate is in the paper's title, and the paper does not say what it does. On a two-dimensional scan the gate decays the state before each patch is written in, so it may be suppressing the regions the downstream task has to recognize, or it may be doing exactly what it is meant to do; the paper gives no evidence either way. What the gate actually does on images has to be explained.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4bRT (1)

Weakness 1: Novelty. The paper is technically sound: the kernel, the TensorRT deployment, and the heads and recipes for six tasks are complete. Composition can be a contribution, for example when it redefines a problem or adapts its parts to a new setting. This paper offers nothing of that kind. The problem statement is the one already posed by ViG and EfficientViT, both cited in the introduction, and every part is used as is: GLA and its chunkwise algorithm come from Yang et al., the layer-wise alternating scan from ZigMa, the four directions from VMamba, Spatial SwiGLU is a mirrored variant of ConvGLU, and the distillation is plain final-feature L2. The one part that would need adaptation to images, the behavior of the gate on a two-dimensional scan, is neither modified nor analyzed. What remains is an inference-only kernel and a TensorRT deployment. That is an engineering contribution.

The composition is not forced, and the workload is real. The main contribution is that workload. The difference from ViG is small, and what has been added on top of ViG, the scan order, the channel mixer, the distillation, has each been done before; the backbone in Sec 2, taken part by part, exists in prior work. A paper of this kind neither leads the field nor poses a new problem, and whether it meets the bar of ICLR has to be weighed.

We thank the reviewer for the careful reading. We respectfully suggest that architectural novelty should not be assessed solely based on whether individual components have appeared in prior work. Rather, it should also consider the underlying design motivation, the way existing components are organized to address a particular problem, and the resulting empirical benefits. For example, RF-DETR [11], accepted at ICLR 2026, demonstrates that effective architectural design and systematic optimization of existing components can constitute a valuable contribution, even when the individual building blocks are not entirely new. Its substantial community adoption, reflected in over 10,000 GitHub stars and 302 citations, further illustrates the practical impact of this design philosophy. Likewise, GTR's novelty lies in its problem-driven architectural design, rather than simply combining existing components.

1. GTR poses a different problem and adopts a different computational paradigm.

ViG and EfficientViT ask how to make non-causal global token mixing linear in cost. GTR asks a different question: can a purely causal recurrence with a fixed-size state and one scan per block, with no softmax attention anywhere, serve as the backbone of a modern real-time detector? ViG cannot answer this question, because it relies on bidirectional recurrence and injects local features into the recurrence itself:

O_{\mathrm{local}} &= \mathrm{DWConv}(X), \\\\ O_{\mathrm{global}} &= \mathrm{BiGLA}(O_{\mathrm{local}}), \\\\ O &= G_{2D} \odot O_{\mathrm{local}} + (1 - G_{2D}) \odot O_{\mathrm{global}}. \end{aligned}$$ ```GTR places DWConv in the value branch of an independent Spatial SwiGLU block, outside the recurrence```: $$\begin{aligned} {[G, V]} &= \mathrm{Linear}_{g,v}(X), \\\\ V_{\mathrm{local}} &= \mathrm{DWConv}_{2D}(V), \\\\ Y_{\mathrm{spatial}} &= \mathrm{Linear}\big(\mathrm{SiLU}(G) \odot V_{\mathrm{local}}\big), \end{aligned}$$ while in GLA each token accesses only preceding positions in the scan order. Separating local spatial interaction from recurrent token mixing removes the need for bidirectional recurrence within each block while keeping a fixed-size recurrent state for efficient inference. ```VMamba [12] needs four scans per block, ViG needs two, and GTR needs one. This is not a "small difference from ViG"; it is a different computational decomposition.``` **2. ```The characterization that "every part is used as is" does not fully reflect our design.```** | Component | Prior work | GTR | |---|---|---| | Scan directions | VMamba: 4 scans within every block; ViG: 2 | ```1 causal scan per block.``` | | Layer-wise alternation | ZigMa [13] alternates zigzag patterns so the scan itself preserves spatial continuity | ```The scan is not responsible for continuity; DWConv is. The recurrence handles only long-range propagation.``` | | Spatial SwiGLU | ConvGLU [14]: DWConv on the gate branch | ```DWConv on the value branch.``` | | Distillation | Multi-stage alignment or multiple objectives [1–3] |``` A single final-feature L2 loss bridges the causal/non-causal gap (Table 3). The simplicity is the finding.``` |
AuthorsOpenReview ↗

Response to Reviewer 4bRT (2)

3. Three insights beyond composition.

Insight 1: Long-range dependency modeling and local spatial modeling should be decoupled.

Recurrence captures long-range dependencies and DWConv models local interactions. Handling locality explicitly removes the need for additional scan directions and enables a single causal recurrence per block without loss in accuracy (Table 4). The contribution is not "adding DWConv" but showing that this decoupling makes a purely causal architecture effective. The single-scan design therefore follows from this principle; it was not found by search or hyperparameter tuning.

Insight 2: Knowledge transfers across computational paradigms with a single L2 loss.

Transferring representations from non-causal Transformers to causal recurrent models is difficult because their computational structures differ, and existing approaches use multi-stage alignment or multiple objectives [1–3]. We show that a single feature-level L2 loss is sufficient (Table 3). The contribution is not a new loss but the finding that such simple supervision bridges this architectural gap.

Insight 3: Pure linear attention is sufficient for modern real-time detection.

Most linear-attention backbones [4–8] are validated only in legacy frameworks (Mask R-CNN, Faster R-CNN), which cannot reveal their upper bound, and recent LLMs adopt 3:1 linear–softmax hybrids [9,10]. GTR reaches strong accuracy in a modern real-time detection setting with no softmax attention; a 3:1 hybrid changes AP by only ±0.1 while increasing latency by ~10% (Table 5).

[1] G. Wei and R. Chellappa. ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

[2] Y. Li et al. ViT-AdaLA: Adapting Vision Transformers with Linear Attention. In Advances in Neural Information Processing Systems (NeurIPS), 2026.

[3] H. Qin et al. Copy the Same, Distill the Difference: Initializing Linear Vision Transformers. arXiv preprint arXiv:2609.35745, 2026.

[4] D. Han et al. FLatten Transformer: Vision Transformer Using Focused Linear Attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

[5] D. Han et al. Agent Attention: On the Integration of Softmax and Linear Attention. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

[6] D. Han et al. Demystify Mamba in Vision: A Linear Attention Perspective. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[7] B. Liao et al. ViG: Linear-Complexity Visual Sequence Learning with Gated Linear Attention. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025.

[8] W. Meng et al. PolaFormer: Polarity-Aware Linear Attention for Vision Transformers. In International Conference on Learning Representations (ICLR), 2025.

[9] Qwen Team. Qwen3-Next: Towards Ultimate Training & Inference Efficiency. Blog post, 2025.

[10] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv preprint arXiv:2510.26692, 2025.

[11] I. Robinson, P. Robicheaux, M. Popov, D. Ramanan, and N. Peri. RF-DETR: Neural Architecture Search for Real-Time Detection Transformers. In International Conference on Learning Representations (ICLR), 2026.

[12] Y. Liu et al. VMamba: Visual State Space Model. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[13] V. T. Hu et al. ZigMa: A DiT-Style Zigzag Mamba Diffusion Model. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

[14] D. Shi. TransNeXt: Robust Foveal Visual Perception for Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

AuthorsOpenReview ↗

Response to Reviewer 4bRT (3)

Q1. The accuracy–efficiency trade-off has to be shown to exist. The paper claims an accuracy–efficiency trade-off for the GTR architecture. The evidence does not attribute the accuracy to the architecture rather than to distillation (Weaknesses 2 and 3), so the trade-off itself has to be shown. The natural control is already in the paper: ECDet is a softmax-attention student of the same teacher family, and its published numbers exceed GTR's at every scale without Objects365. A comparison between GTR and ECDet under one protocol, with the data, the teacher, the distillation objective, and the downstream schedule held fixed, and with the configurations and checkpoints of the control released as the paper already promises for its own models, would settle this. The same comparison at the resolutions where GTR's latency advantage appears would also settle Weakness 3.

Weakness 2. Source of the accuracy. The paper's accuracy comes mostly from distillation, and the distillation is not the paper's contribution. Table 3 shows GTR-S at 31.1 AP from scratch and 50.7 AP after distillation, a gap of 19.6, while no architectural ablation in Table 4 moves AP by more than 1.3. The teacher and the distillation data follow EdgeCrafter, and the objective is plain final-feature L2. ECDet, a plain ViT student distilled from the same teacher family, reaches higher AP than GTR at all four scales without Objects365 (the ‡ rows of Table 1). Whether the accuracy comes from the architectural design or from the training design, the teacher, the distillation objective, and the Objects365 pre-training, is not verified by any experiment in the paper.

Weakness 3. Evidence versus claim. The paper's experimental data does not support its conclusion. The claim of "best AP across all four scales" compares Objects365-pre-trained GTR against ECDet without Objects365, while Objects365 alone is worth 2.9 AP (Table 15). With Objects365 removed, ECDet, a softmax-attention student distilled from the same teacher family, outperforms GTR at every scale (the ‡ rows of Table 1). Tables 4 and 5 compare GTR variants against each other only, and the paper contains no controlled GLA-versus-softmax comparison under the same recipe and schedule, so the evidence supports a latency advantage but not the claimed accuracy–efficiency advantage of the architecture. The core conclusion, that linear attention can effectively replace softmax attention, is at present supported by a systems result held up jointly by a strong teacher, strong pre-training, and kernel engineering, and by nothing that isolates the attention itself.

Response to Q1, Weakness 2, and Weakness 3

We thank the reviewer for raising these concerns. However, we believe the above comments are largely rooted in a misunderstanding of our contributions and, more importantly, an inaccurate interpretation of our experimental results. We clarify both points below.

(1) Distillation is part of our contribution, not an alternative explanation that invalidates it.

The reviewer repeatedly argues that GTR's accuracy mainly comes from distillation and that distillation is not our contribution. We respectfully disagree with this premise. As explicitly stated in our Insight 3 (Weakness 1 response), our contribution is precisely to demonstrate that pure linear attention, together with a simple and efficient distillation strategy, is sufficient for modern real-time detection. The fact that distillation improves accuracy does not invalidate the architectural contribution. More importantly, the original scratch result in Table 3 was affected by an unintended learning-rate configuration, as explained below.

Reviewer_bq9UOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Gated Token Recurrence (GTR), a linear-complexity vision backbone for dense prediction tasks. GTR replaces quadratic softmax attention with a softmax-free, single-scale recurrent architecture based on Gated Linear Attention (GLA). It introduces three key design choices: (1) alternating four-directional scans across depth to capture 2D context, (2) a "Spatial SwiGLU" block that injects depthwise convolution for local mixing, and (3) a simple distillation strategy that aligns only the final patch tokens with a frozen, detection-specialized DINOv3 teacher. The authors demonstrate strong performance across six dense prediction tasks (detection, segmentation, pose, oriented detection, semantic seg, depth) and deploy the model on NVIDIA DRIVE AGX Thor, achieving low-latency FP16 inference with a custom CUDA operator.

Critical strengths
  1. The paper goes beyond single-task evaluation by demonstrating that a single backbone design transfers effectively to six diverse dense prediction tasks—object detection, instance segmentation, pose estimation, oriented object detection, semantic segmentation, and monocular depth estimation.
  2. The integration of a custom chunkwise CUDA operator (achieving a 4.0× speedup over FLA v0.5.0) and TensorRT deployment on automotive hardware represents a significant practical contribution.
Critical weaknesses
  1. The core components are largely incremental combinations of existing ideas (GLA and ViG). The primary novelty lies in the specific combination and the distillation/deployment recipe. The main difference is "one scan per block" and "alternates directions across depth", which is also adopted in Swin Transformer. Is this difference significant enough to warrant a new architecture name, or is it a hyperparameter tuning?
  2. In Table 1/Table 7, GTR uses 640 * 640 inputs, while RF-DETR uses native scale-specific resolutions. The paper notes this but does not provide an ablation of GTR at these native resolutions.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer bq9U (1)

Weakness 1: The core components are largely incremental combinations of existing ideas (GLA and ViG). The primary novelty lies in the specific combination and the distillation/deployment recipe. The main difference is "one scan per block" and "alternates directions across depth", which is also adopted in Swin Transformer. Is this difference significant enough to warrant a new architecture name, or is it a hyperparameter tuning?

We thank the reviewer for raising this important concern. We respectfully clarify that GTR is not an incremental combination of GLA and ViG, nor is its novelty merely a different scan configuration obtained through hyperparameter tuning. Its design is motivated by a fundamentally different computational paradigm and three key insights.

  1. GTR adopts a different computational paradigm from ViG and Swin Transformer.

ViG integrates 2D depthwise convolution into the input of bidirectional GLA, allowing local features to participate directly in bidirectional global context modeling:

O_{\mathrm{local}} &= \mathrm{DWConv}(X), \\\\ O_{\mathrm{global}} &= \mathrm{BiGLA}(O_{\mathrm{local}}), \\\\ O &= G_{2D} \odot O_{\mathrm{local}} + (1-G_{2D}) \odot O_{\mathrm{global}}. \end{aligned}$$ Swin Transformer, in contrast, relies on non-causal self-attention within regular and shifted windows. GTR adopts a different design by placing DWConv in the value branch of an independent Spatial SwiGLU block, rather than in the input of GLA: $$\begin{aligned} {[G, V]} &= \mathrm{Linear}_{g,v}(X), \\\\ V_{\mathrm{local}} &= \mathrm{DWConv}_{2D}(V), \\\\ Y_{\mathrm{spatial}} &= \mathrm{Linear}\big(\mathrm{SiLU}(G) \odot V_{\mathrm{local}}\big). \end{aligned}$$ where each token accesses only preceding positions in the scan order. **By separating local spatial interaction from recurrent token mixing, GTR eliminates the need for bidirectional recurrence within each block, while retaining a fixed-size recurrent state for efficient inference.** 2. **GTR is motivated by three complementary insights.** ```Insight 1: Long-range dependency modeling and local spatial modeling should be decoupled.``` We argue that **long-range propagation and local spatial interaction are distinct problems** best addressed by complementary operators: recurrence captures long-range dependencies, while DWConv models local interactions. By explicitly handling locality, GTR eliminates the need for additional scan directions, enabling a single causal recurrence per block without sacrificing performance **[Table 4]**. Thus, our contribution goes beyond simply adding DWConv; we demonstrate that decoupling locality from recurrence enables an effective purely causal architecture. ```Insight 2: GTR demonstrates an effective knowledge transfer strategy across different computational paradigms.``` Transferring representations from non-causal Transformers to causal recurrent models is challenging due to their different computational structures. Existing approaches often require multi-stage alignment or multiple distillation objectives ([1],[2],[3]). In contrast, we demonstrate that a single L2 loss is sufficient to transfer knowledge **[Table 3]**. ```The contribution is not a new loss function, but the finding that such a simple feature-level supervision strategy effectively bridges this architectural gap.``` ```Insight 3: GTR validates the effectiveness of purely linear-attention modeling for modern real-time detection.``` GTR is a purely linear-attention architecture. Most linear-attention backbones [4–8] are validated only on legacy frameworks (e.g., Mask R-CNN,Faster-RCNN), which cannot reveal their upper bound, and recent LLMs often adopt 3:1 linear–softmax hybrids [9,10]. In contrast, GTR achieves strong performance in a modern real-time detection setting without any softmax attention. As shown in [Table 5], the 3:1 hybrid changes AP by only ±0.1 while increasing latency by ~10%.
AuthorsOpenReview ↗

Response to Reviewer bq9U (2)

  1. The single-scan design is principle-driven rather than search-driven.

Unlike architecture search approaches that explore multiple configurations to identify favorable architectures, GTR's single-scan design follows directly from our architectural principle of decoupling local spatial interaction from long-range recurrent modeling. It is therefore not an isolated hyperparameter choice, but a consequence of the proposed computational decomposition.

[1] Guoyizhe Wei and Rama Chellappa. ViT-Linearizer: Distilling quadratic knowledge into linear-time vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20737–20747, 2025.

[2] Yifan Li, Seunghyun Yoon, Viet Dac Lai, Franck Dernoncourt, Jason Kuen, Yu Kong, and Trung Bui. ViT-AdaLA: Adapting vision transformers with linear attention. To appear in Advances in Neural Information Processing Systems (NeurIPS), 2026.

[3] Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Shiqi Huang, Min Kass Chong, Wahyu Wiratama, Peng Hu, Chen Gong, Wu Liu, Xi Peng, Chun Jian Ho, and Hongyuan Zhu. Copy the same, distill the difference: Initializing linear vision transformers. arXiv preprint arXiv:2609.35745, 2026.

[4] Dongchen Han et al. FLatten Transformer: Vision Transformer using Focused Linear Attention. ICCV, 2023.

[5] Dongchen Han et al. Agent Attention: On the Integration of Softmax and Linear Attention. ECCV, 2024.

[6] Dongchen Han et al. Demystify Mamba in Vision: A Linear Attention Perspective. NeurIPS, 2024.

[7] Bencheng Liao et al. ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention. AAAI, 2025.

[8] Weikang Meng et al. PolaFormer: Polarity-aware Linear Attention for Vision Transformers. ICLR, 2025.

[9] Qwen Team. Qwen3-Next, 2025.

[10] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture, 2025.

AuthorsOpenReview ↗

Response to Reviewer bq9U (3)

Weakness 2: In Table 1/Table 7, GTR uses 640 * 640 inputs, while RF-DETR uses native scale-specific resolutions. The paper notes this but does not provide an ablation of GTR at these native resolutions.

Thank you for your thoughtful comment. We clarify that 640×640 is a widely adopted standard input resolution for real-time object detection [1–8]. Therefore, our choice follows common benchmarking practice. In contrast, RF-DETR's scale-specific resolutions (512/576/704) are selected through its weight-sharing neural architecture search (NAS). Its search explores 6,468 configurations across 11 resolutions, with an estimated computational cost of approximately 10,000 GPU-hours (200 NVIDIA T4 GPUs × 48 hours) [9].

Nevertheless, following your thoughtful suggestion, we additionally evaluate GTR using the corresponding RF-DETR input resolutions (512/576/704). The results are reported below. Note that all GTR models are trained under a unified training configuration, without model-specific hyperparameter tuning. In particular, GTR requires fewer than one-third of the training epochs used by RF-DETR, demonstrating its training efficiency.

ScaleModelResolutionEpochsParams (M)GFLOPsLatency (min / mean / median, ms)APAP$_{50}$AP$_{75}$AP$_S$AP$_M$AP$_L$
SGTR-S5123012.0722.391.055 / 1.069 / 1.06852.169.356.433.156.970.3
SRF-DETR-S51210032601.207 / 1.285 / 1.29952.971.957.032.058.373.0
MGTR-M5763022.6951.431.414 / 1.421 / 1.42056.574.361.438.261.674.1
MRF-DETR-M57610034791.326 / 1.428 / 1.43154.773.559.236.159.773.8
LGTR-L7043037.21127.552.085 / 2.096 / 2.09659.477.064.943.864.276.0
LRF-DETR-L704100341261.805 / 1.952 / 1.96856.575.161.339.061.073.9

[1] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLOv8. GitHub repository, 2023.

[2] Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, and Guiguang Ding. YOLOv10: Real-Time End-to-End Object Detection. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[3] Glenn Jocher and Jing Qiu. Ultralytics YOLO11. GitHub repository, 2024.

[4] Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. DETRs Beat YOLOs on Real-time Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16965–16974, 2024.

[5] Qiang Chen, Xiangbo Su, Xinyu Zhang, Jian Wang, Jiahui Chen, Yunpeng Shen, Chuchu Han, Ziliang Chen, Weixiang Xu, Fanrong Li, Shan Zhang, Kun Yao, Errui Ding, Gang Zhang, and Jingdong Wang. LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection. arXiv preprint arXiv:2406.03459, 2024.

[6] Longfei Liu, Yongjie Hou, Yang Li, Qirui Wang, Youyang Sha, Yongjun Yu, Yinzhi Wang, Peizhe Ru, Xuanlong Yu, and Xi Shen. EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation. Transactions on Machine Learning Research (TMLR), 2026.

[7] Shihua Huang, Zhichao Lu, Xiaodong Cun, Yongjun Yu, Xiao Zhou, and Xi Shen. DEIM: DETR with Improved Matching for Fast Convergence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15162–15171, 2025.

[8] Yansong Peng, Hebei Li, Peixi Wu, Yueyi Zhang, Xiaoyan Sun, and Feng Wu. D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement. In International Conference on Learning Representations (ICLR), 2025.

[9] Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, and Neehar Peri. RF-DETR: Neural Architecture Search for Real-Time Detection Transformers. In International Conference on Learning Representations (ICLR), 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank all reviewers for their detailed and constructive reviews. We are encouraged that reviewers find that:

  • a single backbone design "transfers effectively to six diverse dense prediction tasks" across four model scales (bq9U, pg2P);
  • the chunkwise CUDA operator and the TensorRT deployment are "a significant practical contribution" (bq9U, pg2P), and the latency numbers "can be trusted" (4bRT);
  • GTR "has a clear advantage under resolution scaling" (4bRT, pg2P);
  • the paper is clearly written (bq9U, pg2P), with detailed training, deployment, and timing protocols (pg2P).

During the rebuttal, we added the following experiments and revisions (details in the individual responses):

  1. Matched comparison with Objects365-pretrained ECDet at all four scales, now in Table 1 (p. 6).
  2. Backbone swap with ViG and EdgeCrafter under identical from-scratch training (responses to pg2P and 4bRT).
  3. Corrected from-scratch baseline (thanks to Reviewer 4bRT): after fixing a learning-rate error, it reaches 44.9 AP instead of 31.1(Table 3, p. 8).
  4. Comparison with RF-DETR at its native resolutions (512/576/704)(responses to bq9U and pg2P).
  5. Full-pipeline runtime for sliding-window segmentation and multi-view depth estimation (response to pg2P).

We hope these responses address the reviewers' concerns, and we thank all reviewers and the AC again for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.