acceptodds
Withdrawn from ICLR 2027

QORLA: A MEASURE-AWARE STATISTICAL SE- QUENCE OPERATOR WITH EXACT FAST EVALUATION AND INNOVATION MEMORY

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformer-based sequence models rely on dense pairwise interactions whose computa- tional cost grows rapidly with context length. QORLA replaces dense query-key scoring with measure-aware statistical estimation over learned scalar geometry, while preserving full-vector values through Laplacian localization and local-linear correction. Its struc- tured formulation admits exact efficient evaluation without materializing the full pairwise interaction matrix, and innovation memory preserves information beyond the statistical predictor. Across diverse sequence-modeling domains, QORLA shows competitive predictive performance, strong transfer across architectures, and favorable long-context computational scaling. These properties position QORLA as an efficient statistical sequence-modeling primitive for scalable alternatives to dense pairwise attention.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Efficient Sequence Modeling, Statistical Sequence Modeling, Attention, Long-Sequence Modeling, Kernel Methods, Local Regression, Innovation Memory, Linear Attention, Efficient Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wrLVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a class of subquadratic attention mechanism as alternatives to dense softmax-based attention. However, the submitted manuscript has clear formatting issues and may not use the ICLR 2027 submission template, e.g., the first page lacks the conference header, and line numbers are missing. Furthermore, the appendix contains clear signs of pure AI-generated content, particularly in the mathematical derivations in Appendix A, but this use of AI is not disclosed in their AI assistance statement. Therefore, I suggest a clear rejection.

Critical strengths

Please see summary.

Critical weaknesses

Please see summary.

Questions for authors

Please see summary.

Reviewer_wmdiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes QORLA, a sequence-mixing operator that replaces dense query–key scoring with kernel-weighted statistical estimation over a learned 1-D scalar geometry: Laplacian localization, a positive learned token measure μ, and a local-linear correction R over a local-constant base, while values remain full vectors. Because the geometry is scalar, the estimator's sufficient statistics are computable exactly via sort + prefix/suffix scans (O(ND log N)/O(ND)), without materializing the N×N matrix; an innovation-memory module (EIDM/IDM) retains residuals beyond the smoother. Validation spans: (i) 30+ mechanisms across six domains; (ii) 1K→16K length extrapolation on synthetic regression; (iii) statistical audits (affine reproduction, error rates, decay); (iv) forward/gradient exactness and latency on T4/L4/A100 up to 65K; (v) local attention replacement in Qwen3-0.6B/Gemma 3 270M; (vi) 2B-token LM pretraining at ~10.2M params vs Transformer++, Mamba-2, Gated DeltaNet. Findings: exactness to ~1e-14; latency crossover vs SDPA at 8K; 1.4×/2.2× throughput vs DeltaNet/Mamba-2; but Transformer++ retains better perplexity, and the authors scope all claims carefully.

Critical strengths
  1. Exact efficient evaluation derived from the estimator's structure, and directly verified. The 1-D geometry makes sort+scan sufficient statistics possible, giving non-quadratic evaluation that is exact, not approximate. This is a genuinely distinct and principled design axis versus low-rank, sparse, kernel-approximation, or recurrent families, and the paper proves the property rather than asserting it.
  2. Unusually rigorous and honest empirical protocol. Statistical audits tie behavior to the claimed mechanism (affine reproduction to ~4e-15 under asymmetry; local-error exponent 1.97 vs predicted 2; distant-token decay slope ≈ −3.0); systems claims are measured on three GPUs; and the discussion explicitly states where SDPA/Transformer++ win. This substantially raises confidence in what is reported.
Critical weaknesses
  1. The quality gap persists in the regimes that matter most. At matched 2B-token scale Transformer++ is stronger in absolute perplexity and raw throughput; powered 8K–16K regression and short-context causal LM favor SDPA; vision is the weakest domain. Wins concentrate in long-context latency and vs recurrent baselines, narrowing the practical value proposition.
  2. Breadth and expressivity claims are weaker than they appear. The headline cross-domain rank uses the best family member per domain, not one fixed configuration, and the expressivity cost of the scalar geometry is under-characterize, layer-0 replacement fails and multi-layer conversion accumulates error, leaving open how much interaction structure is sacrificed.
Questions for authors

What does the learned scalar geometry actually encode, position, semantics, or task-specific structure? Do you ever observe geometry collapse during training, and if so, what prevents it?

In large-scale LM training, QORLA starts from a higher-loss regime than the baselines. What causes this—geometry initialization, bandwidth/ridge hyperparameters? Is early optimization sensitive to these choices?

Why does replacing Qwen layer 0 degrade severely while layer 14 transfers well? Is there a measurable diagnostic (e.g., attention entropy, geometry fit quality) that predicts which layers are QORLA-replaceable before attempting conversion?

Were recent adaptive sparse-attention methods (e.g., Native Sparse Attention, MoBA) included among the 30+ cross-domain mechanisms? If not, how would you expect QORLA to compare against them at matched long-context compute budgets?

Reviewer_YWnVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QORLA replaces softmax attention scores with a statistical estimator. Each query and key is reduced to one learned scalar coordinate per head. Tokens are weighted by a Laplacian kernel on that coordinate times a learned positive weight, and a local-linear correction is added. Because the Laplacian kernel factorizes, the operator can be computed exactly in sorted order in roughly N log N time without forming the full attention matrix. Optional memory variants add a fast-weight store.

The experiments cover a cross-domain survey against many attention alternatives, long-context regression, numerical checks of the estimator's properties, latency and memory profiling on several GPUs, single-layer replacement in pretrained decoders, and small-scale language-model pretraining.

Critical strengths
  • I like how interpretable the operator is. Its sorted evaluation matches the dense form to numerical precision, and the paper's checks confirm its expected properties as a local-linear estimator.
  • I find the systems result convincing. On a T4 at long context the exact evaluator runs faster than PyTorch's dense attention, and in the cross-GPU study it runs at much longer contexts than the dense reference was timed at.
Critical weaknesses
  • My biggest concern is that the abstract claims competitive predictive performance, and the paper's own results only partly support it.
    • In the long-context regression of Table 2, QORLA has lower mean error than the efficient mixers, and at the longest context the paired intervals favor it over the Transformer++-style baseline in every regime. It also ties dense attention on full-data AG News. On language modeling, though, Transformer++ and Mamba-2 reach lower perplexity in the large-token pretraining, and dense attention and Mamba-2 beat QORLA in the multi-seed causal language-model comparison. In the separate powered regression against dense attention (Appendix F.5), dense attention has lower error at long context.
    • The paper acknowledges these results in Section 3.6, the Discussion and Appendices F.5, F.6 and F.7, so I think the abstract and introduction should say plainly that QORLA trails the dense-attention baseline in both language-model training studies. The two language-model training studies also use very small models.
  • The cross-domain survey also seems too weak to me to support the introduction's claim of strong cross-domain competitiveness. The left panel of Figure 1 and Table 1 come from a survey with one seed, very short training, tiny models and capped data, and the paper notes that its rankings should not be read as official reproductions of the cited systems.
Questions for authors
  • How stable are the survey ranks across seeds? Could you repeat the survey with several seeds, at least for the strongest competitor in each domain?
  • Is there a from-scratch language-model training setting, for example a larger model or longer contexts, in which a model with QORLA in every layer matches dense attention or Transformer++ in perplexity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.