acceptodds
Under review as a conference paper at ICLR 2027

Warped Selective State Spaces: Reusable Computation and Anchor Selection

Abstract

Selective state-space recurrences preserve input-dependent dynamics but complicate reusable computation. This paper connects an exact anchored representation of the unchanged recurrence to two execution decisions: which dynamic transition-summary computations can be removed, and which blocks admit a positive constant anchor under a coordinate-span budget. Feasible-anchor intervals and temporal certificates distinguish repairable failures from failures under every constant anchor. A sharp mean-anchor span bound yields a sharp budget-augmentation guarantee for fixed-tree partitions; a complementary counterexample rules out a width-independent leaf-count approximation at unchanged budget. On 16,384 sampled public-checkpoint roots, block means recover 96.58–99.41% of oracle leaf-count headroom and improve on width-two-only rescue. Matched GPU experiments separate anchor policy from affine/source-only execution. A subsequent evaluation on eight new documents, 130M- and 370M-parameter models, and three layers per model finds that the block-mean policy reduces preparation-inclusive packed-affine latency by 31.3% and 19.5%, respectively, relative to deployed anchors within the same custom executor at the prespecified primary setting. Improvements occur in all 48 model–document–layer cases, with all cases passing full-output numerical checks. Source-only removes specified local-tree arithmetic, while its total-time benefit depends on partition and preparation. These results connect exact coordinate conditions, reusable propagation, and measured policy benefits within the tested execution families.

Primary area:
learning on time series and dynamical systems
Keywords:
state space models, selective recurrence, anchor selection, parallel scan, numerical conditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_miRuOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is outside my area of expertise, so I have limited confidence in my assessment.

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

1 comment

Thank you for evaluating the paper. I have revised the presentation to make its main contribution clearer: exact conditions for feasible anchored coordinates, sharp anchor-policy guarantees, and a precise account of the transition work that source-only execution removes.

I also added a separate experiment to test a practical consequence of the representation. In its exact form, paired corrections restore token-dependent variation around fixed propagation. The new question is whether every layer needs these corrections for task accuracy. I tested a layerwise delta-warp residual-correction mask, retaining independent state rates; this is an approximation, not an exact-execution speedup. I chose two layers to match the published Based/Zoology MQAR reference depth, with one common training recipe for Native and omission models. In this single-seed experiment, omitting the first-layer correction gives 79.43% example-averaged accuracy. Two same-seed Native executions under the same recipe give 77.91% and 79.62%; their difference is disclosed and unexplained. The omission result is therefore comparable, not an established improvement. Omitting the second-layer correction gives 61.16% and also fits training substantially less well. This provides a concrete learning example and motivates further study of correction placement, while leaving general preservation and optimization effects unresolved.

The layerwise definition and conditional depth bound also apply to four or eight layers; quality and cost at those depths remain unmeasured for this approximation.

The compiled prototype reduces static chunk-loop instructions by 19.3% and shared-memory reservation by 34.6%. Additional local storage, slower complete SCAN and trained-input numerical qualification failures are reported alongside these gains. The paper now distinguishes the established mathematical and resource contributions from the unresolved optimization and broader performance questions.

Reviewer_S1VwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies an exact “anchored” reformulation of Mamba-style selective SSMs. With a fixed anchor aa, propagation depends only on segment length through Λm=eAam\Lambda^m=e^{Aam}, while selectivity is absorbed into the coordinate factors Pt=eArtP_t=e^{Ar_t}, eliminating dynamic transition products from the scan. The trade-off is that the condition numbers of these factors grow exponentially with the residual span R(a)R(a), making chunking necessary. The theoretical contributions include an exact characterization of the feasible anchor interval, a temporal-ordering obstruction, the tight bound R(aˉ)≤(2−2/T)R⋆R(\bar a)\le(2-2/T)R^\star for the mean anchor and corresponding partitioning guarantees, and a precise accounting of computational costs. On an H100, the mean-anchor strategy is 1.66× and 1.30× faster than the default anchor in the two tested settings. However, an optimized parallel physical scan is 11–18× faster than the anchored methods.

Critical strengths
  1. Rigorous and thorough theory. The feasibility characterization is exact, the mean-anchor bound is shown to be tight through an explicit construction, and the results are supported by exhaustive checks using rational arithmetic.

  2. Transparent experimental reporting. The paper includes a strong baseline and openly reports unfavorable results.

Critical weaknesses
  1. Limited practical significance. The range constraint is introduced by the anchored representation itself; the physical scan does not have this issue. According to the paper’s own results, the anchored methods are 11–18× slower than the parallel physical scan and use roughly 25–30× as much GPU memory. Appendix B.4 further shows that they are inferior to the corresponding physical-clock algorithms even in operation count alone. The reported 1.3–1.7× speedups are therefore comparisons within a substantially slower family of methods.

  2. Incremental novelty, small-scale experiments, and difficult presentation. The core identity substantially overlaps with GateLoop and GLA, and the theoretical tools are classical. The experiments cover only two small models, a batch size of 1, and eight documents, with no end-to-end evaluation. The writing is overloaded with qualifications, making the main argument difficult to follow.

Questions for authors
  1. Are there settings in which the anchored methods outperform the strongest physical-scan baseline in total cost, such as training, backpropagation, or reuse across sequences?

  2. Beyond introducing a non-unit anchor, what specifically is technically novel relative to GateLoop and GLA?

  3. Do the partitioning results and mean-anchor guarantees extend beyond a fixed binary partition tree?

2 comments

Thank you for recognizing the theory and controls. The revision clarifies the distinction from prior gated recurrences and adds an anchored approximation study.

  1. Relation to GateLoop and GLA (Question 2).

GateLoop develops data-controlled recurrence and a prefix-product attention form; GLA develops efficient parallel/chunkwise evaluation of gated linear attention. Both already absorb cumulative transitions into input/output factors. I credit that shared algebra, rather than claim a new prefix identity.

Anchored (delta-warp) coordinates re-express an unchanged selective SSM, exposing anchor feasibility and reusable propagation. For constant a,

Λ=eAa,rt=∑j=1t(Δj−a),Pt=eArt.\Lambda=e^{Aa},\qquad r_t=\sum_{j=1}^t(\Delta_j-a),\qquad P_t=e^{Ar_t}.

Then eAΔt=PtΛPt−1−1e^{A\Delta_t}=P_t\Lambda P_{t-1}^{-1}. Beyond this identity, the contribution is exact feasible-anchor intervals, temporal-order obstructions, a sharp mean-span factor 2−2/T2-2/T, and partition guarantees distinguishing enlarged-budget control from potentially unbounded same-budget fragmentation. The source-graph ledger identifies removed transition products while charging preparation, transforms and carry. The range obstructions concern these coordinates, not physical FP32 scans.

The revision adds a layerwise delta-warp residual-correction mask. With a layer mask m∈0,1m\in\\{0,1\\}, both coordinate factors are retained or omitted together:

qt(m)=Λqt−1(m)+Pt−mvt,ht(m)=Ptmqt(m),q_t^{(m)}=\Lambda q_{t-1}^{(m)}+P_t^{-m}v_t,\qquad h_t^{(m)}=P_t^m q_t^{(m)}, Ft(m)=PtmΛPt−1−m=exp⁡!(A[a+m(Δt−a)]).F_t^{(m)}=P_t^m\Lambda P_{t-1}^{-m}=\exp\\!\left(A[a+m(\Delta_t-a)]\right).

Here vt=ΔtBtutv_t=\Delta_tB_tu_t is unchanged. Mask 1 recovers Native; mask 0 removes residual-coordinate work while retaining independent state rates, selective writes/readout and consistent carry. It loses token-dependent forgetting. GateLoop already includes fixed-transition special cases, and GLA studies gate ablations. The added contribution is anchor-defined correction selection, its conditional error/carry analysis and quality/cost study, not fixed transitions or generic gate omission themselves.

  1. Beyond a fixed binary tree (Question 3).

The interval and mean-span results are blockwise. Appendix B extends partition guarantees to any fixed finite interval-refinement tree (independent positive block anchors, singleton terminals): the oracle cut minimizes leaf count; the mean cut at enlarged budget coarsens it.

For unrestricted contiguous segmentation,

Kmean,all(γ(L)z)≤K⋆,all(z)≤Kmean,all(z),K_{\mathrm{mean,all}}(\gamma(L)z)\le K_{\star,\mathrm{all}}(z)\le K_{\mathrm{mean,all}}(z),

where KK counts the fewest blocks meeting budget zz under mean or optimal (⋆\star) anchors, and γ(w)=1\gamma(w)=1 for w≤3w\le3, otherwise 2−2/(w−1)2-2/(w-1). These constants use the physical-first-token convention. Dyadic counts and the counterexample retain their scope.

  1. Practical value and new evidence (Question 1).

The policy benefit is measured: mean anchors improve preparation-inclusive PAIR execution by 1.658x/1.298x within the anchored family. It does not overturn the original 11-18x latency disadvantage and unfavorable peak memory against physical controls; exact-execution gains in training/backpropagation/reuse remain unestablished.

The new mask study tests correction placement (Sections 5, 6.5-6.6; Appendix C). Two layers match the Based/Zoology MQAR reference depth; all models use seed 123, LR 10−2.510^{-2.5} and 32 epochs, not the published sweep. Example/query accuracies are 77.91/39.56% for full correction, 61.16/17.74% for second-layer omission and 79.43/42.50% for first-layer omission. Another same-seed Native run gives 79.62/41.41%; both full-correction runs use original Native forward, and their difference is unexplained. First-layer omission supplies a comparable-quality case, not established superiority. Second-layer omission has higher training CE (0.713 versus about 0.003), leaving optimization and expressivity unresolved.

The resource benefit also survives compilation: 19.3% fewer static loop instructions and 34.6% less shared-memory reservation than optimized Native (Table 3). Total peak memory was not measured in this omission screen. Added preparation/local storage accompany a 14.8-20.6% complete-SCAN slowdown (14.9-19.8% in a second session).

Scheduling has been tuned, but coefficient storage/live-value arrangement and a specialized backward remain open. These results do not establish the best attainable latency or total memory, nor that unfinished optimization caused the slowdown. Epoch times use different backends. Only 2/18 trained-input cases qualify all kernels numerically, so no qualified trained-model speedup is claimed.

The definition and conditional bound extend to four/eight layers; quality and cost there are unmeasured. This single-seed study establishes no general quality or end-to-end superiority. The original GPU panel is unchanged (two checkpoints, batch one, eight documents); repeated qualifications are consolidated.

Thank you for your response and for clarifying the wording in the revision posted today. My concerns about the practical benefits relative to the optimized physical scan remain, so I will maintain my original rating.

Reviewer_USPzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates exact anchored execution for selective state-space models. It uses a static anchored representation to substitute input-dependent selective recurrence, shifting selectivity to source and readout terms and removing residual clocks. It presents a set of propositions, corollaries and theorems covering exactness, feasibility, ordering constraints, span bounds, budget properties and FP32 numerical limits, and analyzes the runtime cost of affine and source-only execution paths. H100 GPU experiments show moderate speedups for the proposed anchor policy over baselines, while 64-bit full-parallel controls are far faster and use much less memory. Additional measurements reveal boundary-mean computation is more costly than physical-clock computation.

Critical strengths
  1. It has a rigorous, coherent theoretical foundation. Lemma 1 and Corollary 4 distinguish budget augmentation from fixed-budget fragmentation.
  2. The FP32 numerical impossibility result is neat.
Critical weaknesses
  1. The title shown on the submission panel is inconsistent with the title inside the PDF manuscript.
  2. Anchored execution lacks competitive practical benefits. It needs new planning and transforms for every input, so precomputed plans cannot be reused across different inputs.
  3. SOURCE only brings 0.6–1.2% improvement; anchored execution is much slower and more memory-heavy than the 100%-fidelity parallel baseline. No favorable operating regime is demonstrated.
  4. Excessive internal jargon and disclaimers obscure the core ideas.
  5. The work misses key implementation details, checkpoints and data for independent verification; exported metrics cannot support element-wise auditing.
Questions for authors
  1. Is there any practical selective-SSM scenario where anchored execution beats physical-clock execution?
  2. Since FP32 paths fail some elementwise checks while COEFF64 passes all, is the residual error dominated by rounding of exponential arguments instead of scan accumulation?
1 comment

Thank you for recognizing the theoretical foundation. I have clarified the original execution claims and added a separate experiment motivated by your practical-benefit concern.

  1. Practical value and complete cost (Weaknesses 2-3; Question 1).

The results establish benefits at two levels. First, feasible-anchor intervals turn anchor selection into a certified decision; mean anchors improve preparation-inclusive PAIR execution by 1.658x/1.298x within the same anchored family. Second, the new approximation gives compiled resource savings and a comparable-quality omission case. The exact anchored path still does not beat the strongest qualified physical control overall; its unfavorable latency and peak-memory results remain reported.

The new layerwise mask uses the anchored (delta-warp) separation of fixed state-specific propagation from paired residual corrections. I tested whether selected layers can omit those corrections while retaining task accuracy and independent state rates. Sections 5 and 6.5-6.6, with details in Appendix C, report quality and complete SCAN cost. For rate A, anchor a and mask m,

ht(m)=exp⁡!(A[a+m(Δt−a)])ht−1(m)+vt.h_t^{(m)}=\exp\\!\left(A[a+m(\Delta_t-a)]\right)h_{t-1}^{(m)}+v_t.

Mask 1 recovers the original recurrence (Native); mask 0 keeps state-specific propagation and selective writing/readout but loses token-dependent forgetting. Appendix C gives consistent carry and conditional error bounds.

Two layers match the published Based/Zoology MQAR reference depth. All models use one common recipe (seed 123, LR 10−2.510^{-2.5}, 32 epochs), not its full sweep. Mask digits follow layer order:

Model/executionMaskExample accuracy (%)Query accuracy (%)
Native, earlier execution A1179.6241.41
Full correction, Native execution B1177.9139.56
Omit second-layer correction1061.1617.74
Omit first-layer correction0179.4342.50

The 01 result demonstrates a comparable-quality case with first-layer correction omitted. Both Native runs use original Mamba forward without P and agree in recorded seed, recipe, data and initial tensors; their 1.7/1.8-point difference is unexplained. These are not independent seeds or proof of noninferiority. The weaker 10 also fits training less well (CE 0.713 versus about 0.003), leaving optimization and representational effects unresolved.

The definition and conditional depth bound also apply at four or eight layers; quality and cost there are unmeasured for this approximation.

The resource benefit is concrete: static loop instructions fall from 1496 to 1207 (19.3%), and shared-memory reservation from 1664 to 1088 bytes per thread block (34.6%). These reductions survive compilation; they are not only an algebraic count. Total peak memory was not measured in this omission screen. Preparation and local storage increase, and complete SCAN is currently 14.8-20.6% slower than the faster Native (14.9-19.8% in a second session).

Forward scheduling optimizations are already included, but coefficient storage and live-value arrangement remain open. The learning adapter also lacks a matched specialized backward. The measurements characterize these implementations, not the best attainable latency or total memory of the formulation. They do not isolate the cause of the slowdown. After training, only 2/18 cases qualify all kernels numerically (Native also fails cases), so no qualified trained-model speedup is claimed.

  1. Reuse and precision (Weakness 2; Question 2).

Residuals and transformed operands must be recomputed per input. Appendix B gives conditional plan reuse: Rold(a)+2ε≤zR_{\mathrm{old}}(a)+2\varepsilon\le z, where RoldR_{\mathrm{old}} is the old residual span, zz the budget and ε\varepsilon bounds internal prefix-clock changes. Propagation weights are reusable at unchanged A, anchor and lengths. Certificate checking remains chargeable; no amortized speedup is measured.

The precision panel passes 64/64 with COEFF64/ALL64, versus 54/64 with FP32 and 55/64 with ACC64. ACC64 promotes scan/readout but keeps FP32 preparation: promoting accumulation alone barely changes the pass count. This motivates examining preparation, without isolating its causal contribution or exponential-argument rounding; COEFF64 also changes exponential evaluation.

  1. Presentation, title and artifacts (Weaknesses 1, 4-5).

The revision separates exact theory, complete cost and approximation, and consolidates repeated qualifications. I apologize for the earlier mismatch between the submission-page and PDF titles. The revised manuscript is titled “Warped Selective State Spaces: Exact Anchoring and Structured Approximation.”

The supplementary artifact adds per-example counts, both Native records, source/build evidence, hashes and recomputation scripts. The earlier study's weights, corpus assets and raw outputs remain excluded, so that study still cannot be audited element-wise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.