acceptodds
Under review as a conference paper at ICLR 2027

SUPPORT GEOMETRY BOUNDS DELAYED COPYING IN COMPACT SEQUENCE MIXERS

Abstract

A sequence mixer can have a receptive field longer than a copying task's required delay and still have no access to the token it must recover. We identify this failure through the reachable lag set , obtained by Minkowski addition of the architectural taps in a residual stack. For finite-bandwidth linear time-invariant mixers, the network's Jacobian support is contained in . On delayed copying with independent payload tokens, a required lag outside therefore limits expected accuracy to the optimal payload-marginal baseline, regardless of the learned weights. We test this exclusion while holding support diameter fixed at under a shared, parameter-matched backbone. Schedules that miss the required lag 54 remain at chance; schedules that cover it exceed 99.9% copying accuracy. Replacing one sparse layer with dense taps restores copying without increasing diameter, whereas coprime dilation rates help only when their finite tap sets cover the required lag. These results separate architectural access from soft influence energy and establish a training-free failure screen for compact mixers on tasks with explicit lag requirements.

Primary area:
learning on time series and dynamical systems
Keywords:
sequence mixers, delayed copying, receptive field, dilated convolution

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_hdKvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on structural limitations of sequence mixers on delayed copying tasks. It characterizes reachable temporal lags through Minkowski sums of layerwise supports and derives a chance-accuracy bound when the required lag is inaccessible. Synthetic experiments confirm this prediction for different dilation schedules.

Critical strengths

The paper provides a clean formulation of architectural reachability and controlled experiments illustrating the difference between receptive-field diameter and exact lag coverage.

Overall the paper is easy to read.

Critical weaknesses
  • Close to no novelty: Missing dependencies within a large receptive field and how to mitigate this phenomenon through heterogeneous dilation schedules were already established, notably by Wang et al. (2018) [1]. The Minkowski-sum characterization is standard support composition, and the copying-accuracy bound follows directly from input independence.

  • Lack of scientific significance. The central experiments essentially confirm what is already implied by the architectural connectivity. They are restricted to small synthetic copying settings and do not demonstrate a substantial new empirical phenomenon or practically consequential methodology. The empirical validation is thus too narrow to establish a significant new contribution independently of the elementary theory in my opinion.


[1] Wang P, et al. Understanding Convolution for Semantic Segmentation WACV 2018

Questions for authors
  • What substantive theoretical or conceptual insight does this work establish beyond existing dilation-gridding and receptive-field analysis?

  • What evidence establishes the practical value of the proposed certificate beyond the synthetic setting where failure is predetermined by graph connectivity?

Reviewer_AVLhOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies architectural limitations of sequence mixers on delayed copying tasks. The main observation is that a sufficiently large receptive-field diameter does not guarantee access to every position within the receptive field. For residual stacks of finite-bandwidth linear time-invariant (LTI) mixers, the authors characterize the set of reachable lags by the Minkowski sum of the layerwise tap sets. They show that if the required copying lag is not reachable, then the model cannot achieve accuracy better than the payload-marginal baseline, assuming independent payload tokens. The authors validate this observation through experiments comparing architectures with the same receptive-field diameter but different reachable lag sets. They further propose using this structural criterion as a training-free screening method and compare it with Jacobian-based influence diagnostics.

Critical strengths
  1. The paper presents a clear and intuitive observation that the maximum receptive-field diameter alone is insufficient to characterize the dependencies accessible to a sequence model. The theoretical results appear correct under the stated assumptions.

  2. The fixed-diameter experiments provide useful evidence for the proposed structural criterion. In particular, the comparisons between sparse and dense convolutional architectures, as well as the cross-layer fill experiments, illustrate how changing the reachable set can restore copying performance without increasing the receptive-field diameter.

Critical weaknesses
  1. The main theoretical results appear relatively straightforward. The characterization of the reachable lag set through Minkowski sums is a standard property of composing convolutional operators, and Lemma 2 follows directly from the chain rule and the architecture's dependency structure. Corollary 1 then follows from the observation that an output cannot depend on an inaccessible input token, together with the independence assumption on the payload. While the resulting impossibility statement is valid, it is unclear what substantial new theoretical insight is provided beyond existing work on receptive fields and dilation gridding. I would encourage the authors to clarify the novelty relative to this literature.

  2. Although the central support-set quantities are formally defined, several additional terms and experimental measures are introduced without adequate explanation in the main text. For example, the precise operator used for computing the Jacobian-based diagnostics is unclear, and quantities such as adaptivity, occupancy, and the different MQAR protocols are not sufficiently motivated or defined before their results are discussed. The paper also introduces two closely related lag quantities, τ\tau and ℓreq\ell_{\mathrm{req}}, which complicates the presentation. A more self-contained experimental setup and clearer organization would significantly improve readability.

Questions for authors

What is the main theoretical novelty compared with existing analyses of receptive-field composition and dilated convolution gridding?

1 comment

Dear Reviewer AVLh,

We sincerely apologize for the delayed response during this busy rebuttal period, and we truly appreciate your patience. We thank you for your careful reading, for confirming the mathematical soundness of our theoretical results, and for highlighting the value of our fixed-diameter and cross-layer fill experiments. Below, we address your question on theoretical novelty and clarify all specific terms raised in Weakness 2.


1. Main Theoretical Novelty & Relation to Dilation Gridding (W1 & Question)

Your reading of the math is accurate: the Minkowski-sum support characterization uses standard support composition, and the independence bound follows cleanly once a missing ancestor is identified. Our contribution lies in combining this support geometry with a task-level impossibility exclusion and strict fixed-diameter falsification:

  • From Geometric Blind Spots to a Pre-Training Certificate (Corollary 1): Existing receptive-field and dilation-gridding analyses warn that spatial blind spots exist in continuous signals. Our result proves that, for discrete causal sequences with independent payloads, missing a single required lag (ℓreq∉R\ell_{\text{req}} \notin R) severs all paths in the unrolled computation graph, locking expected accuracy at the payload-marginal baseline (Pr⁡{x^t=xj}≤max⁡cppay(c)=1/V\Pr\{\hat{x}_t = x_j\} \le \max_c p_{\text{pay}}(c) = 1/V) for every learned weight setting. This converts a geometric observation into a zero-cost, per-task feasibility certificate before training.
  • Positioning Relative to Prior Gridding Work: We position our work relative to Yu & Koltun, Wang et al., and RF-Next in Sections 1–2 and Appendix C. Unlike those works, we do not propose a dilation search heuristic or claim heterogeneous schedules are new. Instead, we test exact bounded membership under controlled alternatives that common heuristics fail to resolve: dense versus sparse at equal Dw=64Dw=64, single-layer fill without increasing diameter, and coprime pairs where one covers lag 54 and one does not. Notably, the (k=5,d=7)+(k=5,d=9)(k=5, d=7) + (k=5, d=9) pair in Table 3 is coprime and has Dw=64Dw=64, yet still misses lag 54 and remains at 0.7% chance accuracy—demonstrating that coprimeness alone is insufficient and bounded finite-set membership is the true criterion.

2. Definitions of Diagnostics, Terms, and MQAR Protocols (W2)

Your concern about self-containment is fair, and we have clarified these definitions upfront in Sections 3.4, 4, and 5.4 before their results are discussed:

  • Jacobian Operator and Diagnostics: The main theorem depends only on TℓT_\ell, RR, DwDw, and ℓreq\ell_{\text{req}}. The Jacobian diagnostics are secondary and defined formally in Appendix B: we compute full-stack input-to-output sensitivity energy Ets(X)=∥∂g(X)t/∂Xs∥F2E_{ts}(X) = \|\partial g(X)_t / \partial X_s\|_F^2 across the entire residual mixer stack gg (skipping inactive rows where ∑s≤tEts≤10−12\sum_{s \le t} E_{ts} \le 10^{-12}), from which r90r_{90} is the nearest-first lag enclosing 90% of causal energy, and mτm_\tau is the normalized energy near diagnostic lag τ\tau. They are intentionally not used as certificates because they can fail both ways: Table 3 shows identical r90=0.73r_{90} = 0.73 for both a chance hole and a successful fill, while Appendix F.3 shows mτ=0m_\tau = 0 for a kernel-63 convolution that still trains successfully.
  • Occupancy and Adaptivity: Occupancy refers to our broader 29-mixer screening suite used to evaluate initialization diagnostics across architectures; adaptivity measures input-dependent variation in the realized Jacobian profile across different probe inputs (separating data-dependent routing in softmax attention from static LTI convolutions, which have zero adaptivity).
  • MQAR Protocols: As summarized in Section 5.4 and detailed in Appendix F.3 (Table 10), the default ("unrepaired") atlas protocol compressed MQAR accuracy across all families (~28%), whereas the "repaired" protocol adjusts the associative-recall task setup so that content-dependent softmax attention reaches 99.8% while LTI convolutions remain near 27%—cleanly separating content-based associative retrieval from fixed-delay copying.

3. Distinction Between the Two Lag Quantities (W2)

The distinction is straightforward: ℓreq=L−P\ell_{\text{req}} = L - P is the true geometric lag from a scored output to its payload source, whereas τ(L)=L−P−1\tau(L) = L - P - 1 is an archived diagnostic coordinate used by earlier initialization-screening protocols (mτm_\tau). Our proofs and membership tests rely exclusively on ℓreq\ell_{\text{req}}; the paper retains τ\tau solely to report historical screening baselines faithfully without altering their original protocol.

We hope these clarifications fully address your concerns, and we respectfully invite you to consider raising your score.

Reviewer_HmwYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies sequence models in which each layer reads only from a fixed set of past positions, such as dilated convolutions. Stacking such layers can leave some past positions that the output never reads, even if those positions lie within the receptive field. The authors show that a model cannot copy a token from such a position better than random guessing. Small-scale experiments support this.

Critical strengths

Strengths

  • The paper includes a wide range of experiments and analyses to validate the core claim, covering many model types and diagnostic tests.
Critical weaknesses
  1. The writing quality is very poor, and the paper is not self-contained, which significantly hurts readability. The core idea is very simple, but the writing hides it, and much of the text seems to fill space without adding new information. Many terms are never clearly defined, for example "adaptivity repair", "Round 2 of task repair", and "status labels refer to the recorded gates". The same quantity appears under different names throughout the paper. For example, the receptive field is also called the "support diameter", DwDw, and "the maximum lookback". The paper would need a major rewrite to be accessible to the ICLR audience.

  2. The core claim is trivial and the contribution is very minor. The paper itself acknowledges that both the skipped positions and the way they are computed come from prior work. What the paper adds is only the fact that a model cannot copy a token it cannot read, which holds by construction. Moreover, the scope of the paper is very narrow and outdated, as it does not apply to modern sequence mixers such as full attention or Mamba. Overall, I do not see a clear empirical or theoretical contribution in this paper.

Questions for authors

Questions

  • Is there a practical setting where the proposed method tells us something beyond simply checking whether the receptive field is long enough?
Reviewer_Jn9bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that simply ensuring the sequence mixer's receptive field diameter covers the required lag (Dw ≥ ℓ_req) is insufficient to guarantee the model can access the target token. The authors establish a precise mathematical framework that spans from tap sets to reachable lag sets, and further to task feasibility. It is proven that if the required lag is not within R, the expected accuracy is locked at a random baseline, regardless of how the weights are trained. The experiments show that in a controlled design with a fixed Dw=64, sparse scheduling remains at a random level of 0.7% due to missing lag 54, while dense or even-step scheduling that covers this lag achieves over 99.9% accuracy.

Critical strengths

Strengths

  1. This paper focuses on an interesting work that addresses arithmetic holes inside the receptive field.
  2. The authors include a solid experimental design isolating the mechanism, including fixed Dw=64, Cross-layer, etc.
  3. The authors include many experiments to support their claims.
Critical weaknesses

Weaknesses:

  1. This paper proved an inventive point about dilated convolutions, but it offers limited practical value, as it was not evaluated on enough real tasks.
  2. Another issue is that the experiments have limited scales; it only tests depth of 2 and length  L with 64, only two random seeds, and other situations are not reported.
  3. Some results are not explained; for example, why the SSM model only achieves 2.2% accuracy is still not well justified.
  4. This paper did not compare with enough adaptive dilation search or other architecture optimization methods, and it did not prove the advantages over other SOTA methods.
  5. Why dense configurations have more FLOPs than some sparse configurations is not a natural conclusion, and it raises concerns that the success of covering architectures may show the effect of greater compute, not the structural access alone.
  6. The so-called delayed copying with independent payloads is a simple task, which cannot reflect real-world situations where tokens are related, and tasks have complex dependencies.
  7. The observation that sparse dilated convolutions have arithmetic holes missing specific lag positions is a fairly straightforward inactivation of dilated convolutions, and the formalization does not yield new design insights.
Questions for authors

Please see the weaknesses. In particular, I want to know the results from: 1) different settings; 2) Failures of other models; 3) application to some real-world applications; 4) the FLOPs comparisons and 5) Comprisons with SOTA methods.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Jn9b (1)

Dear Reviewer Jn9b,

We sincerely apologize for the delayed response, as we were occupied with running extensive supplementary experiments for other submissions during this rebuttal period, and we truly appreciate your patience.

We thank you for your constructive evaluation, for recognizing that our paper "focuses on an interesting work that addresses arithmetic holes inside the receptive field", and for commending our "solid experimental design isolating the mechanism" and extensive experiments. Below, we address your specific questions and concerns.


1. Practical value, real-task scope, and design insights (W1, W6, W7 & Q3)

The certificate is intended to answer a narrower, foundational question rather than general architecture ranking: can a fixed-support mixer possibly access the source token required by a delayed-copy task?

  • Necessity vs. Sufficiency: As stated in Sections 1 and 3.4, a failed membership test rules out above-baseline copying before training, while passing it is a necessary precondition rather than a guarantee of optimization success.
  • Why Independent Payloads Are Required: In real-world text, correlated context allows models to partially guess missing targets (formalized in Appendix C.4 as the conditional Bayes ceiling). Independent payloads eliminate contextual leakage so we can prove an exact population ceiling (1/V1/V). To mark where this hard screen ends and statistical modeling begins, Table 4 and Appendix H show that soft reach does not reliably rank MQAR or real-world enwik8 validation loss across 1M, 10M, and 100M parameter scales.
  • Concrete Design Insights (W7): Our formalization yields two actionable rules that overturn standard heuristics: (i) coprime dilation rates alone are insufficient under finite tap bounds (rates (7,9)(7,9) at Dw=64Dw=64 still miss lag 54 and stay at 0.7%, Table 3), and (ii) Cross-Layer Fill—replacing just one sparse layer with a dense kernel—repairs arithmetic holes and restores 100% accuracy without enlarging diameter DwDw.

2. Scale, seeds, and settings (W2 & Q1)

  • Arbitrary Depth & Length in Theory: The proof is not limited to depth 2 or length 64. Lemma 2 holds at arbitrary depth DD via R=T1+⋯+TDR = T_1 + \cdots + T_D, and Corollary 1 applies to any sequence length and scored position whose required lag is missing.
  • Controlled Mechanism Suites & Extended Results: The main experiments are intentionally compact because they are strict controlled-variable tests: at fixed Dw=64Dw=64 and ℓreq=54\ell_{\text{req}}=54, all models pass the diameter check, isolating interior support membership across 10 predicted configurations (hole, cover, fill, and coprime controls in Tables 2–3). Furthermore, Appendix D.4 reports length-128 screening (29 mixers across 3 seeds {11, 29, 47}), Appendix F.2 summarizes in-distribution copying across 5 seeds and 3 lengths, and Appendix E.5 (Table 9) verifies causal interventions across 6 seeds.

3. Selective SSM failure and other model families (W3 & Q2)

The 2.2% selective-SSM result in Table 4 is not used as evidence for the fixed-support theorem. As explicitly delimited in Section 3.2, selective SSMs and dynamic gates rely on input-dependent state transitions outside the fixed finite-LTI exclusion. Appendix F.5 reports a pre-registered tuning grid over SSM state dimensions ({64, 128}), selection strengths ({0.5, 1.0}), and learning rates ({3e-4, 1e-3}), where the best cell reaches 9.1%—still far below covering finite-support configurations. Alongside Appendix F.2 (which reports GRU at 5.4% and Diagonal SSM at ~0.8%), these rows serve as empirical context on recurrent memory bottlenecks, not certified topological failures.

AuthorsOpenReview ↗

Response to Reviewer Jn9b (2)

4. FLOPs and SOTA / architecture-search comparisons (W4, W5 & Q4, Q5)

  • FLOPs vs. Structural Access (W5 & Q4): The FLOPs concern is fair when comparing successful covering models, and Appendix C.5 explicitly notes that cross-layer fill changes connectivity alongside width, FLOPs, and optimization dynamics. However, our theorem establishes an impossibility screen: when ℓreq∉R\ell_{\text{req}} \notin R, the target Jacobian is identically zero, so no amount of FLOPs or training can create the missing path. Empirically, Table 3 compares two sparse coprime schedules at the same parameter budget and comparable compute: (k=9,d=3)+(k=9,d=5)(k=9, d=3)+(k=9, d=5) covers 54 and reaches 100%, whereas (k=5,d=7)+(k=5,d=9)(k=5, d=7)+(k=5, d=9) misses 54 and remains at 0.7%, confirming that reachability governs solvability.
  • SOTA & Search Comparisons (W4 & Q5): We do not claim leaderboard superiority over SOTA architectures; rather, we provide an O(1)O(1) training-free front-end screen. As noted in Appendix E.4, this screen complements adaptive dilation search methods (such as RF-Next) by filtering out topologically dead candidates at zero cost before training. Per the ICLR Reviewer Guidelines, establishing a rigorous impossibility boundary does not require SOTA leaderboard claims.

We hope these clarifications resolve your concerns, and we respectfully invite you to reconsider your score.

Thanks for the authors' rebuttal. Some of my concerns remain, but they haven't been specifically addressed. I will keep my original rate.

Thank you for your time, your follow-up and your engagement throughout the review process. We truly appreciate your feedback on our experimental design and respect your final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chairs and Reviewers

We sincerely thank the Area Chair, Senior Area Chairs, and all four reviewers (hdKv, Jn9b, AVLh, and HmwY) for their time, careful reading, and constructive engagement throughout the review and discussion period.

Because our theoretical proofs and controlled experiments are complete in the submitted manuscript, we summarize below the key technical consensus established during the discussion and our core contributions relative to prior literature.


1. Consensus on Technical Soundness and Experimental Rigor

Across the initial reviews and follow-up discussions, reviewers affirmed the technical validity of our framework and experimental design:

  • Mathematical Correctness: Both Reviewer AVLh (Soundness: 3/3) and Reviewer hdKv (in their follow-up comment) explicitly confirmed that our theoretical results (Lemmas 1–2 and Corollary 1) are mathematically sound under their stated assumptions.
  • Controlled Experimental Design: All four reviewers recognized the rigor of our fixed-diameter (Dw=64Dw=64) and cross-layer fill controls, which isolate interior arithmetic reachability from outer receptive-field length across diverse sequence-mixer families.

2. Core Contributions Beyond Prior Gridding & Receptive-Field Heuristics

A central theme in the discussion was how our work advances beyond standard receptive-field composition and 2D dilation-gridding heuristics (e.g., Wang et al., 2018). Our submission establishes three distinct contributions that prior heuristics neither predict nor resolve:

  1. From Continuous Anti-Aliasing Heuristics to a Pre-Training Impossibility Certificate (Corollary 1):
    Prior dilation-gridding literature addresses spatial aliasing in 2D continuous signals using empirical rate-selection rules to improve segmentation metrics. In contrast, for 1D discrete causal sequences with independent payloads, we prove an exact information-theoretic bound: whenever ℓreq∉R=T1+⋯+TD\ell_{\text{req}} \notin \mathcal{R} = T_1 + \cdots + T_D, all computational paths to the target token are severed, locking expected accuracy at the random payload-marginal ceiling (1/V=0.8%1/V = 0.8\%) regardless of learned weights, depth, or compute budget. This converts a geometric property into a zero-cost, pre-training feasibility screen.
  2. Falsifying the "Coprime Dilation" Heuristic Under Finite Tap Bounds:
    A prevailing rule in prior gridding work is that choosing coprime dilation rates avoids blind spots. We prove and empirically demonstrate that under finite causal tap bounds, coprimeness is neither necessary nor sufficient: in Table 3, the coprime schedule (k=5,d=7)+(k=5,d=9)(k=5, d=7) + (k=5, d=9) satisfies gcd⁡(7,9)=1\gcd(7,9)=1 and achieves full outer diameter Dw=64Dw=64, yet its bounded Minkowski sum omits lag 54 and remains locked at 0.7% chance accuracy. Conversely, (k=9,d=3)+(k=9,d=5)(k=9, d=3) + (k=9, d=5) or single-layer Cross-Layer Fill covers lag 54 and reaches 100% accuracy without increasing diameter DwDw. Exact bounded Minkowski membership is strictly necessary.
  3. Exposing Soft-Reach Blindness & Delineating Model Boundaries:
    Gradient-based effective receptive-field diagnostics (r90r_{90}) cannot distinguish topological access from structural holes: in Table 3, a dead missing-lag schedule (0.7% accuracy) and a working cross-layer fill (100% accuracy) exhibit identical r90=0.73r_{90} = 0.73. Furthermore, across MQAR (Appendix F.3) and real-world enwik8 language modeling (1M–100M parameters, Appendix H), alongside multi-seed and extended-length controls (Appendices D–F), our paper rigorously delineates where hard topological feasibility ends and learned statistical modeling begins.

We thank the Area Chair and reviewers again for their thoughtful engagement and consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.