acceptodds
Under review as a conference paper at ICLR 2027

HoloRoPE: Holographic RoPE for Hierarchical Wireless Foundation Models

Abstract

Wireless foundation models (WFMs) extend the pretraining paradigm of foundation models to wireless communications, learning high-quality representations from large-scale channel data that serve diverse downstream wireless tasks. The input to WFMs is typically channel state information (CSI), which has two axes that index antennas and subcarriers and sample the propagation channel in space and in frequency, respectively. Adjacent subcarriers in wireless data typically lie within the coherence bandwidth, and we exploit this property by introducing Asymmetric Hybrid-Resolution Attention (AHRA), a hierarchical attention design that better suits wireless signals. AHRA aggregates local signals through pooling, after which prevailing positional encodings assign each new token a single fixed position from the original grid. However, this assignment is limiting for wireless signals, which are sensitive to positional inductive bias. By constructing the response difference between different ways of computing the positional encoding, we decompose it into the main factors that drive the positional difference. On this basis, we propose Holographic Rotary Position Embedding (HoloRoPE), which scales regional keys with learnable gains based on the geometric and content factors. Extensive experiments on large-scale wireless datasets show that both AHRA and HoloRoPE improve performance over the baseline models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hierarchical Wireless Foundation Models, Rotary Position Embedding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bjprOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HoloRoPE, a rotary positional encoding method designed for hierarchical wireless foundation models. First, the authors introduce Asymmetric Hybrid-Resolution Attention (AHRA), which aggregates neighboring subcarrier features in selected attention heads while preserving fine-grained information in the remaining heads. Second, they analyze the positional encoding mismatch introduced by regional aggregation and propose HoloRoPE, which applies content-dependent, frequency-aware amplitude modulation to regional keys while preserving their rotary phases. The proposed methods are evaluated on simulated channel datasets generated using DeepMIMO and Sionna, covering channel extrapolation, channel estimation, LoS/NLoS classification, and beam prediction. Experimental results demonstrate consistent improvements over conventional RoPE and other positional encoding baselines across different antenna configurations and model scales.

Critical strengths
  1. The paper identifies a meaningful limitation of conventional RoPE when applied to pooled regional representations. Its analysis of geometric and content-dependent factors provides a reasonable theoretical motivation for the proposed HoloRoPE design, moving beyond purely empirical modifications to positional encoding.

  2. The experimental evaluation is relatively comprehensive, covering multiple wireless tasks, antenna configurations, channel datasets, and model scales. The consistent improvements over conventional RoPE and other positional encoding methods suggest that HoloRoPE can enhance representation learning in wireless foundation models.

Critical weaknesses
  1. The theoretical analysis does not sufficiently justify the proposed HoloRoPE design. Although the identified positional encoding mismatch involves both amplitude and phase, HoloRoPE only adjusts the amplitude while preserving the original phase. Moreover, comparisons with simpler learnable gating mechanisms and alternative regional position assignments are missing. These limitations raise concerns about whether the proposed formulation offers a substantive advantage over simpler alternatives.

  2. The experimental evaluation lacks sufficient evidence of practical applicability and generalization. All experiments rely on simulated channels without validation using real-world CSI measurements. Essential system parameters, including antenna array sizes, subcarrier counts, and bandwidths, are not clearly specified. Furthermore, the channel extrapolation protocol does not clearly distinguish interpolation from genuine extrapolation. These limitations make it difficult to assess the effectiveness and scalability of HoloRoPE in realistic 5G and emerging 6G systems.

Questions for authors
  1. The authors should clarify how amplitude-only modulation addresses positional encoding mismatches involving both amplitude and phase differences.

  2. Comparisons with simpler learnable key-gating mechanisms and alternative regional position assignments are needed to establish the necessity of HoloRoPE.

  3. Essential wireless system parameters, including antenna array sizes, subcarrier counts, carrier frequencies, and bandwidths, should be explicitly reported.

  4. The channel extrapolation protocol should be clearly specified, particularly whether it involves genuine frequency or spatial extrapolation rather than interpolation or masked reconstruction.

  5. Experiments using real-world measured CSI datasets are needed to demonstrate the practical effectiveness of HoloRoPE.

  6. Cross-scenario and cross-dataset evaluations are needed to assess generalization across propagation environments, frequency bands, and antenna configurations.

  7. Additional experiments should examine robustness under varying coherence bandwidths, delay spreads, and channel estimation noise levels, particularly when adjacent subcarriers exhibit weak correlation.

7 comments
AuthorsOpenReview ↗

Response to Reviewer bjpr (1/5)

We thank the reviewer for the detailed comments and suggestions, and for the positive assessment of our motivation and experiments. We have further clarified in the main text the reasoning behind the motivation and purpose of HoloRoPE, which we believe addresses the reviewer's questions about the method. We have also added capacity-matched gating experiments and the broader experimental validation suggested by the reviewer, and specified the system parameter settings in the paper, making the overall argument and validation more complete. We respond to each question below. The first weakness corresponds to Q1–Q2 and the second to Q3–Q7.

Q1. How does amplitude-only modulation relate to a difference that involves both amplitude and phase?

The comparison between the common-coordinate and constituent-coordinate encodings is meant to reveal how regional encoding depends on geometry, rotary frequency, and content, and the learnable modulation is built on these dependencies. We do not presuppose the constituent-coordinate encoding as the optimal target, nor do we take removing the magnitude and phase differences between the two encodings as a training objective (Appendices A.5–A.6). We thank the reviewer for raising this question, and we have clarified this logic in the revised manuscript. Section 2.3 now includes a paragraph on phase and magnitude effects, which the argument below follows.

Common-coordinate encoding assigns the same positional phase to all constituent contributions. For the geometric factor of a regular window, the phase corresponds to a shift to the window center, while its magnitude falls below one as the rotary frequency grows from zero in the low-frequency regime when the coordinate variance is nonzero (Eq. (10), Appendix A.4). For this geometric component, the remaining phase difference only amounts to moving the representative coordinate to the window center. We added this variant to the revision (RoPE-Center, Table 2), and it changes CExt only marginally, from −40.12/−38.37 dB for RoPE to −40.30/−38.35 dB (ULA/UPA). Across the evaluated rotary families and arrays, the change is at most 0.32 dB (Section 3.2).

The phase of the content–position residual depends on regional features and cannot generally be represented by a position-only rotation (Phase and Magnitude Effects, Section 2.3).

Changing the representative coordinate alone cannot adjust component magnitudes. This motivates content-conditioned magnitude modulation while retaining the common-coordinate phases. HoloRoPE adaptively modulates regional magnitudes and improves CExt to −41.14/−39.19 dB (Table 2). Table 3 directly compares common-coordinate RoPE, constituent-coordinate encoding, and HoloRoPE. The real-valued CExt results are:

Table R1: Common- and constituent-coordinate encoding on DeepMIMO with real-valued models (Table 3, Section 3.2).

DeepMIMO, real-valuedCExt ULACExt UPA
RoPE (Bℓ,jB_{\ell,j})−40.12−38.37
RoPE-Constituent (Tℓ,jT_{\ell,j})−40.54−38.68
HoloRoPE (aℓGℓ,jBℓ,ja_\ell G_{\ell,j}B_{\ell,j})−41.14−39.19

RoPE-Constituent improves CExt over common-coordinate RoPE by 0.42/0.31 dB under ULA/UPA, respectively, supporting our motivation to account for positional information within aggregated regions. RoPE-Constituent retains the individual positional phases of the constituent contributions before summation, whereas HoloRoPE changes only magnitudes and keeps the common-coordinate phases. HoloRoPE still improves on RoPE-Constituent by a further 0.60/0.51 dB and is the best of the three in all 24 entries of Table 3, so the observed gains can be achieved without changing the common-coordinate phases.

AuthorsOpenReview ↗

Response to Reviewer bjpr (2/5)

Q2. Comparisons with simpler key gating and alternative regional position assignments.

We thank the reviewer for this suggestion and have added both kinds of alternatives to the revision (Tables 2 and 3). They are described in Section 3.2 and Appendix B.2. All variants share the same backbone, pooling, and training recipe and differ only in how the regional keys are encoded.

  • RoPE-Center encodes each region at the center of its window instead of at its first patch.
  • RoPE-Constituent rotates each patch's contribution to the pooled key at that patch's own coordinate and then sums the contributions.
  • RoPE-ContGate scales each regional key using a learned scalar per region and regional head, predicted from the pooled content.
  • RoPE-Gate does the same, but predicts the factor from both the pooled content and how much the features vary within the region. It is HoloRoPE's gate on its own.
  • RoPE-Spec lets the gain change across rotary frequencies, with a strength predicted from how much the features vary within the region. It is HoloRoPE's spectral factor on its own.
  • HoloRoPE combines the gate of RoPE-Gate with the frequency profile of RoPE-Spec.

Table R2: Regional position assignments and modulation controls on DeepMIMO with real-valued models (Tables 2 and 3, Section 3.2).

DeepMIMO, real-valuedCExt ULACExt UPABPred-128 ULABPred-128 UPA
RoPE (window-start coordinate)−40.12−38.3766.9060.38
Position assignment
RoPE-Center (window-center coordinate)−40.30−38.3567.3361.51
RoPE-Constituent (each patch at its own coordinate)−40.54−38.6868.7362.54
Simpler gating
RoPE-ContGate (gate on pooled content)−40.37−38.5468.7762.79
RoPE-Gate (gate on content and dispersion)−40.63−38.7968.2463.11
Frequency profile only
RoPE-Spec (learned spectral factor)−40.96−38.9970.0862.32
HoloRoPE−41.14−39.1970.3165.30

All alternatives improve CExt under both arrays except RoPE-Center, which yields a small improvement under ULA but a slight degradation under UPA. HoloRoPE is the best variant in all 24 entries of Table 2 (4 tasks × 2 arrays × real/EH/EC). In CExt, the spectral factor alone improves on RoPE by 0.84/0.62 dB with only 0.03%/0.02% additional parameters under ULA/UPA, respectively (Tables 2 and 9, Appendix C.2), more than either gate, which adds about 3.3% and gives at most 0.51/0.42 dB. These results support the effectiveness of adaptive spectral modulation beyond increasing gating capacity.

Additional controls apply content-only gating to native-resolution keys (RoPE-NatContGate). HoloRoPE outperforms both native-resolution and regional content-only gating across all 24 task–array–family combinations in Table 6 (Appendix C.1), further supporting benefits beyond content-only gating.

AuthorsOpenReview ↗

Response to Reviewer bjpr (3/5)

Q3. System parameters.

We thank the reviewer for pointing this out. Appendix B.3 specifies the dataset configurations and frequency sampling procedures. The reference system parameters are summarized below.

Table R3: Reference system configurations for DeepMIMO, Sionna, and DICHASUS (Appendix B.3).

ParameterDeepMIMOSionnaDICHASUS (measured)
Channel modelSite-specific ray tracing3GPP TR 38.901Outdoor measurements
Array32-element ULA or 8×8 UPA32-element ULA or 8×8 UPA2×8 receive array
Antenna spacing0.5λ0.5λApproximately 0.5λ
Carrier frequency3.5 GHz3.5 GHz1.272 GHz
Nominal bandwidth11.52 MHz11.52 MHz50 MHz
Input frequency samples3232128
Effective frequency sampling interval360 kHz360 kHz390.625 kHz

For Sionna, the original grid contains 384 subcarriers spaced 30 kHz apart. Selecting the first subcarrier from each of 32 resource blocks yields the input frequency samples listed above (Appendix B.3).

These are reference configurations rather than the full range of settings evaluated. Appendices C.5–C.6 report additional experiments across propagation domains, antenna and frequency configurations, and channel conditions. Table 16 evaluates ULA configurations with {8, 16, 32, 64} antennas and {16, 32, 64, 128} frequency samples. Table 17 evaluates frequency sampling intervals of 90, 180, 360, and 720 kHz under both ULA and UPA (Appendix C.5, discussed further in Q6).

Q4. Channel extrapolation protocol.

We agree that the protocol should be stated explicitly. The random target-selection pattern used in the main benchmark evaluates recovery of distributed missing observations. We additionally evaluate three structured patterns defined in Appendix B.1: frequency extrapolation selects a contiguous block at the upper end of the subcarrier axis, spatial extrapolation selects a contiguous block at the end of the antenna axis, and joint extrapolation selects a rectangular corner block spanning the ends of both axes. The target-selection ratio ranges from 0.25 to 0.50 for ULA and from 0.50 to 0.75 for UPA.

Table 10 in Appendix C.3 reports the full results for the real-valued, equal-head-count (EH), and equal-cache-size (EC) variants under all four target-selection patterns. The real-valued results for the three structured extrapolation settings are summarized below:

Table R4: Structured channel extrapolation with real-valued models (Table 10, Appendix C.3).

PatternCExt ULA, RoPE → HoloRoPECExt UPA, RoPE → HoloRoPE
Frequency−26.49 → −27.04−24.50 → −25.61
Spatial−18.61 → −19.58−27.35 → −27.89
Joint−24.02 → −24.71−29.22 → −30.22

HoloRoPE improves performance in all three settings under both ULA and UPA. The same improvements hold for both EH and EC variants, covering all 18 paired comparisons across the three structured patterns, two array configurations, and three rotary families in Table 10. The results support its effectiveness beyond random target selection.

Reviewer_QazSOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript hypothesizes that attention-based wireless foundation models can improve their performance by combining native-resolution and pooled-resolution representations. Furthermore, it argues that rotary positional embeddings are a natural choice for capturing the relative positional relationships between input patches. To combine native- and reduced-resolution representations, the paper proposes an extension of spatial-reduction attention. The central contribution of the work lies in combining rotary positional embeddings with spatial-reduction attention and introducing a learned gain adjustment that aims to mitigate the positional encoding mismatch caused by pooling features from different spatial positions.

Critical strengths
  • The use of native resolution heads together with reduced resolution heads is convincingly motivated with the properties of wireless channels.

  • The work propose a carefully designed positional encoding that adjusts rotary positional encodings for pooled features. The method is well motivated by a theoretical analysis that suggests that an adjustment should incorporate a rotational frequency dependent as well as a content dependent factor.

  • The proposed attention variant together with the positional encoding adjustment outperforms existing baselines

Critical weaknesses

One minor weakness is the specificity of the task: The aggregation of features via Asymmetric Hybrid-Resolution Attention and holographic rotary position embeddings should also be applicable to more general data types, such as images, which can, for instance, contain redundant information in homogeneous regions. The chosen wireless channel task seems highly specific, although the proposed method itself is not inherently restricted to this domain. Evaluating in more common scenarios than radio foundation models might help place the performance of the system in a broader context. In general, the arguments made to justify HoloRoPE for wireless could be identically applied to computer vision.

A bigger concern is the lack of standard deviations across runs which might hide some run-to-run variance introduced by the additional weights.

The paper also does not properly distinguish between what is specific to the positional embedding and what is specific to AHRA: e.g. couldn't the HoloRoPE embedding be similarly used on a Swin-Transformer (or Hiera https://proceedings.mlr.press/v202/ryali23a.html, or CrossFormer https://arxiv.org/pdf/2108.00154, or Multiscale Pooling attention as in MViT and MViTv2, or very directly Shunted Self-Attention)? This changes the scope of this paper: Is this work about positional embeddings (in which case you should demonstrate that the positional embedding helps across hierarchical attention schemes), or is it about this specific architecture?

Please provide either benchmarks for other hierarchical transformer schemes, or narrow the claim. Right now, this is more an architecture paper, and less a positional embedding one.

Finally, I would like to see what happens if you directly use equation 10's proposed quadratic expansion: If I understand the claim correctly, this should perform similarly to having a content unaware PE factor (with the benefit of having a fixed coefficient).

Questions for authors

see weaknesses section

4 comments
AuthorsOpenReview ↗

Response to Reviewer QazS (1/2)

We thank the reviewer for the detailed suggestions, and for the positive assessment of our motivation, method, and performance. Overall, HoloRoPE is a positional encoding for attention whose keys are pooled, and AHRA is the architecture we propose for wireless data and in which we evaluate it. Following the reviewer's suggestions, we have added multi-seed experiments, experiments with HoloRoPE on an image task and on other models with regional aggregation (SSA and MViTv2), and an experiment that directly uses the expansion in Eq. (10). These experiments position and validate the scope of HoloRoPE more clearly. We respond to each comment below.

W1. Task specificity and applicability to images.

We thank the reviewer for this suggestion. We equipped MViTv2, whose pooling attention aggregates keys and values, with RoPE and with HoloRoPE, and trained both on CIFAR-100 with the same recipe. HoloRoPE improves top-1 accuracy from 68.38% to 68.50% (Appendix C.2). We regard this as an initial evaluation on image data. Wireless channel modeling remains our primary focus.

Taken as a whole, we still see our design, AHRA together with HoloRoPE, as primarily tailored to wireless communication. Wireless channels often exhibit local correlation across adjacent subcarriers, motivating selective regional aggregation. AHRA pools keys and values in selected heads while keeping the remaining heads at native resolution to retain local channel detail.

The relation between two CSI patches depends on their relative offset, so RoPE is the natural encoding, and HoloRoPE aims to remove the bottleneck RoPE meets when it encodes aggregated information. Image models with local aggregation do face a similar problem, but in CSI the axes sample physical quantities, the spatial wavefront along the antennas and the frequency response along the subcarriers, so the scale over which neighboring samples share structure is set by the channel itself. The coherence bandwidth characterizes the frequency range over which channel samples remain correlated, motivating local aggregation along the subcarrier axis. We therefore believe this design fits wireless communication better and keep it as the main setting of the paper.

W2. Run-to-run variance.

We thank the reviewer for raising this point and have added experiments with three random seeds (42, 3407, and 5678), as described in Appendix B.2. Table 1 now reports the mean and sample standard deviation for the full-resolution RoPE baseline and, under AHRA, for RoPE, its two complex-valued variants, and the corresponding HoloRoPE variants, on both datasets, both arrays, and all five metrics. The real-valued results are:

Table R1: Three-seed results for real-valued RoPE and HoloRoPE (Table 1 and Appendix B.2).

Dataset and arrayCExt, RoPECExt, HoloRoPEBPred-128, RoPEBPred-128, HoloRoPE
DeepMIMO ULA−39.97 ± 0.17−41.03 ± 0.1666.40 ± 0.5270.57 ± 0.35
DeepMIMO UPA−38.42 ± 0.20−39.20 ± 0.2860.85 ± 0.4865.74 ± 0.42
Sionna ULA−15.67 ± 0.16−16.66 ± 0.0565.07 ± 0.4373.78 ± 0.45
Sionna UPA−16.23 ± 0.17−16.94 ± 0.1159.65 ± 0.4365.55 ± 0.39

Across the 60 paired comparisons in Table 1, HoloRoPE's improvement exceeds twice the larger of the two standard deviations in 59 of the 60 comparisons. The exception is LoS/NLoS accuracy of the EC variant on DeepMIMO-UPA, where both models are already near 99% (98.26 ± 0.37 vs. 98.86 ± 0.35).

Full results for the real-valued, EH, and EC variants are reported in Table 1, with the repeated-seed protocol described in Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer QazS (2/2)

W3 & W4. Positional embedding or architecture, and other hierarchical schemes.

We thank the reviewer for these two related questions and answer them together. AHRA determines which heads aggregate regional keys and values, whereas HoloRoPE modulates the magnitudes of those rotary-encoded keys while preserving their phases and can be applied beyond AHRA. Following the reviewer's suggestion, we evaluated HoloRoPE on other hierarchical attention schemes, building wireless backbones on Shunted Self-Attention (SSA) and on the pooling attention of MViTv2 (Table 8, Appendix C.2). In both backbones, HoloRoPE improves on the corresponding rotary baseline in all 48 paired comparisons (2 backbones × 3 model variants × 4 tasks × 2 arrays).

Within each backbone and rotary family, paired RoPE and HoloRoPE variants share the same backbone configuration, pooling settings, and training protocol. The following tables summarize the real-valued results on DeepMIMO. Full results for the real-valued, EH, and EC variants are reported in Table 8 of Appendix C.2.

Table R2: Performance on the SSA-based backbone with real-valued models (Table 8, Appendix C.2).

Array, RoPE → HoloRoPECExtCEstL/NBPred-128
ULA−42.15 → −43.05−20.10 → −20.5396.56 → 97.6967.73 → 71.05
UPA−38.64 → −39.45−20.28 → −20.8296.75 → 98.5960.77 → 66.23

HoloRoPE reduces CExt NMSE by 0.90/0.81 dB and CEst NMSE by 0.43/0.54 dB under ULA/UPA, respectively, while also improving both classification and beam prediction accuracy.

Table R3: Performance on the MViTv2-based backbone with real-valued models (Table 8, Appendix C.2).

Array, RoPE → HoloRoPECExtCEstL/NBPred-128
ULA−40.76 → −41.50−19.37 → −19.7696.73 → 97.8467.28 → 70.52
UPA−34.95 → −35.77−19.26 → −19.7296.37 → 97.6860.25 → 64.89

HoloRoPE reduces CExt NMSE by 0.74/0.82 dB and CEst NMSE by 0.39/0.46 dB under ULA/UPA, respectively, with improvements in both accuracy metrics. Together, these comparisons show that the benefits extend beyond AHRA to other hierarchical attention backbones.

We chose these two because both use pooled keys and values, allowing us to evaluate HoloRoPE across different hierarchical attention architectures. The gains also hold under both average and learned semantic pooling (Table 4, Section 3.2) and in the image experiment of W1. Since these results all concern attention with pooled keys, we narrow the claim for HoloRoPE to this setting (Sections 1 and 4). Within this family, AHRA is the variant we design for wireless data, using subcarrier-only pooling in the default configuration.

W5. Using the quadratic expansion of Eq. (10) directly.

We thank the reviewer for this suggestion and have evaluated RoPE-Quad, which applies the fixed factor Gℓ,j=exp⁡(−12σℓ2θj2)G_{\ell,j}=\exp(-\tfrac12\sigma_\ell^2\theta_j^2) motivated by the low-frequency expansion in Eq. (10), with aℓ=1a_\ell=1. Here, σℓ2\sigma_\ell^2 is the coordinate variance of the pooling window. For a shared pooling geometry, the factor is identical across regions and independent of feature content. It therefore implements the content-unaware spectral profile suggested by the reviewer without introducing additional learnable parameters. Implementation details are provided in Appendix B.2.

The real-valued CExt results on DeepMIMO are summarized below:

Table R4: Fixed and adaptive spectral modulation on DeepMIMO with real-valued models (Table 7, Appendix C.1, and Table 9, Appendix C.2).

DeepMIMO, real-valuedParams (×), ULA / UPACExt ULACExt UPA
RoPE1.0000 / 1.0000−40.12−38.37
RoPE-Quad1.0000 / 1.0000−40.62−38.67
RoPE-Spec1.0003 / 1.0002−40.96−38.99
HoloRoPE1.0331 / 1.0330−41.14−39.19

Under the evaluated training setup, RoPE-Quad reduces NMSE by 0.50 dB under ULA and 0.30 dB under UPA. Thus, the fixed approximation yields an empirical improvement over the corresponding RoPE baseline in both array configurations. This comparison directly evaluates the proposed geometric factor without content-dependent adaptation. Full results across tasks and rotary families are reported in Table 7 (Appendix C.1), with parameter-count comparisons in Table 9 of Appendix C.2.

The adaptive RoPE-Spec variant further reduces CExt NMSE to −40.96/−38.99 dB under ULA/UPA, respectively, with only 0.03%/0.02% additional parameters, while full HoloRoPE reaches −41.14/−39.19 dB (Tables 7 and 9). RoPE-Spec outperforms RoPE-Quad in all 24 comparisons across tasks, rotary families, and array configurations (Table 7, Appendix C.1). These comparisons support adaptive spectral modulation beyond the fixed geometric approximation.

We hope these answers address your concerns. We are happy to clarify anything further.

Thank you for your additional work weakening our concerns. There are still limitations (estimating variance with 3 seeds is not really sound), but gaps appear to be large enough that this probably does not matter too much (differences are significantly larger than measured run-to-run noise). For future work, I recommend authors utilize e.g. bootstrap confidence intervals for robustness and more seeds for general statistical cleanliness. I would recommend changing the abstract though: Right now the main contribution reads as holorope with AHRA as purely incidental. Refining the abstract/title might help here.

We generally believe that this has improved the work and we would increase our rating to a 4.

AuthorsOpenReview ↗

Response to Reviewer QazS

Thank you for your follow-up comments and your positive assessment of our work. We have revised the abstract according to your suggestion to better reflect the role of AHRA. In future work, we will incorporate bootstrap confidence intervals and use more random seeds. Your constructive feedback has helped us improve the paper, and we appreciate your careful review.

Reviewer_LPo7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses positional encoding for hierarchical attention in wireless foundation models and proposes AHRA and HoloRoPE. AHRA pools keys and values in only a subset of attention heads, while the remaining heads keep the native resolution. HoloRoPE rescales the magnitudes of pooled keys with content-adaptive, frequency-dependent gains, to mitigate the limitation of single-coordinate RoPE in modeling intra-region positional information. Experiments show improvements on multiple tasks on DeepMIMO and Sionna.

Critical strengths
  1. The observation that encoding pooled keys at a single coordinate with RoPE discards intra-region positional information is a specific and valuable point.

  2. AHRA is well-motivated by wireless channel properties, aggregating within the coherence bandwidth while keeping native-resolution heads for deep fades.

  3. Experiments span different simulators and multiple model scales, with consistent improvements.

Critical weaknesses
  1. A large part of the paper analyzes the response difference between the common-coordinate and constituent-coordinate encodings, but it seems that no experiment shows that HoloRoPE actually reduces this difference. Could the authors directly compare the response error before and after HoloRoPE, or compare against a variant that uses the coordinates of each patch within the region?

  2. To separate the contribution of positional correction from that of the extra gating capacity, could the authors add a fair control with the same capacity but no positional component, e.g., applying the same gate to native-resolution keys or under NoPE?

  3. Native-resolution heads are kept to capture deep fades, but channel estimation and extrapolation report only the overall NMSE. Could the authors report the error on deep-fade regions separately? In addition, pooling relies on subcarrier correlation within the coherence bandwidth, and the data cover both urban microcell and rural macrocell scenarios, yet results are not reported per scenario. Is 1*2 pooling optimal in both?

  4. The gating network in HoloRoPE adds extra overhead. A comparison of actual end-to-end inference time would help assess its practical deployability.

  5. Key configurations needed for reproduction are missing, such as the proportion of regional heads, which layers use them, and the patch size.

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer LPo7 (1/3)

We thank the reviewer for the detailed comments and suggestions, and for the positive assessment of our motivation and experiments. In response to the reviewer's comments, we have further clarified in the main text the reasoning behind the motivation and purpose of HoloRoPE, and added a capacity-matched gating ablation together with experiments on deep-fade regions, different scenarios, and computational cost, which make our argument clearer and more rigorous. We respond to each comment below.

W1. Does HoloRoPE reduce the Bℓ,jB_{\ell,j}–Tℓ,jT_{\ell,j} encoding difference? Comparison with a constituent-coordinate variant.

HoloRoPE is not designed to reduce the difference between Bℓ,jB_{\ell,j} and Tℓ,jT_{\ell,j} (Appendices A.5–A.6). We compare Bℓ,jB_{\ell,j} and Tℓ,jT_{\ell,j} to determine which quantities the modulation should depend on (rotary frequency and local feature dispersion). We thank the reviewer for raising this question, and we have revised the paper to make this logic clearer. The revised Section 2.3 states this role when the two encodings are introduced.

Specifically, Bℓ,jB_{\ell,j} places the whole pooled key at one representative coordinate, while Tℓ,jT_{\ell,j} keeps the same pooled contributions but places each at the coordinate of its own patch (Eqs. (5)–(6), Appendix A.3). Comparing the two reveals how regional encoding depends on geometry and content. First, this depends on rotary frequency. Near zero rotary frequency, the constituent phase factors are close. For a fixed region with nonzero coordinate variance, their dispersion produces geometric attenuation. The leading geometric attenuation term is quadratic in frequency in the low-frequency expansion (Eq. (10)). Second, it depends on content. When the constituent contributions are identical, the residual vanishes and the encoding difference is determined by the geometric factor. Variations in these contributions introduce a content–position residual, whose feature-deviation component is bounded in terms of within-region feature dispersion (Appendix A.5). HoloRoPE follows these two observations: its gain uses the same quadratic rotary-frequency basis within a bounded nonlinear mapping, with its strength set by within-region feature dispersion. A region-wide scale is predicted from pooled content and dispersion (Eqs. (38)–(41), Appendix A.5). Whether a component is amplified or attenuated, and by how much, is left to the downstream task. Tℓ,jT_{\ell,j} itself is only one way of assigning positions to the same pooled contributions, so there is no reason to expect the task-optimal encoding to coincide with Tℓ,jT_{\ell,j}, any more than with Bℓ,jB_{\ell,j}.

The constituent-coordinate variant the reviewer suggested tests this directly, and we have added it (RoPE-Constituent, Table 3). It realizes Tℓ,jT_{\ell,j} exactly: each constituent contribution is rotated at its own coordinate before aggregation, while the bias contribution retains the regional coordinate (Eq. (6)), with pooling, projections, and head allocation unchanged. The table below shows two of the four tasks. Table 3 reports all four.

Table R1: Common- and constituent-coordinate encoding on DeepMIMO with real-valued models (Table 3, Section 3.2).

DeepMIMO, real-valuedCExt ULACExt UPABPred-128 ULABPred-128 UPA
RoPE (Bℓ,jB_{\ell,j})−40.12−38.3766.9060.38
RoPE-Constituent (Tℓ,jT_{\ell,j})−40.54−38.6868.7362.54
HoloRoPE−41.14−39.1970.3165.30

Realizing Tℓ,jT_{\ell,j} generally improves over Bℓ,jB_{\ell,j}, indicating that intra-region geometry carries useful information. HoloRoPE does not reproduce Tℓ,jT_{\ell,j}, yet it improves CExt by a further 0.60/0.51 dB and is the best of the three in all 24 entries of Table 3 (4 tasks × 2 arrays × real/EH/EC). Reproducing the constituent coordinates is therefore not necessary to exploit intra-region structure. In our experiments, a modulation that depends on rotary frequency and feature dispersion, as the analysis suggests, works better.

AuthorsOpenReview ↗

Response to Reviewer LPo7 (2/3)

W2. Separating position-dependent modulation from extra gating capacity.

We thank the reviewer for this suggestion and have added this experiment. The control comes in two forms: a capacity-matched gate on the regional keys with no positional component, and the same gate on native-resolution keys, as the reviewer suggested. Implementation details of these ablation variants are provided in Appendix B.2. We summarize the gate configurations below. Let DD be the model width and HRH_R the number of regional heads per layer (a quarter of the heads in the real-valued model).

  • HoloRoPE (Eqs. (38)–(39), Appendix A.5): k^ℓ,j=aℓGℓ,jBℓ,j\hat k_{\ell,j}=a_\ell G_{\ell,j} B_{\ell,j}. The regional gate is aℓ=2 sigmoid(W2 GELU(W1[zℓ∥sℓ]))∈(0,2)a_\ell=2\,\mathrm{sigmoid}(W_2\,\mathrm{GELU}(W_1[z_\ell \Vert s_\ell]))\in(0,2) with W1∈R(D/4)×2DW_1\in\mathbb{R}^{(D/4)\times 2D} and W2∈RHR×(D/4)W_2\in\mathbb{R}^{H_R\times(D/4)}, giving one scalar per region and regional head, shared by all rotary frequencies. Here, ∥\Vert denotes feature concatenation. The spectral factor is Gℓ,j=1+tanh⁡(12λνℓθj2)G_{\ell,j}=1+\tanh(\tfrac12\lambda\nu_\ell\theta_j^2) with νℓ=Softplus(Wνsℓ)\nu_\ell=\mathrm{Softplus}(W_\nu s_\ell), Wν∈RHR×DW_\nu\in\mathbb{R}^{H_R\times D}, and one learned λ\lambda per layer and regional head.
  • RoPE-ContGate (capacity-matched, no positional component): aℓ=2 sigmoid(W2 GELU(W1zℓ))a_\ell=2\,\mathrm{sigmoid}(W_2\,\mathrm{GELU}(W_1 z_\ell)) with W1∈R(D/2)×DW_1\in\mathbb{R}^{(D/2)\times D} and W2∈RHR×(D/2)W_2\in\mathbb{R}^{H_R\times(D/2)}. We double the hidden width so that W1W_1 has the same D2/2D^2/2 entries as in HoloRoPE's gate. It scales each regional key after RoPE by one scalar, takes no dispersion input, and has no θj\theta_j dependence (G≡1G\equiv1).
  • RoPE-NatContGate (the reviewer's native-key example): the same MLP as ContGate, applied to each native patch feature xmx_m and scaling the keys of the heads that would otherwise pool. These heads stay at native resolution, so no key summarizes a region.
  • RoPE-Gate (HoloRoPE's gate alone): the regional gate aℓa_\ell above with G≡1G\equiv1.
  • RoPE-Spec (positional component alone): HoloRoPE's spectral factor Gℓ,jG_{\ell,j} with aℓ≡1a_\ell\equiv1. It adds only WνW_\nu and λ\lambda.

Backbone, head allocation, and training settings are shared across variants. Regional variants use the same pooling configuration, whereas NatContGate retains native-resolution keys.

Table R2: Spectral modulation and gating controls on DeepMIMO with real-valued models (Table 2, Section 3.2, and Table 6, Appendix C.1).

DeepMIMO, real-valuedCExt ULACExt UPABPred-128 ULABPred-128 UPA
RoPE−40.12−38.3766.9060.38
RoPE-Spec−40.96−38.9970.0862.32
RoPE-NatContGate−40.19−38.3367.5361.29
RoPE-ContGate−40.37−38.5468.7762.79
RoPE-Gate−40.63−38.7968.2463.11
HoloRoPE−41.14−39.1970.3165.30

With comparable parameter counts and computational costs (1.0329× vs. 1.0331× parameters and 1.0169× vs. 1.0170× FLOPs for ULA, Table 9 in Appendix C.2), HoloRoPE improves on ContGate by 0.77/0.65 dB in CExt and 1.54/2.51 percentage points in BPred-128, and it is better in all 24 entries of Table 6 (Appendix C.1, 4 tasks × 2 arrays × real/EH/EC). Of HoloRoPE's 1.02/0.82 dB gain over RoPE, ContGate achieves 0.25/0.17 dB. Starting from HoloRoPE's own gate (RoPE-Gate), adding the spectral factor increases the parameter ratio from 1.0328× to 1.0331× under ULA and from 1.0328× to 1.0330× under UPA. It further improves CExt by 0.51/0.40 dB and BPred-128 by 2.07/2.19 percentage points, respectively (Tables 2 and 9). The converse also holds: RoPE-Spec, HoloRoPE's spectral factor alone, improves CExt by 0.84/0.62 dB with only 0.03%/0.02% additional parameters under ULA/UPA, respectively (Tables 2 and 9).

The native-key gate reaches −40.19/−38.33 dB, still 0.95/0.86 dB behind HoloRoPE. Since its gated heads no longer pool, its no-gate reference is the full-resolution RoPE in Table 1 (three-seed means of −38.94/−37.37 dB). Gating is useful in its own right, but the same gate capacity does not close the gap to HoloRoPE. Detailed parameter-count and computational-cost comparisons are reported in Table 9 of Appendix C.2. We used ContGate rather than a NoPE backbone as the main control because it retains the same rotary encoding while adding content-only gating. Under AHRA, NoPE is 22.9/11.2 dB worse than RoPE in CExt (Table 1), so a gate added there would be measured against a very different baseline.

AuthorsOpenReview ↗

Response to Reviewer LPo7 (3/3)

W3. Deep-fade regions and per-scenario pooling.

We thank the reviewer for this suggestion and have added both evaluations.

Deep fades. We define DF regions as the 10% of target channel coefficients with the lowest power in each sample and report NMSE separately on DF and the remaining coefficients (Table 11, Appendix C.3). To assess sensitivity to the DF definition, we also restrict the region to the lowest-power 5% and expand it to 20% (Tables 12 and 13, Appendix C.3). Real-valued results at 10%:

Table R3: Extrapolation in deep-fade and remaining regions at the 10% threshold with real-valued models (Table 11, Appendix C.3).

Dataset / arrayDF: RoPE → HoloRoPERemaining: RoPE → HoloRoPE
DeepMIMO / ULA−37.87 → −39.03−40.31 → −41.30
DeepMIMO / UPA−35.72 → −36.67−38.60 → −39.38
Sionna / ULA−6.98 → −8.01−15.98 → −17.02
Sionna / UPA−7.92 → −8.66−16.56 → −17.29

HoloRoPE improves DF NMSE by 0.74–1.16 dB, comparable to or larger than its gain on the remaining coefficients (0.73–1.04 dB). At both the 5% and 20% thresholds, HoloRoPE and its complex-valued variants retain lower DF NMSE than their corresponding rotary baselines across both datasets and arrays. Together with the 10% results, this gives improvements in all 36 matched DF comparisons (3 thresholds × 2 datasets × 2 arrays × real/EH/EC). These results show that the gains extend to low-power coefficients and persist across the evaluated DF definitions.

Per-scenario pooling. Figures 6 and 7 (Appendix C.1) repeat the antenna × subcarrier pooling sweep of Figure 4 (factors 1, 2, 4, 8 on each axis) separately for urban microcell and rural macrocell scenarios. In both scenarios, 1×2 pooling is the best configuration for both RoPE and HoloRoPE, and coarser subcarrier pooling degrades performance. At this configuration, HoloRoPE reaches a lower NMSE than RoPE in both scenarios.

W4. End-to-end inference time.

We thank the reviewer for this suggestion and have added the measurement (Table 9, Appendix C.2). For each rotary family and array, we report parameters, FLOPs, and end-to-end inference latency relative to the corresponding rotary baseline. Latency is reported as the mean and sample standard deviation over ten measurement rounds.

Table R4: Relative model complexity and end-to-end inference latency for real-valued HoloRoPE (Table 9, Appendix C.2).

Real-valued, relative to RoPEParamsFLOPsLatency
ULA1.0331×1.0170×1.0615× ± 0.0382
UPA1.0330×1.0170×1.0752× ± 0.0413

Across HoloRoPE, HoloRoPE++EH, and HoloRoPE++EC, the latency increase ranges from 6.15% to 7.55%. The extra cost comes from the gate and the spectral scale, which act only on the pooled keys of the regional heads. Attention itself is unchanged. In the real-valued setting, RoPE-Spec offers a cheaper point on the trade-off: it adds 0.03%/0.02% parameters, its latency ratio is 1.0361×/1.0399×, and it retains 0.84/0.62 dB of HoloRoPE's 1.02/0.82 dB CExt gain over RoPE (Tables 2 and 9).

W5. Configuration for reproduction.

We thank the reviewer for pointing this out. Appendix B.2 describes the model configurations and training settings, and Appendix B.3 describes the channel configurations. The main configurations are summarized below.

  • Regional heads. In the real-valued model, 75% of the heads in each layer retain native resolution and 25% perform regional aggregation, giving a native-to-regional head ratio of 3:1 (Appendix B.2). Figure 1 varies the share of native-resolution heads over 0/25/50/75%, and HoloRoPE improves on RoPE at every allocation.
  • Layers. AHRA is used in every encoder layer (Appendix B.2).
  • Patches and pooling. Input channels are partitioned into 2×2 patches along the antenna and subcarrier dimensions, as specified in Appendix B.2. Regional heads pool 1×2 patches (antenna × subcarrier, Figure 4), so each regional key summarizes four adjacent frequency samples spaced 360 kHz apart in the reference grid (Appendix B.3).
  • Model sizes (Table 5, head dimension 64 throughout):

Table R5: Configurations of the real-valued models (Table 5, Appendix B.2).

ModelLayersWidthHeadsFFN widthHead dim.
3M42564102464
25M85128204864
85M1276812307264
302M24102416409664
708M36128020512064

We hope these answers address your concerns. We are happy to clarify anything further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Author–Reviewer Discussion (1/2)

We thank the reviewers and the Area Chair for their time and effort in reviewing our submission. As the author–reviewer discussion comes to a close, we provide a brief summary of the reviewers' comments and our responses, which we hope will be helpful for the subsequent discussion.

The initial reviews recognized that our work addresses a meaningful problem: the paper "identifies a meaningful limitation of conventional RoPE when applied to pooled regional representations" (bjpr), "a specific and valuable point" (LPo7). They also recognized the motivation behind AHRA, which is "well-motivated by wireless channel properties" (LPo7) and "convincingly motivated with the properties of wireless channels" (QazS), and behind HoloRoPE, "a carefully designed positional encoding" that is "well motivated by a theoretical analysis" (QazS), an analysis that "provides a reasonable theoretical motivation for the proposed HoloRoPE design, moving beyond purely empirical modifications to positional encoding" (bjpr). Finally, they recognized the experiments, which "span different simulators and multiple model scales, with consistent improvements" (LPo7) and are "relatively comprehensive" (bjpr), and in which the proposed design "outperforms existing baselines" (QazS).

The reviewers also raised questions and suggestions, summarized below with our responses.

Reviewer LPo7 asked whether HoloRoPE reduces the difference between the common-coordinate and constituent-coordinate encodings. HoloRoPE is not designed to do so; we compare the two encodings to identify the key factors that shape regional key encoding, rotary frequency and within-region feature dispersion, on which HoloRoPE builds its modulation. We revised Section 2.3 to make this reasoning clearer. Following the reviewer's suggestions, we added a constituent-coordinate variant (RoPE-Constituent, Table 3), capacity-matched gating controls (Table 6), deep-fade and per-scenario evaluations (Tables 11–13, Figures 6–7), and latency measurements (Table 9), and specified the model configurations (Appendix B.2). HoloRoPE outperforms the constituent-coordinate variant and the matched gates, so its gains are not merely due to extra parameters; it also reduces the error in deep-fade regions, 1×2 pooling remains the best configuration in both scenarios, and the latency overhead is modest. In the discussion, the reviewer stated that the rebuttal and the added experiments address several of their earlier concerns, and raised a reservation about the contribution, as RoPE-Spec already captures most of the CExt gain. In our response, we explained that this result supports our analysis: RoPE-Spec is HoloRoPE's own spectral factor, which follows from the analysis in Section 2.3 and, at negligible parameter cost, outperforms gates of far greater capacity in channel extrapolation. Its other component, the regional gate, is conditioned on the regional content and feature dispersion identified by the same analysis; our experiments show that downstream tasks are sensitive to the two components to different degrees, so they are complementary. Identifying these factors through the analysis, together with the form in which they enter the model, is therefore one of our main contributions.

AuthorsOpenReview ↗

Summary of the Author–Reviewer Discussion (2/2)

Reviewer QazS asked whether the work is about a positional embedding or an architecture. We clarified that HoloRoPE is a positional encoding for attention with pooled keys, whereas AHRA is the architecture we design for wireless data, and narrowed the claim for HoloRoPE accordingly (Sections 1 and 4). Following the reviewer's suggestions, we added an image classification experiment (Appendix C.2), three-seed means and standard deviations (Table 1), backbones built on SSA and MViTv2 (Table 8), and the fixed quadratic factor motivated by Eq. (10) (Table 7). HoloRoPE improves on RoPE on images and on both backbones, its gains are almost always well above the run-to-run variation, and its adaptive spectral factor outperforms the fixed factor. In the discussion, the reviewer responded positively to our rebuttal, acknowledged that the differences are "significantly larger than measured run-to-run noise", and recommended bootstrap confidence intervals and more seeds for future work, and we will follow this recommendation. We also revised the abstract as suggested to better reflect the role of AHRA.

Reviewer bjpr asked how modulating only the amplitude addresses a mismatch in both amplitude and phase. HoloRoPE does not aim to remove these differences; we revised the corresponding description and added a paragraph to Section 2.3 analyzing phase and magnitude separately. For the geometric component, the phase difference amounts to moving the representative coordinate to the window center, which changes CExt only marginally (RoPE-Center). We therefore keep the common-coordinate phases and modulate the magnitudes; HoloRoPE also outperforms RoPE-Constituent, which retains the individual constituent phases. As further requested, we added comparisons with simpler gates (Table 2), specified the system parameters (Appendix B.3), defined structured extrapolation to contiguous blocks at the end of either or both axes (Appendix B.1, Table 10), and evaluated measured CSI (DICHASUS), cross-domain transfer, different channel configurations, delay spreads, subcarrier correlations, and noise levels (Tables 14–19, Figures 9–10). HoloRoPE outperforms the simpler gates, as does its spectral factor alone in CExt at negligible parameter cost, and improves on its rotary baseline in all of these evaluations, including under weak subcarrier correlation. In the discussion, the reviewer responded positively to our rebuttal, noting that the additional results addressed some of their main concerns.

We are grateful for the reviewers' constructive comments, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.