acceptodds
Under review as a conference paper at ICLR 2027

Dense Local Operators on the Sphere

Abstract

Spherical representations are ideal for wide-field-of-view (FoV) perception because they describe local visual geometry independently of the viewing direction. A spherical local operator should respect this geometric consistency by producing consistent responses to the same upright local pattern wherever it is relocated on the sphere—a property we call *translation consistency*. Achieving this property on a 2D grid is challenging, because no global 2D parameterization can preserve the spherical metric everywhere. On an equirectangular projection (ERP) grid, a common representation for spherical data, the same physical neighborhood has a latitude-dependent grid footprint. Prior spherical local operators recover this neighborhood through location-dependent sampling, but the resulting sparse memory access patterns limit GPU utilization. We propose **D**ense **L**ocal Operator on the **S**phere (DLS), a spherical local attention mechanism realizing *fully dense*, *approximately translation-consistent* operation on ERP grids. DLS combines distance-corrected longitude positional encoding (PE) with linear attention using separate row and column key–value (KV) aggregation. This separation cancels the cross-term involving global longitude, allowing the global PE to encode local spherical geometry therefore enabling KV-first aggregation without explicitly constructing local KV fields through sparse operations. The formulation yields approximately consistent responses to the same upright spherical patch at different locations, using only dense operations. On the panoramic segmentation task, the DLS-based network achieves a 4.9× speedup while improving mIoU by 2.9 points over the prior sparse-sampling-based network.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
spherical attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_M3b1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Proposes DLS, a dense local attention operator for spherical images. Combines geometry-aware positional encoding with row-column KV aggregation to avoid irregular sparse sampling. Achieves approximately translation-consistent features with improved accuracy and inference speed across multiple spherical perception tasks.

Critical strengths
  1. Novel and well-motivated formulation Geometry-aware dense attention avoids irregular sparse sampling while maintaining approximate translation consistency

  2. Strong accuracy and efficiency. Improves segmentation mIoU by 2.9 points with 4.9× speedup. Consistent improvements across multiple tasks (Tables 1–2).

Critical weaknesses
  • DLS requires substantially more GPU memory despite lower latency (Table 2). May limit practical deployment at high resolutions.

  • Translation consistency relies on a first-order approximation, with increasing errors near the poles and for larger receptive fields (Sec. 3.3). Robustness under these conditions needs further validation.

Questions for authors
  1. Can the memory overhead be reduced without sacrificing inference speed?
  2. How does performance change near the poles or with larger receptive fields?
  3. How much of the speedup comes from the dense formulation versus the optimized Triton implementation?
1 comment

Thank you for the practical questions about memory, polar behavior, and implementation efficiency. We agree that practical efficiency should account for peak memory as well as latency, and your suggestions helped us improve the implementation and clarify its behavior.

Reducing memory overhead

Our original inference implementation prioritized latency. We now provide two optimized variants: latency-optimized (L) and latency- and peak-memory-optimized (LM). LM shares identical read-only buffers, reschedules independent operations, and processes attention heads in groups to reduce peak memory usage.

For the trained 256×512256\times512 segmentation model, buffer sharing and rescheduling alone reduce peak allocated memory to approximately 0.93 GB with latency essentially unchanged. Grouping attention heads further reduces memory to 0.472 GB, about 55% below the latency-optimized variant's 1.049 GB. Latency changes from 13.11 to 13.20 ms, an increase of only 0.09 ms (0.7%). The L and LM variants produce bit-identical outputs on all 40 validation panoramas for this trained model, preserving the evaluated accuracy.

Across the four tested resolutions, the final LM variant reduces peak memory by approximately 41-56%, with measured latency changes between -1.0% and +3.3%. Tables 2 and F now report both variants, and Appendices C and E describe the implementation and measurements. These results directly address reducing memory overhead with little latency cost and without changing the model.

Performance at high latitudes

Stanford2D3D has limited valid coverage near the poles, so we additionally evaluated DLS-SegFormer, S2\mathcal{S}^2-SegFormer, and SphereUFormer on the full-latitude Structured3D dataset. Figure C in Appendix D reports segmentation accuracy in 15∘15^\circ latitude bands over 1,687 test images. Section 4.4 now summarizes these findings.

DLS exceeds S2\mathcal{S}^2-SegFormer in every plotted band, while SphereUFormer performs better in several high-latitude bands. The overall mIoU values are 53.7, 53.2, and 51.7 for DLS, S2\mathcal{S}^2, and SphereUFormer, respectively. The continuous S2\mathcal{S}^2 attention formulation respects spherical geometry, but this property alone does not guarantee higher segmentation accuracy at high latitudes. We hypothesize that latitude-dependent responses can provide useful cues to latitude-specific scene statistics. Their usefulness may depend on the dataset and task, and the present comparison does not isolate this mechanism.

Larger angular supports

The default angular-support schedules closely match those of the baseline, as reported in Section 4.1. In preliminary segmentation experiments at 256×512256\times512, doubling the angular supports of all layers reduced mIoU by 0.89 percentage points for DLS and 0.84 percentage points for S2\mathcal{S}^2. We therefore retain the original, closely matched angular-support schedules for the main experiments. We report this sensitivity check and its numerical results in Section 4.4 and Appendix D.

The similar decreases indicate comparable sensitivity to this support enlargement. They do not isolate curvature error from dataset- and task-dependent effects of enlarging the receptive field.

Contribution of implementation optimization to the speedup

Appendix E reports both PyTorch-reference and optimized timings to examine the computational advantage under both implementation settings. Table F gives the following default-model SegFormer latencies at 256×512256\times512:

ImplementationS2\mathcal{S}^2-SegFormerDLS-SegFormerLatency ratio (S2\mathcal{S}^2/DLS)
PyTorch reference2,030 ms189.5 ms10.7×10.7\times
Optimized CUDA / Triton (L / LM)64.3 ms13.11 / 13.20 ms4.9×4.9\times / 4.9×4.9\times

Relative to their respective reference implementations, optimization accelerates S2\mathcal{S}^2 by approximately 31.6×31.6\times and DLS by 14.5×14.5\times (L) or 14.4×14.4\times (LM). Both algorithms benefit substantially from the implementation improvements, although the relative speedup is larger for S2\mathcal{S}^2.

For the spherical Transformer, we additionally applied pointwise/GELU and resampling fusion using Triton to the already optimized S2\mathcal{S}^2 implementation. At 256×512256\times512, latency decreases from 11.2 ms to 10.9 ms (Table G). This modest additional gain shows that the original CUDA implementation already captures most of the measured speedup over its PyTorch reference.

Reviewer_aiHyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DLS, a dense local attention operator for spherical signals on ERP grids. It combines distance-corrected positional encoding, row–column factorization, and KV-first aggregation to achieve approximate translation consistency while avoiding irregular sparse operations.

Critical strengths

The paper addresses an important problem: balancing geometric consistency and computational efficiency in spherical local operators. The motivation is clear and practically relevant.

The proposed formulation is computationally efficient, achieving a 4.9× speedup and a 2.9-point mIoU improvement over the spherical attention baseline at 256×512 resolution.

The paper provides theoretical analysis of approximation errors and evaluates the method across multiple tasks, supporting its practical effectiveness.

Critical weaknesses

The method primarily combines established techniques, including axial attention, RoPE, and KV-first linear attention. The main novelty lies in their geometry-aware integration, but its distinction from prior spherical linear attention methods, particularly SoLA, could be better established.

Equation (4) demonstrates an undesirable absolute-longitude dependence of a particular positional encoding. However, this does not establish that row–column factorization is necessary. Alternative two-dimensional spherical positional encodings could avoid this issue without restricting interactions to individual rows or columns.

Section 3.3 establishes translation consistency only under a first-order approximation in the continuous domain. The O(α2)O(\alpha^2) error bound relies on regularity assumptions, and its practical tightness across different latitudes and support sizes remains unclear.

A single DLS layer aggregates features along a cross-shaped support rather than a full two-dimensional geodesic neighborhood. Although stacking layers expands the receptive field, the implications for spatial expressiveness are not sufficiently investigated.

The cumulative ablations do not fully isolate the contributions of the proposed components. Comparisons with stronger geometry-aware axial or linear-attention baselines would better establish whether the gains arise from the geometric formulation or the computational architecture.

Questions for authors

Please refer to the Weaknesses section.

1 comment

Thank you for the thoughtful assessment. We agree that the contribution should be distinguished from established techniques, and that the scope of both the factorization argument and the approximation needs to be carefully stated.

Technical contribution and relation to prior spherical attention

DLS couples distance-corrected positional encoding, latitude-adaptive support, and separate row and column KV aggregation to obtain a fully dense, approximately translation-consistent local operator on ERP grids. We do not claim that rotary positional encoding, axial attention, or KV-first attention is individually new. Our contribution is this coupled construction for approximately translation-consistent local processing using dense operations, together with its first-order consistency analysis. We have clarified this distinction in Sections 1-3.

Eq. (4) shows that the particular corrected-longitude phase considered here introduces an absolute-longitude term when queries and keys lie on different rows. Our factorization is a sufficient means of removing that term while retaining independent query/key modulation and dense KV aggregation. Within our construction, distance correction aligns rotary phases with physical offsets, adaptive support preserves physical neighborhood spans, and row-column separation removes the cross-row absolute-longitude term. The revised discussion in Section 3.2 explicitly states this scope.

SoLA and DLS target different symmetries. SoLA seeks attention equivariant to arbitrary SO(3)\mathrm{SO}(3) rotations. DLS assumes a known vertical axis and seeks consistent responses when upright local patterns are relocated on the sphere, under a first-order approximation. Section 2 now makes this difference in goals explicit.

Computational efficiency is a second motivation: we seek compact query/key features and fully dense local aggregation on ERP grids. SoLA's KV-first formulation lifts DD-dimensional queries and keys to 4D4D through modulation by bias and direction, whereas DLS's rotary modulation retains DD dimensions per branch. SoLA's experiments use icosphere neighborhoods, with direct ERP processing identified as future work. On ERP grids, spherical caps have latitude-dependent footprints; directly aggregating over their indexed neighborhoods requires location-dependent sparse operations, as in DISCO. DLS instead uses a local row-column construction to enable approximately translation-consistent local ERP computation using solely dense operations, without feature lifting. We have made these computational goals explicit in Sections 2 and 3.1.

Additional component ablations

We agree that a single cumulative ablation sequence does not isolate the individual components. The expanded Table 3 includes one-at-a-time removals of adaptive support, distance correction, and row-column separation, as well as additional combinations. We have updated Section 4.7 and the protocol in Appendix F to explicitly describe the configurations.

For the single-removal comparisons at 256×512256\times512, segmentation mIoU decreases from 73.5 for complete DLS to 70.7 without adaptive support, 72.7 without distance correction, and 66.8 without row-column separation. These results support the contribution of each component within the tested construction.

Approximation error and the tested second-order correction

The Oη(α2)O_\eta(\alpha^2) result in Section 3.3 is a local bound under boundedness and Lipschitz assumptions on a latitude band away from the poles. Its constants depend on the band and the layer's feature and positional mappings. We do not interpret it as a uniform bound at the poles, for arbitrarily large support, or for an entire trained network. The empirical consistency experiments complement the analysis, but do not establish tightness of the bound in all these regimes.

We also evaluated the second-order correction derived in Appendix A; Table C reports its task-level results, and Section 4.4 now summarizes the trade-off. At 256×512256\times512, segmentation mIoU increases from 73.5 to 74.1. Depth results are mixed. The correction increases latency by 10.9-11.3% and peak allocated GPU memory by 76-84%.

Given the mixed task-level changes and increased resource costs, we retain the first-order formulation.

Cross-shaped support and spatial expressiveness

DLS adopts a cross-shaped support to ensure translation consistency. The one-at-a-time removal of row-column separation in Table 3 retains adaptive support and distance correction but uses two-dimensional aggregation. This restores off-axis interactions while reintroducing the absolute-longitude dependence of the corrected-longitude encoding. Its mIoU is 66.8, compared with 73.5 for DLS. This suggests that the benefit of removing absolute-longitude dependence can outweigh the restriction to axial interactions in this configuration. The ablation evaluates this combined trade-off, since both the interaction support and positional dependence change.

Reviewer_6yjiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DLS, which combines distance-corrected positional encoding, row–column decomposition, and latitude-adaptive KV aggregation to balance local response consistency and efficiency on ERP grids. It provides a single-layer analysis under a continuous approximation and compares against an optimized official spherical baseline.

Critical strengths
  1. By avoiding explicit construction of irregular spherical neighborhoods, DLS offers a promising way to balance spherical geometry and efficiency.

  2. Using row–column decomposition to remove the absolute-longitude cross-term is an interesting idea. The appendix also provides a consistency proof and curvature error analysis.

Critical weaknesses
  1. The source of the performance gains remains unclear. DLS-SegFormer doubles the parameter count from 3.2 M to 6.4 M while also changing the number of attention heads and several spatial modules. It is therefore hard to separate the benefits of the geometric design from those of increased capacity and architectural changes. The cumulative ablation in Table 3 also mixes several changes. Parameter-matched comparisons and independent component ablations are needed.

  2. The single-layer theory does not fully explain the behavior of the full network. The authors should report each layer’s angular support and test full-network consistency across latitudes, support sizes, and resolutions. The limits of the polar boundary handling should also be clarified.

  3. Depth evaluation should include valid-pixel-only results. Invalid pixels are currently counted as perfect predictions and kept in the denominator, lowering errors and raising threshold accuracy. The authors should add valid-pixel-only results, report the valid-pixel ratios, and use the same evaluation protocol across methods.

Questions for authors

Seen above.

1 comment

Thank you for the constructive feedback. We agree that capacity-controlled comparisons, network-level consistency measurements, and transparent depth evaluation are important for assessing DLS.

Parameter-matched comparisons

The default DLS-SegFormer retains the baseline's stage embedding dimensions, hierarchy, and task heads. To realize a fully dense end-to-end network, we replace the DISCO-based spatial mixers with DLS modules. This replacement accounts for most of the parameter increase from 3.2M for S2\mathcal{S}^2-SegFormer to 6.4M for DLS-SegFormer.

We examine capacity in both directions: widening S2\mathcal{S}^2 and reducing DLS through mixer replacement.

  • Wide Q/K projections (6.3M parameters): the query and key projection dimensions are expanded to 6C6C, while the value projection dimension is unchanged.
  • Wide embedding (6.4M parameters): the stage widths increase from [32,64,128,256][32,64,128,256] to [46,92,184,368][46,92,184,368], approximately a 1.4-fold increase.
  • DLS-SegFormer with DISCO mixers (3.4M parameters): DLS attention is retained, while the spatial mixers are replaced with those of the baseline. This approximately matches the original S2\mathcal{S}^2 model's 3.2M parameters.

Section 4.4 summarizes these controls, with full results in Table D. At 256×512256\times512, default DLS achieves 73.5 mIoU, compared with 72.5 for wide Q/K and 71.8 for wide embedding. Depth performance at native resolutions is broadly comparable, with mixed rankings across metrics.

The fully dense DLS model also offers a favorable accuracy-latency trade-off. At 256×512256\times512, its latency is 13.1 ms versus 64.3 ms for the original optimized S2\mathcal{S}^2 baseline, a 4.9×4.9\times speedup. Our preliminary timings show that wide Q/K and wide embedding increase latency by approximately 10% and 15%, respectively, relative to the original baseline. Higher segmentation accuracy, broadly comparable depth performance, and substantially lower latency therefore support a favorable accuracy-latency trade-off at comparable parameter counts.

The converse DISCO-mixer control achieves 70.9 mIoU versus 70.6 for the original S2\mathcal{S}^2 baseline, with similar depth performance. Its peak memory and latency are also close to the baseline's: 0.40 versus 0.39 GB and 65.1 versus 64.3 ms, respectively (Table F). Restoring DISCO mixers thus reduces parameter count and peak memory at the cost of the fully dense DLS configuration's speed and accuracy. This motivates our default end-to-end design.

Feature consistency in the full network

We have added full-network measurements across latitudes at encoder stages 1, 2, 3, and 4 (Figure B, summarized in Section 4.2). Using complete spherical images, we compare intermediate features with their equatorial references in both randomly initialized and trained networks. DLS generally maintains higher cosine similarity than joint 2D RoPE, providing evidence of improved feature consistency across the four encoder stages. Appendix D ("Feature consistency in the full network") details the protocol and discusses the results.

Angular support and the scope of the approximation

The stagewise angular-support schedule of DLS-SegFormer reported in Section 4.1 is [5.6,11.1,22.3,44.5]∘[5.6,11.1,22.3,44.5]^\circ, closely matching the [5.7,11.2,22.6,45.9]∘[5.7,11.2,22.6,45.9]^\circ schedule of the S2\mathcal{S}^2-SegFormer baseline.

We agree that the small-support example in Section 3.3 does not characterize every network layer. The curvature mismatch increases with support size and latitude. The local analysis assumes that the relevant supports remain away from the poles; it does not provide a uniform guarantee for the full network. We have made these limits explicit in the revised Section 3.3.

Appendix C describes the boundary conventions: longitude wraps periodically with the appropriate unwrapped rotary phases, whereas out-of-range latitude indices are clamped to the nearest available row. Latitude clamping can introduce additional inconsistency near the poles.

Depth evaluation using only valid pixels

Table 1 follows SphereUFormer's released depth evaluation convention for direct comparison with published scores. Under this convention, predictions and targets at invalid locations are both set to zero before clipping, so those locations count as perfect predictions and remain in the denominator. This lowers errors and raises threshold accuracy relative to a valid-pixel-only reduction.

Table D additionally reports valid-pixel-only depth results, excluding invalid pixels from the numerator and denominator for every method. The reported spherical-area-weighted valid-depth coverage is approximately 89.00%; the table also provides coverage statistics by latitude band. Section 4.4 and Appendix D now describe these evaluation conventions explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.