acceptodds
Under review as a conference paper at ICLR 2027

Structural Radial Aggregation for Cross-View Yaw under Position Uncertainty

Abstract

Ground-to-satellite localization estimates a street-level camera's position and yaw (heading) from an aerial image. Yaw is the harder part, because a small angle error moves distant scene content far. Position uncertainty turns yaw estimation into aggregating complementary evidence over all candidate poses. Prior work matches each ground-view column to aerial pixels. All positions on the ray through a matched pixel imply the same yaw, so each match votes for a yaw without knowing its distance. Each vote is precise but checks one viewing direction, which leaves many yaws nearly as plausible as the true one at each position. The scene layout is coarser but covers the whole view. Structural Radial Aggregation for Yaw (S-RAY) combines both. A structural score, learned from pose labels only, rates each yaw at each position by how well the directions where the ground view shows structure line up with the aerial structure axes, leaving a few promising yaws per position. A Fourier score of continuous match votes decides among them. Hard-bin voting, which bins every position–match pair separately, is costly and makes the score piecewise constant. Instead, each match adds a learned bump at its implied yaw. Since the implied yaw is a difference of angles, a bump of a few cosines turns scoring all positions and yaws into a few FFT correlations that form no position–match pairs; this scorer is 23× faster than prior hard-bin voting. S-RAY raises yaw recall within 1° over the prior method from 51.07% to 87.65% on KITTI Same Area, from 48.44% to 72.91% on KITTI Cross Area, and from 34.81% to 55.22% on MGL, and has the lowest mean yaw error on both VIGOR splits. As a yaw initialization, it also improves Loc² localization on KITTI and MGL.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
cross-view geo-localization, ground-to-satellite localization, 3-DoF pose estimation, yaw estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7djGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Structural Radial Aggregation for Yaw (S-RAY), a framework for cross-view yaw estimation under camera position uncertainty. Building on the radial-invariant column-matching and yaw-voting framework of LAYS, S-RAY introduces two complementary components. First, a structural score aligns ground-view structural direction distributions with aerial structure-axis representations to identify promising yaw candidates at each possible camera position. Second, Fourier-based continuous voting replaces discrete hard-bin accumulation with a learned band-limited angular kernel. By exploiting the angular structure of the voting formulation, the method computes scores efficiently through FFT-based correlations without explicitly constructing position–match pairs. Experiments on KITTI, MGL, and VIGOR report substantial improvements in fine-grained yaw estimation. The paper also presents ablation studies, computational comparisons, and downstream localization results using the estimated yaw for initialization.

Critical strengths
  1. An elegant and computationally meaningful formulation of continuous yaw voting.

The Fourier-based voting formulation is a notable technical contribution. By representing the angular voting kernel using a finite cosine series and exploiting the angle-difference structure of the induced yaw, the method transforms expensive position–match voting into a small number of FFT-based correlations.

For the selected band-limited kernel, this factorization provides an exact computation of the corresponding continuous voting score rather than an additional numerical approximation of the direct sum. The resulting formulation is mathematically well motivated and offers a practical way to scale yaw scoring across candidate positions and orientations.

The integration of structural orientation information with radial matching also provides a reasonable approach to resolving ambiguous yaw hypotheses that may remain difficult to distinguish using individual correspondence votes alone.

  1. Substantial empirical improvements supported by useful ablations and downstream evaluation.

The reported results demonstrate substantial improvements in fine-grained yaw estimation across multiple benchmarks. In particular, R@1° increases from 51.07% to 87.65% on KITTI Same Area, from 48.44% to 72.91% on KITTI Cross Area, and from 34.81% to 55.22% on MGL relative to the reported LAYS baseline.

The component ablations provide useful evidence that structural guidance and continuous voting contribute to the overall performance. The paper also reports computational advantages for the voting module and demonstrates that the estimated yaw can improve downstream Loc² localization when used for initialization.

The inclusion of failure-case analysis, particularly for large angular ambiguities, further helps characterize the remaining limitations of the approach.

Critical weaknesses
  1. The contributions of structural alignment, continuous voting, and training differences are not sufficiently isolated.

Although the overall performance improvements are substantial, the method builds directly on the column matching, radial geometry, and voting framework established by LAYS. Therefore, carefully isolating the contributions of the newly introduced components is important for assessing the methodological novelty.

The ablation results in Table 4 raise a specific attribution concern. The hard-bin variant, which retains structural scoring while replacing continuous voting with hard-bin accumulation, already achieves 78.69% R@1° on KITTI Same Area, compared with 51.07% for the reported LAYS baseline. The full method further increases this value to 87.65%.

Thus, a substantial portion of the overall improvement is already present before replacing hard-bin voting with the proposed continuous formulation. However, this difference cannot be attributed exclusively to structural alignment because other training or implementation changes may also contribute.

In particular, the structural score incorporates an existing aerial-only pose plausibility component, while the new alignment term and the additional training objective have not been sufficiently separated in the reported comparisons.

A matched evaluation of Fourier voting with the original LAYS plausibility score, the complete structural score, and the original LAYS model under equivalent training settings would help determine which design elements are responsible for the gains. A separate analysis of the additional training loss would further strengthen the attribution.

This issue is consequential because the individual contributions of structural alignment and Fourier voting constitute the main methodological claims.

  1. The comparison protocol and efficiency claims require more precise qualification.

Some compared methods appear to operate under different yaw assumptions or prediction conventions. In particular, methods that estimate orientation modulo 180° are not directly comparable to methods evaluated over the complete ±180° range without an appropriate treatment of the angular ambiguity.

The paper should more clearly distinguish methods designed for full-range yaw estimation from those relying on narrower orientation assumptions. Furthermore, while S-RAY demonstrates strong improvements in fine-grained recall, its reported mean yaw error on KITTI Cross Area is not uniformly better than that of the strongest competing method. Presenting these trade-offs more explicitly would provide a more balanced interpretation of the results.

The computational comparison also requires clarification. The reported 23× acceleration corresponds to the finest-level voting module, where runtime decreases from approximately 19.16 ms to 0.83 ms. However, the complete inference time decreases from approximately 103 ms to 84 ms, corresponding to an end-to-end speedup of about 1.23×.

The module-level improvement remains valuable, but emphasizing the 23× figure without consistently distinguishing it from full-system latency may overstate the practical efficiency advantage.

Finally, some implementation details of the structural score, including the orientation representation, gating mechanism, and auxiliary supervision, are deferred to the appendix. A clearer main-text explanation would improve reproducibility and make the structural component easier to assess.

Questions for authors

Q1. Attribution of performance improvements: Can the authors clarify the results of using Fourier voting with the original LAYS pose plausibility score but without the proposed structural alignment term? Are matched results available for LAYS under the same backbone and training configuration, including the additional loss described in Section 3.5? These comparisons would help establish the independent contributions of structural alignment and continuous voting.

Q2. Structural alignment mechanism: Could the authors clarify how the structural alignment term learns correspondence between ground-view structural directions and aerial orientation axes? In particular, how should the statement that training learns the relationship between angular axes be interpreted when Eq. (4) already specifies a fixed angular correspondence? A concise explanation of the learned quantities and the role of the structural gate would help clarify this mechanism.

Q3. Angular resolution and Fourier representation: Is the final yaw estimate restricted to a 1° angular grid, or is sub-bin refinement performed? How sensitive are R@1° and the reported sub-degree median errors to the output grid resolution? Additionally, can the authors clarify the learned angular kernel's shape and whether the selected Fourier order introduces side lobes or other effects that influence the voting distribution?

Q4. Baseline comparability and efficiency: How are baselines with different yaw parameterizations or orientation priors handled in the evaluation? Can the authors clarify the difference between module-level and end-to-end acceleration and describe the implementation and hardware configuration used for hard-bin voting? These details would help assess the fairness and practical significance of the reported improvements.

4 comments
AuthorsOpenReview ↗

Response 1/3

We thank you for the thorough review and for recognizing the exactness and efficiency of our continuous Fourier voting formulation. Paper changes are marked in red.

Q1 / W1, attribution of the gains

Fourier voting with the LAYS plausibility score vs. our structural score SS. "F only" in Table 4 drops the second scoring term altogether and does not compare SS (Eq. 4) directly against LAYS's aerial-only pose plausibility score. Retraining with Fourier voting and LAYS's plausibility score reaches 25.04 yaw recall within 1° (R@1°, %; median 2.21°) on VIGOR Same Area, against 55.01 (0.88°) with SS. The full 2×2 with hard bins, at fixed backbone, objective and support (R@1° / median, Same Area):

Voting ruleVIGOR, Structural score SSVIGOR, Plausibility score (LAYS)KITTI, Structural score SSKITTI, Plausibility score (LAYS)
Fourier voting55.01 / 0.8825.04 / 2.2187.65 / 0.4985.32 / 0.51
Hard-bin voting41.90 / 1.2622.21 / 2.6478.69 / 0.5679.46 / 0.55
  • VIGOR: the ground roll needs SS. Each ground panorama is circularly shifted (rolled) by a random yaw in ±180°, which randomizes the ground view's "forward" direction. LAYS's plausibility score uses only the aerial view and cannot infer this orientation. Aligning the ground view's direction distribution with the aerial axes in SS adds 29.97 points with Fourier voting and 19.69 with hard bins, while Fourier voting adds 13.11 points with SS vs. 2.83 with the plausibility score.
  • KITTI: the gain comes from Fourier voting. Fourier voting adds 8.96 points over hard bins with SS (87.65 vs. 78.69) and 5.86 with the plausibility score (85.32 vs. 79.46). The car camera faces along the road, so the top-down view alone already narrows the yaw to the road axis in most cases, and SS and the plausibility score perform similarly under hard bins (78.69 vs. 79.46).

Separating the method from the backbone. We retrained S-RAY with LAYS's trained VGG16 U-Net in place of frozen DINOv2-B (±180° prior, R@1° % / median yaw error °):

SplitS-RAY (DINOv2-B)S-RAY (VGG16 U-Net)LAYS (VGG16 U-Net)
KITTI Same Area87.65 / 0.4980.39 / 0.5651.07 / 0.96
KITTI Cross Area72.91 / 0.5962.52 / 0.7448.44 / 1.07
MGL55.22 / 0.8746.57 / 1.1034.81 / 1.79
VIGOR Same Area55.01 / 0.8846.55 / 1.1034.16 / 1.94

With the same VGG16 U-Net backbone, S-RAY is ahead of LAYS on KITTI and MGL and of VGG hard-bin voting on VIGOR, by 29.32, 14.08, 11.76, and 9.60 points in the four rows. On KITTI Same Area, the step from published LAYS (51.07) to S-RAY (87.65) reflects both the method and the backbone: in our setup, DINOv2-B hard bins reach 78.69 with SS and 79.46 with the plausibility score (LAYS: 51.07), and DINOv2-B adds 7.26 points to S-RAY (80.39 to 87.65), which leaves hard-bin voting nearly 9 points behind.

The additional loss of Sec. 3.5 (retrained, VIGOR Same Area).

  • Without the position-marginalized term, R@1° is higher (58.40 vs. 55.01), but the mean yaw error is 13.71° vs. 8.74° and yaw<5° is 85.68 vs. 88.92. The term trades fine recall for fewer large errors.
  • Without the structural alignment loss Lstructureℓ\mathcal{L}^\ell_{\text{structure}} (defined under Q2 below), SS still improves recall by 24.7 points over the plausibility score (25.04 to 49.77), and the loss adds a further 5.2 points (55.01). So most of the gain (24.7 of 29.97 points) comes from SS itself.

The revised paper reports the 2×2, the backbone results and both loss ablations in Sec. 4.3, with full tables in Appendix E.

AuthorsOpenReview ↗

Response 2/3

Q2. Our wording in Sec. 3.3 ("training learns how their angle axes correspond") was inaccurate: the correspondence is fixed, and what is learned are the two quantities it compares.

  • What Eq. (4) fixes. If the camera has yaw θ\theta, a structure that the ground image shows at direction ψ\psi relative to the camera runs along the absolute direction θ+ψ\theta+\psi in the aerial image. Eq. (4), C(x,y,θ)=∑ψπ(ψ) A(x,y,θ+ψ)C(x,y,\theta)=\sum_\psi \pi(\psi)\,A(x,y,\theta+\psi), applies this geometric rule; nothing about it is learned.
  • What training learns. The two inputs of the rule: the ground direction distribution π\pi, i.e. in which directions the ground image shows structure (a 1D CNN over ground columns, softmax over direction bins), and the aerial axis scores AA, i.e. along which absolute directions the aerial image has structure at each position (a small CNN on aerial features). CC is high at yaws where the two line up.
  • Role of the gate. λ\lambda is one learned scalar per pyramid level that weights CC within SS. It starts at zero, so CC enters the score only as far as it lowers the training loss.
  • How it is trained. Two losses reach CC, and both use only the ground-truth pose: (i) the pose losses on the total score, like every other term; (ii) the structural alignment loss on CC alone, a circular cross-entropy over yaw between CC at the ground-truth position and a von Mises target (κ=8\kappa=8) at the ground-truth yaw, weight 0.5. CC cannot separate θ\theta from θ+180°\theta+180°, so (ii) concentrates it on that pair, and FF decides the direction.

The revised Sec. 3.3 states the fixed correspondence and the learned π\pi and AA and defines the structural alignment loss; Sec. 3.5 gives the full objective, and Appendix B.3 describes the gate (this also addresses the last point of W2).

Q3, output grid. S-RAY evaluates scores on a coarse-to-fine grid of 4° (90 bins), 2° (180 bins), and 1° (360 bins) across the three pyramid levels, and outputs the argmax bin center at the finest 1° level directly, without post-hoc interpolation (Appendix C). Sub-degree median errors (e.g., 0.49° on KITTI) occur naturally because the ground-truth yaw is continuous: when a prediction selects the nearest 1° bin center, its error against the continuous ground truth lies in [0∘,0.5∘][0^\circ, 0.5^\circ] with an expected median near 0.25°. To test sensitivity to the readout grid, we re-decoded the final score of trained models, without retraining, on coarser grids (4° and 2°), and with a 0.5° sub-bin readout that linearly interpolates the 1° score, which places the estimate between bin centers toward the stronger neighbouring bin. R@1° in % (median yaw error):

Readout4°2°1° (paper)0.5° sub-bin
VIGOR Same35.3 (1.41°)50.1 (1.00°)55.1 (0.88°)55.6 (0.87°)
VIGOR Cross29.7 (1.67°)39.0 (1.38°)40.3 (1.31°)42.0 (1.26°)
KITTI Same47.3 (1.06°)72.8 (0.59°)87.8 (0.48°)90.9 (0.47°)

Grids coarser than 1° cost many points, while the sub-bin readout adds 0.5 points on VIGOR Same Area, 1.7 on Cross Area and 3.1 on KITTI, so the 1° grid limits the reported recall only slightly. The paper's numbers keep the 1° readout; this table is in Appendix E.

Q3, kernel shape and side lobes. The supplementary material of the submission already described the kernel shape; it is now Appendix B.1 of the revised PDF, which includes the appendix. With N=8N=8 the initial kernel falls to 43% of its peak at ±15°, crosses zero near ±24°, and its ripples beyond stay within 16% of the peak. The learned kernels keep this width, 26°, 27° and 28° at half maximum on KITTI, MGL and VIGOR (figure in Appendix B.1), against 120° for N=1N=1. We now also report their side lobes, computed from the learned coefficients: the first zero crossing is at 21°, 22° and 23°, the largest positive side lobe is 14%, 11% and 0.2% of the peak, and the deepest negative lobe is −60%, −39% and −25%, located at 180° on KITTI and MGL and at 35° on VIGOR. A negative lobe is a penalty, so on KITTI and MGL the learned kernel penalizes the opposite heading, most strongly on KITTI.

AuthorsOpenReview ↗

Response 3/3

Q4 / W2, yaw conventions of the baselines; mean error; scorer vs. end-to-end speed

Yaw conventions. CCVPE, SliceMatch, Loc², VIRD, OGH-Net and LAYS report full-circle (±180°) yaw. ARC-Loc is the exception. Its line solver defines yaw modulo 180° and outputs a value in [−90°, 90°). Our full-circle metrics evaluate that output, so we agree its yaw columns are not directly comparable with the other rows.

Scored modulo 180° with ARC-Loc's own evaluator, ARC-Loc reaches 22.61% within 1° and 66.50% within 5° on KITTI Same Area (median 2.83°), and 11.59% and 41.29% on Cross Area (median 7.97°). Under the same axial metric, S-RAY reaches 79.42% and 89.92% on Cross Area (median 0.53°), against 72.91% and 82.37% full-circle. The ranking therefore does not depend on the yaw convention.

The revised Appendix D.1 states the yaw convention of every baseline and gives these modulo-180° results.

Mean error. On KITTI Cross Area, S-RAY's mean yaw error (23.00°) is slightly higher than VIRD's (22.03°), while its median is lower (0.59° vs. 1.87°) and its R@1° higher (72.91 vs. 30.39). The mean is dominated by two groups of large errors: 9.92% of queries near 90° and 7.56% near 180°. Sec. 4.2 states this comparison and Appendix A.1 analyzes the two groups.

Scorer vs. end-to-end speed. We agree that the 23× figure covers only the finest-level scorer. We measured both methods end to end on an NVIDIA RTX 6000 Ada: S-RAY is 2.50× faster with each method's own backbone and 4.23× faster with the matched VGG16 U-Net backbone. The submission's end-to-end times (about 103 vs. 84 ms) included implementation overhead outside the scorer, in inference code that both methods share. We removed it for both methods and re-measured them on one GPU (RTX 6000 Ada): S-RAY now takes 40.3 ms and LAYS 100.8 ms (2.50×). LAYS changes little because its time is dominated by hard-bin scoring (81.8 ms after feature extraction, against 6.0 ms for S-RAY).

ConfigurationS-RAYLAYSSpeedup
Finest-level scorer (Table 5)0.834 ms19.162 ms22.97×
End to end (default backbones)40.3 ms100.8 ms2.50×
Post-backbone scoring6.0 ms81.8 ms13.6×
End to end (matched VGG16 U-Net)23.7 ms100.3 ms4.23×

Protocol and hardware. All comparisons run with batch size 1, gradients disabled, and inputs resident in GPU memory. The finest-level scorer row reproduces Table 5 on an NVIDIA RTX PRO 6000 Blackwell over 102×102 candidate positions and 360 yaw bins, reporting the median of 5 rounds of 21 timed runs after 5 warm-up runs. The three end-to-end rows time one MGL query on an otherwise idle NVIDIA RTX 6000 Ada.

Hard-bin implementation. The hard-bin scorer is the official LAYS implementation, which we obtained from its authors. It computes a bin index for every position–match pair and scatters the votes. Both methods run on the same GPU with identical inputs, excluding data loading.

Where the time goes. After feature extraction, scoring across the three-level pyramid takes 6.0 ms for S-RAY and 81.8 ms for LAYS (13.6×), so the speedup comes from the Fourier scorer. Feature extraction takes 34.6 of S-RAY's 40.3 ms with DINOv2-B against 17.6 ms for LAYS's VGG16 U-Net, which reduces the end-to-end ratio to 2.50×. With the shared VGG16 U-Net it takes 20.1 ms for S-RAY and 19.7 ms for LAYS, and the ratio is 4.23×. Both S-RAY configurations are also more accurate than LAYS (backbone table under Q1).

In the revised paper, the abstract, introduction and conclusion state the end-to-end speedup alongside the scorer speedup and recall gain, Sec. 4.3 reports the timings above, and Appendix D.3 describes each configuration and the timing protocol.

Thank you for your rebuttal. After careful consideration of the manuscript quality, I have decided to retain my original score.

Reviewer_GTbPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Structural Radial Aggregation for Yaw (S-RAY), a framework for cross-view yaw estimation under camera position uncertainty. Building upon the radial voting formulation of LAYS, the method introduces two main components. First, a structural alignment score compares ground-view directional distributions with aerial structural orientations to constrain plausible yaw candidates. Second, a Fourier-based voting mechanism replaces discrete hard-bin accumulation with continuous angular kernels. By exploiting the separability of Fourier harmonics, the voting process can be efficiently implemented using FFT-based correlations without explicitly enumerating position–match pairs. The two scores are combined within a coarse-to-fine pose estimation framework. Experiments on KITTI, MGL, and VIGOR demonstrate substantial improvements in yaw estimation accuracy over existing methods, particularly under fully unknown orientations. The proposed Fourier voting also significantly reduces scoring time compared with the original hard-bin implementation. Additionally, the estimated yaw improves downstream localization when integrated with Loc2.

Critical strengths
  1. A well-motivated and computationally efficient formulation of yaw voting. The paper addresses an important challenge in cross-view yaw estimation under position uncertainty. Building upon the radial voting formulation of LAYS, the proposed Fourier voting reformulates discrete yaw accumulation using a continuous angular kernel represented by a small number of Fourier harmonics. The resulting factorization enables efficient FFT-based correlations without explicitly enumerating position–match pairs. This is a meaningful algorithmic improvement, supported by the substantial reduction in scoring time from 19.16 ms to 0.83 ms.

  2. Promising empirical improvements across multiple benchmarks. The method achieves substantial improvements in fine-grained yaw estimation on KITTI, MGL, and VIGOR. In particular, the yaw recall within 1° improves from 51.07% to 87.65% on KITTI Same Area and from 48.44% to 72.91% on KITTI Cross Area compared with LAYS. The ablation results also indicate that the proposed continuous voting improves accuracy over hard-bin voting within the S-RAY framework. Furthermore, the integration with Loc2 demonstrates potential benefits for downstream localization.

Critical weaknesses
  1. Insufficiently controlled comparisons with the most closely related method, LAYS. My primary concern is that the substantial performance improvement over LAYS cannot be clearly attributed to the proposed components. Although S-RAY builds directly upon LAYS, several important aspects of the framework have changed simultaneously. Specifically, LAYS employs a VGG16-based U-Net backbone and a joint pose-volume classification loss, whereas S-RAY adopts DINOv2-B features and a different yaw-oriented training objective. The voting support region is also modified. These differences make it difficult to determine how much of the reported improvement comes from the proposed Structural Score and Fourier Voting, rather than from the stronger feature backbone or revised training strategy. For example, Table 4 shows that replacing Fourier Voting with hard-bin voting within S-RAY still achieves 78.69% R@1° on KITTI Same Area, compared with 51.07% reported for the original LAYS. This substantial remaining gap cannot be attributed solely to the Structural Score because other components also differ. Moreover, the "F only" ablation removes the entire structural branch, which includes the aerial-only pose plausibility score inherited from LAYS, and therefore does not provide a controlled comparison of the proposed structural alignment mechanism. A more rigorous comparison under a unified backbone, training objective, and voting configuration is necessary to establish the independent contributions of the proposed components. This is particularly important beause the paper's primary empirical claim is its substantial improvement over LAYS.

  2. Limited analysis of the proposed scoring mechanisms and their respective contributions. Although the Structural Score is motivated by the alignment of meaningful scene structures across ground and aerial views, its structural interpretation is not sufficiently validated. Both the aerial axis scores and ground direction distributions are learned without explicit structural direction supervision. Therefore, it remains unclear to what extent the learned responses represent geometrically meaningful structures rather than dataset-specific orientation priors. More direct analysis of structural alignment accuracy would strengthen this claim. In addition, the improvement from Fourier Voting should be more clearly separated from the general benefit of using continuous, differentiable vote accumulation. FFT provides an efficient computation of the Fourier-based score, but the reported accuracy gains over hard-bin voting may also arise from the smoothing and learnability of the angular kernel. Comparisons with simpler soft-bin or smooth-kernel voting alternatives are needed to establish whether the Fourier parameterization offers advantages beyond computational efficiency.

Questions for authors

Q1. Can the authors provide a more controlled comparison with LAYS to isolate the contributions of the proposed components? In particular, it would be helpful to evaluate a 2×2 combination of hard-bin/Fourier voting and with/without the proposed structural alignment, while keeping the backbone, training objective, voting support region, and original aerial-only pose plausibility score consistent. Additionally, could the authors clarify how much improvement is obtained by upgrading the original LAYS backbone and changing its training objective independently? These results would significantly strengthen the attribution of the reported performance gains.

Q2. To what extent is the improvement of Fourier Voting attributable to the Fourier representation itself, rather than simply replacing hard assignment with continuous voting? Could the authors compare against a differentiable soft-bin voting scheme or a conventional smooth angular kernel, such as a Gaussian or von Mises kernel, under otherwise identical settings? It would also be helpful to clarify how the number of harmonics affects the learned kernel and the accuracy–efficiency trade-off.

Q3. How can the authors verify that the Structural Score learns meaningful cross-view structural alignment rather than an implicit orientation prior? Since the structural alignment has an inherent 180° ambiguity, an evaluation of orientation accuracy modulo 180°, or the coverage of the ground-truth orientation among the highest-scoring structural candidates, would help clarify its effectiveness. A comparison with a simple road-orientation prior would also be informative.

I would be willing to reconsider my assessment if the authors can provide convincing evidence addressing these concerns, particularly the controlled comparison with LAYS.

4 comments
AuthorsOpenReview ↗

Response 1/3

We thank you for the constructive feedback and for noting our Fourier voting formulation and empirical gains. We ran all requested retrained controls below and added them to the revised paper (marked in red).

Q1 / W1, attribution: controlled 2×2, backbone and objective

We demonstrate more controlled experiments, since the "F only" in Table 4 is not a direct comparison against LAYS's aerial-only pose plausibility score, as you pointed out. We retrained the missing cells to compare our structural score SS (Eq. 4) directly against that plausibility score under both voting rules. The DINOv2-B backbone, the training objective, the ring support and the output grid are fixed across all cells (the structural alignment loss applies only to SS). The ring support is LAYS's own: the LAYS paper states only the outer radius, but its official implementation, which we obtained from its authors, also leaves out aerial pixels nearer than the closest one the ground image sees (20 m on VIGOR). The submitted S-RAY uses the same inner and outer radii (Sec. 2.5, Appendix C). Values are yaw recall within 1° (R@1°, %) / median yaw error (°) on Same Area:

Voting ruleVIGOR, Structural score SSVIGOR, Plausibility score (LAYS)KITTI, Structural score SSKITTI, Plausibility score (LAYS)
Fourier voting55.01 / 0.8825.04 / 2.2187.65 / 0.4985.32 / 0.51
Hard-bin voting41.90 / 1.2622.21 / 2.6478.69 / 0.5679.46 / 0.55
  • VIGOR: the ground roll requires cross-view alignment. Each ground panorama is circularly shifted (rolled) by a random yaw in ±180°, which randomizes the "forward" direction of the ground view. LAYS's plausibility score takes only the aerial view and cannot infer this orientation ("F only" in Table 4, 25.29, is close to the plausibility-score cell, 25.04). Aligning the ground view's direction distribution with the aerial axes in SS adds 29.97 points under Fourier voting and 19.69 under hard bins. Fourier voting adds 13.11 points with SS vs. 2.83 with the plausibility score (interaction: +10.28), as continuous votes help most once SS has narrowed each position to a few candidate yaws.
  • KITTI: the gain comes from Fourier voting. Fourier voting adds 8.96 points over hard bins with SS (87.65 vs. 78.69) and 5.86 with the plausibility score (85.32 vs. 79.46). The car camera faces along the road, so the top-down view alone already narrows the yaw to the road axis in most cases, and SS and the plausibility score perform similarly under hard bins (78.69 vs. 79.46).

Gap to published LAYS: backbone. We retrained S-RAY with LAYS's trained VGG16 U-Net on both views in place of DINOv2-B, every other setting unchanged (R@1° % / median yaw error °, ±180° yaw prior, same protocol as Tables 1–3):

SplitS-RAY (DINOv2-B)S-RAY (VGG16 U-Net)LAYS (VGG16 U-Net)
KITTI Same Area87.65 / 0.4980.39 / 0.5651.07 / 0.96
KITTI Cross Area72.91 / 0.5962.52 / 0.7448.44 / 1.07
MGL55.22 / 0.8746.57 / 1.1034.81 / 1.79
VIGOR Same Area55.01 / 0.8846.55 / 1.1034.16 / 1.94

With the same VGG16 U-Net backbone, S-RAY is ahead of LAYS in all four rows, by 29.32, 14.08, 11.76 and 12.39 points. On KITTI Same Area, the step from published LAYS (51.07) to S-RAY (87.65) reflects both the method and the backbone: in our setup, DINOv2-B hard bins reach 78.69 with SS and 79.46 with the plausibility score (LAYS: 51.07), and DINOv2-B adds 7.26 points to S-RAY (80.39 to 87.65), which leaves hard-bin voting nearly 9 points behind.

Objective. Besides the per-position yaw loss, the objective of Sec. 3.5 and Appendix B.3 has two terms. We removed each and retrained (VIGOR Same Area):

  • Structural alignment loss Lstructureℓ\mathcal{L}^\ell_{\text{structure}} (the loss on SS, Sec. 3.3): without it, SS still improves recall by 24.7 points over LAYS's plausibility score (25.04 to 49.77), and the loss adds a further 5.2 points (55.01). So most of the gain (24.7 of 29.97 points) comes from SS itself.
  • Position-marginalized yaw term: without it R@1° is higher (58.40 vs. 55.01), but the mean yaw error rises from 8.74° to 13.71° and yaw<5° falls from 88.92 to 85.68. The term trades fine recall for fewer large errors.

In the revised paper. Sec. 4.3 reports the 2×2, the VGG-backbone results and the two loss ablations, with full tables in Appendix E. The appendix states that the hard-bin control isolates the scorer and that the LAYS rows of the main tables compare complete systems.

AuthorsOpenReview ↗

Response 2/3

Q2 / W2, Fourier representation vs. continuous voting; smooth kernels; number of harmonics

The main advantage of this reformulation over binned voting is speed. We agree that a Gaussian or von Mises kernel also has a cosine series, so the factorization of Sec. 3.4 applies to any band-limited smooth kernel: the sum of continuous votes over all positions and yaws factors exactly into 4N+14N+1 real FFT correlations, without forming position–match pairs. Which kernel and which vote placement are most accurate is an empirical question, which the retrained controls below answer.

Speed. Binned voting, hard or smoothed with a von Mises or learned kernel, still computes a bin index and scatters a vote for every position-match pair. The Fourier factorization forms no pairs. We re-measured the runtime with author-provided LAYS code:

ScopeS-RAYLAYSSpeedup
Finest-level scorer (Table 5)0.83 ms19.16 ms22.97×
Multi-level scoring after the backbone6.0 ms81.8 ms13.6×
End to end, each method's backbone40.3 ms100.8 ms2.50×
End to end, both on VGG16 U-Net23.7 ms100.3 ms4.23×

The end-to-end rows are re-measured on one RTX 6000 Ada after removing implementation overhead in inference code that both methods share, which the submission's end-to-end times included (breakdown in Appendix D.4).

Accuracy. Retrained controls (VIGOR Same Area, R@1° in %; each differs from S-RAY only in the scorer):

VariantKernelVote placementR@1°
Hard bin (Table 4)indicatorrounded to bin41.90
Hard bin + von Mises smoothingvon Mises, learnable concentrationrounded50.12
Hard bin + learned-kernel smoothinglearned, N=8N=8rounded50.45
Nearest-bin Fourierlearned, N=8N=8match angle rounded48.83
Fixed kernelnot learned, N=8N=8continuous51.69
S-RAYlearned, N=8N=8continuous55.01
  • Smoothing gives 8.2 to 8.6 of the 13.1 points over hard bins, and a von Mises kernel does as well as our learned kernel once votes are binned (50.12 vs. 50.45).
  • Continuous placement adds a further 4.6 to 4.9 points with the same kind of kernel. Nearest-bin Fourier rounds only the match-side angle and keeps the offset direction continuous; it loses 6.18 points (on KITTI Same Area: hard bin 78.69, nearest-bin 85.93, S-RAY 87.65).
  • Learning the kernel adds 3.32 points over the fixed initial kernel.

Smoothing thus accounts for most of the gain over hard bins and continuous placement for the rest, and the Fourier form provides both without per-pair binning. The revised paper reports these controls in Sec. 4.3 and Appendix E.

Number of harmonics. NN sets the cost (4N+14N+1 correlations) and bounds how sharp the kernel can be: N=1N = 1 is a single cosine, 120° wide at half maximum whatever is learned, while the learned N=8N = 8 kernels are 26° to 28° wide (Appendix B.1). We trained N=2,4N = 2, 4 and 1616 in the configuration of the submitted model, changing only NN (VIGOR Same Area):

Harmonics NN1248 (S-RAY)16
Scorer time relative to N=8N = 80.42×0.50×0.67×1×1.66×
R@1° (%)49.7051.7549.8655.0148.89
Median yaw error (°)1.010.961.000.881.03

N=8N = 8 is the best value tested, 3.3 points above the next; N=16N = 16 costs more and does not improve accuracy. Scorer times are measured on the GPU model of the paper's scorer row (RTX PRO 6000 Blackwell) with inputs of the finest-level shape. The sweep is in Appendix E.

AuthorsOpenReview ↗

Response 3/3

Q3 / W2, structural alignment or orientation prior

An orientation prior that ignores the ground image would be unaffected by replacing the ground evidence. We test this on VIGOR, where the random ground-view roll requires cross-view structural alignment SS (Q1). π\pi is the ground direction distribution that SS aligns with the aerial axis scores (Eq. 4).

(a) Swapped ground evidence (trained models evaluated at test time without retraining, Same / Cross Area). R@1° is:

  • 55.1 / 40.3 with the query's own π\pi (Structural score SS),
  • 11.2 / 10.7 with π\pi computed from a different query's ground image,
  • 15.4 / 14.1 with a uniform π\pi,
  • 15.9 / 14.3 without alignment (Cℓ=0C_\ell = 0).

A wrong ground image is worse than no alignment, so SS depends on the query's own ground image.

(b) Road-orientation prior. We replaced π\pi by a learned constant vector, so the alignment scores each yaw from the aerial axis scores and a fixed, image-independent direction profile. Every other term is kept. Because a constant ground profile reduces SS to an aerial-only score that resembles LAYS's plausibility score in form, its accuracy on VIGOR Same Area closely matches LAYS's plausibility score (26.23% with median 2.11° vs. 25.04% with median 2.21°, compared with 55.01% for S-RAY). An aerial orientation prior of this form recovers only 1.2 of the 30.0 points that SS adds; the rest requires the query's ground directions to resolve the panorama roll.

(c) Modulo-180° accuracy of the structural score. SS is π-periodic (Appendix B.3), so we take its own yaw peak at the ground-truth position and count it correct if it lies within two bins of the true yaw modulo 180°. At the coarsest level (4° bins, so within ±8°), where SS first narrows each position to a few candidate yaws, this holds for 92.6 / 92.2% of VIGOR Same / Cross queries and 96.3% of KITTI Same queries, against 11.1% by chance; equivalently, the true yaw is among the two top candidates of SS, θ and θ+180°. Over the full circle, the same peak is correct for about half the queries (46.8 / 49.2 / 50.3%), as expected for a π-periodic score. The non-periodic Fourier score then picks the direction: R@1° of the final prediction modulo 180° is 56.0 / 41.5 / 88.2% against 55.1 / 40.3 / 87.8% full-circle, and only 1.8 / 3.0 / 0.4% of queries land within 5° of the opposite heading.

Summary of Experiments. The gain from SS requires the query's own ground view: another query's ground evidence or a constant profile removes it, so SS performs cross-view alignment rather than acting as an orientation prior. These diagnostics are in Sec. 4.3 and Appendix E of the revised paper.

Thank you for the detailed responses and additional experiments. I have checked the revised manuscript and confirmed that the major experiments discussed in the rebuttal have been incorporated. The new controlled comparisons and ablation studies address my main concerns regarding the contributions of the Structural Score and Fourier Voting. One minor suggestion is to clarify more explicitly that the voting support region is consistent with the original LAYS implementation, to improve reproducibility. Overall, the revision has substantially strengthened the paper, and I am satisfied with the authors' responses. I therefore raise my rating from Weak Reject to Weak Accept.

Reviewer_t2FrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper estimates ground-camera yaw when position is uncertain and yaw is unknown over the full circle, building on LAYS’s radial column-match votes. It adds (i) a structural score that aligns ground structure directions with aerial structure axes to prune implausible yaws at each position, and (ii) a Fourier score that replaces hard yaw bins with continuous cosine votes. Experimental results show that the proposed method achieves state-of-the-art performance in yaw estimation.

Critical strengths
  1. Methodological Contribution: Compared to the prior baseline LAYS, there are genuine methodological innovations introduced, though it remains an incremental body of work.
  2. Top-Tier Performance: The experimental results demonstrate the best overall performance in yaw estimation.
Critical weaknesses
  1. Discrepancy Between Yaw and Position Accuracy: Why is yaw estimation exceptionally accurate, yet the position estimation significantly worse than other baselines? This trade-off requires further clarification and analysis.
  2. Incremental Nature: The paper is indeed an incremental work that inherits and builds heavily upon many structural designs of LAYS.
  3. The evaluation surface is narrower than that of the baseline. LAYS evaluates on Mapillary (= MGL), Ford, KITTI, VIGOR and reports an MGL ±45° yaw prior; this paper reports KITTI/MGL/VIGOR under ±180° only — Ford and ±45° are missing.
Questions for authors
  1. Please address all the questions and concerns raised in the Weaknesses section, with a particular focus on providing the missing evaluations and comparisons on the datasets/settings consistent with LAYS.
1 comment

We thank you for the constructive review and for noting our methodological innovations and state-of-the-art yaw estimation results. Paper changes are marked in red.

W1, why yaw is more accurate than position. Yaw voting tolerates position error along the viewing ray. A matched aerial pixel lies in the same direction from every camera position on the ray through it, so each of these positions implies the same yaw for that match (Sec. 3.2). Voting thus constrains yaw and the position perpendicular to each ray, but only weakly the position along it. S-RAY shares this geometry with LAYS, which is evaluated as a yaw-only method (Tables 1 to 3).

On KITTI, where the forward camera's rays run mostly along the road, localization accuracy shows this pattern. S-RAY is precise laterally and ambiguous along the road: 97.56% of Same Area queries are within 5 m laterally and 42.91% longitudinally (Cross Area: 82.83% and 32.22%), while yaw R@1° is 87.65% and 72.91%.

The revised abstract, introduction and Sec. 4.2 state the tolerance of yaw to position error along the ray, and Sec. 4.2 gives the position accuracy on each dataset.

W2, relation to LAYS. S-RAY keeps LAYS's column matching and radial invariance (Sec. 1, Sec. 3.2). What it changes is how yaw evidence is gathered and scored, in two components that LAYS has no counterpart for:

  • A cross-view structural score. LAYS's plausibility score rates a pose from the aerial image alone, so it can tell which yaws are plausible at a location but not which one the photo shows. Our score SS compares the viewing directions where the ground photo shows structure, under each candidate yaw, with the structure axes predicted from the aerial image at the candidate position. It is learned from pose labels only and judges the whole view at once, while each column match checks a single direction. SS keeps a few yaws per position, and the matches choose among them.
  • Fourier voting computed as correlations. Hard bins assign each position–match pair separately, so the cost grows with the number of pairs, and a vote's training credit jumps at bin edges. S-RAY replaces the bin with a learned bump of NN cosines at the implied yaw. By angle addition, each cosine factorizes into a yaw term, a match term, and an offset term that is the same ring-shaped kernel around every position. Scoring every position and yaw then reduces exactly to 4N+14N+1 FFT correlations of the match map with that kernel: no position–match pairs are formed, and the score is continuous in yaw with a smooth gradient.

Each component is tested by swapping LAYS's version back in and retraining (yaw recall within 1°, VIGOR Same Area):

Voting ruleStructural score SSPlausibility score (LAYS)
Fourier voting55.0125.04
Hard-bin voting (LAYS's rule)41.9022.21
  • SS vs. plausibility score (+29.97 points): VIGOR panoramas are randomly rolled, so their "forward" direction cannot be inferred from the aerial view alone; aligning ground structure with aerial axes recovers it.
  • Fourier vs. hard bins (+13.11 points, 23× faster scorer): 0.83 vs. 19.16 ms at the finest level (Table 5). On KITTI, where the camera faces along the road, the aerial view alone already narrows the yaw (SS 78.69 vs. plausibility 79.46 under hard bins), and Fourier voting lifts R@1° to 87.65.
  • Backbone control: Retrained with LAYS's VGG16 U-Net, S-RAY stays above published LAYS and VGG hard-bin voting under the same ±180° protocol: 80.39 vs. 51.07 on KITTI Same Area, 62.52 vs. 48.44 on KITTI Cross Area, 46.57 vs. 34.81 on MGL, and 46.55 vs. 36.95 on VIGOR Same Area, while running 4.23× faster end to end (23.7 vs. 100.3 ms).

W3, evaluation scope: MGL with the ±45° yaw prior and Ford.

  • MGL ±45°: Under the LAYS ±45° protocol (training rotations and yaw bins limited to ±45°), S-RAY reaches 78.96 / 95.64 / 98.13% yaw recall within 1° / 2° / 4° (median 0.56°), compared with 72.10 / 90.16 / 95.63% reported in Table 1 of LAYS. Large yaw errors are 2.3× rarer: 1.87% of samples exceed 4° against 4.37% for LAYS. Under ±180° the 1° recall is 55.22% vs. 34.81% (Table 2).
  • Ford Multi-AV: Ford was not in the submission because recent works rarely use it, and only with bounded priors (±10° in Song et al., 2023; ±45° in LAYS). We have now trained and tested S-RAY on Ford Log 1 under the LAYS protocol (±45° yaw, ±20 m translation). S-RAY reaches 74.29 / 95.43 / 98.38% yaw recall within 1° / 2° / 4° (median 0.63°), compared with 67.05 / 85.14 / 91.76% reported by LAYS (Table 6) and 41.36 / 62.74 / 75.11% for CCVPE. Large yaw errors are 5.1× rarer: 1.62% of test samples exceed 4° against 8.24% for LAYS.

Comments on the paper

1 comment

We thank all reviewers for their feedback and for recognizing our continuous Fourier voting formulation (GTbP, 7djG) and yaw accuracy across benchmarks (GTbP, 7djG, t2Fr).

The main shared concern is attribution: how much of the gain over LAYS comes from our structural score SS and Fourier score FF, given that the backbone and objective also differ. Below, numbers are yaw recall within 1° (R@1°, %) on VIGOR Same Area (unknown prior) unless noted.

1. What S-RAY changes in LAYS (t2Fr W2). S-RAY keeps LAYS's column matching and radial invariance and changes how yaw evidence is scored. (i) A cross-view structural score SS, learned from pose labels only, aligns the directions where the photo shows structure with aerial structure axes; LAYS's plausibility score sees only the aerial image. (ii) Fourier voting replaces hard bins with continuous cosine votes and factorizes them exactly into 4N+14N+1 FFT correlations, so no position–match pairs are formed. In this rebuttal, we further demonstrate the ablation and efficiency impact of this new formulation.

2. Controlled 2×2 (GTbP Q1, 7djG Q1). Voting rule × structural score, with backbone, objective, support and grid fixed:

Voting ruleStructural score SSPlausibility score (LAYS)
Fourier voting55.0125.04
Hard-bin voting41.9022.21

On VIGOR, rolled panoramas have no fixed "forward" direction for an aerial-only score to infer; cross-view alignment in SS adds 29.97 points under Fourier voting, and Fourier voting adds 13.11 points with SS vs. 2.83 with the plausibility score. On KITTI Same Area, Fourier voting adds 8.96 points over hard bins (87.65 vs. 78.69); because the car camera faces along the road, the top-down view alone narrows the yaw (SS 78.69 vs. plausibility 79.46 under hard bins).

3. Backbone (GTbP Q1, 7djG Q1). With LAYS's VGG16 U-Net backbone, S-RAY beats LAYS and VGG hard bins: 80.39 vs. 51.07 on KITTI Same Area, 62.52 vs. 48.44 on KITTI Cross Area, 46.57 vs. 34.81 on MGL, and 46.55 vs. 36.95 on VIGOR Same Area.

4. Loss terms (7djG Q1). Removing the structural alignment loss Lstructureℓ\mathcal{L}^\ell_{\text{structure}} (Sec. 3.3), SS still adds 24.7 points (25.04 to 49.77) and Lstructureℓ\mathcal{L}^\ell_{\text{structure}} adds 5.2 more (55.01). Without the position-marginalized term: 58.40, with mean error rising from 8.74° to 13.71°; it reduces large errors.

5. Fourier vs. smoothing (GTbP Q2). Smoothing hard-bin votes with a learned or von Mises kernel gives 50.45 / 50.12 (hard bins: 41.90), while continuous Fourier voting reaches 55.01. Smoothed hard bins still bin every position–match pair. The scorer is 22.97× faster (0.83 vs. 19.16 ms). Harmonics N=1,2,4,8,16N = 1, 2, 4, 8, 16 give 49.70, 51.75, 49.86, 55.01, 48.89.

6. Alignment or orientation prior (GTbP Q3). On VIGOR, giving SS a different query's ground evidence drops R@1° from 55.1 to 11.2, and retraining with a learned constant ground distribution reaches 26.23, close to the plausibility score (25.04), as both reduce to aerial-only scoring.

7. Efficiency (7djG Q4). Post-backbone scoring: 6.0 vs. 81.8 ms (13.6×). End to end, with shared overhead removed for both, S-RAY takes 40.3 vs. 100.8 ms (2.50×) with default backbones and 23.7 vs. 100.3 ms (4.23×) with matched VGG16 U-Net, with higher recall in both cases.

8. Yaw vs. position accuracy and evaluation scope (t2Fr W1, W3). Why yaw is more accurate than position (W1): a matched aerial pixel lies in the same direction from every camera position along its ray, so position error along the ray leaves yaw unchanged; voting thus constrains yaw and position perpendicular to the ray. Evaluation scope (W3): under ±45° prior, S-RAY has far fewer large errors: 1.87% of MGL samples exceed 4° yaw error (LAYS 4.37%) and 1.62% on Ford (LAYS 8.24%), 2.3× and 5.1× fewer, with higher recall at 1° / 2° / 4° (MGL 78.96 / 95.64 / 98.13 vs. LAYS 72.10 / 90.16 / 95.63; Ford 74.29 / 95.43 / 98.38 vs. 67.05 / 85.14 / 91.76).

Revisions (red in the updated paper)

  1. 2×2 table (voting rule × SS vs. LAYS's plausibility score) on VIGOR and KITTI, plus ground-swap, constant-prior, loss and VGG controls (Sec. 4.3, Appendix E).
  2. Smoothing, nearest-bin and fixed-kernel controls and the NN sweep (Sec. 4.3, Appendix E).
  3. Sec. 3.3 and 3.5: what Eq. (4) fixes vs. what training learns, structural alignment loss, full objective, and gate λ\lambda (Appendix B.3).
  4. Modulo-180° accuracy of the structural score, output-grid sweep (Appendix E), and kernel statistics (Appendix B.1).
  5. End-to-end and scorer speedups paired with recall (abstract, intro, Sec. 4.3, App. D.4).
  6. Baseline yaw conventions and modulo-180° results (Appendix D.1); MGL and Ford results with a ±45° yaw prior against the baselines reported by LAYS (Appendix D.2).
  7. Ray-direction invariance of yaw vs. position error and per-dataset position accuracy (abstract, intro, Sec. 4.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.