acceptodds
Under review as a conference paper at ICLR 2027

LoDEOT: Low-Dimensional and Efficient Offset Tokens for Building Footprint Extraction from Off-Nadir Imagery

10 views

Abstract

Instance-level roof-to-footprint offset (RFO) prediction is central to extracting building footprints from off-nadir imagery. Query-based pipelines commonly use high-dimensional instance tokens to predict signed two-dimensional RFOs. We investigate whether RFO prediction can instead use a compact offset token. Under local pinhole projection and vertical-extrusion assumptions, the idealized RFO map admits a five-parameter sufficient descriptor comprising intrinsic shape, composite amplitude, and relative geometry. This factorization provides a structural prior for a five-dimensional offset token, whose channels learn task-relevant latent representations through end-to-end training. Based on this design, we propose LoDEOT, which retains high-dimensional instance tokens for detection and segmentation but maps instance-token, concentration-gated roof, and box–mask evidence to a five-dimensional offset token followed by an independent two-dimensional readout. Known denoising-query target indices further align each supervised decoder-layer estimate with the same clean instance RFO, organizing successive predictions as target-aligned recovery under perturbed query conditions. Experiments on five real-world building datasets demonstrate the effectiveness of LoDEOT for building footprint extraction. Experiments on real-world building datasets demonstrate that a five-dimensional offset token can support accurate RFO prediction. On BONAI, LoDEOT achieves the best roof-detection bAP and bAP50 and leads all five offset-corrected footprint metrics among the evaluated end-to-end methods, with FAP50 of 54.58 and mEPE of 5.23 pixels. Its FAP50 exceeds those of the evaluated end-to-end baselines by 7.56–16.85 percentage points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Off-Nadir Imagery, Building Footprint Extraction, Offset Tokens, Denoising Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L5nUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the intrinsic relationship between RFO and key physical parameters in the photogrammetric process and tailors this analysis to the off-nadir remote sensing setting. Building on the photogrammetric analysis and insights from modern foundation models, the authors propose LoDEOT and equip it with a customized denoising training framework.

Critical strengths

The paper demonstrates a commendably cautious approach to acknowledging its limitations.

Critical weaknesses
  1. The lack of a coherent motivation–method–evidence chain makes it difficult for readers to understand why the proposed design is necessary to address a specific problem and whether the experimental results directly demonstrate that the design effectively resolves that problem.
  2. Due to the lack of the latest comparison methods, it is impossible to fully demonstrate the superiority of this method.
Questions for authors

Please refer to the weaknesses listed above.

4 comments
AuthorsOpenReview ↗

Meta Response

Meta Response:

We sincerely thank you for dedicating your valuable time to reviewing our manuscript. We deeply regret that our current presentation did not clearly convey the connection between our underlying motivation and empirical validation. We take full responsibility for this clarity gap in our writing—as reflected by the fact that while Reviewer hVff grasped the geometric motivation and focused mainly on experimental verification, our exposition left these links insufficiently transparent in your reading.

We also appreciate your note regarding our discussion of the paper's limitations ("The paper demonstrates a commendably cautious approach to acknowledging its limitations"). To avoid any potential misunderstanding, we would like to clarify that this work is entirely human-conceived and executed—not an auto-research or AI-generated study (AI tools were used strictly for minor language polishing)—and our cautious tone stems solely from a commitment to scientific rigor rather than defensiveness.

Because our approach bridges two distinct subfields, fully appreciating the motivation requires combining domain knowledge in remote sensing segmentation with recent foundational findings in general domain, e.g., 3D vision and Video World Models (specifically, how Transformer architectures implicitly parse and encode 3D camera geometry without explicit physical supervision). To make this cross-domain foundation clear and accessible, we have summarized these core premises in our Meta Response to Reviewer fYgp and added a dedicated Related Work section in the revised manuscript. We kindly invite you to refer to that overview alongside our point-by-point responses below.

AuthorsOpenReview ↗

Response Q1

Q1: The lack of a coherent motivation–method–evidence chain makes it difficult for readers to understand why the proposed design is necessary to address a specific problem and whether the experimental results directly demonstrate that the design effectively resolves that problem.

Thank you for your feedback. Building upon the foundational concepts outlined in our Meta Response above (and in our Response to Reviewer fYgp), we have restructured our presentation and added targeted controlled experiments to establish a clear, unbroken motivation–method–evidence chain:

Step 1: From Core Observations to Architectural Design (Motivation →\rightarrow Method)

Our architecture is directly motivated by three foundational observations established in the Introduction:

  • Observation 1 (Implicit 3D Modeling in Transformers): Recent advances in 3D vision (e.g., AC3D) demonstrate that Transformer architectures naturally and implicitly parse and encode 3D imaging geometry during training, without requiring explicit channel-wise supervision or direct disentanglement into raw physical parameters.
  • Observation 2 (Compact Geometric Parameterization in Off-Nadir Remote Sensing): Due to the high-altitude, near-parallel imaging characteristics of off-nadir satellite/aerial imagery, the 3D photogrammetric projection δ=Φ(K,R,r,H)\delta = \Phi(K, R, r, H) can be analytically simplified into a minimal 5-parameter representation η5=(a,κ~,γ,βx,βy)⊤\eta_5 = (a, \tilde{\kappa}, \gamma, \beta_x, \beta_y)^\top.
  • Observation 3 (Layer-Wise Denoising Dynamics): Deep Transformer decoders iteratively refine predictions through a layer-by-layer denoising process.

How These Observations Dictate Our Method:

  1. Why a Compact 5D Offset Token (ToffsetT_{\text{offset}})? Combining Observations 1 & 2, because both the physical projection Φ(K,R,r,H)\Phi(K, R, r, H) and the network readout Φ∗(Toffset)\Phi^*(T_{\text{offset}}) model the exact same geometric mapping, the token's degrees of freedom must satisfy rank(Toffset)≥5\text{rank}(T_{\text{offset}}) \ge 5 to span the displacement space without geometric information loss. Thus, the 5D derivation acts as a principled geometric bottleneck—allowing us to decouple high-dimensional semantic/mask queries from a compact 5D recurrent offset state, where the Transformer implicitly encodes combinations of the underlying physical factors.
  2. Why Layer-Wise Denoising Supervision? Guided by Observation 3, injecting controlled additive perturbations into target-indexed offset tokens and applying layer-wise clean-target supervision trains the decoder to progressively denoise and refine the 5D geometric state across layers rather than relying on single-step final-layer regression.

Step 2: Explaining Empirical Dimensional Behavior Across Three Regimes (Method →\rightarrow Evidence)

To connect our theoretical 5D bound to the empirical results in Table 3 (where neighboring dimensions perform comparably on constrained subsets), we analyze the token width dd across three distinct regimes:

  • 1. High-Dimensional Regime (d∈[256,+∞)d \in [256, +\infty)): Prior baselines uniformly default to over-parameterized embeddings (d≥256d \ge 256). Our work breaks this convention by demonstrating that geometric offset regression can be solved within a compact low-dimensional subspace.
  • 2. Over-Parameterized Low-Dimensional Regime (d∈[5,256)d \in [5, 256)): In Table 3, accuracy across d∈[5,12)d \in [5, 12) and d∈[12,256)d \in [12, 256) shows no statistically significant variance because once d≥5d \ge 5, the token already satisfies the full-rank DoF bound (rank(Toffset)≥5\text{rank}(T_{\text{offset}}) \ge 5). Increasing dd beyond 5 merely introduces redundant latent capacity without new geometric information.
  • 3. Sub-Rank Regime (d∈[2,5)d \in [2, 5)) — Conditional Dimensionality Reduction: Real-world benchmarks cover a finite set of satellite sensors, and off-nadir viewing angles are bounded within narrow operational ranges (e.g., θv∈[2∘,30∘]\theta_v \in [2^\circ, 30^\circ], far from 90∘90^\circ horizon grazing angles). Substituting real satellite parameters into our equations confirms conditional dimensionality reduction, collapsing the effective degrees of freedom from 5D onto an approximate 2D–2.5D manifold. Hence, 2D–2.5D reflects the dataset-specific numerical approximation, whereas 5D guarantees full-rank geometric completeness across unconstrained viewing geometries.

(Quantitative tables verifying the physical sensitivity, SVD spectral alignment, 5D vs. Direct 2D head, and layer-wise denoising ablation follow in Part 2/2 below.)

AuthorsOpenReview ↗

Response Q1

To directly substantiate every link in the chain above, we added four controlled quantitative experiments to the revised Appendix:

1. Real-World Satellite Physical Sensitivity Verification Substituting real orbital/camera parameters from five satellites (Jilin-1 GF03D, QuickBird, WorldView-2, WorldView-3, Sentinel-2) with empirical building heights (3m3\text{m}–102m102\text{m}), focal anisotropy (a∈{0.99,1.00,1.01}a \in \{0.99, 1.00, 1.01\}), normalized skew (κ~∈{−0.01,0,0.01}\tilde{\kappa} \in \{-0.01, 0, 0.01\}), and viewing angles (θv∈[2∘,30∘]\theta_v \in [2^\circ, 30^\circ]) into η5\eta_5 yields sensitivity indices S(q)=σ(q)Ew[∥∂δ/∂q∥22]S(q) = \sigma(q)\sqrt{\mathbb{E}_w[\|\partial \delta / \partial q\|_2^2]} (px):

SatelliteS(a)S(a)S(κ~)S(\tilde{\kappa})S(γ)S(\gamma) (Height)S(βx)S(\beta_x)S(βy)S(\beta_y)S(B)S(B) (Geometric)
Jilin-1 GF03D0.02240.02503.32982.89972.00722.2434
QuickBird0.02590.02883.84213.34582.31602.5886
WorldView-20.03660.04075.42884.72763.27253.6577
WorldView-30.05420.06048.05587.01534.85605.4276
Sentinel-20.00060.00070.08540.07450.04930.0532
  • Physical Finding: Height-related S(γ)S(\gamma) and geometric magnitude S(B)=∥β∥2S(B)=\|\beta\|_2 dominate intrinsic shape terms S(a),S(κ~)S(a), S(\tilde{\kappa}) by two orders of magnitude, proving conditional collapse onto an approximate 2D–2.5D manifold under bounded viewing angles.

2. SVD Spectral Distribution & Principal Component Truncation Performing SVD on learned dd-dimensional offset features before the readout layer and truncating minor components at inference time shows:

Dim ddTop-1 Energy E1E_1 (%)Top-2 Energy E2E_2 (%)r99r_{99}Full mEPE (px)Top-1 Only mEPE (px)Top-2 Only mEPE (px)
280.90100.0022.82393.28682.8239
387.7099.99822.76753.54672.7665
494.9499.99122.75633.37432.7574
580.9399.99222.67233.42732.6745
1680.1399.97322.69463.30152.6958
25679.5199.96022.64173.29862.6819
  • Spectral Finding: For all d>2d>2, the top-2 components capture 99.960%–99.998% variance (r99=2r_{99}=2). For our 5D model, dropping components 3–5 changes mEPE by only +0.0022 px+0.0022\text{ px} (2.6723→2.67452.6723 \rightarrow 2.6745), while dropping the 2nd component degrades mEPE to 3.4273 px3.4273\text{ px}, matching the 2 dominant physical factors (γ,B\gamma, B).

3. Direct Attribution to the 5D Offset Representation (vs. Same-Framework Direct 2D Head) To isolate the exact gain of our 5D geometric bottleneck from backbone or training recipes, we replaced our offset branch with a standard Direct 2D Regression Head within the exact same framework and training schedule:

Offset Head DesignFootprint AP ↑\uparrowFootprint AP50\text{AP}_{50} ↑\uparrowOffset mEPE (px) ↓\downarrow
Direct 2D Regression Head19.7942.083.21
LoDEOT (5D Offset Token, Ours)22.7247.222.81
Net Gain (Δ\Delta)+2.93+5.14-0.40 (-12.46%)
  • Finding: With all other variables held strictly constant, our 5D offset representation alone improves Footprint AP50\text{AP}_{50} by +5.14+5.14 and reduces offset error by 12.46%12.46\%.

4. Independent Ablation of Layer-Wise Denoising Supervision To verify that our layer-wise denoising framework (Observation 3) actively resolves iterative error accumulation, we independently varied supervision depth and perturbation type under the exact same backbone:

Supervision DepthPerturbation TypeFootprint AP ↑\uparrowFootprint AP50\text{AP}_{50} ↑\uparrowOffset mEPE (px) ↓\downarrow
Final-layerNone20.8943.843.30
Final-layerAdditive20.4242.883.32
Layer-wiseNone19.4242.013.30
Layer-wiseMultiplicative21.0444.163.02
Layer-wiseAdditive (Ours)22.7247.222.81
  • Finding: Adding noise only at the final layer degrades performance, whereas pairing layer-wise supervision with additive noise reduces mEPE by 0.49 px0.49\text{ px} (3.30→2.81 px3.30 \rightarrow 2.81\text{ px}) over the un-noised baseline.

Together, these four controlled experiments establish a complete, unbroken chain from physical observation to architectural design and empirical improvement.

AuthorsOpenReview ↗

Response Q2

Q2: Due to the lack of the latest comparison methods, it is impossible to fully demonstrate the superiority of this method.

To the best of our knowledge, Table 2 in the main text already evaluates all publicly available, reproducible open-source baselines in off-nadir building footprint extraction (including externally guided methods such as OBM and PolyFootNet, and end-to-end methods such as LOFT, Cas.LOFT, and MLS-BRN).

We also thoroughly examined the most recent literature in this area—specifically DG-BRF (ISPRS J. Photogramm. Remote Sens., 2026) and DragRoof (arXiv:2607.25210, 2026)—and checked their official repositories prior to submission and during the rebuttal period:

  • DG-BRF (https://github.com/zhenghaohu/DG-BRF): Although the paper was published earlier this year, its official repository has not yet released any source code or pretrained weights (community requests in its GitHub Issues remain open).
  • DragRoof (https://github.com/likaiucas/DragRoof): Currently a preprint whose official repository similarly lacks public code or model checkpoints.

Because neither method provides official code or pretrained weights, re-implementing them from scratch without verified architectural details or hyperparameters would risk an unfair comparison. Should the authors release their official implementations—or if there are any other open-source, reproducible baselines we should include, we would be very glad to add them to our benchmark comparisons.

In other words, if there are any other open-source, reproducible baselines we may have overlooked, we would be grateful if you could point us to them.

Reviewer_hVffOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoDEOT addresses building footprint extraction from off-nadir imagery by predicting roof masks and signed roof-to-footprint offsets. It preserves high-dimensional instance queries while maintaining a compact recurrent offset token, updated using query features, mask-pooled appearance, and box-mask geometry. A five-parameter factorization of an idealized projection model motivates a five-dimensional token, although its learned channels are not constrained to represent these physical factors. Training additionally uses perturbed, target-indexed offset tokens with layer-wise clean-target supervision. Experiments include a BONAI system comparison, cumulative training across multiple sources, and width and denoising ablations on IRSA.

Critical strengths
  1. A practical formulation with geometric analysis. Separating displacement prediction from the richer representation required for detection and segmentation is sensible. The explicit projection factorization and conditional approximation bounds provide useful context for studying compact offset representations.

  2. Encouraging task-level results and a substantial width study. On BONAI, LoDEOT reports FAP50 of 54.58 versus 47.02 for Cas.LOFT, alongside mEPE of 5.23 versus 5.52 pixels. The study tests 18 widths over five seeds, and the denoising-path ablation reports a sizable FAP50 improvement from 37.21 to 46.79. Evaluation on multiple domains, including held-out Huizhou, adds useful context.

Critical weaknesses
  1. The architectural benefit of the physical factorization remains unclear. The five token channels are not tied to the derived physical parameters, and the final linear prediction layer does not explicitly use the factorized projection equation. Table 3 shows that several compact widths, including two dimensions, work well. The experiments do not establish what the derivation contributes beyond selecting the width of a generic compact state.

  2. Controlled evidence for the denoising mechanism and efficiency is incomplete. Table 4 changes perturbation and supervision jointly, leaving the benefit of noisy offset initialization unresolved. Smaller offset tokens and affine readouts do not establish whole-model efficiency, which remains unmeasured. A same-framework direct two-dimensional regressor, with matched supervision and training budget, is needed for accuracy, latency, and peak GPU-memory comparisons. Compatible evaluations against DG-BRF and DragRoof are also missing.

Questions for authors
  1. Core concern: the justification for five dimensions. In Table 3, the two-dimensional token has higher mean FAP50/F1 than the five-dimensional token (45.67/51.60 versus 45.20/50.93), although the differences are not statistically significant. What evidence supports choosing five dimensions on physical grounds, and what benefit does the factorization provide beyond a generic compact representation? This is central to my assessment. Convincing evidence connecting the derivation to a practical architectural benefit would make me consider raising my score.

  2. Comparison with recent methods. The main end-to-end comparison is limited to LOFT, Cas.LOFT, and MLS-BRN. Could the authors compare with recent related methods such as DG-BRF and DragRoof under a consistent evaluation protocol? This would help establish whether LoDEOT remains competitive with recent approaches to geometric offset estimation.

  3. Practical efficiency. The reported savings concern offset tokens and their readout rather than the complete model. Could the authors compare LoDEOT with a same-framework direct two-dimensional regression head, reporting accuracy, end-to-end inference latency, and peak GPU memory under identical conditions? This would establish whether the compact offset branch offers a practical accuracy-cost advantage.

References relevant to Question 2 (already cited by the submission):

  • Hu et al. 3D building reconstruction from monocular remote sensing imagery via diffusion models and geometric priors. ISPRS Journal of Photogrammetry and Remote Sensing, 232:124-137, 2026. DG-BRF.
  • Li et al. ObliCity: A Benchmark and Baseline for Roof-to-Ground Projection Displacement Correction. arXiv:2607.25210, 2026. DragRoof.
7 comments
AuthorsOpenReview ↗

Meta Response to Reviewer hVff

Meta Response:

We sincerely thank you for your thoughtful, thorough, and constructive review. Among all reviewers, we deeply appreciate that your comments reflect a strong background in 3D vision and a clear understanding of our physical imaging formulation and conditional geometric bounds. I believe your insights are invaluable to us and to the Area Chair in evaluating the contributions of this work.

If you would still like further background on the foundational motivation behind binding 3D imaging geometry to compact token design, please refer to our Meta Response to Reviewer fYgp, where we reiterate two core premises: (1) the implicit 3D modeling capability of Transformer architectures established in recent 3DV literature, and (2) the degrees-of-freedom rank alignment between the physical projection process and the learned offset representation.

Building directly upon your shared understanding of these principles (given your background in this domain, you clearly recognize our core motivation, and your primary concerns center on rigorous experimental validation.), our point-by-point responses below will focus squarely on your core questions—grounding our theoretical justification in Appendix A ("Conditional Reduction of the RFO Map under Restricted Imaging Geometry") and presenting newly added physical sensitivity verifications using real-world satellite parameters, controlled denoising/backbone ablations, and end-to-end runtime/memory benchmarks.

AuthorsOpenReview ↗

Response to Q1 (Part 1/2)

Q1 Core concern: the justification for five dimensions. In Table 3, the two-dimensional token has higher mean FAP50/F1 than the five-dimensional token (45.67/51.60 versus 45.20/50.93), although the differences are not statistically significant. What evidence supports choosing five dimensions on physical grounds, and what benefit does the factorization provide beyond a generic compact representation? This is central to my assessment. Convincing evidence connecting the derivation to a practical architectural benefit would make me consider raising my score.

Response to Q1 (Part 1/2: Physical Justification & Architectural Benefits of the 5D Factorization)

We sincerely thank you for your openness to reconsidering your score if we provide convincing evidence linking our physical derivation to practical architectural benefits—this is deeply encouraging to us.

As noted in our Meta Response, for the foundational motivation behind implicit 3D modeling in Transformers and DoF rank alignment, please refer to our Response to Q1.1 for Reviewer fYgp (which aligns with the physical intuition you noted in your Strengths). Here, we directly address your two central questions: (1) why 5D is justified on physical grounds despite 2D achieving comparable mean FAP50\text{FAP}_{50} in Table 3, and (2) what concrete architectural benefits the factorization provides beyond selecting a generic compact width.


1. Practical Architectural Benefits Beyond Width Selection

Our 5-parameter physical factorization η5=(a,κ~,γ,βx,βy)⊤\eta_5 = (a, \tilde{\kappa}, \gamma, \beta_x, \beta_y)^\top is not merely a post-hoc justification for picking d=5d=5; it shapes our architecture in three concrete ways:

  1. Principled Bottleneck Sizing Without Blind Search: Rather than relying on heuristic width tuning or defaulting to over-parameterized 256D+256\text{D}+ embeddings, the factorization establishes the exact minimal full-rank geometric bottleneck (rank(Toffset)≥5\text{rank}(T_{\text{offset}}) \ge 5) required to span unconstrained off-nadir projections.
  2. Decoupled Dual-Stream Architecture: By isolating the 5D geometric state from high-dimensional instance queries, the architecture separates semantic/boundary segmentation (handled by high-dimensional queries) from recurrent displacement refinement (handled by the compact ToffsetT_{\text{offset}}). This decoupling allows lightweight affine readouts and layer-wise denoising updates to operate directly on a low-dimensional geometric manifold without destabilizing mask segmentation.
  3. Predictable Spectral Hierarchy & Generalization Safety: The factorization predicts an exact hierarchy between two dominant physical factors (γ,B\gamma, B) and two secondary intrinsic modulations (a,κ~a, \tilde{\kappa}). In practice, this explains why d=2d=2 performs competitively on narrow benchmarks (where a,κ~a, \tilde{\kappa} barely vary) while 5D guarantees zero geometric information loss when generalizing across diverse sensors and viewing geometries—at virtually zero extra latency or memory cost (see Response to Q3).

2. Why d=2d=2 and d=5d=5 Perform Comparably in Table 3: Three Dimensional Regimes

To state our core conclusion directly: 5D is derived as a theoretically sufficient full-rank upper bound for general off-nadir geometry, rather than an empirically exclusive optimum on constrained subsets. Under operational remote sensing constraints, the effective intrinsic dimensionality collapses to approximately 2D–2.5D:

  • High-Dimensional Regime (d∈[256,+∞)d \in [256, +\infty)): Standard baselines default to d≥256d \ge 256. Our work breaks this convention by proving that offset regression does not require over-parameterized embeddings.
  • Over-Parameterized Low-Dimensional Regime (d∈[5,256)d \in [5, 256)): In Table 3, performance across d∈[5,12)d \in [5, 12) and d∈[12,256)d \in [12, 256) shows no statistically significant variance because d≥5d \ge 5 already satisfies the full-rank DoF bound (rank(Toffset)≥5\text{rank}(T_{\text{offset}}) \ge 5). Extra dimensions merely add redundant capacity.
  • Sub-Rank Regime (d∈[2,5)d \in [2, 5)) — Conditional Dimensionality Reduction: In real-world benchmarks, datasets cover a finite set of satellite sensors, and off-nadir viewing angles are bounded within narrow operational ranges (e.g., θv∈[2∘,30∘]\theta_v \in [2^\circ, 30^\circ], far from 90∘90^\circ horizon grazing angles). As formally analyzed in Appendix A ("Conditional Reduction of the RFO Map under Restricted Imaging Geometry"), these operational bounds induce conditional dimensionality reduction, collapsing the effective degrees of freedom from 5D onto an approximate 2D–2.5D manifold. Thus, 2D–2.5D reflects the dataset-specific numerical approximation, whereas 5D preserves full-rank geometric completeness.

(Quantitative physical sensitivity and SVD spectral truncation proofs follow in Part 2/2 below.)

AuthorsOpenReview ↗

Response to Q1 (Part 2/2)

Response to Q1 (Part 2/2: Quantitative Physical & Spectral Evidence Added to the Revision)

To substantiate Part 1/2 and directly link our physical factorization to the learned network representation, we added two quantitative verification experiments to the revised Appendix:

1. Real-World Satellite Physical Sensitivity Verification We substituted real orbital and camera parameters from five representative satellites (Jilin-1 GF03D, QuickBird, WorldView-2, WorldView-3, and Sentinel-2) alongside empirical building heights (3m3\text{m}–102m102\text{m}), focal anisotropy (a∈{0.99,1.00,1.01}a \in \{0.99, 1.00, 1.01\}), normalized skew (κ~∈{−0.01,0,0.01}\tilde{\kappa} \in \{-0.01, 0, 0.01\}), and operational off-nadir angles (θv∈[2∘,30∘]\theta_v \in [2^\circ, 30^\circ]) into our 5-parameter formulation η5=(a,κ~,γ,βx,βy)⊤\eta_5 = (a, \tilde{\kappa}, \gamma, \beta_x, \beta_y)^\top. Computing the sensitivity index S(q)=σ(q)Ew[∥∂δ/∂q∥22]S(q) = \sigma(q)\sqrt{\mathbb{E}_w[\|\partial \delta / \partial q\|_2^2]} (in pixels) yields:

SatelliteS(a)S(a)S(κ~)S(\tilde{\kappa})S(γ)S(\gamma) (Height)S(βx)S(\beta_x)S(βy)S(\beta_y)S(B)S(B) (Geometric)
Jilin-1 GF03D0.02240.02503.32982.89972.00722.2434
QuickBird0.02590.02883.84213.34582.31602.5886
WorldView-20.03660.04075.42884.72763.27253.6577
WorldView-30.05420.06048.05587.01534.85605.4276
Sentinel-20.00060.00070.08540.07450.04930.0532

Physical Finding: Across all five platforms, the height-related scale factor S(γ)S(\gamma) and geometric magnitude S(B)=∥β∥2S(B)=\|\beta\|_2 dominate the offset response by two orders of magnitude over the intrinsic shape terms S(a)S(a) and S(κ~)S(\tilde{\kappa}). Under bounded satellite viewing angles, the effective geometric degrees of freedom collapse onto an approximate 2D–2.5D manifold, explaining why d=2d=2 performs closely to d=5d=5 on constrained benchmarks.

2. SVD Spectral Distribution & Principal Component Truncation To verify whether the trained network internalizes this exact physical hierarchy, we performed SVD on the centered dd-dimensional offset features right before the 2D readout layer and evaluated inference accuracy when truncating minor components (projecting back to dd dimensions with all weights frozen):

Dim ddTop-1 Energy E1E_1 (%)Top-2 Energy E2E_2 (%)r99r_{99}Full mEPE (px)Top-1 Only mEPE (px)Top-2 Only mEPE (px)
280.90100.0022.82393.28682.8239
387.7099.99822.76753.54672.7665
494.9499.99122.75633.37432.7574
580.9399.99222.67233.42732.6745
1680.1399.97322.69463.30152.6958
25679.5199.96022.64173.29862.6819

Empirical Finding:

  • Spectral Concentration: Across all widths d>2d>2 (even up to d=256d=256), the top two principal components capture 99.960%–99.998% of the cumulative variance (r99=2r_{99}=2).
  • Truncated Inference: For our 5D model, zeroing out components 3–5 at inference time changes mEPE by only +0.0022 px+0.0022\text{ px} (2.6723→2.6745 px2.6723 \rightarrow 2.6745\text{ px}), whereas dropping the second component degrades mEPE sharply to 3.4273 px3.4273\text{ px}. Notably, the 5D model also achieves a lower full-feature mEPE (2.6723 px2.6723\text{ px}) than d=2d=2 (2.8239 px2.8239\text{ px}). This directly connects our physical factorization to the learned representation: the 2 dominant empirical modes correspond to the 2 dominant physical factors (γ,B\gamma, B), while 5D serves as the theoretically complete full-rank bound that preserves secondary intrinsic modulations (a,κ~a, \tilde{\kappa}) without extra computational overhead.
Reviewer_fYgpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LoDEOT for building footprint extraction from off-nadir imagery. It introduces a geometry-motivated 5D offset token to predict roof-to-footprint displacement, together with mask-guided feature aggregation and layer-wise denoising supervision. Experiments on five building datasets report improved footprint extraction accuracy over selected baselines, with ablation studies on token dimensions and denoising strategies.

Critical strengths

The paper derives a compact offset representation from the physical imaging geometry of off-nadir imagery and uses it to guide the design of offset tokens for building footprint extraction. This is an interesting and potentially valuable direction for integrating physical principles into deep learning.

Critical weaknesses
  1. The theoretical motivation for the 5D token is not convincingly connected to the learned representation, and experiments show no significant advantage over other token dimensions. Key components lack independent ablations, while the claimed efficiency is not supported by end-to-end computational measurements.

  2. Comparisons involve different architectures and training settings, making the reported improvements difficult to attribute to the proposed method. Qualitative results lack clear ground-truth comparisons and detailed boundary visualization, while quantitative evaluation insufficiently demonstrates geometric accuracy. These limitations substantially weaken the paper's conclusions.

  3. The manuscript is difficult to follow due to unclear explanations, redundant text, and confusing figures. In particular, the relationship between the geometric derivation and the actual network design is poorly illustrated. The experimental setup and comparisons are also insufficiently explained, making the paper difficult to understand and evaluate independently.

Questions for authors
  1. How does the derived 5D physical representation relate to the learned offset tokens, given that their channels are not explicitly constrained to represent physical parameters? What evidence supports choosing 5D when other dimensions achieve comparable or better performance?

  2. Can the authors provide clearer qualitative comparisons with ground-truth footprints, including enlarged examples and challenging cases? How accurately are footprint boundaries reconstructed beyond the reported detection-based metrics?

  3. What are the end-to-end parameter counts, FLOPs, and inference times compared with the baselines? How much of the reported accuracy improvement can be attributed specifically to the proposed offset representation rather than differences in backbone architecture and training?

8 comments
AuthorsOpenReview ↗

Meta Response to Reviewer fYgp

We sincerely thank the reviewer for the valuable and constructive feedback. Before addressing each question point-by-point, we would like to clarify several foundational concepts (outlined in Observation 1 of the Introduction) that form the core motivation of our paper and explain why we physically model the actual imaging geometry:

1. Implicit 3D Physical Modeling in Transformers: Recent advances in general computer vision (e.g., 3D vision and Video World Models) and extensive linear-probing studies have demonstrated that Transformer-based architectures naturally and implicitly parse and model underlying 3D imaging properties—such as camera parameters—during standard training. Crucially, this implicit physical representation emerges without requiring explicit supervision from ground-truth physical camera parameters. This phenomenon is explicitly established in the literature cited in Observation 1 (e.g., AC3D). To ensure this background is crystal clear to all readers especially for researchers only from remote sensing society, we have added a dedicated Related Work section elaborating on this mechanism.

2. Degrees-of-Freedom (DoF) Alignment Between Physical Geometry and Token Dimensions: Extracting building footprints from off-nadir imagery is inherently a 3D geometric problem. Let the explicit physical imaging process be formulated as δ=Φ(K,R,r,H)\delta = \Phi(K, R, r, H) and the network-level prediction be formulated as δ=Φ∗(Toffset)\delta = \Phi^*(T_{\text{offset}}). Because both functions model the exact same physical projection problem, the degrees of freedom (DoF) of the learned representation must satisfy the necessary condition:

rank(Toffset)≥rank(K,R,r,H)\text{rank}(T_{\text{offset}}) \ge \text{rank}(K, R, r, H)

Due to this physical coupling, the offset token ToffsetT_{\text{offset}} must be designed with sufficient dimensionality to satisfy the minimal necessary conditions required to recover the roof-to-footprint displacement δ\delta. This directly motivates binding the 3D imaging geometry to our token design and deriving a minimal, compact parameterization tailored to off-nadir remote sensing.


Clarification on Experimental Verification and Token Dimensionality

At the macro-experimental level, LoDEoT achieves state-of-the-art (SOTA) accuracy across five benchmarks while maintaining high computational efficiency.

To further verify the alignment between our theoretical derivation and the learned token dimensions, we conducted controlled ablation studies across multiple random seeds on a compact subset. Due to limited computing resources (4–6 RTX 3090 GPUs), running multi-seed experiments even on this subset required over a month of training. While the results on this smaller subset did not show a statistically significant margin over neighboring dimensions, we chose to report the findings transparently and analyzed the root cause in the Appendix: because the subset covers a restricted geographic/viewing distribution, global variations in camera parameters are insufficiently represented, causing the effective intrinsic dimensionality required by the model to collapse below 5 (specifically, reducing to an effective rank of 4). We further validated this dimensional collapse via Singular Value Decomposition (SVD) on the final-layer offset token ToffsetT_{\text{offset}}.

To make this theoretical and empirical connection even more rigorous in the revision, we have added physical verification calculations grounded in real-world satellite imaging parameters, directly confirming this conditional dimensionality reduction under constrained viewing geometries. For the table management, we rearrange them in terms of the metrics.

AuthorsOpenReview ↗

Response Q1.1

The official response to your comments are as follow:

Q1.1: How does the derived 5D physical representation relate to the learned offset tokens, given that their channels are not explicitly constrained to represent physical parameters?

As noted in our foundational clarification above regarding 3D vision (3DV) principles, our design is directly guided by three core observations presented in the Introduction:

  • Observation 1 (Implicit 3D Modeling): Transformer architectures naturally learn to parse and encode 3D imaging geometry implicitly during training, without requiring explicit channel-wise supervision or direct disentanglement into raw physical parameters.
  • Observation 2 (Compact Geometric Parameterization in Off-Nadir Remote Sensing): Due to the unique imaging characteristics of off-nadir satellite/aerial remote sensing, the complex 3D camera parameters and scene geometry can be analytically simplified into a minimal 5-dimensional representation.
  • Observation 3 (Layer-wise Denoising Dynamics): Deep Transformer decoders iteratively refine predictions through a layer-by-layer denoising process.

Connection Between the 5D Derivation and the Learned Offset Token: Rather than forcing each individual channel of ToffsetT_{\text{offset}} to explicitly correspond to a specific physical variable (e.g., pitch, roll, or building height), the 5D physical derivation serves as an information-theoretic and geometric bottleneck—establishing the minimal degrees of freedom (rank(Toffset)≥5\text{rank}(T_{\text{offset}}) \ge 5) required to span the roof-to-footprint displacement space under general off-nadir viewing conditions (combining Observations 1 & 2). Within this compact 5D latent subspace, the Transformer leverages its implicit 3D representation capability to encode a linear or nonlinear combination of the underlying physical factors necessary for accurate displacement regression. Meanwhile, Observation 3 directly motivates our layer-wise denoising supervision architecture, enabling the network to progressively refine this compact representation across decoder layers.

AuthorsOpenReview ↗

Response Q1.2

Q1.2: What evidence supports choosing 5D when other dimensions achieve comparable or better performance?

Response to Q1.2 (Part 1/2: Theoretical Analysis & Dimensional Regimes)

To begin with, we state the core conclusion directly: 5D is derived as a theoretically sufficient full-rank upper bound for general off-nadir geometry, rather than an empirically exclusive optimum on constrained subsets. Under practical remote sensing constraints, the effective intrinsic dimensionality collapses to approximately 2D–2.5D.

To clarify why 5D is principled and why neighboring dimensions exhibit comparable accuracy, we analyze the offset token dimension dd across three regimes (referencing Table 3 in the main text):

1. High-Dimensional Regime (d∈[256,+∞)d \in [256, +\infty)): Standard foundation models and prior baselines uniformly default to high-dimensional embeddings (d≥256d \ge 256). A primary contribution of our work is breaking this convention by proving that geometric offset regression can be accurately solved within a compact low-dimensional subspace.

2. Over-Parameterized Low-Dimensional Regime (d∈[5,256)d \in [5, 256)): As shown in Table 3, performance across d∈[5,12)d \in [5, 12) and d∈[12,256)d \in [12, 256) remains comparable without statistically significant variance. Once d≥5d \ge 5, the token already satisfies the full-rank degrees-of-freedom condition (rank(Toffset)≥5\text{rank}(T_{\text{offset}}) \ge 5). Increasing dd beyond 5 merely introduces redundant latent capacity without providing additional geometric information, hence yielding no substantial accuracy gain.

3. Sub-Rank Regime (d∈[2,5)d \in [2, 5)) — Conditional Dimensionality Reduction: Why do dimensions d∈[2,5)d \in [2, 5) still perform competitively on small training subsets?

  • Operational Bounds in Remote Sensing: Real-world datasets cover a finite set of satellite sensors, and off-nadir viewing angles are physically bounded within narrow operational ranges (e.g., θv∈[2∘,30∘]\theta_v \in [2^\circ, 30^\circ], far from 90∘90^\circ horizon grazing angles where sensors point toward space). Substituting real satellite parameters into our equations confirms that these physical constraints induce conditional dimensionality reduction, collapsing the effective degrees of freedom from 5D onto an approximate 2D–2.5D manifold.
  • 5D vs. 2.5D Relationship: 2.5D reflects the dataset-specific numerical approximation under constrained viewing geometries, whereas 5D represents the theoretically guaranteed full-rank upper bound required to preserve secondary intrinsic modulations (a,κ~a, \tilde{\kappa}) and ensure cross-domain generalization without extra computational cost.

(Quantitative verification tables and experiments added to the revised paper follow in Part 2/2 below.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.