acceptodds
Withdrawn from ICLR 2027

SAGS: Structure-Aware Geometry-Guided Stitching for Cost-Effective Glass Curtain Wall Inspection with Consumer UAVs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image stitching for glass curtain walls is critical for unmanned aerial vehicle (UAV)-based infrastructure inspection, yet remains challenging due to specular reflections, transmitted background interference, and repetitive window structures. These factors introduce unreliable appearance cues and periodic correspondence ambiguity, degrading existing feature-based and learning-based matching methods. In this paper, we identify periodic matching ambiguity as a key geometric challenge and derive a sufficient condition for reliable correspondence under bounded displacement uncertainty. Based on this principle, we propose Structure-Aware Geometry-Guided Stitching (SAGS), which exploits facade structures as geometric priors. SAGS employs a fine-tuned Segment Anything Model (SAM)-based structural extractor to obtain facade masks, dense-frame geometric calibration to estimate the scale factor and geometric priors, and mask-based cross-correlation for geometry-guided alignment. A drift-aware recalibration strategy further maintains consistency during long-range UAV inspection. Experiments on a real-world glass curtain wall dataset demonstrate that SAGS achieves reliable facade panorama stitching using consumer-grade UAVs without real-time kinematic (RTK) positioning or LiDAR sensors by leveraging velocity telemetry as motion priors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image stitching, Geometric priors, Periodic correspondence ambiguity, Structure-aware alignment, UAV inspection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MYRsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed SAGS (Structure Aware Geometry-guided Stitching), which align between segmented structural prior and frame geometric calibration for glass curtain wall inspection.

Critical strengths

The proposed methodology is easy to understand and outperforms the baselines, with qualitative results further supporting its effectiveness.

Critical weaknesses
  • Although the paper describe that SAGS leveraging a fine-tuned SAM3, the specific fine-tuning process is not provided. I think the methodological contribution is limited. Since the proposed method appears to simply apply SAM3 rather than learning representations specifically tailored to image stiching.
  • I have concerns about the Alignment Success Rate (ASR) metric. The authors should provide a detailed description of the blind visual assessment procedure and clarify what measures were taken to minimize subjectivity and ensure the objectivity and reliability of the evaluation.
  • The proposed method focus on a specific problem of curtain wall image stitching. Howerver, I believe that curtain wall image stitching represents a relatively narrow research scope. In particular, I have concerns that the stiching problem may be oversimplified in this setting, as SAM3 can produce relatively clear masks for curtain wall structures.
Questions for authors
  • Please provide quantitative and qualitative comparisons between the zero-shot(pretrained) SAM3 and the fine-tuned SAM3 for curtain wall segmentation. In particular, please clarify the fine-tuning procedure used to improve segmentation performace such as dataset, loss function, etc.
  • I believe that inaccurate segmentation prediction from SAM3 could negatively affect image stiching qulaity. Could the authors conduct a robustness analysis to evaluate segmentation quality affects (finetuned SAM3 vs zero-shot SAM3, or other segmentation method, etc.) stitching performance and report the results?
Reviewer_Fx4JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses image stitching for UAV based inspection of glass curtain walls, where specular reflections, transmitted backgrounds, and repetitive window layouts make correspondence unreliable. It formulates the latter issue as periodic correspondence ambiguity and derives a sufficient condition, E≤B<W/2E \le B < W/2, under which a search interval of radius BB contains the true displacement while excluding neighboring periodic alternatives, where WW is the structural period in pixels and EE bounds the displacement prediction error.

SAGS consists of four stages. First, a fine tuned SAM 3 model using the prompt “window frame” extracts facade structural masks. Second, dense frame calibration estimates a pixel per metre scale factor KK from mask correlation and integrated UAV velocity, with a first to last frame correction. Third, sparse frame stitching uses KK to convert telemetry displacement into an image displacement prediction and refines it through mask correlation within a local search interval. Fourth, structure aware seam placement uses dynamic programming with a cost that favors seams along structural regions, with λ=3.0\lambda=3.0.

Experiments use a real world dataset of 2,307 image pairs from 21 buildings with timestamp aligned flight logs recorded at 10 Hz. Of these, 174 pairs are used to fine tune SAM 3 and 2,133 for evaluation, with evaluation buildings excluded from fine tuning. Alignment Success Rate combines a 30 pixel structural correspondence check with a blinded visual assessment. SAGS achieves 90.76% overall ASR, compared with 57.81% to 61.98% for SIFT, LoFTR, UDIS++, and PixelStitch. The largest gains occur under strong reflection, with 91.21% versus 47.94% for the best baseline, and severe repetition, with 93.59% versus 59.13%. ASR drops to 69.15% without structural masks and 62.82% without motion guided search. Section 4.7 further provides a synthetic periodic mask experiment whose results are consistent with the proposed sufficient condition.

Critical strengths

1. A real, well characterized dataset with an interpretable challenge taxonomy. The paper evaluates on 2,307 synchronized image pairs from 21 buildings, with 174 pairs used for SAM 3 fine tuning and 2,133 for evaluation, while excluding evaluation buildings from fine tuning to prevent scene level leakage. The evaluation set is further annotated by normal facade, strong reflection, strong transmission, and severe repetition conditions, with challenge labels explicitly allowed to overlap. This design makes the aggregate result substantially more interpretable. The largest gains occur under strong reflection, where SAGS achieves 91.21% ASR versus 47.94% for the best baseline, and severe repetition, where it achieves 93.59% versus 59.13%. In contrast, the gain on normal facades is small, at 87.64% versus 86.65% for LoFTR. This pattern is consistent with the proposed mechanism, since the largest improvements occur on two central failure modes that SAGS explicitly targets. More broadly, the challenge taxonomy and condition specific reporting provide a useful evaluation framework for facade inspection.

2. A simple dense to sparse calibration strategy, supported by unusually explicit limitations. SAGS estimates a single pixel per metre scale factor for each sequence from a densely sampled calibration segment and reuses it for subsequent sparse frame pairs. This is a simple and well motivated way to convert consumer grade UAV velocity telemetry into a useful motion prior without requiring RTK or LiDAR. The implementation is also described concretely: dense and sparse sampling intervals are typically 0.30 m and 2.5 m, respectively, the physical search tolerance is 0.45 m, stitching is performed at 3840 × 2160 resolution, and flight logs are recorded at 10 Hz. The SAM 3 fine tuning configuration further specifies a frozen text encoder, AdamW optimization, and 50 training epochs. These details make the calibration and stitching pipeline comparatively easy to reproduce and adapt.

A further strength is that the paper states the main geometric assumption explicitly. Reusing the calibrated scale factor assumes approximately stable camera orientation and camera to facade distance within a calibration segment. Section 4.8 then directly discusses the associated failure modes, including substantial viewpoint or distance changes, curved or irregular facades, and the lack of validation on other facade materials. This transparency is important because it clearly delineates where the proposed geometric prior is expected to remain reliable and where more flexible calibration or alignment models would be required.

Critical weaknesses

1. The sufficient condition is not instantiated on the real data, leaving the theoretical argument weakly connected to the reported ASR.

Eq. (1), (E \le B < W/2), is presented as a central conceptual contribution and as the justification for using imperfect UAV motion priors. However, the paper does not report the quantities needed to verify whether this condition actually holds on the evaluation set.

First, no numerical values of the structural period (W) are reported. Section 4.3 states that the 30 pixel ASR tolerance is smaller than the repetition periods in the evaluated images, but these periods are not quantified. Consequently, it is not possible to verify either this statement or whether the actual search radius satisfies (B < W/2).

Second, the displacement prediction error is not characterized. Section 3.3 explicitly states that SAGS uses a prescribed search tolerance without estimating (W) or (E) online. The paper therefore never reports the empirical distribution of (|d_{gt}-\hat d|), even though this residual error is directly relevant to whether (E \le B) is plausible on the real data.

Third, the mapping of the search radius across units is underspecified. In Eqs. (8) and (9), (B) is expressed in the image displacement domain, whereas Table 3 reports a physical search tolerance of 0.45 m. Since the calibrated factor (K) provides a pixel per metre conversion, this is conceptually resolvable, but the paper does not explicitly state how the reported 0.45 m tolerance is converted into the (B) used by the correlation search or report its value in pixels.

Finally, Section 4.7 provides only a synthetic consistency check. The simulation varies (W/(2B)) and (E/B), but the distribution used to generate motion prior errors is not specified. Figure 8 also shows substantial recovery outside the sufficient region, which is not inconsistent with a sufficient condition, since violating the condition need not cause failure. However, it means that the figure does not establish the tightness of the condition or demonstrate that it holds on the real dataset. Its interpretation depends on how errors are sampled within the stated bound.

This issue is consequential because the condition is presented as a key explanation for why motion constrained matching resolves periodic ambiguity. Reporting the empirical window periods, the residual displacement error distribution, and the effective search radius in pixels would directly connect the theoretical condition to the real experiments.

2. The headline comparison is confounded by unequal inputs, while the ASR metric does not expose its objective and subjective components separately.

SAGS receives information unavailable to the four headline baselines. Section 4.4 explicitly states that SAGS additionally uses UAV velocity telemetry, dense frame calibration, and a facade specific fine tuned SAM 3 extractor. The paper does provide useful internal ablations: appearance based correlation retains the same motion guided search, and global mask correlation removes the motion restriction. These experiments show that both structural masks and motion priors contribute within SAGS. However, they do not provide a controlled comparison in which a standard baseline such as SIFT or LoFTR is given the same calibrated motion prior. Such a control would help determine how much of the improvement over conventional baselines comes from the proposed structural alignment design and how much comes from access to additional sensing and calibration information.

The evaluation protocol introduces a second concern. ASR is the conjunction of a reproducible structural test and a blinded visual assessment, but only the final combined success rate is reported. The manuscript does not report the number of pairs rejected at each stage for each method, the number or identity of visual assessors, the assessment procedure beyond hiding method identities, or any inter assessor agreement statistic. It also does not report how many outputs were counted as failures because they were missing or because annotated points could not be mapped. In addition, the number of annotated correspondences per pair, the annotation procedure, and the meaning of the stated “common evaluation scale” are not specified.

These omissions make the 30 pixel criterion difficult to reproduce precisely and make it impossible to determine how much of each method's ASR is determined by the structural threshold versus the visual assessment. The visual criterion also explicitly targets distortions such as bent or discontinuous window frames, so additional protocol details are needed to establish that it treats translation based and flexible warp based methods comparably. Finally, although the paper motivates the fixed 30 pixel threshold as smaller than the repetition period, no sensitivity analysis is reported.

Because the overall ASR gap is the paper's primary empirical evidence, these issues affect both attribution and reproducibility. Reporting the two ASR stages separately and adding a standard matcher equipped with the same motion constrained search would substantially strengthen the comparison without changing the proposed method.

Questions for authors
  1. Please report the empirical quantities required to validate the uniqueness condition in Eq. (1) on the 2,133 evaluation pairs. In particular, please provide the distribution of the structural period WW in pixels and, where possible, in metres, together with the empirical distribution of the displacement prediction error ∣dgt−d^∣\lvert d_{\mathrm{gt}}-\hat d\rvert and the corresponding choice of EE. Please also report the actual search radius BB in pixels. Table 3 specifies a physical search radius of 0.45 m, whereas Eqs. (8) and (9) operate in pixel coordinates, so the conversion from 0.45 m to the pixel domain should be stated explicitly, for example through Bpx=0.45KB_{\mathrm{px}}=0.45K when KK is expressed in px/m. Using these measurements, please report how often E≤B<W/2E \le B < W/2 is actually satisfied on the real evaluation data. Finally, since KK is calibrated once per sequence and subsequently reused, please report its variation across sequences and its stability within each sequence.
  2. Please decompose the reported ASR for SIFT, LoFTR, UDIS++, PixelStitch, and SAGS into the following mutually exclusive outcomes: pairs that fail the 30 pixel structural check, pairs that pass the structural check but are rejected by the blinded visual assessment, and pairs counted as failures because the output is missing or the annotated correspondences cannot be mapped. In addition, please define the “common evaluation scale” at which the 30 pixel threshold is applied, including its image resolution and coordinate convention. For methods whose native output coordinate system or stitching canvas differs from that of the input images, please describe the exact transformation used to map the annotated points and the resulting outputs into this common evaluation coordinate system.
  3. Please specify the exact input resolution and all resizing or rescaling operations used for each baseline. The source videos are recorded at 3840×21603840\times2160, and Table 3 lists the SAGS stitching resolution as 3840×21603840\times2160, but Section 4.4 does not state whether SIFT, LoFTR, UDIS++, and PixelStitch operate directly at this resolution or at method specific inference resolutions. This information is important because any subsequent coordinate rescaling also affects the interpretation of the 30 pixel evaluation threshold. In addition, the SIFT and LoFTR baselines use only RANSAC based global homography estimation. Since the paper itself discusses more flexible alignment models such as APAP and robust elastic warping, please consider evaluating a stronger matching based pipeline, such as LoFTR followed by APAP or elastic refinement. A LoFTR variant constrained by the same calibrated motion prior would also be valuable for separating the benefit of motion restriction from that of the structural mask representation.
  4. Please provide substantially more detail on the SAM 3 fine tuning data and the resulting structural extractor. How were the target masks for the 174 fine tuning pairs obtained, for example through manual annotation, model assisted annotation, or automatic generation, and how many buildings contributed to this subset? Please report segmentation accuracy on held out evaluation buildings and compare the fine tuned extractor with the original SAM 3 model using an appropriate mask metric such as IoU or F1. More importantly, because the theoretical uniqueness condition does not guarantee correct peak selection in the presence of mask errors, please quantify how alignment reliability depends on mask quality. For example, ASR could be stratified by mask IoU or evaluated under controlled mask perturbations and different search radii.
  5. Please report ASR at the building level, and preferably also at the sequence level, together with uncertainty estimates that account for clustering, such as 95% confidence intervals obtained by cluster bootstrap over buildings or sequences. Although the evaluation contains 2,133 image pairs, these pairs are drawn from only 21 buildings and are therefore not necessarily independent observations. This is particularly important for the Normal Facade subset, where SAGS achieves 87.64% ASR compared with 86.65% for LoFTR, a difference of only 0.99 percentage points. Reporting only pair aggregated ASR makes it difficult to determine whether this margin is consistent across buildings or is driven by a small number of sequences.
  6. Does the search radius BB need to adapt to the facade periodicity WW? Table 3 uses a fixed physical search radius of 0.45 m, whereas the uniqueness condition requires B<W/2B<W/2. Because window pitch varies across buildings, a fixed physical tolerance can approach or violate this condition on facades with sufficiently small structural periods. Please report the distribution of WW in physical units and the resulting margin W/2−BW/2-B across the evaluation data, together with the fraction of pairs or sequences for which the condition is violated. A real data sensitivity analysis of ASR as a function of BB, preferably also expressed through the normalized ratio 2B/W2B/W, would directly test how robust the method is to this design choice and whether an adaptive BB is necessary.
Reviewer_XbYdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper deals with UAV-based image stitching of glass curtain walls. The difficulty of this task is due to multiple factors such as specular reflections, repetitive patterns and partly transmitted interior content. There are a number of strong assumptions being adopted (planar surfaces, 1D translational motion, constant orientation, constant distance to objects) which are generally reasonable for this specific application, but which decrease significantly the potential transfer of the ideas in the paper to other tasks. A new dataset which contains 2,307 pairs from 21 buildings is proposed as well. Against SIFT, LoFTR, UDIS++ and PixelStitch, SAGS reports 90.76% Alignment Success Rate (ASR) versus 57.81–61.98% for the baselines.

Critical strengths
  • the applicative context is clear (although narrow), and the approach of avoiding to rely on RTK positioning and LIDAR makes sense

  • the ablations in Table 2 are useful to demonstrate that masking and windowing helps

  • the dataset would be useful for the community, and its collection is not trivial

Critical weaknesses
  • the applicative context is clear (although narrow), and the approach of  avoiding to rely on RTK positioning and LIDAR makes sense- the ablations in Table 2 are useful to demonstrate that masking and windowing helps- the dataset would be useful for the community, and its collection is not trivial

  • a large chunk of related works relying on telemetry-guided and pose prior guided matching is ignored, and they should be present in the related works and as baselines (e.g. "On-board GNSS/IMU assisted feature extraction and matching for oblique UAV images", "A multi-sensor fusion approach for rapid orthoimage generation in large-scale uav mapping"); there is also an entire community around repetition-aware matching on facades (e.g. "Doppelgangers: Learning to Disambiguate Images of Similar Structure")

  • the presented method receives three advantages that make the comparison with the selected baselines largely irrelevant : a strong telemetry prior that decreases significantly the hypothesis space, the 1-DOF model, and the in-domain adaptation of SAM 3 (the baselines run with pretrained weights)- the cost-effective assertion is not quantified, and the runtimes are not reported; in order to discuss fairly the cost effectiveness, the hardware cost should be discussed against RTK/LIDAR or photogrammetry solutions. the processing is offline on a pretty beefy machine; so the "consumer" aspect does not imply cheap or fast

  • the ML contribution is very thin for a conference focused on learning representations. The only learning step is fine-tuning SAM 3 on 174 pairs with a single text prompt. The authors do not show that fine-tuning was necessary (this requires just a zero-shot SAM 3 ablation and it would be an easy, useful addition). Overall, the justification for sending this to ICLR is nowhere to be found. Arguably, the most valuable contribution is the dataset, and that hints either to a dataset track, or even better to an applications track of an applied conference such as WACV

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.