acceptodds
Withdrawn from ICLR 2027

PAGER: Partial-to-global Alignment via Geometric and Relational Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Pretrained 3D encoders are typically developed on globally reconstructed scenes expressed in a consistent world coordinate frame, whereas embodied systems must reason from partial, viewpoint-dependent observations in camera coordinates. We show that this shift from globally learned 3D feature spaces to realistic partial observations exposes a severe representation mismatch, which we find consistently across representative state-of-the-art encoders, including Sonata and Concerto. A frozen Sonata encoder with a global linear probe achieves 72.47 mIoU on full ScanNet scenes, but only 2.57 mIoU on single-frame camera-coordinate inputs. Training-free gravity alignment recovers performance to 41.64 mIoU, showing that coordinate-frame mismatch is a dominant source of degradation but cannot be fully resolved through canonicalization alone. We introduce PAGER, a label-free adaptation method that aligns partial-view features with a frozen global 3D semantic space using only paired partial/global geometry. It learns lightweight adaptation modules while keeping the pretrained encoder and global segmentation probe frozen. Matched-point feature alignment anchors partial features to their global counterparts, while relational supervision preserves their similarity structure with respect to the global representation. Global geometry provides supervision only during training. Inference operates directly on the partial observation. Without partial-view labels, PAGER outperforms label-supervised PEFT on both Sonata and Concerto, and in zero-shot ScanNet->ScanNet++ transfer surpasses fully fine-tuned Sonata (53.93 vs. 48.09 mIoU), suggesting that preserving the frozen global representation can improve cross-dataset transfer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Understanding, 3D Domain Adaptation, 3D Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_upKWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PAGER to study why 3D encoders pretrained on full scenes fail on camera-frame partial point clouds. The authors show that the coordinate frame, not missing geometry, causes most of the collapse. The PAGER method trains a lightweight adapter on top of the frozen encoder and probe and needs no partial-view labels. Instead, it distills from matched global-scene features using a point-wise alignment loss and a relational similarity loss. PAGER outperforms supervised Parameter-efficient fine-tuning (PEFT) methods on ScanNet and surpasses full fine-tuning baselines.

Critical strengths
  • The Partial Semantic Segmentation is a timely problem. The gap between scene-level pretraining and single-view, camera-frame deployment is real and increasingly relevant.
  • The paper provides a clean diagnostic. The Camera-Global condition isolates the coordinate-frame effect from missing geometry.
  • The develop method is simple and backbone-agnostic. The encoder and probe stay frozen with a small adapter. And the same recipe works on Sonata and Concerto.
Critical weaknesses
  •  Some key baselines are missing. Frozen Sonata already scores 67.45 on world-frame single frames, exceeding PAGER’s 65.46. An oracle or IMU-based gravity-alignment control is therefore essential to isolate PAGER’s contribution. The evaluation should also include logit-level distillation from the global probe and the same adapter trained with cross-entropy through the frozen probe.
  •  The cross-dataset protocol is questionable. To my knowledge, Sonata’s pretraining includes ScanNet++, which makes the “zero-shot” description questionable. The full fine-tuning baseline may also be undertuned. In Table 8, the reported mIoU averages over three classes absent from ScanNet++, making the comparison harder to interpret.
  • Limited novelty of the objective. The relational loss resembles established anchor-similarity distillation (SEED, ICLR 2021) and multi-frame-to-single-frame point-cloud distillation (SCPNet, CVPR 2023), neither of which is cited. The adapter is also drawn from GEM. The main contribution appears to be the problem setting and its per-point, correspondence-based implementation.
  • The "Label-free" claim is overstated. The frozen probe is trained with labels from the same global ScanNet scenes. Section A.3 also shows that partial-view labels can be derived through the correspondences PAGER already requires. The more meaningful unlabeled training setting on unlabeled and posed RGB-D data is not evaluated.
  • Overbroad performance claims. The statement that PAGER “outperforms supervised PEFT on both backbones” (line 364) holds only on ScanNet-20. With Sonata, GEM performs better on ScanNet++ (33.27 vs. 27.38) and ScanNet200 (21.97 vs. 19.25). PAGER also performs poorly in several small classes. For example, its smoke-detector score is 0.25 versus GEM’s 47.56.
Questions for authors

Please see the weakness part

4 comments
AuthorsOpenReview ↗

Response to Reviewer upKW (1/2)

We thank the reviewer for the constructive feedback and specific suggestions. We address each point below, including additional experiments.

W1 (missing baselines). We evaluated gravity alignment (tables in the general response). With ground-truth gravity, the frozen model reaches 66.49 mIoU and remains stable up to σ = 5° of tilt noise (66.21), but degrades at larger errors (61.52 at 10°). RGB-estimated gravity yields 41.64. Without pose or gravity, PAGER achieves 65.46 (KL) and 67.27 (Gram, App. A.4), the latter exceeding the frozen model with perfect gravity and approaching the ground-truth-pose oracle (67.45). Combining PAGER-KL with gravity alignment further improves performance to 68.27, indicating benefits beyond frame correction alone (see our response to iWKC, W2 and Q1).

We use feature rather than logit distillation to keep adaptation independent of the semantic taxonomy. Logit distillation would depend on a particular pretrained probe, whereas feature distillation allows the same adapter to serve both ScanNet-20 and ScanNet200 probes. Cross-entropy supervision would require partial-view labels, as in GEM. We are also evaluating GEM on gravity-aligned clouds.

W2 (cross-dataset protocol). Sonata's self-supervised pretraining includes ScanNet++ scenes, including validation scenes, as unlabeled complete reconstructions. This applies equally to all compared methods, which share the same pretrained encoder. However, this exposure alone does not yield effective partial-view predictions: the frozen encoder reaches only 0.69 mIoU on ScanNet++. We will replace "zero-shot" with the more precise "cross-dataset transfer without ScanNet++ labels or adaptation."

All baselines, including full fine-tuning, use the same 100-epoch training protocol (App. B.2). Our ~19.6K partial-view training frames yield approximately twice as many sample presentations as Sonata's official 800-epoch schedule on 1,201 ScanNet scenes, although the training samples and protocols differ. Full fine-tuning achieves the highest in-domain performance (69.26 mIoU), indicating effective optimization on the source domain, but generalizes less effectively to ScanNet++. We will document the differences from Sonata's official fine-tuning recipe in App. B.2.

For class mapping, we follow the ScanNet++ benchmark mapping and ScanNet v2 category table. The 26 ScanNet++ categories map to 17 ScanNet-20 classes: 11 directly and six through merges (wall, cabinet, chair, door, picture, otherfurniture). Unmapped categories are ignored. Desk, shower curtain, and bathtub have no corresponding ScanNet++ points and score zero for all methods, capping the reported 20-class mIoU at 85%. Evaluating only the 17 present classes preserves the ranking: Sonata (+ linear) 31.07, Adapter 36.11, Sonata (late blocks + linear) 40.17, GEM 54.23, Sonata (full + linear) 56.58, and PAGER 63.44.

AuthorsOpenReview ↗

Response to Reviewer upKW (2/2)

W3 (novelty). Our contribution consists of a finding and an adaptation method. First, we show that coordinate-frame changes account for substantially more degradation in frozen 3D encoders under partial observations than missing geometry alone (−67 vs. −5 mIoU, Sec. 3). Second, we introduce self-supervised, correspondence-based adaptation that approaches oracle-pose performance without pose, labels, or backbone/probe modification (PAGER-Gram 67.27 vs. 67.45). With gravity alignment, PAGER-KL further exceeds the frozen pose oracle (68.27 vs. 67.45).

We acknowledge the connections to prior distillation methods. SEED matches image-level similarity distributions for model compression. SCPNet distills pairwise voxel similarities from a supervised multi-frame LiDAR teacher into a single-frame student for semantic scene completion. PAGER instead relates observed points to keys sampled across the complete scene, including unobserved regions, using the same frozen encoder to establish relational targets across coordinate frames. Its adaptation requires no semantic labels or backbone retraining. The relational loss improves mIoU by +2.15 (KL) and +3.96 (Gram) over pointwise alignment, and random global keys outperform nearest-neighbor keys by 15.75 mIoU. We will expand the related work to include also SEED and SCPNet.

W4 (label-free). We will use "self-supervised adaptation" to distinguish the adaptation stage from supervised probe training. Adaptation uses only feature-based losses from a frozen self-supervised encoder, without semantic labels or probe predictions. The linear probe is trained separately on labeled complete scenes and used only for downstream readout. This follows the standard evaluation setting for self-supervised 3D encoders. Although ScanNet allows partial-view labels to be derived from annotated meshes, acquiring such annotations for new environments requires additional effort. Adaptation on never-annotated geometry would test this setting directly, and we will discuss it as a limitation.

W5 (scope of performance claims). We will clarify that L364 refers specifically to ScanNet-20. On finer taxonomies, results depend on the backbone. With Concerto, PAGER outperforms GEM on all three benchmarks (71.67 / 33.97 / 26.73 vs. 69.90 / 32.86 / 25.06). With Sonata, GEM performs better on ScanNet++ and ScanNet200. The training requirements also differ: GEM uses task-specific semantic labels, while PAGER reuses the same adapter trained on ScanNet without 200-class supervision, together with the frozen 200-class probe.

Tables 9–10 report the ten largest class-wise differences in each direction. Across all 200 ScanNet200 classes, PAGER outperforms GEM on 66, ties on 24 (mostly rare classes with zero scores), and underperforms on 110, consistent with GEM's higher overall mIoU.

We thank the reviewer again for the suggestions, which have helped clarify the scope and contributions of the work. We hope the additional results address the main concerns and would appreciate reconsideration of the score.

Reviewer_upKWOpenReview ↗

Official Comments by Reviewer upKW

Thank you for the detailed response and additional experiments. The gravity results and the recomputed 17-class results address my questions about the oracle control and Table 8. But they also strengthen my main concern. With ground-truth gravity, the frozen model scores 66.49, above the main-table PAGER-KL result of 65.46. This suggests that the pose-free gain is most relevant to platforms without an IMU while the abstract’s claim that canonicalization cannot close the gap is no longer supported. The additional gain over frame correction (+1.78 with gravity) is demonstrated only relative to the frozen model. Controlled comparisons with GEM, the same adapter trained with cross-entropy on gravity-aligned clouds, and logit distillation are still missing, as is the never-labelled adaptation experiment. Thus I would keep my rating unchanged.

Thank you for the follow-up and for confirming that the oracle control and Table 8 are resolved.

The new results separate the two sources of degradation identified in Sec. 3: the coordinate frame, the dominant one (−67 mIoU), and partial observation, the smaller one (−5 mIoU). Frame correction addresses only the first. Even the full ground-truth pose, which restores the exact frame the encoder was pretrained in, reaches 67.45, while complete scenes reach 72.47: no frame correction can recover what the partial view does not contain. Frame correction is therefore a step toward global performance but cannot reach it, consistent with the abstract.

For the dominant source, accurate gravity recovers most of the frame gap for the frozen model (66.49), while gravity estimated from images does not (41.64). This relies on a property of the benchmark rather than of the task: ScanNet scenes are upright, and the encoder was pretrained on gravity-aligned reconstructions. It also requires an accurate gravity sensor aligned with the camera. A scene tilted relative to gravity, such as a ramp or a vehicle interior, acts like a gravity error, and with a 10° error the frozen model drops to 61.52. Ground-truth gravity is thus effectively an upper bound for the gravity-alignment baseline. Without any gravity, PAGER reaches this level: 65.46 with the KL objective and 67.27 with the Gram objective, which exceeds the best case of the gravity-alignment baseline.

For the remaining source, PAGER distills what the partial view lacks from the complete scene during training: with the same gravity alignment, it improves the frozen model from 66.49 to 68.27, above the ground-truth-pose oracle and thus beyond what any frame correction can reach. We therefore respectfully disagree that PAGER's relevance is limited to platforms without an IMU: without a gravity sensor, PAGER replaces it; with one, it addresses the part of the gap that frame correction leaves. Such a sensor is also unavailable for point clouds reconstructed from images, uncalibrated add-on cameras or archival data.

Regarding the suggested controls, we note what they would and would not isolate. Logits are a linear projection of the features PAGER already matches, so feature matching is the stronger constraint; logit distillation would additionally bring the labelled probe into training and tie the adapter to one label set. In a closely related dense-to-sparse 3D setting, SCPNet (Table 4), which the reviewer cites, finds that logit distillation lowers performance below no distillation (33.8 vs. 34.4), while relational distillation improves it (37.2). A cross-entropy adapter on gravity-aligned clouds would not isolate distillation either: on this benchmark, partial-view labels are copied from annotations of the complete reconstruction, so such an adapter also receives complete-scene information, through labels. Finally, since PAGER's adapter reads neither labels nor the probe, it is identical whether or not the adaptation scenes were ever annotated.

In summary: gravity alignment addresses the dominant, frame-related source of degradation but cannot recover the gap due to partial observation. PAGER improves over the frozen model both without gravity and with it.

Reviewer_iWKCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks why 3D encoders pretrained on complete, world-frame scans fail on the partial, camera-frame clouds that an embodied agent actually sees. With a frozen Sonata encoder and its global linear probe, ScanNet mIoU drops only from 72.47 to 67.45 when a full scene is replaced by one frame in world coordinates, but to 5.14 when that same complete scene is expressed in camera coordinates, and to 2.57 for a real single frame. Gravity alignment recovers 41.64. PAGER then trains a GEM-style spatial adapter (1.3M parameters) against a frozen encoder and a frozen probe. Paired partial/global geometry supplies a matched-point cosine loss and a KL match of similarities to random global keys. Inference uses only the partial cloud. On ScanNet-20, PAGER reaches 65.46 mIoU with Sonata and 71.67 with Concerto, above supervised GEM (59.81 and 69.90) and close to fully fine-tuned Sonata (69.26). Trained on ScanNet and evaluated on ScanNet++ mapped to 20 classes, it reaches 53.93, above fully fine-tuned Sonata (48.09).

Critical strengths

1.Section 3 isolates the failure mode cleanly. Completeness (global, room slice, view crop, single frame) and coordinate frame (world vs. camera) are varied separately, with the encoder and probe held fixed. Changing the frame of a complete scene costs about 67 mIoU; removing geometry inside a fixed world frame costs about 5. PCA and gravity alignment, at 8.38 and 41.64, show that a training-free canonical frame does not restore the frozen representation. This measurement is what makes the adaptation problem concrete, and it is the result I trust most.

2.Distilling that frozen representation, rather than updating it, is a reasonable response, and the transfer result tests the right consequence. PAGER does not touch the Sonata weights, yet under the ScanNet to ScanNet++ shift it beats source-domain full fine-tuning (53.93 vs. 48.09), which falls from its in-domain 69.26. The same adapter and losses also improve Concerto, so the gain is not one checkpoint. The loss ablation is informative: cosine alignment alone reaches 63.31, and the relational term adds 2.15.

Critical weaknesses

1.The headline comparison does not isolate “no labels.” PAGER is trained to match features of the complete scene at corresponding points. Those features come from a pretrained encoder, and the frozen probe that reads them was fit with semantic labels on global scans. Appendix C builds the partial-view labels used by GEM and full fine-tuning by copying mesh labels within 0.01 m, so on this benchmark those labels are a byproduct of the same reconstructions and poses PAGER already uses. Cosine matching to the teacher accounts for nearly all of the ScanNet-20 gain over GEM (63.31 vs. 59.81; both losses 65.46). The teacher sees full-scene context that the supervised baselines never receive, so a higher mIoU does not show that partial labels are unnecessary. It shows that full-scene feature targets are a strong signal. The claim is also not uniform: with Sonata, supervised GEM is ahead on ScanNet++ (33.27 vs. 27.38) and ScanNet200 (21.97 vs. 19.25). The abstract does not say this. The narrower claim that still holds is that matching frozen global features transfers better than partial-view cross-entropy. That claim is worth publishing only if it is stated at that scope.

2.The paper’s own evidence says the dominant nuisance is the coordinate frame, but no control learns that frame. Single-frame clouds transformed into world coordinates with ground-truth pose score 67.45; gravity alignment scores 41.64; PAGER scores 65.46. PAGER may be recovering an implicit upright frame, repairing relational features, or both. Sonata is built on an absolute-coordinate backbone, so a large drop under a rigid change of frame is the expected behavior of that architecture. The missing comparisons are the same adapter trained with cross-entropy after gravity alignment, a small rotation or gravity head in front of the frozen model, and training under camera-frame rotations of world-coordinate clouds. Without one of these, Section 4 does not yet show that geometric and relational distillation is what closes the gap. The in-the-wild figures do not fill this hole: they have no ground truth, and every quantitative number is from ScanNet-style RGB-D with poses and meshes used to form training pairs.

Questions for authors

1.After gravity-aligning the partial cloud, how much does PAGER still improve over the frozen probe, and over a GEM adapter trained with cross-entropy on those aligned clouds? If most of the 41.64 to 65.46 gap remains, the representation-mismatch claim is substantially stronger. If the gap collapses, the method is mostly learned canonicalization and the claim should be restated.

2.Please define the ScanNet++ to ScanNet-20 mapping behind Table 3: which classes are merged, dropped, or ignored in the mIoU. Please also report mean and standard deviation over at least three seeds for PAGER, GEM, and full fine-tuning on ScanNet-20 and on that transfer. The Concerto margins over GEM are about 1–2 mIoU, and a single run is not enough to judge them.

3.In which training regime is paired partial/global geometry available while partial semantic labels are not? If mesh projection makes those labels cheap whenever PAGER can be trained, the label-free claim needs a different experiment: adaptation on reconstructions that were never semantically labeled, using a probe trained on a separate source, or a supervised baseline given the same global correspondences.

1 comment

We thank the reviewer for a precise and constructive review, and for the explicit criterion in Q1, which we used to structure the new experiments.

W1 (scope of "no labels"). PAGER's adaptation sees no labels of any kind: its losses compare features only, its targets come from a self-supervised encoder (Sonata, Concerto), and the labelled linear probe, the standard read-out for self-supervised 3D encoders, is used only to read out predictions after adaptation. GEM and full fine-tuning instead require labels for every partial view they are trained on. We agree that on ScanNet these partial-view labels are cheap, because they are copied from hand-annotated meshes, and we adopt the reviewer's scope: matching frozen complete-scene features is a stronger and more transferable signal than partial-view cross-entropy. A1 supports this (cosine alone 63.31 vs. GEM 59.81). Consistent with this, PAGER transfers better (ScanNet→ScanNet++, 17 classes: 63.44 vs. 54.23 for GEM and 56.58 for full fine-tuning, see Q2). Because PAGER reads out through the frozen probe, it is limited by that probe, whereas GEM retrains its head on partial-view labels. With Sonata, GEM therefore leads on the fine-grained benchmarks, ScanNet++ (33.27 vs. 27.38) and ScanNet200 (21.97 vs. 19.25). With the stronger Concerto representation, PAGER leads GEM on all three benchmarks. We will state this scope and these results in the abstract. A practical consequence of feature-level targets is that one adapted encoder serves both the ScanNet-20 and ScanNet200 probes without retraining.

W2 / Q1 (frame control). We ran the requested control. With ground-truth gravity the frozen model reaches 66.49, with gravity estimated from the image, 41.64. By the reviewer's criterion, the gap remains fully with estimated gravity (41.64 vs. 65.46 for PAGER without any sensor), while with perfect gravity it largely closes for the frozen model. We will therefore make Sec. 3 more precise: for the frozen model, most of the coordinate-frame gap closes once the vertical axis is known exactly, but not when it must be estimated from the image. We agree that a large drop is expected for an encoder that takes absolute coordinates and normals as input.

PAGER goes beyond this canonicalization in two ways. Without a sensor, it learns it from data: PAGER-Gram reaches 67.27, above the frozen model with perfect gravity. This also bounds the suggested gravity or rotation head in front of the frozen model, which could at best match perfect gravity (66.49). Given the same gravity, PAGER-KL improves from 66.49 to 68.27, above the ground-truth-pose oracle (67.45), recovering about 30% of the remaining gap due to partial observation.

Q2 (mapping, seeds). We use ScanNet++'s benchmark mapping and the ScanNet v2 category table: 26 ScanNet++ classes map to 17 ScanNet-20 classes (11 one-to-one; merges: wall + shower wall, cabinet + kitchen cabinet, chair + office chair, door + doorframe, picture + poster + painting, otherfurniture = heater, trash can, crate, rack). All other classes are ignored. Desk, shower curtain and bathtub have no ScanNet++ points; Table 3 previously counted them as zero, scaling every method by 17/20 and leaving the ranking unchanged. Over the 17 present classes: Sonata (+ linear) 31.07, Adapter 36.11, Sonata (late blocks + linear) 40.17, GEM 54.23, Sonata (full + linear) 56.58, PAGER 63.44

Retraining every method over multiple seeds did not fit in the rebuttal period, but our ablations provide independent training runs. On Concerto, PAGER with two different relational objectives reaches 71.67 (KL) and 74.14 (Gram), both above GEM (69.90). On Sonata, all five configurations we trained (cosine only, KL with M = 2048, 4096, 8192, Gram; Tables 4 and 11) exceed GEM by 3.5–7.5 mIoU. Mean and standard deviation over three seeds for PAGER, GEM and full fine-tuning will be in the camera-ready.

Q3 (when are labels unavailable?). Partial-view labels are cheap in ScanNet only because its meshes were annotated by hand, which is the expensive step in any new environment. Posed RGB-D, by contrast, is produced routinely without annotation by RGB-D SLAM systems and capture apps. PAGER needs only such geometry and a probe trained once on any labelled dataset, and the adapter is not tied to that probe's taxonomy. The setting the reviewer proposes, adaptation on never-labelled reconstructions with a probe from a separate source, is the direct test of this, and we will add it.

We hope these results address your concerns and would be grateful if you would consider updating your score.

Reviewer_KKQ8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the performance drop that occurs when 3D encoders trained on complete, world-coordinate scenes are applied to single-frame point clouds in camera coordinates. PAGER keeps the pretrained encoder and semantic head frozen and trains lightweight spatial adapters using paired partial/global geometry. Its loss combines point-wise feature alignment with relational distillation against features sampled from the complete scene.

Critical strengths
  • The diagnostic experiment is clear and useful. It separates missing geometry from coordinate-frame mismatch and shows that the latter causes most of the performance collapse.
  • PAGER is lightweight and does not require semantic labels for partial-view adaptation. It obtains strong ScanNet results with only 1.3M trainable parameters and works with both Sonata and Concerto.
  • The method is simple at inference time: it only needs the current camera-coordinate point cloud, without the complete scene or camera pose.
Critical weaknesses
  1. The embodied motivation is overstated. PAGER processes each frame independently, without pose, temporal memory, or an accumulated map, while practical embodied systems often estimate motion and fuse observations over time.

  2. The need for PAGER depends on the pose-free assumption. Transforming a frame to world coordinates with ground-truth pose already achieves 67.45 mIoU, above PAGER's 65.46. Comparisons with estimated or noisy poses are missing.

  3. The technical novelty is limited. The method mainly combines an existing spatial adapter with point-wise feature matching and relational distillation. The relational loss adds only 2.15 mIoU over point-wise alignment, and closely related work such as M2SKD [1] and C2P [2] is not discussed.

  4. The generalization evidence is narrow. The only quantitative cross-domain test is ScanNet to ScanNet++, which remains a similar indoor reconstruction domain. Results on real robot data are qualitative, and PAGER does not consistently outperform supervised GEM on fine-grained benchmarks.

[1] “Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation.” ICRA, 2023.

[2] “Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning.” CVPR, 2023.

Questions for authors

See the above weaknesses for details.

1 comment

We thank the reviewer for the careful review and for recognising the diagnostic experiment and the simplicity of PAGER at inference.

W1 (embodied motivation). PAGER is not proposed as an alternative to pose estimation or mapping, but as the single-view component they build on. Many embodied tasks act on the current view: a robot passing through a doorway or placing an object on a table needs per-point semantics of the door or table surface in front of it, not a room-scale map that must be rebuilt and re-segmented whenever objects move. PAGER provides these semantics from a single view, at constant per-frame cost and without pose, and its predictions can be fused into a map when one is maintained. We will soften L012–013 and L041–043 ("must" → "often must") and add this positioning to the related work.

W2 (pose). On ScanNet, gravity alone recovers nearly all of the frame gap for the frozen model (66.49 vs. 67.45 with full pose), so the realistic comparison is gravity alignment with noisy or estimated gravity (tables in the general response). Without pose or gravity, PAGER-Gram (67.27) exceeds the frozen model with perfect gravity and comes within 0.2 mIoU of the ground-truth pose. The frozen model degrades with gravity error (64.83 at σ = 7.5°, 61.52 at 10°), and gravity estimated from RGB gives only 41.64. When a calibrated IMU is available, gravity alignment is a strong training-free baseline, above PAGER-KL without gravity (65.46), and PAGER should be combined with it: PAGER-KL + gravity reaches 68.27, above the pose oracle. We will state this in the limitations.

W3 (novelty). The contribution is a finding and a recipe. The finding: frozen pretrained 3D encoders fail on partial views mainly because of the coordinate frame the input is expressed in, and far less because of missing geometry (−67 vs. −5 mIoU, Sec. 3). The recipe: a self-supervised, correspondence-based adaptation that comes within 0.2 mIoU of oracle-pose performance (PAGER-Gram 67.27 vs. 67.45) without pose, labels, or changes to the encoder or probe, and that with gravity also recovers part of the gap due to missing geometry (68.27 vs. 67.45). The relational term adds +2.15 (KL) and +3.96 (Gram) mIoU over pointwise alignment alone, and random keys beat k-NN keys by 15.75 mIoU.

M2SKD and C2P share the complete-to-partial distillation idea but differ in their objectives. M2SKD trains LiDAR teacher and student networks with semantic supervision, using semantic and instance information for hard-class fusion and affinity distillation. C2P performs self-supervised pretraining of 4D sequence encoders. PAGER instead adapts a frozen pretrained encoder and probe to partial observations without labels or backbone retraining. Its adaptation is taxonomy-independent: the same encoder supports ScanNet-20 and ScanNet200 probes, unlike M2SKD's predefined semantic classes. We will cite and discuss both works.

W4 (generalization). We test generalization along four axes. (1) Dataset shift: ScanNet++ is captured with different hardware than ScanNet (laser-scanned geometry with registered iPhone and DSLR capture, vs. handheld RGB-D reconstructions). Trained only on ScanNet, PAGER exceeds full fine-tuning by 6.86 mIoU (63.44 vs. 56.58) and GEM by 9.21 over the 17 shared classes. (2) Encoders: the same recipe works on Sonata and Concerto, and with Concerto PAGER leads GEM on all three benchmarks (71.67 / 33.97 / 26.73 vs. 69.90 / 32.86 / 25.06). (3) Taxonomies: one adapted encoder serves both the 20- and 200-class probes without retraining. (4) Input sources: PAGER runs on VGGT point maps and robot stereo clouds in unseen buildings (Fig. 5), shown qualitatively since no ground truth exists. With Sonata, PAGER trails GEM on the 100- and 200-class benchmarks, a likely reason is that it reads out through the frozen probe, while GEM trains on fine-grained partial-view labels. Even so, without any labels, it closes 75.6% and 78.2% of the gap between the frozen model and full fine-tuning.

We hope these results address your concerns and would be grateful if you would consider updating your score.

Comments on the paper

1 comment

General response

We thank the reviewers for their careful reading. All three highlight the diagnostic of Section 3 as clean and convincing: it separates missing geometry from coordinate-frame mismatch with the encoder and probe held fixed, and Reviewer iWKC calls it the result they trust most. Reviewer upKW notes that the problem is timely, with a real and increasingly relevant gap between scene-level pretraining and single-view deployment. The reviewers describe the method as simple and lightweight, with 1.3M trainable parameters, no partial-view labels, and only the current camera-frame point cloud at inference (KKQ8, upKW), and note that it works with both Sonata and Concerto (all three). Reviewer iWKC finds distilling the frozen representation a reasonable response, and the ScanNet→ScanNet++ transfer the right test of it.

The main shared question is whether PAGER does more than correct the coordinate frame, and how it compares with sensor-based and oracle alternatives. The new results answer both.

1. Without any sensor, PAGER exceeds gravity-based alignment and comes within 0.2 mIoU of the ground-truth-pose oracle. With gravity, it exceeds that oracle.

Method (ScanNet-20, single frames, Sonata)Pose / gravity at inferencemIoU
Frozen + gravity estimated from RGBestimated41.64
Frozen + ground-truth gravityoracle66.49
Frozen + ground-truth 6-DoF poseoracle67.45
PAGER-KL (main tables)none65.46
PAGER-Gramnone67.27
PAGER-KL + gravity (trained and evaluated aligned)oracle gravity68.27

PAGER-KL is the variant used in the main tables. PAGER-Gram is the similarity-regression variant of the same relational loss, reported in App. A.4. Using only the camera-frame point cloud, PAGER-Gram exceeds the frozen model given perfect gravity (67.27 vs 66.49) and comes within 0.2 mIoU of the full ground-truth pose (67.45). The frozen model tolerates moderate gravity error but not large error:

Gravity error σ (per axis)5°7.5°10°
Frozen + noisy gravity66.21 ± 0.0264.83 ± 0.0261.52 ± 0.14

Mean ± std over 3 noise draws.

Gravity estimated from the image (41.64) falls far below this entire range, so the frozen model effectively requires a physical gravity sensor. PAGER requires none. The strength of gravity alignment also relies on a property of the benchmark rather than of the task: ScanNet scenes are upright, and the encoder was pretrained on gravity-aligned reconstructions. A scene tilted relative to gravity, such as a ramp or a vehicle interior, acts like a gravity error.

2. PAGER goes beyond frame correction. The ground-truth-pose result restores the exact frame the encoder was pretrained in, so it is the natural upper bound for any correction of the input frame. With the same gravity alignment as the frozen model, PAGER-KL improves from 66.49 to 68.27 and exceeds that bound. It recovers about 30% of the remaining gap to complete scenes (72.47), which is due to partial observation itself. This gain comes from the full-scene context distilled from the teacher.

3. PAGER sees no labels during adaptation. PAGER is trained without any labels: no partial-view labels, no complete-scene labels, and no class information of any kind. Its two losses compare features only, matching the adapted features of a partial view to the frozen encoder's features of the same points in the complete scene. Those target features come from Sonata and Concerto, which are themselves pretrained without labels. Neither the encoder nor the adapter ever receives a gradient from a label. The only labelled component is the global linear probe, the standard read-out for evaluating self-supervised 3D encoders. It is never used during adaptation, only to read out predictions afterwards. This is also why one adapted encoder serves both the ScanNet-20 and ScanNet200 probes without retraining: the adaptation never learns which label set will be used.

4. Cross-dataset evaluation. Desk, shower curtain and bathtub have no ScanNet++ points and scored zero for every method. Averaged over the 17 present classes, the ranking is unchanged:

ScanNet → ScanNet++ (17 classes)mIoU
GEM54.23
Full fine-tuning56.58
PAGER63.44

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.