- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes PAGER to study why 3D encoders pretrained on full scenes fail on camera-frame partial point clouds. The authors show that the coordinate frame, not missing geometry, causes most of the collapse. The PAGER method trains a lightweight adapter on top of the frozen encoder and probe and needs no partial-view labels. Instead, it distills from matched global-scene features using a point-wise alignment loss and a relational similarity loss. PAGER outperforms supervised Parameter-efficient fine-tuning (PEFT) methods on ScanNet and surpasses full fine-tuning baselines.
Critical strengths
- The Partial Semantic Segmentation is a timely problem. The gap between scene-level pretraining and single-view, camera-frame deployment is real and increasingly relevant.
- The paper provides a clean diagnostic. The Camera-Global condition isolates the coordinate-frame effect from missing geometry.
- The develop method is simple and backbone-agnostic. The encoder and probe stay frozen with a small adapter. And the same recipe works on Sonata and Concerto.
Critical weaknesses
- Some key baselines are missing. Frozen Sonata already scores 67.45 on world-frame single frames, exceeding PAGER’s 65.46. An oracle or IMU-based gravity-alignment control is therefore essential to isolate PAGER’s contribution. The evaluation should also include logit-level distillation from the global probe and the same adapter trained with cross-entropy through the frozen probe.
- The cross-dataset protocol is questionable. To my knowledge, Sonata’s pretraining includes ScanNet++, which makes the “zero-shot” description questionable. The full fine-tuning baseline may also be undertuned. In Table 8, the reported mIoU averages over three classes absent from ScanNet++, making the comparison harder to interpret.
- Limited novelty of the objective. The relational loss resembles established anchor-similarity distillation (SEED, ICLR 2021) and multi-frame-to-single-frame point-cloud distillation (SCPNet, CVPR 2023), neither of which is cited. The adapter is also drawn from GEM. The main contribution appears to be the problem setting and its per-point, correspondence-based implementation.
- The "Label-free" claim is overstated. The frozen probe is trained with labels from the same global ScanNet scenes. Section A.3 also shows that partial-view labels can be derived through the correspondences PAGER already requires. The more meaningful unlabeled training setting on unlabeled and posed RGB-D data is not evaluated.
- Overbroad performance claims. The statement that PAGER “outperforms supervised PEFT on both backbones” (line 364) holds only on ScanNet-20. With Sonata, GEM performs better on ScanNet++ (33.27 vs. 27.38) and ScanNet200 (21.97 vs. 19.25). PAGER also performs poorly in several small classes. For example, its smoke-detector score is 0.25 versus GEM’s 47.56.
Questions for authors
Please see the weakness part
4 comments
Response to Reviewer upKW (2/2)
W3 (novelty). Our contribution consists of a finding and an adaptation method. First, we show that coordinate-frame changes account for substantially more degradation in frozen 3D encoders under partial observations than missing geometry alone (−67 vs. −5 mIoU, Sec. 3). Second, we introduce self-supervised, correspondence-based adaptation that approaches oracle-pose performance without pose, labels, or backbone/probe modification (PAGER-Gram 67.27 vs. 67.45). With gravity alignment, PAGER-KL further exceeds the frozen pose oracle (68.27 vs. 67.45).
We acknowledge the connections to prior distillation methods. SEED matches image-level similarity distributions for model compression. SCPNet distills pairwise voxel similarities from a supervised multi-frame LiDAR teacher into a single-frame student for semantic scene completion. PAGER instead relates observed points to keys sampled across the complete scene, including unobserved regions, using the same frozen encoder to establish relational targets across coordinate frames. Its adaptation requires no semantic labels or backbone retraining. The relational loss improves mIoU by +2.15 (KL) and +3.96 (Gram) over pointwise alignment, and random global keys outperform nearest-neighbor keys by 15.75 mIoU. We will expand the related work to include also SEED and SCPNet.
W4 (label-free). We will use "self-supervised adaptation" to distinguish the adaptation stage from supervised probe training. Adaptation uses only feature-based losses from a frozen self-supervised encoder, without semantic labels or probe predictions. The linear probe is trained separately on labeled complete scenes and used only for downstream readout. This follows the standard evaluation setting for self-supervised 3D encoders. Although ScanNet allows partial-view labels to be derived from annotated meshes, acquiring such annotations for new environments requires additional effort. Adaptation on never-annotated geometry would test this setting directly, and we will discuss it as a limitation.
W5 (scope of performance claims). We will clarify that L364 refers specifically to ScanNet-20. On finer taxonomies, results depend on the backbone. With Concerto, PAGER outperforms GEM on all three benchmarks (71.67 / 33.97 / 26.73 vs. 69.90 / 32.86 / 25.06). With Sonata, GEM performs better on ScanNet++ and ScanNet200. The training requirements also differ: GEM uses task-specific semantic labels, while PAGER reuses the same adapter trained on ScanNet without 200-class supervision, together with the frozen 200-class probe.
Tables 9–10 report the ten largest class-wise differences in each direction. Across all 200 ScanNet200 classes, PAGER outperforms GEM on 66, ties on 24 (mostly rare classes with zero scores), and underperforms on 110, consistent with GEM's higher overall mIoU.
We thank the reviewer again for the suggestions, which have helped clarify the scope and contributions of the work. We hope the additional results address the main concerns and would appreciate reconsideration of the score.
Official Comments by Reviewer upKW
Thank you for the detailed response and additional experiments. The gravity results and the recomputed 17-class results address my questions about the oracle control and Table 8. But they also strengthen my main concern. With ground-truth gravity, the frozen model scores 66.49, above the main-table PAGER-KL result of 65.46. This suggests that the pose-free gain is most relevant to platforms without an IMU while the abstract’s claim that canonicalization cannot close the gap is no longer supported. The additional gain over frame correction (+1.78 with gravity) is demonstrated only relative to the frozen model. Controlled comparisons with GEM, the same adapter trained with cross-entropy on gravity-aligned clouds, and logit distillation are still missing, as is the never-labelled adaptation experiment. Thus I would keep my rating unchanged.
Thank you for the follow-up and for confirming that the oracle control and Table 8 are resolved.
The new results separate the two sources of degradation identified in Sec. 3: the coordinate frame, the dominant one (−67 mIoU), and partial observation, the smaller one (−5 mIoU). Frame correction addresses only the first. Even the full ground-truth pose, which restores the exact frame the encoder was pretrained in, reaches 67.45, while complete scenes reach 72.47: no frame correction can recover what the partial view does not contain. Frame correction is therefore a step toward global performance but cannot reach it, consistent with the abstract.
For the dominant source, accurate gravity recovers most of the frame gap for the frozen model (66.49), while gravity estimated from images does not (41.64). This relies on a property of the benchmark rather than of the task: ScanNet scenes are upright, and the encoder was pretrained on gravity-aligned reconstructions. It also requires an accurate gravity sensor aligned with the camera. A scene tilted relative to gravity, such as a ramp or a vehicle interior, acts like a gravity error, and with a 10° error the frozen model drops to 61.52. Ground-truth gravity is thus effectively an upper bound for the gravity-alignment baseline. Without any gravity, PAGER reaches this level: 65.46 with the KL objective and 67.27 with the Gram objective, which exceeds the best case of the gravity-alignment baseline.
For the remaining source, PAGER distills what the partial view lacks from the complete scene during training: with the same gravity alignment, it improves the frozen model from 66.49 to 68.27, above the ground-truth-pose oracle and thus beyond what any frame correction can reach. We therefore respectfully disagree that PAGER's relevance is limited to platforms without an IMU: without a gravity sensor, PAGER replaces it; with one, it addresses the part of the gap that frame correction leaves. Such a sensor is also unavailable for point clouds reconstructed from images, uncalibrated add-on cameras or archival data.
Regarding the suggested controls, we note what they would and would not isolate. Logits are a linear projection of the features PAGER already matches, so feature matching is the stronger constraint; logit distillation would additionally bring the labelled probe into training and tie the adapter to one label set. In a closely related dense-to-sparse 3D setting, SCPNet (Table 4), which the reviewer cites, finds that logit distillation lowers performance below no distillation (33.8 vs. 34.4), while relational distillation improves it (37.2). A cross-entropy adapter on gravity-aligned clouds would not isolate distillation either: on this benchmark, partial-view labels are copied from annotations of the complete reconstruction, so such an adapter also receives complete-scene information, through labels. Finally, since PAGER's adapter reads neither labels nor the probe, it is identical whether or not the adaptation scenes were ever annotated.
In summary: gravity alignment addresses the dominant, frame-related source of degradation but cannot recover the gap due to partial observation. PAGER improves over the frozen model both without gravity and with it.
Response to Reviewer upKW (1/2)
We thank the reviewer for the constructive feedback and specific suggestions. We address each point below, including additional experiments.
W1 (missing baselines). We evaluated gravity alignment (tables in the general response). With ground-truth gravity, the frozen model reaches 66.49 mIoU and remains stable up to σ = 5° of tilt noise (66.21), but degrades at larger errors (61.52 at 10°). RGB-estimated gravity yields 41.64. Without pose or gravity, PAGER achieves 65.46 (KL) and 67.27 (Gram, App. A.4), the latter exceeding the frozen model with perfect gravity and approaching the ground-truth-pose oracle (67.45). Combining PAGER-KL with gravity alignment further improves performance to 68.27, indicating benefits beyond frame correction alone (see our response to iWKC, W2 and Q1).
We use feature rather than logit distillation to keep adaptation independent of the semantic taxonomy. Logit distillation would depend on a particular pretrained probe, whereas feature distillation allows the same adapter to serve both ScanNet-20 and ScanNet200 probes. Cross-entropy supervision would require partial-view labels, as in GEM. We are also evaluating GEM on gravity-aligned clouds.
W2 (cross-dataset protocol). Sonata's self-supervised pretraining includes ScanNet++ scenes, including validation scenes, as unlabeled complete reconstructions. This applies equally to all compared methods, which share the same pretrained encoder. However, this exposure alone does not yield effective partial-view predictions: the frozen encoder reaches only 0.69 mIoU on ScanNet++. We will replace "zero-shot" with the more precise "cross-dataset transfer without ScanNet++ labels or adaptation."
All baselines, including full fine-tuning, use the same 100-epoch training protocol (App. B.2). Our ~19.6K partial-view training frames yield approximately twice as many sample presentations as Sonata's official 800-epoch schedule on 1,201 ScanNet scenes, although the training samples and protocols differ. Full fine-tuning achieves the highest in-domain performance (69.26 mIoU), indicating effective optimization on the source domain, but generalizes less effectively to ScanNet++. We will document the differences from Sonata's official fine-tuning recipe in App. B.2.
For class mapping, we follow the ScanNet++ benchmark mapping and ScanNet v2 category table. The 26 ScanNet++ categories map to 17 ScanNet-20 classes: 11 directly and six through merges (wall, cabinet, chair, door, picture, otherfurniture). Unmapped categories are ignored. Desk, shower curtain, and bathtub have no corresponding ScanNet++ points and score zero for all methods, capping the reported 20-class mIoU at 85%. Evaluating only the 17 present classes preserves the ranking: Sonata (+ linear) 31.07, Adapter 36.11, Sonata (late blocks + linear) 40.17, GEM 54.23, Sonata (full + linear) 56.58, and PAGER 63.44.