- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes the View-Consistent Panoptic Quality (VC-PQ) metric for panoptic segmentation. It builts up on the PR metric using its matching and unmatched-instance penalties, but replaces area-weighted overlap with equally weighted per-view overlap. It also changes class filtering and dataset aggregation. Experiments on ScanNet++ and ScanNetv2 evaluate four feed-forward methods and use controlled perturbations to examine how these choices affect scores and their dependence on the number of input views.
Critical strengths
S1: The paper identifies a meaningful evaluation issue in multi-view panoptic segmentation: correctly segmenting a large appearance can outweigh missing the same object in several smaller views. Giving appearances equal weight is a reasonable way to emphasize consistency across views. Further, the work separates the effects of view weighting, scene aggregation, and class filtering. Especially the substantial effect of class filtering is useful for understanding existing evaluations.
Critical weaknesses
W1: Limited methodological novelty.
VC-PQ makes a reasonable but small adjustment to PQ/VPQ: it changes how per-view overlaps are weighted while retaining the existing matching and counting scheme. The visibility-consistency term is a factorization of this average, and the remaining changes concern filtering and aggregation. I find the metric change alone too limited to support acceptance at a top-tier conference.
W2: Insufficient accompanying contribution.
The evaluation covers four feed-forward methods on two quite related indoor datasets, with unchanged main method rankings. A broader benchmark could include Hypersim [1] and Replica [2], and established per-scene methods such as Panoptic Lifting [3] and Contrastive Lift [4], with their additional inputs stated clearly. These comparisons already have precedent in PanSt3R [5] and would test different annotation sources and approaches to cross-view consistency. The careful ablations and changed trend with input-set size are useful, but do not establish enough additional insight. A broader benchmark, or a method addressing the exposed failures, could make the proposed metric change part of a stronger contribution. The current paper provides neither.
[1] Roberts et al., “Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding,” ICCV 21
[2] Straub et al., “The Replica Dataset: A Digital Replica of Indoor Spaces,” arXiv:1906.05797, 2019
[3] Siddiqui et al., “Panoptic Lifting for 3D Scene Understanding with Neural Fields,” CVPR 23
[4] Bhalgat et al., “Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion,” NeurIPS 23
[5] Žust et al., “PanSt3R: Multi-view Consistent Panoptic Segmentation,” ICCV 25
Questions for authors
Q1: VC-PQ gives small appearances of an object the same weight as large ones. Figure 2 shows why this can be useful, but Table A7 also shows sensitivity to boundary errors. On real predictions, how can we distinguish useful penalties for missed appearances or inconsistent IDs from penalties caused by uncertain annotations?
Q2: The metric is formulated for static multi-view scenes. Does it actually require the scene to be static, or could it be extended to a more general formulation to evaluate video and dynamic 3D scenes?
5 comments
Response to Reviewer iw5X (2/3): W1/W2 — per-scene methods
Broader benchmark: per-scene methods. We evaluated the per-scene methods that have released code and checkpoints on the Panoptic Lifting benchmark. These are three methods, Panoptic Lifting, Contrastive Lift, and MVC-PSU. We rendered the three methods from the checkpoints released by the authors and scored PQscene and VC-PQ together against the same ground-truth label maps as PanSt3R v2. Following this benchmark, all held-out test views of a scene form a single input set, and the scoring is at 512×512. PQscene of the three per-scene methods computed with the official evaluation code (58.8, 62.9, 63.3) was within 0.6 of the published values (58.9, 62.3, 63.1).
The inputs used by each method are as follows. All three per-scene methods require an optimization for each scene and take the camera poses of the training views and the predictions of Mask2Former as input. PanSt3R v2 uses neither an optimization for each scene, nor camera poses, nor the predictions of an external segmenter. For PanSt3R, however, the code of the per-scene optimization (LUDVIG) that uplifts the predictions to 3D Gaussian Splatting is not released, so its predictions were made feed-forward on test views rendered from a 3D Gaussian Splatting built from the posed training views.
On Replica and HyperSim, the released checkpoints of Panoptic Lifting and MVC-PSU cover a single scene each (Replica room_0 and HyperSim ai_001_008), and no checkpoint of Contrastive Lift is released for the two datasets. We therefore trained Contrastive Lift on these two scenes ourselves (in the same way as the released checkpoints of the authors, starting from the released checkpoint of Panoptic Lifting and training the instance branch for 3 epochs) and report the results on the two scenes together (the four right columns of Table R15).
Table R15. The Panoptic Lifting benchmark. The four left columns are the values on the 12 scenes of ScanNet, and the four right columns are the values on a single scene each, Replica room_0 and HyperSim ai_001_008. PQscene and VC-PQ are over all classes. The columns and divide over thing classes and therefore do not sum to 100 with VC-PQ over all classes. The column is the sum of the view-consistency term and the matching threshold term, with the values of the two terms in parentheses (the sum is computed before rounding, so some cells are off by 0.1 from the sum of the values in parentheses). of VC-PQ and of the decomposition is 0.1% of the image area (Common 1). PQscene is the value of the same scorer as Table 1. On the 12 scenes of ScanNet, this value is 1.1 to 1.2 higher than that of the official evaluation code for the three per-scene methods. The predictions of PanSt3R v2 contain void, so under the official evaluation code its value lies between 61.5 and 69.7 depending on how void is handled. † marks the values of models that we trained.
| Method | ScanNet (12 scene) PQscene | ScanNet (12 scene) VC-PQ | ScanNet (12 scene) | ScanNet (12 scene) | Replica room_0 (1 scene) PQscene | Replica room_0 (1 scene) VC-PQ | HyperSim ai_001_008 (1 scene) PQscene | HyperSim ai_001_008 (1 scene) VC-PQ |
|---|---|---|---|---|---|---|---|---|
| Panoptic Lifting | 59.9 | 36.3 | 56.9 | 5.8 (2.4 + 3.5) | 62.1 | 58.5 | 67.9 | 48.8 |
| Contrastive Lift | 64.1 | 38.3 | 50.7 | 8.3 (3.3 + 5.0) | 65.2† | 58.5† | 63.2† | 49.9† |
| MVC-PSU | 64.5 | 40.9 | 50.7 | 10.1 (2.6 + 7.5) | 59.6 | 51.0 | 77.9 | 60.9 |
| PanSt3R v2 | 66.4 | 45.0 | 33.9 | 5.2 (4.9 + 0.3) | 64.1 | 61.7 | 54.6 | 40.8 |
In the error decomposition on the 12 scenes of ScanNet, the difference between the three per-scene methods and PanSt3R v2 lies mostly in the mean IoU term ( 50.7 to 56.9 versus 33.9). The sum of the view-consistency term and the matching threshold term () is 5.8 to 10.1 and 5.2.
Response to Reviewer iw5X (3/3): W1/W2 — fine-tuning and rankings; Q1, Q2
A method addressing the exposed failures. For the second direction, we fine-tuned the released PanSt3R v1 with the pair score of VC-PQ and the pair score of PQscene each added as a loss (design and intervals in Common 3). At , adding a single loss term raised VC-PQ from 20.8 for the model fine-tuned without an additional loss (control) to 21.7 (PQscene loss) and 22.2 (VC-PQ loss).
Unchanged rankings and additional insight. As the reviewer says, the main method rankings did not change in the submission (the reason is given in Common 2 (b)), and in the additional results evaluated over several scenes (Table R14, and the 12 scenes of ScanNet in Table R15) the rankings were also the same except for one cell of Replica (PQscene at , 48.9 and 48.8). Even with the same rankings, however, there were two cases in which the judgment depends on the metric used.
- Whether to add input views. When the views given to PanSt3R v2 (ScanNet++) increase from 5 to 40, the four existing metrics lead to the judgment that the result is unchanged (a change of no more than 0.7), and VC-PQ leads to the judgment that the result per object has become worse (from 29.1 to 23.8, Common 2).
- Which loss to choose. In the fine-tuning experiment above, the models fine-tuned with the two losses have the same area-weighted score (26.9 and 26.9), so this score alone leads to the judgment that either loss can be used. The difference between the two showed in of the error decomposition (13.3 and 11.5, a difference of −1.79 [−2.69, −0.75]).
In the revision, we will include Tables R14 and R15 in the appendix and state the evaluation conditions of this benchmark in Sec. 5.1.
Q1 — Penalties caused by uncertain annotations
This question is answered in part 6 of Common 1.
Q2 — Extension to video and dynamic scenes
The metric does not require the scene to be static. Its formulas use neither 3D geometry nor the order of the views, and it can be computed whenever the instance IDs of the ground truth and of the prediction are given consistently over the whole input set. The Conclusion of the submission states that VC-PQ is defined for a static scene, but this restates the scope of the task we evaluated (Sec. 3.1) and is not a condition that the formulas require.
When the metric is applied to video, what differs from VPQ is the range over which pairs are matched. VC-PQ takes a whole video as a single input set and matches only once, so that an object is scored with a single ID over the whole video.
On the validation set of VIPSeg (343 videos, 124 classes), which contains moving objects, we scored the outputs of seven released video panoptic segmentation methods without changing the definition, together with the official VPQ and STQ of VIPSeg (Table R16).
Table R16. VIPSeg validation set (343 videos, all classes). VPQ and STQ are the values of the official VIPSeg evaluation code, and of VC-PQ is 0.1% of the image area (Common 1). The two right columns are the error decomposition over thing classes (the paper's test order).
| Method | VPQ | STQ | VC-PQ | ||
|---|---|---|---|---|---|
| CAVIS ViT-L online | 56.8 | 51.0 | 53.6 | 40.1 | 5.0 |
| LVMT DINOv2 online | 56.6 | 51.3 | 54.0 | 40.4 | 4.3 |
| DVIS++ ViT-L online | 56.4 | 50.6 | 53.3 | 41.4 | 4.5 |
| DVIS++ R50 offline | 44.1 | 43.9 | 42.4 | 53.4 | 3.7 |
| DVIS++ R50 online | 42.2 | 38.5 | 40.4 | 56.7 | 3.4 |
| TarViS R50 | 33.4 | 43.0 | 30.2 | 57.1 | 8.6 |
| Clip-PanoFCN R50 | 24.1 | 28.7 | 18.2 | 65.3 | 14.8 |
The order of the methods was the same as under VPQ except for the top two methods (CAVIS and LVMT) (the two methods differ by no more than 0.4 under both metrics). The error decomposition of the paper (Eq. (7)) is also computed on video as it is, and most of the lost score of all seven methods lies in the mean IoU term (40.1 to 65.3). The sum of the view-consistency term and the matching threshold term, which is lost across frames, is 3.4 to 5.0 for five of the methods and 8.6 and 14.8 for TarViS and Clip-PanoFCN. These two methods are also the ones whose score falls most in the official VPQ when the window grows from 1 frame to 8 frames (3.2 and 3.8; 1.0 to 1.2 for the other five methods). That is, the difference that shows in VPQ as a fall with the window length shows in VC-PQ as this sum. VC-PQ can therefore be used on video without changing its definition, and along with the score it shows where that score is lost. We could not evaluate on data of dynamic 3D scenes. In the Conclusion of the revision, we will state that the formulation of the metric is not limited to static scenes.
Response to Reviewer iw5X (1/3): W1/W2 — answer and broader benchmark
We thank the reviewer for the specific suggestions on the benchmark and the methods. The points raised by several reviewers are answered in Common 1 to 3, and this response points to the relevant parts.
W1 / W2 — Novelty and accompanying contribution
VC-PQ keeps the matching and counting of PQ so that it reduces to standard PQ for a single view and can be compared directly with existing PQ results. Within that scheme, what we intended was to formalize what it means for a single object to be matched in the multi-view setting. When views are counted with equal weight, the pair score is the product of the mean of the per-view IoU and the Jaccard index of the sets of views in which the prediction and the ground truth are visible (Sec. 4.1 of the paper). In other words, PQ is redefined as the problem of getting the IoU right in every view and, at the same time, getting right the views in which the object appears. Under this definition, when the views given to PanSt3R v2 (ScanNet++) increase from 5 to 40, VC-PQ falls by 5.3, and VC-PQ$_1$, in which only the weighting is changed to the area, falls by 1.7 (Common 2). As the reviewer points out, however, the evaluation in the submission was narrow in scope. We therefore carried out additional experiments in both of the directions mentioned and report them in turn below.
Broader benchmark: HyperSim and Replica. We agree that the evaluation was limited to two closely related indoor datasets. The two datasets of the submission (ScanNet++ and ScanNetv2) are both indoor datasets acquired by scanning.
The two proposed datasets (HyperSim and Replica) were evaluated under the same protocol as Table 1 (N images are given as an input set and scored on the input views) (Table R14).
Table R14. HyperSim and Replica, all classes. PQscene is computed as in Table 1, and η of VC-PQ is 0.1% of the image area (Common 1). PanSt3R v1 is evaluated with the standard post-processing of PanSt3R in place of the simulated annealing (QUBO) post-processing of the submission.
On these two datasets as well, where the ground truth is rendered together with the images, VC-PQ of PanSt3R v2 does not rise as N grows from 5 to 40 (HyperSim from 25.5 to 24.3, Replica from 32.1 to 30.9).
This table, however, covers two methods, PanSt3R v1 and v2. SegVGGT and SIU3R are trained to output the classes of ScanNet only and could not be evaluated on these two datasets.