acceptodds
Under review as a conference paper at ICLR 2027

Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation

Abstract

Multi-view panoptic segmentation assigns a semantic class and a scene-level instance ID to every pixel of an unordered set of images, and recent feed-forward 3D models predict these labels for the input views in a single forward pass. Their predictions, however, have been evaluated with the scene-level PQ (PQ) borrowed from per-scene optimization methods, typically on rendered held-out views. PQ tiles all views of a scene into a single image, so that a missed appearance or a change of ID lowers the score of the matched pair only in proportion to its area. We propose View-Consistent Panoptic Quality (VC-PQ), which extends PQ from a single image to a set of input views, counts equally every view in which an instance is visible, and penalizes a prediction that is not visible in the same views as its ground truth. A decomposition of VC-PQ attributes the score a method loses to mask accuracy, view consistency, and the matching threshold. A single additional parameter recovers the area weighting of tiling for comparison. Under a fixed evaluation protocol on ScanNet++ and ScanNetv2, recent feed-forward methods are evaluated with VC-PQ and PQ, and the decomposition shows where each of them loses its score. Controlled perturbations of the ground truth show that VC-PQ responds to the number of views in which an instance is missed or changes ID, whereas PQ responds to their area. The aim of this work is to make view consistency part of the evaluation of multi-view panoptic segmentation, with VC-PQ reported alongside PQ. The evaluation code will be released.

Primary area:
datasets and benchmarks
Keywords:
multi-view panoptic segmentation, evaluation metric, panoptic quality, view consistency, feed-forward 3D reconstruction, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iw5XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the View-Consistent Panoptic Quality (VC-PQ) metric for panoptic segmentation. It builts up on the PR metric using its matching and unmatched-instance penalties, but replaces area-weighted overlap with equally weighted per-view overlap. It also changes class filtering and dataset aggregation. Experiments on ScanNet++ and ScanNetv2 evaluate four feed-forward methods and use controlled perturbations to examine how these choices affect scores and their dependence on the number of input views.

Critical strengths

S1: The paper identifies a meaningful evaluation issue in multi-view panoptic segmentation: correctly segmenting a large appearance can outweigh missing the same object in several smaller views. Giving appearances equal weight is a reasonable way to emphasize consistency across views. Further, the work separates the effects of view weighting, scene aggregation, and class filtering. Especially the substantial effect of class filtering is useful for understanding existing evaluations.

Critical weaknesses

W1: Limited methodological novelty.

VC-PQ makes a reasonable but small adjustment to PQ/VPQ: it changes how per-view overlaps are weighted while retaining the existing matching and counting scheme. The visibility-consistency term is a factorization of this average, and the remaining changes concern filtering and aggregation. I find the metric change alone too limited to support acceptance at a top-tier conference.

W2: Insufficient accompanying contribution.

The evaluation covers four feed-forward methods on two quite related indoor datasets, with unchanged main method rankings. A broader benchmark could include Hypersim [1] and Replica [2], and established per-scene methods such as Panoptic Lifting [3] and Contrastive Lift [4], with their additional inputs stated clearly. These comparisons already have precedent in PanSt3R [5] and would test different annotation sources and approaches to cross-view consistency. The careful ablations and changed trend with input-set size are useful, but do not establish enough additional insight. A broader benchmark, or a method addressing the exposed failures, could make the proposed metric change part of a stronger contribution. The current paper provides neither.

[1] Roberts et al., “Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding,” ICCV 21

[2] Straub et al., “The Replica Dataset: A Digital Replica of Indoor Spaces,” arXiv:1906.05797, 2019

[3] Siddiqui et al., “Panoptic Lifting for 3D Scene Understanding with Neural Fields,” CVPR 23

[4] Bhalgat et al., “Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion,” NeurIPS 23

[5] Žust et al., “PanSt3R: Multi-view Consistent Panoptic Segmentation,” ICCV 25

Questions for authors

Q1: VC-PQ gives small appearances of an object the same weight as large ones. Figure 2 shows why this can be useful, but Table A7 also shows sensitivity to boundary errors. On real predictions, how can we distinguish useful penalties for missed appearances or inconsistent IDs from penalties caused by uncertain annotations?

Q2: The metric is formulated for static multi-view scenes. Does it actually require the scene to be static, or could it be extended to a more general formulation to evaluate video and dynamic 3D scenes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iw5X (1/3): W1/W2 — answer and broader benchmark

We thank the reviewer for the specific suggestions on the benchmark and the methods. The points raised by several reviewers are answered in Common 1 to 3, and this response points to the relevant parts.

W1 / W2 — Novelty and accompanying contribution

VC-PQ keeps the matching and counting of PQ so that it reduces to standard PQ for a single view and can be compared directly with existing PQ results. Within that scheme, what we intended was to formalize what it means for a single object to be matched in the multi-view setting. When views are counted with equal weight, the pair score is the product of the mean of the per-view IoU and the Jaccard index of the sets of views in which the prediction and the ground truth are visible (Sec. 4.1 of the paper). In other words, PQ is redefined as the problem of getting the IoU right in every view and, at the same time, getting right the views in which the object appears. Under this definition, when the views given to PanSt3R v2 (ScanNet++) increase from 5 to 40, VC-PQ falls by 5.3, and VC-PQ$_1$, in which only the weighting is changed to the area, falls by 1.7 (Common 2). As the reviewer points out, however, the evaluation in the submission was narrow in scope. We therefore carried out additional experiments in both of the directions mentioned and report them in turn below.

  • A broader benchmark: we evaluated HyperSim and Replica (Table R14) and evaluated three per-scene methods on the Panoptic Lifting benchmark (Table R15).
  • An experiment in the direction of addressing the exposed failures: we fine-tuned PanSt3R with the pair score of VC-PQ turned into a loss (Common 3).

Broader benchmark: HyperSim and Replica. We agree that the evaluation was limited to two closely related indoor datasets. The two datasets of the submission (ScanNet++ and ScanNetv2) are both indoor datasets acquired by scanning.

The two proposed datasets (HyperSim and Replica) were evaluated under the same protocol as Table 1 (NN images are given as an input set and scored on the input views) (Table R14).

  • HyperSim: the 46 scenes of the official validation split. The data are synthetic, so the ground truth comes directly from rendering. The classes are the 37 that remain of the 40 NYU classes after the three "other" classes are removed. Depending on the number of images a scene has, we used 120, 80, or 40 images per scene (Table 1 uses 120 per scene).
  • Replica: 7 scenes, 120 images per scene. The images and the ground truth are rendered from the same mesh. The classes are the 48 classes provided by Replica that occur in these scenes, with wall, floor, and ceiling as stuff.

Table R14. HyperSim and Replica, all classes. PQscene is computed as in Table 1, and η\eta of VC-PQ is 0.1% of the image area (Common 1). PanSt3R v1 is evaluated with the standard post-processing of PanSt3R in place of the simulated annealing (QUBO) post-processing of the submission.

DatasetMethodPQscene (NN = 5 / 10 / 20 / 40)VC-PQ (NN = 5 / 10 / 20 / 40)
HyperSimPanSt3R v135.2 / 36.1 / 37.7 / 38.124.3 / 23.1 / 22.1 / 21.3
HyperSimPanSt3R v237.3 / 38.5 / 39.5 / 41.725.5 / 25.1 / 24.0 / 24.3
ReplicaPanSt3R v143.0 / 48.9 / 47.1 / 49.727.5 / 27.1 / 25.0 / 25.3
ReplicaPanSt3R v245.1 / 48.8 / 52.6 / 53.232.1 / 31.5 / 31.6 / 30.9

On these two datasets as well, where the ground truth is rendered together with the images, VC-PQ of PanSt3R v2 does not rise as NN grows from 5 to 40 (HyperSim from 25.5 to 24.3, Replica from 32.1 to 30.9).

This table, however, covers two methods, PanSt3R v1 and v2. SegVGGT and SIU3R are trained to output the classes of ScanNet only and could not be evaluated on these two datasets.

AuthorsOpenReview ↗

Response to Reviewer iw5X (2/3): W1/W2 — per-scene methods

Broader benchmark: per-scene methods. We evaluated the per-scene methods that have released code and checkpoints on the Panoptic Lifting benchmark. These are three methods, Panoptic Lifting, Contrastive Lift, and MVC-PSU. We rendered the three methods from the checkpoints released by the authors and scored PQscene and VC-PQ together against the same ground-truth label maps as PanSt3R v2. Following this benchmark, all held-out test views of a scene form a single input set, and the scoring is at 512×512. PQscene of the three per-scene methods computed with the official evaluation code (58.8, 62.9, 63.3) was within 0.6 of the published values (58.9, 62.3, 63.1).

The inputs used by each method are as follows. All three per-scene methods require an optimization for each scene and take the camera poses of the training views and the predictions of Mask2Former as input. PanSt3R v2 uses neither an optimization for each scene, nor camera poses, nor the predictions of an external segmenter. For PanSt3R, however, the code of the per-scene optimization (LUDVIG) that uplifts the predictions to 3D Gaussian Splatting is not released, so its predictions were made feed-forward on test views rendered from a 3D Gaussian Splatting built from the posed training views.

On Replica and HyperSim, the released checkpoints of Panoptic Lifting and MVC-PSU cover a single scene each (Replica room_0 and HyperSim ai_001_008), and no checkpoint of Contrastive Lift is released for the two datasets. We therefore trained Contrastive Lift on these two scenes ourselves (in the same way as the released checkpoints of the authors, starting from the released checkpoint of Panoptic Lifting and training the instance branch for 3 epochs) and report the results on the two scenes together (the four right columns of Table R15).

Table R15. The Panoptic Lifting benchmark. The four left columns are the values on the 12 scenes of ScanNet, and the four right columns are the values on a single scene each, Replica room_0 and HyperSim ai_001_008. PQscene and VC-PQ are over all classes. The columns εIoU‾\varepsilon_{\overline{\mathrm{IoU}}} and εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q divide 1−VC-PQ1 - \text{VC-PQ} over thing classes and therefore do not sum to 100 with VC-PQ over all classes. The column εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q is the sum of the view-consistency term and the matching threshold term, with the values of the two terms in parentheses (the sum is computed before rounding, so some cells are off by 0.1 from the sum of the values in parentheses). η\eta of VC-PQ and of the decomposition is 0.1% of the image area (Common 1). PQscene is the value of the same scorer as Table 1. On the 12 scenes of ScanNet, this value is 1.1 to 1.2 higher than that of the official evaluation code for the three per-scene methods. The predictions of PanSt3R v2 contain void, so under the official evaluation code its value lies between 61.5 and 69.7 depending on how void is handled. † marks the values of models that we trained.

MethodScanNet (12 scene) PQsceneScanNet (12 scene) VC-PQScanNet (12 scene) εIoU‾\varepsilon_{\overline{\mathrm{IoU}}}ScanNet (12 scene) εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_qReplica room_0 (1 scene) PQsceneReplica room_0 (1 scene) VC-PQHyperSim ai_001_008 (1 scene) PQsceneHyperSim ai_001_008 (1 scene) VC-PQ
Panoptic Lifting59.936.356.95.8 (2.4 + 3.5)62.158.567.948.8
Contrastive Lift64.138.350.78.3 (3.3 + 5.0)65.2†58.5†63.2†49.9†
MVC-PSU64.540.950.710.1 (2.6 + 7.5)59.651.077.960.9
PanSt3R v266.445.033.95.2 (4.9 + 0.3)64.161.754.640.8

In the error decomposition on the 12 scenes of ScanNet, the difference between the three per-scene methods and PanSt3R v2 lies mostly in the mean IoU term (εIoU‾\varepsilon_{\overline{\mathrm{IoU}}} 50.7 to 56.9 versus 33.9). The sum of the view-consistency term and the matching threshold term (εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q) is 5.8 to 10.1 and 5.2.

AuthorsOpenReview ↗

Response to Reviewer iw5X (3/3): W1/W2 — fine-tuning and rankings; Q1, Q2

A method addressing the exposed failures. For the second direction, we fine-tuned the released PanSt3R v1 with the pair score of VC-PQ and the pair score of PQscene each added as a loss (design and intervals in Common 3). At N=40N = 40, adding a single loss term raised VC-PQ from 20.8 for the model fine-tuned without an additional loss (control) to 21.7 (PQscene loss) and 22.2 (VC-PQ loss).

Unchanged rankings and additional insight. As the reviewer says, the main method rankings did not change in the submission (the reason is given in Common 2 (b)), and in the additional results evaluated over several scenes (Table R14, and the 12 scenes of ScanNet in Table R15) the rankings were also the same except for one cell of Replica (PQscene at N=10N = 10, 48.9 and 48.8). Even with the same rankings, however, there were two cases in which the judgment depends on the metric used.

  • Whether to add input views. When the views given to PanSt3R v2 (ScanNet++) increase from 5 to 40, the four existing metrics lead to the judgment that the result is unchanged (a change of no more than 0.7), and VC-PQ leads to the judgment that the result per object has become worse (from 29.1 to 23.8, Common 2).
  • Which loss to choose. In the fine-tuning experiment above, the models fine-tuned with the two losses have the same area-weighted score (26.9 and 26.9), so this score alone leads to the judgment that either loss can be used. The difference between the two showed in εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q of the error decomposition (13.3 and 11.5, a difference of −1.79 [−2.69, −0.75]).

In the revision, we will include Tables R14 and R15 in the appendix and state the evaluation conditions of this benchmark in Sec. 5.1.

Q1 — Penalties caused by uncertain annotations

This question is answered in part 6 of Common 1.

Q2 — Extension to video and dynamic scenes

The metric does not require the scene to be static. Its formulas use neither 3D geometry nor the order of the views, and it can be computed whenever the instance IDs of the ground truth and of the prediction are given consistently over the whole input set. The Conclusion of the submission states that VC-PQ is defined for a static scene, but this restates the scope of the task we evaluated (Sec. 3.1) and is not a condition that the formulas require.

When the metric is applied to video, what differs from VPQ is the range over which pairs are matched. VC-PQ takes a whole video as a single input set and matches only once, so that an object is scored with a single ID over the whole video.

On the validation set of VIPSeg (343 videos, 124 classes), which contains moving objects, we scored the outputs of seven released video panoptic segmentation methods without changing the definition, together with the official VPQ and STQ of VIPSeg (Table R16).

Table R16. VIPSeg validation set (343 videos, all classes). VPQ and STQ are the values of the official VIPSeg evaluation code, and η\eta of VC-PQ is 0.1% of the image area (Common 1). The two right columns are the error decomposition over thing classes (the paper's test order).

MethodVPQSTQVC-PQεIoU‾\varepsilon_{\overline{\mathrm{IoU}}}εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q
CAVIS ViT-L online56.851.053.640.15.0
LVMT DINOv2 online56.651.354.040.44.3
DVIS++ ViT-L online56.450.653.341.44.5
DVIS++ R50 offline44.143.942.453.43.7
DVIS++ R50 online42.238.540.456.73.4
TarViS R5033.443.030.257.18.6
Clip-PanoFCN R5024.128.718.265.314.8

The order of the methods was the same as under VPQ except for the top two methods (CAVIS and LVMT) (the two methods differ by no more than 0.4 under both metrics). The error decomposition of the paper (Eq. (7)) is also computed on video as it is, and most of the lost score of all seven methods lies in the mean IoU term (40.1 to 65.3). The sum of the view-consistency term and the matching threshold term, which is lost across frames, is 3.4 to 5.0 for five of the methods and 8.6 and 14.8 for TarViS and Clip-PanoFCN. These two methods are also the ones whose score falls most in the official VPQ when the window grows from 1 frame to 8 frames (3.2 and 3.8; 1.0 to 1.2 for the other five methods). That is, the difference that shows in VPQ as a fall with the window length shows in VC-PQ as this sum. VC-PQ can therefore be used on video without changing its definition, and along with the score it shows where that score is lost. We could not evaluate on data of dynamic 3D scenes. In the Conclusion of the revision, we will state that the formulation of the metric is not limited to static scenes.

Reviewer_7EyVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the evaluation of multi-view panoptic segmentation, where a model assigns a class and a scene-level instance ID to every pixel of an unordered set of views. Existing methods are evaluated with PQscene, which concatenates all views of a scene and computes one IoU per object, so each view contributes to the object's score in proportion to the object's area in that view. The authors argue that every view should contribute equally, and that a prediction should be penalized for every view in which it is missing while the ground truth is visible, or visible while the ground truth is not. They propose VC-PQ, which averages the per-view IoU over all views where either the ground truth or the prediction is visible, with zero where only one of them is, and uses this score instead of IoU in standard PQ. They also decompose the score lost under VC-PQ into mask accuracy, view consistency and matching threshold, and analyze PQscene in a controlled way, attributing the difference between the two metrics to area weighting, per-scene averaging, and a filter in the official PQscene code that drops classes covering less than 0.5% of the pixels.

Critical strengths

S1. Identifies a real weakness of PQscenePQ_{\text{scene}}, with a thorough analysis. Since views are area-weighted, a good prediction in a large view can compensate for missing the same instance in smaller views. The paper disentangles area weighting, per-scene averaging and the class filter, with a useful error decomposition.

S2. A clean metric, compatible with PQ, that penalizes predictions visible in the wrong views.

S3. Well written and easy to follow. The metric is formally defined, and its difference from PQscene is illustrated with clear examples.

Critical weaknesses

W1. Equal view weighting is a design choice, not an objectively correct definition of multi-view quality. Full equality means a 100-pixel, mostly occluded view of an object weighs as much as a 100,000-pixel frontal view, which is hard to justify.

W2. Sensitivity to annotation noise. Ground-truth labels projected from 3D scans are noisy at small scales.

In Table A7, the authors shrink the ground-truth masks of thing classes on ScanNet++ by one pixel and score them against the original ground truth. A perfect prediction would score 100; this almost perfect one gets only 61.3 VC-PQ, against 85.9 for PQscene, partly because 13.2% of the object views become too small to count and are scored as missing. Even when views are weighted by their area (γ=1\gamma = 1), it scores only 70.3.

Since real labels contain errors of this size, VC-PQ may partly measure annotation noise, as the authors acknowledge (Sec. 6). The paper offers a minimum pixel count (η) and a mild area weighting (γ\gamma) as remedies, but these introduce arbitrary choices.

W3. The practical benefit of VC-PQ is not demonstrated.

  • (a) VC-PQ ranks methods as PQscene does in all settings but one (Tables 1, A4). The only change in order is PanSt3R v2 vs. EPS3D on ScanNetv2 stuff classes (80.6 vs. 80.9 PQscene; 71.5 vs. 69.3 VC-PQ).
  • (b) The comparison rests mostly on PanSt3R, plus SegVGGT, which is only evaluated on ScanNetv2 thing classes. As the paper notes (Sec. 5.4), EPS3D and IGGT lose nearly all of their score within single images, before view consistency comes into play (over 90% on ScanNet++, Tables 3 and A12), so they barely test what VC-PQ is designed to measure.

W4. Related work: the novelty of combining equal view weighting with a view-consistency term is limited. The per-object score is, as the paper acknowledges (Sec. 2.2), conceptually very close to IGGT and G2TAM, which average IoU equally over views and check whether an object is tracked in every view, and to the following metrics, which the paper does not cite:

  • HOTA (Luiten et al., IJCV 2021): its association score is an IoU over the frames in which the prediction and the ground truth appear, close to the view-consistency term of VC-PQ (although HOTA matches per frame rather than once per instance).
  • IDF1 (Ristani et al., ECCV Workshops 2016): matches trajectories once per sequence and counts every frame equally, as VC-PQ does for views.
Questions for authors

Q1. What evidence shows that equal view weighting reflects multi-view quality better than area weighting? Concretely: is there a case, beyond the single swap in W3a, where VC-PQ changes which method is preferred? And in which settings should VC-PQ, PQscene, or an intermediate γ\gamma be used, given W1 and W2?

Q2. SIU3R is a feed-forward panoptic method cited in the paper, and its code is public. Could you also evaluate it? A method other than PanSt3R that performs well on both datasets would make the comparison more convincing (W3b).

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7EyV (1/3): W1/Q1, W2, W3 (a)

We thank the reviewer for the specific suggestions, including the evaluation of SIU3R. The points raised by several reviewers are answered in Common 1 to 3, and this response points to the relevant parts.

W1 / Q1 — Equal view weighting is a design choice

As the reviewer says, equal view weighting is a design choice, and we do not regard it as the only correct definition of multi-view quality. A small mask has a less accurate ground truth, and its IoU moves by a large amount for a difference of a few pixels, so counting such a view with the same weight as a large view clearly has a cost. Sec. 4.2 of the paper defines the family VC-PQ$_\gamma$, in which the weighting of views is given by a single parameter γ\gamma (γ=0\gamma = 0 is the equal weighting of views, γ=1\gamma = 1 is the area weighting, and the matching rule and the counting are the same at every γ\gamma). What this choice changes can therefore be measured within the same framework, and in this measurement the method rankings were the same at all six values of γ\gamma (Common 1, part 5).

Within this family, we recommend γ=0\gamma = 0 as the reported value for two reasons.

  • A change that the area weighting hides was present in real predictions. For PanSt3R v2 (ScanNet++), as the input views increase from 5 to 40, the share of objects matched in more than half of the views in which they are visible falls from 47.7% to 42.8% (the share of individual views that are matched stays the same, at 56.6% and 55.8%). With this change, VC-PQ$_1$ falls by 1.7 and VC-PQ by 5.3 (Common 2 (c) and Table R7). The reason is that under the area weighting, as S1 of the review states, a view in which an object appears large compensates for a miss in a view in which it appears small.
  • On the predictions of real methods, switching to the area weighting did not reduce the drop caused by annotation noise. At the recommended η\eta, a 4 px shift of the ground truth lowers VC-PQ from 26.6 to 24.6 and VC-PQ$_1$ from 31.4 to 29.4 (Common 1, part 5 and Table R6).

We also answer the two other questions of Q1.

  • A case in which VC-PQ changes which method is preferred: we did not find one in the evaluations on ScanNet++ and ScanNetv2, which are the main evaluations of the paper (Common 2 (b)).
  • Which metric to use in which setting: as the reported value we use VC-PQ (γ=0\gamma = 0) as above, and as stated in the Conclusion we hope that it is reported alongside PQscene. For annotation noise, the measurements of Common 1 confirmed that setting η\eta to 0.1% of the image area lowers the sensitivity, and we therefore recommend this η\eta.

We will add this explanation to Sec. 4.2 in the revision.

W2 — Sensitivity to annotation noise

This point is answered in parts 1, 2, 4, and 5 of Common 1.

W3 — The practical benefit of VC-PQ

(a) As the reviewer points out, the method rankings do not change except in the one setting mentioned (PanSt3R v2 and EPS3D on the stuff classes of ScanNetv2). The rankings also stayed the same when we additionally computed per-image PQ, VPQ, and STQ on the same predictions and added SIU3R, as requested in Q2 (Table R8 of Common 2).

The rankings were the same because the methods evaluated so far already differ widely in per-view panoptic quality (Common 2 (b)).

Although the rankings did not change, VC-PQ shows differences that the rankings alone do not show.

  • Two methods that are close in per-view panoptic quality are far apart under VC-PQ. On ScanNetv2, PanSt3R v2 and SIU3R have a per-image PQ of 51.3 and 48.3, whereas their VC-PQ is 43.3 and 27.6 (Common 2 (b)).
  • The judgment on adding input views changes. For PanSt3R v2 (ScanNet++), as NN grows from 5 to 40, the four existing metrics move by no more than 0.7, whereas VC-PQ falls from 29.1 to 23.8. Judged by the existing metrics alone, adding views leaves the result unchanged, and judged by VC-PQ, it makes the result worse (Common 2 (a) and (c)).

To check the practical benefit more directly, we added an experiment that uses the pair score of a metric as a training loss. In this experiment, which fine-tunes the released PanSt3R v1, VC-PQ at N=40N = 40 rose over the control by +0.98 [+0.01, +1.49] for the model fine-tuned with an additional loss built from the pair score of PQscene and by +1.45 [+0.83, +2.27] for the model fine-tuned with an additional loss built from the pair score of VC-PQ (95% intervals). That is, the pair score of VC-PQ can be considered as a loss as it is once the IoU is replaced by the soft IoU, and within the scope of this experiment it could be used not only as an evaluation metric but also as a training signal. In addition, the two models were not distinguished by the area-weighted score and differed in the sum of the view-consistency term (εVC\varepsilon_{\mathrm{VC}}) and the matching threshold term (εq\varepsilon_q) (Common 3 (b)).

AuthorsOpenReview ↗

Response to Reviewer 7EyV (2/3): W3 (b)

(b) We agree that the comparison rests mostly on PanSt3R and that EPS3D and IGGT barely test what VC-PQ is designed to measure. To address it, we evaluated SIU3R, as requested in Q2.

In our evaluation, SIU3R, unlike PanSt3R and SegVGGT, lost a large part of its score to view consistency. On the thing classes of ScanNetv2, for NN from 5 to 40, εIoU‾\varepsilon_{\overline{\mathrm{IoU}}} of SIU3R is 45.5 to 46.8, close to that of PanSt3R v2 (48.1 to 48.6), and the sum of the view-consistency term and the matching threshold term (εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q) is 24.7 to 36.3, larger than for PanSt3R v2 (9.5 to 12.5) and SegVGGT (6.7 to 10.6) (Table R12).

Table R12. ScanNetv2, thing classes. For each method the three rows are VC-PQ, εIoU‾\varepsilon_{\overline{\mathrm{IoU}}}, and εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q, with the values of the two terms in parentheses. The three rows sum to 100 (this is Table 3 with its last two terms added, and some cells are off by 0.1 because of rounding). η\eta is 0.1% of the image area (Common 1). N=2N = 2 is a column added to match the number of views SIU3R was trained with. With two views, εq\varepsilon_q is 0 by definition.

N=2N = 2N=5N = 5N=10N = 10N=20N = 20N=40N = 40
PanSt3R v2
VC-PQ47.742.540.039.139.1
εIoU‾\varepsilon_{\overline{\mathrm{IoU}}}46.848.148.248.448.6
εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q5.5 (5.5 + 0.0)9.5 (7.6 + 1.9)11.8 (8.2 + 3.5)12.5 (6.8 + 5.7)12.4 (5.9 + 6.5)
SegVGGT
VC-PQ54.651.349.848.747.8
εIoU‾\varepsilon_{\overline{\mathrm{IoU}}}42.242.141.841.441.6
εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q3.2 (3.2 + 0.0)6.7 (5.3 + 1.3)8.4 (5.8 + 2.6)9.8 (5.4 + 4.4)10.6 (5.2 + 5.3)
SIU3R
VC-PQ41.229.722.918.716.8
εIoU‾\varepsilon_{\overline{\mathrm{IoU}}}44.345.546.046.446.8
εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q14.5 (14.5 + 0.0)24.7 (20.3 + 4.4)31.2 (23.8 + 7.4)35.0 (25.3 + 9.6)36.3 (24.2 + 12.1)

For all three methods, as NN grows from 5 to 40, VC-PQ over thing classes falls, εIoU‾\varepsilon_{\overline{\mathrm{IoU}}} stays nearly the same, and εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q grows. The change is largest for SIU3R (VC-PQ from 29.7 to 16.8, and εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q from 24.7 to 36.3).

The results of SIU3R should nevertheless be read with care. The released SIU3R is a model trained with two views, so N≥5N \ge 5 feeds it more views than it was trained with. Even at N=2N = 2, the number of views used in training, εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q of SIU3R is 14.5, larger than for PanSt3R v2 (5.5) and SegVGGT (3.2) under the same condition (Table R12). Our results cannot tell whether this difference is a property of the method itself or comes from its use outside the training range or from our running conditions (scoring the 2D outputs as they are without camera poses, and the input size).

We also evaluated three per-scene optimization methods (Panoptic Lifting, Contrastive Lift, and MVC-PSU) on the Panoptic Lifting benchmark (ScanNet, 1 scene of Replica, and 1 scene of HyperSim). In the error decomposition over thing classes on the 12 scenes of ScanNet, the sum of the view-consistency term and the matching threshold term was 5.8 for Panoptic Lifting, 8.3 for Contrastive Lift, and 10.1 for MVC-PSU, and 5.2 for PanSt3R v2. It can thus be seen that the score lost across views is not specific to PanSt3R and is present at a similar or larger size in the three methods, which follow a different approach.

Even with these additions, the limitation remains that the feed-forward methods compared under the same protocol are PanSt3R, SIU3R, and SegVGGT, the last of which predicts thing classes only.

AuthorsOpenReview ↗

Response to Reviewer 7EyV (3/3): W4, Q2

W4 — Relation to HOTA, IDF1, IGGT, and G²TAM

We agree that the four metrics mentioned (IGGT, G²TAM, HOTA, and IDF1) are close in form to the pair score of VC-PQ. Not citing HOTA and IDF1 was an omission on our part, and in the revision we will cite both metrics and state the relation to these metrics in Sec. 2.2. We will also add to the appendix a comparison table that writes the formulas of HOTA, IDF1, IGGT, G²TAM, VPQ, and STQ in the same notation.

VC-PQ differs from these metrics as follows.

  • IGGT, G²TAM. The two metrics report a mean IoU (T-mIoU, S-mIoU) and a success rate (T-SR, S-SR) as two separate values and do not combine them into a single score. The success rate of an object is 1 only when it is tracked in every view and 0 when it is not tracked in at least one view. VC(p,g)\mathrm{VC}(p,g) of VC-PQ is the share of the views in which both the prediction and the ground truth are visible among the views in which either of them is visible, and it is multiplied by IoU‾(p,g)\overline{\mathrm{IoU}}(p,g) into a single pair score, on which the matching is decided. In addition, in the two metrics the objects to be evaluated are given, there is no class-wise matching, and unmatched predictions do not enter the score.
  • HOTA. Its association score falls when an ID is split or some frames are missed, and so does VC(p,g)\mathrm{VC}(p,g). HOTA, however, matches separately in each frame and computes the score per detection. VC-PQ matches once over the whole set of input views and computes the score per instance.
  • IDF1. IDF1 scores by the number of correctly identified frames. VC-PQ counts TP, FP, and FN per instance with a pair score that averages the values of the per-view IoU.

Q2 — Evaluating SIU3R

We evaluated SIU3R as requested (Table R13). We used the checkpoint released by the authors as it is and scored its predictions on the input views over the 312 validation scenes of ScanNetv2 under the same protocol as Table 1 (NN = 5, 10, 20, 40).

There are four conditions that we should state first.

  • We could not evaluate it on ScanNet++. The only released checkpoint is trained to output the 20 classes of ScanNet and cannot predict the 100 classes of ScanNet++. The condition of "both datasets" is therefore not met.
  • The main model of the SIU3R paper is trained with two views, and since it is the only released checkpoint, we used this checkpoint. The values at N≥5N \ge 5 below are therefore obtained by feeding the model more views than it was trained with.
  • The main results of the SIU3R paper are obtained by rendering 3D Gaussians with camera poses. Our protocol does not give poses as input, so we scored the 2D outputs of the network on the input views as they are.
  • The input was resized so that its shorter side is 256, without cropping (256×352). The preprocessing of the authors is a 256×256 center crop.

Table R13. ScanNetv2, all classes. PQscene is computed as in Table 1, and η\eta of VC-PQ is 0.1% of the image area (Common 1). EPS3D could not be evaluated at N≥20N \ge 20 (out of memory).

MethodPQscene (NN = 5 / 10 / 20 / 40)VC-PQ (NN = 5 / 10 / 20 / 40)
PanSt3R v250.1 / 52.5 / 55.2 / 57.445.5 / 43.3 / 42.5 / 42.4
SIU3R44.0 / 43.1 / 41.8 / 41.433.9 / 27.6 / 23.7 / 22.0
EPS3D21.7 / 21.5 / -- / --10.5 / 9.1 / -- / --

SIU3R lies between PanSt3R v2 and EPS3D under both metrics. As NN grows from 5 to 40, the PQscene of SIU3R falls by only 2.6, from 44.0 to 41.4, whereas its VC-PQ falls by 11.9, from 33.9 to 22.0 (the VC-PQ of PanSt3R v2 falls by 3.1, from 45.5 to 42.4). That is, as input views are added, the fall of SIU3R is much larger under VC-PQ than under PQscene, and Table R12 shows in which terms this fall arises.

We also scored the predictions at N=2N = 2, the number of views used in training. The pairs of views were drawn at random following our protocol, which differs from the training of SIU3R, where pairs with overlap are used. At N=2N = 2, VC-PQ (all classes) is 44.4 for SIU3R and 50.3 for PanSt3R v2. It can be seen that the difference between the two methods is smallest, 5.9, at N=2N = 2, the number of views SIU3R was trained with, and widens to 20.4 at N=40N = 40.

In the revision, we will add a SIU3R row to the ScanNetv2 block of Table 1, to Table A4, and to Table 3, and state the four conditions above with it.

Reviewer_WYu5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes View-Consistent Panoptic Quality (VC-PQ), a new evaluation metric for multi-view panoptic segmentation. The authors argue that the commonly used scene-level Panoptic Quality (PQscene) does not adequately capture cross-view consistency because it concatenates all views into a single image before computing segmentation quality. Consequently, predictions in views where an object occupies a large area can dominate the evaluation, while missed detections or inconsistent instance identities in smaller views receive relatively limited penalties.

To address this limitation, VC-PQ extends conventional PQ by explicitly incorporating per-view segmentation accuracy and visibility consistency. Specifically, it computes the average per-view IoU over jointly visible views and combines it with a visibility-consistency factor based on the overlap between predicted and ground-truth visibility supports. Each visible view contributes equally, regardless of object size. The authors further introduce a parameterized family of metrics to analyze the influence of area weighting and propose an error decomposition that attributes performance degradation to mask accuracy, view consistency, and matching-threshold effects.

The proposed metric is evaluated on ScanNet++ and ScanNetv2 using several recent feed-forward segmentation methods, including PanSt3R, EPS3D, IGGT, and SegVGGT. Experiments demonstrate substantial differences between VC-PQ and PQscene, particularly as the number of input views increases. Controlled perturbations involving missed appearances and instance identity changes further illustrate how VC-PQ responds to cross-view inconsistencies. The authors suggest reporting VC-PQ alongside PQscene to provide a more comprehensive evaluation of multi-view panoptic segmentation.

Critical strengths

(1) The paper identifies an important limitation of existing scene-level evaluation protocols, where area-weighted aggregation can obscure segmentation failures in views with smaller object appearances. Explicitly accounting for consistency across input views is well motivated for multi-view panoptic segmentation. (2) VC-PQ extends conventional PQ through a straightforward combination of per-view mask accuracy and visibility consistency. The proposed parameterized formulation and error decomposition provide useful tools for understanding how different evaluation choices affect the final score. (3) The paper evaluates multiple feed-forward methods on two established benchmarks and includes controlled perturbations to study the behavior of the proposed metric. The analysis of area weighting, aggregation strategies, and class filtering provides useful insights into the limitations of the existing PQscene protocol.

Critical weaknesses

(1) The proposed equal-view weighting makes VC-PQ potentially sensitive to inaccurate segmentation annotations, particularly when objects occupy small image regions or are heavily occluded. This concern is supported by Table A7, where eroding ground-truth masks by only one pixel reduces VC-PQ from 100 to 61.3, compared with 85.9 for PQscene. Although the authors investigate different visibility thresholds in Table A9, the robustness of the resulting method rankings and diagnostic conclusions under realistic annotation noise remains unclear. This is particularly important because the evaluated datasets rely on projected 3D annotations, which may introduce boundary inaccuracies. (2) The visibility-consistency factor penalizes both missing object appearances and inconsistent instance identities, although these correspond to different failure mechanisms. In particular, Table A6 shows that approximately 55–92% of object appearances are not correctly covered by the evaluated methods, whereas only around 0.4–1.2% are covered by another instance ID. These results suggest that the observed penalties are largely influenced by per-view detection failures rather than explicit cross-view identity switching. While the controlled perturbations demonstrate that VC-PQ can detect identity inconsistencies, its additional diagnostic value for identity association in real model predictions is not sufficiently established. (3) Although VC-PQ produces substantially lower scores than PQscene, a considerable portion of this difference arises from changes beyond the proposed view-consistency formulation. For example, Table 2 shows that, for PanSt3R v2, the 29.2-point difference is attributed to area weighting (7.8), per-scene averaging (6.0), and class filtering (15.3). Therefore, the large overall discrepancy should not be interpreted primarily as evidence of cross-view inconsistency. Moreover, while the paper discusses video panoptic and tracking metrics, it does not provide direct empirical comparisons demonstrating the additional value of VC-PQ over existing evaluation measures combined with appropriate aggregation and filtering. (4) Although the decomposition in Eq. (7) is mathematically valid, the attribution of errors depends on specific design choices. In particular, unmatched instances are categorized through sequential tests, while the interaction between mask accuracy and visibility consistency is not allocated symmetrically. As acknowledged in the paper, the view-consistency term therefore represents a lower bound rather than a complete attribution of consistency-related errors. Consequently, interpreting these components as distinct and independent sources of failure may be misleading, especially when using the decomposition to guide improvements in segmentation models.

Questions for authors

(1) Given the substantial sensitivity to mask erosion reported in Table A7, could the authors clarify whether the relative rankings of evaluated methods and the error decomposition remain stable under realistic annotation noise? In particular, how does the choice of visibility threshold affect these conclusions across different image resolutions and object sizes? (2) Table A6 suggests that missed appearances are considerably more frequent than explicit identity inconsistencies in the evaluated predictions. Could the authors clarify how much of the additional penalty introduced by VC-PQ is attributable to actual identity association failures rather than per-view detection errors? (3) Given that Table 2 attributes a substantial portion of the difference between VC-PQ and PQscene to class filtering and aggregation, could the authors clarify what additional conclusions are uniquely enabled by the proposed view-consistency formulation? (4) Could the authors clarify how sensitive the proposed error decomposition is to the ordering of the tests used to categorize unmatched instances and the allocation of interaction terms? Would alternative attribution strategies lead to similar conclusions about the dominant sources of error?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WYu5 (1/2): W1/Q1, W2/Q2, W3/Q3

We thank the reviewer for the specific questions. The points raised by several reviewers are answered in Common 1 to 3, and this response points to the relevant parts.

W1 / Q1 — Sensitivity to annotation noise and the choice of η

Our answer to this weakness and question is given in parts 1, 3, and 4 of Common 1.

W2 / Q2 — Share of identity association failures

As the reviewer notes, for the evaluated methods most of the penalty of VC-PQ comes from per-view detection failures. The submission did not show what share of it comes from identity association failures.

To check this, we split 1−VC-PQ1 - \text{VC-PQ} into three parts by cause (Table R10). This splits the same quantity as Eq. (7) of the paper (1−VC-PQ1 - \text{VC-PQ}), not by the ordering of the tests but by which prediction covers the object in each view. η\eta is the value recommended in Common 1 (0.1% of the image area).

  • Detection: the part lost because, in a view in which the ground-truth object is visible, no prediction of the same class covers the object with an IoU above 0.5, or one does but its mask is inaccurate.
  • Identity: the part lost because a prediction covers the object with an IoU above 0.5 in that view but is not the prediction matched to the object (the IoU of that prediction minus the IoU of the matched prediction).
  • Unmatched predictions: the penalty of the predictions that are not matched, and the part lost because a matched prediction appears in a view in which the ground truth is not visible.

An unmatched ground-truth object is counted as a whole on one side. For each view in which the object is visible, the IoU of the prediction that covers it best in that view is taken, and these values are averaged; the object goes to Identity when this average exceeds 0.5 and to Detection otherwise. The three values are divided by the same denominator as PQ, so together with VC-PQ they sum to 1.

Table R10. 1−VC-PQ1 - \text{VC-PQ} split by cause (×100, thing classes, N=10N = 10). Before rounding, VC-PQ and the three columns sum to 100. ScanNet++ pools DSLR and iPhone. SIU3R is a method newly evaluated for this response.

DatasetMethodVC-PQDetectionIdentityUnmatched predictions
ScanNet++PanSt3R v122.757.22.018.2
ScanNet++PanSt3R v227.346.61.524.6
ScanNet++EPS3D0.587.80.211.5
ScanNet++IGGT4.880.00.714.5
ScanNetv2PanSt3R v240.029.53.826.7
ScanNetv2SegVGGT49.824.71.923.6
ScanNetv2SIU3R22.939.813.124.1
  • Q2: For the four methods on ScanNet++, Identity is 0.2 to 2.0 and Detection is 46.6 to 87.8. The part that comes from identity association failures is small.
  • W2: For the three methods on ScanNetv2, Identity is 1.9 for SegVGGT, 3.8 for PanSt3R v2, and 13.1 for SIU3R, and thus differs by method. For PanSt3R v2 the value is small as a class average, 1.5 (ScanNet++) and 3.8 (ScanNetv2). Class by class, however, for chair, the class with the most objects, the loss due to Identity switching is 6.0 and 7.8, the largest among all classes in both datasets.

PanSt3R v2, which is evaluated on both datasets, has values that differ by dataset (Detection 46.6 and 29.5). This is because the values in the table are class averages and the two datasets differ in their classes (the class average covers 83 and 18 thing classes). For chair alone, the loss due to Detection is 41.2 and 33.6, a smaller gap.

In the revision we will add Table R10 to the appendix.

W3 / Q3 — Direct comparison with existing metrics

As the reviewer says, the whole of the difference between PQscene and VC-PQ should not be read as evidence of cross-view inconsistency. Per-scene averaging and class filtering raise the value of PQscene, and in Table 2 of the paper 21.3 of the 29.2 points of difference between PQscene and VC-PQ come from them.

The direct comparison with existing metrics that the reviewer asks for is given in Common 2. On Q3, for PanSt3R v2 (ScanNet++), as NN grows from 5 to 40, per-image PQ, VPQ, STQ, and PQscene (tiling only) move by at most 0.7, and these values alone lead to the judgment that the quality is kept as views are added. On the same predictions VC-PQ falls from 29.1 to 23.8, and it can be seen that the cases in which an object is matched in all of the views in which it is visible have decreased.

AuthorsOpenReview ↗

Response to Reviewer WYu5 (2/2): W4/Q4

W4 / Q4 — Order dependence of the decomposition

As the reviewer says, the attribution of Eq. (7) depends on the ordering of the tests and on the allocation of the cross term, and we agree with this point. In the decomposition we applied the mean IoU test first. What changes with the ordering and the allocation is the part in which both the mean IoU and the view consistency are low. For matched pairs it is the cross term (1−IoU‾)(1−VC)(1 - \overline{\mathrm{IoU}})(1 - \mathrm{VC}), and for unmatched instances it is the instances that pass neither of the two tests (lower left of Fig. A3). Table R11 separates this part as εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}} and the rest of εIoU‾\varepsilon_{\overline{\mathrm{IoU}}} as εIoU‾-up\varepsilon_{\overline{\mathrm{IoU}}\text{-up}}.

Table R11. The decomposition with the mean IoU term of Eq. (7) split in two (thing classes). η\eta is 0.1% of the image area (100 px in Table 3). The five columns sum to 100, and εIoU‾-up+εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-up}} + \varepsilon_{\overline{\mathrm{IoU}}\text{-down}} is εIoU‾\varepsilon_{\overline{\mathrm{IoU}}} of Eq. (7).

DatasetMethodN=10N = 10: VC-PQεIoU‾-up\varepsilon_{\overline{\mathrm{IoU}}\text{-up}}εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}}εVC\varepsilon_{\mathrm{VC}}εq\varepsilon_qN=40N = 40: VC-PQεIoU‾-up\varepsilon_{\overline{\mathrm{IoU}}\text{-up}}εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}}εVC\varepsilon_{\mathrm{VC}}εq\varepsilon_q
ScanNet++PanSt3R v122.719.251.04.62.620.417.853.04.34.6
ScanNet++PanSt3R v227.321.641.76.72.824.119.043.17.66.3
ScanNet++EPS3D0.56.891.80.60.3----------
ScanNet++IGGT4.87.584.82.20.7----------
ScanNetv2PanSt3R v240.023.324.98.23.539.124.424.25.96.5
ScanNetv2SegVGGT49.821.620.25.82.647.820.221.45.25.3

εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}} is 20.2 to 91.8 in the table. Of this, the cross term of the matched pairs is at most 0.7, so changing the allocation of the cross term changes each term by no more than 0.7. If the view-consistency test is applied first, εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}} is attributed to view consistency. The score attributed to view consistency is then at most εVC+εIoU‾-down\varepsilon_{\mathrm{VC}} + \varepsilon_{\overline{\mathrm{IoU}}\text{-down}}, and this value is larger than εIoU‾-up\varepsilon_{\overline{\mathrm{IoU}}\text{-up}} in every row. Which term is the largest therefore depends on the ordering of the tests.

Our reason for applying the mean IoU test first is the following. εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}} holds the instances that pass neither of the two tests. For such an instance no prediction overlaps its mask by more than half and no prediction agrees with it in more than half of the views in which either is visible, and the instances that the predictions miss altogether are also included. If the view-consistency test is applied first, all of these instances are counted as failures of view consistency. VC(p,g)\mathrm{VC}(p,g), however, compares only the sets of views in which the two instances are visible and does not look at the masks. Saying "the visible views do not agree" of a prediction whose mask does not even overlap the instance tells nothing about the view consistency of that instance. We therefore applied the mean IoU test first, so that εVC\varepsilon_{\mathrm{VC}} means only "the score lost by instances that the prediction has found by mask and that disagree in their views" (Appendix E). εVC\varepsilon_{\mathrm{VC}} defined in this way is the narrowest measure of the score lost to view consistency (a lower bound). εVC+εIoU‾-down\varepsilon_{\mathrm{VC}} + \varepsilon_{\overline{\mathrm{IoU}}\text{-down}}, which also assigns to view consistency all of εIoU‾-down\varepsilon_{\overline{\mathrm{IoU}}\text{-down}}, where the two causes cannot be told apart, is the widest measure (an upper bound).

In the revision we will rewrite, throughout the main text, the sentences that describe the three terms as if they were independent sources of failure, in line with this range.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Common 3: Can VC-PQ be used for more than exposing failures? (7EyV W3 · iw5X W1/W2)

As the reviewers point out, the submission went only as far as exposing failures with VC-PQ and did not show what changes when the metric is used in training. To check this, we fine-tuned the released PanSt3R v1 with the pair score of PQscene and the pair score of VC-PQ each used as a training loss.

Experimental design. We kept the backbone frozen and fine-tuned only the panoptic decoder for 20 epochs on the training scenes of ScanNet++ (excluding the 50 evaluation scenes), with all other training settings as released for PanSt3R. To these we added one loss term for every matched pair. The VC-PQ loss carries the pair score of VC-PQ over to the soft IoU, which is computed in every view in which the ground truth or the prediction is visible and then averaged. The PQscene loss carries the pair score of PQscene over in the same way, with the points of all views concatenated into a single soft IoU. We compared the two models, each with one of the two losses added, with the model fine-tuned without an additional loss (control); the three models differ only in the added loss term, and each model was trained once. The evaluation follows the same ScanNet++ protocol as Table 1, with the standard post-processing of PanSt3R.

Results.

Table R9. The three models fine-tuned from PanSt3R v1 (ScanNet++, η\eta = 0.1% of the image area; the η\eta of the loss used in training is 100 px). The three score columns are over all classes and the two error decomposition columns over thing classes (the paper's test order). The last column is the sum of the view-consistency term and the matching threshold term, with the two terms in parentheses.

NNFine-tunePQscene (tiling only)VC-PQ$_1$VC-PQεIoU‾\varepsilon_{\overline{\mathrm{IoU}}}εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q
10control25.327.923.767.28.6 (6.0 + 2.6)
10+ PQscene loss26.729.424.965.29.3 (6.6 + 2.7)
10+ VC-PQ loss26.829.625.265.58.7 (6.1 + 2.7)
40control25.126.520.866.912.1 (6.1 + 6.0)
40+ PQscene loss26.928.021.764.613.3 (6.8 + 6.5)
40+ VC-PQ loss26.928.122.265.911.5 (5.5 + 6.0)

(a) Adding only one loss term built from the pair score of a metric raised the scores. With either loss, PQscene (tiling only), VC-PQ$_1$, and VC-PQ rose above the control (Table R9). At N=40N = 40, VC-PQ rose by +0.98 [+0.01, +1.49] for the model with the additional PQscene loss and by +1.45 [+0.83, +2.27] for the model with the additional VC-PQ loss, and at N=10N = 10 by +1.24 [+0.50, +2.03] and +1.55 [+0.86, +2.51] (95% intervals, the same scene-level bootstrap as Table A3 of the paper).

(b) The difference between the two losses was on the view-consistency side. The score lost across views grows with NN (for the control εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q is 8.6 at N=10N = 10 and 12.1 at N=40N = 40), so we looked at N=40N = 40. Here the two models with an additional loss rose by the same amount in the area-weighted score (PQscene (tiling only) from 25.1 to 26.9 and 26.9, +1.8). VC-PQ, which counts views with equal weight and is harder to raise, rose by +0.98 for the model with the additional PQscene loss and by +1.45 for the model with the additional VC-PQ loss (difference between the two losses +0.47 [−0.12, +1.22]). In the error decomposition, the model with the additional PQscene loss has the smaller mean IoU term (εIoU‾\varepsilon_{\overline{\mathrm{IoU}}}) (64.6 and 65.9), and the model with the additional VC-PQ loss has the smaller sum of the view-consistency term and the matching threshold term (εVC+εq\varepsilon_{\mathrm{VC}} + \varepsilon_q) (13.3 and 11.5, difference −1.79 [−2.69, −0.75]). For the model with the additional PQscene loss this sum increased over the control (+1.20 [+0.21, +1.98]). At the other NN (5, 10, 20) as well, this sum was larger for the model with the additional PQscene loss. This difference is in the direction expected from the definitions of the two losses. The PQscene loss concatenates the points of all views into a single soft IoU and therefore grows only slightly when there is a view in which only one of the prediction and the ground truth is visible, whereas the VC-PQ loss enters such a view into the average with equal weight and therefore grows when the two sets of views disagree.

Summary. Within the scope of this experiment, the pair score of VC-PQ could be used not only as an evaluation metric but also as a training signal.

In the revision we will include the design and results of this experiment in the main text.

AuthorsOpenReview ↗

Common 2 (1/2): What does VC-PQ show that existing metrics do not? (WYu5 W3/Q3 · 7EyV W3/Q1 · iw5X W1/W2) — (a), (b)

The question the three reviewers share is what VC-PQ shows beyond existing metrics. The submission contained no direct comparison with existing metrics. To check this, we computed per-image PQ, VPQ, and STQ on the same predictions and computed PQscene with its aggregation matched to that of VC-PQ and without the class filter (tiling only) (Tables R7 and R8). VC-PQ$_1$ is VC-PQ with only the weighting of views changed to the area (γ=1\gamma = 1).

(a) Change with the number of input views. For PanSt3R v2 (ScanNet++) and SegVGGT (ScanNetv2), the four existing metrics do not show how many of the views in which an object is visible are not matched, whereas with VC-PQ this can be seen more clearly. For PanSt3R v2, as NN grows from 5 to 40, per-image PQ, VPQ, STQ, and PQscene (tiling only) move by at most 0.7 (Table R7), and these values alone lead to the judgment that the quality is kept as views are added. On the same predictions VC-PQ falls from 29.1 to 23.8, and it can be seen that the cases in which an object is matched in all of the views in which it is visible have decreased. For SegVGGT (thing classes) as well, the four metrics move by at most 1.2, and VC-PQ falls from 51.3 to 47.8.

Table R7. The same predictions scored in six ways, by the number of input views NN. VPQ is the average of the values over the window lengths. η\eta (0.1% of the image area) applies only to VC-PQ$_1$ and VC-PQ. SegVGGT predicts thing classes only, and in its block STQ is computed with the segmentation term replaced by the IoU averaged over thing classes.

MetricN=5N = 5N=10N = 10N=20N = 20N=40N = 40
PanSt3R v2 (ScanNet++, all classes)
per-image PQ28.128.328.028.1
VPQ28.328.327.927.6
STQ34.734.634.434.7
PQscene (tiling only)28.628.628.629.1
VC-PQ$_1$32.231.430.630.5
VC-PQ29.126.624.823.8
SegVGGT (ScanNetv2, thing classes)
per-image PQ56.356.757.257.5
VPQ55.355.255.154.9
STQ60.360.461.061.2
PQscene (tiling only)54.254.254.353.9
VC-PQ$_1$55.855.855.955.6
VC-PQ51.349.848.747.8

VC-PQ$_1$ and VC-PQ differ only in the weighting of views. As NN grows from 5 to 40, VC-PQ$_1$ falls by 1.7 for PanSt3R v2 and by 0.2 for SegVGGT, and VC-PQ falls by 5.3 and 3.5. The fall with NN thus shows not in the aggregation or the class filter but in counting views with equal weight.

(b) Comparison across methods (N=10N = 10, Table R8).

Table R8. The values of the same six ways of scoring at N=10N = 10, by method. STQ in the thing classes group is computed as in Table R7.

Methodper-image PQVPQSTQPQscene (tiling only)VC-PQ$_1$VC-PQ
ScanNet++ (all classes)
PanSt3R v123.924.128.024.627.022.7
PanSt3R v228.328.334.628.631.426.6
EPS3D2.62.67.12.63.02.2
IGGT5.35.711.76.16.65.2
ScanNetv2 (all classes)
PanSt3R v251.350.251.449.850.243.3
SIU3R48.343.135.539.139.427.6
EPS3D10.510.426.410.211.69.1
ScanNetv2 (thing classes)
SegVGGT56.755.260.454.255.849.8
PanSt3R v248.747.151.746.446.940.0
SIU3R45.539.435.634.735.022.9

With the aggregation matched to that of VC-PQ and without the class filter, PQscene of PanSt3R v2 (ScanNet++, N=10N = 10) goes from 51.3 in Table 1 to 28.6, at the level of per-image PQ (28.3) and VPQ (28.3). That is, the high PQscene of Table 1 comes from the aggregation and the class filter. The method rankings are the same under all six ways of scoring, because the methods evaluated here already differ widely in per-view panoptic quality. The four methods on ScanNet++ are spread over 28.3, 23.9, 5.3, and 2.6 in per-image PQ, which scores each view independently, so their ranking is decided before view consistency separates them. Over the thing classes of ScanNetv2 as well, the order in per-image PQ (SegVGGT 56.7, PanSt3R v2 48.7, SIU3R 45.5) is the same as the order in VC-PQ (49.8, 40.0, 22.9). SIU3R, however, is a model trained on two views, and the results of its authors are obtained on outputs rendered from the predicted 3D Gaussians with camera poses. We gave no poses, fed the views of N=10N = 10, and scored the 2D outputs of the network on the input views as they are. Under these conditions SIU3R was close to PanSt3R v2 in per-image PQ and yet far lower in VC-PQ. That is, two methods that are close in the quality of single views were separated under VC-PQ.

AuthorsOpenReview ↗

Common 2 (2/2): (c) and Summary (WYu5 W3/Q3 · 7EyV W3/Q1 · iw5X W1/W2)

(c) Why existing metrics do not show the fall with NN. The reason existing metrics do not show the fall of PanSt3R v2 lies in the unit that is counted. Per-image PQ scores each view independently, so the scoring result of an object does not carry over from one view to another. VPQ evaluates the pixels of the views within a window pooled together, and STQ and PQscene those of all views, so that, as the three reviewers wrote of PQscene, the views in which an object appears large dominate the score and a miss in a view in which it appears small is hidden. VC-PQ takes a single object as its unit and counts the views in which that object is visible with equal weight. When the set of views in which the ground truth is visible and the set of views in which the prediction is visible disagree, the view-consistency factor lowers the pair score by that ratio, and when the pair score does not exceed 0.5 the object is counted as missed. This criterion is set by the fraction of views that are matched, so the score of a method that matches an object evenly over the views in which it is visible does not fall as NN grows. We attribute the fall of the score of PanSt3R v2 not to predictions that become worse as views are added but to the exposure of objects that were being matched in some views and missed in others. For PanSt3R v2 (ScanNet++), the fraction of single views that are matched (the fraction of views in which the prediction that fits the object best has an IoU above 0.5 in that view) is the same, 56.6% at N=5N = 5 and 55.8% at N=40N = 40. Among the objects visible in five or more views at N=40N = 40, however, 60% were matched in only some of their views. At N=5N = 5 an object is visible in only 2.0 views on average, which leaves almost no room for matched and missed views of the same object to show together, whereas at N=40N = 40 it is visible in 9.8 views on average and they show as they are. As a result, the objects matched in more than half of the views in which they are visible decrease from 47.7% to 42.8%.

For SegVGGT (ScanNetv2, thing classes) as well, the fraction of single views that are matched is the same, 70.9% at N=5N = 5 and 70.7% at N=40N = 40. The views in which an object is visible increase from 1.9 to 11.0 on average. Among the objects visible in five or more views, those matched in all of their views are 64.6% at N=5N = 5 and 30.0% at N=40N = 40, and those matched in only some of their views increase from 31.0% to 58.4%. For SegVGGT as well, it can thus be seen that the fraction of single views that are matched stays the same while, as the views in which an object is visible increase, the objects matched in only some of their views increase.

Summary. For PanSt3R v2 (ScanNet++), the four existing metrics (per-image PQ, VPQ, STQ, PQscene (tiling only)) move by at most 0.7 as NN grows from 5 to 40, and on the same predictions VC-PQ falls from 29.1 to 23.8. PQscene evaluates the pixels of all views pooled together, so a prediction that is right only in the views in which an object appears large still receives a high score, and the consistency across views is not properly reflected in the score. A high score under VC-PQ requires that the set of views in which the prediction is visible agree with the set of views in which the ground truth is visible (VC(p,g)\mathrm{VC}(p,g), the Jaccard index of the two sets) and that the IoU be high in each of those views. When both conditions are met, the prediction is in fact consistent across views. VC-PQ therefore asks, in addition to the quality of the per-view masks, that the agreement of the two sets of views be reflected in training.

In the revision we will add Tables R7 and R8 to the appendix and a paragraph on the comparison with existing metrics to Sec. 5.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.