acceptodds
Under review as a conference paper at ICLR 2027

W3D: Where, Who, and Why? Diagnosing Multi-View 3D Fusion

Abstract

Multi-view fusion assessment should explain local geometric errors rather than merely assign reliability scores. We introduce W3D, a query-centric framework that formulates fusion assessment as three coupled diagnostic objectives: Where localizes unreliable geometry, Who traces view-level influence through leave-one-view-out effects, and Why predicts multi-label observation factors. W3D reconnects each fused-surface query to its original multi-view observations using a point-normal-residual (PNR) representation that captures relative depth, viewing geometry, and surface orientation. A shared Set Transformer supports gated Where Who Why message passing, while stop-gradient operations decouple task-head updates through these messages without preventing joint learning of shared features. Supervision combines reference geometry with leave-one-view-out re-fusion, whereas inference requires only the fused surface, original observations, and camera calibration. On Skoltech3D, W3D consistently improves all six diagnostic metrics over the matched parallel Base with identical PNR inputs, remains effective across varying numbers of input views, and supports multiple fusion algorithms under the same diagnostic formulation through separate training. These results show that local fusion quality can be expressed as a traceable diagnosis linking geometric errors to their supporting observations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-view 3D Fusion, Structured Fusion Diagnosis, View Tracing, Multi-label Factor Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7ZfkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes W3D, a query-centric framework for diagnosing errors in multi-view 3D fusion through three related tasks: Where, which localizes unreliable fused geometry; Who, which identifies views whose removal improves local reconstruction quality; and Why, which predicts four predefined observation factors: Bias, Residual Outlier, Noise, and Multi-view Conflict. Each fused-surface query is linked back to its multi-view observations using an 11-dimensional point-normal-residual (PNR) representation. A Set Transformer encodes the observation set, followed by gated Where→Who→Why message passing with stop-gradient connections between task heads. Supervision is constructed using reference geometry and leave-one-view-out re-fusion, while inference requires only the fused surface, observations, and camera calibration. Experiments on Skoltech3D show substantial gains from the PNR representation and smaller additional gains from the proposed detached-gated architecture. The paper also evaluates different view counts, four fusion procedures, local view-removal utility, DTU, and ETH3D.

Critical strengths
  1. The Where/Who/Why formulation links a fused-surface error to both its supporting observations and interpretable observation-pattern labels. This is more informative than simply predicting a scalar uncertainty or confidence value and constitutes the most distinctive contribution of the paper.

  2. The progressive PNR ablation shows that relative-depth features provide a substantial contribution, while the paper additionally evaluates multiple fusion procedures, different input-view counts, frozen cross-count transfer, geometric baselines, per-seed results, coverage statistics, factor-level performance, and an explicit negative ETH3D boundary case. The authors also appropriately acknowledge that the Who and Why targets do not establish physical causality.

Critical weaknesses
  1. At identical PNR inputs, W3D improves the parallel Base on all six reported metrics, but the model introduces additional adapters and parameters while simultaneously changing gating, cross-task conditioning, and gradient routing. The manuscript itself states that the comparison is not parameter matched and that the architectural factor evaluates gating and gradient decoupling jointly. Consequently, the current results demonstrate an advantage for the complete larger architecture, but do not establish that the proposed Where→Who→Why detached hierarchy is the cause. A parameter-matched baseline and path-specific ablations are important because the architectural gains are relatively modest for most metrics compared with the much larger gains obtained from PNR.

  2. Who requires a Bad query, complete leave-one-view-out evidence, and a positive removable effect, so only a fraction of constructed queries are Who-valid; this becomes particularly restrictive for Screened Poisson. In addition, the Bias and Noise Why labels are almost always positive on seven-view TSDF, making their AP values close to one largely because of class prevalence. The paper acknowledges these issues in the appendix and limitations, but they should be integrated into the main evaluation because they materially affect how the reported Who and Why performance should be interpreted.

Questions for authors
  1. Can the authors provide a parameter-matched Base+PNR comparison, or an adapters-only control, to determine whether the fixed-PNR improvements remain after controlling for the additional parameters introduced by W3D? This would materially affect my assessment of the claimed architectural contribution.

  2. Please provide targeted ablations separating the effects of cross-task message passing and stop-gradient, for example adapters without messages, messages without stop-gradient, and removal of individual Where→Who or Who→Why paths? The current experiment evaluates these mechanisms jointly, so it is not possible to determine whether the proposed hierarchy itself is responsible for the improvements.

  3. Please report Who coverage alongside Who Top-1/Mass in the main results, particularly for the four fusion methods? The method appears to provide valid Who supervision for only a subset of reconstructed queries, and I would like to understand how much of the reconstructed surface can actually receive meaningful view attribution.

  4. If Why performance is evaluated using only the less-degenerate Residual Outlier and Conflict factors, does W3D retain a clear advantage over Base+PNR? This would help determine how much of the reported Why AP is driven by the extremely high positive prevalence of Bias and Noise.

Reviewer_zTP6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To the best of my understanding, the paper presents an approach for fusing multi-view depth maps in a way that can trace errors and identify their causes. In some unclear way, this is facilitated by computing all leave-one-out fusion outputs. The approach consists of three stages, named Where, Who and Why. As I have already indicated, following the descriptions in the paper is hard. Understanding the validation is not easier: experiments are presented in a scattered fashion where different baselines are compared to the proposed approach, named W3D, on different datasets.

Critical strengths

The proposed approach to fusion is novel; it aims to detect errors and diagnose their causes, while the majority of the literature aims to prevent errors from being fused or to reduce their weights in the fusion process. I am not convinced that the proposed approach is better, but it is along an unexplored path.

Critical weaknesses

I apologize in advance for stretching the definition of this section. The two weaknesses I discuss below are broad, but I think that they should be considered in the decision-making process.

The first major weakness of the paper is presentation. There is no section that is sufficiently clear and can be understood easily. I had to reach Section 5.7 to become convinced that the inputs being fused are depth maps estimated using Multi-View Stereo (MVS). Up to that point, I also considered the case of the inputs being multiple RGB-D scans, based on the citation to the work of Voynov et al. (2023) that introduces the dataset used for training. (The dataset contains RGB-D scans from various sensors as well as data that can be used as input to MVS.) The output is also poorly defined. It consists of 6 per-view confidence/diagnostic maps, two for each of the Ws, but these metrics are introduced in Section 5.1 without being defined. A paper must specify the inputs and outputs of the algorithm it proposes!

Section 4.1 begins with: “The raw7 baseline contains normalized image coordinates, absolute observed depth, incidence cosine, and three view-slot codes: linear, squared, and sinusoidal functions of the view index. PNR replaces absolute depth with relative depth, removes slot codes, and adds normalized camera coordinates and camera-frame normals. Section 5.3 evaluates these cumulative changes.” This is entirely undecipherable: raw7 has not been mentioned before and is not defined later either, “view-slot” does not appear again in the paper. The term “incidence cosine kappa” appears in the next paragraph, as if incidence cosine is typically used in this context. Acronyms and symbols such as PMA, ISAB, rFF and z (Line 186 on) should be defined before being used.

The entire paper is written in this style. Line 218: “Reference geometry defines offline supervision.” Line 225: “Who requires Bad queries, complete counterfactual evidence, and positive effects representable by the input mask.” I am aware that Who is the name of a module, but these sentences make no sense. Queries are mentioned often but not defined, while counterfactual evidence and the input mask are mentioned once in the main paper and once in the appendix without definitions.

I will not copy Section 6 here, but I am having large difficulty understanding any of the sentences.

The second major weakness is the lack of comprehensible validation. The 6 metrics “introduced” here have not been used in any other publication, so they do not give a sense of whether W3D performs well. More importantly, W3D is not compared to any other confidence/uncertainty estimators. Table 3 shows W3D applied on various fusion methods without specifying the input depth maps, while Table 5 shows results of one uncertainty estimator on one dataset and a mixed set of different MVS backbones and datasets. According to Lines 426-427: “We compare Where with Trust3R uncertainty on ETH3D, Who with native Vis-MVSNet and PatchMatchNet view-score proxies on DTU, and Why with a Bayesian-last-layer variant on Skoltech3D.”

Questions for authors

I do not think that the paper can be explained sufficiently well in the discussion phase, but I do have a question for the authors. I was wondering why do they choose to fuse all inputs first and then detect errors, instead of preventing errors via selection or robustness like previous fusion methods? I also wonder whether leave-one-out is sufficient or whether one would have to also consider leave-N-out variants when there are many input depth maps or the noise level is high.

Reviewer_MT3oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes the use of three rule-based logics Where, Who, and Why to diagnose the multi-view 3D fusion process. This approach can predict low-quality points with high uncertainty within the fusion results, identify the specific views associated with these points, and analyze the underlying causes. Experiments demonstrate that this method can successfully perform such predictions on the Skoltech3D dataset.

Critical strengths

None

Critical weaknesses
  1. This paper proposes a diagnostic framework for multi-view fusion, designed to pinpoint issues that arise during the fusion process. However, the paper fails to explain how these diagnostic findings might be meaningfully utilized in subsequent stages, nor does it clearly articulate the specific purpose of such an approach. Consequently, I have reservations regarding the rationale behind this work.
  2. Figure 1 in the paper is highly perplexing. There is strange white background noise in the images; furthermore, the shape of the rabbit’s surface varies each time it appears—which is quite confusing, as shouldn't these all be derived from the same dataset sample? Additionally, the red annotations in the figures are consistently placed in the same locations; in particular, subfigure #1 clearly contains some extraneous and irrelevant content. I hope the authors can provide a reasonable explanation for these issues.
  3. The technical contribution of this paper is also quite limited. All baseline methods are merely ablation variants of W3D, and the paper fails to provide comparisons with existing uncertainty/confidence estimation methods trained under identical supervision conditions.
  4. The data in the experimental table is too densely packed, making it difficult to read. Optimizing the way it is presented would be an improvement.
  5. Figure 3 is rather difficult to interpret. The authors previously emphasized that their method can identify which specific views—among those being fused—are the source of problems. What exactly are they trying to demonstrate here?
Questions for authors
  1. Explain the soundness of this paper.
  2. Explain figure 1 and figure 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.