- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces MAGIC, a multi-baseline framework for unsupervised stereo matching.
The main challenge for unsupervised stereo models consists of retrieving a proper supervisory signal in occluded regions, where the classical photometric consistency loss cannot help due to the lack of correspondences across views. To this end, MAGIC assumes the availability of multi-baseline data and exploits the different occlusion patterns occurring between the same reference image and two distinct target views and , collected respectively at baseline distances and .
MAGIC employs an EMA teacher-student scheme, in which the teacher and the students process the two stereo pairs and , respectively. This allows the student to obtain reliable supervision on occluded regions in that appear to be disoccluded in the pair, by properly identifying these regions and weighting the loss function there accordingly. To train stereo models with the MAGIC recipe, a new multi-baseline dataset rendered with CARLA is proposed (MBS20K).
Experimental results with different stereo backbones validate the effectiveness of the proposed approach under different settings: zero-shot generalization across datasets and weather conditions, and unsupervised fine-tuning on specific datasets. Some ablation studies are conducted to properly assess the impact of the different loss terms and design strategies.
Critical strengths
Two major strengths characterize this paper:
-
The main strength of this paper definitely lies in the quantitative results reported in Table 1. Indeed, state-of-the-art models like MonSter and FoundationStereo are slightly improved when trained in an unsupervised manner within the MAGIC framework, particularly on occluded regions. If the code and trained models are made publicly available upon acceptance, this would be the main contribution of this work.
-
A secondary strength of this work is the proposed MBS20K dataset: indeed, the effectiveness of MAGIC-trained models is probably also due to the quality of the data being rendered. In particular, the improvements across weather conditions shown in Tab. 2 probably stem from the diverse weather and lightning conditions exposed by MBS20K.
Critical weaknesses
The paper, unfortunately, exposes several critical weaknesses:
-
Although the proposed solution is sound and principled, it is not novel. The use of multi-baseline data for unsupervised or semi-supervised stereo has already been proposed by SMV (Yuan et al., 2021) and NeRFStereo (Tosi et al., 2023). Both introduce loss terms specifically designed to handle occlusions by exploiting photometric consistency across multiple target images compensating each others for occlusions.
-
A fair comparison against SMV and NeRFStereo should be provided to support that the proposed technique is more effective than both. Such a comparison is missing in the paper and is a critical omission: in Tab. 1, an entry for NeRFStereo is reported, using a different backbone (RAFT-Stereo) and without any SceneFlow initialization (which is, in turn, used by any MAGIC variant), while no entry for SMV is reported at all.
-
The need for the MBS20K dataset is unclear, and not sufficiently supported by experiments. NeRFStereo dataset already provides multi-baseline data with three different baselines, so using this dataset would have been a natural choice. SMV also provides multi-scopic data, still with constant baselines. A comparison between the same models trained on the different datasets is needed to validate the quality of the proposed MBS20K data.
-
Experiments reported in Tab. 3 are inherently flawed. Using MonSter weights to warp images and generate multi-baseline data breaks the unsupervised assumption, since MonSter is trained with ground-truth supervision. The whole comparison against unsupervised methods becomes, therefore, unfair.
Questions for authors
The critical weaknesses raised in the previous form must be addressed to make the paper acceptable at this venue.
Therefore, my main questions for the authors are:
-
Compare against NeRFStereo and SMV methods using the same backbone (MonSter or FoundationStereo), without any SceneFlow initialization, and on the same dataset, e.g., MBS20K. This is crucial to prove that the minimum photometric consistency loss proposed by the former, or the cross-view supervision proposed by the latter, is insufficient to handle occlusions during training.
-
Highlight the need for MBS20K by repeating the previous experiment with both NeRFStereo data and SMV data. This is crucial to prove that the MBS20K dataset plays a role in the results achieved, and to decouple the impact of the data from that of MAGIC.
-
For Tab. 3, either 1) just fine-tune the model on ground truth as usual and submit to the benchmark. This would prove that MAGIC allows for better pre-training, which transfers into better performance even after fine-tuning; or 2) generate the multi-baseline using MAGIC models themselves (or clarify if this is the case already). The simulated multi-baseline setup is flawed, since it stems from the availability of a model (MonSter) pre-trained with ground truth, making the whole process no longer unsupervised. Furthermore, DualNet (previous state-of-the-art in this experiment) also uses a different backbone (CFNet). A comparison using the same backbone would be appreciated
Post-discussion update
During the discussion period, the authors provided the requested missing comparisons with prior work, as well as addressed my concern about the unsupervised adaptation setting, proposing a new experiment replacing MonSter rendering with 4DGS-based rendering, which can be considered effectively unsupervised.
The authors also uploaded a revised version of the paper, implementing any of the changes that were discussed.
In light of this, I am glad to adjust my initial recommendation (Clear Reject) and increase it to Weak Accept.
8 comments
Table I. Supplementary results to Table 1 in the main paper. Quantitative comparisons across different methods and datasets are provided.
| Method | Training Data | KITTI 2015 OCC | NOC | ALL | KITTI 2012 OCC | NOC | ALL | Middlebury OCC | NOC | ALL | ETH3D OCC | NOC | ALL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SMV-CFNet | MBS20K | 20.23 | 9.38 | 9.64 | 49.38 | 11.61 | 12.49 | 53.63 | 19.89 | 24.31 | 22.54 | 13.80 | 14.36 |
| NeRFStereo-CFNet | MBS20K | 11.78 | 6.10 | 6.26 | 28.70 | 5.46 | 5.99 | 44.54 | 12.13 | 16.56 | 17.11 | 12.00 | 12.31 |
| MAGIC-CFNet | MBS20K | 10.95 | 4.29 | 4.42 | 27.48 | 3.46 | 4.00 | 35.55 | 10.47 | 13.97 | 8.66 | 2.93 | 3.23 |
| SMV-CFNet | NeRFStereo | 18.55 | 5.38 | 5.68 | 32.75 | 5.05 | 5.68 | 43.20 | 13.13 | 17.17 | 35.97 | 21.79 | 22.46 |
| NeRFStereo-CFNet | NeRFStereo | 17.67 | 4.80 | 5.20 | 29.26 | 3.98 | 4.54 | 35.06 | 9.05 | 12.61 | 16.67 | 5.52 | 6.01 |
| MAGIC-CFNet | NeRFStereo | 13.18 | 4.68 | 4.88 | 31.59 | 3.60 | 4.24 | 28.46 | 8.13 | 10.93 | 13.46 | 5.33 | 5.74 |
| MAGIC-IGEV | MBS20K | 10.37 | 21.49 | 7.98 | |||||||||
| MAGIC-IGEV | NeRFStereo | 3.98 | 4.15 | 2.68 | 3.11 | 22.14 | 5.48 | 7.84 | 2.10 | 2.35 |
Table II. Supplementary results to Table 2 in the main paper. Quantitative comparisons across different methods and datasets are provided.
| Method | Training Data | Sunny | Cloudy | Rainy | Foggy |
|---|---|---|---|---|---|
| SMV-CFNet | MBS20K | 8.19 | 8.25 | 12.91 | 12.29 |
| NeRFStereo-CFNet | MBS20K | 5.65 | 3.78 | 8.04 | 8.92 |
| MAGIC-CFNet | MBS20K | 1.97 | 1.60 | 4.68 | |
| SMV-CFNet | NeRFStereo | 3.84 | 3.86 | 9.68 | 6.32 |
| NeRFStereo-CFNet | NeRFStereo | 3.08 | 3.68 | 6.79 | 4.23 |
| MAGIC-CFNet | NeRFStereo | 2.55 | 2.75 | 7.99 | 3.40 |
| MAGIC-IGEV | MBS20K | 1.64 | |||
| MAGIC-IGEV | NeRFStereo | 2.70 | 2.39 | 7.52 |
Q2: Necessity of MBS20K.
As shown in Tables I and II, MAGIC trained on MBS20K outperforms its counterpart trained on the NeRFStereo dataset on most occlusion-region metrics and consistently achieves better results across all four weather conditions. We attribute these improvements to the complementary advantages of MBS20K. The NeRFStereo dataset relies on NeRF-based novel-view reconstruction, which may introduce reconstruction artifacts, particularly in occluded regions, potentially affecting the learning of occlusion completion. In contrast, MBS20K is collected using simulation engines, providing geometrically consistent multi-view observations while covering diverse weather conditions. These properties help improve stereo matching robustness in challenging occlusion regions and adverse weather. We further evaluate MAGIC with the IGEVStereo backbone and observe consistent trends.
Q3. Evaluation on the public KITTI benchmark.
(1) Clarification of the two fine-tuning settings. We first evaluate MAGIC under standard binocular unsupervised fine-tuning. This setting, using MAGIC-IGEV$^\ast$, provides a direct and fair comparison with existing unsupervised methods and achieves state-of-the-art performance on KITTI 2015 and 2012, demonstrating the effectiveness of our pre-training. We further explore whether additional multi-baseline observations can bring improvements during target-domain adaptation. Since KITTI does not provide multi-baseline images, we synthesize additional views through warping and inpainting using MonSter-estimated disparities. We acknowledge that MonSter introduces a supervised-model prior in constructing these input views. However, neither ground-truth disparities nor MonSter predictions are used as supervision targets during the subsequent MAGIC fine-tuning, which is optimized entirely with our unsupervised objectives. Thus, we consider MAGIC-IGEV$^\S$ an unsupervised fine-tuning setting, as the additional information is introduced only through the input views, without using ground-truth or MonSter-estimated disparities as supervision targets.
(2) Comparison with CFNet on KITTI 2015 benchmark. Following the reviewer's suggestion, we further fine-tune MAGIC-pretrained CFNet using ground-truth disparities from the KITTI 2015 training set. As shown in Table III, MAGIC-CFNet consistently outperforms the original CFNet across all six metrics. In particular, D1-all decreases from 1.88% to 1.76% in ALL regions and from 1.73% to 1.60% in NOC regions. These results demonstrate that the benefits of MAGIC pre-training persist after conventional supervised fine-tuning, providing a better initialization for subsequent target-domain adaptation.
Table III. Comparison on the KITTI 2015 benchmark.
| Method | ALL D1-bg | ALL D1-fg | ALL D1-all | NOC D1-bg | NOC D1-fg | NOC D1-all |
|---|---|---|---|---|---|---|
| CFNet | 1.54 | 3.56 | 1.88 | 1.43 | 3.25 | 1.73 |
| MAGIC-CFNet | 1.48 | 3.20 | 1.76 | 1.36 | 2.82 | 1.60 |
We thank the reviewer for recognizing the effectiveness of MAGIC, particularly its improvements in occluded regions, and the value of the MBS20K dataset. We also appreciate the constructive feedback, which helps improve the clarity and strengthen the experimental validation of our work. Below, we clarify the methodological contributions of MAGIC, provide controlled comparisons with existing multi-view methods, further examine the contribution of MBS20K, and address the concerns regarding target-domain fine-tuning.
Q1. Comparison with existing multi-view methods.
We acknowledge that SMV and NeRFStereo also exploit complementary visibility across multiple views to alleviate occlusion. Building on these works, the key distinction of MAGIC lies in how multi-baseline visibility complementarity is converted into progressively refined disparity supervision through an EMA teacher–student framework, explicit baseline-ratio alignment, and occlusion-aware weighting.
(1) Novelty clarification.
Visibility-asymmetric teacher–student learning. By sharing the same reference image but using different target views, the teacher and student naturally exhibit complementary visibility patterns. MAGIC exploits this visibility asymmetry: the teacher uses valid correspondence cues in its visible regions to provide reliable disparity supervision for student-occluded regions. As the student progressively improves its disparity completion capability in occluded regions, this capability is transferred back to the teacher through EMA updates, further improving the teacher's subsequent predictions.
Geometric disparity alignment. Both SMV and NeRFStereo adopt a fixed three-view configuration, with a central reference view and two target views on either side at equal baselines, ensuring that the resulting disparity maps share the same scale. In contrast, MAGIC imposes no such restriction on the relative positions of the reference and target views. By exploiting geometric consistency and rescaling teacher disparities according to the baseline ratio, MAGIC aligns disparity predictions from arbitrary reference–target pairs with different baselines. Consequently, an N-view set can generate up to N(N−1)2 teacher–student triplets, substantially expanding the available training samples.
Occlusion-aware weighting. As illustrated in Fig. 4, even with ground-truth supervision, the learning signal from occluded regions can be overwhelmed by the abundant co-visible pixels due to their highly imbalanced proportions. This imbalance limits the model's ability to learn accurate disparity completion in occluded regions. To address this issue, OAW explicitly increases the supervision weight of student-occluded but teacher-visible pixels, ensuring that their sparse but valuable supervision contributes more effectively to optimization.
(2) Quantitative comparison. We adopt CFNet as the shared backbone for MAGIC, SMV, and NeRFStereo. SMV predicts disparity and uncertainty through parallel heads attached to the generated cost volume, making CFNet's cost-volume filtering architecture a natural choice for preserving its original design. For a fair comparison of multi-view supervision, we use only NeRFStereo's triplet photometric loss, excluding its rendered disparity loss. All three methods use the same SceneFlow-pretrained CFNet initialization, learning rate, and iteration steps. Since the publicly provided SMV training data is no longer accessible, we evaluate all three methods on MBS20K and NeRFStereo datasets.
As shown in Tables I and II, MAGIC achieves the best overall performance among the three methods under the matched CFNet backbone and training configuration. On MBS20K, MAGIC achieves the best performance across all 16 reported metrics, outperforming both SMV and NeRFStereo. On the NeRFStereo dataset, MAGIC also outperforms SMV across all metrics and achieves better results than NeRFStereo on 14 out of 16 metrics. These results demonstrate the effectiveness of MAGIC under a unified backbone and initialization, providing a fair comparison with existing multi-view stereo methods.