acceptodds
Under review as a conference paper at ICLR 2027

LESS THAN MEETS THE EYE: ACTION-SET IDENTIFIABILITY AND PRINCIPLED ABSTENTION FOR 3D PHYSICAL GEOMETRY

Abstract

A depth prediction can match its reference measurement and still describe a surface that no hand would touch. A corridor, its reflection in a planar mirror, and a photograph of that corridor on a screen can produce one image, while a hand stops in three places. Existing work addresses such scenes by enlarging what a predictor reports, through multi-layer formulations, detectors for mirrors and glass, and calibrated uncertainty. Each such quantity, however, describes a prediction or a predictor at a fixed viewpoint. None of them registers that the observation carries no evidence, because one image fits every explanation. We therefore ask a different question. Given a bounded set of camera motions, can an observer recover the physical explanation at all? Our approach, Intervene3D, calls two explanations separable to the extent that they predict different observations after the camera moves. A scene counts as identifiable once some available motion separates every remaining explanation, and the system abstains otherwise. A maximin rule then selects the motion separating the weakest contending pair. The experiments we conducted begin on a benchmark whose explanations are pixel-identical at the reference view, where abstention lowers false physical certainty from to . Maximin selection there reaches a committed accuracy of across four mechanisms, whereas a belief-weighted objective reaches on mirrors. Turning to real photographs, our experiments on seven families reveal that all of them fail on to of images, while their confidence stays below chance of predicting these failures. Choosing the viewpoint that some hypothesis explains best then lowers ambiguous-region error by on all thirteen checkpoints. Moreover, a gate over frozen encoder features reaches out of fold and lowers the failure rate from to at coverage. Asking what the available views can settle therefore precedes any answer about the physical geometry of the scene.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Depth Prediction, LLM, Camera Motion, Vision LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RkeROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper mainly analyze if monocular depth estimation models will predict fake 3d patterns as 3d geometry. The paper compare the predicted depths from two stereo images which can be viewed as a virtual camera move, and if the discrepancy between the depth after warping can be a indicator of ambiguous region. The proposed method can identify some ambiguous regions and eliminate false physical certainty.

Critical strengths

The motivation to address the hallucination of 3d prediction is clear, which is important for applications on robots and cars.

The proposed method is reasonable where the consistency across views can be useful to identify ambiguity.

The proposed method show effectiveness to identify planar surface hallucination.

Critical weaknesses

The paper makes a strong assumption on the planar geometry of the fake 3d, which is not general for real-world cases.

The proposed method has not been validated on real world setup despite a simplified stereo-image-based evaluation.

For the proposed method, it is not clear if it will lead to false alarms of ambiguity prediction on general 3d scenes without ambiguous regions like reflection.

Questions for authors

Could you please clarify for the proposed method, how to disentangle error prediction or inconsistent prediction across views, and the ambiguous regions?

Could you please detail the simulation setup on the diversity and realism of the simulated planar geometry?

1 comment

Thank you for your careful review and for recognizing the motivation and the effectiveness of the method. We agree that the method needed validation beyond stereo pairs and a clearer explanation. Guided by your comments, the revision evaluates Intervene3D on real camera motion and in a hand-held capture, and it explains the method step by step.

Q1 (prediction error, cross-view inconsistency and ambiguity): Thank you for this question. The revised Section 2.6 and Figure 1 now walk through this step explicitly: the depth network runs only once, on the first image, and each hypothesis turns this single prediction into a prediction of the second view. Two comparisons follow:

  • Separability compares the predictions of two hypotheses for a motion aa, Δij(a)=D(F^i,a,F^j,a)\Delta_{ij}(a) = D(\hat F^{i,a}, \hat F^{j,a}) (Equation 2), so it needs no observation.
  • The residual compares each prediction with the observation after the selected motion, ek=D(F^k,a⋆,Fobs)e_k = D(\hat F^{k,a^\star}, F_{\mathrm{obs}}) (Section 2.5).
CaseSeparabilityLowest residualOutput
Ambiguous regionΔ<ϵ\Delta < \epsilonnot usedabstain
Prediction error on a panelΔ≥ϵ\Delta \ge \epsilonflat panelpanel surface replaces the network depth
Correct predictionΔ≥ϵ\Delta \ge \epsilondirectnetwork surface kept

On sheets, posters and monitors in the frame, panel answers lower the median surface error from 9.4%, 16.3% and 4.1% to 4.2%, 7.6% and 1.1% (Table 3). Inconsistency across views is a weaker signal: two predictions fooled in the same way still agree, and cross-view disagreement detects failures poorly (area under the ROC curve 0.56–0.62, Section 3.6).

Q2 (diversity and realism of the simulation): We agree that these details matter, and Appendix A.3 lists every parameter of the generator, drawn uniformly per base scene:

ParameterRange
Interface distance, tilt1.2–2.5 m, up to 8°
Interface width, aspect0.45–1.15 of the image half-width, 0.60–0.95
Content depth behind the interface0.02–0.80 m, spread 0.02–2.50 m
Camera offset, pointing±0.30 m, ±6°
Scenes96 base scenes per seed (192 with four mechanisms), 11 seeds

The benchmark is built so that all mechanisms produce exactly the same first view, verified below 3×10⁻¹³ pixels. Photometric realism would break this guarantee, so the simulation stays geometric. Instead, realism comes from real data: 404 stereo photographs, 1,620 executed-motion combinations, 123 Mirror3D frames and the hand-held capture.

W1 (planarity assumption): We agree that planarity is an assumption, and we clarify its scope. It applies only to the interface, that is, the panel, the mirror or the glass pane. The content shown on it can have any 3D structure. Printed sheets, posters, monitors and flat mirrors are planar in fact, and a real image enters the evaluation only when its reference disparity fits a plane (Section 2.6). Moreover, curved interfaces are handled as well: the residual-adequacy test lowers wrong answers on strongly curved mirrors from 56.8% to 12.4%, and a mirror hypothesis with known curvature answers those scenes at accuracy 1.000 (Section 3.3).

W2 (real-world validation): We agree, and the revision adds two real settings with executed camera motion (Section 3.4, Figure 4):

Real settingCombinationsCorrect decisionMedian surface error, network → ours
Executed motion, 53 scenes1,620–4.35% → 2.98%
Hand-held, monitor56490.2%11.5% → 6.3%
Hand-held, mirror32487.7%–

W3 (false alarms on ordinary scenes): This is an important check, and the results answer it: on scenes without displays or mirrors, Intervene3D does not report a false panel or mirror. On the direct scenes of the synthetic benchmark, every answer it gives is correct (154 of 154), and an abstention returns the network's own depth. Likewise, on real panels, Intervene3D makes the surface worse on only 0.8% of stereo combinations, compared with 52.7% for a detector that replaces every flagged region by a plane (Section 3.6).

Finally, we take the presentation concern seriously and are grateful for it: Figure 1 now shows the whole pipeline on one real stereo pair, Figure 2 gives the decision order at the start of Section 2, each term is defined once, and Appendix A.1 collects the notation. We hope the clearer explanation and the real-world validation make both the method and its evidence easy to follow. We would appreciate your reconsidering the score in that light, and we are happy to answer any further question.

Reviewer_EqcZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between 3D visual predictions and true physical geometry, highlighting that existing depth estimation models may incorrectly interpret visual content in images as actual physical surfaces. To address this issue, the authors propose Intervene3D, which constructs four candidate physical explanations and predicts how the observations would change under different hypotheses following camera movements. A Maximin strategy is then used to select actions that help distinguish between these hypotheses. Through synthetic experiments and a real-world visual illusion dataset, the paper demonstrates the limitations of existing depth estimation models in understanding physical geometry and explores a prediction rejection strategy based on identifiability.

Critical strengths
  1. The paper addresses an interesting problem concerning the gap between visual geometry and true physical geometry, particularly the fact that prediction confidence does not necessarily reflect physical ambiguity. This is a meaningful issue for 3D understanding and robotic perception.
  2. The authors combine active observation, physical hypothesis discrimination, and prediction rejection, offering a perspective that differs from conventional depth estimation. The overall approach is clear and well-motivated.
  3. The experimental evaluation is fairly comprehensive. The authors evaluate 13 depth estimation checkpoints and conduct synthetic experiments, real-world evaluations, and several ablation studies.
Critical weaknesses
  1. The predefined physical hypothesis space is rather limited, raising concerns about the generalizability of the method. Real-world phenomena are often more complex; for example, a water surface may involve reflection, refraction, and dynamic geometric changes simultaneously. More importantly, distinguishing between predefined hypotheses does not necessarily imply that the true physical geometry is identifiable. Therefore, the method seems closer to discrimination within a limited hypothesis set than to physical geometry identifiability in a general sense.
  2. The most important contribution of this paper is the use of actively selected camera motions to resolve physical ambiguities. However, the complete multi-hypothesis action selection framework is mainly evaluated on synthetic data. The real-world experiments focus more on detecting incorrect predictions, which is not sufficient to demonstrate the effectiveness of the method in complex real-world environments.
  3. The practical advantages of the proposed method are still unclear. In the synthetic experiments, higher prediction accuracy is accompanied by a relatively high rejection rate. In addition, some simple feature-based and learning-based strategies also achieve good error detection performance in the real-world experiments. The paper does not sufficiently explain the unique advantages of complex physical hypothesis modeling over these simpler approaches.
Questions for authors
  1. How does the system determine whether its hypothesis space is insufficient when the underlying physical mechanism of a real-world scene falls outside the four predefined hypotheses? Have the authors considered unknown or combined physical mechanisms, such as a water surface involving both reflection and refraction?
  2. Could the authors include additional experiments involving active camera movements in real-world scenes to verify whether the proposed method can effectively resolve physical geometry ambiguity?
  3. Compared with simpler geometric consistency checks or feature-based failure prediction methods, what are the main advantages of Intervene3D?
1 comment

Thank you for finding the problem meaningful, the perspective distinct, and the approach clear and well motivated. We agree that the complete framework should be shown on real camera motion, and that its advantage over simpler strategies should be explicit. Your comments guided the revision: we kept the method as submitted and ran its full active loop on real executed motion and in a hand-held capture, and we compared it directly with simpler strategies.

Q1 (mechanisms outside the hypothesis family): Section 3.3 handles such scenes in three ways. First, a residual-adequacy test abstains when no hypothesis explains the new image, that is, when min⁡kek>η\min_k e_k > \eta with η=2\eta = 2 pixels. On strongly curved mirrors, it lowers wrong answers from 56.8% to 12.4%. Second, combined mechanisms are already modelled: a glass pane that both transmits and reflects is a family member, and committed accuracy stays 1.000 on all four mechanisms. Third, a new mechanism only needs its transition. A curved mirror of known curvature, added as a fifth hypothesis, is answered at accuracy 1.000 with the same decision rule, and a refractive hypothesis for water follows the same recipe.

Q2 (active camera motion in real scenes): Yes (Section 3.4, Figure 4).

Real settingCombinationsCorrect decisionMedian surface error, network → ours
Executed motion1,620–4.35% → 2.98%
Hand-held, monitor56490.2%11.5% → 6.3%
Hand-held, mirror32487.7%–

On the mirror, the camera's light serves as the cue that separates a mirror from an opening. Without a cue, ScanNet++ mirrors leave that pair inseparable, and the system abstains as identifiability predicts (Appendix C.7).

Q3 (advantage over simpler methods): Intervene3D returns the physical surface, whereas simpler methods only flag a possible error. Each is therefore scored on the surface it returns (Section 3.6):

Flag, then replace by a planeBaseline worsensIntervene3D worsens
OWLv2 detector52.7%0.8%
Learned failure classifier4.8%0.6%

Cross-view disagreement detects failures poorly (area under the ROC curve 0.56–0.62) and gives no surface. Moreover, Intervene3D names the mechanism and the motion that settles it.

W1 (scope of identifiability): We agree with this reading, and Section 2.3 now states this scope in the definition (Equations 4–5): a scene is identifiable when min⁡(i,j)∈Cmax⁡a∈AΔij(a)≥ϵ\min_{(i,j)\in\mathcal{C}} \max_{a\in\mathcal{A}} \Delta_{ij}(a) \ge \epsilon. Identifiability therefore holds for a stated family, action set A\mathcal{A} and threshold ϵ\epsilon, as in T-optimal design. This gives a precise guarantee, and the family stays open, as the response to Q1 shows.

W2 (action selection on real data): We agree that this was the main gap, and the complete framework now runs on real data. On the hand-held mirror, three hypotheses compete:

Motion ruleMaximin (ours)Largest motionRandom
Correct87.7%55.6%33.3%

W3 (abstention rate and practical advantage): This is a fair concern, and the design of the benchmark explains the rate. The benchmark deliberately includes scenes that no motion resolves, and abstention is what protects them: without it, 76.4% of these scenes receive a confident answer, and with it 0.0%. On resolvable scenes, maximin abstains on only 3.0%, against 30.9% for summed selection, and on real executed motion abstention is 0.0% (Table 2). The response to Q3 covers simpler strategies.

We hope the real-motion results and the comparison with simpler strategies resolve your concerns about the practical value of the method. If they do, we would be grateful for a reconsideration of the rating, and we welcome further discussion.

Reviewer_F1nZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that a single image can be explained by several physical mechanisms, so a depth estimator might make false predictions. It argues that this challenge can be addressed by verifying against several camera motions. They propose Intervene3D, which predicts what each mechanism would observe after each motion. It abstains when some pairs cannot be separated. It achieves 100% accuracy on the cases it answers on the synthetic benchmark. On real images, the authors show that 13 depth models make wrong predictions on displays and mirrors, and on displays their separability score predicts these failures better than model confidence.

Critical strengths
  1. The problem is real and well-defined. The paper treats ambiguous depth estimation as a lack of evidence instead of a capability issue of existing depth estimators. It offers a principled solution as well.
  2. The study of existing depth estimators is extensive and insightful. The experiments are done across 13 checkpoints and the models are fooled more than half the time. Additionally, the models’ own confidence cannot tell when they are fooled.
Critical weaknesses
  1. The real-image experiments are limited. The maximin action selection is only validated on synthetic data. A large proportion of the real-image results measures the failures of existing depth estimators rather than the performance of Intervene3D. Also, there is no comparison with existing methods for detecting mirrors or glass.
  2. (Presentation) The paper is somewhat hard to follow. Key information, such as the pipeline figure (Fig. 7), appears only in the appendix. At the same time, the terminology is dense. There are also some consistency issues: Fig. 2 has three panels, but its caption and the text refer to four.
Questions for authors
  1. Could the authors provide more quantitative and qualitative results of Intervene3D on real images?
  2. How does Intervene3D compare with existing mirror or glass detectors?
  3. How often does the method abstain on ordinary images without displays or mirrors?
3 comments

Thank you for finding the problem real and well defined, the solution principled, and the study of 13 checkpoints extensive. Following your comments, the revision runs the full loop of Intervene3D on real camera motion and compares it with a detector. Twelve released checkpoints enter the statistics, and the thirteenth, which needs a depth prompt, is shown in Figure 11. All twelve are fooled, on 21.8–68.3% of illusion images and on 69.1% of Mirror3D frames on average.

Q1 (results on real images): Yes. The revision adds two real settings, and in both of them Intervene3D selects a motion, observes the result and decides (Section 3.4, Table 2, Figures 3c and 4).

Real settingCombinationsCorrect decisionMedian surface error, network → Intervene3D
Executed motion, 53 scenes1,620–4.35% → 2.98%
Hand-held capture, monitor56490.2%11.5% → 6.3%
Hand-held capture, mirror32487.7%–

In both settings with a surface error, the error drops on all twelve checkpoints. Figure 4 shows one example per recording: the network depth copies the room in the photograph, whereas Intervene3D reports the flat panel or the mirror plane. In addition, on 404 stereo photographs, the flat-panel answers lower the median surface error in every panel category (Table 3):

Stereo categoryNetworkIntervene3D
Printed sheet on a table9.4%4.2%
Poster beside an object16.3%7.6%
Monitor in the frame4.1%1.1%

Q2 (comparison with mirror and glass detectors): Intervene3D returns a surface, so both methods are scored on the surface they return (Section 3.6). An open-vocabulary detector (OWLv2) is used because sheets, posters and monitors are not classes of a mirror detector. A detector returns a mask and no surface, so every flagged region is replaced by the panel plane.

Surface made worse than the networkStereo photographsExecuted motion
OWLv2 detector, then plane52.7%25.8%
Intervene3D0.8%9.3%

Moreover, the two approaches combine well: a detector output can serve as the prior p0p_0 of the belief update p1(Hk)∝exp⁡(−βek) p0(Hk)p_1(H_k) \propto \exp(-\beta e_k)\, p_0(H_k) (Equation 8), and the motion then supplies the evidence. This surface test is stricter than the confidence comparison of the submission, which it replaces.

Q3 (abstention on ordinary images): The synthetic benchmark contains direct scenes with no display or mirror. On the 162 resolvable direct scenes, maximin never abstains, and every answer is correct (Section 3.2), so no false panel or mirror is reported. It abstains on a direct scene only when the benchmark pairs it with a display or mirror that no allowed motion can tell apart, which is exactly the purpose of the abstention rule. Moreover, an abstention returns the network's own depth, so it costs nothing on an ordinary image.

W1 (real-image experiments and maximin selection): Maximin selection is now validated on real data. On the hand-held mirror recording (27 regions, twelve checkpoints), three hypotheses compete, so the choice of motion matters (Figure 4c):

Motion ruleMaximin (ours)SummedLargest motionFirst frameRandom
Correct, 324 combinations87.7%86.4%55.6%37.0%33.3%

In addition, the real-image results now measure Intervene3D itself (response to Q1), and the detector comparison is given in the response to Q2.

W2 (presentation): First, Figure 1 now shows the full pipeline in the main text on one real stereo pair, and Figure 2 gives the decision order at the start of Section 2. Second, each term is defined once in Section 2, and Appendix A.1 collects the notation. Third, the main-results figure (Figure 3) has three panels, and its caption and the text now match. Fourth, Table 1 states, for each protocol, the contending pairs and when each check is applied.

We hope these real-data results address your concerns and merit a higher rating, and we are glad to clarify any remaining point.

Thank you for the detailed response and the revised version. However, the hand-held capture covers only one monitor and one mirror. A larger real-world capture set with more scenes would be needed to support the claims. Maximin also shows little advantage over simpler rules such as summed selection on real data, so I will keep my original rating.

Thank you for reading the revision and for the follow-up. Both points are fair. We address each with results on public real benchmarks, including new scene-level statistics computed for this reply.

1. Real-world evidence on public benchmarks. We agree that one monitor and one mirror cannot carry the claims, and they do not: the hand-held capture only adds the three-hypothesis case. The real-world claims rest on public data, summarized here per independent scene and with bounds from resampling whole scenes or photographs.

Public benchmarkSizeResultBounds
3D-Visual-Illusion, executed motion44 scenes, 1,620 combinationsanswer better than the network in 42 of 44 scenes; error reduced by 29.8%[16.5%, 39.3%]
3D-Visual-Illusion, stereo404 photographs, 4,848 combinationssurface made worse on 0.8%, against 52.7% for a detector[0.4%, 1.3%] against [49.3%, 56.1%]
ScanNet++ mirrors, registered video30 scenes, 1,433 regions, 17,196 combinationsconfidently wrong surfaces fall from 35.3% to 5.0%[29.5%, 41.0%] to [3.3%, 7.0%]

On the executed-motion data, the answer is better in 100% of 5,000 scene resamples and on all twelve checkpoints. On the stereo data, Intervene3D harms fewer checkpoints than the detector on 361 of 404 photographs, and more on 1. On ScanNet++, no observer cue is available, so a mirror and an opening cannot be told apart; the system abstains on 75.8% of regions, as identifiability predicts, instead of committing to a wrong surface. A surface counts as confidently wrong when it is committed and more than 10% off the scanned mesh, and Intervene3D lowers that share in 30 of 30 scenes and on all twelve checkpoints. For this reply we extended this benchmark from four to twelve checkpoints.

2. Mirror and glass detectors. Your first review asked for this comparison, and we now report dedicated detectors as well. A flagged region is replaced by the panel plane, and each method is scored on the surface it returns.

Surface made worse than the networkStereoExecuted motion
OWLv252.6%25.8%
Grounding DINO52.3%27.8%
HetNet (mirror detector, AAAI 2023) + OWLv252.1%24.3%
GDNet (glass detector, CVPR 2020) + OWLv253.6%29.0%
Intervene3D0.8%9.3%

The result therefore does not depend on the detector. HetNet and GDNet fire on 96.7% and 87.8% of the real mirrors of Mirror3D, so both work as intended.

3. Maximin against summed selection. You are right that the two rules are close when every recorded motion is available: 87.7% against 86.4%. With two hypotheses they are the same rule, since a single pair makes the minimum and the sum select the same action, so only the three-hypothesis mirror can separate them. There, one motion happens to separate every pair when all 33 motions are available. A robot rarely has that freedom, so we re-ran the selection with the motions limited to random subsets (20 subsets per region, 324 region–checkpoint combinations):

Motions availableMaximinSummedGap, mean [min, max] over subsets
556.4%54.4%+1.9 [+0.3, +4.3]
1263.0%60.0%+2.9 [+1.2, +4.0]
2475.3%73.5%+1.7 [+0.3, +3.4]
33 (all)87.7%86.4%+1.2

Across 160 configurations of subset size and seed, maximin is better in 146, equal in 14, and worse in none. Per checkpoint, it is equal on seven and better on five, by up to 14.8 points for Metric3D-Large, the networks this mirror fools most and for which more pairs stay in contention. This matches the synthetic benchmark, where maximin answers 61.0% of scenes against 43.1% at committed accuracy 1.000 and abstains on 3.0% of resolvable scenes against 30.9% (Section 3.2). Against no selection, the gap on the real mirror is larger still: 87.7% against 55.6% for the largest motion and 33.3% for a random motion.

These statistics will be added to the paper. We are grateful for the time you have given the submission. We hope the scene-level evidence on public benchmarks and the selection analysis address both points, and we are glad to answer anything further.

Comments on the paper

1 comment

We thank the three reviewers for their careful reading. All three asked for the same thing: evidence that the full method works on real camera motion, and a clearer presentation. The revision provides both, and every number below is in the revised PDF.

What was askedWhat the revision addsWhere
The active loop on real dataOne executed motion lowers the median surface error from 4.35% to 2.98% on 1,620 combinations, on all twelve checkpointsSection 3.4, Table 2
A real multi-hypothesis testHand-held RGB-D capture: 90.2% correct on a monitor, 87.7% on a mirror; maximin 87.7% against 33.3% for a random motionSection 3.4, Figure 4
Comparison with detectors and simpler strategiesA detector followed by a plane worsens the surface on 52.7% of combinations, Intervene3D on 0.8%Section 3.6
Scenes outside the hypothesis familyAn adequacy test lowers wrong answers on curved mirrors from 56.8% to 12.4%, and a fifth hypothesis answers them at accuracy 1.000Section 3.3
PresentationFull pipeline in Figure 1, decision order in Figure 2, one definition per term, a Method paragraph per protocolSections 1 and 2

One note on the evaluation: twelve released checkpoints enter the statistics, and the thirteenth, a transparency variant that requires a depth prompt, appears qualitatively in Figure 11. All of these additions build on the same core idea as the submission: a single image admits several physical mechanisms, a selected camera motion can separate them, and the system answers only when the evidence supports the answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.