- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
- The distinction between oracle opportunity and achievable routing performance is closely related to existing work on adaptive routing and selective prediction. The main contribution seems to be a reformulation of familiar ideas rather than a fundamentally new methodology.
- The experiments identify potential bottlenecks but never demonstrate that following the proposed diagnostic recommendations improves an actual controller. This leaves the practical value of the framework insufficiently validated.
- The central 47.13% versus 3.04% comparison relies on retrospective validation-set analysis. The strongest observable-coverage result is not independently confirmed on the test set.
- Experiments use relatively small evaluation sets, yet confidence intervals and statistical significance analyses are largely absent. This makes the robustness of the reported differences difficult to assess.
Critical strengths
- This work reveals a important issue in adaptive multimodal inference that detecting modality degradation or observing model complementarity does not necessarily imply that a useful and reliable intervention policy can be constructed. And this diagnostic perspective is practically relevant and provides a reasonable motivation for the work.
- Also this work has a clear problem decomposition that separating recoverability from pre-action actionability helps pinpoint why adaptive interventions may fail.
- Some empirical findings are interesting like the three benchmarks illustrate distinct failure modes.
Critical weaknesses
- The distinction between oracle opportunity and achievable routing performance is closely related to existing work on adaptive routing and selective prediction. The main contribution seems to be a reformulation of familiar ideas rather than a fundamentally new methodology.
- The experiments identify potential bottlenecks but never demonstrate that following the proposed diagnostic recommendations improves an actual controller. This leaves the practical value of the framework insufficiently validated.
- The central 47.13% versus 3.04% comparison relies on retrospective validation-set analysis. The strongest observable-coverage result is not independently confirmed on the test set.
- Experiments use relatively small evaluation sets, yet confidence intervals and statistical significance analyses are largely absent. This makes the robustness of the reported differences difficult to assess.
Questions for authors
See weakness above
2 comments
Thanks for the response and additional experiment results. Even though the additional results can improve the robustness and validity of the methods, I feel like this work is more like a engineering effort and the presentation of the work still has a large room for improvement. In addition, the practicality of the work remains ambiguous. Therefore, I maintain the attitude that this paper needs more effort to be a qualified work.
We thank the reviewer for recognizing the importance of separating intervention opportunity from the ability to identify beneficial actions before execution. We agree that the methodological contribution, statistical evidence, and practical limitations require clearer presentation.
1. Contribution beyond established techniques .
We agree that counterfactual outcomes, grouped out-of-fold evaluation, and selective decision rules are individually established techniques. Our contribution is not a new estimator or routing architecture, but a registered intervention-eligibility methodology. Specifically, we evaluate full-information safety feasibility and pre-action legal safety observability on the same frozen proposals, utility definition, and benefit/harm constraints, and use their separation to determine whether further controller development is warranted.
Unlike reporting oracle–policy performance gaps alone, the Harness distinguishes whether safe intervention opportunities exist from whether admissible evidence can expose them at sufficient coverage. Its registered STOP/REFINE/validation-eligibility decisions make these distinctions actionable within a specified system contract. We will clarify this methodological scope and avoid claims of fundamentally new routing algorithms.
2. Statistical robustness of the central finding .
We conducted additional scene-group bootstrap analyses of the frozen DeLiVER VAL audit (2,005 samples). The full-information safe ceiling selects 945/2,005 samples (47.1322%), while SOFT-R1 exposes 61/2,005 (3.0424%). Across 2,000 resamples, the descriptive 95% intervals are [41.9044%, 52.2825%] and [0.2718%, 4.5269%], respectively, with a paired difference interval of [39.6782, 50.2496] percentage points.
Only 17/2,000 resamples reach the registered 5% continuation threshold, and none reaches the 15% operational target. These results characterize the stability of the observed gap under scene-level resampling. Importantly, they remain conditional on the frozen OOF scores and retrospective threshold-scanning procedure; they are not post-selection guarantees or independent estimates of deployment coverage.
3. Independent confirmation and practical validation.
We acknowledge that the strongest SOFT-R1 observability result has not been independently confirmed on TEST. The historical B2 TEST evaluation uses a different proposal family and cannot serve as that confirmation. We will explicitly distinguish these results and avoid interpreting hindsight-safe representability as prospective policy performance.
The practical purpose of Harness is diagnostic triage before controller construction, rather than guaranteed improvement after controller training. Under the registered DeLiVER contract, sufficient safe opportunity exists, but the tested legal evidence fails the predefined continuation threshold. The resulting STOP decision is therefore supported for this specific diagnostic contract; it does not establish that static execution is universally optimal or that following the diagnostic improves a downstream controller. Demonstrating such improvement remains an open validation step.
4. Additional diagnostic findings and corrected interpretation.
Our new MMAct audit identifies a +0.3931 percentage-point retain-or-mask hindsight oracle, correcting the earlier interpretation that the registered action family lacked recoverable opportunity. Forced masking remains harmful on average, but some decision units benefit from masking. This correction reinforces the need to distinguish action-family opportunity from the effectiveness of a fixed intervention rule.
Additional DeLiVER scorer comparisons indicate that ranking choices influence safe representability, although they do not isolate which learning component explains the difference. We will report these findings with their limitations and revise the manuscript to separate established diagnostic conclusions from unvalidated controller-improvement claims.
We will also improve presentation by explicitly distinguishing the full-information oracle, hindsight-safe representability, and prospectively validated controller performance, and by clarifying the decision associated with each diagnostic stage.