acceptodds
Under review as a conference paper at ICLR 2027

From Multimodal Indicators to Intervention Eligibility: Testing the Assumptions Behind Adaptive Intervention

Abstract

Multimodal inference increasingly makes per-input decisions about which sensors, modality subsets, or experts to execute. Yet when adaptation underperforms, aggregate accuracy or routing gain does not reveal whether the problem lies in the action space, intervention usefulness, or the evidence used for selection. We introduce Perception Harness, a diagnostic framework that evaluates executable alternatives before controller development. With actions, fallback, evidence, utility, and harm conditions held fixed, it separates action identity, recoverability, and pre-action actionability. Three benchmarks reveal distinct development needs. On MMAct, degradation is detectable but masking is harmful on average, motivating fallback retention or masking redesign. On MCubeS, identical modality labels yield different learned action landscapes, calling for model-specific action-space profiling. On DeLiVER, substantial safe opportunity exists within frozen proposals, but admissible pre-action evidence exposes little of it, motivating evidence and proposal redesign. Locating the failed transition directs the next development step. The diagnosis motivates redesign or eligibility for separate controller validation. Eligibility is not deployment authorization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal learning, adaptive routing, safe model selection, selective prediction, robustness evaluation, deployment-time decision making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_j6WjOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary
  1. The distinction between oracle opportunity and achievable routing performance is closely related to existing work on adaptive routing and selective prediction. The main contribution seems to be a reformulation of familiar ideas rather than a fundamentally new methodology.
  2. The experiments identify potential bottlenecks but never demonstrate that following the proposed diagnostic recommendations improves an actual controller. This leaves the practical value of the framework insufficiently validated.
  3. The central 47.13% versus 3.04% comparison relies on retrospective validation-set analysis. The strongest observable-coverage result is not independently confirmed on the test set.
  4. Experiments use relatively small evaluation sets, yet confidence intervals and statistical significance analyses are largely absent. This makes the robustness of the reported differences difficult to assess.
Critical strengths
  1. This work reveals a important issue in adaptive multimodal inference that detecting modality degradation or observing model complementarity does not necessarily imply that a useful and reliable intervention policy can be constructed. And this diagnostic perspective is practically relevant and provides a reasonable motivation for the work.
  2. Also this work has a clear problem decomposition that separating recoverability from pre-action actionability helps pinpoint why adaptive interventions may fail.
  3. Some empirical findings are interesting like the three benchmarks illustrate distinct failure modes.
Critical weaknesses
  1. The distinction between oracle opportunity and achievable routing performance is closely related to existing work on adaptive routing and selective prediction. The main contribution seems to be a reformulation of familiar ideas rather than a fundamentally new methodology.
  2. The experiments identify potential bottlenecks but never demonstrate that following the proposed diagnostic recommendations improves an actual controller. This leaves the practical value of the framework insufficiently validated.
  3. The central 47.13% versus 3.04% comparison relies on retrospective validation-set analysis. The strongest observable-coverage result is not independently confirmed on the test set.
  4. Experiments use relatively small evaluation sets, yet confidence intervals and statistical significance analyses are largely absent. This makes the robustness of the reported differences difficult to assess.
Questions for authors

See weakness above

2 comments

We thank the reviewer for recognizing the importance of separating intervention opportunity from the ability to identify beneficial actions before execution. We agree that the methodological contribution, statistical evidence, and practical limitations require clearer presentation.

1. Contribution beyond established techniques .

We agree that counterfactual outcomes, grouped out-of-fold evaluation, and selective decision rules are individually established techniques. Our contribution is not a new estimator or routing architecture, but a registered intervention-eligibility methodology. Specifically, we evaluate full-information safety feasibility and pre-action legal safety observability on the same frozen proposals, utility definition, and benefit/harm constraints, and use their separation to determine whether further controller development is warranted.

Unlike reporting oracle–policy performance gaps alone, the Harness distinguishes whether safe intervention opportunities exist from whether admissible evidence can expose them at sufficient coverage. Its registered STOP/REFINE/validation-eligibility decisions make these distinctions actionable within a specified system contract. We will clarify this methodological scope and avoid claims of fundamentally new routing algorithms.

2. Statistical robustness of the central finding .

We conducted additional scene-group bootstrap analyses of the frozen DeLiVER VAL audit (2,005 samples). The full-information safe ceiling selects 945/2,005 samples (47.1322%), while SOFT-R1 exposes 61/2,005 (3.0424%). Across 2,000 resamples, the descriptive 95% intervals are [41.9044%, 52.2825%] and [0.2718%, 4.5269%], respectively, with a paired difference interval of [39.6782, 50.2496] percentage points.

Only 17/2,000 resamples reach the registered 5% continuation threshold, and none reaches the 15% operational target. These results characterize the stability of the observed gap under scene-level resampling. Importantly, they remain conditional on the frozen OOF scores and retrospective threshold-scanning procedure; they are not post-selection guarantees or independent estimates of deployment coverage.

3. Independent confirmation and practical validation.

We acknowledge that the strongest SOFT-R1 observability result has not been independently confirmed on TEST. The historical B2 TEST evaluation uses a different proposal family and cannot serve as that confirmation. We will explicitly distinguish these results and avoid interpreting hindsight-safe representability as prospective policy performance.

The practical purpose of Harness is diagnostic triage before controller construction, rather than guaranteed improvement after controller training. Under the registered DeLiVER contract, sufficient safe opportunity exists, but the tested legal evidence fails the predefined continuation threshold. The resulting STOP decision is therefore supported for this specific diagnostic contract; it does not establish that static execution is universally optimal or that following the diagnostic improves a downstream controller. Demonstrating such improvement remains an open validation step.

4. Additional diagnostic findings and corrected interpretation.

Our new MMAct audit identifies a +0.3931 percentage-point retain-or-mask hindsight oracle, correcting the earlier interpretation that the registered action family lacked recoverable opportunity. Forced masking remains harmful on average, but some decision units benefit from masking. This correction reinforces the need to distinguish action-family opportunity from the effectiveness of a fixed intervention rule.

Additional DeLiVER scorer comparisons indicate that ranking choices influence safe representability, although they do not isolate which learning component explains the difference. We will report these findings with their limitations and revise the manuscript to separate established diagnostic conclusions from unvalidated controller-improvement claims.

We will also improve presentation by explicitly distinguishing the full-information oracle, hindsight-safe representability, and prospectively validated controller performance, and by clarifying the decision associated with each diagnostic stage.

Thanks for the response and additional experiment results. Even though the additional results can improve the robustness and validity of the methods, I feel like this work is more like a engineering effort and the presentation of the work still has a large room for improvement. In addition, the practicality of the work remains ambiguous. Therefore, I maintain the attitude that this paper needs more effort to be a qualified work.

Reviewer_ZXuqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Perception Harness, a novel pre-controller diagnostic framework based on formal registered contract. The authors claim that this auditing approach systematically decouples and validates the intervention chain of adaptive multimodal systems under counterfactual constraints before committing to policy optimisation. The approach is evaluated across three failure modes (such as Recoverability, Actionability, and Eligibility) on the MMAct, MCubeS, and DeLiVER benchmarks. These benchmarks use sensor configurations spanning video, wearable IMUs, polarisation, and active adverse-weather sensors. In this paper, authors claim a real-world observability bottleneck, demonstrating that retrospective opportunity ceilings (e.g., 47.13% on DeLiVER) collapse to minimal prospective safe coverage (3.04%) when realistically constrained by legal pre-acquisition evidence.

Critical strengths
  1. One main strength of this paper is that rather than rushing to optimise complex dynamic routers or attention gating layers, it establishes a formalised contract that decouples retrospective opportunity from prospective actionability to audit system viability before model training.

  2. Unlike prior evaluation methods, which evaluate routing performance under unconstrained settings, this framework strictly restricts the intervention decision to pre-acquisition evidence (OOF predictors or proxies), ensuring high real-world fidelity.

  3. The authors provide finite witnesses formulation (Appendix A) to prove why adjacent steps of the intervention chain (Indicator, Opportunity, Actionability, Eligibility) do not imply one another.

  4. This paper provides experimental evaluation using three qualitatively distinct benchmarks (MMAct, MCubeS, and DeLiVER), and demonstrates different failure modes across multiple sensing modalities.

Critical weaknesses
  1. In this paper, the core approach relies on standard counterfactual utility and out-of-fold cross-validation, which are well-established techniques in causal inference and policy evaluation. While applied to sensor degradation, the formulation lacks algorithmic novelty, offering a classic offline auditing pipeline rather than a novel online routing solution.

  2. To demonstrate the action ineffectiveness, the authors evaluate a weak binary fallback (a₀ vs. standalone IMU) on MMAct. Recent action recognition networks (e.g., C3T [1], PSKD [2]) never discard vision; they use cross-modal distillation or continuous latent alignment to recover degraded video streams. Evaluating an isolated IMU model is an unfair comparison.

  3. Similarly, the reported observability bottleneck on DeLiVER is also not fair. By forcing the decision model to use low-capacity tabular metadata or basic SOFT-R1 spatial proxies, the authors withhold the high-capacity, multi-scale cross-attention maps (e.g., CMNeXt [3] or MAGIC [4]) that recent models natively use to resolve pre-acquisition observability.

  4. Also, the models used to evaluate implementation-dependence on MCubeS are legacy CNN and standard transformer backbones. For a fair comparison, the diagnostic gate must be tested against recent architectures, such as MMSFormer [5], FuseAdapt [6], or MAGIC++ [7].

  5. Last but not the least, this paper primarily focuses on comparisons with static, frame-by-frame fallback baselines while largely overlooking more sophisticated, task-adaptive approaches such as EQUISeg [8] or MLE-SAM [9]. Experimental comparison with these continuous, dynamic routing and missing-modality adaptation approcahes would significantly improve the paper’s positioning and more clearly demonstrate the utility-safety trade-offs of the proposed diagnostic framework under sequential, real-time closed-loop constraints.

[1] C3T: "Cross-Modal Transfer Through Time for Unsupervised Multimodal Action Representation Learning", TPAMI 2024

[2] PSKD: "Predictive Skeleton Knowledge Distillation for Wearable Action Recognition", CVPR 2023

[3] CMNeXT: "Modern Fusion and Gating for Arbitrary-Modal Segmentation", CVPR 2023

[4] MAGIC: "Centering the Value of Every Modality: Towards Efficient and Resilient Modality-Agnostic Semantic Segmentation", ECCV 2024

[5] MMSFormer: "Multimodal Material Segmentation with Mix Transformers", PR 2024

[6] FuseAdapt: "Parameter-Efficient Routing in Frozen Multimodal Foundation Spaces.", ICLR 2026

[7] MAGIC++: "Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection", ECCV 2024

[8] EQUISeg: "Robustness Benchmarking for Missing and Noisy Modalities in Adverse Conditions", CVPRW 2025

[9] MLE-SAM: "Dynamic Mixture of LoRA Experts for Adapting Segment Anything Models to Missing Modalities", NeurIPS 2024

Questions for authors
  1. How does the system reliably assess or guarantee operational benefit during live deployment when ground-truth labels do not exist? Could the authors provide evidence that delegating runtime decisions to an auxiliary pre-evidence predictor remains sufficient when that predictor itself suffers from severe observability bottlenecks?
  2. Since no modern SOTA literature advocates completely discarding degraded video to run an IMU classifier alone, how can the authors claim action ineffectiveness flaw in existing architectures by evaluating only non-viable action space?
  3. How will the model perform if the decision model were granted access to intermediate multi-scale cross-attention features rather than being restricted to low-capacity tabular metadata? does the observed safe coverage gap still persist?
  4. What is the measured impact of high-frequency modality switching on temporal tracking consistency, state-reset latency, and hardware wear in sequential video (MMAct) and driving (DeLiVER) streams?
  5. Why did the authors restrict their evaluation to static, frame-by-frame fallback baselines while completely overlooking more sophisticated, task-adaptive and missing-modality approaches?
2 comments

We thank the reviewer for the detailed assessment. We agree that stronger multimodal architectures and deployment constraints deserve explicit discussion. We clarify several experimental misunderstandings and the scope of our contributions.

1. Methodological contribution and deployment scope.

Perception Harness is not a new online routing architecture, nor do we claim novelty in counterfactual estimation or out-of-fold prediction individually. Its contribution is a registered, fail-closed diagnostic protocol that separates recoverability, safety feasibility, legal safety observability, and controller authorization under fixed executable-action and evidence contracts.

The framework does not guarantee runtime benefit without ground-truth labels. In particular, DeLiVER's 3.0424% is the maximum hindsight-safe representability of the best tested legal grouped-OOF score on VAL, with the threshold selected using outcomes only after scores were frozen. It is neither prospective controller coverage nor a deployment guarantee. Temporal tracking, switching latency, state-reset costs, hardware wear, and closed-loop intervention are not evaluated. Extending the framework to sequential settings would require explicitly registering these transition costs and sequence-level utility and safety criteria.

2. MMAct action definition and corrected recoverability.

The MMAct experiment does not compare a multimodal classifier against a standalone IMU-only model. The registered baseline is a retained weighted four-stream classifier, and the alternative is fault-informed mask-and-renormalize execution, which may retain a watch-only path under certain conditions.

We also identified an important distinction in the recoverability analysis. Forced masking yields an average gain of −1.4573 accuracy percentage points, but improves 26 of 76 decision units. Including static retention in the sample-wise oracle gives a corrected retain-or-mask oracle gain of +0.3931 points. We therefore revise the original interpretation: the tested masking intervention is harmful on average, but the restricted action family contains some recoverable opportunity. This result does not establish the ineffectiveness of modern cross-modal distillation or adaptive architectures.

3. Existing architectures and richer legal evidence.

MCubeS already evaluates frozen, published MMSFormer and CMNeXt specialists under the same four modality-subset labels. Their TEST oracle gains are +1.45 and +1.02 macro-IoU points, respectively. This experiment examines implementation-dependent action landscapes, not architecture-level superiority.

For DeLiVER, cross-attention features requiring an auxiliary sensor that has not yet been acquired cannot inform its acquisition decision under the registered RGB-first contract. However, stronger multiscale RGB-only representations available before the decision are admissible, subject to computational-cost accounting. We have not tested whether such representations close the observed observability gap. Thus, our result identifies a limitation of the tested legal evidence, not a universal bound on pre-acquisition predictability.

4. Dynamic multimodal systems and fair comparison.

We agree that MAGIC++, FuseAdapt, EQUISeg, MLE-SAM, and related methods are important approaches to multimodal robustness and adaptation. Their model-level results address a related but distinct question from our evaluation of selected-switch utility and harm under a fixed action contract.

These architectures could be incorporated as executable candidates in a new registered action family. Such an evaluation would require matched fallback, acquisition timing, decision units, utility, and selected-population safety criteria. We do not claim superiority over these methods or that they necessarily exhibit the same failure modes.

We will revise the manuscript to clarify the action definitions, correct the MMAct recoverability interpretation, distinguish tested observability limitations from universal claims, and explicitly position Harness as a complementary diagnostic methodology rather than an online adaptive-fusion solution.

I thank the authors for their response and for clarifying the MCubeS and MMAct configurations. However, significant concerns remain. As the authors acknowledged, they have not evaluated whether high-capacity, legally admissible unimodal representations would close the observed observability gap, nor does the static framework account for sequential closed-loop dynamics or offer empirical comparisons against modern task-adaptive baselines like EQUISeg or MLE-SAM. Since these gaps limit the framework's practical applicability, I will maintain my original score.

Reviewer_Wto6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces perception harness, a new tool to test and evaluate adaptive intervention systems.Usually,developers build these systems first and then test them. This paper says we should test our assumptions before we build the actual system. It has 4 stage: detectability, recoverbility, predictability, and validation eligibility. The paper use 3 real world datasets for testing the effective of the proposed mehtods.

Critical strengths
  • The problem itself is interesting, it separates "having a good backup option" from "being able to predict the right option in advance". This helps developers know if their system is failing because of a bad choice algorithm, or simply because they do not have enough clues beforehand.
  • The paper does not just show math. It proves the concept works on three different real datasets, finding different types of errors in each one.
  • The paper is clearly written and logically easy to udnerstand.
  • The experiment settings are thorough and effective, demonstrating 3 different failure modes.
Critical weaknesses
  1. The framework assumes an pipeline: first you extract clues, then you decide what to do. This is correct for many ML systems, but in many of recent AI experiment settings (e.g., with transformers, MoE,, etc), the training is e2e and the model does not separate "looking for clues" from "making a decision". If you test the clues with a simple proof-of-concept model/tool before training, the tool might fail and tell you to stop, but a deep neural network might find hidden patterns that your audit tool could never see.
  2. When a test fails and the framework says STOP, it does not give the developer clear instructions or tips on how to find better clues or improve their system. It only tells them that they need to redesign it.
  3. How to create an objective registered contract? the intermediate tools and thresholds are defined by human, therefore it does not remove human subjectivity, it just moves it from model tuning to configs. and these human biases could affect the system's stabliity or even flip the results.
Questions for authors

see weakness

2 comments

Thank you for raising the applicability of our diagnostic to jointly trained and end-to-end adaptive systems, as well as the subjectivity of its registered criteria.

1. End-to-end models and stronger representations. We agree that a weak auxiliary predictor cannot establish that a deeper or jointly trained decision model would fail. Our STOP result applies only to the tested action family, decision-time evidence, proposal and scorer family, and evaluation contract. It does not rule out end-to-end routing, richer RGB representations, or future adaptive policies. A jointly trained model can be evaluated within the same framework provided its decision evidence is available at the specified time. If it requires acquiring additional sensors first, the decision timing and cost contract must change.

2. Subjectivity and robustness of the criteria. Registration makes design choices explicit and fixes them before inspecting the audited outcomes; it does not eliminate human judgment. We agree this distinction should be clearer.

For the original contract, SOFT-R1's legal observable safe coverage is 3.0424%. The scene-bootstrap 95% interval is 0.2718%–4.5269%, and 17/2,000 resamples reach the 5% continuation threshold. Nine harm-tolerance settings produce coverage between 0.6983% and 3.7905%. However, reducing the continuation threshold to 3% would change the original point decision from STOP to REFINE. Thus, STOP is stable under the tested scene perturbations and harm settings, but not invariant to development-policy choices.

3. Actionable redesign guidance. The diagnostic identifies a specific failed comparison: substantial outcome-aware safe opportunity is not recovered by the evaluated legal pre-action rankings. One appropriate next experiment would hold the actions, proposals, RGB evidence budget, scene folds, and utility contract fixed while changing only the risk-ranking objective. A separate controlled comparison could then change the legal RGB representation.

These are concrete development hypotheses, not validated improvements. Our present work diagnoses the tested candidate's limitation; it does not establish the optimal redesign or an information-theoretic inability of more expressive models to succeed.

We will narrow the STOP interpretation and make the distinction between registered decision criteria and universal operating standards explicit.

Thanks the authors for the response. My questions are addressed, however I remain concerned about the methods generalizability to more complex systems. As the author acknowledged, some decision changes may be required, and I think more rigorous tests are needed. Therefore I will keep my score.

Reviewer_coGuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Perception Harness, a diagnostic framework that evaluates executable alternatives before controller development in adaptive multimodal inference. Under fixed actions, fallback, evidence, utility, and harm conditions, it separates action identity, recoverability, and pre-action actionability. Experiments on MMAct, MCubeS, and DeLiVER show, respectively, that masking is harmful on average despite detectable degradation, identical modality labels yield different learned action landscapes, and substantial safe opportunity is poorly exposed by the tested pre-action evidence.

Critical strengths
  1. The paper usefully separates opportunity, whether executable actions contain safe gains over a fallback, from actionability, whether pre-action evidence can identify them. Formalizing both under one contract, with actions defined beyond modality labels, lets weak adaptation be traced to the action space or the evidence before effort goes into routers.

  2. The DeLiVER comparison fixes proposals, utility, scene groups, and harm constraints, and computes the full-information ceiling exactly. The 47.13% versus 3.04% gap shows how little of the available opportunity the tested scores recover.

Critical weaknesses
  1. The paper's organization and writing are weak. Dense terminology and the shifting discussion of B2, fixed-MLP, and SOFT-R1 make their roles and evaluation settings hard to track. Figures 1 and 2 do little to clarify the actual diagnostic procedure.

  2. The contribution is mainly an organized diagnostic workflow, with limited methodological novelty. The findings are useful, but the analysis does not sufficiently explain whether failures stem from inadequate information, weak predictors, or other factors, or show how the diagnosis leads to better evidence or actions.

  3. The STOP recommendation rests on 3.04% coverage falling below 5%, without confidence intervals or repeated grouped evaluations. Its sensitivity to harm tolerances is also unclear, leaving the stability of the recommendation under different samples or constraints untested.

  4. The current submission provides no code link or implementation files. Combined with incomplete feature and training details, this makes the main results difficult to reproduce independently.

Questions for authors
  1. How large is DeLiVER VAL, and is STOP stable under scene-level resampling and different harm tolerances?

  2. Does the MLP check also test the risk scorer? How do you distinguish weak predictors from insufficient evidence?

  3. Could you specify Pool7's six non-static experts, the 573-dimensional RGB feature construction, and predictor training settings?

1 comment

Thank you for identifying the difficulty of tracking the different experimental roles and the uncertainty in attributing the observed failure.

1. Clarifying the empirical design. B2 is the primary utility proposer evaluated in the historical one-shot TEST study. The fixed MLP is a separate grouped-VAL proposer used to construct a common candidate population. SOFT-R1 is a legal pre-action risk score evaluated on those fixed MLP proposals. The 47.1322% versus 3.0424% comparison uses the same 2,005 VAL images, 130 scene groups, 1,211 non-static proposals, utility, and harm constraints. It must not be conflated with the B2 TEST result.

2. Methodological contribution and attribution. Our contribution is not a new routing architecture. It is the paired assessment of an exact full-information safe-subset ceiling and a frozen-score observability limit under the same intervention contract. This separates available intervention opportunity from what a specified decision-time ranking can reveal. We agree that the current analysis does not establish the causal source of the remaining gap.

In particular, the fixed-MLP study does not isolate risk-scorer capacity. Matched frozen comparisons show that risk-learning objectives and ranking choices affect achievable coverage. The secondary hard-map score reaches 3.1421%, slightly above the primary SOFT-R1 progression's 3.0424%; both remain below the registered continuation threshold.

3. Statistical stability. Scene-group bootstrap yields a descriptive 95% interval of 0.2718%–4.5269% for SOFT-R1 coverage and 39.6782–50.2496 percentage points for the paired full-information/observable gap. Only 17/2,000 resamples reach 5%, and no single-scene deletion changes STOP. Across nine audited harm-tolerance settings, the fixed-score point coverage ranges from 0.6983% to 3.7905%. These are conditional retrospective sensitivity results, not prospective safety guarantees.

4. Reproducibility. Pool7 consists of the static RGB-D-E-L fallback and six CMNeXt-B2 alternatives: RGB, RGB-D, RGB-E, RGB-L, RGB-D-E, and RGB-D-L. The 573-dimensional features comprise 29 RGB appearance/gradient statistics, 32 RGB-only prediction-map summaries, and 512 pooled RGB encoder coordinates. We recovered the exact ordering, transformations, grouped fitting protocol, and scorer configurations for documentation. A local replay package exists, but no public code or data release is claimed.

We will revise the manuscript to make the experimental roles, numerical scope, and implementation details explicit.

Reviewer_JvzpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Perception Harness, a diagnostic protocol for assessing multimodal interventions before controller development. It distinguishes the existence of beneficial executable alternatives from their identification using pre-action evidence, and evaluates selected interventions under benefit, harm and coverage constraints. MMAct, MCubeS and DeLiVER illustrate different gaps between detection, action specification and selection.

Critical strengths

The distinction between outcome-aware opportunity and opportunity identifiable before acting is useful. The paper also uses grouped out-of-fold evaluation and explicitly separates its retrospective diagnostics from deployment guarantees. These choices help avoid interpreting oracle gains as achievable policy performance.

Critical weaknesses
  1. Figure 1 contains low-resolution raster icons that appear to be AI-generated, with textures resembling those of ChatGPT-generated images. Together with the extensive AI assistance disclosed by the authors, this raises concerns that the paper may have been produced through an automated research workflow.
  2. I do not find the MMAct conclusion convincing. Masking hurts performance on average, but it helps in 26 of the 76 evaluated units. An adaptive selector could potentially identify those cases and leave the others unchanged. Before recommending intervention redesign over further router optimization, the authors need to show how much this selective strategy could gain and whether the existing health signals can identify the beneficial cases.
  3. The DeLiVER results leave the source of failure unresolved. Low safe coverage could reflect inadequate features, a poorly fitted risk model, or an unsuitable ranking objective. The MLP experiment tests proposal capacity, not whether the risk score makes effective use of the available information. I therefore do not see enough evidence that the framework can tell researchers which component actually needs redesign.
  4. The STOP decision rests on comparing 3.0424% with a 5% threshold, yet its stability across scene groups is not evaluated. Independent verification is also difficult because the paper does not fully specify the 573-dimensional features, expert pool or several experimental alternatives. Repeated references to frozen results and artifact hashes cannot replace the definitions and per-sample records needed to reproduce the central finding.
Questions for authors
  1. Which elements of Figure 1 were generated or modified using AI, and can you provide the editable figure sources and clarify how the scientific content was checked?
  2. What is the MMAct oracle gain when keeping the original input remains an option? Can the existing health signals identify the 26 units that benefit from masking, and what supports prioritizing intervention redesign over selection?
  3. With DeLiVER evidence and proposals fixed, how does safe coverage change across risk-learning objectives and ranking models? Why does the proposal-capacity experiment support the diagnosis about evidence or proposals?
  4. Can you provide complete feature and expert definitions, per-unit proposals, gains, OOF scores and scene groups, and quantify the stability of the STOP decision under group-respecting resampling?
1 comment

Thank you for the detailed review. We have re-audited the central claims and identified an important correction to the MMAct interpretation.

1. Figure provenance. Figure 1 includes nine raster icons alongside editable text and shapes. AI assistance in figure preparation was disclosed, but the available source archive does not establish the creation history of each icon, and we cannot certify unsupported provenance details. We have prepared an editable vector-only replacement with the scientific labels checked against the registered decision contract. We will include it in the revised manuscript.

2. MMAct recoverability and selective masking. We agree that negative average masking gain does not establish the absence of recoverable opportunity. Across 76 held-out subject–condition units, always applying the fault-informed masking rule gives −1.4573 accuracy points, whereas a hindsight retain-or-mask oracle yields +0.3931 points. Thus, our original recommendation to prioritize action redesign over selective routing was too strong.

We further evaluated the existing legal sensor-health signals. The mean maximum bad-modality probability gives benefit-prediction AUROC 0.4431 and AP 0.3338, compared with a positive prevalence of 0.3421. Three frozen health-based policies also have negative mean gains. These results do not establish that selective routing is impossible, particularly with only four held-out subjects. We will revise the claim accordingly: detecting degradation does not by itself establish the usefulness of applying a fixed masking response, although selective recovery remains possible.

3. DeLiVER scorer versus representation. We agree that the fixed-MLP experiment tests proposal capacity rather than risk-scorer capacity. Holding 1,211 non-static proposals fixed, the audited dual LR and HGB risk rankings yield 0.2993% and 1.1471% hindsight-safe coverage; an HGB one-point ranking reaches 2.2444%. An existing secondary hard-map ranking reaches 3.1421%, compared with SOFT-R1's 3.0424%. This confirms that the ranking mechanism matters. It does not establish whether the remaining limitation is caused by representation, score learning, or both.

4. STOP stability and reproducibility. The original SOFT-R1 result is reproduced exactly: 61/2,005 legal observable safe coverage versus a 945/2,005 full-information ceiling, across 130 scenes. In 2,000 scene-group bootstrap resamples, the descriptive 95% interval for observable coverage is 0.2718%–4.5269%; 17 resamples reach the 5% continuation threshold. None of the 130 leave-one-scene-out evaluations changes the STOP classification. These analyses condition on frozen OOF predictions and do not constitute independent TEST confirmation.

We also recovered the seven executable expert definitions, the ordered 573-dimensional RGB features, and local gain/score/scene records for reproducibility. We will document these definitions explicitly; the code and records have not yet been publicly released.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.