- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MECHVAR, a lightweight acquisition rule for sequential mechanism discrimination in autonomous ML workflows. The core theoretical result shows that, under shared-Gaussian predictive variance, the posterior-weighted response variance is exactly proportional to the Box–Hill pairwise-KL criterion, enabling O(KE) vectorized scoring and an additive mechanism-pair audit. Experiments include a synthetic stress audit, a semi-synthetic Digits-Shortcut loop, and a retrospective replay on existing intervention-response tables.
Critical strengths
The MDV–Box–Hill equivalence is simple but operationally consequential: it simultaneously enables vectorized rescoring, pairwise attribution, and a formal local connection to EIG. The proofs are clear and the presentation is well-organized.
The paper is commendably transparent about what it does not do and carefully distinguishes confirmatory from exploratory analyses. The paired Wilcoxon tests with Holm correction, exact sign tests at ceiling, and explicit acknowledgment of unresolved MECHVAR–EIG comparisons reflect strong experimental methodology.
Critical weaknesses
No experiment evaluates MECHVAR in a genuine ML research scenario. This is the central limitation. The paper frames itself around autonomous ML workflows and research agents, however, the stress audit is entirely synthetic (random response matrices, artificial noise). The Digits experiment uses planted mechanisms and pre-designed interventions on a toy dataset. While it involves real training, the mechanisms are artificial constructs, not genuine competing explanations that a researcher would face. The retrospective replay operates on fixed, pre-existing intervention-response tables with no actual experiment execution, yielding near-indistinguishable results across all policies. The motivating diagnostics (FB01–FB06) involve real ML training but, as the paper acknowledges, "do not evaluate the planner."
There is no experiment where MECHVAR is used to resolve a genuine ambiguity in a real ML system. The hardest parts of the pipeline are assumed away. MECHVAR requires three inputs: a finite mechanism set , a probe library , and a response-template matrix . In practice, these are precisely the most difficult components to obtain.
The empirical contribution over EIG is purely computational, not statistical. Across all conditions—synthetic, Digits, and replay—MECHVAR and EIG are statistically indistinguishable in mechanism-identification performance. This is theoretically expected given Propositions 1 and 2. The paper is honest about this, but it means the contribution reduces to: "MECHVAR is a faster way to compute (approximately) the same thing as EIG under shared-Gaussian assumptions." Whether this speedup matters depends on the setting, and for the realistic regime where probe execution dominates time, it likely does not.
Questions for authors
Could you demonstrate MECHVAR on at least one genuine ML debugging scenario where mechanisms and templates are estimated from real preliminary experiments, rather than planted or synthetically generated?
2 comments
Thanks for your response! The scenario you provided did make it a bit more realistic. As my score is already the highest among the reviewers, I will stick with it.
Response to Reviewer zeAx: Executed ML Experiments and Calibration Guarantees
Thank you for the constructive review and for recognizing the statistical care of the original work. Your central request—a genuine ML experimentation setting in which response templates are estimated from preliminary runs—directly motivated the principal revision.
A scenario with executable mechanisms and estimated templates. Sections 5.3–5.4 and Appendix E introduce a new training-time probe study with the following properties:
Beyond linear models (Sec. 5.5, App. H). We additionally evaluate a small convolutional network trained with Adam. Its candidate mechanisms include common training defects such as excessive learning rate, missing input normalization, label noise, insufficient data, and excessive weight decay. This study contains 6,048 executed training runs.
All certificates on the CNN were correct. Increasing calibration from 16 to 64 seeds improves identification accuracy by 3.3 points, while an adaptive race reaches within 1.3 points of the 64-seed result using 43% of its fits.
Removing the Gaussian assumption. Theorem 2 provides an order-statistic-based certificate valid for arbitrary i.i.d. response differences. Across all eight libraries, this distribution-free procedure issued 1,317 certificates, none incorrect, at a cost of 1–41% more fits than the Student-based version.
We acknowledge the remaining limitation: these experiments use planted pipeline configurations rather than naturally occurring failures. Section 6 states this explicitly.
The contribution relative to EIG. We agree that computational speed alone would not constitute a meaningful contribution and have removed that claim.
The revision instead addresses calibration. Matched seeds reduce the cost of obtaining trustworthy probe rankings, while the certificate remains valid under adaptive posterior updates and stopping decisions.
On identification accuracy, our findings are consistent with your expectation: once templates are adequately calibrated, MDV, EIG, and the races perform within approximately one point on the main tasks (Table 3). Under strongly heteroscedastic noise, EIG and variance-aware criteria can outperform MDV (Table 4).
Retrospective replay and motivating diagnostics. We have removed the retrospective replay that could not distinguish policies and the FB01–FB06 case studies that did not evaluate the planner. The synthetic experiments are now restricted to controlled statistical validation, including near-tie and calibration-drift failure modes (Secs. 5.1–5.2).
Thank you again for the constructive suggestions. We would value your assessment of whether the executed training-time probe and CNN experiments address your concern about a genuine ML scenario, or which remaining property would make the evidence more convincing.