acceptodds
Under review as a conference paper at ICLR 2027

MECHVAR: Variance-Guided Mechanism Discrimination for Autonomous ML Experiment Selection

Abstract

A reproduced benchmark gain does not by itself identify the mechanism that caused it. We study finite-library mechanism discrimination: given posterior-weighted candidate mechanisms, executable probes, and a limited budget, which experiment should be run next to separate the remaining explanations? MECHVAR scores each probe by the posterior-weighted variance of its predicted responses. Under a shared-Gaussian predictive model, this quantity is exactly proportional to the classical Box–Hill posterior-weighted pairwise-KL criterion, while admitting an vectorized evaluation and an additive mechanism-pair audit. A local expansion further relates the score to expected information gain (EIG) for small response separations. In a 25-block stress audit, MECHVAR outperforms confirmation-first in several moderate misspecification regimes; its primary comparisons with EIG are statistically unresolved. In a held-out Digits loop, normalized mechanism-identification AUC is 0.8975 for MECHVAR, 0.7825 for a score-greedy policy, and 0.9092 for EIG. At , median single-thread full-library scoring is 10.36 for MECHVAR and 57.69 ms for six-node quadrature EIG in the recorded environment. MECHVAR is therefore a lightweight, auditable rule for finite-library experiment selection when a shared predictive scale is a defensible approximation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
autonomous experimentation, mechanism discrimination, experiment selection, Bayesian experimental design, expected information gain, active learning, model discrimination, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zeAxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MECHVAR, a lightweight acquisition rule for sequential mechanism discrimination in autonomous ML workflows. The core theoretical result shows that, under shared-Gaussian predictive variance, the posterior-weighted response variance is exactly proportional to the Box–Hill pairwise-KL criterion, enabling O(KE) vectorized scoring and an additive mechanism-pair audit. Experiments include a synthetic stress audit, a semi-synthetic Digits-Shortcut loop, and a retrospective replay on existing intervention-response tables.

Critical strengths

The MDV–Box–Hill equivalence is simple but operationally consequential: it simultaneously enables vectorized rescoring, pairwise attribution, and a formal local connection to EIG. The proofs are clear and the presentation is well-organized.

The paper is commendably transparent about what it does not do and carefully distinguishes confirmatory from exploratory analyses. The paired Wilcoxon tests with Holm correction, exact sign tests at ceiling, and explicit acknowledgment of unresolved MECHVAR–EIG comparisons reflect strong experimental methodology.

Critical weaknesses

No experiment evaluates MECHVAR in a genuine ML research scenario. This is the central limitation. The paper frames itself around autonomous ML workflows and research agents, however, the stress audit is entirely synthetic (random response matrices, artificial noise). The Digits experiment uses planted mechanisms and pre-designed interventions on a toy dataset. While it involves real training, the mechanisms are artificial constructs, not genuine competing explanations that a researcher would face. The retrospective replay operates on fixed, pre-existing intervention-response tables with no actual experiment execution, yielding near-indistinguishable results across all policies. The motivating diagnostics (FB01–FB06) involve real ML training but, as the paper acknowledges, "do not evaluate the planner."

There is no experiment where MECHVAR is used to resolve a genuine ambiguity in a real ML system. The hardest parts of the pipeline are assumed away. MECHVAR requires three inputs: a finite mechanism set H\mathcal{H}, a probe library E\mathcal{E}, and a response-template matrix mh(e)m_h(e). In practice, these are precisely the most difficult components to obtain.

The empirical contribution over EIG is purely computational, not statistical. Across all conditions—synthetic, Digits, and replay—MECHVAR and EIG are statistically indistinguishable in mechanism-identification performance. This is theoretically expected given Propositions 1 and 2. The paper is honest about this, but it means the contribution reduces to: "MECHVAR is a faster way to compute (approximately) the same thing as EIG under shared-Gaussian assumptions." Whether this speedup matters depends on the setting, and for the realistic regime where probe execution dominates time, it likely does not.

Questions for authors

Could you demonstrate MECHVAR on at least one genuine ML debugging scenario where mechanisms and templates are estimated from real preliminary experiments, rather than planted or synthetically generated?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zeAx: Executed ML Experiments and Calibration Guarantees

Thank you for the constructive review and for recognizing the statistical care of the original work. Your central request—a genuine ML experimentation setting in which response templates are estimated from preliminary runs—directly motivated the principal revision.

A scenario with executable mechanisms and estimated templates. Sections 5.3–5.4 and Appendix E introduce a new training-time probe study with the following properties:

  • Executed preliminary experiments. Six candidate pipeline configurations are trained under eight interventions on four datasets: retraining, half-data training, label noise, feature noise, weaker or stronger regularization, forced standardization, and early stopping. Each response is obtained from a separate fit, totaling 92,160 training runs.
  • Estimated rather than assumed templates. Templates and observation variances are estimated only from matched-seed pilot runs of candidate pipelines. The hidden target uses disjoint seeds and reveals one probe outcome at a time.
  • Certification and diagnosis results. The paired races issued 2,036 certificates, none incorrect, and achieved empirical coverage of all reference scores (Table 2). Across 1,200 end-to-end episodes, templates estimated from 4–8 pilot seeds reduced accuracy by 1.5–5.0 points and raised confident wrong diagnoses from 1.9% to 3.7–5.9%. Informed selection required approximately 2.5–2.6 target probes, compared with 3.9 for random selection (Table 3, Fig. 3b).

Beyond linear models (Sec. 5.5, App. H). We additionally evaluate a small convolutional network trained with Adam. Its candidate mechanisms include common training defects such as excessive learning rate, missing input normalization, label noise, insufficient data, and excessive weight decay. This study contains 6,048 executed training runs.

All certificates on the CNN were correct. Increasing calibration from 16 to 64 seeds improves identification accuracy by 3.3 points, while an adaptive race reaches within 1.3 points of the 64-seed result using 43% of its fits.

Removing the Gaussian assumption. Theorem 2 provides an order-statistic-based certificate valid for arbitrary i.i.d. response differences. Across all eight libraries, this distribution-free procedure issued 1,317 certificates, none incorrect, at a cost of 1–41% more fits than the Student-based version.

We acknowledge the remaining limitation: these experiments use planted pipeline configurations rather than naturally occurring failures. Section 6 states this explicitly.

The contribution relative to EIG. We agree that computational speed alone would not constitute a meaningful contribution and have removed that claim.

The revision instead addresses calibration. Matched seeds reduce the cost of obtaining trustworthy probe rankings, while the certificate remains valid under adaptive posterior updates and stopping decisions.

On identification accuracy, our findings are consistent with your expectation: once templates are adequately calibrated, MDV, EIG, and the races perform within approximately one point on the main tasks (Table 3). Under strongly heteroscedastic noise, EIG and variance-aware criteria can outperform MDV (Table 4).

Retrospective replay and motivating diagnostics. We have removed the retrospective replay that could not distinguish policies and the FB01–FB06 case studies that did not evaluate the planner. The synthetic experiments are now restricted to controlled statistical validation, including near-tie and calibration-drift failure modes (Secs. 5.1–5.2).

Thank you again for the constructive suggestions. We would value your assessment of whether the executed training-time probe and CNN experiments address your concern about a genuine ML scenario, or which remaining property would make the evidence more convincing.

Thanks for your response! The scenario you provided did make it a bit more realistic. As my score is already the highest among the reviewers, I will stick with it.

Reviewer_Cd8BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed MECHVAR, which is an acquisition rule for sequentially selecting experiments to discriminate between competing model hypotheses. MECHVAR selects the experiment that maximizes the posterior-weighted variance of predicted model responses under each hypothesis. The authors show that MECHVAR, under certain conditions, is equivalent to the Box-Hill model-discrimination criterion, and show a relationship between MECHVAR and Expected Information Gain. They compare the performance of MECHVAR to EIG and other heuristic baselines.

Critical strengths

None

Critical weaknesses
  1. Lack of novelty: MECHVAR is demonstrated to be equivalent to the existing Box-Hill acquisition under the authors' assumptions. None of the main experiments violate these assumptions, so I am not sure what the value is in performing them beyond an evaluation of an existing criterion. Box-Hill is already well established for model discrimination, and the paper has only a very limited discussion of prior work and where this contribution fits among it. For example [1] presents a closely related model discrimination criterion, including a formulation that reduces to Eq. (6) of this paper under the authors' assumptions. On the theoretical side, the computational complexity claims follow directly from the elementary properties of weighted variance. I do not see a sufficiently novel methodological or theoretical contribution here.

  2. Insufficient evidence of practical significance: The main claim is a computational speedup, but the reported speedups are not compared to existing Box-Hill implementations, instead only to a specific numerical EIG implementation. The speedup is impressive, but only in relative terms -- the EIG computation in this setting is quite fast and likely negligible compared to the broader cost of running the experiment and constructing the responses. It appears to me that the only potential novel contribution of this paper is the O(KE) computation of the Box-Hill acquisition, but the authors do not demonstrate their formulation provides any speedup over existing methods for evaluating Box-Hill.

  3. Presentation: I found the presentation of this paper quite inscrutable. Much of the manuscript is written in highly compressed, telegraphic language that introduces new terminology, mathematical objects, and theoretical/experimental claims without sufficient definition or context. Many sentences combine methodology, results, conclusions, and experiments into one sentence. The figures were similarly difficult to follow: for example, Figure 3 has MECHVAR in blue with EIG in green in the left panel; in the right panel, MECHVAR is now in orange and EIG is in blue, so at first glance it appears that EIG significantly outperforms MECHVAR despite the opposite being true.

[1] Schwaab, M., Monteiro, J. L. F., and Pinto, J. C. (2008). Sequential experimental design for model discrimination: Taking into account the posterior covariance matrix of differences between model predictions. Chemical Engineering Science, 63(9), 2408–2419.

Questions for authors
  1. Novelty: Can the authors clarify what is novel about the proposed method, and discuss how their proposed method fits into existing literature on model discrimination, particularly prior formulations of Box-Hill??
  2. Practical significance: Can the authors provide evidence that the hypothesis scoring provides a meaningful computational bottleneck (or perhaps a different type of meaningful bottleneck!) that is solved by this method?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Cd8B: Box–Hill, Schwaab et al., and Practical Significance

Thank you for the detailed review, the reference to Schwaab et al. (2008), and the observation regarding the inconsistent figure colors.

Novelty and relation to Box–Hill and Schwaab et al. We agree that MDV is equivalent to the classical Box–Hill criterion under the stated shared-Gaussian assumptions. Proposition 1 is now explicitly labeled classical, and the O(KE) score evaluation is no longer claimed as a contribution.

Section 2 now positions the work in the literature on model-discrimination design (Hunter & Reiner, 1965; Box & Hill, 1967; Schwaab et al., 2008), common random numbers, sequential ranking-and-selection, and best-arm identification.

The distinction from Schwaab et al. is important. Their posterior covariance of prediction differences arises from parameter uncertainty in fitted models and enters the experiment-design criterion. Our uncertainty arises from matched pilot runs and is used for finite-sample certification of which probe maximizes the selected criterion (Theorem 1).

The certificate holds simultaneously for all posteriors and stopping times, allowing a single calibration to support a sequential diagnosis. Theorem 2 further removes the Gaussian assumption.

We also implemented a diagonal Schwaab-style criterion as a baseline. With eight pilot seeds on the original tasks, it performs similarly to MDV (94.6% versus 94.5%; Table 3), so accounting for template uncertainty inside the criterion alone does not resolve under-calibration when posterior updates still use noisy templates.

Under strongly heteroscedastic noise, however, the Schwaab-style criterion outperforms MDV by 6.3 points on Digits-200 and 6.7 points on Breast Cancer-120 (Sec. 5.5). This supports your observation that variance-aware discrimination matters. A fully covariance-aware Schwaab implementation remains beyond the current study (Sec. 6, App. K).

Practical significance: identifying the meaningful bottleneck. We agree that scoring runtime is not the relevant bottleneck. The runtime claim has therefore been removed. Appendix I reports per-call scoring times below 1 ms, negligible compared with running training experiments.

The revised work focuses on calibration cost: how many pilot fits are required before the selected probe can be trusted?

Two findings are particularly relevant:

  • Paired calibration reduces certification cost. On Wine, where both methods certified successfully, the paired race required 37% fewer fits than the cellwise race. On Breast Cancer, it certified 77% of replications, while the cellwise race certified none. No certificate was incorrect (Table 2).
  • Under-calibration causes measurable identification failures. With 4–8 pilot seeds, accuracy falls by 1.5–5.0 points and confident wrong diagnoses rise to 3.7–5.9%, compared with 1.9% for the paired race (Table 3).

We also report negative findings. On the four original libraries, a fixed budget of 16 seeds matches the race with 417 fewer fits. In three noisier libraries, additional calibration provides little benefit and variance-aware rules outperform MDV by approximately 6–8 points (Sec. 5.5).

On the CNN, the situation differs: calibration from 16 to 64 seeds improves accuracy by 3.3 points, and the adaptive race comes within 1.3 points of the 64-seed result using 43% of its fits.

We therefore present the certificate as a checkable, problem-adaptive stopping rule for the pilot phase, not as a guaranteed cost saving relative to a well-chosen fixed budget.

Presentation and figure consistency. We have simplified the writing, clarified units and uncertainty intervals, and assigned consistent colors and markers to policies across figures. In particular, the paired race is consistently blue and the cellwise race orange. Figure 1 now illustrates the workflow and information boundary, and Algorithm 1 specifies the procedure.

Does this more precise positioning relative to Box–Hill and Schwaab et al. address your novelty concern? If a particular comparison remains especially important, we would be happy to address it during the discussion.

While I appreciate the authors' efforts to address my concerns, I think the revisions made to the paper are too substantial to be properly assessed in a rebuttal period and instead warrant undergoing the full review process. I will maintain my score at 1, and encourage the authors to submit the revised work to a future conference.

Reviewer_JFsyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to choose the next experiment when several mechanisms could explain an observed result and only a few experiments can be run. The authors assume a finite set of candidate mechanisms, a library of probes, and a predicted response for each mechanism under each probe. They propose MECHVAR, which selects the probe whose predicted responses have the largest posterior-weighted variance, and updates the posterior after each observed response. They show that, under a Gaussian likelihood with a shared variance, this score is proportional to the classical Box–Hill criterion and agrees with expected information gain (EIG) for small response differences. Experiments on synthetic tasks and on a Digits task with planted shortcuts show that MECHVAR performs similarly to EIG and better than a confirmation-first rule in some settings, while being much cheaper to compute.

Critical strengths
  1. Choosing which ablation or diagnostic experiment to run next is a relevant problem for ML research, especially for research agents with limited compute.
Critical weaknesses
  1. The technical contribution is limited. Proposition 1 follows from two standard facts: a weighted variance equals half the weighted sum of pairwise squared differences, and the KL divergence between two Gaussians with the same variance is the squared mean difference divided by 2σ22\sigma^2. With equal variances, the Box–Hill criterion is therefore already a weighted sum of squared response differences, so writing it as a variance does not give a new selection rule. Proposition 2 is the standard small-signal expansion of mutual information in a Gaussian channel. The experiments confirm this: MECHVAR is never better than EIG, and EIG is slightly better on Digits.

  2. The setting is far from autonomous ML experiment selection. The method assumes that the candidate mechanisms, the probes, and the predicted response of each mechanism under each probe are all given. In practice, these are the hard parts. In the Digits experiment, the predicted responses are estimated from calibration runs in which the planted mechanism is known, which is not available in a real investigation. The experiments are also small (five mechanisms, ten probes, a budget of four) and do not involve a research agent or a real ML question. Finally, the speed advantage over EIG (microseconds versus milliseconds) does not matter when each probe is a training run, as the paper itself notes.

The writing could also be improved. Parts of the text read like automatically generated prose, with many qualifiers in each sentence and very precise numbers for incidental details (for example, timings in microseconds with hardware and library versions), which makes the main points harder to follow. The six studies in Appendix C are not used to evaluate the method and could be removed. The paper also does not follow the ICLR template and bibliography style.

Questions for authors
  1. How would the mechanisms, probes, and predicted responses be obtained in a real investigation where the true mechanism is unknown?

  2. Is there a realistic setting where the cost of computing EIG is a bottleneck compared with the cost of running the probes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer JFsy: Novelty, Calibration, and Experimental Evidence

Thank you for the precise summary and the two important questions. We agree with your main technical observation and have revised the paper accordingly.

Novelty of Propositions 1–2. You are correct: Proposition 1 follows from the weighted-variance identity and equal-variance Gaussian KL, while Proposition 2 is the low-SNR expansion of the I–MMSE relation. The revision explicitly acknowledges both as classical results (Sec. 3.2), and we no longer claim a new acquisition rule.

The revised contribution instead concerns calibrating and certifying probe selection using executed pilot experiments (Sec. 4). We do not claim novelty for the individual ingredients, including matched designs, common random numbers, Student intervals, union bounds, and LUCB-style racing. Their combination provides a certificate for a posterior-weighted quadratic score based on pairwise response differences. Crucially, the certificate remains valid for every posterior and stopping time, so one calibration can support the entire sequential diagnosis.

Q1: How are mechanisms, probes, and predicted responses obtained when the true mechanism is unknown?

We now distinguish the three inputs explicitly (Sec. 3):

  • Mechanisms and probes are supplied by a researcher or an agent; MECHVAR does not generate them.
  • Predicted responses are estimated by executing each candidate pipeline on matched calibration seeds before examining the hidden target. This does not require knowing which candidate is correct.

In the original Digits study, the templates were obtained from calibration runs involving known planted mechanisms. The new studies remove this dependency: the templates come only from executable candidate pipelines, while the hidden target is evaluated using disjoint seeds and reveals only the outcome of each executed probe (Secs. 5.3–5.4, Apps. E and J).

The candidate mechanisms remain controlled, planted configurations. We explicitly acknowledge this limitation in Sec. 6.

Q2: Is EIG computation a meaningful bottleneck compared with probe execution?

Not in the settings we study, and we have removed the runtime claim. All scoring rules take less than 1 ms per call, whereas even the smallest model fit takes 11–34 ms (App. I). The relevant cost is instead the number of pilot training runs required before a probe decision can be trusted.

Sections 4–5 now investigate this calibration cost directly. Table 3 reports calibration fits, target probes, and identification accuracy together.

Experimental realism and scale. The revised evaluation contains 216,048 executed training runs. The new training-time probe study applies eight interventions to six candidate configurations across four datasets, with 92,160 fits in which each response is a separate retrain (Secs. 5.3–5.4).

End-to-end evaluation demonstrates a concrete failure mode: with only 4–8 pilot seeds, identification accuracy falls by 1.5–5.0 points, while confident wrong diagnoses rise from 1.9% to 3.7–5.9% (Table 3).

We also add a CNN trained with Adam, with candidate configurations representing common defects such as excessive learning rate, missing input normalization, label noise, and excessive weight decay, together with three noise-dominated libraries (Sec. 5.5). These remain controlled benchmarks rather than diagnoses of naturally occurring failures, and we do not claim to evaluate an autonomous research agent.

Writing and presentation. We have rewritten the manuscript for clarity, removed the unrelated case studies from the former Appendix C, and adopted the official ICLR 2027 style and bibliography files.

We would be grateful to know whether reframing the contribution from a new acquisition rule to certified calibration of a classical criterion addresses your soundness concern, or which aspect remains the main obstacle.

Reviewer_JFsyOpenReview ↗

Official Comment by Reviewer JFsy

Thank you for the clear response. However, my main concern about novelty still remains, as the revised contribution is a combination of standard tools for calibrating a classical criterion. I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary and Responses to Shared Reviewer Concerns

We thank all three reviewers for their careful and candid reviews. They agree on three important points, and we accept all three: (i) under a shared Gaussian variance, MDV is the classical Box–Hill criterion, and its local EIG link follows from the standard I–MMSE expansion; (ii) computational speedups over EIG are not practically meaningful when each probe requires a training run; and (iii) the original work assumed predicted response templates rather than explaining how to obtain them from actual ML experiments.

We have uploaded a revised manuscript and supplementary material. The title and core MECHVAR selection framework remain unchanged. The revision addresses these concerns as follows.

1. Positioning and novelty (Secs. 1–3). Propositions 1–2 are now explicitly presented as classical identities, not new acquisition results. We have removed the runtime advantage claim and expanded the related work to discuss Box–Hill, Schwaab et al. (2008), common random numbers, sequential ranking-and-selection, and best-arm identification.

2. Calibration and certification (Sec. 4). Predicted responses are estimated from pilot runs of candidate pipelines, rather than assumed known. Because MDV depends on pairwise response differences, matched-seed calibration permits exact Student confidence intervals for these differences. Theorem 1 provides simultaneous score bounds valid for all posteriors and stopping times, allowing the same calibration to serve the entire sequential diagnosis. Algorithm 1 implements a confidence race that either certifies the MDV-maximizing probe or abstains. Theorem 2 further provides a distribution-free certificate.

3. Executed ML experiments (Secs. 5.3–5.5, Apps. E–H). The revised evaluation contains 216,048 executed training runs. In the new main study, eight training-time interventions are applied to six candidate pipeline configurations on four datasets, producing 92,160 fits in which every response is obtained from a separate training run.

The key results are:

  • The paired races issued 2,036 certificates, none incorrect, with empirical coverage of all reference scores (Table 2).
  • Across 1,200 end-to-end diagnosis episodes, using only 4–8 pilot seeds reduced identification accuracy by 1.5–5.0 percentage points and increased confident wrong diagnoses from 1.9% to 3.7–5.9% (Table 3).
  • In three noise-dominated libraries, additional calibration offered little benefit, while variance-aware criteria outperformed MDV by up to 8 percentage points. On a small CNN, however, adaptive calibration reached within 1.3 points of a 64-seed budget using 43% of its fits (Sec. 5.5, Table 4).
  • The distribution-free certificate issued 1,317 certificates across all eight libraries, none incorrect.

We also report the limitations clearly: a well-chosen fixed calibration budget can be cheaper than racing, no selection rule shows a clear accuracy advantage over EIG in the adequately calibrated main evaluation, and strongly heteroscedastic settings favor variance-aware criteria.

The abstract and Sec. 6 now reflect these findings and limitations. Appendix K maps each reviewer concern to the corresponding revision and remaining limitations. The supplementary material provides the scripts, measured responses, and data needed to regenerate the reported results.

We appreciate the reviewers' feedback and welcome further discussion of any concerns that remain unresolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.