acceptodds
Desk rejected at ICLR 2027

Matched Counterfactual Attribution: A Controlled Evaluation of Data Selection for Video Instruction Tuning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting instruction data is an increasingly important way to control multimodal post-training cost, yet reported gains are difficult to interpret when subset size, baseline choice, training variation, and evaluator behavior change together. We revisit this controlled policy-comparison setting with selector-specific matched counterfactuals, a frozen 10,656-example budget, and a crossed subset-draw by training-seed design. The design separates the total policy contrast (S-U), the shift into the realized data and processing envelope (M-U), and residual ranking within that envelope (S-M). We apply it to a frozen mixed image-video pool with Qwen3-VL-8B training and response-invariant evaluation on six benchmarks. In a homogeneous-H20 3-draw by 3-seed study, ScalSelect yields +1.137 percentage points (pp) for S-U (95% CI [+0.312, +1.963]), but its S-M estimate changes from -0.233 pp under three arbitrary diagonal pairs to +0.263 pp under the crossed design (CI [-0.131, +0.657]). Thus the pairing reverses the attribution estimate without establishing a positive ranking effect. A mechanism-distinct, homogeneous-A800 XMAS adaptation gives +0.530 pp for S-U and -0.009 pp for S-M, with wide intervals spanning both signs. ScalSelect intervals use three training-seed marginals after averaging three control draws within seed; XMAS intervals use 5 marginals after averaging 5 draws within seed. The results do not rank or reject selectors in general. They show that a selector gain is interpretable only relative to an explicit counterfactual resolution, stochastic design, and training-evaluation contract.

Primary area:
datasets and benchmarks
Keywords:
matched counterfactuals, data efficiency, video instruction tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xh6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a methodological paper on evaluating data-selection methods for multimodal instruction tuning. It addresses a core ambiguity in standard evaluation practice: comparing a selected subset S against an equal-size unrestricted random subset U conflates the overall shift in data composition/processing envelope caused by the selector with the residual ranking gain from picking better samples within that envelope. The paper proposes a matched counterfactual attribution framework that decomposes the total contrast into three interpretable components, and advocates a crossed draw-by-training-seed experimental design to avoid unstable, sign-reversible estimates from naive diagonal pairing.

Key contributions:

  1. A three-layer decomposition framework using selector-specific matched counterfactual controls that separate total policy effect (S−US-U), envelope shift effect (M−UM-U), and within-envelope ranking effect (S−MS-M), with explicit reporting guidelines for bounded, reproducible claims.
  2. Empirical demonstration that arbitrary one-to-one diagonal pairing between random control draws and training seeds can reverse the sign of attribution estimates, and that averaging multiple draws per seed with seed-level marginals as independent units produces more stable, less biased estimates.
  3. A rigorous, strictly validated experimental pipeline with frozen budgets, run-admission checks, and hash-verified artifacts, demonstrated on two mechanistically distinct selectors (ScalSelect, XMAS adapter) with Qwen3-VL-8B across six benchmarks.

Validation:

On ScalSelect (H20 hardware), the total policy contrast S−US-U is significantly positive (+1.137 pp, 95% CI [+0.312, +1.963]), but the within-envelope ranking contrast S−MS-M flips sign from −0.233 pp under diagonal pairing to +0.263 pp under the crossed design, with a confidence interval spanning zero. For the XMAS adapter (A800 hardware), all main contrasts have wide intervals crossing zero. Additional sensitivity tests across candidate pools, model stacks, hardware, and reduced grids confirm that results are strongly conditional on the full experimental contract, not universal properties of selectors. The work does not rank selectors; it provides a disciplined evaluation protocol for future data-selection research.

Critical strengths
  1. Rigorous attribution decomposition framework with empirical demonstration of pairing‑induced bias

The paper addresses a fundamental, long‑standing ambiguity in data‑selection evaluation: the standard S−US-U total contrast conflates envelope‑level distribution shifts with within‑envelope ranking gains. It introduces a matched counterfactual decomposition that separates total policy effect, envelope shift, and residual ranking, each mapping to a distinct scientific claim. Most consequentially, it demonstrates empirically that naive diagonal one‑to‑one pairing of control draws with training seeds can reverse the sign of the estimated ranking effect — a finding that calls into question the reliability of many published selector evaluations.

  1. Transparent, contract‑bound experimental discipline with strong exemplary value

The work establishes and follows unusually strict experimental protocols: frozen budgets, stable subset identities, run‑admission gates, hash‑verified artifacts, and an explicit statement that all results are conditional on the full experimental contract (candidate pool, model, hardware, evaluation suite). It further includes systematic sensitivity analyses across candidate pools, model stacks, hardware, and reduced grid sizes to quantify how conclusions shift with contract boundaries.

Critical weaknesses
  1. Low statistical power and inconclusive core ranking estimates weaken the methodological demonstration

The paper’s flagship empirical finding — that naive diagonal pairing reverses the sign of the within‑envelope ranking effect — rests on very low‑powered estimates. ScalSelect has only 3 independent seed‑level marginals and the XMAS adapter only 5, producing wide confidence intervals that consistently cross zero for the core S−MS-M contrast. For XMAS, all main intervals span zero, yielding no conclusive evidence of either a positive or negative ranking effect. Critically, the observed sign reversal could partly arise from high sampling noise rather than being purely an artifact of diagonal pairing. This undermines the strength of the paper’s central methodological warning, because sign instability is expected in low‑powered settings even under correct estimation procedures. Since the core empirical demonstration drives much of the work’s novelty, its inconclusiveness materially weakens the contribution.

  1. Limited practical generalizability and residual confounding in the matching framework

First, matched counterfactuals must be constructed selector‑specifically with no universal matching rule, which adds substantial evaluation overhead for each new selector and makes the protocol difficult to standardize across studies. Second, the M control matches only aggregate distributional features — source quotas and joint token‑cost bins — and leaves many potential confounders unbalanced, so the S−MS-M contrast still does not isolate a pure causal ranking effect; it remains a descriptive decomposition rather than a causal identification. Third, the framework is demonstrated on only two selectors run under separate hardware contracts with small sample sizes, providing limited evidence that it produces stable, actionable conclusions across a broader range of settings. For a methodological paper, limited generalizability and residual ambiguity about what the decomposition actually measures reduce its practical impact on the field.

Questions for authors

The paper’s central methodological claim is that naive diagonal pairing of control draws with training seeds can reverse the sign of the within‑envelope ranking estimate. However, the core S−MS-M estimates have very low statistical power, with wide confidence intervals that consistently cross zero. Could the authors clarify whether there are additional simulation analyses or expanded experiments with more training seeds/control draws that demonstrate the sign reversal is primarily an artifact of diagonal pairing bias, rather than natural sampling variability in low‑powered settings?

The current matched control M only balances aggregate distributional features such as source quotas and joint token‑cost bins, so the S−MS-M contrast remains a descriptive decomposition rather than a pure causal estimate of ranking ability. Additionally, the framework is validated on only two selectors run under separate hardware contracts with small sample sizes. Could the authors clarify whether they have explored finer‑grained matching (e.g., on per‑example difficulty or semantic attributes) to further reduce residual confounding, and whether there are supplementary results across more selectors or candidate pools demonstrating the stability and generalizability of the decomposition?

Reviewer_xETzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the evaluation of data selection policies through matched counterfactual comparisons. It introduces a controlled evaluation protocol involving matched controls and crossed draws and random seeds, aiming to separate the apparent contribution of a selection policy from the effects of baseline construction and experimental randomness. The paper highlights a sign change between a diagonal estimate and a fully crossed estimate as a central empirical finding.

The paper addresses a relevant problem: data selection methods can be difficult to compare fairly when experimental outcomes depend on the construction of control subsets and stochastic training runs. However, I have substantial concerns about the statistical support for the paper’s main conclusion, the sensitivity of its reported estimates, and the novelty and practical significance of the proposed methodology. In its current form, I do not believe the paper provides sufficiently strong evidence for its central claims.

Critical strengths

Relevant evaluation problem. The paper addresses the important issue of confounding and variability in the evaluation of data selection policies.

Explicit treatment of experimental variation. The crossed draw–seed design is potentially useful for revealing how conclusions depend on the choice of experimental pairing.

Attention to reproducibility. The paper describes execution manifests, source hashes, validation procedures, and scripts for reproducing reported results. These are useful elements of a reproducible evaluation pipeline.

Critical weaknesses
  1. The central sign-reversal claim is not adequately supported by the statistical evidence.

The paper emphasizes the change from a diagonal estimate of −0.233 percentage points to a fully crossed estimate of +0.263 percentage points as a central empirical finding. However, the reported 95% confidence interval for the crossed estimate, [−0.131, +0.657], includes zero. The three seed-level marginal estimates are also all non-negative (+0.404, +0.293, and +0.092 percentage points). These results do not establish a reliable positive effect or demonstrate a meaningful reversal in the underlying ranking of selection policies.

The distinction between a change in point estimates and evidence of a genuine effect is essential here. The results may illustrate sensitivity to the evaluation estimator, but the paper does not provide sufficient evidence to elevate this sensitivity into a central empirical finding. The main conclusion should be substantially narrowed unless stronger statistical evidence can be provided.

  1. The results are sensitive to the choice of pairing, but this sensitivity is not adequately characterized.

The diagonal estimate corresponds to one particular pairing of draws and seeds, whereas the fully crossed estimate averages across all combinations. The paper reports that the six one-to-one pairings produce mean differences ranging from −0.233 to +0.878 percentage points, with five of the six estimates being non-negative. This indicates that the sign and magnitude of the estimated effect depend substantially on the pairing and aggregation procedure.

The paper does not sufficiently characterize this dependence or establish how robust its conclusions are to individual cells. In particular, a systematic leave-one-cell-out analysis and uncertainty estimates for the relevant contrasts would help determine whether the reported pattern is stable. Since the same observations contribute to the different estimators, uncertainty in their difference should also be evaluated directly rather than inferred from the separate confidence intervals. Without such analyses, the paper’s interpretation of the sign change remains fragile.

  1. The methodological novelty relative to existing evaluation protocols is insufficiently established.

The decomposition (S-U=(M-U)+(S-M)) is an algebraic identity and does not, by itself, constitute a methodological contribution. The potentially meaningful contribution instead lies in the combination of matched controls, crossed draw–seed comparisons, and a standardized reporting protocol.

However, the paper does not clearly establish the incremental value of this combination over existing approaches based on fixed protocols, repeated random baselines, or mixture controls. It is unclear which specific methodological component is essential, what failure mode it resolves beyond these alternatives, and how much the proposed protocol improves the reliability of practical conclusions. The paper would need a more explicit comparison with relevant existing evaluation practices and evidence isolating the benefits of its individual design choices to substantiate its methodological contribution.

  1. The computational cost of the selection procedure is not adequately measured.

The paper describes matching based on joint processor-cost bins and aggregate token counts within a stated tolerance. However, it does not report measured training FLOPs, wall-clock time, GPU-hours, or the computational overhead introduced by the selection procedure itself.

This omission limits the practical interpretation of the evaluation. Matching selected cost proxies does not necessarily establish that competing policies incur comparable realized computational costs, and the selection procedure may itself introduce overhead that affects its practical utility. The paper should report these quantities or explicitly restrict its claims to the cost proxies it actually controls. Without this information, the practical trade-offs associated with the proposed evaluation remain insufficiently characterized.

  1. Figure 1 and the presentation of the methodology need substantial improvement.

Figure 1 is difficult to follow because some text elements overlap, which reduces readability. This is particularly problematic for a paper whose main contribution depends on distinguishing several estimators, controls, and sources of variation.

More broadly, the paper introduces specialized terminology and methodological distinctions without always making their relationships immediately accessible. A clearer Figure 1, a concise worked numerical example demonstrating how the key estimates are calculated, and a more operational description of the subset-matching procedure would improve both readability and reproducibility. These presentation issues make it harder to assess and independently implement the proposed protocol.

Questions for authors

What evidence supports interpreting the sign change as a substantive empirical finding rather than sensitivity to the estimator and pairing?

How do the main conclusions change under leave-one-cell-out analysis and alternative aggregation schemes?

What specific advantage does the proposed protocol provide over existing repeated-baseline and matched-control evaluation practices?

How do the policies compare in realized training cost and total computational overhead?

Reviewer_vCbWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Matched Counterfactual Attribution evaluation framework for video instruction fine-tuning data selection. By constructing randomized control groups at different levels, the framework decomposes the overall performance improvement of data selection methods into overall strategy gain, gain from changes in data composition and processing features, and residual gain from sample ranking under matched conditions. The authors conduct experiments on two data selection methods, ScalSelect and XMAS. The results show that different counterfactual control groups may significantly change the interpretation of the effectiveness of data selection methods, providing a more fine-grained evaluation perspective for multimodal data selection.

Critical strengths
  1. Meaningful research motivation and a more interpretable evaluation framework for data selection. The paper points out that existing data selection methods are usually validated by comparison with randomly sampled subsets of equal size, but this evaluation approach cannot distinguish gains brought by changes in data composition from gains brought by sample ranking itself. To address this issue, the paper proposes a stratified matched counterfactual evaluation framework that decomposes overall performance improvement into contributions from different sources. This design helps to more accurately understand the effectiveness of data selection methods and provides a more interpretable evaluation protocol for subsequent multimodal data selection research.

  2. Relatively comprehensive experimental evaluation and fine-grained performance analysis. The paper validates the proposed evaluation framework on two methods with different selection mechanisms, ScalSelect and XMAS, and evaluates it using six benchmarks: MVBench, Video-MME, MLVU, LVBench, MMMU, and RealWorldQA. In addition, the paper provides detailed results on overall performance, performance differences under matched conditions, and different tasks, and analyzes the scope of applicability of the evaluation conclusions through additional experiments on data pools, model configurations, and training budgets. These experiments provide valuable empirical evidence for understanding data selection gains under different counterfactual control conditions.

Critical weaknesses
  1. Meaningful research motivation and a clear evaluation framework. The paper points out that traditional random baselines make it difficult to distinguish gains brought by changes in data distribution from those brought by sample ranking, and proposes a stratified matched counterfactual framework that makes the performance improvement of data selection methods more interpretable.

  2. Relatively comprehensive experimental validation. The paper conducts experiments on two different data selection methods, ScalSelect and XMAS, covers six multimodal benchmarks, and provides detailed analyses of overall performance and task-level performance, enhancing the credibility of the experimental conclusions.

Questions For Authors

Questions for authors
  1. Can the authors further clarify the core methodological innovation of this framework compared with existing matched control and data distribution control methods? In addition, can they provide a deeper mechanistic explanation than the existing performance decomposition by further analyzing factors such as data quality, semantic diversity, or sample difficulty?

  2. Can the authors further explain why data source, temporal category, and token distribution were chosen as the main matching conditions? Are these features sufficient to control the key factors affecting data selection effectiveness? In addition, can ablation experiments with different combinations of matching features verify the sensitivity of the final attribution conclusions to the matching strategy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.