acceptodds
Under review as a conference paper at ICLR 2027

When Marginal Stability Is Not Enough: Auditing Benchmark Conclusions under Composition Shift

Abstract

Multimodal benchmarks aggregate performance across heterogeneous evaluation contexts, yet it remains unclear how far their conclusions extend beyond the empirical mixture of those contexts. We introduce Composition-Conditional Reliability Auditing (CCRA), which keeps questions and model responses fixed and varies only the weights of observed context cells under bounded total-variation (TV) budgets. We audit four multimodal large language models (MLLMs) on MMAD, an industrial anomaly-detection benchmark, and a fixed three-benchmark panel spanning visual robustness (MVI-Bench), multidisciplinary reasoning (MMMU), and spatial reasoning (MMSI-Bench). Within the audited bounds, a one-percentage-point score change requires moving as little as 1.4–3.7% of evaluation mass on MMAD, MVI-Bench, and MMMU, whereas MMSI-Bench remains within 1.4 points. At the prespecified +1pp selection margin, joint reweighting reaches the endpoint in 6 of 12 native MMAD pair–prompt settings while matched product-only, task-only, and additive spaces do not; one of six MMMU pairs also reaches the endpoint. Larger challenger leads are substantially less common, and no five-point lead is reached within the audited scopes. Separately, on MMAD, joint stress increases held-out error among accepted answers by 6.9–9.7 percentage points for acceptance rules selected on development data and then frozen. These results show that benchmark robustness is conclusion- and margin-dependent, motivating reports that state the composition conditions supporting each score, model-selection, or acceptance conclusion.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, composition shift, multimodal large language models, distribution shift, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fB52OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Composition-Conditional Reliability Auditing (CCRA), a framework for examining the robustness of multimodal benchmark conclusions under changes in evaluation data composition. By reweighting observed context cells while keeping questions and model responses fixed, CCRA investigates score stability, model selection, and confidence-based oversight under bounded distribution shifts. Experiments across four multimodal benchmarks demonstrate that joint composition shifts can affect benchmark scores, model rankings, and selective prediction reliability.

Critical strengths
  • CCRA provides a systematic framework for comparing marginal, additive, and joint composition shifts under bounded total-variation constraints.
  • The paper evaluates four multimodal models across four benchmarks covering different reasoning and visual understanding tasks.
Critical weaknesses
  • The paper does not sufficiently demonstrate how CCRA advances beyond existing distribution-shift and benchmark-sensitivity evaluation methods.
  • The framework identifies adversarial reweightings within predefined constraints, but it remains unclear whether these shifts correspond to realistic changes in deployment distributions.
  • Although the paper controls for total-variation distance, this does not fully establish that the observed differences arise from meaningful interactions between evaluation dimensions.
  • While the paper reports several pairwise ranking reversals, some involve models that are not the overall best-performing model and therefore do not change the final model-selection decision.
  • Although six native MMAD pairwise settings reach the prespecified +1pp challenger lead in the full-data audit, the held-out final challenger gaps range only from -0.2 to +0.64 percentage points.
  • The primary selective-risk experiments are conducted only on MMAD using verbalized confidence scores. It remains unclear whether similar conclusions hold for other multimodal benchmarks, alternative confidence estimation methods, or realistically observed distribution shifts.
  • Additional evaluations on broader model families and open-ended tasks would help establish the general applicability of the framework.
Questions for authors

please see the weaknesses.

Reviewer_i1k3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for auditing benchmark conclusions under composition shifts. It reweights observed evaluation groups while keeping model predictions fixed, and examines score stability, model rankings, and selective risk. Experiments on four multimodal benchmarks show that relatively small composition changes can affect evaluation outcomes, particularly under joint reweighting.

Critical strengths
  1. The paper goes beyond aggregate accuracy and examines how benchmark composition affects model selection and confidence-based decisions.

  2. The experiments cover multiple benchmarks and models, with bootstrap uncertainty estimates and held-out evaluation.

  3. The distinction between score changes and actual ranking reversals is particularly useful.

Critical weaknesses
  1. The framework mainly applies established distributional reweighting techniques and joint reweighting has substantially more flexibility than marginal or additive reweighting, making larger worst-case changes expected. It remains unclear how representative these adversarial compositions are of realistic distribution shifts.

  2. The MMSI reversal results change from 0/6 to 6/6 depending on the cell accuracy estimator, limits the strength of the model-selection claims.

Questions for authors
  • Please compare the optimized composition shifts with naturally occurring benchmark or deployment shifts to assess their practical relevance?

  • How should readers interpret the +1pp model-selection results given that the held-out gaps do not reach this margin? Can the authors further clarify the strong estimator sensitivity on MMSI?

Reviewer_gGsjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method for characterizing the robustness or lack of robustness of conclusions over reweightings of categories of items in a benchmark. This paper uses products and tasks in an industrial anomaly detection as its primary example, but also examines 3 other multimodal benchmarks with different types of category structure. In essence their method selects a reweighting that gives a worst-case outcome for one of several measures of robustness, by constraining this optimization in ways that make it less extremely adversarial. Its constraints are limiting how much of the weight mass that can be shifted and limiting the reweighing to certain families of functions. Meanwhile their metrics for robustness which they worst-case optimize against are change with a reweighting in 1) benchmark score for a given model, 2) gap in score between a given pair of models, 3) error rate given a decision threshold chosen on the benchmark. Their results show that score for a given model can change a lot with minimal weight shift, but that comparisons between pairs of models are either not shifted or need larger weight shifts to flip. They show that an unconstrained reweighting function is key to achieving these model comparison shifts (more constrained reweightings do not flip the comparisons). Finally, they show that error rates for a given decision threshold can increase by up to 10% under worst-case reweighting.

Critical strengths
  1. The issue of characterizing evaluation stability over benchmark composition is very important. Much existing work approaches this from the standpoint of showing variation over item in the benchmark; this work takes a creative approach by treating instability as an optimization target for recovering adversarial benchmark recompositions using structured categories often available for items.
  2. Typically evaluation stability is just considered for the use of evaluations of comparing one model to another, but this work highlights an important and under examined used case of evals that also requires stability: selecting a decision threshold on the benchmark as a dev set.
Critical weaknesses
  1. The work does not compare its method for charactering evaluation stability against other methods from previous work. Even simple baselines such as bootstrap resampling of items or random reweightings would help show what this method shows that others don't. Without this the authors are going to be hard pressed to convince any readers to adopt their convoluted method over many simpler options. Based on the results that are presented, it seems the paper's method more or less reproduces the common finding for foundation models that improvement on evaluations of different kinds are highly correlated (https://arxiv.org/abs/2606.24020 inter alia). That is the reviewed paper shows that reweighing shifts the scores within models but much more rarely shifts the rankings between models. It is interesting to see this finding arrived at from a different method, but it rather contradicts the paper's title that "Marginal Stability Is Not Enough". Meanwhile the decision threshold results are much more compelling but again there is no comparison to simpler methods that might reveal the same instability.
  2. The paper is almost unreadable due to the amount of "neuralese" that mints jargon such as "stress weights" or "frozen ledger" to describe simple things like the worst-case reweighting or the table of model-benchmark results. This becomes more than a stylistic concern when as is often the case these neologisms are not defined anywhere in the body of the paper. Besides rendering the descriptions of methods ambiguous, it makes it unclear what exactly the paper is concludes beyond the need to monitor benchmark instability. I sympathize, since making claims about evaluations is doubly difficult because one must not only show a result but show why the measurement that lead to the result is more interesting than other possible measurements. So I'd accept relatively loose claims like our method shows x and this other method shows y and isn't it interesting that they are different.
Questions for authors

Are the y positions in fig 3 and and 4 meaningful or just jitter within the rows to avoid having the markers overlap?

Did you explore selecting the decision threshold theta for the experiments in fig 5 again after reweighing? Your 80% of answers selection criteria doesn't require labels so it would be realistic that the decision threshold chosen that way could continue to be tuned after deployment.

What baselines do you think would be best to compare your method against as alternative ways to characterize benchmark stability under composition shift?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.