acceptodds
Under review as a conference paper at ICLR 2027

Certifying Data Rankings Under Deployment Uncertainty: When Benchmark Agreement Transfers and When to Abstain

Abstract

Data products are often ranked before the downstream learner is fixed. In a prospectively frozen study on untouched human-activity data, histogram boosting and a standard multilayer perceptron (MLP) significantly prefer opposite acquisition products, drawn from different subjects, under familywise simultaneous intervals. The comparison averages over the registered acquisition and training randomness while the archive, training pools, and test cohort stay fixed, so it does not claim that the reversal will recur for new subjects or sites. The result shows why a single "best dataset" is not always the right thing to acquire. We then ask when agreement among benchmark learners can still certify a ranking under deployment uncertainty. In an exact Bayesian linear-Gaussian model with a shared target, each dataset induces an information operator and each executable learner induces a trace response. Preferring one dataset for every learner in a declared family is then exactly a dual-cone order. No finite benchmark catalog covers all responses in dimension two or more. Projection onto the benchmark cone gives the exact largest normalized reversal that every benchmark accepts, and finite datasets realize every such separating direction. On the positive side, if the convex hull of the benchmarks covers the deployment family to within , a benchmark margin transfers whenever , and the auditor abstains otherwise. Learners whose response depends on the data add an exact one-sided drift penalty . In controlled families the certificate is sound and non-vacuous: a registered sign-aware rule certifies 236 of 480 ridge pairs, which is 76.1% of the 310 pairs where a conservative oracle confirms a uniform preference, with no false certificate. The natural certificate arms return no positive certificates, and a second prospective dataset is ruled uncovered because its samples miss classes. The theorem thus certifies rankings only inside an explicitly covered response family. Learner-free scores remain useful proxies, but a ranking that holds for every deployment requires a declared family, measured coverage, and honest abstention.

Primary area:
learning theory
Keywords:
data valuation, dataset selection, deployment uncertainty, learning theory, selective certification, benchmark coverage, ridge regression, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Mv2xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when agreement among a suite of "benchmark" learners about which of two data products is better can be certified to transfer to an unknown downstream learner, versus when the auditor should abstain. It opens with a prospectively frozen study on human-activity data in which histogram boosting and a standard MLP significantly prefer opposite acquisition products under family-wise simultaneous intervals, motivating why a single "best dataset" need not exist. In an exact Bayesian linear–Gaussian model (shared target θ ~ N(0, I_d)), each dataset induces an information operator K_D and each learner a PSD trace response A, with utility U = tr(A K_D). The theory delivers: a dual-cone characterization of preferring one dataset for every learner in a declared family (Thm 7); a finite-benchmark incompleteness result — no finite benchmark catalog covers all responses in d ≥ 2 — with an exact normalized Frobenius escape margin via projection onto the benchmark cone (Thms 8–9, Cor. 10); a positive coverage certificate (if the hull covers the deployment family to within ε with margin γ, the ranking transfers iff γ > εM, plus a one-sided drift penalty ρ₋ for data-dependent learners; Thms 4, 11); and a minimax-regret abstention rule (Thm 12). Validation is a registry of controlled synthetic-Gaussian studies and natural-data studies (HAR, Covertype, Credit, Bank, Heart, Gas): on a continuous ridge family a sign-aware rule certifies 236/480 pairs (76.1% of 310 oracle-positive pairs) with no false certificate, while the natural certificate arms return no positive certificates and a second prospective dataset (Covertype) is ruled uncovered.

Critical strengths
  1. Intellectual honesty about negative results. The paper explicitly reports that the certificate fires on no natural dataset — Covertype uncovered because size-256 samples miss classes, Bank/Heart/Gas producing no positive certificates, and the HAR Gaussian arm yielding "zero false certificates but also zero positive certificates" — and refuses to recast these nulls as success. This is consequential because it makes the soundness assessment tractable and keeps the paper's framing ("a ranking that holds for every deployment requires a declared family, measured coverage, and honest abstention") matched to the evidence rather than over-claimed.
  2. A clean, correctly-scoped impossibility result. Theorem 8 (finite-benchmark incompleteness for d ≥ 2), with the exact Frobenius escape margin (Thm 9, Cor. 10) and the fact that finite datasets realize every separating direction, is a crisp contribution that is genuinely proved and correctly positioned as a qualitative obstruction plus an exact per-instance cone distance — not an over-general claim. It gives the "when to abstain" half of the paper real teeth.
Critical weaknesses
  1. The positive certificate is only ever demonstrated where the model holds by construction, and certifies nothing where it would matter. Every positive certificate in the paper comes from synthetic Gaussian designs built to satisfy the linear–Gaussian, trace-response, and convex-hull-coverage assumptions; on all natural data the method certifies nothing, which the authors concede leaves its natural-data usefulness open. For a paper titled "Certifying Data Rankings Under Deployment Uncertainty," the method provides no demonstrated certification under any realistic deployment uncertainty — the delivered artifact is a theory plus a stress test showing the theory does not bite in practice. This is the load-bearing limitation: an honest negative result presented in the frame of a certification method.
  2. The headline 76.1% is measured against a self-defined denominator, and the metric is not deployment transfer. The "236 of 480 ... 76.1% of the 310 pairs where a conservative oracle confirms a uniform preference" uses, as its denominator, the authors' own conservative ridge-family oracle — not external ground truth — computed on the same in-distribution ridge sweep the rule certifies over. So 76.1% is in-sample recall against the method's own oracle, and "no false certificate" is defined relative to that same synthetic oracle, neither of which measures whether a ranking transfers to a learner outside the declared family — the quantity the title and intro promise. This metric–claim mismatch is only partially signalled by the "controlled" labeling in the registry and is not flagged where the 76.1% headline is stated.
Questions for authors
  1. Is the conservative ridge-family oracle that defines the 310-pair denominator computed on the same ridge family and designs that the sign-aware rule certifies over? If so, 76.1% is in-sample recall against your own oracle; can you provide any held-out or prospective check in which a certified preference is validated against a learner not in the declared family? This is the single result that would make "transfers under deployment uncertainty" empirical rather than definitional.
  2. Does any theorem-covered certificate (as opposed to a diagnostic classifier result such as the Credit crossover) ever fire positively on any non-synthetic dataset in the paper? If the answer is no, the abstract and title should say so directly, since the current framing invites the opposite reading.
  3. Theorem 4 is deterministic, and the Appendix C uncertainty wrapper requires four simultaneous confidence bounds that, by your own statement, your natural studies with observed labels do not meet. On what basis is the result a "certificate" for a real deployment rather than a diagnostic — concretely, what would a practitioner need to supply to obtain a probabilistic guarantee on real data?
Reviewer_khxSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the problem of choosing which dataset to acquire or buy before knowing which model will be trained on it. Specifically, the authors question the assumption that if Dataset A outperforms Dataset B on a set of popular models, such as Random Forests or MLPs, it will also outperform Dataset B on other models. As I understand it, the paper aims to show that this assumption does not hold in general.

Critical strengths

Interesting problem statement for foundational understanding of ML models.

Critical weaknesses

Clarity of presentation

My main concern is that the paper is difficult to understand, and it gives me a certain degree of uncertainty in the evaluation. The storytelling and presentation are unclear in several places, with sentence often seem disconnected. Starting from the abstract, introduction to the experimental results discussion, the logical presentation flow is chaotic. While reading, I struggled to identify a logical path connecting the motivation, research questions, methodology, and results. Consequently, I found it difficult to understand the precise contribution and assess whether the conclusions are supported. I do not believe the paper to be ready in its current form for publication and would require a major intervention to rewrite it.

I’ll be happy to engage in a discussion with the authors, possibly giving me another key to read their paper and check the opinion of other colleagues. Therefore, I’ll be ready to significantly change my score if convincing.

Lack of background and motivation

The paper does not provide sufficient background or preliminaries to facilitate accessibility to the readers. Furthermore, the research questions are also not clearly formulated and justified. In particular, the authors do not explain why the assumption being challenged is relevant, where it is adopted, and what is already known about its limitations. We would help readers understand the purpose of the analysis.

Use of AI tools

Some passages gave me the impression, because of their logical disconnection, that the paper may have relied substantially on AI tools during writing. Indeed, the AI usage statement reports extensive use of them, from writing and mathematical proofs to coding and interpretation of the results. I would appreciate more transparency about how much, in percentage, the text in this paper is the result of an LLM and how much is the contribution of a researcher. My concern is that the resulting text and reasoning do not always appear sufficiently connected or clearly explained, raising suspicion of intense use of these tools.

Limited experimental evaluation

Considering again, if I understood correctly, the question relates to the assumption that if Dataset A outperforms Dataset B on a set of popular models, such as Random Forests or MLPs, it will also outperform Dataset B on other models. In this regard, the experimental evaluation is limited in scale and relies on small datasets, making the results statistically not significant to respond to such a general problem statement. Furthermore, they are considering only a single modality of data, there is no uncertainty estimation supporting their analysis or heterogeneity across the models utilised.

Furthermore, showing that two architectures perform better on different subsets of the data does not, by itself, establish that the difference is due to the architecture. Other factors can affect performance, including the optimizer, hyperparameters, and initialization. I therefore believe that the ambition is currently greater than the evaluation setup proposed in the paper.

Questions for authors

Can the authors isolates model architecture from other training dynamics (optimizers, initialization, etc.)? This would ensure the observed reversals are not simply artifacts of the training configuration.

Reviewer_RicyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper considers the problem of dataset ranking without knowing the deployment learners in advance. Theoretical results are proved under a Bayesian linear Gaussian model of the learners, and Blackwell informativeness inequalities establish an ordering amount benchmarks.

Critical strengths

Data valuation is an important problem. The paper attempts to solve this problem without prior knowledge of the downstream learners. This could be beneficial in pre-training as well as post-training if achieved.

Critical weaknesses

Poor Writing. The paper's exposition is the number one problem:

  • Very frequently, the terms of art are not defined before use
  • There is no logical continuity between many paragraphs in the intro
  • The paper's marquee exposition figure on page 2 is not only not discussed in the intro, but there is no mention of it until page 5 in Section 5

As an example: Contribution list cites a contribution from Section 8; then Sections 4-5, and then Sections 3-6. The first one reads "A prospectively frozen finding of deployment sensitivity." What is frozen? What is sensitivity? None of this is explained in the paragraphs leading up to this list.

Novelty and Contribution. The method reads as a simple linear model for data valuation with projection into a space of benchmarks with a Blackwell informativeness formulation. None of this is properly presented (see prior point) and cited. For example, the concept of a response (from a game formation) is raised without any attempt at presenting the game. Some games are standard in learning theory, but it is hard to parse what the authors put forward in Section 3 without any problem formulation. The result of this is that establishing novelty is also non-trivial.

As a result, this paper, in my opinion, does not pass the basics of good scientific writing for even a 2nd tier conference, let alone ICLR. I recommend authors withdraw the paper, and reconsider their core problem and introduce it with as few concepts as possible.

Questions for authors

I have no questions that would materially affect my assessment of this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.