acceptodds
Under review as a conference paper at ICLR 2027

Release-State Audits for Retrospective Machine Learning

Abstract

A dataset name or event-time cutoff does not uniquely specify the evidence available to a retrospective machine-learning evaluation. We introduce release-state audits: a closed reference specification linking exact source-state commitments, support accounting, estimands, and admissible claims. It separates support migration from retained-unit change and specifies typed non-results when identity, label, or missing-support requirements fail. The current v0.2 one-input checker validates only closed aggregate types and listed algebra over caller-supplied declarations and always withholds; it does not establish source replay or authorization. Three formal boundaries motivate the specification: a terminal event-time view need not identify earlier availability; finite agreeing label states do not establish permanence; and union-paired contrasts with release-exclusive support depend on registered missing-side restrictions. A separate closed synthetic adapter/evaluator fixture supports only finite implementation conformance, not real-world detection performance. We examine one exploratory pre-merge BIG-bench task-state case using three exact model revisions and a fixed census of 998 effective inputs. The complete four-cell panel exhibits descriptive grade differences between task states. Under no-appended-choice rendering, all three models have identical aggregate grades near the cell-specific uniform-choice expectation, without establishing identical predictions. This limited vignette illustrates why task state belongs to the evaluation object; it does not establish post-release benchmark drift, causal repair effects, or general model capability.

Primary area:
datasets and benchmarks
Keywords:
dataset versioning, benchmark auditing, retrospective evaluation, data provenance, machine learning reproducibility, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_audUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper appears to ask whether a retrospective machine-learning study uses only evidence that was available at the historical time under study. The authors propose an audit framework for limiting claims when the historical record is incomplete. I am not confident that this reading is right.

Critical strengths

I could not identify any.

Critical weaknesses

The paper reads as unedited AI-generated text. The key problem is not a lack of clarity. The clarity is so poor that I cannot engage with the paper at all: I cannot understand its motivation, and I cannot assess the soundness or significance of its contribution. Specifically:

  • W1. The paper never explains in plain language what a retrospective machine-learning evaluation is.
  • W2. It does not say what is being evaluated, what evidence is used, or what the evaluation is for.
  • W3. I could not tell what the proposed solution is or whether I would ever apply it.
  • W4. The terminology and Figure 1 do not give the reader a concrete picture of the framework.
Questions for authors
  • Q1. Can you give a concrete example of a real-world problem that you are solving?
  • Q2. In that example, what is being evaluated, what evidence is available, and what decision must be made?
  • Q3. In plain terms, what is the problem, and what is your solution?
  • Q4. Why should anyone use it?
  • Q5. How does this paper advance science?
  • Q6. Please walk through Figure 1 using the same example.
Reviewer_2qmaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that same dataset and cutoff date does not mean that a ML is evaluated on the same evidence. The data and other information may have changed over time and can affect the evaluation. The paper propose "release-state audit": to record what data was available at different times and how changes affect the evaluation. The authors built a tool called v0.2 one-input checker. The paper includes experiments on BIG-bench with three different models distilGPT2, T5-small and T5-base.

Critical strengths
  1. The author's method of tracking unchanged, remove and newly added examples into the dataset provides a practical checklist because researchers evaluating models on datasets can assume that data available today is the same as past.
  2. The authors are very transparent and reported everything -- one finding that score changed by only 1.3 percentage, even though grades of approx. 34% examples changed is interesting shows small change in average score can hide large change in individual results.
Critical weaknesses
  1. All 998 input remain unchanged across versions, methods for handling added or removed data and estimating their impact are never tested; also the planned DBLP study remains imcomplete.
  2. All three model achieve 16.83 accuray in certain setting; the authors did not investigate this anomaly or established the score differences is due to genuine changes in model behavior.
Questions for authors
  1. Can you show any real dataset or benchmark pair with non-empty E or X?
  2. Why is a checker that always returns "withheld" meaningful? What would a user learn by running it that they could not learn from checking by hand?
  3. In q1q_1 cell what is the distribution of selected choices per model and degree of agreement across the three model?
Reviewer_4iaPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces release-state audits to compare retrospective evaluations across different versions of datasets and benchmarks. The core argument is that a dataset name or event-time cutoff does not uniquely specify the evidence available to an evaluation, so cross-state comparisons require a closed reference specification binding exact source-state commitments, support partitions, estimands, and admissible claim types. The framework is instantiated on one pre-merge BIG-bench task pair with three small models (DistilGPT2, T5-small, T5-base) on 998 inputs, yielding observed-state grade differences of -1.1% to -1.5%.

Critical strengths

Benchmark versioning silently changes what an evaluation measures, and the ML community routinely conflates dataset names with stable evidence objects. Separating event time from evidence time (Proposition 1) and distinguishing retained-unit change from support migration are genuine conceptual contributions.

The decomposition in Equation 2 cleanly separates retained-unit change from entry/exit composition and handles empty sets without fabricating means. Table 2's coverage comparison against related work is an effective positioning device.

Critical weaknesses

The general writing flow in the paper and style can be greatly improved for clarity. I found myself needing to reread the same paragraph multiple times at a lot of spots. There is also a lot of text on what the approach does not do or not show that makes it very clunky to read. It honestly reads like generated text and this shoud be addressed.

The v0.2 checker validates only caller-declared aggregates and always returns WITHHELD. Bundle replay, the stronger specification that would establish source-derived claims, is not implemented. The authors therefore describe a specification rather than delivering a working system, and the specification itself is not validated beyond its own synthetic fixtures (54 designs, 24 regression tests). This makes the contribution closer to a position paper than a systems or methods paper.

Content-hashing data files, recording the commit SHA, and reporting scores on the intersection already captures much of what C/E/X partitioning formalizes. The paper never shows where this simpler practice breaks down and the full typed-claim apparatus becomes necessary. Without such a comparison, the overhead of the verification bundle, seven-constructor grammar, and ordered reference semantics is not clearly justified.

The planned DBLP production-snapshot study is prominently discussed but contributes zero results. Mentioning unexecuted future work at this length creates an expectation the paper does not meet. It would be better either to execute it or to cut the discussion entirely.

One pre-merge task pair, three small/older models, and a fixed census of 998 inputs in a near-uniform-reference scoring regime (all q1 grades collapse to ~17%, barely above uniform-choice expectation). The paper acknowledges this extensively, but acknowledgment does not solve the problem. The BIG-bench vignette demonstrates that task state can change a grade, which is unsurprising. It does not demonstrate that the audit framework catches something that a simpler diff-based approach would miss, or that the typed claim system adds value beyond careful bookkeeping.

The writing needs improvement. It does not currently read like

Questions for authors

Can you demonstrate a concrete scenario where the full typed-claim apparatus detects or prevents a misinterpretation that a simpler hash-and-intersect workflow would miss?

What would it take to implement BUNDLE_REPLAY and move the checker from "always withholds" to actually issuing claims? Is this planned for a revision?

How should practitioners decide which claim type (Native, Retained, Union, etc.) is appropriate for a given evaluation? The paper defines the grammar but gives no guidance on when each type is the right choice in practice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.