acceptodds
Under review as a conference paper at ICLR 2027

ViewCert: Verifying Model Selection under Evaluator Ambiguity

Abstract

Model selection is usually reported under one evaluator, even when the same public data admit several defensible choices. ViewCert verifies whether a model-selection claim holds for every program in an explicitly declared, mechanically complete typed language while reusing fixed outputs. Unstructured verification is coNP-complete. Factored margins reduce exactly to classical min-sum elimination, with cost exponential only in interaction width . ViewCert returns a replayable worst-case program, handles the full aggregation simplex without a grid, and jointly covers the program, competitor, weights, and budget selected after search. As a post-outcome algorithmic closure on Open Images, an audited width-two compiler verifies Grounding DINO Base against five competitors over evaluator programs without enumerating their leaves; its worst-case margin is AP. Prospectively on WMT25, five official metrics certify Shy over Gemini-2.5-Pro, while two registered LLM judges certify the reverse among 15,120 covered comparisons. Across six regimes, ViewCert distinguishes stable selection, certified ranking dependence, and principled non-identification.

Primary area:
datasets and benchmarks
Keywords:
machine learning evaluation, model selection, evaluator ambiguity, benchmark robustness, statistical certification, post-selection inference, partial identification, structured verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ey6zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

ViewCert formalizes model selection under evaluator ambiguity by applying a declared typed program language to fixed, per-model prediction banks. It provides mechanical completeness checks, exact factored verification, grid-free optimization over aggregation weights, replayable counterexamples, and post-search confidence curves under stated assumptions. Experiments across vision and translation benchmarks show stable winners, certified ranking reversals, and non-identification. Conformance tests, locked resampling, synthetic exactness checks, and an AP-native audit support the implementation; the Open Images program closure is post-outcome, whereas the WMT transfer is prospective.

Critical strengths

The paper formulates a useful and precise decision problem. Separating evaluator ambiguity from sampling uncertainty, and distinguishing a certified reversal from failure to identify a winner, is conceptually valuable for benchmark methodology. The completeness checker, replayable witnesses, and explicit dependence on the declared language make the resulting claims substantially more auditable than a conventional sensitivity table.

The exact simplex and transport argument avoids a weight grid, the structured solver is checked against exhaustive enumeration, and the paper clearly labels conditional guarantees, post-outcome closures, and the human-evaluation extension that was halted at its pre-registered metadata gate. The prospective WMT lock and shared-output design provide a credible demonstration that the method can expose evaluator-dependent conclusions without rerunning models.

Critical weaknesses

The experiments cover object detection and machine translation, but the framework’s applicability to other task types remains unclear. Additional evaluation on tasks such as question answering or summarization, where evaluator ambiguity arises from different correctness criteria or reference choices, would help establish its broader usefulness. In particular, it would be valuable to show whether the evaluator language can be specified and the verification procedure applied without substantial task-specific engineering.

Questions for authors

How much task-specific engineering is required to apply ViewCert beyond object detection and machine translation? Could the authors provide evidence from another task, such as question answering or summarization, to demonstrate that the evaluator language and verification procedure remain applicable?

Since every certificate is conditional on the registered language, its practical strength depends on how that language is chosen. How do the authors justify the boundary of the registered evaluator language and the granularity of its semantic atoms, and which plausible evaluator choices are excluded before certification?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ey6z (1/2): QA and summarization

Thank you for the review. We answer the two questions in turn.

Q1. Task-specific engineering, and evidence from another task.

We ran both tasks you suggested, on the shared code unchanged, and QA produces all three answers. What is task-specific is the scoring code (exact match, ROUGE-L, and so on); the completeness check, the simultaneous intervals and the certificates are shared and were not modified. In numbers: 40 lines of scoring code for QA and 29 for summarization, plus one runner script per study (242 and 156 lines) that loads the data, declares the list and calls the shared functions. Both studies use the finite-list part of the method, and QA also the weighting of its two datasets. In both, the list of evaluators was fixed and hashed before we read any score from our own runs (the published findings were known; see the caveat below).

Open-domain QA (EVOUNA; Wang et al., NeurIPS 2023). Answers of five systems (FiD, GPT-3.5, ChatGPT, GPT-4, Bing Chat) to Natural Questions and TriviaQA questions, with human correctness labels. The 16 evaluators cross {exclude, include} flagged questions × {all gold aliases, first alias} × {exact match, token F1, gold containment, human label}. We keep the 5,008 questions with complete answers and labels, and use 95% simultaneous intervals over 160 margins.

  • Under exact match, FiD is certified (63.9 against at most 12.8), and no reweighting of the two datasets changes this.
  • Under first-alias containment, Bing Chat is certified instead: the winner itself is certified to depend on the criterion.
  • Replacing only the criterion by the human label, FiD − GPT-4 goes from +62.0 [59.7, 64.4] to −9.3 [−11.4, −7.2], a certified reversal with a single edit.
  • Under the human label, GPT-4 versus Bing Chat is not identified (−0.3 [−2.0, 1.5]).
  • Nine of the ten pairs reverse with certified signs somewhere in the list.

Summarization (SummEval; Fabbri et al., TACL 2021). Summaries of 16 systems for 100 articles, with 11 references per article and three expert ratings per summary. The 10 evaluators cross {first reference, all eleven} × {ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, expert relevance}; 9 are distinct, since the expert ratings do not depend on the reference.

  • With 100 articles no overall winner is certified. Pegasus, T5 and BART are not separated under ROUGE or under expert relevance.
  • Twelve pairs reverse with certified signs. All involve the GPT-2-based system, which is last of 16 under ROUGE-L, 15th under expert relevance and 2nd under BERTScore with all references. Against Lead-3 its margin is −5.71 [−9.69, −1.73] under ROUGE-L (first reference), −0.84 [−1.22, −0.46] under expert relevance, and +1.54 [0.52, 2.56] under BERTScore (all references).

A Bonferroni critical value instead of max-t changes none of these answers.

One caveat. That lexical metrics disagree with human judgments on these benchmarks was known before our analysis. We therefore present the two studies as evidence that the method transfers to other tasks with little code, and we make no new claim about the systems themselves. Details are in Table 4 and App. H–I.

AuthorsOpenReview ↗

Response to Reviewer ey6z (2/2): boundary of the evaluator list

Q2. How is the boundary of the list justified, what is the granularity of atoms, and which choices are excluded?

We fix the boundary with three admission rules (§6.1), list the plausible choices each study leaves out (App. J, Table 19), and test the most consequential ones: under AP averaged over IoU 0.50 to 0.95, Grounding DINO Base remains certified, and growing the SummEval list to 16,380 evaluators changes the certified reversals only from 12 to 14. Choosing the list remains a judgment call; the method makes it explicit and checkable.

  1. Public provenance. An operator must be used or documented by the benchmark maintainers or by a published protocol, for example the official partial-verification, hierarchy and group-of rules of Open Images with their documented alternatives, or the official metrics, normalizations and aggregation levels of WMT25. The 2502^{50} Open Images family goes beyond this rule, and we report it only as a stress test.
  2. No dependence on model outputs. Static checks ensure that no operator takes predictions or scores as input. They cannot detect an operator chosen after looking at results; the next rule addresses that.
  3. Fixed before outcomes. For confirmatory claims the list is hashed before the scores are read (the detection replications and the three WMT studies); later extensions are marked post-hoc. For WMT25 the organizers' own rankings were already public, so the lock only precedes our access to the score files, and one metric's sign was corrected afterwards (general response, 2/2).

Granularity. One edit is one documented decision that a maintainer could change on its own, such as whether category hierarchies are expanded. This is a convention of the registry: the same evaluator can cost one edit in one registry and several in another, so edit budgets are comparable only within a registry. What we can guarantee is that encoding the same decision with several grammar nodes changes neither the edit count nor the certificate (Prop. 7, App. A).

Exclusions. Table 19 lists, for each study, plausible choices left outside the declared list, for example IoU thresholds other than 0.5 for Open Images, any metric or LLM judge beyond the five official ones for WMT25, and other reference subsets, unstemmed ROUGE, BERTScore with idf and the other expert dimensions for SummEval.

What happens when excluded choices are added. Adding evaluators can never strengthen a stable-winner certificate, since the set of uniform winners can only shrink (Prop. 8); it can create or remove reversal certificates. We ran it twice, with specifications hashed before any new score was computed.

  • SummEval, all four kinds of excluded choices added in three cumulative steps: the list grows from 9 distinct evaluators to 12, 20 and 16,380, the critical value from 3.94 to 4.95, the pairs ordered with a certified sign go 79, 95, 95, 92 (of 120), and certified reversals 12, 12, 12, 14. Selection stays "not identified" (App. I, Table 18).
  • Open Images, AP averaged over IoU 0.50 to 0.95 instead of AP at IoU 0.5. This is the choice that could hurt the certificate that matters; it is analyzed as a separate two-program family (official rules, both criteria) with one simultaneous band over its 30 margins. GD-Base remains certified: its macro AP is 49.57 (against 60.40 at IoU 0.5), and its closest competitor, YOLO-World, trails by 16.08 AP (simultaneous 95% interval [14.78, 17.39]); the point weight-change radius is 0.297, against 0.240 at IoU 0.5 (App. J).

We hope the two new studies and these rules answer both of your questions, and would be grateful if you would take them into account in your assessment.

I appreciate the authors running the QA and summarization studies, and the explicit admission rules help clarify how they choose the evaluator list. I will keep my score.

Reviewer_BZTBOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the model that wins a benchmark would still win under other reasonable evaluation choices. With model predictions held fixed, ViewCert searches a declared set of evaluation rules and aggregation weights for changes that could overturn the ranking. It can certify that a model remains ahead within the allowed changes, produce a configuration that reverses the ranking, or report that the data are insufficient for a reliable conclusion. Experiments cover object detection and machine translation.

Critical strengths

The paper addresses a practically important question: how much does a model's apparent advantage depend on the way we evaluate it? The framework goes beyond reporting rankings under several settings by identifying changes that overturn a result and accounting for sampling uncertainty. Its applications to real evaluation tasks demonstrate the value of examining the reliability of model selection.

Critical weaknesses

The experiments do not yet fully demonstrate the combined framework in a complex real setting. The largest discrete search example has a structure that makes its choices easy to optimize separately, while the analysis of continuous weights and statistical confidence is mainly demonstrated on a smaller set of evaluation programs. This leaves some uncertainty about the computational cost and statistical reliability when these capabilities are used together.

Questions for authors
  1. In the large Open Images example, category-level choices can be optimized independently once the global options are fixed. How does the general solver compare with a simple method that directly exploits this structure? Is there a real evaluation setting with more substantial interactions between choices?

  2. Could the authors demonstrate discrete rule search, continuous weight changes, and statistical confidence bounds together in one real setting? In particular, how does the existing coverage check for pairwise score differences support the reliability of confidence bounds on the amount of change needed to overturn a ranking?

4 comments
AuthorsOpenReview ↗

Response to Reviewer BZTB (1/2): separable structure and interacting choices

Thank you for the review. Both questions pointed at weak spots: the first led to three new analyses, and the second made us rerun our own validation and correct two statements (general response, 2/2).

Short answers. Q1a: on this family the solver has no advantage over a direct per-category computation, and the revision says so. Q1b: yes, in three real settings; in each, treating the choices one at a time misses reversals or overstates robustness, so they must be solved exactly, although none needs variable elimination specifically. Q2a: yes, on the registered Open Images grammar, and the 2502^{50} family now has confidence bounds too. Q2b: it did not; the revision reports four direct checks of the radius bounds, three of them new (part 2).

Q1a. Comparison with a method that exploits the separable structure.

Your reading is correct. The support rule of category c only changes the AP of c, so once the two global choices are fixed the worst program is found category by category. This direct computation returns the same worst margin as variable elimination for all five competitors (+11.953 AP in the worst case, against YOLO-World), with the same program and edit count at every budget, and it reads about as many table entries (384 against 390). The revision says so, removes the "1.1×10151.1\times10^{15} leaves" comparison from the figure, credits bucket elimination to Dechter (1999), and states that this per-category family is our own post-hoc construction.

Q1b. A real setting with stronger interactions.

The declared lists. If the stages of a list were separable, picking the worst option of each stage on its own would find the worst evaluator. On WMT25 (36 evaluators) and on SummEval's declared list this stage-by-stage search fails: for 7 and 8 pairs it returns an evaluator that does not reverse the pair although the list contains one, and it misses the worst evaluator for 238 of 420 and 38 of 240 ordered pairs (point estimates; App. G, Table 14). On Open Images (8 evaluators) and QA (16) the interactions are mild and no reversal is missed. These lists are small enough for ViewCert to enumerate exactly.

Reference subsets in summarization. SummEval has 11 references per article, and a summary's score is its maximum over the references used, so the choice of references has eleven yes/no parts that interact through the maximum. We evaluated all 2,047 subsets under 8 automatic criteria, plus 4 expert criteria: 16,380 evaluators under one simultaneous band. Three searches that treat the eleven parts one at a time (adding references, dropping references, an additive fit) miss the worst subset in 81% to 97% of the 1,920 criterion-pair combinations; in the 1,690 where some subset puts the second system ahead (point estimates), they find no such subset in 7% to 30%. Under ROUGE-L, 11 different systems lead for some subset, yet with 100 articles no pair reverses with certified signs between two subsets; the certified reversals (14, against 12 in the declared list) all come from changing the criterion (App. I, Table 18).

Edit budget and reweighting together on Open Images. In the 2502^{50} family the category choices stop being separable once an edit budget is combined with reweighting, because which categories are worth switching depends on which keep their weight. A small mixed-integer program solves this exactly (checked against enumeration on small instances): the smallest weight change that overturns GD-Base with at most H edits is 0.240, 0.204, 0.200, 0.196 for H = 0 to 3, and 0.178 from H = 20 on. The per-category rule always errs on the unsafe side: it is suboptimal in 106 of 255 competitor-budget cells and overstates the radius by up to 0.018 (App. G, Table 13).

None of these real cases needs variable elimination itself: the SummEval family is solved by enumeration and the budgeted problem by integer programming. Elimination has run only on the separable Open Images family and on synthetic controls of induced width up to four (Table 11), and the paper presents it for declared structures too large to enumerate. All three analyses are post-hoc; their specifications were hashed before they were run, and the WMT25 line was recomputed after the sign correction.

AuthorsOpenReview ↗

Response to Reviewer BZTB (2/2): combined demonstration and reliability of the radius bounds

Q2a. Discrete search, continuous weights and confidence bounds in one real setting.

Yes: the registered 8-program Open Images grammar combines all three on real data. The submission had this result, but it was hard to find; it now has its own paragraph. Over its programs and all weightings of the 48 categories, the radius curve is B(H) = (0.240, 0.204, 0.204, 0.204), and the joint 95% lower curve, the smallest over three steps of the numerical delta method, is L(H) = (0.160, 0.140, 0.135, 0.135) (Fig. 3, right).

For the 2502^{50} family the submission only had point values. The revision adds the following, from the stored 10,000 paired bootstrap resamples (no model is rerun).

  • Margins. In each draw we take the largest centered error over all 2502^{50} programs. Because GD-Base was itself selected on these data, the maximum also runs over all 30 ordered pairs of models. Its 95% quantile is 2.80 AP, so with asymptotic 95% confidence the margin of GD-Base is at least 11.95 − 2.80 = 9.15 AP for every program and every edit budget simultaneously.
  • Weights. Lowering any category margin can only lower the radius (Lemma 1), so the smallest radius over the family takes one transport computation per competitor and global setting: 0.178, against 0.240 for the official program, with 95% lower bound 0.133. With an edit budget the exact curve comes from the integer program of part 1; since the radius can only decrease with H, 0.133 is a valid lower bound at every budget.

On a desktop CPU the band takes under a second, the 265-target radius inference 29 seconds, and the 1,000 integer programs of part 1 take 19 seconds in total.

Q2b. Does the pairwise coverage check support the bounds on the radius?

No: the audit in App. D concerns pairwise margins only. The submission had one check of the radius bounds, the Gaussian limit experiment (item 1); the revision adds three (App. G): an AP-based simulation, a limit experiment at the observed Open Images margins, and a resampling of the real images. In the first three, the curve at the least conservative step, n−0.35n^{-0.35}, covers in at least 97.7% of trials. On resampled real images it covers in 90.1% and fails its pre-specified rule, while the reported bound covers in all 131 valid trials (item 4).

  1. Gaussian limit experiment (in the submission; now Table 16): five regimes built around ties in the transport step, three sample sizes, 2,000 repetitions, protocol frozen before the simulations. Coverage is 97.7% to 99.4%.
  2. AP-based simulation: the simulator of App. D extended to eight categories. Coverage is 199/200 and 200/200. The bound is conservative, about 40% of the true radius at H = 0, which fails our original two-sided calibration rule on the conservative side.
  3. Limit experiment at the observed Open Images margins (Table 15, 1,000 trials). With step n−0.35n^{-0.35} the curve covers in 99.1% of trials, but the 40 program-competitor targets jointly in only 84%. With either smaller step, and hence with the reported bound, both hold in every trial.
  4. Resampling of the real images (new, App. G). The 9,000 Open Images images are taken as the population; each trial draws 3,000 of them and reruns the whole inference, AP included, on that sample. Of 200 trials, 69 were invalid (a category with no positive image in the sample or in one of its inner bootstrap draws, so AP is undefined) and are reported, not replaced. In the 131 valid trials the lower curve at step n−0.35n^{-0.35} covered the population curve in 118 (90.1%, Wilson 95% interval [0.838, 0.941]). This fails the rule fixed before the run (coverage at least 0.90, upper Wilson limit at least 0.95). The specification tested only that step, not the bound we report (adopted before this result was known), so after seeing this result we replayed the same 200 trials (same seeds; the first run is reproduced exactly; amendment hashed before the replay). The reported bound, the smallest over the three steps, covered in all 131 (Wilson 95% interval [0.972, 1.000]), which passes the same rule. It is conservative: its mean at H = 0 is 0.118, against 0.240 for the population curve.

In short: with search, weights and confidence bounds used together on real data, the reported bound covered in every valid real-image trial, erred on the conservative side, and the whole analysis takes under a minute on a CPU. We hope this resolves the uncertainty about cost and reliability that you raised, and would be grateful if you would take it into account in your assessment.

Thank you for the helpful additions and clarifications. The new examples better demonstrate the value of jointly considering evaluation choices and largely address my concerns. Some uncertainty remains about the calibration of the robustness-radius confidence bounds, but overall I maintain my Weak Accept recommendation.

AuthorsOpenReview ↗

Thank you; on the calibration of the radius bounds

Thank you for the follow-up and for maintaining your recommendation. On the remaining point, the short answer is that the reported radius bound is valid in every check we ran but not calibrated: it errs on the conservative side, so it understates how far the weights can move rather than overstating it. It also does not enter the three verdicts, which are defined from the simultaneous margin intervals (Section 3); the radius bound only measures how far the category weights can move before a certified winner loses.

What the checks show. The reported bound, the smallest of the joint lower curves at the three steps, covered in at least 97.7% of repetitions in the Gaussian limit experiment (Table 16), in at least 199 of 200 trials in each of the two AP-native simulations, in all 1,000 trials at the observed Open Images margins (Table 15), and in all 131 valid trials on resampled real images (App. G). On real images its mean at H=0H=0 is 0.118, against 0.240 for the population radius. Two caveats remain: the theorem is asymptotic and pointwise, so these checks, not the theorem, support the bound at our sample sizes; and the real-image test of the reported bound was specified after a first test of one step had failed.

Where the conservativeness comes from. Both sources were chosen to protect validity. First, the bound covers all 40 program–competitor targets jointly, which is more than the curve needs (Section 3, third remark after Theorem 1). Second, it takes the smallest of three steps, because the least conservative step, n−0.35n^{-0.35}, under-covered on its own: the 40 targets were jointly covered in only 84.0% of trials at the observed margins, and the curve in 90.1% of valid trials on resampled real images.

A tighter bound close to nominal. The intersection–union bound (Section 3; App. F) covers the curve only, as the minimum of per-target one-sided bounds. At step n−0.35n^{-0.35} it covered in 93.9% of the 1,000 trials at the observed margins (Wilson 95% interval [0.922, 0.952]), close to the nominal 95%, and on the real data it gives (0.202, 0.170, 0.170, 0.170), against (0.160, 0.140, 0.135, 0.135) for the reported bound and (0.240, 0.204, 0.204, 0.204) for the point curve. We keep it as a comparator rather than the reported bound for two reasons: it supports no statement about individual program–competitor pairs, and it has not been tested on resampled real images, where the same step under-covered for the joint bound.

For the final version. We will run a confirmatory real-image test with fresh seeds, specified and hashed before the run, of both the reported bound and the intersection–union bound, and report it in App. G whatever the outcome.

Reviewer_jgVSOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about finding the best evaluators or programs (?) for models. But the paper organization was quite bad, mixing all kinds of ideas. The co-NP completeness seems like a general (known) result without any connection to what the experiments and the motivation of the paper is lacking.

Critical strengths

None

Critical weaknesses

I really tried understanding the paper and I could not make out what it does other than robust model ranking (maybe?). It is very convoluted and has a lot of jargon to not much effect. This level of presentation is not acceptable for a conference.

Questions for authors

What is the contribution of this paper? It seems like it is trying all possible evaluators and checking if the winner changes. That level of technique is not something novel.

What do "decision object", "mechanically complete language" and "replayable proof obligation" mean?

1 comment
AuthorsOpenReview ↗

Response to Reviewer jgVS

Thank you for the review. The presentation was the main problem, and we have rewritten the abstract, the introduction and the method section; we would be grateful if you could look at the first two pages of the revision. Our answers follow.

1. What is the contribution beyond trying all evaluators and checking whether the winner changes?

ViewCert does not look for the best evaluator; it tests whether the choice of evaluator changes which model wins. Given the models' saved outputs, a declared list of legitimate evaluators and every weighting of the benchmark's parts (categories, language pairs, datasets), it returns, at 95% confidence, one of three answers: the same winner under all of them (certified), an order reversed under a named evaluator (certified reversal), or not identified. For example, on WMT25, Shy is certified ahead of Gemini-2.5-Pro under the five official metrics, and Gemini-2.5-Pro is certified ahead under the two LLM judges among them, on the same translations (Fig. 1b).

You are right that one starts by trying every evaluator: for a short list, ViewCert is that table plus simultaneous intervals. It goes further in three situations that occur in our data (new Table 1).

(a) A change of winner can be noise. On WMT24 the top system changes with the reference translation, but once sampling error is accounted for neither ordering is supported, and the answer is "not identified". On WMT22, WMT25 and QA the reversal is supported. Checking whether the winner changes cannot tell these cases apart.

(b) Weightings cannot be enumerated. Component weights vary continuously: a grid with step 0.1 over the 48 Open Images categories has 4.3×10104.3\times10^{10} points, and 3 million random weightings find no reversal closer than distance 0.363 from uniform weights. The actual minimum, 0.240, is computed exactly by a transport step that moves weight from the categories with the largest margins to the category with the smallest margin.

(c) The worst case is found on the same data. After a search, an ordinary interval on the worst margin is no longer valid. The simple safe alternative, an interval on each of the candidate's 1,920 raw per-category differences, gives a lower bound of only 0.04 on the Open Images radius; ours gives 0.160, with an asymptotic guarantee that holds after the search (Theorem 1).

2. The coNP-completeness result.

We agree. It is a textbook reduction from 3-SAT, and its only role is to explain why very large lists must come with a declared structure. It is now a one-sentence remark in §3, with the proposition and proof in App. A (Prop. 6), and it is removed from the abstract and the contributions. A single result is now stated as a theorem (validity of the confidence bounds after the search), and we say that it applies the numerical delta method. The search is presented as bucket elimination (Dechter, 1999), not as a new algorithm. What we claim as new is not an algorithm but the three-answer test, its validity after search, and the evidence on eight benchmark settings.

3. Terminology.

The three expressions are removed.

  • "Decision object" was the output: one of the three answers above.
  • "Mechanically complete language" meant that the admissible evaluators are declared as a grid, e.g. support ∈ {official partial labels, closed world} × hierarchy ∈ {expand, leaf only} × matching ∈ {group-of, plain IoU}, i.e. 8 evaluators, and that a script checks that each cell was computed exactly once. Completeness is relative to the declared grid, not to every reasonable evaluator.
  • "Replayable proof obligation" meant that when a ranking can be reversed, the output includes the evaluator and the weights that reverse it, so that anyone can rerun that one evaluation.

Table 2 defines the terms that remain.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what ViewCert finds and what the revision adds

We thank the three reviewers; each review is answered in its own comment. The second general comment reports three corrections to the submission. None changes a headline answer; the largest change is that the certified WMT25 reversals of Shy fall from 98 to 14.

What ViewCert does. A benchmark often supports several legitimate evaluators: labeling rules, references, metrics, weights of its parts. Given the models' saved outputs and a declared list of such evaluators, ViewCert says whether a ranking holds under all of them (certified), reverses under a named evaluator (certified reversal), or cannot be decided from the data (not identified). Its 95% confidence bounds stay valid although the worst case was found by searching the same data.

What it finds (model outputs fixed; only the evaluator changes)

  • Certified reversal, WMT25. Shy is certified ahead of Gemini-2.5-Pro under the five official metrics, and Gemini-2.5-Pro is certified ahead under the two LLM judges among them. All 14 certified reversals of Shy use evaluators containing the GPT-4.1-based judge (8 against Gemini-2.5-Pro, 6 against GPT-4.1 itself). These data cannot tell whether the judge favors its own model family; an evaluator audit should surface such patterns.
  • The winner itself depends on the evaluator, QA (post-hoc). FiD is certified under exact match, Bing Chat under first-alias containment, and nine of the ten pairs reverse with certified signs somewhere in the list.
  • A changed winner is not a reversal, SummEval (post-hoc). Under ROUGE-L, 11 of the 16 systems lead for some subset of the 11 references, and Lead-3 ranks anywhere from 1st to 15th; with 100 articles, none of these reorderings is certified.
  • Certified winner, Open Images (post-hoc family). Grounding DINO Base stays ahead under all 2502^{50} evaluators of a per-category family (simultaneous 95% lower bound 9.15 AP), and remains certified under AP averaged over IoU 0.50 to 0.95.

What the revision adds

  • Presentation (jgVS, BZTB): rewritten around the WMT25 example, with a terminology table, a single theorem, bucket elimination credited to Dechter (1999), and Table 1, which shows what the obvious alternatives give on our data: 3 million random weightings overstate the Open Images radius (0.363 against the exact 0.240).
  • Interacting choices (BZTB Q1): three real cases; in each, treating the choices one at a time misses reversals or overstates robustness (on WMT25 it misses the worst evaluator for 238 of 420 ordered pairs).
  • Search, weights and confidence together (BZTB Q2): bounds for the 2502^{50} family and three new checks of the radius bounds. On resampled real images the reported bound covers the full-data radius in all 131 valid trials, conservatively; the whole computation takes under a minute on a CPU.
  • Other tasks (ey6z Q1): question answering and summarization, with 40 and 29 lines of task-specific scoring code and the shared code unchanged.
  • Boundary of the list (ey6z Q2): admission rules, a table of excluded choices, and two studies rerun with excluded choices added.

Scope. All pre-registered results concern lists of at most 36 evaluators; the larger families and the QA and summarization studies are post-hoc. Variable elimination itself is exercised on the separable Open Images family and on synthetic controls only. Each new analysis has a specification hashed before it was run, with dated amendments in the supplement, except four: two baselines (random search, the comparator table), one diagnostic (the orientation check), and the band over all 30 ordered pairs behind 9.15 AP, which is more conservative than the specified band (9.41 AP).

AuthorsOpenReview ↗

General response (2/2): three corrections to the submission

We found three errors in the submission while preparing this response. Each is corrected in the revised PDF, and the original artifacts stay in the supplement.

  1. WMT25, sign of one metric. The organizers release MetricX-24 already negated (higher is better), and our pre-registered script negated it again, so 12 of the 36 evaluators, including the nominal one, used MetricX with the wrong sign. The error showed up when we laid out all 36 evaluators side by side: under MetricX alone, Qwen2.5-7B, last under the nominal evaluator, ranked first. Our conformance checks did not compare orientations. A later check, released with the code, shows MetricX correlating at −0.95 with the other four metrics before the fix and all five agreeing at 0.95 or more after it. We hashed a correction specification and reran the registered analysis with nothing else changed. Shy remains certified under the five official metrics: its margin over Gemini-2.5-Pro goes from 0.093 to 0.166 [0.141, 0.190], and the reversal under the two LLM judges is unchanged ([−0.098, −0.030]). Certified (evaluator, competitor) reversals of Shy go from 98 to 14 (84 of the 98 used the mis-signed MetricX), and the point weight-change radius goes from 0.581 to "no reweighting reverses Shy". App. E reports both versions. The other studies compute their metrics from text or labels and use no orientation table.

  2. Open Images, step size of the numerical delta method. The submission said that the step n−0.35n^{-0.35} was hashed before the radius inference was applied. Our file times do not support this, and we withdraw the statement. Every Open Images radius bound is now the smallest over three steps (n−0.35n^{-0.35}, n−0.40n^{-0.40}, n−0.45n^{-0.45}): 0.160 instead of 0.187 for the registered grammar, 0.133 for the 2502^{50} family. The smallest of three lower bounds lies below each of them, so this choice can only weaken our claims. It was adopted after the limit experiments and before the real-image test. For the registered grammar, in a limit experiment at the observed margins, this bound held for the curve and for all 40 targets in every one of 1,000 trials; since the rule was chosen after that experiment, this is not an independent check. When the 9,000 real images were resampled (3,000 per trial), the bound covered in all 131 valid trials; the step n−0.35n^{-0.35} alone covered in 118, failing the pre-specified rule, and the replay that tests the reported bound was specified after that first result was seen (BZTB, part 2).

  3. Direct bootstrap. The submission said that a direct bootstrap of the minimum under-covers. Neither that limit experiment (98.3%) nor our near-tie simulations (lowest cell 93.2%, App. F) show this, so we withdraw the claim; we still do not use it, because it is not uniformly valid near ties.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.