- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2(−1)
Read review
Paper summary
ViewCert formalizes model selection under evaluator ambiguity by applying a declared typed program language to fixed, per-model prediction banks. It provides mechanical completeness checks, exact factored verification, grid-free optimization over aggregation weights, replayable counterexamples, and post-search confidence curves under stated assumptions. Experiments across vision and translation benchmarks show stable winners, certified ranking reversals, and non-identification. Conformance tests, locked resampling, synthetic exactness checks, and an AP-native audit support the implementation; the Open Images program closure is post-outcome, whereas the WMT transfer is prospective.
Critical strengths
The paper formulates a useful and precise decision problem. Separating evaluator ambiguity from sampling uncertainty, and distinguishing a certified reversal from failure to identify a winner, is conceptually valuable for benchmark methodology. The completeness checker, replayable witnesses, and explicit dependence on the declared language make the resulting claims substantially more auditable than a conventional sensitivity table.
The exact simplex and transport argument avoids a weight grid, the structured solver is checked against exhaustive enumeration, and the paper clearly labels conditional guarantees, post-outcome closures, and the human-evaluation extension that was halted at its pre-registered metadata gate. The prospective WMT lock and shared-output design provide a credible demonstration that the method can expose evaluator-dependent conclusions without rerunning models.
Critical weaknesses
The experiments cover object detection and machine translation, but the framework’s applicability to other task types remains unclear. Additional evaluation on tasks such as question answering or summarization, where evaluator ambiguity arises from different correctness criteria or reference choices, would help establish its broader usefulness. In particular, it would be valuable to show whether the evaluator language can be specified and the verification procedure applied without substantial task-specific engineering.
Questions for authors
How much task-specific engineering is required to apply ViewCert beyond object detection and machine translation? Could the authors provide evidence from another task, such as question answering or summarization, to demonstrate that the evaluator language and verification procedure remain applicable?
Since every certificate is conditional on the registered language, its practical strength depends on how that language is chosen. How do the authors justify the boundary of the registered evaluator language and the granularity of its semantic atoms, and which plausible evaluator choices are excluded before certification?
6 comments
Response to Reviewer ey6z (2/2): boundary of the evaluator list
Q2. How is the boundary of the list justified, what is the granularity of atoms, and which choices are excluded?
We fix the boundary with three admission rules (§6.1), list the plausible choices each study leaves out (App. J, Table 19), and test the most consequential ones: under AP averaged over IoU 0.50 to 0.95, Grounding DINO Base remains certified, and growing the SummEval list to 16,380 evaluators changes the certified reversals only from 12 to 14. Choosing the list remains a judgment call; the method makes it explicit and checkable.
- Public provenance. An operator must be used or documented by the benchmark maintainers or by a published protocol, for example the official partial-verification, hierarchy and group-of rules of Open Images with their documented alternatives, or the official metrics, normalizations and aggregation levels of WMT25. The Open Images family goes beyond this rule, and we report it only as a stress test.
- No dependence on model outputs. Static checks ensure that no operator takes predictions or scores as input. They cannot detect an operator chosen after looking at results; the next rule addresses that.
- Fixed before outcomes. For confirmatory claims the list is hashed before the scores are read (the detection replications and the three WMT studies); later extensions are marked post-hoc. For WMT25 the organizers' own rankings were already public, so the lock only precedes our access to the score files, and one metric's sign was corrected afterwards (general response, 2/2).
Granularity. One edit is one documented decision that a maintainer could change on its own, such as whether category hierarchies are expanded. This is a convention of the registry: the same evaluator can cost one edit in one registry and several in another, so edit budgets are comparable only within a registry. What we can guarantee is that encoding the same decision with several grammar nodes changes neither the edit count nor the certificate (Prop. 7, App. A).
Exclusions. Table 19 lists, for each study, plausible choices left outside the declared list, for example IoU thresholds other than 0.5 for Open Images, any metric or LLM judge beyond the five official ones for WMT25, and other reference subsets, unstemmed ROUGE, BERTScore with idf and the other expert dimensions for SummEval.
What happens when excluded choices are added. Adding evaluators can never strengthen a stable-winner certificate, since the set of uniform winners can only shrink (Prop. 8); it can create or remove reversal certificates. We ran it twice, with specifications hashed before any new score was computed.
- SummEval, all four kinds of excluded choices added in three cumulative steps: the list grows from 9 distinct evaluators to 12, 20 and 16,380, the critical value from 3.94 to 4.95, the pairs ordered with a certified sign go 79, 95, 95, 92 (of 120), and certified reversals 12, 12, 12, 14. Selection stays "not identified" (App. I, Table 18).
- Open Images, AP averaged over IoU 0.50 to 0.95 instead of AP at IoU 0.5. This is the choice that could hurt the certificate that matters; it is analyzed as a separate two-program family (official rules, both criteria) with one simultaneous band over its 30 margins. GD-Base remains certified: its macro AP is 49.57 (against 60.40 at IoU 0.5), and its closest competitor, YOLO-World, trails by 16.08 AP (simultaneous 95% interval [14.78, 17.39]); the point weight-change radius is 0.297, against 0.240 at IoU 0.5 (App. J).
We hope the two new studies and these rules answer both of your questions, and would be grateful if you would take them into account in your assessment.
I appreciate the authors running the QA and summarization studies, and the explicit admission rules help clarify how they choose the evaluator list. I will keep my score.
Response to Reviewer ey6z (1/2): QA and summarization
Thank you for the review. We answer the two questions in turn.
Q1. Task-specific engineering, and evidence from another task.
We ran both tasks you suggested, on the shared code unchanged, and QA produces all three answers. What is task-specific is the scoring code (exact match, ROUGE-L, and so on); the completeness check, the simultaneous intervals and the certificates are shared and were not modified. In numbers: 40 lines of scoring code for QA and 29 for summarization, plus one runner script per study (242 and 156 lines) that loads the data, declares the list and calls the shared functions. Both studies use the finite-list part of the method, and QA also the weighting of its two datasets. In both, the list of evaluators was fixed and hashed before we read any score from our own runs (the published findings were known; see the caveat below).
Open-domain QA (EVOUNA; Wang et al., NeurIPS 2023). Answers of five systems (FiD, GPT-3.5, ChatGPT, GPT-4, Bing Chat) to Natural Questions and TriviaQA questions, with human correctness labels. The 16 evaluators cross {exclude, include} flagged questions × {all gold aliases, first alias} × {exact match, token F1, gold containment, human label}. We keep the 5,008 questions with complete answers and labels, and use 95% simultaneous intervals over 160 margins.
Summarization (SummEval; Fabbri et al., TACL 2021). Summaries of 16 systems for 100 articles, with 11 references per article and three expert ratings per summary. The 10 evaluators cross {first reference, all eleven} × {ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, expert relevance}; 9 are distinct, since the expert ratings do not depend on the reference.
A Bonferroni critical value instead of max-t changes none of these answers.
One caveat. That lexical metrics disagree with human judgments on these benchmarks was known before our analysis. We therefore present the two studies as evidence that the method transfers to other tasks with little code, and we make no new claim about the systems themselves. Details are in Table 4 and App. H–I.