acceptodds
Under review as a conference paper at ICLR 2027

Correct Scores, Unsupported Conclusions: Benchmarking Evaluation under Incomplete Semantic Reference

Abstract

An evaluation score can be computed exactly against the available reference and still fail to support the scientific conclusion drawn from it. We formulate model comparison under incomplete semantic reference as an evidence-qualified evaluation problem: when observational proxies, current snapshots, or incomplete records substitute for the required semantic reference, what evidence is sufficient to certify which model is better? Holding predictions fixed across structured domains reveals that proxy distortion alone does not determine conclusion validity: selected-only reference drops chess forest set from to while leaving the model winner unchanged, yet the identical substitution reverses model selection in Planning Domain Definition Language (PDDL) tasks. This divergence establishes that reference fidelity and conclusion fidelity are distinct: exact reference identification is sufficient but not necessary if every compatible reference induces the same model comparison. We introduce a typed benchmark across public and executable sources spanning stable, reversed, recoverable, certifiable, and uncertifiable regimes, and operationalise an executable certification protocol that determines whether available evidence already certifies the comparison, target-matched recovery suffices, additional evidence must be acquired, or the comparison must remain \Unknown. In process panels, target-matched recovery resolves ranking conclusions despite imperfect reference reconstruction; in controlled and natural-history panels, exact query certificates determine whether evidence certifies the query, requires witnesses or exhaustion, or yields \Unknown. Holding outputs fixed, stronger references can preserve rankings, alter pairwise comparisons, or reverse top-system selection, with the latter observed in both formal planning and a contemporary speech-recognition panel evaluated against independently corrected transcripts. Evaluators should therefore determine whether available evidence is sufficient to certify the declared model comparison, rather than forcing an ungrounded point reference and mistaking predictive confidence for semantic warrant.

Primary area:
datasets and benchmarks
Keywords:
model evaluation, benchmark validity, semantic reference, incomplete information, model selection, evaluation reliability, process mining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4ZEsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when incomplete evaluation references still support a reliable model comparison. It introduces a benchmark and a protocol that considers all references compatible with the available evidence, certifies comparisons when they agree, and otherwise seeks more evidence or returns UNKNOWN. Experiments across structured tasks, code evaluation, and speech recognition show that changing the reference can preserve rankings, change pairwise comparisons, or reverse the winner.

Critical strengths
  1. The paper distinguishes estimated answers from evidence-supported conclusions. I think this is a meaningful research perspective in ML/DL research. Besides, allowing UNKNOWN and reporting coverage alongside certified error provides a useful discipline for evaluation under incomplete information.

  2. The fixed-output design isolates the effect of the evaluation reference. This makes the ranking changes easier to interpret. The speech-recognition winner reversal is particularly consequential because it shows that reference quality can affect practical model selection.

Critical weaknesses
  1. The contribution beyond established ideas is not sufficiently clear. The benchmark and synthesis could still be valuable, but many experiments illustrate expected consequences of these principles rather than establish substantially new insights.
  2. The strongest practical examples and the certification results remain somewhat disconnected. The modern ML panels show what happens after stronger references become available. The certification demonstrations mainly use controlled or explicitly structured settings. This leaves unclear how much the proposed protocol helps with realistic model selection when the reference is still incomplete.
Questions for authors
  • What is the main contribution beyond applying existing certain-answer and decision-region ideas to evaluation? Which reported finding or capability best establishes that additional contribution?
Reviewer_pA9COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies model evaluation under incomplete semantic references, showing that accurately computed scores may still fail to support reliable model comparisons. It distinguishes reference fidelity from conclusion fidelity: substantial reference errors may preserve model rankings, while smaller discrepancies may reverse them. The authors introduce a benchmark spanning structured domains and contemporary machine learning evaluations, together with an evidence-qualified protocol that considers all references compatible with the available evidence. A comparison is certified when these references yield the same conclusion; otherwise, the protocol calls for reference recovery, additional evidence acquisition, or an UNKNOWN outcome. Experiments illustrate stable rankings, pairwise ranking changes, and top-system reversals, highlighting the importance of assessing whether available evidence supports the intended evaluation claim.

Critical strengths
  1. The experiments cover diverse evaluation scenarios. The paper demonstrates stable rankings, changes in pairwise comparisons, and reversals in the top-performing system across structured tasks, code evaluation, and speech recognition, providing a broad account of how reference changes affect evaluation outcomes.

  2. The evaluation problem is relatively novel. The paper examines the design of reference answers from a broader perspective, offering a fresh approach to thinking about evaluation.

Critical weaknesses
  1. The evaluation metrics appear to be designed to produce a predetermined outcome. If you evaluate using F1 and then change the reference answers, the scores will naturally decrease.

  2. There is a gap between the core certification protocol and the experiments on real-world models. The experiments primarily compare scores and rankings before and after changing the reference, demonstrating that reference changes can affect model comparisons. However, they do not sufficiently validate the proposed certification protocol: how to construct the compatible reference set when the complete reference is unavailable, determine whether existing evidence is sufficient, and obtain certification by acquiring additional evidence. Thus, the experiments provide substantial evidence for the problem but insufficient validation of the proposed solution.

  3. The key assumptions underlying certification lack validation in realistic settings. The method relies on a compatible reference set that covers all possible references, but the paper does not sufficiently explain how to construct and validate this set in open-ended, real-world tasks. Omitting possible references can undermine the reliability of certification, whereas an overly broad set may lead to frequent UNKNOWN outcomes.

  4. The figures and tables are clearly AI-generated, yet this is neither disclosed nor followed by any refinement. The writing also contains excessive parenthetical explanations, unnecessarily long decimal values, and unusual italicization commonly associated with AI-generated text, resulting in very poor readability.

  5. The paper does not demonstrate the practical value of this work. In what scenarios would reference data need to be constructed this way? The paper does not test whether training models on such data would have adverse effects or whether the models could still learn the correct answers. Evaluation and training should be aligned: models should be evaluated according to how they are trained. Otherwise, this evaluation design has no practical value and offers no contribution to the community.

Questions for authors

Why there is no citation in Introduction?

Reviewer_V4BeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether model comparison conclusions still hold when evaluation references are incomplete. The authors distinguish between the questions of "whether the references are accurate" and "whether the comparison conclusions are reliable." The comparison can be validated as long as all references compatible with existing evidence give the same conclusion. Experiments covering structured tasks, code, and speech recognition demonstrate that changing references can maintain, change, or even reverse model rankings.

Critical strengths

(1) It addresses an important evaluation problem: correct score calculation does not necessarily mean reliable conclusions, providing a useful analytical perspective.

(2) The design of fixing model outputs and only changing evaluation references is more convincing; it reports both stable rankings and shows winner reversals in PDDL and speech recognition.

Critical weaknesses

(1) The conclusion of "restoring ranking" is too strong. Table 12 shows that Hospital and Road still have strict ranking discrepancies of 7/75 and 4/75, respectively. Although the differences are small, Appendix O explicitly uses strict ranking as a presupposed indicator, and the tolerance for smaller differences cannot replace this indicator, making the related conclusions less convincing.

(2) The demonstration of the actual authentication process is still insufficient. Modern ML experiments primarily involve re-scoring after a reference change, and haven't fully addressed how to construct a compatible reference set, obtain additional evidence, and complete authentication when the complete reference is unknown.

Questions for authors

(1) Which recovery results satisfy strict order consistency, and which only hold true when small differences are allowed?

(2) Can you demonstrate authentication examples when the complete reference is unknown, and report the evidence gathering and computational costs?

(3) When the score difference range includes 0, how do you distinguish between a tie and a ranking reversal?

1 comment

Thank you for your response. The rebuttal partially addresses my concerns. I am maintaining my original rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.