acceptodds
Under review as a conference paper at ICLR 2027

DEPLOYRANK: RANKING AMBIGUITY AND CERTIFIED CHOICE UNDER MISSING CONTEXT LINKAGE

Abstract

Benchmarks may publish outcomes and all candidates' predictions for each evaluation record, yet omit the records' links to context available at deployment. We study what these releases identify and what evidence justifies model choice. For any finite number of candidates, we construct one complete-joint prediction release that preserves a strict standalone Blackwell order yet admits every contextual Blackwell order arbitrarily close to a fixed redundancy completion. Each requested order holds in one compatible world simultaneously for all regular strictly proper losses. Ranking ambiguity, however, need not prevent an -optimal choice. For frozen rules and binary context, DeployRank exactly tests zero-link decision adequacy and certifies choices through selectively disclosed row–context links. We derive exact pairwise minimum-disclosure frontiers and multi-candidate bounds, distinguishing holder-known proof size from online discovery cost. In a post-hoc minimax analysis of one protocol-frozen UCI Heart task, the release-only choice incurs nats of aligned regret at . An alignment-informed holder certifies the corrected choice with 86 of 417 links; the precommitted policy uses 278 paid links, counting quota-implied labels as free. Other audits demonstrate zero-link sufficiency and guarantee-only repair.

Primary area:
learning theory
Keywords:
leaderboard nonidentifiability, partial identification, context-assisted evaluation, proper scoring rules, model selection, information acquisition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LPG4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to select the model when the test data are from heterogeneous groups and the group membership is missing. The paper first presents a negative result that missing linkage can result in non-identification (Theorem 1) and then proposes an algorithm to use available and selectively disclosed links to verify that a fixed candidate is within a prescribed tolerance of the best candidate. A numerical study evaluates the proposed method.

Critical strengths
  1. Interesting problem and useful framework. The problem itself is interesting and can have some useful applications. The paper's idea is to move from recovering a complete ranking to certifying an acceptable choice. This distinction is valuable and makes the problem more practical.

  2. Clear theoretical results. The first negative results on ranking ambiguity are clear and motivate the later focus on certifying an acceptable choice. Section 4 on selective linkage provides concrete procedures for certifying an acceptable model choice and clearly quantifying the disclosure required.

Critical weaknesses
  1. The presentation is difficult to follow. Dense notation, terminology, and frequent transitions between ranking ambiguity, fixed-rule certification, minimum proof size, and online discovery make the paper extremely difficult to understand. A consistent example and clearer separation of these concepts would make the paper easy to understand.

  2. The practical usefulness of the proposed method remains unclear. Although certificate verification is efficient, discovering the required links can be costly. Thus, the theoretical certification results do not provide a convincing practical method.

Questions for authors
  1. Can the authors provide a simple illustrative example that explains the setting and the certification procedure without extensive notation? Can the authors also clearly express the logic behind the theoretical contribution?

  2. How are informative links selected? If the missing linkage is due to privacy issues, how is information leakage assessed when additional group memberships are revealed? If the motivation is the cost of linkage, can the authors provide some examples to motivate the problem?

  3. How practical is the method in a realistic application, considering the cost of acquiring and authenticating links and the computational burden of link selection? Could the numerical study more clearly evaluate different tolerances, comparing the proposed method against a fully informed oracle and simple baselines?

Reviewer_LoqZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper investigates whether benchmark releases that publish model predictions and output-context statistics but withhold row-level links between prediction and context are sufficient to rank models and justify deployment decisions. Paper shows that same released statistics can be consistent with different contextual model rankings. It then develops criteria for determining when deployment decisions can be guaranteed without additional data and methods for calculating the minimum number of row-level links needed to strengthen those guarantees. Experiments across datasets show that some releases already support reliable decisions without additional links while other require limited disclosure to strengthen guarantees.

Critical strengths
  1. Paper demonstrates that ambiguity in model rankings does not necessarily prevent reliable deployment decisions. Theorem 3 provides a zero-link decision test and Theorem 4 gives an exact pairwise disclosure frontier.
  2. Paper carefully distinguishes between different guarantee scopes and between proof size, online discovery cost and reports failed evaluation gates.
Critical weaknesses
  1. Paper relies on specific information release and trust model whose practical relevance is not demonstrated through a real deployment.
  2. Novelty of the paper is limited, several results are build on established ideas. Focusing on Theorem 3-4 and few well motivated case studies could make the contribution clearer.
Questions for authors
  1. The paper motivates link disclosure by replayability: aggregate aligned risks would settle selection but cannot be verified by evaluator but disclosed links can be checked against public losses. However the paper also states that link authencity is external trust asumption so verification still depends on trusting the holder. It is unclear why authenticated links are more credible or easier to share than authenticated aggreate risks -- individual linker are more privacy sensitive. Please explain the asymmetry with real setting.
  2. Was τ=0.005\tau=0.005 based on domain grounds before any heart analysis? How sensitive is the "material repair" conclusion to calibration choices?
Reviewer_s3DXOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of model selection when only aggregate performance information is available, and the linkage between deployment-relevant context is missing, making context-dependent performance rankings ambiguous. In particular, from my understanding, they investigate the question of whether a reasonable selection can be made without revealing any link, and if not, what minimal set of links can be revealed. I understand the contributions to be:

  • proving that a ranking without context-link can simultaneously be compatible with every possible contextual ranking (Blackwell order)
  • showing that we can, in some cases, identify an ϵ\epsilon-optimal model without revealing any context links, and derive an exact number of links to be revealed in the other cases.
  • empirically investigating those theoretical findings on decision audits with the UCI Heart task dataset.

My background does not allow me to understand the terminology used in the paper, and thus the problem and contributions in their details. I flagged this to the AC shortly after receiving the assignment. In my opinion, this is further exacerbated by the paper's extremely jargon-heavy writing style, which makes it very difficult for non-experts to read.

Critical strengths

None. However, this assessment should be considered in light of the mismatch between my expertise and the topic of this paper

Critical weaknesses

It seems to me that this paper is written in a way that does not allow it to be read by non-experts. Even the introduction fails to provide a clear and understandable definition of the problem in my opinion, and include undefined mathematical notation starting from the second paragraph (Y,Z,C, I, Z_m). I believe that the paper's lack of clarity raises significant concerns regarding its suitability for publication at ICLR. However, this assessment should be considered in light of the mismatch between my expertise and the topic of this paper.

Questions for authors

In my opinion, the paper needs major improvements in readability, including less jargon-heavy language and proper definitions for the mathematical notation used. As this requires substantial rewriting, it seems to me that it can hardly be addressed during the rebuttal phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.