acceptodds
Withdrawn from ICLR 2027

Ghosts Without Target Shift : Exact Admissibility for Stochastic Candidate Reuse

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model evaluation often assigns an expensive judgment only to the response selected from a stochastic candidate batch. Unselected responses can supply control information, but proxy alignment or variance reduction alone does not establish that reuse preserves the selected-response target, respects an externally certified score envelope, and repays its full cost. We introduce Target-Preserving Ghost Qualification (TPGQ), which combines a policy-centered correction, Support Projection, and qualification on independent calibration data. Preselection centering preserves the target, and projection enforces support; independently supplied conditional slot means give variance optimality within the certified feasible class. We derive exact range-sensitive and componentwise cost boundaries under registered planning and ledger assumptions. In a controlled clustered instance, projection lowers variance by and the continuous planning cap by relative to feasible Global Coefficient Shrinkage. An independent-split study obtains an admissible projection-active decision and a disjoint variance ratio of . A 2,304-cell audit identifies constraints missed by simpler rules; two sealed reward-model reconstructions explain headroom rejection. The results establish controlled feasibility and interpretable real-pipeline boundaries.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
language model evaluation, control variates, variance reduction, constrained estimation, prediction-powered inference, cost-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6MmTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a setting where the real annotation f(X) would be expensive while people have a proxy annotation g(X), and people wish to get a data-efficient estimation of the real value. So that the authors proposed to use an unbiased estimation as f(X)+b(−g(X)+E[g(X)])f(X)+b(-g(X)+\mathbb E[g(X)]) to reduce the variance. They further proposed a constrained projection version of this unbiased estimation to control the range. They also derived some criterions on when should people use this estimation. They conducted some experiments on toy cases and real experiments involving LLMs such as Gemini and LLama.

Critical strengths
  • They proposed a constrained optimization framework, which might be of some novelty.
Critical weaknesses
  • The authors claimed that "the preparation of figures were independently completed by the authors", however, the Figure 2 is AI-generated w.h.p.
  • The theoretical part is trivial, as it's a basic idea in statistical inference to control variance, for example, see [1]. The only contribution might be to proposed the constrained projection method, however, the experimental advantage of projection is very marginal, as shown in the LLM experiments.
  • The experiments are mostly based on manually designed toy cases. In the only experimental part involving modern LLMs, the authors admitted that "A favorable variance comparison therefore informs the control signal but does not establish cost-saving qualification." and "Every real-model setting remains unqualified, with no measured accepted four-resource ledger." It can be seen that the improvement in practice is very marginal, or sometimes even worse: " Full-context Skywork gives gains of .67% and .54% on Llama–Gemini and Llama–Claude, though both remain worse than raw.".
  • Their proposed criterion doesn't hold in practice as admitted in the paper, "All combinations return HOLD; no fresh evaluation runs."
  • The paper writing seems AI-native.

[1] Prediction-Powered Inference. https://arxiv.org/abs/2301.09633.

Questions for authors

I don't have questions. I think this paper cannot get accepted.

Reviewer_88JFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to reuse unselected candidate responses (referred to as "ghosts") to improve the statistical efficiency of stochastic LLM evaluation. The authors propose Target-Preserving Ghost Qualification (TPGQ), which constructs policy-centered corrections from cheap proxy scores and projects them onto a constrained support region to preserve the original evaluation target.

Critical strengths

None

Critical weaknesses
  1. The manuscript is excessively concise, often at the expense of clarity and readability. In particular, the Introduction consists of unusually short, fragmented paragraphs, with little explanation or logical development between ideas. Technical concepts and claims are introduced in a highly compressed manner, making the paper difficult to follow

  2. The paper's strongest positive results come primarily from controlled synthetic settings, while none of the six real-world generator–judge combinations passes the proposed qualification gate. Consequently, no fresh independent evaluation is conducted in these settings, and the paper does not demonstrate actual end-to-end cost savings in a realistic LLM evaluation pipeline.

Questions for authors

None

Reviewer_tTaQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TPGQ, a framework that reuses proxy scores from unselected candidate responses to estimate a fixed stochastic policy’s value. It projects policy-centered corrections into a prescribed score range and uses independent calibration to assess variance, workload, and full-cost benefits.

Critical strengths
  1. The paper addresses an important distinction between reducing estimator variance and evaluation cost, considering the score-range constraints and calibration overhead while preserving the policy’s target.
Critical weaknesses
  1. The linear correction coefficients are fitted before nonlinear projection and may be suboptimal for the estimator. Besides, the optimality theorem assumes a conditional-mean oracle and fails to provide guarantees for the multi-stage procedure.
  2. The practical improvements lack independent validation. All variance reductions rely on exploratory analyses using development data, without rigorous model validation or evaluation on fresh data.
  3. The cost savings lack empirical validation. Economic benefits are estimated using a constructed cost ledger and amortized allocation across invariant tasks, without demonstrating actual savings under specified precision requirements in an operational pipeline.
  4. The poor quality of the manuscript significantly hinders comprehension and verification. Several figures exhibit signs suggestive of AI generation or post-processing. Specifically, the strange fonts and layout in Fig. 3 raise concerns that this plot might have been AI-manipulated (If the authors contend otherwise, please provide the anonymous code used to precisely generate this figure for verification). Other figures show similar suspicious visual characteristics that contradict the claims in 'Statement'. I highly recommend that the authors provide a clear explanation.
Questions for authors
  1. Authors should establish a guarantee for the fitted-and-projected estimator, or show that directly optimizing the coefficient after projection improves its variance.
  2. Please provide a real-model setting to demonstrate gains on fresh, disjoint data, and achieve measured cost savings at matched precision.
  3. Please clarify how the figures, particularly Fig. 3, were produced and whether any AI tools were used. Can you provide the anonymous plotting code and underlying numerical data to resolve the visual concerns and verify consistency with the Statement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.