acceptodds
Under review as a conference paper at ICLR 2027

A Reference Alone Is Not a Bias Estimator: Identifiability in Logit Correction

Abstract

Blank images and content-free text are often queried to expose a model's class preference, and their logits are then subtracted from ordinary predictions. We ask when such a reference identifies a target-optimal correction. For affine readouts, reference subtraction is exactly geometric recentering, but the induced zero point need not match the optimum selected by a target risk. We prove two complementary non-identifiability results: off-support responses are unconstrained by in-domain behavior, and fixed responses cannot track optima that move with the target distribution. We also give a sufficient condition for identification. Frozen-checkpoint audits in long-tailed semi-supervised learning, including a three-seed replication, show why references can nevertheless help: heterogeneous responses share a training-induced prior-like direction, while their scale and orthogonal content are unreliable. Controlled experiments with two language-model families reproduce the target-risk gap under prior and domain interventions. References are useful inductive probes, but target-optimal estimation requires structure or target information.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Identifiability, Model Calibration, Logit Correction, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YS2qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reference-based logit correction, where predictions are adjusted using the logits produced by a content-free input such as a blank image or neutral text. It shows that reference subtraction geometrically recenters the model’s logits but does not, by itself, identify the correction that minimizes risk on a target distribution. The paper establishes two non-identifiability results, gives a sufficient condition for successful reference correction, and evaluates the analysis in long-tailed vision models and language-model classification. The experiments suggest that references can provide a useful prior-like correction direction, while the appropriate scale and objective require additional target-related information.

Critical strengths
  1. Clear separation of geometry from statistical identification The strongest contribution is the paper’s precise distinction between what reference subtraction mechanically does and what would be required for it to be target optimal. The affine-readout identity makes the geometric action transparent, while the target-risk result identifies the missing distributional argument. This directly addresses a common but insufficient justification for reference-based correction.

  2. Careful empirical decomposition with meaningful controls The vision audit goes beyond showing that some blank or synthetic references work. It separates the shared prior-like direction from reference-specific magnitude and orthogonal content, and supports this account with held-out evaluation, permutation controls, seed replication, and independently trained checkpoints. The language-model prior intervention is also well matched to the theoretical claim: the target optimum changes while the fixed reference does not. These experiments materially strengthen the paper beyond the elementary observation that different references can produce different outputs.

Critical weaknesses
  1. The theoretical results are correct but relatively direct consequences of underspecification and target-distribution dependence. The paper should more clearly explain which claims made by prior reference-correction methods are challenged.
  2. The vision experiments mainly analyze frozen logits. Since reference correction may alter pseudo-labeling and subsequent training dynamics, an end-to-end LTSSL evaluation would strengthen the paper’s practical conclusions.
  3. The empirical scope is limited to a small number of vision checkpoints and binary, single-token language-model classification, leaving broader generalization uncertain.
Questions for authors
  1. After fixing the prior direction, how much additional benefit comes specifically from the reference rather than from a reference-free scale?
  2. Do the conclusions from the frozen-logit analysis still hold when the correction is applied during end-to-end LTSSL training?
  3. Can the authors clarify which claims in prior reference-correction methods are directly challenged by the non-identifiability results?
Reviewer_1fQwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses the relationship between subtracting a reference, the model output given a neutral or context-free input, and the theoretical geometric recentering of a neural network's output. The paper proves some propositions under assumptions about the model's structure and tests them on image classification and language modeling.

Critical strengths
  • The paper studies an interesting and relatively less explored scientific problem.
  • The paper attempts to address both theoretical support and more realistic experiments.
Critical weaknesses

Overall, the paper seems to present an interesting idea, however, it is basically not very readable. The presentation neither explains the idea well nor provides the necessary information. Here are some examples:

  • It's very difficult to understand what problem the paper wants to solve. I can only start guessing after paragraph 2 of the introduction.
  • The paper is highly based on the mentioned LTSSL . However, the paper does not introduce LTSSL's content and concepts in the whole paper. The paper is highly recommended to have at least a background section for LTSSL.
  • Many notations are not defined. I would like to know how the authors will clarify all the notations in the rewritten paragraphs. I only name a few here: (There are more. This is not an extensive list.)
    • L76: RCR^C, what is the C? and what is fθf_\theta? I guess the C is the number of class, and fθf_\theta is the “used model” with learnable parameter set θ\theta
    • Eq1: Never define what is rr and what is fcorrf_{corr}. I guess fcorrf_{corr} is to define the model output after an introduced correction, and r(a)r(a) is an intermediate output after apply PP matrix to the original model output.
    • Eq2: What is 11\mathbf{11}? The paper never defined that matrix. Only claiming that it “removes the softmax-null common-logit direction”. I guess it's an all-one matrix, but I rarely see such notation.
    • The π\pi and ρ\rho are never defined before, but directly said to be class priors and used as πy\pi_y and ρy\rho_y in Eq3. They need formal definition before using them.
    • Eq3 also never defines c(x) and directly just says it’s class independent.
    • The used x and y are also never defined. The x and y can only be guessed throughout the paper or refer to other papers like the often-used CDMAD (Lee & Kim, 2024).
  • Many wordings are not very precise and lead to ambiguity. Such as in L86, what is the train and test the paper wants to mean here? Are they “train and test sets” or “training and testing stages”?
  • Figure 2 with its associated Appendix B is highly uninformative. None of the used models are defined in the paper and appendix. Also, the Appendix B mentions “9000 points in Figure 2”, but there’s not many points in Figure 2.
  • Many aliases were never described but directly used, such as the repetitively used bAcc, which should be at least described as something like balanced accuracy.

Soundness and consistency:

  • Different propositions seem to use different assumptions of classifier types, which are also different from the experiments’ used models, but the connections between the proposition and experiments are yet unclear.
Questions for authors

To clarify:

  • This paper is highly based on prior work in image classification. What is the motivation to discuss the problem in LM, other than trying other modality? An auto-regressive LM has different types of input-output relationship compared to CNN, but the paper hasn't touched on the LM's properties.
  • Not really sure why L153, proposition 1 the g(x)v is called additive output branch here, while it’s not additive in that form?
Reviewer_7NABOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether logits from a supposedly ``content-free reference" identify an additive correction minimizing a specified target risk. "Affine readouts" give an exact recentering identity.

Critical strengths

The work appears to propose an approach to correct the bias when the training and test set differ. There might be a chance that the identification problem is deeper than my impression.

Critical weaknesses

I find it challenging to parse the work and extract any possibly useful information due to the poor writing. There are too many uncommon terms introduced here and there.

The identifiability properties are more or less trivial: the affine recentering identity in Eq. (5) follows directly by substitution. Proposition 1assumes closure under an output perturbation that vanishes on the data support while changing the reference response. Therefore, the claimed non-identifiability then follows immediately from that assumption.

Questions for authors
  1. What is the theoretical novelty of Propositions 1–3 beyond target-dependent risk minimization?
  2. How does reference projection compare with a reference-free correction using the same labeled-prior direction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.