acceptodds
Under review as a conference paper at ICLR 2027

Silence Is Not Absence: A Probe Null Requires More Than Format Competence

Abstract

A probe that asks what a language model memorised reports a null when the model answers at chance, and that null is evidence of absence only if the probe could have shown the fact. Two separate conditions have to hold, and checking the first does not establish the second. The first is that the model can use the readout. Asked thirty questions with no domain content, of the form “which one is a fruit, an apple or a hammer”, each in both option orders so that a fixed letter preference scores exactly one half, seven of nineteen open-weight models cannot do it. The failures are not the small models: every rung of the Pythia ladder fails from 0.16B to 6.9B, and Llama-2-7B fails while Llama-3.2-1B passes at a sixth of its size; inside one laboratory OLMo-1-7B scores 0.517 and OLMo-3-7B scores 1.000. Two of the seven answer a yes/no frame at 0.983, consistent with a label-binding failure rather than an inability to answer, and the excluded models cannot be read even when the answer is printed in the prompt, at 0.507 and 0.535. The second is that information of the kind being probed can reach that readout, which no control built from questions a model already knows can establish. We test it by writing invented dated facts into training text at four exposure counts, in three models from three laboratories. Under the form in which they were learned every model separates the true value from the probe's own foil, increasingly with exposure and against flat untrained controls, while under five forced choices — relabelled, list-free, and with demonstrations — the same facts remain at chance in every model. Interface validity does not imply construct validity. Turning the instrument on the question that motivated it, whether small open-weight models memorise market history closely enough to contaminate a backtest, the probe that scores about 0.90 on which company a ticker belongs to scores about 0.50 on every date-specific fact about that company, and we exclude effects larger than 1.75 points in the direction memorisation predicts. By the second condition, that bounds what a forced choice retrieves, not what the models hold. We release the panel, the probes, the estimators and the power calculator.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evaluation methodology, probe validity, multiple-choice symbol binding, format sensitivity, memorization, training data cutoff, language models, regression discontinuity, finance

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_66nMOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The quality of the paper is very poor that I cannot understand what the paper is doing.

Critical strengths

none

Critical weaknesses

The writing for the paper is confusing.

Questions for authors
Reviewer_ywG9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper describes conditions for accepting a null result when probing whether models have memorized facts or not. First, we must test if the model can handle the structure of a question and respond in the expected format using basic facts. A variety of weaker models cannot. Second, we must identify whether financial facts from training can even be surfaced by forced choice questions. To explore this, small adapters are trained over a small dataset of synthetic financial facts. Models are shown to memorize those facts, but memorization does not improve their ability to use the facts when answering forced choice questions. Together, the paper argues that probes of financial fact memorization are only valid when a model performs sufficiently well in these test conditions.

Critical strengths

The goal of this paper is to establish when we can accept a null probe result as evidence that the model has failed to memorize some fact. The paper identifies confounds to drawing this conclusion and methods for excluding these confounding factors. It may inspire future work to improve probing strategies.

Critical weaknesses

The synthetic data training task is not sufficient to conclude that real models cannot generalize knowledge from prose to a QA task. At most, what is shown is that the models cannot use the parameters of a small adapter trained on 340 documents to generalize prose facts to QA. This may be due to the fact that embeddings and linear layers are untuned, or to the damage to the models’ general reasoning capabilities effected by the presence of the adapter, or by overfitting the adapter-modified synthetic company representations to the prose domain.

The writing of this paper is extremely confusing. Please explain what you are doing and why in terms that a larger segment of the ICLR audience could understand. Define what you mean by “readout” and “backtest”.

Questions for authors

The paper finds that weak (older or smaller base) models cannot be effectively probed using forced choice QA formats, which is known. With this finding, the possibility remains that such models have memorized facts but lack the ability to generally surface those facts. Can this be empirically verified using the true-foil log-prob experiments?

Reviewer_MaFuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks when a language model’s chance-level answer on a probe can be read as “the model doesn’t know this.” It argues two conditions must hold. First, the model must be able to use the answer format. Using simple multiple-choice questions, the authors find that 7 of 19 open-weight models cannot, and this is not explained by model size. Second, the probe must be able to reach the kind of fact being asked about. The authors train invented dated stock prices into three models and find that a fill-in-the-sentence readout recovers them while multiple-choice probes stay at chance. Finally, they apply the method to whether small models memorise stock market history. The models know which company a ticker belongs to (~90%) but are at chance on all date-specific questions.

Critical strengths
  • Careful and transparent design. Every question is asked in both option orders, so chance is exactly 50% and position bias cannot create a false signal.

  • A useful practical message. The paper shows that a chance-level probe result can come from the model failing the answer format rather than lacking the knowledge. It also shows that the standard design in the finance literature lacks the power to detect effects of the size at issue.

Critical weaknesses

W1. The construct-validity experiment may measure familiarity, not dated recall. The declarative margin in Section 5.3 may only show that the model has seen a price alongside the company, not that it knows which date the price belongs to. The paper’s own diagnostic on never-written facts points this way. If so, the readout gap concerns familiarity and does not transfer to the date-specific financial probes. The authors should report the margin on pairs where both prices were written. They should also confirm that the forced-choice readout still works on the synthetic companies after training.

W2. The financial application cannot answer its motivating question. Either Section 5.3 holds, in which case a forced-choice null says little about what the models hold, as the authors concede. Or, per W1, the readout’s validity for dated facts is untested. Either way, the Section 6 null does not tell us whether the models memorise market data. Applying the readout that recovers facts in Section 5.3 to real market data would be the natural fix.

W3. Limited novelty. Prior work on multiple-choice evaluation and knowledge probing has already established two things: base models often fail multiple-choice formats, and forced-choice questions can underreport what a model encodes. The contribution is therefore incremental. The claim that format competence is not monotone in model size is also unsupported, since the compared models differ in training data, not just size.

W4. Presentation. The paper is hard to follow for an ICLR audience. Key results are in the appendix, many numbers are reported in long prose passages instead of tables, and econometric terms are used in the main text without definition.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.