acceptodds
Under review as a conference paper at ICLR 2027

Right Number, Wrong Question: LLM Probability Substitution in Incomplete Searches

Abstract

Suppose an agent fails to find the target in a partial search and must report the chance the target still exists in the unsearched share. How do language models respond to a required probability field when the prompt does not contain the base rate for the search target? In this setting, when the prompt states the search's coverage as the share of records each searched location holds, we find that 22 of 46 substitute a different probability event for the requested posterior. Supplying the base rate removes the substitution in most models, and asking the model for its assumed prior removes the substitution in half of the models scored. Even when the prompt asks for the prior and for the event the number describes, relabeling the probability field as “the number you report” brings the substitution back. The substitution is a default rather than a misunderstanding. When asked in separate inference calls for the probability that the target exists and for the probability that the search missed it, 18 of 30 models answer the same number to both questions. This two-call test detects 19 of 20 substituting models and none of the seven that report the posterior, so a pipeline can screen a model before trusting its probabilities. The substitution persists even when some models run the search themselves with tools. A calibration score cannot tell which event this default probability refers to, and no recalibration of the reported probability can correct to the posterior at every base rate. We simulate a thresholding pipeline on two datasets and show how thresholding on these LLM-reported probabilities leads to silent failures in downstream applications at low base rates. As self-reported model probabilities are commonly used to make decisions and orchestrate agent pipelines, careful evaluations should test what events those probabilities are actually estimating.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, abstention, uncertainty quantification, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m5kSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines how LLMs estimate the probability that a target exists after a partial search finds nothing. Some models report the unsearched fraction as this probability, whereas the correct answer requires applying Bayes’ rule with the prior probability that the target exists.

The experiments show this behavior does not stem simply from an inability to compute the posterior: providing the prior or changing the output format alters the models’ answers. When the prompt lacks sufficient information but still demands a probability, some models substitute a probability they can compute rather than stating that the requested one is undetermined.

Critical strengths
  1. The authors identify a useful distinction between reporting an inaccurate probability and reporting the probability of a different event, which matters when a system must make high-stakes decisions.
  2. The experiments are comprehensive, covering varied search coverage, priors, and output formats.
Critical weaknesses
  1. The motivation and presentation are difficult to follow and do not accurately reflect the work.
  2. The assumption that the percentage of the searched set is known is unrealistic.
  3. The paper is hard to read because notation and key terms are used before they are defined, which makes it difficult for first-time readers to follow.
  4. The original question does not specify a prior. A prior of 0.5 may serve as a useful reference, but the paper should state its prior assumption explicitly.
  5. The paper reports different behavior when coverage is given as a running percentage, which makes it unclear how broadly the findings generalize.
Questions for authors
  1. In real-world systems, how would the exact fraction of records already searched be known?
  2. If all records or logs are available, how does this setting differ from standard retrieval? Could the target not be located directly by indexing the records in a vector database and querying it?
  3. Since the prior is not given, how should an answer that relies on an explicitly stated prior be evaluated?
  4. Why does presenting coverage as a running percentage produce different behavior from presenting location shares?
5 comments

We thank the reviewer, whose summary states our claim accurately. Beyond Bayes' rule, the paper shows which wrong number appears and that calibration does not identify the event behind that number, and gives a check without labels that detects the substitution (general comment). A substituting model is defined in the general comment.

W2 and Q1. Is a known fraction searched realistic, and how would it be known? A pipeline has to measure the coverage. The paper supplies the coverage and does not test how a pipeline would estimate it, and Sec. 5 now says so. The coverage is the probability that the search finds a record that exists, which equals the share of records searched only when the target is equally likely to be any record and a search of a location finds every record in it. In the clinical records we measured it on the patients known to have the record, as the fraction for whom the searched tables contain it: 0.75 to 0.86 (App. B.2). The benchmark states δ\delta because the test requires it. If the model estimated the coverage itself, the miss likelihood at its estimate and the posterior at some other coverage could give the same number (Sec. 2).

Q2. How does this differ from indexing everything? We agree that an exhaustive, exact lookup over a complete store resolves the question: a failed lookup then means the record is absent, and no probability is needed. The paper covers searches that stay partial or approximate. Search backends flag such searches in their responses. Elasticsearch by default returns partial results when shards fail or time out, GitHub's search API sets incomplete_results when a query times out, and Google Drive's sets incompleteSearch when not all documents were searched. These flags say a search was partial and do not give its coverage, which the pipeline must still measure. In our agentic run, models search an enterprise collection with tools under a budget, and 8 of the 21 substituting models (38%) report 1−δ1-\delta when the section shares are stated (App. B.4). A prompt that states the coverage as a retrieval recall, drops the candidate list and states a base rate gets the posterior from 16 of 19 substituting models (App. D.4). The coverage also matters for decisions. At base rates where the coverage moves the posterior across the threshold, computing the posterior from each search's coverage makes up to 9.7 fewer wrong decisions per 100 failed searches on the enterprise data than the better of creating every record or no record (App. H.2).

W5 and Q4. Why does a running percentage behave differently? On the benchmark a running percentage does not behave differently. Under the total after each scan, the 21 substituting models give 1−δ1-\delta in 91% of 1,344 responses, 92% under the share table (Fig. 5(a), App. D.4). The run the reviewer cites differs in its search: its agents stopped at a median coverage of 13%, mostly because the budget ran out. Only 1 of its 176 responses (11 models, 16 episodes each) was at 1−δ1-\delta. When a live search states the total after each scan (Table 6), GPT-5.5 gives 1−δ1-\delta in all 128 episodes, Haiku 4.5 in most and Grok 4.20 Reasoning in 3 of 128 (App. B.5). The two runs also differ in budget and model set, so which of these differences removes the substitution is open.

W4 and Q3. The prior, and how a stated prior is scored. We score a stated prior against the posterior at that prior, within the same tolerance. Since the question does not state a base rate, 0.5 is only a reference prior, and an "undetermined" response or a stated dependence on the prior also counts as correct (Sec. 2). Given a required prior field, 6 of 11 models give the posterior at the prior they state (App. D.2). Urged to state a prior, two models that keep 1−δ1-\delta, GPT-5.5 and GPT-5.4, give in 59 of 64 episodes a prior that contradicts their own number (App. D.7).

W1 and W3. Motivation, notation and terms. The introduction opens on one concrete scene, defines the coverage where it first appears, and Sec. 3.1 states the claim with its condition: given a coverage stated as location shares and no base rate, 58% of the 60 models fill the probability field with 1−δ1-\delta (47% with the episodes that withhold the coverage pooled in). Sec. 2 defines rule and the kinds of prior in one place, and Fig. 1 shows example responses without rates.

If this answers how the coverage would be known (Q1) and why indexing leaves the question open (Q2), we hope the reviewer will reconsider the score. If a point still stands, please tell us which, and we will answer it.

Thank you for the clarifications.

Addressed: W3 through the described notation revisions; W4/Q3 through prior-dependent scoring and acceptance of “undetermined” answers; and Q2 through the distinction between partial and exhaustive retrieval.

Remaining concerns:

W1: The connection between the controlled finding and the broader practical motivation remains insufficiently clear.

W2/Q1: The clinical example is helpful, but reliable coverage estimation for new searches and the effect of uncertainty in those estimates remain insufficiently addressed.

W5/Q4: I accept that the two coverage formats produce similar benchmark results. However, the unexplained differences across live-agent experiments leave the extent of generalization unclear.

I therefore maintain my rating of weak rejection, based on these remaining concerns.

We thank the reviewer for the follow-up and for listing their remaining concerns. Below we add new results and clarify the scope of our earlier findings.

W5/Q4: "... the unexplained differences across live-agent experiments leave the extent of generalization unclear." Our earlier reply left open why one live run shows almost no substitution, and we ran new live searches to address this. In the run the reviewer refers to, the coverage is given as a fraction of all documents. When we instead list the locations with their location shares and state the coverage as a fraction of those locations, 6 of 11 models give 1−δ1-\delta on the same search (App. B.5). This comparison changes two things at once, so it cannot tell us whether the added information or the wording is the cause. A further live search lists the locations without their location shares and gives only the cumulative coverage after each search. In this setting, 5 of the 13 scored models match 1−δ1-\delta in at least 90% of their 32 episodes (App. B.5). We therefore narrowed the claim to coverage stated as a share of the possible locations.

W2/Q1: "... reliable coverage estimation for new searches and the effect of uncertainty in those estimates remain insufficiently addressed." We agree that estimation matters, and we now test estimated inputs. In a split-sample analysis, we estimate coverage and base rate jointly on one half of the episodes and score decisions on the other half (App. H.2). Up to a base rate of 0.40, the posterior from the estimated inputs makes as many errors as with the true inputs. At a base rate of 0.60 on the emergency-department data, wrong decisions rise from 37 to ~44 per 100 failed searches, slightly worse than the model’s own number (~40). Even when we tell four models that the coverage is estimated and may be off by up to 0.10, they still report 1−δ1-\delta in almost every response (App. D.4). We agree that these results measure sensitivity to estimated inputs on our datasets, and do not validate an estimator for deployment or for new datasets, which we leave to future work (Sec. 5).

W1: "The connection between the controlled finding and the broader practical motivation remains insufficiently clear." We agree that the link should be more explicit. We ran the deployed classifier of Google’s Agent Development Kit on our inputs. 6 of the 8 scored models meet the substitution criterion, including one that computes the posterior on our benchmark (Table 8). This shows that the failure can pass through existing interfaces that require a probability, although we have not measured how often it occurs in practice.

We hope the answers above address the three remaining comments. If so, we again thank the reviewer for the detailed feedback and would ask them to kindly re-evaluate our work in light of the narrowed claim.

Reviewer_rDM8OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks the question of what LLMs reports when you prompt them for the probability that a target exists after a partial search, giving the coverage but not the base rate. The benchmark is synthetic, so ground-truth coverage is known. In their 22 of 46 models return the unsearched fraction rather than the posterior, which is a different quantity entirely. The authors figure out that which rule each model follows by varying coverage and watching how the answer moves. They also propose a two-call diagnostic that eventaully catches substituting models, and they simulate a thresholding pipeline where the substitution silently drops records at low base rates.

Critical strengths

The identification trick is the best part in my reading. Coverage is set and varied while everything else is held fixed, so the slope of a model’s answer directly reveals which quantity it’s computing, no calibration assumptions needed. The coverage is good with 46 models, many prompt variants, external datasets, tool.

Critical weaknesses

The effect looks tied to the prompt format. Models get a table of location shares summing to one, basically a probability distribution over where the record would be, and many just report one minus the searched portion.

The question is also kind of deliberately underdetermined. The authors themselves say you shouldn’t ask for a posterior without a base rate. Even the posterior at a flat 0.5 prior misses roughly a fifth of records when the true base rate is 0.01. So the takeaway is “give the base rate,” which people already know, not very suprising.

Questions for authors
  • How much does the share-table format drive the effect? Does substitution still show up when coverage is described more naturally, say as a plain list of locations checked?
  • Can you report the headline numbers by family or provider rather than per checkpoint?
3 comments

We thank the reviewer for two concrete questions, and new experiments answer both. The effect does not need the share table, and substituting models come from 12 of the 14 providers. We also explain what the paper adds to the advice to supply a base rate. A substituting model is defined in the general comment.

Q1 and W1. How much the share table drives the effect, and whether it shows up with a plain list of locations checked. The substitution occurs without the table, and the wording of the coverage still matters. On the same 64 episodes, we gave the 21 substituting models the coverage in other forms (Sec. 3, Fig. 5(a), App. D.4):

Coverage given asResponses at 1−δ1-\delta
share table (submitted)92% of 1,344
total after each scan, no table91% of 1,344
one single-probability sentence52% of 1,344

Under the plain list of equally likely locations the reviewer asks about, 11 of 18 scored models keep 1−δ1-\delta, and the other 7 give 1−δ1-\delta again once the shares are written out. Most also keep 1−δ1-\delta with the shares written as sentences (21 of 22 scored) or the coverage in words (18 of 20, scored against the coverage stated in words). No model keeps 1−δ1-\delta when a retrieval tool's recall appears in the search log beside the list of candidate locations, and we report only this one recall wording. These models were chosen because they substitute under the pooled label, so the counts do not estimate prevalence over all 60. No reference-posterior model keeps 1−δ1-\delta in any coverage form of App. D.4). Sec. 3.1 and the abstract now state this condition.

Q2. Headline numbers by family or provider. Of the 60 models, 35 (58%) substitute on the episodes that state the coverage, and 28 (47%) under the submitted version's criterion, which pools in the episodes that withhold the coverage (App. A.6). With each family weighted equally, the share of substituting models is 57%, and with each provider weighted equally it is 59%, against 58% over all 60. At least one model substitutes in 14 of the 17 families and 12 of the 14 providers. Of the 35, 17 are OpenAI's and 18 are not (Table 3, App. A.6, new).

W2. The question is underdetermined, and the advice to give the base rate is known. We agree the advice is standard, and it works for over half of the substituting models: given the base rate, 59% of the 29 asked give the posterior at that rate in at least 60% of their responses, and 41% do not (App. D.1). We made the question underdetermined on purpose. Without a base rate, Sec. 2 counts an "undetermined" response, a stated dependence on the prior, or the posterior at a declared prior as correct.

Empirically, the paper measures what models put in a required number field when the question is underdetermined. They return a different probability, 1−δ1-\delta, while saying the probability is undetermined. Asked whether the information given determines the probability, all 19 substituting models asked say no, and 11 of them choose "undetermined" when it is allowed yet give 1−δ1-\delta when a number is required (App. C.5). Calibration does not identify the event behind this number, because at each coverage δ\delta, 1−δ1-\delta equals the posterior at the base rate 1/(1+δ)1/(1+\delta) (Fig. 3). A two-question check detects this substitution without knowing whether the record exists. The check flags 24 of the 26 substituting models scored at coverages 0.12 and 0.78 and none of the 7 reference-posterior models, and comparing the existence answers alone with the stated 1−δ1-\delta gives the same flags (App. C.1). The substitution also causes wrong decisions. As the reviewer notes, the posterior at 0.5 also misses records at a base rate of 0.01. However, when a pipeline thresholds the model's number at that base rate, the substituting models are wrong on about 42 more of every 100 failed searches than the reference-posterior models on the enterprise data, and about 9 more on the emergency-department data (App. H.3). At that data's own base rate of 0.89, the reference-posterior models cost more than 4 of the 5 substituting models (Table 1).

If Fig. 5(a) and Table 3 answer Q1 and Q2, we hope the reviewer will reconsider the score. If a concern remains, please tell us which, and we will answer it.

We thank the reviewer again for their two concrete questions.

"How much does the share-table format drive the effect?" Since our earlier comment, we ran a new live search that lists locations without their location shares and only shows the model cumulative coverage so far. Because locations vary between episode, the coverage takes 24 to 28 different values for each model, which allows us to test whether the responses track 1−δ1-\delta. Five of 13 models match 1−δ1-\delta in at least 90% of their 32 episodes (App B.5). Therefore, a share table is not needed. We also agree with the reviewer that the effect depends on how coverage is stated and we narrow the claim accordingly (1−δ1-\delta is uncommon when coverage is given as a fraction of all documents as mentioned in Section 5).

Thanks the authors for the response. I will choose to remain my score.

Reviewer_EMNsOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks 46 LLMs for P(target exists | failed partial search) when the coverage δ is stated but the base rate π is not. It finds that 22 report the miss likelihood 1−δ (“substitution”), 7 report the posterior at π = 0.5, and 17 are mixed. It also varies the output schema, proposes a two-call check, tests transfer, and simulates a thresholding pipeline.

Critical strengths

The identification design is clean: coverage is known by construction and the rule is read off the δ-curve. Model coverage is broad, and the appendix is candid.

Critical weaknesses
  1. The normative frame is not justified. The question is underdetermined by design, so the 0.5-prior posterior is also just a default, not the correct answer. It is worse calibrated than 1−δ at π = 0.9 (ECE 0.47 vs 0.26, Table 20). Read as a likelihood, 1−δ is the more useful report, because Prop. 1's h(q, π) recovers the exact posterior from it. The real problem is a mislabeled field, and the paper's own remedy never uses the model's number.

  2. "Default, not misunderstanding" is unsupported. The event-field form rewords the field to "the number you report" and lets the model choose the quantity (Table 17), so reporting 1−δ there is compliance, not substitution. Yet this form underlies both the abstract's "relabeling" claim and the 256/265 labels in Sec. 3.1. In the clean two-turn test, 3 of 10 models call 1−δ the existence probability (C.5). Models that keep 1−δ state a prior of 1.0 (D.2, D.6), i.e., they presuppose that the target exists.

  3. The effect is narrow and depends on prompt format. It disappears in three settings:

    • when coverage is given as a running percentage (B.4);
    • when coverage is withheld on MIMIC (0/64);
    • when a decline is allowed: GPT-5.5, GPT-5.6-sol, Grok 4.20 and Kimi K2.7 decline 100% (Table 13).

    With live tools, only 2 of 5 1−δ models persist, and only at a tolerance loosened to 0.05 (Table 9). The MIMIC and enterprise-RAG results rest on GPT-5.5 alone. Sec. 3.1 omits the bug-form rows in which non-1−δ models also substitute (Table 10).

4 Significance: even granting the results, the practical lesson (do not request a posterior without its inputs; allow abstention or compute it downstream) is standard practice, and the paper shows no deployed pipeline that violates it.

Questions for authors

NA

3 comments

We thank the reviewer for a detailed review. The review was right about the event field and the "default" wording, and the revision adopts both. The reviewer is also right that the response depends on how the coverage is worded and that a decline lowers the substitution for some models, and the revision states these limits. New results show that the main effect holds without the permissive form and without the 0.5 prior. A substituting model is defined in the general comment.

W2. Adopted: the event field and "default, not misunderstanding". A form that lets the model choose its event ("the number you report") is no longer counted as substitution, and the phrase is removed (Sec. 3, App. D.3). With the existence question kept and an event label added, 747 of 768 responses are 1−δ1-\delta in a field that still asks for the existence probability, against 716 of 768 without the label (App. D.3). The second-turn labels do not identify the event. Asked in a second turn, the 19 substituting models call 65% of their 580 answers at 1−δ1-\delta the existence probability, and for some models the label follows the order of the options (App. C.5). Therefore, a pipeline does not learn the event from the second-turn label, and the two-question check compares numbers. A stated prior of one presupposes the record exists, as the reviewer says, and then the posterior after a failed search is one (App. D.7).

W1. The normative frame. We agree that a likelihood is useful once it is identified as one: a correctly identified miss likelihood, combined with the base rate, gives the posterior. We also agree that 0.5 is only a reference prior (Sec. 2 now says so). The check does not depend on 0.5, because the check compares the existence answer with the miss likelihood. However, identifying the likelihood is difficult. A calibration score does not separate a likelihood from a posterior, because each rule scores best at the base rate that agrees with the rule (App. E.4). The number is also not always the pipeline's miss likelihood. When a search of a location finds a record only with a stated probability, only 32% of the 22 substituting models asked (pooled label) move to the new miss likelihood (App. E.2), and under a retrieval wording some flagged models give the unsearched share of a list (App. C.3). Mapped with the base rate, the 6 flagged models' numbers make about 3 more wrong decisions per 100 failed searches than the posterior from the search log on the emergency-department data at its own base rate, and all 9 models about 8 more (App. H.6). We recommend computing the likelihood from the search log whenever possible.

W3. "The effect is narrow and depends on prompt format." A single-probability sentence lowers the share of responses at 1−δ1-\delta to 52%, and no model keeps 1−δ1-\delta under the one recall wording we report (Sec. 3, App. D.4). In a diagnostic-test record with a stated sensitivity, on the bug form's 11 models, no scored model gives the miss likelihood in 60% of its responses, most substituting models give the posterior at the reference prior, and 9 of 11 use the stated prevalence (App. B.8). A decline helps some models: allowed "undetermined", substituting models' responses at 1−δ1-\delta fall from 92% to 33%. With live tools, GPT-5.5 and Grok 4.20 Reasoning keep 1−δ1-\delta within 0.001 (App. B.6), and App. B.7 now reports the bug-form rows where other models also substitute. Withholding the coverage also lowers the substitution, from 58% to 47% of the 60 models when those episodes are pooled in. On clinical records, 4 of the 5 scored models give 1−δ1-\delta in more than half of their 320 episodes, and two reach the 60% criterion (App. B.2). In the agentic enterprise search with the shares stated, 8 of 21 keep 1−δ1-\delta (App. B.4). With the coverage withheld on clinical records, the 5 scored models give 1−δ1-\delta in only 9 of 1,600 episodes (App. B.2), and in the live run with a running percentage only 1 of 176 responses is at 1−δ1-\delta (App. B.5).

W4. The lesson is standard, and no deployed pipeline is shown. We agree, and we do not measure harm in a deployed system. Google ADK's LlmClassifier, in the library's labs module, requires "the probability that the answer is true"; run on our failed searches with its prompt and schema unchanged, 6 of the 8 scored models return 1−δ1-\delta, including the open-weight ones (Sec. 3.1, App. B.9). This is a case study on our inputs. As motivation, 16 of 100 sampled public repositories have a model fill a confidence field beside a search call, and in all 16 the code omits the base rate (App. B.10). However, the finding that models fill the field with this one wrong number is new, and the two-question check tests a pipeline's wording for this number without labels (Sec. 3.4).

If the W1 answer meets the normative concern, we hope the reviewer will reconsider the score, and we will answer any follow-up.

Thank you for the thorough revisions. W1 and W2 are partially resolved. W3 and W4 remain. Outside the share-table format the effect largely disappears: 1/176 with a running percentage, 9/1,600 with coverage withheld, and 33% with a decline option. No deployed harm is shown, and the mapped cost is only about 3 wrong decisions per 100 failed searches.

We thank the reviewer for their follow up response and hope to address any remaining concerns.

"Outside the share-table format the effect largely disappears: 1/176 with a running percentage, 9/1,600 with coverage withheld, and 33% with a decline option." We agree that the effect depends on how coverage is stated, and the reviewer's three numbers help us state the claim more narrowly. First, the 1/176 run states the coverage as a fraction of all documents, which our earlier reply did not make clear. We agree that 1−δ1-\delta responses are rare in this case and Section 5 now mentions this beside the recall result of our earlier reply (W3). A share table format is not needed for 1−δ1-\delta and a new live search shows locations without their location shares and only gives cumulative coverage after each search. There, 5 out of 13 models match 1−δ1-\delta in at least 90%90\% of their 32 episodes across varying coverage (App. B.5). Second we agree that the count is near zero when coverage is withheld, so we limit the claim to settings where coverage is given or can be inferred. Third, we agree that a decline option helps, and now recommend allowing "undetermined" (Sec. 3.3). Thus, our claim is about a required number after a search where coverage is shown as a share of where records can be found and Section 1 now makes this clear.

"No deployed harm is shown, and the mapped cost is only about 3 wrong decisions per 100 failed searches." We agree that we do not show harm in a deployed system and Sec. 5 now states this limitation. The ~3 extra errors are remaining errors after the correction. They are for the 6 models our check flagged on the emergency department dataset at a base rate of 0.89 (App H.6), where we agree the correction works well. The 45 per 100 is not the same setting before the correction. It comes from a different dataset and base rate. On the enterprise dataset with a base rate of 0.01, without the correction, substituting models make about 45 more errors per 100 failed searches (Sec. 3.3, Fig. 6). Finally stating the base rate does not correct every model. Under the ADK classifier of our previous reply, Haiku 4.5 gives 1−δ1-\delta in 60 out of 64 episodes and only states the posterior in 5 of 64 once the base rate is stated (App. B.9).

We again thank the reviewer for their careful reading. If this addresses the remaining concerns, we kindly ask the reviewer to consider raising their score, and we would be happy to answer any further questions.

Comments on the paper

1 comment

We thank the reviewers for the discussion. We summarize the main finding and new results below.

Our main finding is that many LLMs answer a different question from the one they are asked. Suppose a search fails, and its search coverage δ\delta is the probability that it would have found the record if the record existed. When we ask for the probability that the record exists without stating a base rate, 58% of the 60 models report 1−δ1-\delta, the probability that the search misses an existing record (Sec. 3.1). The revision explicitly accepts "undetermined" and the posterior at a declared prior as correct answers, and it excludes prompts that let the model choose the event from the substitution counts (Sec. 2, App. D.3).

A shared concern was whether this effect needs our share table. On the same episodes, replacing the table with a running total after each search leaves the substitution almost unchanged (Fig. 5a). We also ran a live search without a share list, and several models still give 1−δ1-\delta in almost every episode (App. B.5). In a controlled replay (see reply to m5kS), most of the selected models also keep 1−δ1-\delta when the coverage is stated directly as a detection probability. We agree that the wording of the coverage matters, and 1−δ1-\delta is rare when the coverage is stated only as a share of all documents (Sec. 5).

The revision also addresses three other weaknesses. To test whether the finding carries through an existing probability interface, we ran Google's Agent Development Kit classifier on our inputs, and found that most models return 1−δ1-\delta (App. B.9). To test whether the effect is specific to one model provider, we extended the evaluation to 14 providers, and around half of the models outside OpenAI report 1−δ1-\delta (Fig. 4). Because the coverage must be estimated in practice, we also evaluated decisions with jointly estimated coverage and base rate (App. H.2). In the four models we tested, a warning that the coverage may be inaccurate does not remove the substitution (App. D.4).

What we find surprising is that models can correctly say that the probability is undetermined and still report the miss likelihood when required to output a number (App. C.5). The new experiments show that removing the share table does not reliably prevent this, and stating the missing base rate corrects only about 60% of the tested substituting models (Sec. 3.2). We also provide a two-question check for the substitution that does not require labeled outcomes (App. C.1).

We once again thank the reviewers for their feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.