acceptodds
Under review as a conference paper at ICLR 2027

BlindToChange: AI Agents Can Find Changes When Asked But Do They Look When Needed?

Abstract

Given that AI agents can search the web broadly and deeply in real time, it is tempting to conclude that their outputs account for recent world events. We show that this conclusion does not hold in practice. While agents can search and easily find recent changes, the correctness of their responses depends on whether they search and what they search for. Popular agentic search benchmarks like BrowseComp cannot expose failures wherein agents do not recognize their own assumptions that should be verified, since these benchmarks explicitly state what needs to be found. In contrast, many user requests state only the final information need (e.g., an itinerary for a trip), leaving the agent to discover relevant changes (e.g., the temporary closure of a museum) and account for their consequences. To highlight such gaps, we introduce BlindToChange, a benchmark that measures whether agents proactively discover recent events that affect their responses. Our benchmark comprises requests for recommendations where an obvious candidate is no longer applicable due to a recent real-world event. These events are drawn automatically from Wikidata, allowing BlindToChange to be periodically refreshed. Strikingly, we find a big gap—as high as 27.5 points, even at high reasoning effort—in systems' performance, when the recent event is not explicitly mentioned compared to when we ask about it directly. Analyzing search trajectories, we learn that more than half of the failures can be attributed to the fact that agents do not directly search to verify whether a relevant fact has changed. Our benchmark highlights, and tracks, a new axis of progress, wherein systems account for recent information that is not explicitly mentioned in the user request. We will publicly release the benchmark and evaluation code upon publication.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Agentic Search, Retrieval-Augmented Generation, Temporal Knowledge, Implicit Information Needs, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FzGKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BlindToChange, which pairs direct questions with indirect recommendation tasks about the same event to test whether search agents proactively find and use up-to-date information. The data comes from deaths and ends of tenure recorded in Wikidata. Experiments on eight models show that accuracy is generally higher on direct questions than on indirect recommendation tasks.

Critical strengths
  • The research question matters in practice: users usually do not spell out every time-sensitive fact that needs to be checked.

  • The paired questions, native and shared search setups, and search trajectory analysis provide useful evidence about how models check facts.

  • The authors checked the answer judge against human ratings and provided concrete examples of failures.

Critical weaknesses
  • The dataset covers only deaths and ends of tenure, both of which make a person no longer eligible for a particular role or activity. Real-world changes also include venue closures, discontinued products, service outages, rescheduled events, and rule changes. These may require different checks and different adjustments to the answer. The current experiments show problems with recommending people, but they do not establish that agents generally struggle to handle changes in the world. I suggest adding more types of events and reporting results for each type.

  • The model may simply find better candidates, produce a shorter list, or give a more cautious answer. It could therefore pass without knowing that the person has died or left office. Recommending the person in three consecutive runs without search does not rule out these possibilities. Also, the questions are filtered only at low reasoning effort, and there is no high-effort baseline without search. Some gains may therefore come from changes in whom the model recommends. The authors should check the search evidence in successful responses to confirm whether the model actually found the update.

  • Four of the eight models have no reported knowledge cutoff. Assigning them a common cutoff date does not establish that they are unaware of later events. Also, making a mistake in a recommendation task does not mean the model cannot answer correctly when asked directly. I suggest testing the direct questions without search. Otherwise, it is unclear whether correct answers come from newly retrieved information or knowledge the model already has.

  • A direct question asks about one named person’s status. An indirect recommendation request also requires finding candidates, comparing their qualifications, and checking several people. Lower accuracy may therefore partly reflect the more complex task. I suggest using the same recommendation question with and without a reminder to check current status, to measure how much that reminder matters.

  • Each model’s test set is selected based on its own answers without search. The events and question difficulty may therefore differ across models. Table 2 partly addresses this by using shared questions, but it covers only five models. I suggest adding a fixed test set for all models and reporting its results separately from those on model-specific test sets.

Questions for authors

1.Among the indirect answers marked correct, how many trajectories actually retrieved evidence of the update? Could you distinguish between cases where the model excluded someone after finding the update and cases where it simply did not recommend them without finding the update?

2.Could you add a high-reasoning-effort baseline without search, and also test the direct questions for the same events without search? This would help check whether the gains come from more reasoning and whether the models already know about the events.

3.Could you test the same recommendation task under four conditions: no extra hint, a reminder to check current eligibility, a specific target to verify, and the update evidence provided directly? This would help separate whether the model knows when to check, can find the information, and can use it correctly.

4.Could you provide anonymous code and datasets during the review period? These materials are currently unavailable, he supplementary materials only contain prompts, without executable code or datasets, making it difficult to independently check question generation and filtering, model-specific test sets, and scoring. Since the experiments rely on live web search, please also share model and tool settings, run dates, search-result snapshots, and full trajectories so the results can be checked and reproduced.

5.After adding more types of events, could you also test newer representative models, such as GPT-6 Astra, Opus 5.5, Gemini 4 Argon, Grok 4.7, GLM-5.3, and Kimi K3, and examine whether they still show the same weaknesses?

6.The evaluation prompts in the authors’ supplementary code reveal a gap between the paper’s claim and its scoring rules. The paper links avoiding an outdated recommendation to discovering the update, but the judge accepts answers that simply omit the target person, without checking whether the agent found any updated information.

7.The authors’ supplementary code also shows a mismatch between question generation and judging. The question-generation prompts require someone currently holding a position, while the judging prompts can accept recommendations based on a former holder’s experience or informal advice.

3 comments
AuthorsOpenReview ↗

Response to Reviewer FzGK (1/2)

We thank the reviewer for their feedback. We are encouraged that they found our research question well-motivated, our insights useful, and our analysis rigorous. Please see our response to common concerns above. We address the remaining concerns and questions specific to the reviewer below:

Four of the eight models have no reported knowledge cutoff. __ making a mistake in a recommendation task does not mean the model cannot answer correctly when asked directly. I suggest testing the direct questions without search.

For models without reported knowledge cutoff—Qwen3.7-Max, DeepSeek-V4-Pro-0423, Kimi-K2.6, and GLM-5.1 we tested direct probing accuracy without search for both categories and effort levels. We found the accuracy to be 0 using the criterion that the date of the event mentioned by the models should be right (upto the month) to avoid rewarding guesses.

the questions are filtered only at low reasoning effort, and there is no high-effort baseline without search

Could you add a high-reasoning-effort baseline without search, and also test the direct questions for the same events without search? This would help check whether the gains come from more reasoning and whether the models already know about the events.

For direct questions (no search and high reasoning): if models have a reported knowledge cutoff, we are sure from dates of the events that accuracy is 0 and for the remaining 4 models, as discussed in the above question we find accuracy to be 0. This establishes that models are unaware of these events.

We believe your concern is: filtering at low effort could result in the gap between low and high effort being driven by non-zero indirect accuracy of high effort without search. We test this by comparing the gap on all probes (Table 1) vs on only the ones where high effort without search gets 0/3 correct. From the table below, we see that gap(0/3) is at most ~2 points less than gap(all probes) implying that the gap is unaffected by whether filtering happens only using low effort or both.

ModelHarnessEventGap (all probes)Gap (0/3 probes)
GPT-5.4NativeDemises34.533.5
GPT-5.4SharedDemises59.058.5
GPT-5.4NativeTenures17.018.4
GPT-5.4SharedTenures10.58.8
Gemini-3.1-ProNativeDemises33.535.7
Gemini-3.1-ProSharedDemises31.535.1
Gemini-3.1-ProNativeTenures25.526.6
Gemini-3.1-ProSharedTenures9.08.5

Could you test the same recommendation task under four conditions: no extra hint, a reminder to check current eligibility, a specific target to verify, and the update evidence provided directly? This would help separate whether the model knows when to check, can find the information, and can use it correctly.

We thank the reviewer for this suggestion. The three capabilities it aims to separate are already measured in our setup, on the unmodified task.

  • Does the agent know when to check? Our error analysis (§6) identifies failures where no search could reveal the change. These make up at least 75% of demise failures in 13 of 15 configurations.
  • Can the agent find the information? This is what the direct probe measures. Agents reach at least 96.5% accuracy on demises and 91.0% on tenures at high effort in nearly all configurations (Table 1).
  • Can the agent use the information correctly? The same analysis identifies reasoning errors, where the update is in the agent’s context yet the answer is outdated. These account for at most 15% of demise failures in every configuration, and a median of 24% of tenure failures (Figure 3).

could you also test newer representative models, such as GPT-6 Astra, Opus 5.5, Gemini 4 Argon, Grok 4.7, GLM-5.3, and Kimi K3, and examine whether they still show the same weaknesses?

We evaluated 2 current state-of-the-art models GPT-6 Astra and Gemini 4 Argon (knowledge cutoffs April and March 2026) on 200 probes. For demises we use events from May-June, while for tenures we use May-August (for enough samples). We used the shared harness. Direct accuracy was 100%.

ModelDemises LowDemises HighTenures LowTenures High
GPT-6-Astra89.598.597.5100.0
Gemini-4-Argon78.597.090.595.0

While these models improve substantially over the generation we tested, indirect accuracy still falls short of direct accuracy, particularly at low effort (up to 21.5 pp for Gemini 4 Argon demises). At high effort the gap narrows to under 5 pp, suggesting that newer models discover changes more reliably, but only when given a larger reasoning and search budget.

AuthorsOpenReview ↗

Response to Reviewer FzGK (2/2)

mismatch between question generation and judging. The question-generation prompts require someone currently holding a position, while the judging prompts can accept recommendations based on a former holder’s experience or informal advice.

This difference is intentional and reflects the different roles of the two stages. Question generation ensures that the update disqualifies the affected person. The judge instead checks whether the response is consistent with the update, since BlindToChange measures whether the agent accounts for the change rather than the overall quality of its recommendations. For example, a response that recommends the person explicitly as a former holder for informal advice, may not fully satisfy the request, but it is consistent with the update and indicates that the agent discovered it.

Reviewer_FzGKOpenReview ↗

Response to Authors

Thank you for the clarification. However, the response does not resolve my concerns, so I will maintain my original score.

Reviewer_MGatOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BlindToChange, a benchmark evaluating whether AI agents actively discover and consider real-world changes. The authors design two probe types: Direct Probe and Indirect Probe Built on Wikidata, the dataset covers two event types: demises and end of tenures. The authors evaluate 8 frontier models. Experiments show a significant performance drop on Indirect Probes, even when models correctly answer Direct Probes. In native search environments under high reasoning intensity, the median accuracy gap for the demise task remains 27.5 percentage points. Error analysis further shows many failures stem from agents failing to verify whether existing knowledge is outdated.

Critical strengths
  1. Paired Direct and Indirect Probes help isolate information retrieval ability from proactive change detection.

  2. Using Wikidata and LLMs to generate test samples reduces manual annotation costs and enables continuous future updates.

  3. Covers 8 models, two search environments, and varying reasoning intensities. Search trace analysis clarifies failure modes.

Critical weaknesses
  1. Despite emphasizing real-world dynamic changes, experiments only cover demises and end of tenures. Missing scenarios like product updates, policy changes, or venue closures limit generalizability.

  2. Indirect Probe accuracy relies on whether an invalidated entity is recommended. Not recommending an entity does not guarantee proactive discovery; search bias could simply shift recommendation preferences.

Questions for authors
  1. Does introducing a simple mandatory fact-checking mechanism (e.g., requiring agents to verify candidate status before recommending) significantly narrow the Direct vs. Indirect Probe gap? What is the associated search cost?

  2. How do you ensure an agent omitted a target entity due to proactive change discovery rather than altered search results, random sampling, or other noise?

  3. Does model-specific filtering create inconsistent benchmark difficulty across models? Can supplementary results be provided on a unified, unfiltered test set?

1 comment
AuthorsOpenReview ↗

Response to Reviewer MGat

We thank the reviewer for their thoughtful feedback. We are glad they recognize the benefits of our fully automated benchmark and are pleased they found our experiments and analysis comprehensive. All of the reviewer’s concerns overlap with those of Reviewer FzGK, so we address them in our response to common concerns above: (1) does omission implies discovery (2) model-specific test sets (3) effect of a fact-checking instruction and its search cost. We would be happy to clarify anything further.

Reviewer_6za6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes learning dataset-specific realizations of persistent task-relevant identities for enterprise data agents. Supporting and opposing evidence shape Gaussian prototypes and soft membership profiles. A separate query-prototype model and learned compatibility matrix route queries to a bounded evidence view, which an Arbiter and tool-using agent interpret and execute against. The paper derives conditional assignment-stability bounds and reports 94.67% dataset-macro Pass@1 on DAB, versus 84.13% for a Core system using the same Opus 5 reasoner, over five trials of 54 queries.

Critical strengths

The paper distinguishes hard assignment from soft memberships and limits its stability certificate to perturbations under a fixed prototype bank. Appendix B provides a valid margin/Lipschitz proof structure and explicitly separates assignment stability from semantic correctness. The resource reporting also usefully reveals the cost of the full system rather than presenting quality alone.

Critical weaknesses

To judge what is learned and whether the evaluation tests it, I need to know (i) how the identity inventory, feature map, and supporting/opposing weights are constructed and what labels or task information they use; (ii) what targets train the query encoder, prototypes, and compatibility matrix, and which datasets/queries are available during fitting and selection; and (iii) how the materializer generates its evidence view and which parts are learned, rule-based, or query-dependent. Sections 3 and 5.2 do not make these operational choices sufficiently explicit. Without them, I cannot determine whether the supervision matches the claimed task or whether training and evaluation are separated appropriately. This is not an allegation of leakage or invalid results. Missing exact hyperparameters and source code would primarily impede reproduction; the missing supervision and data boundaries impede assessment of validity itself.

Questions for authors

Please specify the three operational stages: identity/evidence construction and its supervision; query-side training targets and data splits; and evidence-view materialization. For each, identify what benchmark data, queries, labels, and feedback are accessible during fitting, selection, and inference. A worked example and executable-level pseudocode would help. This information would allow assessment of training/evaluation validity even if code and some implementation details remain proprietary.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Common Concerns of Reviewers MGat and FzGK (1/2)

We thank reviewers MGat and FzGK for their thoughtful feedback. We are encouraged that they found our setup of using paired direct (eg: Is person X alive?) and indirect probes (eg: Who can we invite for a talk next week? - where answering with X is wrong since they passed away) a controlled way to isolate the ability to find a fact when asked directly versus discover recent events relevant to the answer. We are pleased they found our experiments and analysis comprehensive, providing useful insights into whether agents verify their assumptions.

We address the reviewers' common concerns below:

Not recommending an entity does not guarantee proactive discovery; search bias could simply shift recommendation preferences. @MGat

How do you ensure an agent omitted a target entity due to proactive change discovery rather than altered search results, random sampling, or other noise? @MGat

It could pass without knowing that the person has died or left office. The authors should check the search evidence in successful responses to confirm whether the model actually found the update. @FzGK

Among the indirect answers marked correct, how many trajectories actually retrieved evidence of the update? Could you distinguish between cases where the model excluded someone after finding the update and cases where it simply did not recommend them without finding the update? @FzGK

The paper links avoiding an outdated recommendation to discovering the update, but the judge accepts answers that simply omit the target person, without checking whether the agent found any updated information. @FzGK

We agree that omitting an outdated entity does not guarantee discovery. However, discovery is not necessary for an answer to be up to date. Our task requires only that the agent not recommend an outdated entity, i.e., someone who is deceased or no longer holds the position. Because we retain only questions where the model without search recommends the outdated entity, omission reflects a genuine change in behavior. We acknowledge that search could still shift the recommendation without the agent encountering the event. Since our evaluation targets closed-source models whose retrieved context is typically not exposed, verifying discovery directly is not feasible. We therefore treat omission as a behavioral measure of being up to date and will clarify this limitation in the revised paper.

While discovery cannot be verified for closed-source agents, our shared harness exposes the full retrieved context. To quantify the fraction of cases where agents are successful without encountering the event, similar to our error analysis (§6), we classify traces where the model was successful: 1) No evidence in context and if evidence was present then 2) Explicit 3) Indirect 4) No/unrelated search resulted in discovery of the evidence. The table below reports the percentage of all indirect probes in which the agent succeeded without evidence.

ModelDemises-LowDemises-HighTenures-LowTenures-High
Opus-4.71.51.55.73.4
GPT-5.48.02.58.03.0
Gemini-3.1-Pro10.03.59.04.5
Qwen3.7-Max15.02.012.01.5
DeepSeek-V4-Pro5.00.01.52.5
Kimi-K2.65.03.05.01.5
MiMo-V2.5-Pro--3.5--1.7
GLM-5.16.02.53.01.5

For all models and categories, at high effort such cases were ≤\leq 4.5% (up to 15% at low effort), implying most omissions are due to discovery. Therefore, our benchmark overestimates agents’ already poor performance by the numbers in the above table, making it all the more concerning. Interestingly, by classifying search behavior resulting in success (Figure 6, Appendix K), we find models discover evidence of demise largely through indirect or unrelated searches while for tenures they explicitly search for the event. This supports our hypothesis that agents are more inclined to explicitly search for events that occur frequently.

Does model-specific filtering create inconsistent benchmark difficulty across models? Can supplementary results be provided on a unified, unfiltered test set? @MGat

Each model’s test set is selected based on its own answers without search. Table 2 partly addresses this by using shared questions, but it covers only five models. I suggest adding a fixed test set for all models and reporting its results separately from those on model-specific test sets. @FzGK

To control for difficulty, Table 2 (§5) compares five models under the shared harness on probes retained by all of them, as reviewer FzGK notes. We limited it to these models to retain a sufficient number of probes in the intersection for a reliable comparison. Model-specific filtering is necessary (Design Decision 3). An unfiltered set would include probes that test nothing for some models making comparison unfair.

AuthorsOpenReview ↗

Response to Common Concerns of Reviewers MGat and FzGK (2/2)

Does introducing a simple mandatory fact-checking mechanism (e.g., requiring agents to verify candidate status before recommending) significantly narrow the Direct vs. Indirect Probe gap? What is the associated search cost? @MGat

Lower accuracy may therefore partly reflect the more complex task. I suggest using the same recommendation question with and without a reminder to check current status, to measure how much that reminder matters. @FzGK

We agree with reviewer FzGK that the indirect task is more complex, and this is by design. Realistic requests rarely state which facts need verification, and recognizing the need to check within such a request is precisely the capability we measure. Complexity would be a confound only if agents attempted to verify the entity’s status and failed due to the added load. Our trajectory analysis (§6) shows this is not the main cause: on demises, at least 75% of failures in 13 of 15 configurations involve no search that could reveal the change.

We interpret the reviewers' suggestion as appending an instruction such as ”Before recommending any candidate, verify its current status” to the prompt. We find the suggestion to include a fact-checking instruction valuable. However, this particular prompt relies on the knowledge that the task at hand is to recommend candidates (which will not always be the case!). Therefore we use a slightly more general instruction: “Before finalizing your answer, verify the facts it relies on.” The following results are with the same indirect probes used in Table 1, and shared harness to report search cost.

ModelEventAccGain (pp)Direct AccGap closed to directQueries
GPT-5.4Demises97.5+3.0100.055%1.14x
GPT-5.4Tenures97.5+3.599.564%0.90x
Gemini-3.1-ProDemises86.0+10.599.544%0.97x
Gemini-3.1-ProTenures93.0+9.5100.058%0.84x
Qwen3.7-MaxDemises89.5+12.0100.053%1.12x
Qwen3.7-MaxTenures95.5+9.599.570%1.03x
DeepSeek-V4-ProDemises97.0+13.0100.081%1.77x
DeepSeek-V4-ProTenures97.5+4.5100.064%1.01x
MiMo-V2.5-ProDemises94.0+16.0100.073%2.14x
MiMo-V2.5-ProTenures99.1+7.0100.089%1.58x

The instruction improves accuracy for every model and event type, closing 44-89% of the gap between direct and indirect probing, yet a gap remains in every setting. Its search cost varies. On demises, most models search more, sometimes substantially: deepseek (1.77×) and mimo (2.14×). On tenures, search volume stays roughly unchanged for most models (0.84–1.03×). We will add these results to the revised paper and discuss the cost–accuracy trade-off.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.