- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BlindToChange, which pairs direct questions with indirect recommendation tasks about the same event to test whether search agents proactively find and use up-to-date information. The data comes from deaths and ends of tenure recorded in Wikidata. Experiments on eight models show that accuracy is generally higher on direct questions than on indirect recommendation tasks.
Critical strengths
-
The research question matters in practice: users usually do not spell out every time-sensitive fact that needs to be checked.
-
The paired questions, native and shared search setups, and search trajectory analysis provide useful evidence about how models check facts.
-
The authors checked the answer judge against human ratings and provided concrete examples of failures.
Critical weaknesses
-
The dataset covers only deaths and ends of tenure, both of which make a person no longer eligible for a particular role or activity. Real-world changes also include venue closures, discontinued products, service outages, rescheduled events, and rule changes. These may require different checks and different adjustments to the answer. The current experiments show problems with recommending people, but they do not establish that agents generally struggle to handle changes in the world. I suggest adding more types of events and reporting results for each type.
-
The model may simply find better candidates, produce a shorter list, or give a more cautious answer. It could therefore pass without knowing that the person has died or left office. Recommending the person in three consecutive runs without search does not rule out these possibilities. Also, the questions are filtered only at low reasoning effort, and there is no high-effort baseline without search. Some gains may therefore come from changes in whom the model recommends. The authors should check the search evidence in successful responses to confirm whether the model actually found the update.
-
Four of the eight models have no reported knowledge cutoff. Assigning them a common cutoff date does not establish that they are unaware of later events. Also, making a mistake in a recommendation task does not mean the model cannot answer correctly when asked directly. I suggest testing the direct questions without search. Otherwise, it is unclear whether correct answers come from newly retrieved information or knowledge the model already has.
-
A direct question asks about one named person’s status. An indirect recommendation request also requires finding candidates, comparing their qualifications, and checking several people. Lower accuracy may therefore partly reflect the more complex task. I suggest using the same recommendation question with and without a reminder to check current status, to measure how much that reminder matters.
-
Each model’s test set is selected based on its own answers without search. The events and question difficulty may therefore differ across models. Table 2 partly addresses this by using shared questions, but it covers only five models. I suggest adding a fixed test set for all models and reporting its results separately from those on model-specific test sets.
Questions for authors
1.Among the indirect answers marked correct, how many trajectories actually retrieved evidence of the update? Could you distinguish between cases where the model excluded someone after finding the update and cases where it simply did not recommend them without finding the update?
2.Could you add a high-reasoning-effort baseline without search, and also test the direct questions for the same events without search? This would help check whether the gains come from more reasoning and whether the models already know about the events.
3.Could you test the same recommendation task under four conditions: no extra hint, a reminder to check current eligibility, a specific target to verify, and the update evidence provided directly? This would help separate whether the model knows when to check, can find the information, and can use it correctly.
4.Could you provide anonymous code and datasets during the review period? These materials are currently unavailable, he supplementary materials only contain prompts, without executable code or datasets, making it difficult to independently check question generation and filtering, model-specific test sets, and scoring. Since the experiments rely on live web search, please also share model and tool settings, run dates, search-result snapshots, and full trajectories so the results can be checked and reproduced.
5.After adding more types of events, could you also test newer representative models, such as GPT-6 Astra, Opus 5.5, Gemini 4 Argon, Grok 4.7, GLM-5.3, and Kimi K3, and examine whether they still show the same weaknesses?
6.The evaluation prompts in the authors’ supplementary code reveal a gap between the paper’s claim and its scoring rules. The paper links avoiding an outdated recommendation to discovering the update, but the judge accepts answers that simply omit the target person, without checking whether the agent found any updated information.
7.The authors’ supplementary code also shows a mismatch between question generation and judging. The question-generation prompts require someone currently holding a position, while the judging prompts can accept recommendations based on a former holder’s experience or informal advice.
3 comments
Response to Reviewer FzGK (2/2)
mismatch between question generation and judging. The question-generation prompts require someone currently holding a position, while the judging prompts can accept recommendations based on a former holder’s experience or informal advice.
This difference is intentional and reflects the different roles of the two stages. Question generation ensures that the update disqualifies the affected person. The judge instead checks whether the response is consistent with the update, since BlindToChange measures whether the agent accounts for the change rather than the overall quality of its recommendations. For example, a response that recommends the person explicitly as a former holder for informal advice, may not fully satisfy the request, but it is consistent with the update and indicates that the agent discovered it.
Response to Authors
Thank you for the clarification. However, the response does not resolve my concerns, so I will maintain my original score.
Response to Reviewer FzGK (1/2)
We thank the reviewer for their feedback. We are encouraged that they found our research question well-motivated, our insights useful, and our analysis rigorous. Please see our response to common concerns above. We address the remaining concerns and questions specific to the reviewer below:
For models without reported knowledge cutoff—Qwen3.7-Max, DeepSeek-V4-Pro-0423, Kimi-K2.6, and GLM-5.1 we tested direct probing accuracy without search for both categories and effort levels. We found the accuracy to be 0 using the criterion that the date of the event mentioned by the models should be right (upto the month) to avoid rewarding guesses.
For direct questions (no search and high reasoning): if models have a reported knowledge cutoff, we are sure from dates of the events that accuracy is 0 and for the remaining 4 models, as discussed in the above question we find accuracy to be 0. This establishes that models are unaware of these events.
We believe your concern is: filtering at low effort could result in the gap between low and high effort being driven by non-zero indirect accuracy of high effort without search. We test this by comparing the gap on all probes (Table 1) vs on only the ones where high effort without search gets 0/3 correct. From the table below, we see that gap(0/3) is at most ~2 points less than gap(all probes) implying that the gap is unaffected by whether filtering happens only using low effort or both.
We thank the reviewer for this suggestion. The three capabilities it aims to separate are already measured in our setup, on the unmodified task.
We evaluated 2 current state-of-the-art models GPT-6 Astra and Gemini 4 Argon (knowledge cutoffs April and March 2026) on 200 probes. For demises we use events from May-June, while for tenures we use May-August (for enough samples). We used the shared harness. Direct accuracy was 100%.
While these models improve substantially over the generation we tested, indirect accuracy still falls short of direct accuracy, particularly at low effort (up to 21.5 pp for Gemini 4 Argon demises). At high effort the gap narrows to under 5 pp, suggesting that newer models discover changes more reliably, but only when given a larger reasoning and search budget.