acceptodds
Under review as a conference paper at ICLR 2027

Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents

Abstract

Deep Research agents synthesize evidence into cited reports, yet a well-cited report can still reach a misleading conclusion. Citation correctness checks whether cited sources support individual claims. It does not show whether adaptive search exposed a representative view of all documents made available for evaluation, which we call the candidate pool. Early findings redirect later queries, document choices, and stopping, so the documents an agent reads form a selective sample. Existing evaluations rarely account for this selection. We formulate the problem as adaptive evidence sampling and introduce Causal Evidence Selection Correction (CESS). CESS predicts each candidate document's evidence direction and corrects the candidate-pool average using the logged probabilities of selecting each document and reaching each search round. Shrinkage stabilizes short searches, while intervals replace point estimates when some documents cannot be sampled. We also prove that estimating the average evidence direction of a common pool differs from measuring how a change in search policy alters the evidence read. The latter requires intervention. On questions from the MS2 systematic-review benchmark, CESS reduces mean absolute error against the candidate-pool average by 9.2% and reduces the estimate's change under opposing document rankings by 39.4% relative to averaging the evidence scores of documents read. Across trajectories from a public Open Deep Research agent, the corresponding reductions reach 60.1% and 87.2%. A further 4,800 trajectories under paired interventions confirm that correcting a pool estimate and measuring a policy effect are different tasks. CESS therefore audits whether the evidence direction underlying a report reflects the documents available for evaluation, while a separate intervention analysis measures the effect of search decisions. Code and configurations are available at https://anonymous.4open.science/r/Auditing-Deep-Research-Bias-5B58/.

Primary area:
interpretability and explainable AI
Keywords:
Evidence Selection, Deep Research Agents, Causal Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TuXrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to address how sequential searches can bias the outcomes of deep research agents. It proposes a method to estimate the average score over a predefined candidate pool of searched items, and the experiments primarily evaluate how accurately and consistently this statistic can be recovered.

Critical strengths

The paper aims to address an important problem of evidence-selection bias in deep research agents. Its controlled experiments systematically vary document rankings, selection policies, and stopping decisions, providing useful empirical insights into how adaptive search can influence the evidence collected by an agent.

Critical weaknesses
  1. There is a disconnection between the motivation and the proposed method. The paper motivates the problem of deep research agents producing misleading conclusions due to biased evidence selection. However, CESS has a pretty big gap to the motivated setting. For example, CESS estimates the average evidence score (Eq. 1) over a predefined candidate pool, but obtaining the complete candidate pool and evidence scores is already difficult in practice. Also, it is unclear why the average evidence score should represent a reliable research conclusion.

  2. The experiments do not directly validate the motivation. Most experiments evaluate whether CESS can recover the candidate-pool average and reduce ranking sensitivity, rather than whether it can detect misleading conclusions in generated research reports. The paper also uses the Opened Mean as a proxy for the agent's conclusion, but an LLM's actual conclusion may be quite different from simply averaging the documents it reads.

  3. Limited methodological novelty and insufficient justification. Eq. (6) is largely based on existing inverse-probability weighting and doubly robust estimation methods. The technical novelty beyond applying these methods to Deep Research is not very clear. Also, the causal model in Eq. (2) is introduced without sufficient explanation of its construction or how its assumptions hold for actual agents.

  4. The contribution of the causal intervention analysis is unclear. The paper introduces a separate analysis of selection and stopping effects in Eq. (4). However, Theorem 2 seems to follow directly from the assumption that the estimators are unbiased for the same target. The intervention experiments mainly confirm that CESS contrasts do not measure policy effects, which is already expected from the estimator's objective. Although studying the causal effects of search policies could provide useful insights into evidence-selection bias, the paper provides limited analysis of these effects, especially their interaction. Therefore, it is unclear what additional contribution this analysis makes beyond distinguishing two different estimation objectives.

Questions for authors
  1. How does the average evidence score in Eq. (1) relate to the actual conclusion of a Deep Research agent? How would the candidate pool be obtained in a realistic open-web setting?

  2. Can the authors evaluate whether CESS can actually detect misleading conclusions in generated research reports, rather than only recovering a predefined candidate-pool average?

  3. What is the main methodological novelty of Eq. (6) compared with existing inverse-probability weighting methods? Could the authors also justify the causal dependencies and assumptions in Eq. (2)?

  4. How does the causal intervention analysis contribute to the proposed CESS method? Why is the interaction effect in Eq. (4) not discussed in detail in the experimental results?

Reviewer_bvbsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies evidence selection bias in Deep Research Agents, focusing on how evidence acquired during early search stages can influence subsequent queries, document selection, and stopping decisions. The authors formulate this problem as adaptive evidence sampling and propose Causal Evidence Selection Correction (CESS), a method designed to estimate the average evidence direction of a predefined candidate document pool from selectively observed documents. CESS combines full-pool outcome predictions with inverse-probability correction for document selection and search continuation, and applies shrinkage to stabilize estimates under limited search budgets. The paper further distinguishes candidate-pool estimation from causal attribution of search-policy effects, using paired interventions to study the latter. Experiments on MS2 and PERSPECTRUM, including trajectories from a public Deep Research Agent operating with a controlled search interface, report improvements in candidate-pool estimation accuracy and sensitivity to document rankings over several baselines.

Critical strengths

A meaningful perspective on evidence selection in Deep Research Agents. The paper highlights a potential limitation of conventional citation correctness and factuality evaluations: a report may accurately summarize the documents it cites while overlooking relevant contradictory evidence during adaptive search. The observation that previously acquired evidence can influence subsequent queries, document selection, and stopping decisions is relevant to understanding the reliability of Deep Research Agents. The distinction between estimating a candidate-pool target and attributing changes in observed evidence to search policies is also conceptually useful.

Critical weaknesses

First, Insufficient justification of the problem formulation and auditing target. The paper motivates its approach by arguing that adaptive evidence acquisition may lead Deep Research Agents toward misleading conclusions. However, the proposed framework ultimately formulates this problem as estimating the equally weighted average evidence direction over a predefined candidate document pool. The paper does not sufficiently justify why this statistical quantity is an appropriate reference for evaluating the reliability of research conclusions. In practice, Deep Research Agents are expected to prioritize documents based on relevance, quality, and evidential value, and such informed selection may legitimately produce an evidence distribution that differs from the overall candidate pool. The paper therefore does not clearly distinguish harmful evidence selection bias from reasonable evidence prioritization, leaving the connection between the motivating problem and the proposed auditing objective insufficiently established.

Second, Limited practical applicability and insufficient validation of the proposed auditing capability. CESS requires a predefined candidate document pool, outcome predictions for all candidate documents, and logged document-selection probabilities, which are difficult to obtain in realistic open-ended Deep Research environments. Although the authors evaluate CESS using a public Deep Research Agent, they replace its original search environment with a controlled, fixed-document environment that explicitly provides these quantities. Moreover, the experiments primarily measure estimation accuracy against the predefined candidate-pool average rather than the reliability of the agent's generated reports. In the public agent experiment, the improvement over full-pool Outcome Regression is relatively modest (MAE of 0.2287 versus 0.2404), and supplementary comparisons show that CESS does not consistently outperform simpler shrinkage baselines. Consequently, the experiments do not sufficiently demonstrate that the proposed statistical correction provides a practically useful auditing capability for real-world Deep Research Agents.

Questions for authors

1. The paper uses the equally weighted average evidence direction of a predefined candidate pool as its auditing target. Could the authors clarify why this quantity is an appropriate reference for assessing the reliability of Deep Research conclusions? In particular, how does the framework distinguish undesirable selection bias from legitimate prioritization of more relevant or higher-quality evidence?

2. How could CESS be applied to an open-ended Deep Research Agent when the complete candidate document pool is unknown and document-selection probabilities are unavailable? Since the public agent experiment replaces the original search environment with a controlled document pool, which aspects of the proposed auditing framework can be transferred to realistic retrieval environments without modifying the search system?

Reviewer_8Y5FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies evidence-selection bias in Deep Research agents: a report may contain accurate citations while relying on an unrepresentative subset of the available evidence. It proposes Causal Evidence Selection Correction (CESS), which combines predictions over a prespecified candidate pool with residual corrections based on logged document-selection and continuation probabilities. Shrinkage stabilizes estimates under short search budgets. The paper also distinguishes estimating the candidate-pool target from measuring the effects of changing search policies. Experiments on MS2, PERSPECTRUM, and a modified Open Deep Research agent report improvements in target-estimation error and ranking sensitivity relative to averaging the evidence scores of opened documents.

Critical strengths
  1. The paper addresses an important reliability problem. It clearly explains why citation correctness alone cannot establish whether an agent has considered representative evidence, providing a useful perspective on evaluating research agents.
  2. The audit objective is clearly defined. Separating the candidate-pool target from the effect of changing a search policy is conceptually useful. This distinction prevents a stable corrected estimate from being misinterpreted as evidence that search decisions had little influence on what the agent read.
  3. The evaluation includes meaningful controls. Paired ranking interventions, separate tuning questions, and question-level bootstrap resampling strengthen the experimental design. The comparisons at matched accuracy also address the possibility that reduced ranking sensitivity simply reflects stronger shrinkage.
Critical weaknesses
  1. The advantage over strong shrinkage baselines is inconsistent. The improvements over Opened Mean are encouraging, but do not by themselves establish the value of the proposed correction. In Table 8, CESS has higher MAE (0.3103) than both the tuning-set mean (0.2787) and the simple shrinkage baseline (0.2427). Although the matched-MAE comparison favors CESS in 10 of 28 settings, the reciprocal comparison in Table 11 shows lower MAE in only 3 settings and higher MAE in 10 at matched sensitivity. These mixed results need a clearer explanation of when probability correction improves the accuracy–stability trade-off beyond prediction and shrinkage alone.

  2. The theoretical contribution appears modest. Equation 6 explicitly follows augmented inverse-probability estimation, while Theorem 2 follows directly from the fact that two estimators with the same conditional expectation have a zero expected difference. The latter is a useful clarification of estimands, but not a substantial new identification result. The paper should more clearly distinguish its application-specific contribution from the established statistical ingredients it combines.

The practical requirements and predictor implementation need clarification. The unbiasedness guarantee assumes accurate controller probabilities and positive selection probabilities for every candidate at every reachable history. The public-agent experiment makes these quantities observable by replacing the search tool with a fixed document pool. This is a valid controlled evaluation, but leaves practical questions about obtaining full-pool predictions and reliable acquisition probabilities. For the real-data experiments, the manuscript does not sufficiently describe the pre-search features, outcome-model training, and task-wise shrinkage calculation. Different experiments also use different shrinkage anchors and clipping orders, without a clear general rule for choosing among them. These details matter for assessing reproducibility and the source of the gains.

Questions for authors
  1. Baseline comparisons: How do you explain the weaker MAE results in Table 8 and the reciprocal matched-sensitivity results in Table 11? Which conditions in the existing experiments distinguish cases where CESS improves over simple shrinkage from those where it does not?
  2. Prediction inputs and costs: What exactly are the pre-search features XiXi and the training procedure for the outcome predictor in each real-data experiment? Which information from unopened documents is available to this predictor, and what additional computation or document-access costs are required?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.