- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper aims to address how sequential searches can bias the outcomes of deep research agents. It proposes a method to estimate the average score over a predefined candidate pool of searched items, and the experiments primarily evaluate how accurately and consistently this statistic can be recovered.
Critical strengths
The paper aims to address an important problem of evidence-selection bias in deep research agents. Its controlled experiments systematically vary document rankings, selection policies, and stopping decisions, providing useful empirical insights into how adaptive search can influence the evidence collected by an agent.
Critical weaknesses
-
There is a disconnection between the motivation and the proposed method. The paper motivates the problem of deep research agents producing misleading conclusions due to biased evidence selection. However, CESS has a pretty big gap to the motivated setting. For example, CESS estimates the average evidence score (Eq. 1) over a predefined candidate pool, but obtaining the complete candidate pool and evidence scores is already difficult in practice. Also, it is unclear why the average evidence score should represent a reliable research conclusion.
-
The experiments do not directly validate the motivation. Most experiments evaluate whether CESS can recover the candidate-pool average and reduce ranking sensitivity, rather than whether it can detect misleading conclusions in generated research reports. The paper also uses the Opened Mean as a proxy for the agent's conclusion, but an LLM's actual conclusion may be quite different from simply averaging the documents it reads.
-
Limited methodological novelty and insufficient justification. Eq. (6) is largely based on existing inverse-probability weighting and doubly robust estimation methods. The technical novelty beyond applying these methods to Deep Research is not very clear. Also, the causal model in Eq. (2) is introduced without sufficient explanation of its construction or how its assumptions hold for actual agents.
-
The contribution of the causal intervention analysis is unclear. The paper introduces a separate analysis of selection and stopping effects in Eq. (4). However, Theorem 2 seems to follow directly from the assumption that the estimators are unbiased for the same target. The intervention experiments mainly confirm that CESS contrasts do not measure policy effects, which is already expected from the estimator's objective. Although studying the causal effects of search policies could provide useful insights into evidence-selection bias, the paper provides limited analysis of these effects, especially their interaction. Therefore, it is unclear what additional contribution this analysis makes beyond distinguishing two different estimation objectives.
Questions for authors
-
How does the average evidence score in Eq. (1) relate to the actual conclusion of a Deep Research agent? How would the candidate pool be obtained in a realistic open-web setting?
-
Can the authors evaluate whether CESS can actually detect misleading conclusions in generated research reports, rather than only recovering a predefined candidate-pool average?
-
What is the main methodological novelty of Eq. (6) compared with existing inverse-probability weighting methods? Could the authors also justify the causal dependencies and assumptions in Eq. (2)?
-
How does the causal intervention analysis contribute to the proposed CESS method? Why is the interaction effect in Eq. (4) not discussed in detail in the experimental results?