- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces DeepSearchDAG, a benchmark for evaluating complex search tasks through claim-level annotations and dependency DAGs. It contains 348 real-world business queries, 10,617 atomic claims, and 2,411 dependency edges. The authors propose NodeF1 to measure final-answer coverage and a trajectory-aware Prerequisite Edge Score (PES) to evaluate whether search agents follow annotated information dependencies. Experiments across eight backbone models and three agent frameworks reveal substantial limitations in both answer coverage and prerequisite-linked search behavior.
Critical strengths
- Fine-grained benchmark design. DSD evaluates complex search tasks at the claim level, providing more informative feedback than final-answer-only benchmarks.
- Extensive benchmark evaluation. The paper evaluates 32 model-framework configurations and includes human annotation procedures, judge consistency analysis, and detailed error diagnostics.
Critical weaknesses
- Unclear validity of prerequisite-edge scoring. PES rewards adherence to annotated dependencies, but valid search strategies may retrieve information through alternative paths. The paper lacks direct validation that higher PES reflects better search reasoning rather than conformity to a predefined structure.
- Limited independence from answer accuracy. NodeF1 and PES are highly correlated across configurations (Pearson r = 0.988). The additional diagnostic value of PES remains insufficiently established.
- Inconsistent fan-in evaluation. Verify nodes are defined as requiring multiple evidence branches, yet the Complete Success Rate requires only one incoming fan-in edge to be satisfied. This appears inconsistent with the intended evaluation of multi-source verification.
- Potential overestimation of answer coverage. Enumeration claims are considered fully covered when their set-level F1 reaches only 0.60. This binary threshold may reward incomplete lists and obscure meaningful differences in information coverage.
- Limited validation of evaluation reliability. Although the paper reports high LLM-judge agreement, cross-judge DSD-Score varies by 0.066 on the tested subset. Independent human validation of PES decisions is not provided.
- Questionable interpretation of structural depth. The reported Depth metric includes artificial ordering edges between claims within the same milestone. Consequently, it may reflect claim serialization rather than genuine search dependency depth.
- The pictures in this paper need to be polished.
Questions for authors
- PES validity. How do the authors verify that annotated prerequisite edges represent necessary information dependencies rather than one possible search strategy? Could they evaluate PES against human judgments of search-process quality?
- Incremental value of PES. Given the strong correlation between NodeF1 and PES (r = 0.988), what additional information does PES provide beyond answer accuracy? Could the authors analyze cases where the two metrics disagree substantially?
- Fan-in consistency. Why does the Complete Success Rate require only one incoming edge for a fan-in verification node, while the DAG formulation emphasizes multi-source verification and PES scores each incoming edge separately?
- Metric sensitivity. How sensitive are model rankings to the enumeration threshold of 0.60 and the DSD-Score weights of 0.7/0.3? Could the authors report results using stricter coverage criteria?
- Judge reliability. How well do LLM-generated PES judgments agree with independent human assessments? Does the observed cross-judge variation affect model rankings?
- Structural depth. How much of the reported Depth comes from actual dependency edges versus artificial within-milestone ordering? Would the benchmark retain its claimed depth advantage under a dependency-only definition?