- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Deep Cognition is a multi-agent system for deep research tasks that exposes plans, subagents, queries, evidence, and drafts so users can intervene mid-process. It has a 2B evidence router and a pipeline that uses an LLM to simulate an expert. The authors report a 50 people user study, benchmark results on 5 datasets and controlled feedback-timing experiments and analysis of 2461 sessions.
Critical strengths
1- Mid-process, artifact level steering of deep research agents is a timely and useful problem. 2- The feedback timing controls in Table 5 and Appendix B fix the feedback content and budget, which is the right experimental setup. 3- Real deployment data and a user study with professionals are rare in this area.
Critical weaknesses
1- Numbers contradict each other across tables: (1) Table 5 reports 0.7681 (structured) and 0.7583 (post-output). Table 10, which claims the same setup, reports 0.6912 and 0.6861. (2) ResearcherBench in Table 4 (0.7281) equals the post-output refinement score in Appendix B.1. (3) The Claude expert row in Table 6 gives 0.5942 on ScholarQA-CS, while DC in Table 4 gives 0.7652. (4) It is unclear which configuration produced which headline number. 2- Table 3 claims n=50, but every mean (e.g., 2.231, 4.385, 4.538, 4.35, 4.73) is a multiple of 1/26. That points to 26 ratings per system, not 50. Please clarify. 3- In Table 6, every expert condition scores below "None" on ScholarQA-CS (0.594–0.658 vs. 0.667), so expert interaction hurts there. The abstract cites only the ResearcherBench and DeepResearch gains, and the text never discusses this. 4- Benchmark comparisons are unfair due to unequal expert feedback, inconsistent judges and GAIA splits, while autonomous DC trails or ties commercial baselines. 5- The timing claim is overstated. The gap is only about 0.005. 6- The user study is biased toward DV due to the interaction metrics favor DC by consutrction since the baselines can't be interrupted. 7- Deployment analysis is weak. A "4.7× quality shift" on a 1–5 scale isn't explained. The judge is an LLM (Claude Sonnet 4.6), the comparison uses 25 within-session pairs, and nothing is controlled.
Questions for authors
1- Which backbone, expert, and checkpoint produced each number in Tables 4, 5, 6, and 10? Why do Tables 5 and 10 disagree for the same setting? 2- How many participants and evaluators participated in Table 3? What was the agreement? 3- Why do all expert conditions hurt on ScholarQA-CS?