- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PANINI-DR is a multi-agent research harness that organizes subquestions, dependencies, findings, and evidence in a shared workspace. Research agents investigate individual subquestions, while a verifier checks candidate findings and provides feedback for further investigation; admitted findings then inform later research and computation. The system is evaluated on BrowseComp-Plus, FRAMES, and MoNaCo using a shared gpt-oss-120b backbone, and it achieves the strongest average answer quality among the compared systems, with particularly large gains on harder questions.
Critical strengths
The improvement on BrowseComp-Plus is substantial and consistent across repeated runs. Specifically, PANINI-DR averages 72.7% accuracy, compared with 58.0% for both SLEUTH and Open Deep Search. The paper also includes useful controls, such as structured state versus prose memory, a common answer-recovery policy, and repeated SLEUTH sampling at comparable estimated computation. The last control is particularly convincing: even with repeated sampling, SLEUTH only reaches 64.7%, remaining below PANINI-DR’s 73.3% reference result.
The evidence-quality analysis is also valuable. The common-reader experiment and the claim-support audit complement the final-answer accuracy. I also appreciate the authors’ discussion of related approaches, inference costs, and practical failure cases.
Critical weaknesses
-
The incremental contribution of the proposed execution protocol remains insufficiently established, and the experimental evidence does not adequately validate the stated contributions. While the paper discusses related concepts (e.g., dependency-aware planning in LLMCompiler, persistent state in IterResearch), my concern is whether this specific integration—and its claimed components such as the shared workspace and typed subquestion review—provides a sufficiently distinct advantage over simpler baselines. The current ablations evaluate the entire package rather than its individual mechanisms. For instance, the planning/state ablation only compares the full package against prose memory, without isolating its advantage over a simpler dependency-aware implementation. More critically, as shown in Table 11, a reduced system with answer recovery already matches the full system's 73.3% BCP accuracy. While the full system yields better evidence support (Obs.) and uses fewer prompt tokens, it remains unclear which specific architectural contributions (e.g., the typed resolver, dedicated Reader, or the complex verification loop) are actually driving these gains.
-
Reproducibility heavily depends on execution details that are difficult to inspect without the implementation. The manuscript promises a public repository only after the review process, but I could not locate an accessible anonymized implementation link in the PDF or supplementary materials. The appendix usefully documents important exceptions: research outside the ready frontier is permitted, unsuccessful verification can lead to a WARNED admission, and final-answer recovery can produce an answer even when the target remains unresolved. Notably, this recovery mechanism supplies 40 of the 110 correct BCP answers. While these design choices are not inherently problematic and the paper includes recovery controls, the inability to inspect the exact execution policy during review makes it difficult to fully verify and interpret the headline results.
-
The verification ablation does not fully separate the quality of feedback from the additional research effort it triggers. Enabling verification improves BCP accuracy by 16 points, but the median prompt usage also rises from 453k to 1,042k tokens, and the introduction of evidence-attachment checks adds another confounding variable. This establishes the effectiveness of the complete policy, but not its superiority over a baseline that simply spends the same extra computation on more search or refinement. The matched-computation SLEUTH comparison evaluates the overall system rather than this specific mechanism. Furthermore, generality is uncertain: the primary experiments use a single backbone and frozen corpora, and the system shows smaller gains on FRAMES and a lower raw MoNaCo F1 than HyMem.
Questions for authors
-
While the paper lists related work, the core components (persistent state, verification, planning) are shared with several baselines (e.g., SLEUTH, Omni-Decision). Could the authors provide a concrete example (e.g., a specific query) demonstrating how PANINI-DR's specific design (such as the GSW-based typed subquestions and admission policy) leads to a different and better execution trajectory than these baselines? Which experiment best isolates this specific architectural advantage?
-
Given the complexity of the execution details, I strongly encourage the authors to provide the anonymized implementation and a subset of reasoning trajectories in the supplementary materials. Specifically, the execution configurations and rules governing WARNED admissions, reopening dependent findings, and answer recovery, along with concrete examples of these paths, should be made available for inspection during the review period to verify the reported results.
-
Beyond the existing system-level computation control, could the authors provide a focused comparison between verifier-guided research and additional search or refinement at approximately matched expenditure?