- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Programmatic Search Agents (PSA), which use executable code to compose search operations, maintain persistent candidate workspaces, and selectively present evidence. Experiments on two benchmarks with five LLMs show improvements in task success rates and final-step token efficiency.
Critical strengths
- The paper provides a controlled comparison of query-based, tool-based, and code-based search agents across five LLMs. By sharing the same underlying retrieval components, the experiments help isolate the impact of different search interfaces on agent performance.
- PSA demonstrates improvements in task success rates and final-step token efficiency across two benchmarks. These results provide useful evidence that programmatic control over search operations can benefit complex information-seeking tasks.
Critical weaknesses
- The paper’s motivation and methodological novelty are not convincing. The authors argue that agents should control how retrieved evidence is processed and presented, but modern search agents already support tools such as visit and fetch for selectively accessing retrieved content. More importantly, Perplexity’s Search as Code (SaC) has already introduced a highly similar approach. Although the authors acknowledge SaC as an inspiration, they do not clearly explain the technical differences between the two approaches. As a result, I find it difficult to identify a substantial methodological contribution.
- The empirical contribution is also insufficient. While PSA outperforms the selected baselines, the experiments do not adequately explain why it performs better. The Query-based Agent baseline lacks common capabilities such as visit/fetch, making the comparison less convincing. The ablations also fail to clearly isolate the contributions of individual components. Moreover, the evaluation is limited to two fixed-corpus benchmarks, without testing on open-web benchmarks such as BrowseComp or WideSearch. The implementation also does not appear to be publicly available, raising reproducibility concerns. Overall, I don’t think the experimental evidence is strong enough to compensate for the limited methodological novelty.
Questions for authors
- What are the key technical differences between PSA and Perplexity’s Search as Code?
- Why was PSA not evaluated against existing search agents equipped with visit or fetch tools? Would the reported improvements still hold against these stronger baselines?