acceptodds
Under review as a conference paper at ICLR 2027

Programmatic Search Agents: Extending Agentic Search Beyond Query Reformulation

Abstract

Search agents adapt their queries, yet fixed search interfaces leave candidate processing and evidence presentation outside the agent's direct control. Our trajectory analysis shows that supporting passages can be retrieved yet never delivered to the agent; a same-page oracle intervention shows that changing the returned evidence can reduce subsequent search. We introduce Programmatic Search Agent (PSA), which makes a local executable computation over candidates the unit of a search action. PSA unifies a persistent candidate workspace, flexible primitive composition, and selective evidence presentation. It incrementally generates program cells that reuse candidates, execute dependent operations, and select what the agent inspects next. The runtime resolves specified data dependencies within each cell, while the agent adapts its search strategy across cells as new evidence arrives. We compare PSA with the Query-based Agent and Tool-based Agent on InfoSeek-Eval and BrowseComp-Plus using five policy backbones without task-specific training. All three interfaces share the search substrate, and the Tool-based Agent also shares PSA's primitives and persistent workspace. Relative to the Query-based Agent, PSA improves macro-averaged task success by 4.00 and 7.56 percentage points on the two benchmarks, respectively; within-backbone reductions in final-step tokens average 28.3% and 33.9%. These results support extending agent control beyond query reformulation to the processing and presentation of retrieved evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Search, LLM Agents, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KKU8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Programmatic Search Agents (PSA), which use executable code to compose search operations, maintain persistent candidate workspaces, and selectively present evidence. Experiments on two benchmarks with five LLMs show improvements in task success rates and final-step token efficiency.

Critical strengths
  1. The paper provides a controlled comparison of query-based, tool-based, and code-based search agents across five LLMs. By sharing the same underlying retrieval components, the experiments help isolate the impact of different search interfaces on agent performance.
  2. PSA demonstrates improvements in task success rates and final-step token efficiency across two benchmarks. These results provide useful evidence that programmatic control over search operations can benefit complex information-seeking tasks.
Critical weaknesses
  1. The paper’s motivation and methodological novelty are not convincing. The authors argue that agents should control how retrieved evidence is processed and presented, but modern search agents already support tools such as visit and fetch for selectively accessing retrieved content. More importantly, Perplexity’s Search as Code (SaC) has already introduced a highly similar approach. Although the authors acknowledge SaC as an inspiration, they do not clearly explain the technical differences between the two approaches. As a result, I find it difficult to identify a substantial methodological contribution.
  2. The empirical contribution is also insufficient. While PSA outperforms the selected baselines, the experiments do not adequately explain why it performs better. The Query-based Agent baseline lacks common capabilities such as visit/fetch, making the comparison less convincing. The ablations also fail to clearly isolate the contributions of individual components. Moreover, the evaluation is limited to two fixed-corpus benchmarks, without testing on open-web benchmarks such as BrowseComp or WideSearch. The implementation also does not appear to be publicly available, raising reproducibility concerns. Overall, I don’t think the experimental evidence is strong enough to compensate for the limited methodological novelty.
Questions for authors
  1. What are the key technical differences between PSA and Perplexity’s Search as Code?
  2. Why was PSA not evaluated against existing search agents equipped with visit or fetch tools? Would the reported improvements still hold against these stronger baselines?
Reviewer_sfr1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Programmatic Search Agent (PSA), a framework that extends agentic search beyond query reformulation by allowing agents to compose retrieval, filtering, reranking, and evidence extraction operations through executable programs. PSA maintains a persistent candidate workspace and supports selective evidence presentation to reduce unnecessary context consumption. Experiments across multiple LLM backbones and benchmarks demonstrate improvements in task success rates and final-step token efficiency.

Critical strengths
  1. The paper addresses a meaningful limitation of existing agentic search interfaces. The paper identifies the limitations of query-based search and provides a flexible interface for agents to control evidence retrieval, processing, and presentation.

  2. The experimental results demonstrate consistent improvements across different models and benchmarks. The evaluation covers five LLM backbones and multiple benchmarks, showing consistent improvements in task success rates. The comparison with tool-based agents also provides a useful baseline.

Critical weaknesses
  1. Limited novelty beyond existing programmatic search frameworks.

PSA shares substantial similarities with existing approaches such as Perplexity's Search as Code, which also allows agents to compose search operations through executable programs. The proposed framework appears to primarily integrate existing mechanisms, including programmatic tool composition, persistent workspaces, and selective evidence presentation. The technical novelty beyond these existing approaches is not sufficiently established.

  1. Potentially unfair computational budget comparisons.

PSA can execute multiple dependent retrieval operations within a single program cell, whereas the Tool-based Agent requires separate decisions for these operations. Consequently, matching the number of agent decisions does not guarantee comparable computational budgets. Moreover, the reported final-step token efficiency does not fully reflect end-to-end computational costs.

  1. Insufficient ablations to isolate the contributions of individual components.

PSA combines programmatic composition, persistent candidate reuse, and selective evidence presentation, but their individual contributions are not sufficiently isolated. Although the Tool-based Agent shares some underlying primitives and workspace functionality, the comparison does not clearly establish which components account for the observed improvements.

Questions for authors
  1. What are the key technical differences between PSA and existing programmatic search approaches, particularly Perplexity's Search as Code? Which capabilities or mechanisms constitute the primary methodological novelty?

  2. Could the authors provide comparisons under matched primitive-call or total computational budgets, including total LLM tokens, retrieval operations, latency, and estimated monetary costs, to establish whether PSA retains its advantages?

  3. Could the authors provide component-level ablations that independently evaluate the contributions of programmatic composition, persistent candidate reuse, and selective evidence presentation to task success and efficiency?

Reviewer_XpCROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

PSA allows models to combine retrieval, filtering, reranking, extraction, and deduplication within executable program cells. Candidate results remain in a persistent workspace, while the model selects which evidence enters its context. Experiments across five models and two benchmarks show higher task success rates and fewer final-step tokens. In a controlled experiment, selective evidence presentation reduced returned text by 37.2% while maintaining or improving accuracy.

Critical strengths
  1. Program cells combine multiple operations, reducing intermediate results in the context and the interactions needed for separate tool calls.
  2. The authors find that 42.7% of retrieved supporting passages never reach the model. Supplying omitted evidence improves task success, directly motivating the proposed approach.
Critical weaknesses

Final-step token counts do not capture total cost. Code generation, execution, retries, and all model interactions should also be included.

Questions for authors

Have you considered fine-tuning or reinforcement learning on successful trajectories to improve operation composition and evidence selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.