- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PathSeek, an inference-time trajectory optimization layer for research agents. It represents executed trajectory prefixes as search-tree nodes and bounded sequences of reasoning, action, and observation as Agent Events. A harness adapter enables event execution across different model–harness combinations. Search is guided by two evaluation modules: Claim–Evidence Auditing, which assesses support for intermediate claims and their dependencies, and Middle-Candidate Verification, which evaluates task relevance and overall trajectory quality. Their scores and corrective feedback influence branch expansion and final selection. Experiments cover five models, three harnesses, and four benchmark sets. The paper reports improvements in the aggregate score for all 15 model–harness combinations, with occasional individual-task regressions. Component ablations support the usefulness of both evaluation modules.
Critical strengths
-
The framework offers a useful integration of executable trajectory search and evidence-based feedback. Evaluating bounded, actually executed prefixes avoids requiring a complete rollout for every branch. Separating claim-level evidential support from task-level adequacy is also sensible: a trajectory can contain supported claims while still failing to answer the question.
-
The breadth of model–harness evaluation supports practical applicability. Table 1 evaluates 15 combinations, rather than demonstrating the method only with one executor. Improvements are also visible in exact-match results, not exclusively LLM-judge scores. For example, DeepSeek-V4 Flash with DeepSeek Harness improves SGI-DR exact match from 21.6 to 43.1. Table 2 further shows that removing either evaluation module reduces performance.
Critical weaknesses
1.The experiments do not sufficiently isolate the method’s benefit from additional inference-time computation. The principal baselines are the original harnesses without PathSeek. Resource accounting shows substantial token increases in several settings, while the variant without both evaluation modules changes branch selection and final selection to random selection. As the paper acknowledges, this is not an equal-compute comparison. A budget-matched best-of-, conventional trajectory-search, or verifier-only baseline would better establish the incremental value of the proposed event-level evidence-guided search.
2.Verifier reliability and evaluation independence are insufficiently characterized. GPT-OSS-120B is used both for online verification and benchmark judging. This creates a risk of correlated evaluation preferences, even though online verification does not access reference answers. The exact-match improvements provide useful independent support, but most benchmark outcomes still rely on LLM judging. The paper also lacks uncertainty estimates adequate to distinguish small improvements from run or judge variability.
Questions for authors
1.Budget clarification: What search limits and stopping criteria are used for each model–harness combination, and which limits are shared with the corresponding baseline? Do you have budget-matched best-of- or conventional search comparisons?
2.Verifier validation: How well do the two verification scores predict independently assessed final-answer quality? Do the improvements persist with a different benchmark judge or a blinded human audit of a representative subset?
3.Component clarification: Can existing results separate the effects of numerical branch scoring, corrective feedback, and final-candidate selection? The current module-removal ablations measure these effects jointly.
Compute Comparisons, Verifier Robustness, and Component Analysis
Response:
DS=DeepSeek-V4-Flash; Q35=Qwen3.5-35B-A3B; Q9=Qwen3.5-9B; DSH=DeepSeek Harness; PS=PathSeek.
Response 1
We added BoN-2, LATS, Reflexion (Shinn et al., 2023), and Self-Refine (Madaan et al., 2023). BoN-2 independently generates two answers with the same backbone/harness and uses the same GPT-OSS-120B final verifier. PathSeek exceeds BoN-2 in all four configurations:
LATS/Reflexion/Self-Refine averages are 33.18/36.07/33.81 for DS and 31.58/18.39/23.52 for Q35, below corresponding PathSeek configurations.
Including verifier calls, DS PathSeek uses 18.7%/28.7% fewer recorded tokens and has 21.1%/33.5% lower costs than BoN-2 with DSH/Codex:
These results support the integrated search/verification design beyond reranking. They establish measured resource advantages; they do not establish identical cumulative-token caps or isolate an individual module’s contribution.
Response 2
On 50 challenging SGI-DR questions, we held the questions, search budget, and benchmark judge fixed and replaced the turn, final, or both verifiers with Q9 or DS:
Current Raw: selected-answer accuracy, including API and other non-framework failures as errors. Current Clean: accuracy after excluding these errors. Oracle Raw: fraction of questions with any candidate judged correct.
Heterogeneous verifiers achieve 28%–32% Current Raw versus 26% originally and 22% without verification. These point estimates provide preliminary robustness evidence. With the benchmark judge fixed, evaluator independence remains unestablished.
Response 3
Verification raises Oracle Raw from 36% to 42% and Current Raw from 22% to 26%. With DS as final verifier, coverage stays at 42% while accuracy reaches 32%, narrowing the oracle–selection gap from 16 to 10 points. Turn-verifier replacements achieve 32% accuracy with 38%/40% coverage, suggesting changes in candidate composition and selection. Numerical branch scores and corrective feedback jointly guide online search; final-verifier results support a distinct selection role.
Response 4
Expanded evaluation shows improvements on most datasets, supporting the broader pattern of gains: