acceptodds
Desk rejected at ICLR 2027

PATHSEEK: MODEL- AND HARNESS-AGNOSTIC AGENTIC TRAJECTORY OPTIMIZATION

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Continued advances in models and agent execution frameworks (harness) have enhanced the capabilities of research agents, yet early-stage trajectory deviations and unsupported intermediate claims can still undermine research quality. We propose PathSeek, which introduces trajectory optimization as a capability-enhancement mechanism that is orthogonal to, and can be combined with, model enhancement and harness improvement, further extending the boundaries of agent capabilities. PathSeek encapsulates one or more consecutive rounds of Reasoning–Action–Observation interaction into a bounded Agent Event, uses executed trajectory prefixes as tree nodes and event-level continuations as search edges, and connects different model–harness combinations through a unified interface. To evaluate unfinished research trajectories, Claim–Evidence Auditing examines the evidence supporting key claims and the dependencies among them, while Middle-Candidate Verification assesses task relevance and overall quality. Together, they provide process-value assessments and corrective feedback at event boundaries to guide branch selection, evidence supplementation, and trajectory adjustment. Across four sets of Deep Research and scientific reasoning benchmarks and multiple combinations of five representative models and three harnesses, PathSeek achieves broad performance improvements and a score of 71.4 on SGI-Bench DeepResearch, demonstrating that our method can more effectively translate existing model capabilities and harnesses into high-quality agent research trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agent System, Trajectory Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZuDFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PathSeek, an inference-time trajectory optimization layer for research agents. It represents executed trajectory prefixes as search-tree nodes and bounded sequences of reasoning, action, and observation as Agent Events. A harness adapter enables event execution across different model–harness combinations. Search is guided by two evaluation modules: Claim–Evidence Auditing, which assesses support for intermediate claims and their dependencies, and Middle-Candidate Verification, which evaluates task relevance and overall trajectory quality. Their scores and corrective feedback influence branch expansion and final selection. Experiments cover five models, three harnesses, and four benchmark sets. The paper reports improvements in the aggregate score for all 15 model–harness combinations, with occasional individual-task regressions. Component ablations support the usefulness of both evaluation modules.

Critical strengths
  1. The framework offers a useful integration of executable trajectory search and evidence-based feedback. Evaluating bounded, actually executed prefixes avoids requiring a complete rollout for every branch. Separating claim-level evidential support from task-level adequacy is also sensible: a trajectory can contain supported claims while still failing to answer the question.

  2. The breadth of model–harness evaluation supports practical applicability. Table 1 evaluates 15 combinations, rather than demonstrating the method only with one executor. Improvements are also visible in exact-match results, not exclusively LLM-judge scores. For example, DeepSeek-V4 Flash with DeepSeek Harness improves SGI-DR exact match from 21.6 to 43.1. Table 2 further shows that removing either evaluation module reduces performance.

Critical weaknesses

1.The experiments do not sufficiently isolate the method’s benefit from additional inference-time computation. The principal baselines are the original harnesses without PathSeek. Resource accounting shows substantial token increases in several settings, while the variant without both evaluation modules changes branch selection and final selection to random selection. As the paper acknowledges, this is not an equal-compute comparison. A budget-matched best-of-NN, conventional trajectory-search, or verifier-only baseline would better establish the incremental value of the proposed event-level evidence-guided search.

2.Verifier reliability and evaluation independence are insufficiently characterized. GPT-OSS-120B is used both for online verification and benchmark judging. This creates a risk of correlated evaluation preferences, even though online verification does not access reference answers. The exact-match improvements provide useful independent support, but most benchmark outcomes still rely on LLM judging. The paper also lacks uncertainty estimates adequate to distinguish small improvements from run or judge variability.

Questions for authors

1.Budget clarification: What search limits and stopping criteria are used for each model–harness combination, and which limits are shared with the corresponding baseline? Do you have budget-matched best-of-NN or conventional search comparisons?

2.Verifier validation: How well do the two verification scores predict independently assessed final-answer quality? Do the improvements persist with a different benchmark judge or a blinded human audit of a representative subset?

3.Component clarification: Can existing results separate the effects of numerical branch scoring, corrective feedback, and final-candidate selection? The current module-removal ablations measure these effects jointly.

1 comment
AuthorsOpenReview ↗

Compute Comparisons, Verifier Robustness, and Component Analysis

Response:

DS=DeepSeek-V4-Flash; Q35=Qwen3.5-35B-A3B; Q9=Qwen3.5-9B; DSH=DeepSeek Harness; PS=PathSeek.

Response 1

We added BoN-2, LATS, Reflexion (Shinn et al., 2023), and Self-Refine (Madaan et al., 2023). BoN-2 independently generates two answers with the same backbone/harness and uses the same GPT-OSS-120B final verifier. PathSeek exceeds BoN-2 in all four configurations:

ModelHarnessBoN-2PSGain
DSDSH44.1557.7513.60
Q35DSH31.7538.236.48
DSCodex51.5355.733.20
Q35Codex33.1340.237.10

LATS/Reflexion/Self-Refine averages are 33.18/36.07/33.81 for DS and 31.58/18.39/23.52 for Q35, below corresponding PathSeek configurations.

Including verifier calls, DS PathSeek uses 18.7%/28.7% fewer recorded tokens and has 21.1%/33.5% lower costs than BoN-2 with DSH/Codex:

HarnessMethodRollout Tokens (K)Verifier Tokens (K)Tokens (K)Cost ($)Time (s)Tool Use (Num.)
DSHVanilla492.00.0492.00.1171206.914.7
DSHPS556.151.5607.60.1412297.324.2
DSHBoN-2744.62.5747.10.1790723.532.6
CodexVanilla830.40.0830.40.1918291.622.4
CodexPS736.4167.0903.40.1953356.626.3
CodexBoN-21263.53.81267.30.2936731.039.6
MCTSLATS105.878.4184.20.0522156.49.7
ReflexionReflexion147.02.0148.90.0472796.07.6
ReflexionSelf-Refine17.22.920.10.0107420.50.0

These results support the integrated search/verification design beyond reranking. They establish measured resource advantages; they do not establish identical cumulative-token caps or isolate an individual module’s contribution.

Response 2

On 50 challenging SGI-DR questions, we held the questions, search budget, and benchmark judge fixed and replaced the turn, final, or both verifiers with Q9 or DS:

Verifier ConfigurationCurrent RawCurrent CleanOracle Raw
Baseline (Both GPT-OSS-120B)26.028.042.0
Weak Final Verifier (Q9)28.032.038.0
Strong Final Verifier (DS)32.034.042.0
Weak Turn Verifier (Q9)32.036.038.0
Strong Turn Verifier (DS)32.032.040.0
Weak All Verifier (Q9)28.030.044.0
Strong All Verifier (DS)28.030.038.0
No Verifier22.024.036.0

Current Raw: selected-answer accuracy, including API and other non-framework failures as errors. Current Clean: accuracy after excluding these errors. Oracle Raw: fraction of questions with any candidate judged correct.

Heterogeneous verifiers achieve 28%–32% Current Raw versus 26% originally and 22% without verification. These point estimates provide preliminary robustness evidence. With the benchmark judge fixed, evaluator independence remains unestablished.

Response 3

Verification raises Oracle Raw from 36% to 42% and Current Raw from 22% to 26%. With DS as final verifier, coverage stays at 42% while accuracy reaches 32%, narrowing the oracle–selection gap from 16 to 10 points. Turn-verifier replacements achieve 32% accuracy with 38%/40% coverage, suggesting changes in candidate composition and selection. Numerical branch scores and corrective feedback jointly guide online search; final-verifier results support a distinct selection role.

Response 4

Expanded evaluation shows improvements on most datasets, supporting the broader pattern of gains:

Model / ConfigurationHLE (subset)OmniMatBench CALOmniMatBench QAFrontierMathSUPERChemAVG.
ExoMind Harness
ExoMind41.08.039.033.329.030.1
ExoMind+PS49.012.439.641.731.334.8
Kimi-K2.647.017.049.050.057.044.0
Kimi-K2.6+PS45.018.746.058.359.145.4
DS52.09.042.025.036.032.8
DS+PS53.010.045.025.035.033.6
Q3537.012.037.016.725.025.5
Q35+PS45.09.038.025.027.028.8
Q941.010.033.00.019.020.6
Q9+PS43.011.236.00.021.022.2
Codex
ExoMind38.010.039.866.739.938.9
ExoMind+PS64.017.557.075.059.054.5
Kimi-K2.640.018.649.941.766.343.3
Kimi-K2.6+PS65.020.053.075.070.056.6
DS45.013.349.650.055.642.7
DS+PS55.015.052.041.758.044.3
Q3513.010.641.58.311.116.9
Q35+PS34.018.042.08.321.524.8
Q97.09.433.00.08.811.6
Q9+PS20.012.035.00.019.117.2
DSH
ExoMind58.010.046.550.019.636.8
ExoMind+PS65.018.358.050.021.342.5
Kimi-K2.655.015.848.033.334.937.4
Kimi-K2.6+PS57.020.052.083.372.056.9
DS51.013.652.141.750.641.8
DS+PS57.013.353.041.759.644.9
Q3518.012.241.616.713.320.4
Q35+PS29.012.843.616.719.224.2
Q912.010.341.50.011.115.0
Q9+PS25.013.139.50.018.019.1
Reviewer_xfQeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PathSeek is a test-time tree search wrapper for research agents.

A node is an executed trajectory prefix, and an edge is an 'Agent Event', one or more reasoning, action and observation rounds run by the original harness through an adapter, so the model and harness stay untouched.

Unfinished prefixes are scored by Claim-Evidence Auditing, which checks whether key claims have evidence and how they depend on each other, and by Middle-Candidate Verification, a holistic LLM check. These scores guide MCTS-style branch selection and feed corrective feedback into new branches.

The evaluation covers five models and three harnesses (ExoMind, DeepSeek Harness, Codex CLI) on four science and deep research benchmarks.

The four-benchmark mean improves for all 15 combinations, by about 6 points on average, and the headline is 71.4 on SGI-Bench Deep Research. The breadth is impressive, but the only baseline is a single plain harness run, so it's hard to tell how much of the gain comes from evidence-guided search itself. The search idea is also close to existing agent tree search work.

Critical strengths
  1. Broad and systematic evaluation. Gains hold across 15 model and harness combinations, including two third-party black-box harnesses, and on SGI-DR they also show up in judge-free exact match (about +7 points on 318 questions). This matters because most agent search papers test one framework, and the exact match gains don't depend on the LLM judge.

  2. A practical wrapper design. Search runs over actually executed prefixes and leaves planning, tools and context to the original harness, so it can sit on top of closed harnesses. The paper is also fairly honest about its caveats. Practitioners could use this without rebuilding their agent stack.

Critical weaknesses
  1. There are no compute-matched test-time scaling baselines, and the paper's own evidence suggests that selection and extra turns, not evidence-guided tree search, drive the gains. The only baseline is one plain harness run, while PathSeek uses up to about 1.75x the tokens, and Sec. 4.3.1 admits there's no equal-compute comparison. In Table 2, removing both modules (random final selection) falls well below the baseline, and each module alone lands near the baseline on the LLM-judge and PRL scores. On the DeepSeek and Codex harnesses the output path is only about two rounds, and the case studies look like best-of-2 or a linear chain. Best-of-N with the same verifier, self-refine, LATS-style search, or 'baseline plus one continuation plus the same finalizer' are all missing. Without them, the central claim can't be separated from 'more compute plus a strong selector'.

  2. Evaluation validity and novelty are both weaker than the framing. GPT-OSS-120B is both the in-loop verifier and the benchmark judge, every result is a single run, and many FS and PRL changes are within noise on 60 to 100 items. The headline 71.4 is a self-defined LLM-judge score that starts from that model's weakest baseline (exact match there is 43.1). On novelty, MCTS over executed agent states with an LLM value and language feedback already exists (LATS, Koh et al. 2024, SWE-Search), and Claim-Evidence Auditing closely mirrors the DRIFT auditor (claim ledger, direct/weak/missing/conflicting support, dependency tracing), which the paper cites only as motivation. This matters because the size and source of the gains are uncertain, and the new part is mostly engineering breadth.

Questions for authors
  1. For at least two configurations, can you report best-of-N over independent full harness runs, selected by the same GPT-OSS-120B verifier, plus a critique-and-revise loop, at a matched token or wall-clock budget? Please also add 'baseline plus one continuation plus the same finalizer'. This would most change my assessment.
1 comment
AuthorsOpenReview ↗

Baseline Comparisons, Search Contributions, and Method Attribution

Response:

DS=DeepSeek-V4-Flash; Q35=Qwen3.5-35B-A3B; Q9=Qwen3.5-9B; DSH=DeepSeek Harness; PS=PathSeek.

Response 1

We added BoN-2, Self-Refine (Madaan et al., 2023), LATS, and Reflexion (Shinn et al., 2023) across two backbones and four benchmarks. BoN-2 generates two independent answers with the same backbone/harness and uses the same GPT-OSS-120B final verifier:

Model / ConfigurationSGI-DR ExactSGI-DR LLMPRLFS-OFS-RAVG.
DSH
DeepSeek-V4-Flash21.642.532.270.054.049.7
Qwen3.5-35B-A3B16.029.625.938.037.432.7
PS (DSH)
DeepSeek-V4-Flash43.171.435.370.054.357.8
Qwen3.5-35B-A3B23.942.829.841.039.338.2
BoN-2 (DSH)
DeepSeek-V4-Flash22.639.034.355.048.344.2
Qwen3.5-35B-A3B23.639.618.640.028.831.8
Codex
DeepSeek-V4-Flash27.450.941.169.052.653.4
Qwen3.5-35B-A3B25.544.325.437.023.132.5
PS (Codex)
DeepSeek-V4-Flash31.853.441.472.056.155.7
Qwen3.5-35B-A3B29.650.325.745.539.440.2
BoN-2 (Codex)
DeepSeek-V4-Flash26.751.240.365.053.652.5
Qwen3.5-35B-A3B25.740.221.043.028.333.1
Model / ConfigurationSGI-DR ExactSGI-DR LLMPRLFS-OFS-RAVG.
Vanilla Model
DeepSeek-V4-Flash11.324.519.377.08.332.3
Qwen3.5-35B-A3B9.414.513.942.52.518.3
Reflexion
DeepSeek-V4-Flash19.539.018.557.029.836.1
Qwen3.5-35B-A3B11.322.05.336.010.318.4
Self-Refine
DeepSeek-V4-Flash14.525.821.060.028.533.8
Qwen3.5-35B-A3B13.225.212.941.015.023.5
LATS
DeepSeek-V4-Flash19.236.58.460.027.833.2
Qwen3.5-35B-A3B21.737.719.648.021.331.7

PathSeek exceeds BoN-2 in all four configurations and the other added baselines with both backbones. Averages use SGI-DR LLM, PRL, FS-O, and FS-R:

ModelHarnessBoN-2PSGain
DSDSH44.1557.7513.60
Q35DSH31.7538.236.48
DSCodex51.5355.733.20
Q35Codex33.1340.237.10

Response 2

Including verifier overhead, DS PathSeek uses 18.7%/28.7% fewer recorded tokens than BoN-2 with DSH/Codex. Resource comparisons do not establish identical token caps.

HarnessMethodRollout Tokens (K)Verifier Tokens (K)Tokens (K)Cost ($)Time (s)Tool Use (Num.)
DSHVanilla492.00.0492.00.1171206.914.7
DSHPS556.151.5607.60.1412297.324.2
DSHBoN-2744.62.5747.10.1790723.532.6
CodexVanilla830.40.0830.40.1918291.622.4
CodexPS736.4167.0903.40.1953356.626.3
CodexBoN-21263.53.81267.30.2936731.039.6
MCTSLATS105.878.4184.20.0522156.49.7
ReflexionReflexion147.02.0148.90.0472796.07.6
ReflexionSelf-Refine17.22.920.10.0107420.50.0

A two-round retained path omits evaluated/pruned branches. Random-selection degradation confirms that selection matters; the combined modules’ gains suggest complementarity. Added baselines support the integrated design without attributing all gains to evidence-guided search.

Response 3

Claim–Evidence Auditing adapts DRIFT’s diagnostics; we will clarify this attribution. DRIFT diagnoses static claim–evidence correspondence. PathSeek guides dynamic expansion/pruning: an unsupported claim becomes a next-event verification task, and new tool results trigger reassessment. Event-level prefix evaluation, complementary trajectory signals, and score/execution feedback address incomplete evidence and distinguish reliability from task relevance.

Response 4

Gains remain modest in some settings, but expanded evaluation shows improvements on most datasets:

Model / ConfigurationHLE (subset)OmniMatBench CALOmniMatBench QAFrontierMathSUPERChem
ExoMind Harness
ExoMind41.08.039.033.329.0
ExoMind+PS49.012.439.641.731.3
Kimi-K2.647.017.049.050.057.0
Kimi-K2.6+PS45.018.746.058.359.1
DS52.09.042.025.036.0
DS+PS53.010.045.025.035.0
Q3537.012.037.016.725.0
Q35+PS45.09.038.025.027.0
Q941.010.033.00.019.0
Q9+PS43.011.236.00.021.0
Codex
ExoMind38.010.039.866.739.9
ExoMind+PS64.017.557.075.059.0
Kimi-K2.640.018.649.941.766.3
Kimi-K2.6+PS65.020.053.075.070.0
DS45.013.349.650.055.6
DS+PS55.015.052.041.758.0
Q3513.010.641.58.311.1
Q35+PS34.018.042.08.321.5
Q97.09.433.00.08.8
Q9+PS20.012.035.00.019.1
DSH
ExoMind58.010.046.550.019.6
ExoMind+PS65.018.358.050.021.3
Kimi-K2.655.015.848.033.334.9
Kimi-K2.6+PS57.020.052.083.372.0
DS51.013.652.141.750.6
DS+PS57.013.353.041.759.6
Q3518.012.241.616.713.3
Q35+PS29.012.843.616.719.2
Q912.010.341.50.011.1
Q9+PS25.013.139.50.018.0
Reviewer_a4LEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies an interesting and practical problem in research agents: how to improve long-horizon reasoning trajectories without modifying the underlying language model or agent harness. The authors propose PathSeek, a trajectory optimization framework that treats intermediate reasoning steps as searchable paths and uses evidence-based verification to guide exploration and correction. In particular, the proposed Claim–Evidence Auditing and Middle-Candidate Verification modules help evaluate unfinished trajectories and detect unsupported claims during the reasoning process. Experiments across multiple models, harnesses, and scientific reasoning benchmarks demonstrate consistent improvements, suggesting that trajectory optimization is a promising direction complementary to model scaling and harness engineering. While the approach introduces additional inference cost and depends on verifier quality, the paper is well-motivated and provides strong empirical evidence for its effectiveness.

Critical strengths
  1. The authors study an interesting and meaningful problem of improving long-horizon agent reasoning trajectories for deepresearch issues. Their proposed PathSeek wrap heterogeneous harnesses without modifying their internal planning, tool use, or context management. It requires no training, so it can be deployed on top of existing systems.

  2. The authors conduct extensive experiments to verify the effectiveness of their method. The evaluation spans 15 model–harness combinations and four science-oriented benchmarks covering several model families and scales, including dense and MoE models.

  3. The writing and logic of this paper is clear and easy to follow. The authors provide enough related works to demonstrate background and their motivations.

Critical weaknesses
  1. I find thart gains shrink as the base model gets stronger on ExoMind Harness. It's better for authors to discuss the reason behind this phenomenon for it's really important to consider this approach for different harnesses.

  2. The cost analysis is incomplete. Calls and rounds count only the retained path rather than the whole search tree, and dollar costs are estimated rather than measured. The search budget, nodes expanded per question, latency, and gain-vs-budget curves are not reported.

  3. The experiments include no external test-time baselines and no compute-matched comparison. Every result is compared only against the same model–harness without PathSeek. The only internal control, random selection in Table 2, falls below the vanilla baseline. Meanwhile, PathSeek uses substantially more compute plus a separate 120B verifier, and the authors acknowledge that their comparisons are not equal-compute. Since the ablation indicates that the gains come mainly from verifier-based selection, the most important missing baseline is Best-of-N or self-consistency with the same GPT-OSS-120B verifier at a matched token budget. Like PathSeek, it is model- and harness-agnostic. Rollout-based MCTS or LATS, which Figure 2 contrasts with PathSeek but never evaluates, and a Reflexion-style revision baseline should also be included. Without these, it is unclear whether event-level tree search and claim–evidence auditing add value beyond simple verifier-based reranking.

Questions for authors

Please refer to weaknesses part.

1 comment
AuthorsOpenReview ↗

Model Dependence, Resource Accounting, and Additional Baselines

Response:

DS=DeepSeek-V4-Flash; Q35=Qwen3.5-35B-A3B; Q9=Qwen3.5-9B; DSH=DeepSeek Harness; PS=PathSeek.

Response 1

Thank you for highlighting this pattern. Stronger ExoMind Harness baselines may leave fewer errors recoverable through retrieval, evidence collection, recalculation, or alternative research directions. Their effective use of the harness may also overlap with PathSeek’s checks and feedback. These are plausible explanations, not established causes. Gains depend on the model–harness combination: DS improves by about 0.9 points with ExoMind Harness and 8.1 with DSH. Model- and harness-agnostic describes applicability through a unified interface; it does not imply uniform gains. We will clarify these mechanisms and limitations.

Response 2

Retained-path calls and rounds understate full-search cost. The following measurements aggregate all executed branches, verification calls, and finalization on matched questions within each harness, including unsuccessful runs:

HarnessMethodRollout Tokens (K)Verifier Tokens (K)Tokens (K)Cost ($)Time (s)Tool Use (Num.)
DSHVanilla492.00.0492.00.1171206.914.7
DSHPS556.151.5607.60.1412297.324.2
DSHBoN-2744.62.5747.10.1790723.532.6
CodexVanilla830.40.0830.40.1918291.622.4
CodexPS736.4167.0903.40.1953356.626.3
CodexBoN-21263.53.81267.30.2936731.039.6
MCTSLATS105.878.4184.20.0522156.49.7
ReflexionReflexion147.02.0148.90.0472796.07.6
ReflexionSelf-Refine17.22.920.10.0107420.50.0

With a three-node budget, maximum depth 3, branch width 2, and a 900-second question limit, PathSeek executes 2.82 research segments per question with DSH and 2.86 with Codex. Mean latency rises from 206.9 to 297.3 seconds and from 291.6 to 356.6 seconds, respectively.

Response 3

We added BoN-2, LATS, Reflexion (Shinn et al., 2023), and Self-Refine (Madaan et al., 2023) across two backbones and four benchmarks. BoN-2 generates two independent candidates with the same backbone and harness, then uses the same GPT-OSS-120B final verifier as PathSeek:

Model / ConfigurationSGI-DR ExactSGI-DR LLMPRLFS-OFS-RAVG.
DSH
DeepSeek-V4-Flash21.642.532.270.054.049.7
Qwen3.5-35B-A3B16.029.625.938.037.432.7
PS (DSH)
DeepSeek-V4-Flash43.171.435.370.054.357.8
Qwen3.5-35B-A3B23.942.829.841.039.338.2
BoN-2 (DSH)
DeepSeek-V4-Flash22.639.034.355.048.344.2
Qwen3.5-35B-A3B23.639.618.640.028.831.8
Codex
DeepSeek-V4-Flash27.450.941.169.052.653.4
Qwen3.5-35B-A3B25.544.325.437.023.132.5
PS (Codex)
DeepSeek-V4-Flash31.853.441.472.056.155.7
Qwen3.5-35B-A3B29.650.325.745.539.440.2
BoN-2 (Codex)
DeepSeek-V4-Flash26.751.240.365.053.652.5
Qwen3.5-35B-A3B25.740.221.043.028.333.1
Model / ConfigurationSGI-DR ExactSGI-DR LLMPRLFS-OFS-RAVG.
Vanilla Model
DeepSeek-V4-Flash11.324.519.377.08.332.3
Qwen3.5-35B-A3B9.414.513.942.52.518.3
Reflexion
DeepSeek-V4-Flash19.539.018.557.029.836.1
Qwen3.5-35B-A3B11.322.05.336.010.318.4
Self-Refine
DeepSeek-V4-Flash14.525.821.060.028.533.8
Qwen3.5-35B-A3B13.225.212.941.015.023.5
LATS
DeepSeek-V4-Flash19.236.58.460.027.833.2
Qwen3.5-35B-A3B21.737.719.648.021.331.7

PathSeek outperforms BoN-2 in all four configurations. The following averages use SGI-DR LLM, PRL, FS-O, and FS-R with equal weights:

ModelHarnessBoN-2PSGain
DSDSH44.1557.7513.60
Q35DSH31.7538.236.48
DSCodex51.5355.733.20
Q35Codex33.1340.237.10

Including verifier overhead, PathSeek uses 18.7%/28.7% fewer recorded tokens and has 21.1%/33.5% lower estimated costs than BoN-2 with DSH/Codex, respectively. These are measured resource comparisons; identical cumulative-token caps were not enforced.

LATS/Reflexion/Self-Refine averages are 33.18/36.07/33.81 for DS and 31.58/18.39/23.52 for Q35, all below the corresponding PathSeek configurations. These comparisons support the integrated design beyond independent sampling and reranking: intermediate verification guides further evidence collection and branch expansion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional Experiments and Clarifications

Dear Area Chair and Reviewers,

We sincerely thank you for your time, careful reading, and constructive feedback. Your suggestions helped us strengthen the evaluation and clarify the scope of our claims. We added four sets of experiments and measurements, summarized below.

  1. Additional test-time baselines. We added BoN-2, LATS, Reflexion, and Self-Refine using two backbones across SGI-DR, PRL, FS-O, and FS-R. These cover independent sampling, conventional search, and iterative revision.

    BoN-2 runs two independent native harness instances with the same backbone, harness, and GPT-OSS-120B final verifier as PathSeek:

    Model / HarnessOriginal AVG.BoN-2 AVG.PathSeek AVG.
    DeepSeek-V4-Flash / DSH49.744.257.8
    Qwen3.5-35B-A3B / DSH32.731.838.2
    DeepSeek-V4-Flash / Codex53.452.555.7
    Qwen3.5-35B-A3B / Codex32.533.140.2

    DSH=DeepSeek Harness. AVG. equally weights SGI-DR LLM, PRL, FS-O, and FS-R; Exact is reported separately.

    Finding: PathSeek exceeds both the original harness and BoN-2 in average score in all four configurations. DeepSeek/DSH gains 13.60 points over BoN-2 (44.15 to 57.75 before rounding). PathSeek also exceeds the added revision and LATS baselines in average score with both backbones. These comparisons reflect performance under the reported execution protocols.

    Completion-rate clarification: The Original scores were obtained after rerunning questions as needed until every question had a valid output. Some BoN-2 evaluations failed to return a valid answer within the allotted time and were counted as failures. This difference can contribute to lower or more variable BoN-2 scores; the comparison therefore reflects both answer quality and completion reliability.

  2. Full-search resource accounting. We measured executed branches, verification calls, and finalization, including unsuccessful runs on matched questions within each harness. For DeepSeek, PathSeek uses 607.6K versus 747.1K total tokens for BoN-2 with DSH, and 903.4K versus 1267.3K with Codex. These are 18.7% and 28.7% reductions, including verifier overhead. Reported cost estimates are also 21.1% and 33.5% lower.

    Interpretation: Higher average scores accompany lower recorded resource use than BoN-2. In this resource setting, PathSeek uses a three-node budget, maximum depth 3, branch width 2, and a 900-second question limit; BoN-2 allows 900 seconds per candidate plus final verification. These compare observed resource use, with different total budget caps. Retained-path length excludes evaluated alternatives and does not describe the entire search.

  3. Heterogeneous-verifier ablations. On 50 challenging SGI-DR questions, we replaced the turn verifier, final verifier, or both with Qwen3.5-9B or DeepSeek-V4-Flash, holding the questions, search budget, and benchmark judge fixed. Raw selected-answer accuracy is 28.0-32.0%, versus 26.0% with GPT-OSS-120B for both roles and 22.0% without verification. Raw accuracy counts judge API failures as incorrect. Replacing both verifiers gives 28.0% with either alternative model.

    Finding: The point estimates provide preliminary evidence of robustness to verifier identity. With DeepSeek as final verifier, selected accuracy rises from 26.0% to 32.0% while aggregate oracle coverage (questions with any candidate judged correct) stays at 42.0%, consistent with a final-selection benefit at unchanged coverage. These complement the original module ablations, which measure scores and corrective feedback jointly. The benchmark judge remains fixed, so this is not an independent-judge evaluation or blind human audit.

  4. Broader benchmark coverage. We added HLE (subset), OmniMatBench CAL, OmniMatBench QA, FrontierMath, and SUPERChem evaluations across five backbones and three harnesses. Of 75 model-harness-benchmark comparisons, PathSeek improves 60, ties 8, and declines on 7. ExoMind/Codex improves from 38.0 to 64.0 on HLE and from 39.9 to 59.0 on SUPERChem. We retain regressions, including DeepSeek/Codex on FrontierMath (50.0 to 41.7).

    Interpretation: The broader results support applicability across settings, with gains depending on the model, harness, and task. We do not claim statistical significance for small improvements.

We also clarify contribution and attribution: Claim-Evidence Auditing adapts DRIFT's diagnostics, and PathSeek integrates them with evaluation of unfinished prefixes and evidence-directed continuation across harnesses. We appreciate your guidance in strengthening the evidence and defining its scope. Detailed tables and analyses are provided in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.