- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies a failure mode of LLM search agents that it calls “lost in search”: errors made in early retrieval steps propagate through later steps instead of being corrected. The authors attribute this to dependencies between steps being tracked only implicitly. SearchMapper has the agent propose sub questions in layers with explicit dependency annotations, which forms a DAG. Two training time mechanisms build on this structure:
- Scoring model. A Qwen3-30B model distilled from Claude Sonnet 4.6 gives each sub-question binary scores and textual feedback on three dimensions: redundancy, clarity, and progressiveness. These outputs drive up to three rounds of sub-question refinement, filter the SFT trajectories, and serve as a dense reward in RL.
- NEAR guided error detection. Given a failed trajectory, an LLM picks a suspicious node, corrects it, and reexecutes its descendants. The new answer is checked against the ground truth. If it is still wrong, the candidate set shrinks to the non-descendants of that node, and the process repeats.
Critical strengths
- The DAG is used consistently across the method: for sub-question quality assessment, for error localization, for building preference data, and for layer-level credit assignment in RL. At inference the system is still a single ReAct policy with no auxiliary models. The Stage I ablation is well controlled: with the same teacher data, removing decomposition costs 4.8 average points. This gives fairly direct evidence that the structure itself helps, beyond better data alone. This is the main reason I think the core idea has merit.
- Good accuracy efficiency trade-off at 30B scale. Within Table 1, the model is the best same-scale agent on four of five benchmarks. It also uses fewer reasoning steps and has lower latency despite issuing more tool calls, which the paper attributes to parallel dispatch within layers. If the comparisons hold up under a controlled protocol (see Weakness 2), this would be a practically useful result.
Critical weaknesses
- NEAR is framed as principled error attribution, but its contraction rule is a heuristic, and the 85.2% recovery figure lacks controls.
The rule makes two exclusions after a failed correction, and neither is justified:
It excludes itself. This assumes the proposed correction was right. A wrong correction of the true error node would wrongly remove it. It excludes . Descendants are only re-executed, not corrected. An independent error inside a descendant can therefore persist, yet that node is removed from the hypothesis space.
Co-occurring errors in independent branches also violate the implicit single-fault assumption. The “convergence guarantee” (Theorem B.1) is trivial: any procedure that removes at least one node per step terminates. It says nothing about whether the procedure localizes the right node.
On the empirical side, recovery is judged against the ground-truth answer, and re-execution is stochastic (sampled decoding plus live web search). Part of the 85.2% could therefore come from resampling combined with oracle filtering rather than from localization. The paper reports no baselines with matched budgets, such as:
correcting a random node, correcting the last node, resampling the full trajectory the same number of times.
There is also an internal inconsistency. Section 4.4 says 74 trajectories “stop before the candidate set is exhausted,” but no value is given.
This matters because NEAR is a headline contribution (abstract and contribution 2). Its downstream benefit is evidenced only by the “w/o high-quality refinement” ablation, which shows +2.1 average points.
- The empirical comparisons are not controlled and have no statistical support.
Uncontrolled baselines. Baseline numbers appear to be copied from the original papers. Those papers use different search and visit tools, summarizers, turn budgets, and context lengths. The model behind the Visit tool’s summarization is not specified, and this could be a significant confound. Omitted baselines. Appendix D.1 lists MiroThinker-v1.5/v1.7-mini, WebResearcher, MiniMax-M2.1, and GLM-4.7, but none of them appear in Table 1. This directly affects the claim of leading all same-scale agents on four of five benchmarks. Small test sets and single runs. Both xBench-DS sets appear to have 100 questions, so one point equals one question. Many reported differences fall within plausible run-to-run variance. Examples: the claim of surpassing GPT-5 and DeepSeek-V3.2 by about 3 points, and several ablation gaps (layer progress reward: −1.2 average; KTO: −1.8 average). Tables 1 and 2 report single runs with no standard deviation. Teacher vs. student framing. DeepSeek-V3.2 is one of the trajectory teachers, which deserves discussion given the claim of surpassing it. Contamination. There is no check for overlap between the training sources (RedSearcher, OpenSeeker) and the evaluation sets.
Questions for authors
- NEAR assumes that (i) the model-declared dependency edges are correct and (ii) the root cause is a single existing node. Neither assumption is validated. A missing edge leaves a true descendant un-re-executed after a correction, so a correct fix can appear to fail and the true error node is wrongly excluded. A spurious edge prunes unrelated nodes. If the root cause is structural — a missing sub-question or a misreading of a clue in the first layer — it is not in at all, so no contraction of can reach it. Could the authors (a) report the accuracy of the declared dependency edges on a human-annotated sample, (b) break down the 14.8% unrecovered trajectories by error type (node-level vs. edge vs. missing node / wrong decomposition), and (c) clarify whether re-executing regenerates the descendant sub-questions or only re-runs retrieval for the existing ones?
- How is aggregated from the sub-question scores and , and with what weights? The advantage of each layer depends only on that layer’s reward, and the outcome reward is added only at the terminal layer. How do earlier layers receive any outcome signal? Is there a return or discounting?
- What are the recovery rates of the budget-matched baselines listed above (random node, last node, full resampling)? On a human-checked sample, how often is the node that NEAR finally corrects actually the true root cause? What is ?
4 comments
Q3: Recovery evidence and contamination
The complete recovery comparison on the same 500 failed trajectories is reproduced below:
| Method | Answer recovery rate | Recovered / 500 |
|---|---|---|
| Random-node selection | 26.8% | 134 |
| Earliest-node selection | 34.6% | 173 |
| Self-diagnosis and repair without NEAR pruning | 47.6% | 238 |
| Last-node selection | 5.2% | 26 |
| Up to three complete trajectory regenerations | 30.4% | 152 |
| NEAR | 85.2% | 426 |
These values measure answer recovery under the stated repair protocols; they do not establish localization correctness. We will report repair cost and clearly state that independent human root-cause accuracy is not yet available.
Before training, our overlap check found zero shared questions between the SFT/KTO/DAPO data prepared for training and each of the five evaluation benchmarks, including the RedSearcher and OpenSeeker subsets. We will include the matching protocol and per-benchmark zero-overlap counts in the revision.
Thanks for your clarification, majority of my concern is solved I will raise my score to 3 accordingly.
We appreciate the reviewer's positive evaluation and feedback. We will continue to improve the quality of our research in future work. Thank you for the time you've spent and the valuable suggestions provided during the review process.
W1: NEAR's assumptions and downstream regeneration
We thank the reviewer for precisely identifying NEAR's limitations. We agree that a failed correction does not logically prove that the selected node and all descendants are error-free. NEAR should be understood as a dependency-guided candidate-search heuristic. Theorem B.1 guarantees finite termination only, not correct root-cause localization. We will revise the abstract, contribution statement, Sec. 3.3, and Appendix B to remove any stronger implication.
NEAR searches for a critical upstream error because a wrong upstream answer may contaminate the downstream reasoning state. Importantly, descendant re-execution does not merely rerun retrieval for fixed old subquestions. We truncate the trajectory at the selected node, retain the preceding prefix, inject correction feedback, and regenerate the subsequent reasoning, subquestions, and tool calls. The regenerated suffix can therefore revise downstream questions and add information missing from the original suffix. Nevertheless, incorrect or missing dependency edges may affect candidate pruning, and structural errors already present in the retained prefix are not guaranteed to be repaired.
The reported 85.2% is the answer-recovery rate of the complete repair pipeline, not root-cause localization accuracy. The recovery comparisons demonstrate NEAR's empirical value for offline repair, but do not replace an independent human audit of dependency-edge or localization accuracy. We do not currently have such annotations and will state this limitation explicitly rather than infer localization correctness from final-answer recovery.
The same distinction applies to the 74 unrecovered trajectories. They may involve a failed correction, incorrect or missing edges, a missing subquestion or wrong decomposition, multiple independent errors, unavailable evidence, or termination after the five-attempt cap or an invalid auditor output. Stopping reasons explain why the procedure ended; they do not by themselves identify the underlying error type. We will report the two separately.
W2: Comparison scope and statistical support
For method attribution, we reproduce the relevant internal controls below. All Stage-I variants use identical training questions and teacher models; the Stage-III variants share the same preceding structured training.
These results provide stage-specific evidence, not a fully matched end-to-end Flat-ReAct SFT-to-KTO-to-DAPO comparison, because NEAR-based KTO construction itself requires the DAG.
For the direct REDSearcher comparison, we use the same Google Search/Search/Visit tools,
gpt-osssummarizer, interaction limit, and context setting. The shared reported benchmarks are reproduced here:Other literature-reported baselines without a reproduced common environment will remain contextual comparisons only. We will clarify why Appendix D.1 models are absent when corresponding comparable benchmark results are unavailable and narrow the “all same-scale agents” claim.
We also reproduce the three-run results used to assess run-level stability:
We will distinguish these repeated results from single-run external numbers and avoid significance claims for small gaps without corresponding repetitions.
Q2: Reward aggregation and credit assignment
The frozen post-KTO checkpoint computes Eq. 7, and the gold answer is visible only during reward calculation. The three binary quality dimensions are averaged per subquestion and then across subquestions in a layer. The subquestion-quality and layer-progress rewards each receive weight 0.3; the binary outcome reward receives weight 1.0 and is added only at the terminal layer. No format reward is used, so the maximum combined reward is 1.6. Earlier layers receive local quality and progress rewards, after which all layer rewards in the rollout group are jointly normalized to form layer-level advantages. Policy ratios are computed per generated token, while tool observations are excluded from the loss.