- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines overly specific queries and premature stopping in LLM search agents, termed the Golden Document Fallacy. It proposes an Exploration–Coverage–Synthesis (E-C-S) strategy and trains SearchPLM through supervised distillation and curriculum GRPO, with BM25-based webpage refinement. Experiments across seven benchmarks report gains over several larger models.
Critical strengths
-
E-C-S provides concrete guidance for query refinement, failure recovery, and source verification. The prompting and guided-trajectory ablations support its usefulness.
-
SearchPLM-14B achieves the highest scores on six of seven benchmarks in the shared search environment. RL further improves performance while reducing average searches from 6.64 to 5.75 relative to the SFT model.
Critical weaknesses
-
Table 10 shows an advantage from E-C-S-guided trajectories after SFT. However, the unguided SFT model is not evaluated after the same RL training, so it is unclear whether this advantage remains in the final model.
-
The averaging rule in Table 3 is unclear. SearchPLM-14B has an unweighted mean of 58.74 and a sample-weighted mean of 67.51 using Table 12, versus the reported 67.43; the corresponding values for MiroThinker are 52.70 and 59.68, versus 62.03.The summarizer is also listed as DeepSeek-V3 in Section 4.2 and V3.1 in Appendix H.
Questions for authors
-
How does the unguided-trajectory SFT model in Table 10 perform after the same RL procedure, using identical RL data and training budgets?
-
Please specify how Avg. is computed, explain or correct the numerical discrepancies, and confirm the summarizer used for all compared models.
3 comments
Dear Reviewers,
We would appreciate your feedback on our rebuttal, additional experiments, and revised manuscript, particularly whether they address your main concerns. If any questions remain, we would welcome the opportunity to discuss them and provide further clarification during the discussion period.
Your feedback will help us further improve the paper. Thank you again for your time and effort.
A Kindly Reminder
Dear Reviewer,
With only one day remaining in the discussion period, we would sincerely appreciate it if you could find time to read our rebuttal and participate in the discussion.
We understand that this is a busy and stressful period for everyone, and we appreciate the time and effort you have already devoted to reviewing our work. In response to your concerns, we have added a controlled intervention, a vanilla-SFT+RL comparison, and fixed-budget evaluations, and incorporated these results into the revised manuscript.
We have not yet received feedback on these additions. Even a brief response indicating whether the new evidence addresses your concerns, and which questions remain, would be very helpful. This would give us an opportunity to clarify unresolved points before the discussion closes.
We remain available for further questions and would greatly value your assessment. Thank you again for your time and consideration.
We thank the reviewer for recognizing the practical value of E-C-S guidance and SearchPLM's benchmark performance. We have added the requested vanilla-SFT+RL comparison and clarified the averaging protocol and summarizer configuration.
The additional SFT+RL comparison is now included in Appendix E, Table 16 (lines 848–877) and reproduced below. The SFT-stage comparison appears in Appendix E, Table 14 (lines 837–845). Both models start from Qwen3-14B and undergo SFT followed by the same RL procedure; SearchPLM uses E-C-S-guided SFT.
Both checkpoints use the same RL set of approximately 1,300 questions from the ASearcher training split, after excluding SFT questions and questions with zero accuracy across repeated teacher sampling (§3.3.2, lines 294–300). All RL hyperparameters and the training budget are identical, following Appendix G.2 (lines 908–921).
SearchPLM improves all seven benchmark scores and raises Micro Avg. by 6.74 percentage points, while reducing average searches by 37.6%. This supports the value of strategy-guided SFT initialization in the full SFT+RL pipeline.
Thank you for identifying these discrepancies. They arose from an aggregation-script bug and manual transcription errors. All numerical entries in the manuscript tables were entered by hand, and some values were mistyped during this process. The script excluded instances without valid scores from the denominator, including runs that reached the maximum number of search iterations without generating a final answer. Such runs have no final answer to judge and now receive zero. We fixed the script, recomputed Micro Avg. over all 2,163 questions with unscored instances contributing zero, and corrected the transcription errors. Before submitting the final version, we will recheck every table entry and reported aggregate against the underlying evaluation outputs. The submitted Table 3 is now Table 5 (lines 324–348). §4.2 (lines 410–413) defines the scores, and Appendix J, Table 19 (lines 1033–1042) gives the evaluation counts.
SearchPLM-14B. The submitted version reported 67.43%. Our recomputed Micro Avg. in Table 5 (line 348) is 67.51%, which agrees with your sample-weighted calculation from the seven benchmark scores.
Qwen3-14B. Three instances without valid scores now contribute zero. The corrected Micro Avg. is 54.90%, with 36.76% on GAIA and 46.91% on WebWalker (Table 5, lines 333–334). SearchPLM-14B improves over this baseline by 12.61 percentage points.
MiroThinker. Two incomplete GAIA runs were excluded from the original aggregation. Counting them as zero changes GAIA from 43.00% to 42.16% and the pooled score from 62.03% to 61.97%. We also corrected the WebWalker entry to 54.71% (Table 5, line 346). The updated benchmark scores agree with the recomputed Micro Avg., using the evaluation counts in Appendix J, Table 19 (lines 1033–1042).
Corrected results in Table 5 (lines 333–334, 346, and 348).
Webpage summarizer. We corrected the typo in §4.2 (lines 403–408) to DeepSeek-V3.1, which summarized webpages for all models, consistent with Appendix H (lines 953–956). DeepSeek-V3 judged final answers (§4.2, lines 407–413); Kimi-K2.5 and GLM-5 independently re-scored predictions (§5, Table 8, lines 432–438 and 482–485).
The added comparison shows that E-C-S-guided initialization retains its advantage after identical RL training. We appreciate your careful examination of the evaluation details and hope the additional results and protocol clarifications address your main concerns.