- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACE, a lightweight trajectory selector for parallel search agents. Instead of using majority voting or generative aggregation, TRACE constructs a query–evidence graph across completed search trajectories, propagates shared evidence through a GNN, and ranks candidate answers using an answer-conditioned readout. Experiments across WebQA and long-horizon search benchmarks demonstrate improvements over voting and competitive performance against LLM-based aggregators, with substantially lower post-rollout computation costs.
Critical strengths
- Efficient trajectory selection. TRACE avoids additional autoregressive generation and achieves 12–258× lower post-rollout GPU cost than LLM-based aggregation methods.
- Well-designed cross-rollout evidence sharing. The occurrence-preserving graph allows trajectories to exchange information while retaining their original retrieval context. The component ablations support the effectiveness of this design.
- Remaining gap to oracle selection. On long-horizon benchmarks, TRACE achieves 78.6% accuracy against a Pass@16 upper bound of 90.9%, leaving a 12.3-point gap. This raises questions about how effectively the selector identifies correct answers already present in the candidate pool.
Critical weaknesses
- Limited gains on stronger search policies. TRACE provides only marginal improvements on RL-trained agents (+0.3 points for Qwen2.5-14B RL). Given the high variance of search benchmarks, the lack of statistical significance analysis raises concerns about the reliability of these gains.
- Selection-only limitation. TRACE can only return an existing answer and cannot synthesize new answers from complementary evidence, limiting its applicability when no trajectory is correct.
- Unverified factual discrimination. TRACE learns to rank candidates using answer-level correctness labels and cosine similarity between learned representations, without explicit supervision for evidence verification. It remains unclear whether the selector can identify subtle factual errors rather than relying on semantic relevance or evidence overlap. Despite a Pass@16 of 90.9%, TRACE achieves only 78.6% selection accuracy, leaving a substantial gap that raises concerns about its ability to reliably identify correct answers
Questions for authors
- Statistical reliability. Could the authors report variance or statistical significance across repeated evaluations, particularly for the small gains on RL-trained agents?
- Selection versus synthesis. How does TRACE handle cases where no candidate is correct but complementary evidence across trajectories could support a correct answer?
- Factual discrimination. How does TRACE learn to distinguish factual correctness from semantic relevance using only answer-level supervision? Could the authors analyze why the selector still misses correct candidates despite the high Pass@16?
3 comments
Thank you again for your detailed review. As the author-reviewer discussion closes on November 18, we would like to check whether our response has addressed your concerns. A short summary of our answers to your three questions:
- Q1, statistical reliability. Paired question-level tests (Appendix K): the gain over majority voting is significant on the Base/SFT WebQA pools (p ≤ 2e-7) and on the long-horizon average (+3.07, 95% CI [+1.64, +4.54]). The RL gain is small because in 57–61% of RL questions all 16 rollouts give the same answer, so every selection method equals voting.
- Q2, selection vs. synthesis. On long-horizon search Pass@16 is 90.9% and voting 75.5%, so 15.4 points are available to selection alone. When no candidate is correct, the generative aggregators recover only 0.3–1.4 points of total accuracy (Appendix J.4).
- Q3, factual discrimination and the gap to Pass@16. On the 2,168 questions whose pool holds the correct answer and a wrong answer of the same type, TRACE reaches 74.2% against 67.1% for voting, and 25.5% against 0.0% when the correct answer is in the minority. When exactly one rollout is correct, TRACE selects it in 38.8% of long-horizon cases against 20.0% for voting.
If any concern still stands, please let us know and we will address it before the deadline. If our response has resolved them, we would be grateful if you could consider updating your score.
Thank you for your rebuttal. I have no other questions. Raised my rating to weak acceptance.
Thank you for the detailed review. Shared controls are in the General Response.
W1 & Q1. RL pools and significance. All methods share questions and pools, so we use question-level paired tests (exact McNemar, paired bootstrap). All six WebQA pools are in Appendix K.
The small RL gain reflects how little there is to select from, not the strength of the policy:
W2 & Q2. Selection vs. synthesis. TRACE targets questions where a correct answer is already in the pool but voting misses it: on long-horizon search Pass@16 is 90.9% and MV 75.5%, so 15.4 points are available to selection alone. When no candidate is correct, the generative aggregators recover only 0.3–1.4 points of total accuracy, mostly by shortening a verbose candidate or from their own knowledge, not by combining evidence across trajectories (Appendix J.4). TRACE still has the higher overall accuracy (Tables 1–2).
W3 & Q3. Factual discrimination. All candidates in a pool answer the same question from evidence retrieved for it, so semantic relevance is nearly constant within a pool. The ranking losses contrast candidates within a pool, so only features that separate a supported answer from an unsupported one can lower them. On the 2,168 questions whose pool holds the correct answer and a wrong answer of the same type (another date, number, person or place):
Example (Bamboogle): "Who was the father of the father of psychoanalysis?" Gold: Jacob Freud. MV returns Schlomo Freud (6 votes), TRACE Jacob Freud (5 votes). Both groups read the same passage, which says Jacob is Sigmund's father and Schlomo is Jacob's. TRACE scores the Jacob rollouts 0.09 to 0.13 and the Schlomo rollouts −0.40 to −0.21; evidence overlap cannot explain this (more examples: Appendix J.5).
The gap to Pass@16. Pass@16 is the ceiling of every selection method. When a correct candidate exists, TRACE returns one in 86.5% of cases (MV 83.1%), and a missed correct answer is usually ranked second (65–75%). Of the remaining gap, 5.0 points are questions where only one or two of the 16 rollouts are correct, which is also where TRACE gains most over voting (single correct rollout: 20.0 → 38.8).