- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TRACE-o1 extends Search-o1-style agentic search with three test-time controls: (1) Evidence planning: a planner builds a plan, search queries and a briefing once per question. All trajectories reuse them. (2) Critic-audited search and repair: after each step a critic LLM chooses NONE/SEARCH/REVISE/REPLAN/RESTRUCTURE/REWRITE and outputs a self-reported confidence . (3) Consistency-guided selection: an LLM aggregator combines completed trajectories. When it flags disagreement, an answer cluster is scored over answer-masked embeddings.
On nine QA, math and biomedical benchmarks with Qwen3-4B, macro accuracy rises from 59.1 (Search-o1) to 63.6, above Search-o1+SC at K=5 (60.5) and the roughly token-matched K=10 (61.5) (Tables 1, 3). A Llama-3.2-3B replication, disagreement diagnostics and a three-dataset ablation follow.
Critical strengths
- Consistent gains over the direct baseline, including at matched compute
TRACE-o1 beats Search-o1 on all nine datasets with Qwen3-4B (e.g., GPQA +8.6, Bamboogle +6.4) and by 8.7 macro points with Llama-3.2-3B. Against Search-o1+SC at K=10, which uses a similar token budget (38.7K vs. 37.9K per question), it still gains +2.04, so the gain is not just more sampling. Specialized baselines (MA-RAG, CRAG, MAD) each win one dataset but stay at 59.0~59.4 macro.
- Reliability diagnostics beyond accuracy
At equal K, mean disagreement falls from 0.187 to 0.108 and unanimous questions rise from 56.7% to 74.6%; GPQA gains are largest on high-disagreement questions (+13.0 vs. +7.2). The controls make trajectories more consistent, not only more accurate.
Critical weaknesses
-
Cost and latency are not characterized, and compute is not adaptive
- TRACE-o1 uses about 9× the tokens of Search-o1 (37.9K vs. 4.2K per question), 10.8× the LLM calls (18.4 vs. 1.7) and 12× the searches (4.3 vs. 0.35), for +4.5 macro points.
- Only a prompt+completion token sum is reported: no wall-clock latency (critic/repair calls and web searches run sequentially) and no prefill/decode split. The plan/briefing prefix is shared across the K trajectories, so prefix/KV caching could cut prefill cost, but this is not discussed.
- Every question runs K=5 plus the critic loop regardless of difficulty, although extra trajectories add <1.2 points on biomedical tasks (Fig. 4) and 56.7% of questions are already unanimous under Search-o1+SC.
- SC at K=10 uses 3.2× the tokens of K=5 for 2× the trajectories (tokens per call 1,001 → 2,297), so how well the "token-matched" control is matched is unclear.
-
The critic is not validated directly, and the evidence for each component is thin
- There are no critic action frequencies, no right-->wrong vs. wrong-->right repair rates, and no test of whether separates correct from incorrect trajectories. Indirect evidence suggests is weak: picking the top- trajectory is -1.43 points below majority vote (Table 13), the full selector gains only +0.42, and Table 8 shows critic–replan loops.
- The ablation covers only GPQA, MedQA and Math500; critic-only lowers Math500 (79.0 --> 78.6) and reranking lowers MedQA (76.2 --> 75.0).
- Results are single runs at temperature 0.7 and the bootstrap resamples questions only, while AIME (30 questions), AMC (40) and the GPQA high-disagreement bucket (46) are small.
Questions for authors
-
What is per-question wall-clock latency vs. Search-o1 and SC at K=10, the prefill/decode split, and the saving from caching the shared plan/briefing prefix? Latency within a small multiple of SC at K=10 would raise Informativeness.
-
How often does each critic action fire, what fraction of repairs flip wrong-->right vs. right-->wrong, and what is the AUROC of for correctness?
3 comments
W2 + Q2 | Critic Discrimination Ability, Repair Effectiveness, Negative Ablation, and Stability
(1) Critic actions and repair behavior.
We analyzed all Critic logs from the main experiments, covering 5,407 questions with K=5 and a total of 48,263 calls, or approximately 1.79 calls per trajectory. Table R3 reports the call-level action distribution, and Table R4 groups trajectories by their first repair action and measures correctness transitions before and after repair.
Table R3. Critic output distribution (48,263 calls).
| Critic Output | Count | Share of All Calls | Share of Material Repair Actions |
|---|---|---|---|
NONE | 20,572 | 42.62% | — |
REVISE | 12,903 | 26.73% | 49.99% |
REPLAN | 7,055 | 14.62% | 27.33% |
SEARCH | 5,842 | 12.10% | 22.63% |
RESTRUCTURE / REWRITE | 7 / 4 | 0.02% | 0.04% |
| Empty output / format error | 1,825 / 55 | 3.90% | — |
| Total | 48,263 | 100% | 100% (material actions only) |
Table R4. Trajectory correctness transitions grouped by first repair action (7,428 trajectories).
| First Repair Action | Trajectories | Wrong→Right | Right→Wrong | Net Change |
|---|---|---|---|---|
SEARCH | 3,775 | 398 | 7 | +391 |
REVISE | 1,933 | 109 | 14 | +95 |
REPLAN | 1,719 | 16 | 3 | +13 |
RESTRUCTURE | 1 | 0 | 0 | 0 |
| Total | 7,428 | 523 | 24 | +499 |
Among these repair trajectories, 523 change from incorrect to correct and 24 from correct to incorrect, a correction-to-regression ratio of approximately 22:1. Repair processes initiated by SEARCH produce the largest net positive transition (+391), showing that retrieval-based evidence supplementation is the most frequent repair mode and contributes the most net corrections in the current process.
(2) Direct discriminative ability of Critic scores.
The score c_i used for terminal-cluster scoring is the uncalibrated trajectory score from the final Critic output (§3.3, Appendix E.5).
On the 27,031 trajectories from the same experimental run, c_i achieves an AUROC of 0.712 for final trajectory correctness.
This result shows that c_i carries a correctness signal, although it is not a perfect verifier. We therefore combine it with the semantic-consistency metrics coh and cen for cluster scoring, and use it for reranking only when need_posthoc=true.
(3) Negative ablation and stage synergy.
We retain and discuss the negative results:
- MedQA: reranking reduces accuracy from 76.2% to 75.0%.
- Math500: the Critic-only setting reduces accuracy from 79.0% to 78.6%.
These results show that the Critic does not improve every task and configuration independently. We therefore position it as an auxiliary component within the multi-stage pipeline rather than as a module that is uniformly effective in isolation.
(4) Randomness stability and confidence intervals.
Appendix B reports confidence intervals obtained by stratified paired resampling over fixed predictions. They quantify uncertainty from question sampling; randomness from regenerating trajectories at temperature=0.7 is outside their coverage. We will make this statistical scope explicit in the revised version and interpret results on small datasets, including AIME, AMC, and the high-disagreement GPQA subset, with appropriate caution.
Thank you for the detailed rebuttal. I will maintain my rating of 3.
We thank the reviewer for raising concerns about realistic deployment cost, Critic effectiveness, and experimental stability. In response, we added end-to-end latency measurements under a unified resource cap (Table R2), analyses of Critic actions and correctness transitions (Tables R3–R4), and an AUROC evaluation of Critic scores. We also clarified the scope of the resource-matched comparison and confidence intervals.
W1 + Q1 | Computational Cost, Wall-Clock Latency, Fixed K, and Cache Benefits
(1) Computational cost and resource-matched comparison.
We acknowledge that TRACE-o1 has substantially higher cost than single-trajectory Search-o1 (37,865 vs. 4,164 Tokens/Q). This cost comes from multi-candidate reasoning, making Search-o1+SC the more informative multi-candidate comparator.
Table R1 summarizes the nine-dataset macro results from Table 3 of the paper. When TRACE-o1 (K=5) and SC (K=10) use similar numbers of tokens (37,865 vs. 38,708), TRACE-o1 retains a 2.04 pp accuracy advantage.
Table R1. Token-matched resource comparison (nine-dataset macro).
TRACE-o1 therefore retains its advantage under similar token consumption. Because Calls and Searches are not fully matched, we refer to this as a token-matched comparison, which tests whether a larger sampling budget alone explains the performance difference.
(2) End-to-end wall-clock latency.
To address deployment efficiency, we added a single-stream latency measurement using the first five questions from each of the nine datasets, for 45 questions in total. Both methods used the same per-question resource cap and retrieval endpoint, with caching disabled, no queueing, and randomized execution order.
Table R2. Latency and resource consumption under a unified per-question resource cap (45 questions).
In this 45-question latency test, TRACE-o1 reduces mean, median, and P90 latency by 47.2%, 47.3%, and 56.8%, respectively, while using fewer total tokens (33,910 vs. 41,066 Tokens/Q). Completion tokens are approximately half as large as those of SC. On this subset, TRACE-o1 and SC answer 30/45 (66.7%) and 29/45 (64.4%) questions correctly, respectively. The main accuracy conclusions continue to come from the full nine-dataset results.
(3) Fixed K, Prefill/Decode, and caching.
Our main focus was the effect of multi-stage collaboration on the accuracy of multi-candidate reasoning, rather than system-level inference optimization. We have not yet separately measured the Prefill/Decode latency breakdown or the benefit of shared-prefix caching. We will add this clarification to the revised version and identify system-level latency analysis and cache optimization as directions for future work.