acceptodds
Under review as a conference paper at ICLR 2027

TRACE-o1: Reliable Search-Augmented Reasoning via Stage-Aligned Trajectories

Abstract

Search-augmented reasoning enables LLMs to consult external evidence during inference, but search access alone does not guarantee reliable reasoning. Existing interleaved search–reasoning systems often make retrieval and reasoning decisions from local trajectory context, which can lead to fragmented evidence acquisition, accumulated intermediate errors, and unstable final predictions. We propose TRACE-o1, a stage-aligned framework that introduces reliability controls throughout the search-augmented reasoning process. Before reasoning, TRACE-o1 plans an evidence path to guide subsequent retrieval; during reasoning, it audits and repairs the evolving trajectory; and after reasoning, it stabilizes predictions across completed candidate trajectories through consistency-guided selection. Experiments across nine datasets show that TRACE-o1 improves overall answer accuracy and cross-trajectory stability relative to Search-o1-based baselines. Under a matched-K=5 comparison, TRACE-o1 improves macro accuracy from 60.5 to 63.6 over Search-o1 with self-consistency, with a 95% paired-bootstrap confidence interval of [+1.4, +5.0] percentage points. These results support the benefit of combining evidence planning, trajectory repair, and consistency-guided selection within a unified framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Search-Augmented Reasoning, Retrieval-Augmented Generation, Reliable Reasoning, Process-Level Critique, Answer Consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rvF3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE-o1 extends Search-o1-style agentic search with three test-time controls: (1) Evidence planning: a planner builds a plan, search queries and a briefing once per question. All trajectories reuse them. (2) Critic-audited search and repair: after each step a critic LLM chooses NONE/SEARCH/REVISE/REPLAN/RESTRUCTURE/REWRITE and outputs a self-reported confidence cic_i. (3) Consistency-guided selection: an LLM aggregator combines K=5K=5 completed trajectories. When it flags disagreement, an answer cluster is scored over answer-masked embeddings.

On nine QA, math and biomedical benchmarks with Qwen3-4B, macro accuracy rises from 59.1 (Search-o1) to 63.6, above Search-o1+SC at K=5 (60.5) and the roughly token-matched K=10 (61.5) (Tables 1, 3). A Llama-3.2-3B replication, disagreement diagnostics and a three-dataset ablation follow.

Critical strengths
  1. Consistent gains over the direct baseline, including at matched compute

TRACE-o1 beats Search-o1 on all nine datasets with Qwen3-4B (e.g., GPQA +8.6, Bamboogle +6.4) and by 8.7 macro points with Llama-3.2-3B. Against Search-o1+SC at K=10, which uses a similar token budget (38.7K vs. 37.9K per question), it still gains +2.04, so the gain is not just more sampling. Specialized baselines (MA-RAG, CRAG, MAD) each win one dataset but stay at 59.0~59.4 macro.

  1. Reliability diagnostics beyond accuracy

At equal K, mean disagreement falls from 0.187 to 0.108 and unanimous questions rise from 56.7% to 74.6%; GPQA gains are largest on high-disagreement questions (+13.0 vs. +7.2). The controls make trajectories more consistent, not only more accurate.

Critical weaknesses
  1. Cost and latency are not characterized, and compute is not adaptive

    • TRACE-o1 uses about 9× the tokens of Search-o1 (37.9K vs. 4.2K per question), 10.8× the LLM calls (18.4 vs. 1.7) and 12× the searches (4.3 vs. 0.35), for +4.5 macro points.
    • Only a prompt+completion token sum is reported: no wall-clock latency (critic/repair calls and web searches run sequentially) and no prefill/decode split. The plan/briefing prefix is shared across the K trajectories, so prefix/KV caching could cut prefill cost, but this is not discussed.
    • Every question runs K=5 plus the critic loop regardless of difficulty, although extra trajectories add <1.2 points on biomedical tasks (Fig. 4) and 56.7% of questions are already unanimous under Search-o1+SC.
    • SC at K=10 uses 3.2× the tokens of K=5 for 2× the trajectories (tokens per call 1,001 → 2,297), so how well the "token-matched" control is matched is unclear.
  2. The critic is not validated directly, and the evidence for each component is thin

    • There are no critic action frequencies, no right-->wrong vs. wrong-->right repair rates, and no test of whether cic_i separates correct from incorrect trajectories. Indirect evidence suggests cic_i is weak: picking the top-cic_i trajectory is -1.43 points below majority vote (Table 13), the full selector gains only +0.42, and Table 8 shows critic–replan loops.
    • The ablation covers only GPQA, MedQA and Math500; critic-only lowers Math500 (79.0 --> 78.6) and reranking lowers MedQA (76.2 --> 75.0).
    • Results are single runs at temperature 0.7 and the bootstrap resamples questions only, while AIME (30 questions), AMC (40) and the GPQA high-disagreement bucket (46) are small.
Questions for authors
  1. What is per-question wall-clock latency vs. Search-o1 and SC at K=10, the prefill/decode split, and the saving from caching the shared plan/briefing prefix? Latency within a small multiple of SC at K=10 would raise Informativeness.

  2. How often does each critic action fire, what fraction of repairs flip wrong-->right vs. right-->wrong, and what is the AUROC of cic_i for correctness?

3 comments

We thank the reviewer for raising concerns about realistic deployment cost, Critic effectiveness, and experimental stability. In response, we added end-to-end latency measurements under a unified resource cap (Table R2), analyses of Critic actions and correctness transitions (Tables R3–R4), and an AUROC evaluation of Critic scores. We also clarified the scope of the resource-matched comparison and confidence intervals.

W1 + Q1 | Computational Cost, Wall-Clock Latency, Fixed K, and Cache Benefits

(1) Computational cost and resource-matched comparison.

We acknowledge that TRACE-o1 has substantially higher cost than single-trajectory Search-o1 (37,865 vs. 4,164 Tokens/Q). This cost comes from multi-candidate reasoning, making Search-o1+SC the more informative multi-candidate comparator.

Table R1 summarizes the nine-dataset macro results from Table 3 of the paper. When TRACE-o1 (K=5) and SC (K=10) use similar numbers of tokens (37,865 vs. 38,708), TRACE-o1 retains a 2.04 pp accuracy advantage.

Table R1. Token-matched resource comparison (nine-dataset macro).

MethodKMacro Acc.Tokens/QCalls/QSearches/Q
Search-o1159.1%4,1641.700.35
Search-o1+SC560.5%11,97111.961.75
Search-o1+SC1061.5%38,70816.853.36
TRACE-o1563.6%37,86518.364.34

TRACE-o1 therefore retains its advantage under similar token consumption. Because Calls and Searches are not fully matched, we refer to this as a token-matched comparison, which tests whether a larger sampling budget alone explains the performance difference.

(2) End-to-end wall-clock latency.

To address deployment efficiency, we added a single-stream latency measurement using the first five questions from each of the nine datasets, for 45 questions in total. Both methods used the same per-question resource cap and retrieval endpoint, with caching disabled, no queueing, and randomized execution order.

Table R2. Latency and resource consumption under a unified per-question resource cap (45 questions).

MethodMean (s)Median (s)P90 (s)Calls/QPrompt tok/QCompletion tok/QSearches/Q
SC (K≤10)493.4244.11549.416.324,50716,5593.38
TRACE-o1 (K=5)260.3128.6669.512.625,5338,3772.02

In this 45-question latency test, TRACE-o1 reduces mean, median, and P90 latency by 47.2%, 47.3%, and 56.8%, respectively, while using fewer total tokens (33,910 vs. 41,066 Tokens/Q). Completion tokens are approximately half as large as those of SC. On this subset, TRACE-o1 and SC answer 30/45 (66.7%) and 29/45 (64.4%) questions correctly, respectively. The main accuracy conclusions continue to come from the full nine-dataset results.

(3) Fixed K, Prefill/Decode, and caching.

Our main focus was the effect of multi-stage collaboration on the accuracy of multi-candidate reasoning, rather than system-level inference optimization. We have not yet separately measured the Prefill/Decode latency breakdown or the benefit of shared-prefix caching. We will add this clarification to the revised version and identify system-level latency analysis and cache optimization as directions for future work.

W2 + Q2 | Critic Discrimination Ability, Repair Effectiveness, Negative Ablation, and Stability

(1) Critic actions and repair behavior.

We analyzed all Critic logs from the main experiments, covering 5,407 questions with K=5 and a total of 48,263 calls, or approximately 1.79 calls per trajectory. Table R3 reports the call-level action distribution, and Table R4 groups trajectories by their first repair action and measures correctness transitions before and after repair.

Table R3. Critic output distribution (48,263 calls).

Critic OutputCountShare of All CallsShare of Material Repair Actions
NONE20,57242.62%—
REVISE12,90326.73%49.99%
REPLAN7,05514.62%27.33%
SEARCH5,84212.10%22.63%
RESTRUCTURE / REWRITE7 / 40.02%0.04%
Empty output / format error1,825 / 553.90%—
Total48,263100%100% (material actions only)

Table R4. Trajectory correctness transitions grouped by first repair action (7,428 trajectories).

First Repair ActionTrajectoriesWrong→RightRight→WrongNet Change
SEARCH3,7753987+391
REVISE1,93310914+95
REPLAN1,719163+13
RESTRUCTURE1000
Total7,42852324+499

Among these repair trajectories, 523 change from incorrect to correct and 24 from correct to incorrect, a correction-to-regression ratio of approximately 22:1. Repair processes initiated by SEARCH produce the largest net positive transition (+391), showing that retrieval-based evidence supplementation is the most frequent repair mode and contributes the most net corrections in the current process.

(2) Direct discriminative ability of Critic scores.

The score c_i used for terminal-cluster scoring is the uncalibrated trajectory score from the final Critic output (§3.3, Appendix E.5).

On the 27,031 trajectories from the same experimental run, c_i achieves an AUROC of 0.712 for final trajectory correctness.

This result shows that c_i carries a correctness signal, although it is not a perfect verifier. We therefore combine it with the semantic-consistency metrics coh and cen for cluster scoring, and use it for reranking only when need_posthoc=true.

(3) Negative ablation and stage synergy.

We retain and discuss the negative results:

  • MedQA: reranking reduces accuracy from 76.2% to 75.0%.
  • Math500: the Critic-only setting reduces accuracy from 79.0% to 78.6%.

These results show that the Critic does not improve every task and configuration independently. We therefore position it as an auxiliary component within the multi-stage pipeline rather than as a module that is uniformly effective in isolation.

(4) Randomness stability and confidence intervals.

Appendix B reports confidence intervals obtained by stratified paired resampling over fixed predictions. They quantify uncertainty from question sampling; randomness from regenerating trajectories at temperature=0.7 is outside their coverage. We will make this statistical scope explicit in the revised version and interpret results on small datasets, including AIME, AMC, and the high-disagreement GPQA subset, with appropriate caution.

Thank you for the detailed rebuttal. I will maintain my rating of 3.

Reviewer_dYqgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TRACE-o1, a search-augmented reasoning framework that requires no changes to the underlying language model’s parameters. Its core idea is to introduce reliability controls before, during, and after reasoning: evidence planning establishes a retrieval plan and background context; a critic examines and repairs candidate reasoning trajectories; and LLM aggregation followed by conditionally triggered reranking determines the final answer.

The authors evaluate TRACE-o1 on nine benchmarks using two open-weight models. With the candidate count matched at (K=5), TRACE-o1 improves macro-average accuracy from 60.5% to 63.6% over Search-o1+SC while reducing disagreement among candidate answers. The authors also report a positive gain over the (K=10) Search-o1+SC configuration under approximately matched token budgets. The paper further provides component ablations, disagreement-stratified analysis, candidate-count analysis, and an audit of retrieval contamination.

Critical strengths
  1. The overall performance gains are reasonably convincing, and the authors provide meaningful resource-controlled comparisons.

  2. The framework clearly distinguishes within-trajectory repair from selection across candidates and provides a broad range of diagnostic analyses.

Critical weaknesses

1.A substantial gap remains between the paper’s central claims about reliability and its direct measurements. Reliability is defined in terms of deliberate evidence acquisition, process-level error repair, and cross-trajectory stability, yet the main quantitative evidence focuses on final-answer accuracy and disagreement among candidate answers. The paper does not adequately report whether evidence omissions are reduced, how effectively the critic identifies errors, or the rates at which repairs turn incorrect answers into correct ones and vice versa. Although the authors clarify that the reported stability is conditional on shared initialization information, consistency still needs to be distinguished more clearly from reliability. This gap directly limits the support for the paper’s central argument.

2.The necessity of the stage-aligned design and its additional complexity has not been sufficiently established. Table 4 demonstrates benefits from combining components, but lacks comparisons against simple self-correction under matched budgets, planning controls using the same retrieved evidence, and intermediate auditing versus final-only auditing. Consequently, it is difficult to determine whether the gains arise from the placement of interventions across stages, access to more evidence, or additional reasoning.

Questions for authors
  1. Could the authors report, based on the existing trajectories, the frequency of each critic action and the numbers of incorrect-to-correct and correct-to-incorrect answer transitions following repair? These results would help assess whether process-level repair yields consistent net benefits.

2.In Table 2, what proportion of questions receive unanimous but incorrect answers under each method? Does the proportion of questions with at least one correct candidate answer increase? Is there evidence distinguishing accuracy gains from the shared initial plan from increased correlation among candidates?

3.What are the candidate counts, retrieval volumes, and token costs for each configuration in Table 4? Are there comparisons with standard self-correction under matched budgets, or with and without explicit planning using the same retrieved evidence, to isolate the contribution of stage-specific controls?

  1. Final-answer accuracy alone is insufficient to assess reliability. Could the authors provide quantitative measures of reliability beyond final-answer accuracy, particularly measures of evidence quality and process-level correctness?
3 comments

We thank the reviewer for the questions about reliability, stage-wise control, and correlated candidate errors. This response presents the added Critic repair analysis (Tables R1–R2) and evidence-quality audit (Table R3). Part 2 addresses stage-wise control, resource comparisons, and candidate consistency.

W1 + Q1 + Q4 | Critic Repair Behavior and Reasoning Quality

Beyond final accuracy, we examined actual repair behavior, evidence acquisition, and critical reasoning steps.

(1) Critic action frequency and repair effectiveness.

We analyzed all Critic logs from the main experiment (5,407 questions × K=5). Of 27,035 candidate trajectories, 27,031 (99.99%) retained Critic records, totaling 48,263 calls (1.79 per trajectory).

Table R1. Critic output distribution (48,263 calls).

Critic outputCountAll callsSubstantive actions
NONE20,57242.62%—
REVISE12,90326.73%49.99%
REPLAN7,05514.62%27.33%
SEARCH5,84212.10%22.63%
RESTRUCTURE / REWRITE7 / 40.02%0.04%
Empty output / format error1,825 / 553.90%—
Total48,263100%100%

The five substantive repair actions account for 25,811 calls (53.48%), mainly REVISE, REPLAN, and SEARCH. RESTRUCTURE and REWRITE occur only 11 times; we will note their low trigger frequency in the revision. The final column is normalized over substantive repair actions only.

To measure repair effectiveness, we compared trajectory correctness before and after the first repair.

Table R2. Correctness transitions grouped by first repair action (7,428 trajectories).

First repairTrajectoriesIncorrect → CorrectCorrect → IncorrectNet
SEARCH3,7753987+391
REVISE1,93310914+95
REPLAN1,719163+13
RESTRUCTURE1000
Total7,42852324+499

Overall, 7,428 trajectories (27.48%) underwent at least one repair. After the first repair, 523 changed from incorrect to correct and 24 from correct to incorrect, a correction-to-regression ratio of approximately 22:1. Repairs initiated by SEARCH contributed the largest net positive transition (+391, or 78.4% of net corrections), highlighting the role of retrieval-based evidence supplementation in the current repair process.

(2) Evidence support and critical reasoning quality.

We conducted a paired audit on the same questions: 12 questions were sampled without replacement from each of nine datasets (108 total). Search-o1 and TRACE-o1 reasoning trajectories were evaluated using an AI-assisted semantic audit, with recorded final-correctness labels hidden.

The evaluation criteria were:

  • Substantive evidence: relevant external retrieval material is present. Query terms, URLs, empty results, model-generated briefings, and materials supplied with the question are excluded.
  • Evidence support (0/1/2): 2 = sufficient support for decisive facts and the conclusion; 1 = partial support or an unverified reasoning bridge; 0 = evidence conflicts with or cannot support the conclusion.
  • Critical-step quality (0/1/2): 2 = valid critical reasoning; 1 = a non-decisive reasoning gap; 0 = a critical error affecting the final conclusion.

Table R3. Paired evidence and process-quality audit (108 questions; 12 per dataset).

MetricSearch-o1TRACE-o1Difference
Final answer correct61 (56.5%)72 (66.7%)+10.2 pp
Substantive evidence present25 (23.1%)51 (47.2%)+24.1 pp
Answer correct and sufficiently supported14 (13.0%)24 (22.2%)+9.3 pp
Among correct answers, sufficiently supported14/61 (23.0%)24/72 (33.3%)+10.4 pp
Critical reasoning valid (step = 2)66 (61.1%)82 (75.9%)+14.8 pp
Incomplete bridge (step = 1)7 (6.5%)7 (6.5%)0
Decisive step error (step = 0)35 (32.4%)19 (17.6%)−14.8 pp

In the paired comparison, 12 questions changed from incorrect to correct and 1 from correct to incorrect. TRACE-o1 increased substantive evidence from 23.1% to 47.2% and valid critical reasoning from 61.1% to 75.9%, while reducing decisive-step errors from 32.4% to 17.6%.

Together with the Critic logs, these results show differences beyond final answers, extending to evidence acquisition and critical reasoning. We will include the analysis and evaluation setup in the revision. Part 2 discusses the mechanism comparisons supported by these observations and the additional controls needed to isolate stage-specific contributions.

W2 + Q3 | Role of Stage-Wise Control and Comparison with Terminal Self-Correction

(1) Mechanistic distinction between stage-wise control and terminal self-correction.

TRACE-o1 is designed to let the Critic identify problems as reasoning unfolds, use SEARCH to supplement external evidence, and use REPLAN to adjust the subsequent plan so that repaired information continues to contribute to reasoning.

By contrast, terminal Self-Correction without external feedback reviews the answer after reasoning is complete, and its corrections rely mainly on reassessing the information already available to the model. Prior work has shown that this type of intrinsic self-correction does not always improve reasoning performance and may introduce new errors [Huang et al., ICLR 2024].

The process logs provide a related observation (Table R2): repairs initiated by SEARCH account for 50.8% of repaired trajectories and contribute 78.4% of net positive transitions. For this group, the incorrect-to-correct rate is 10.5% (398/3,775), while the correct-to-incorrect rate is 0.19% (7/3,775); for REVISE, the corresponding rates are 5.6% and 0.72%.

These results highlight the role of retrieval-based evidence supplementation in the TRACE-o1 repair process. The independent contributions of stage-wise Critic and terminal Self-Correction require further comparison under fixed evidence and matched resource caps.

(2) Resource statistics for the stage ablation and existing budget comparison.

Table 4 compares the accuracy of different stage combinations on GPQA, MedQA, and Math500. The per-configuration Tokens/Q, Calls/Q, and Searches/Q were not recorded under a common accounting protocol, so we report the accuracy results for these configurations and use Table 3 for the available resource comparison.

Table R4. Token-matched resource comparison (nine-dataset macro average).

MethodKMacro Acc.Tokens/QCalls/QSearches/Q
Search-o1159.1%4,1641.700.35
Search-o1+SC560.5%11,97111.961.75
Search-o1+SC1061.5%38,70816.853.36
TRACE-o1563.6%37,86518.364.34

Under similar token consumption (38,708 vs. 37,865), TRACE-o1 has a 2.04 pp accuracy advantage over SC (K=10). This shows that the performance difference is not explained by token count alone. Because Calls and Searches are not fully matched, we treat this as a token-matched comparison, rather than a strict equal-compute or equal-tool-use comparison. It is also not a direct comparison between intermediate Critic and terminal Self-Correction under fixed evidence.

We will distinguish the performance evidence supported by the current experiments from the stage-specific mechanism comparison that requires a dedicated matched-budget study.

Q2 | Correlated Errors among Candidates, Unanimous Agreement, and Oracle@5

We thank the reviewer for noting that agreement across candidate outputs cannot be equated with correctness.

Under the matched K=5 condition, Table 2 shows that TRACE-o1 reduces mean disagreement from 0.187 to 0.108, reduces the high-disagreement rate from 27.73% to 15.47%, and increases unanimous agreement from 56.66% to 74.56%.

These metrics reflect cross-trajectory answer consistency. They separate consistency from accuracy in our reporting, while shared initialization remains a possible source of correlated errors.

The shared (P, B) may induce correlated errors among candidates. TRACE-o1 addresses this through two design mechanisms (§3.1–3.4): first, the shared plan is used only for initialization, after which each trajectory independently performs Critic checks, retrieval, and local REPLAN; second, when need_posthoc=true, terminal aggregation combines Critic scores and semantic features to rerank candidates rather than relying entirely on majority voting, creating an opportunity to retain a correct minority answer. We will report answer consistency and accuracy separately in the revision.

The current aggregate statistics in Table 2 focus on consistency and accuracy; unanimous-correct rate, unanimous-wrong rate, and Oracle@5 require a separate per-question analysis.

Thank you for the detailed rebuttal. I will maintain my rating of 2.

Reviewer_dP6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE-o1, a training-free framework for improving the reliability of search-augmented reasoning through three stage-aligned components: (1) Evidence Planning, which decomposes questions and prepares retrieval evidence before reasoning; (2) Critic-Audited Search and Repair, which identifies and corrects intermediate reasoning failures; and (3) Consistency-Guided Selection, which aggregates multiple candidate trajectories using critic confidence and semantic consistency. Experiments are conducted on nine benchmarks spanning general QA, mathematical reasoning, and biomedical QA, using two open-weight language models. On Qwen3-4B-Instruct, TRACE-o1 achieves 63.6% macro accuracy, compared with 60.5% for Search-o1 with self-consistency at K=5. Under a similar token budget, it also outperforms Search-o1 with K=10 by approximately 2.0 percentage points. Additional ablations and disagreement analyses suggest that the proposed components can improve reasoning accuracy and prediction consistency.

Critical strengths
  1. Comprehensive empirical evaluation with meaningful baseline controls.

The paper evaluates TRACE-o1 across nine diverse benchmarks and two model backbones. Importantly, the authors include both candidate-count-matched and token-budget-matched comparisons, rather than relying solely on improvements over single-trajectory baselines. The reported paired-bootstrap confidence intervals provide additional statistical support for the accuracy gains. These experiments strengthen the empirical credibility of the proposed framework.

  1. A practical framework for identifying and repairing reasoning failures.

The stage-aligned design addresses different failure points throughout the reasoning process. In particular, the critic-guided repair mechanism provides actionable interventions for missing evidence, logical errors, and context-quality issues. The ablation studies and qualitative examples illustrate how planning and intermediate verification can complement each other. The framework is also applicable without additional model training, which increases its practical relevance.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the contribution of the final selection mechanism.

The three components largely combine established techniques, including query planning, LLM-based critique and iterative refinement, and self-consistency-based aggregation. Although their integration is reasonable, the paper does not clearly establish a sufficiently novel algorithmic insight beyond combining these existing mechanisms.

Furthermore, the contribution of the more sophisticated final reranking mechanism appears limited. In Table 4, adding reranking improves GPQA by 1.0 percentage point and Math500 by 0.4 points, but reduces MedQA accuracy by 1.2 points. Appendix F further reports only a 0.42-point improvement over majority voting for the proposed heuristic selector across three datasets. These results raise questions about whether the additional complexity of the final selection mechanism is justified, and whether a simpler planning-and-critique framework with conventional aggregation could achieve comparable performance.

  1. Incomplete control of inference and retrieval costs limits the strength of the efficiency comparison.

Although the authors provide a token-budget-matched comparison, the computational and retrieval budgets are not fully aligned. As shown in Table 3, TRACE-o1 uses approximately 37.9K tokens, 18.36 model calls, and 4.34 searches per question, while Search-o1 with K=10 uses approximately 38.7K tokens, 16.85 calls, and 3.36 searches. Thus, TRACE-o1 benefits from additional retrieval opportunities and model interactions despite having a similar total token budget.

The remaining 2.0-point accuracy advantage is encouraging, but it is difficult to determine how much comes from the proposed reliability-control mechanisms versus the different search and inference allocation. Moreover, the framework incurs substantially higher costs than the original Search-o1 baseline. More rigorous cost-controlled comparisons, including retrieval budgets and end-to-end latency, are needed to establish its practical efficiency.

Questions for authors
  1. Can the authors provide a more strictly budget-matched comparison?

The current token-matched Search-o1 baseline uses fewer search requests and model calls than TRACE-o1. Would the performance advantage persist when both methods are constrained to comparable token budgets, retrieval calls, and overall inference costs? This would help determine whether the improvements primarily arise from the proposed mechanisms or additional retrieval opportunities.

  1. What is the incremental benefit of the final selection mechanism over simpler alternatives?

Could the authors provide a full nine-benchmark comparison of planning and critic-guided repair combined with (a) majority voting, (b) LLM aggregation only, and (c) the complete Consistency-Guided Selection mechanism? It would also be helpful to report how frequently the post-hoc reranking branch is activated and how often it corrects versus introduces errors. These results would clarify whether the final component is essential to the proposed framework.

  1. Does increased trajectory agreement necessarily correspond to improved reliability?

Table 2 reports a substantial reduction in candidate disagreement. However, all trajectories share an initial evidence plan and briefing, which could also increase correlated errors. Could the authors report the accuracy of unanimous predictions, particularly the rate of unanimously incorrect answers, and compare it with the baseline? This would help distinguish genuine improvements in reasoning reliability from increased agreement caused by shared evidence or reasoning biases.

3 comments

We thank the reviewer for questions on novelty, resource fairness, and the value of final-stage selection. Following the suggestion, we compared three selection strategies on a fixed candidate set across nine datasets (Q2 and Table R3).

W1|Novelty and the incremental value of final-stage selection

The three stages of TRACE-o1 have distinct roles rather than functioning as parallel modules: the shared (P, B) determines the evidence path before generation; each candidate can update its own plan during generation through the Critic and local REPLAN; after generation, candidates are first aggregated, and cluster scoring is invoked only when verification is triggered (§3.1–3.4). This ordering separates within-trajectory correction from selection across candidates. Figure 5's AIME example illustrates how the two mechanisms operate. Our claim is the joint design and coordination of these three control points, rather than the first introduction of planning, Critic, or Self-Consistency as individual operators.

For final-stage reranking, we agree that its benefit is limited and task-dependent. In Table 4, TRACE-o1 (w/o) retains LLM Aggregation and disables only posterior reranking: GPQA changes from 65.7→66.7, Math500 from 79.6→80.0, and MedQA from 76.2→75.0. Appendix F.2 reports a +0.42 pp average improvement of one selector over majority voting on three datasets. In the nine-dataset fixed-candidate analysis in Q2, reranking yields +0.93 pp (95% CI [+0.23, +2.08]) under paired-question bootstrap. This means that final-stage selection explains only a small portion of the +3.08 pp gain of TRACE-o1 over SC, while the main gain comes from the full multi-stage process.

W2|Tokens are close, but Calls and Searches are not matched

As shown in Table 3 of the paper (excerpted in Table R1), the K=10 control is close in recorded Tokens, while Calls and Searches are not matched:

Table R1|Token-matched resource comparison (excerpted from Table 3 of the paper; nine-dataset Macro)

MethodKMacro Acc.Tokens/QCalls/QSearches/Q
Search-o1 (single trajectory)159.1%4,1641.700.35
Search-o1+SC560.5%11,97111.961.75
Search-o1+SC1061.5%38,70816.853.36
TRACE-o1563.6%37,86518.364.34

When the Token budgets are close (38,708 vs. 37,865), doubling the number of SC candidates still leaves SC 2.04 pp behind TRACE-o1. This shows that increasing the SC candidate count alone does not explain the full performance difference. Because TRACE-o1 uses more Calls and Searches, we treat this as a token-matched comparison, rather than a strict equal-compute comparison.

Q1|How can comparisons be made fairly across different resource dimensions?

Because the two methods use different reasoning flows and tool schedules, we use two complementary evaluation protocols: Token-matched comparison evaluates accuracy at similar token budgets, while the matched per-question resource test measures practical runtime overhead.

(1) Full token-matched comparison (Table 3 of the paper, shown in Table R1): We compare accuracy when Token consumption is close and report the differences in Calls and Searches. TRACE-o1 achieves a 2.04 pp higher Macro Accuracy.

(2) Matched per-question resource cap (Table R2): To measure practical runtime overhead, we added a single-stream latency measurement using the first five questions from each of the nine datasets, for a total of 45 questions. Both methods use the same per-question resource cap and search endpoint, with caching disabled, no queueing, and randomized execution order.

Table R2|Single-stream latency and resource consumption under a matched per-question resource cap (45 questions; first five questions from each of nine datasets)

MethodMean wall-clock (s)Median (s)P90 (s)Calls/QPrompt tok/QCompletion tok/QSearches/Q
Search-o1+SC (K≤10)493.4244.11549.416.324,50716,5593.38
TRACE-o1 (K=5)260.3128.6669.512.625,5338,3772.02

We imposed the same per-question resource cap on both methods and measured runtime under the same search conditions. TRACE-o1 reduces mean wall-clock latency by 47.2% while consuming fewer total Tokens. These results clarify practical runtime overhead and the scope of conclusions under distinct resource-matching protocols.

Q2|The incremental contribution of majority voting, aggregation only, and full selection

Using the same completed K=5 candidate sets, we compared three strategies: ① Majority (normalized answer voting, with ties resolved by Critic scores and the decision rule); ② Aggregation Only (directly using the LLM-aggregated answer); and ③ Full Selection (after aggregation, reranking only when need_posthoc=true using ConfSum + λ·min(coh, cen), with λ=8 as in the paper). The results are shown in Table R3.

Table R3|Selector comparison over nine datasets (Qwen3-4B, 5,407 questions, fixed K=5 candidates, paired bootstrap)

ComparisonMacro Acc.Macro difference [95% CI] (pp)Wrong→CorrectCorrect→WrongNet questions
Majority → Aggregation Only62.4% → 62.6%+0.18 [+0.06, +0.33]239+14
Aggregation Only → Full Selection62.6% → 63.6%+0.93 [+0.23, +2.08]7649+27
Majority → Full Selection62.4% → 63.6%+1.12 [+0.05, +2.27]9251+41

need_posthoc is triggered for 13.22% of questions. Reranking changes the correctness status of 125 questions (2.31%), improving six datasets, leaving two unchanged, and reducing one (MedQA, consistent with Table 4). These results show that posterior reranking yields a small gain on a fixed candidate set and that its contribution can be distinguished from the multi-stage reasoning process. We will include these results in the revision.

Q3|Could consistency also reflect shared errors?

The reviewer is right to raise this point. Table 2 shows that, under the K=5 matched condition, TRACE-o1 reduces mean disagreement from 0.187 to 0.108, decreases the high-disagreement rate from 27.73% to 15.47%, and increases the unanimous-consistency rate from 56.66% to 74.56%. These metrics measure cross-trajectory answer consistency.

The shared (P, B) may induce correlated errors among candidates. The paper addresses this through two design mechanisms (§3.1–3.4). First, (P, B) is used only for initialization. Each trajectory then calls the Critic, performs retrieval independently, and can modify its plan through local REPLAN; the shared initial plan therefore does not lock all candidates onto one reasoning path. Second, final-stage selection is not simple majority voting: when need_posthoc=true, it combines Critic scores and semantic features to rerank candidates, creating an opportunity to retain a correct minority answer. We will distinguish answer consistency from accuracy in the revision and report the two quantities separately.

Dear Reviewer dP6L,

Thank you again for your thoughtful review. As the discussion deadline is approaching, could you please let us know whether our responses address your concerns? We would be happy to provide further clarification before the discussion closes if any issues remain.

Thank you for your time and consideration.

Warm regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.