- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LogicRoute, a training-free framework that uses LogicSig to characterize query-specific reasoning requirements and model-specific failure patterns. It dynamically selects reasoning demonstrations before inference. Experiments across multiple LLMs and reasoning benchmarks demonstrate substantial accuracy improvements.
Critical strengths
- The solver-specific reasoning framework offers an interesting perspective on adaptive test-time assistance.
- Experiments cover four reasoning benchmarks and seven LLMs from different model families.
- The method achieves substantial accuracy improvements over zero-shot and demonstration retrieval baselines.
- Extensive ablations investigate model-specific profiles, routing strategies, and demonstration-bank configurations.
Critical weaknesses
- The individual contributions of query-side and failure-side logic signatures are not sufficiently isolated.
- The framework relies heavily on stronger LLMs for annotation and demonstration construction.
- End-to-end computational costs, including calibration and routing overhead, are not adequately evaluated.
- Statistical significance analysis is missing, and some evaluation subsets are relatively small.
Questions for authors
- Can the authors separately ablate query-side and failure-side logic signatures?
- How sensitive is the framework to the quality of diagnostic annotations and generated demonstrations?
- What are the end-to-end inference costs compared with the evaluated baselines?
- Can the authors provide statistical significance tests and comparisons with stronger adaptive demonstration selection methods?
2 comments
Question #4. Adaptive selection baselines
Following your suggestion, we have added a bank-routing adaptation of Pattern-Aware CoT (PA-CoT) and a simple error-aware selector, Error-Bridge, using the same target solver and candidate bank as LogicRoute. PA-CoT selects a strategy set based on reasoning patterns extracted from an initial zero-shot rationale. Error-Bridge retrieves similar calibration questions that the solver answered incorrectly, then selects the strategy set whose examples best match those questions.
Sec. 4.1 describes both baselines. Table 1 reports their macro accuracies across five Qwen models and four benchmarks, with per-model results in Appendix C.2, Table 6.
| Method | Macro accuracy (%) |
|---|---|
| Random ICL | 52.80±1.91 |
| DemRAG | 54.34±1.51 |
| PA-CoT | 55.26±1.51 |
| Error-Bridge | 54.63±1.72 |
| LogicRoute-LLM | 62.50±2.30 |
LogicRoute-LLM exceeds PA-CoT and Error-Bridge by 7.24 and 7.87 percentage points, respectively. In the cross-architecture comparison over Qwen3-4B, Llama3.1-8B-Instruct, and Mistral3-3B-Instruct, it reaches 51.77%, compared with 41.32% for PA-CoT and 41.76% for Error-Bridge (Table 2).
Weakness #4 / Question #4. Larger test sets
Thank you for suggesting a broader evaluation. We have expanded AIME from 30 to 60 questions by adding 30 AIME 2026 problems, and GPQA from 88 to 128 questions by adding 40 GPQA Extended questions after excluding all GPQA-Main questions. On the newly added GPQA questions alone, LogicRoute-LLM averages 52.08%, compared with 43.75% for DemRAG, the strongest baseline on average. On AIME 2026 alone, it averages 15.56% and ranks first or ties for first for three solvers.
Appendix C.8, Table 13, reports the following accuracies (%) on the full expanded test sets. For each solver and benchmark, the best-baseline value is the highest accuracy among the seven baselines.
| Target solver | AIME (60), best baseline | AIME (60), LogicRoute-LLM | GPQA (128), best baseline | GPQA (128), LogicRoute-LLM |
|---|---|---|---|---|
| Qwen3-14B | 25.00 | 30.00 | 58.59 | 58.59 |
| Qwen3-8B | 20.00 | 25.00 | 45.31 | 53.12 |
| Qwen3-4B | 23.33 | 25.00 | 41.41 | 49.22 |
| Qwen3-1.7B | 15.00 | 15.00 | 31.25 | 41.41 |
| Llama3.1-8B-Instruct | 5.00 | 6.67 | 39.84 | 47.66 |
| Mistral3-3B-Instruct | 20.00 | 25.00 | 63.28 | 67.19 |
Tables 1–2 now include variability from three independent answer-generation runs on fixed evaluation splits. The central accuracy uses majority aggregation of the three correctness labels for each question. The SD values average the sample SDs of the three run-level accuracies across the corresponding model–benchmark pairs. For example, Table 1 reports 62.50% majority-aggregated accuracy with an average run-level SD of 2.30 percentage points for LogicRoute-LLM, and 50.98% with 2.86 points for Zero-shot.
Weakness #4 / Question #4. Statistical evidence
We also completed paired significance tests on the common support of 20 model–benchmark cells, weighting cells equally and paired questions equally within each cell. We use one-sided robust mean t-tests with Welch–Satterthwaite degrees of freedom, stratified by dataset and clustered by dataset/question index. Each question cluster includes all runs, reroutes, and model evaluations of that question. Holm correction jointly covers the 12 comparisons between two router configurations and six baselines.
The table reports paired gains in percentage points, per-comparison two-sided 95% question-clustered t intervals, and jointly Holm-adjusted p-values.
| Baseline | LogicRoute-Light gain [95% CI], pp | Holm p | LogicRoute-LLM gain [95% CI], pp | Holm p |
|---|---|---|---|---|
| Zero-shot | 4.593 [2.211, 6.976] | 1.07e-4 | 9.806 [6.752, 12.860] | 2.36e-8 |
| Fixed CoT | 7.440 [5.287, 9.594] | 7.21e-10 | 12.653 [9.774, 15.532] | 3.15e-12 |
| Fixed CoD | 12.997 [10.535, 15.459] | 4.03e-19 | 18.210 [15.138, 21.282] | 4.61e-19 |
| DemRAG | 6.396 [4.020, 8.772] | 8.48e-7 | 11.609 [9.234, 13.983] | 2.25e-14 |
| Error-Bridge | 6.765 [4.594, 8.937] | 4.46e-8 | 11.978 [9.196, 14.760] | 1.88e-11 |
| PA-CoT | 7.544 [5.320, 9.769] | 3.65e-9 | 12.757 [10.008, 15.506] | 7.78e-13 |
Both router configurations outperform all six baselines in the aggregate tests after joint Holm correction, with all adjusted p-values at or below 1.07e-4.
Thank you again for helping us make the comparisons more direct and clarify the scope of the evaluation.
Thank you for your careful assessment and for the suggestions on ablations, construction sensitivity, and computational cost. We have added direct controls, a construction-model comparison, routing-token estimates, and evaluations on larger test sets.
Weakness #1 / Question #1. Contributions of the two signatures
Following your suggestion, we have added query-only (β = 0) and failure-only (α = 0) ablations, keeping the target solver, demonstration bank, feature extractor, and distance calculation fixed except for the ablated weight. Appendix C.5, Table 10, reports the results across five Qwen models and four benchmarks.
Joint matching improves macro accuracy by 1.22 percentage points over query-only matching and 2.19 points over failure-only matching. With the profile-informed bank held fixed, these comparisons show how each matching term contributes to pre-response demonstration selection.
Weakness #2 / Question #2. Dependence on stronger LLMs and construction sensitivity
We agree that the role of stronger LLMs deserves explicit discussion. GPT-5.2 supplies the diagnostic annotations and default demonstrations during offline construction. Appendix C.4, Table 9, examines sensitivity at two stages. Across the four Qwen solvers in this comparison, replacing the test-time feature extractor with Qwen3.5-35B-A3B increases LogicRoute-Light's macro accuracy from 55.93% to 57.43%. The bank and distance weights are fixed in this comparison.
For offline construction, we replace GPT-5.2 with GPT-6.1-Sol to generate both the demonstration sets and routing instructions for LogicRoute-LLM. The frozen target solver performs routing and answer generation in both settings. The table below averages each solver's accuracy across the four benchmarks in Table 9.
Across the four Qwen solvers and four benchmarks, replacing GPT-5.2 with GPT-6.1-Sol reduces macro accuracy from 57.75% to 57.43%, a difference of 0.32 percentage points. Qwen3-4B and Qwen3-1.7B improve, and Qwen3-14B and Qwen3-8B decline. The sensitivity of pre-response routing to this bank-construction change varies across solvers.
Weakness #3 / Question #3. Calibration, routing, and inference costs
We agree that the computational cost needs clearer accounting. Appendix C.9, Table 14, adds routing-output estimates to the answer-token measurements. The estimates use each solver's tokenizer and are normalized to one routing call and one answer generation. Across the five Qwen solvers and four benchmarks, the comparison is as follows.
Routing accounts for approximately 5% of LogicRoute's normalized output-token totals. Its routing counts are estimated by re-tokenizing reconstructed routing JSON.