acceptodds
Under review as a conference paper at ICLR 2027

Logic Signatures for Test-Time LLM Reasoning Routing

Abstract

Reasoning in Large Language Models (LLMs) is commonly summarized by benchmark scores and improved with a uniform inference recipe. However, this scalar view neglects the interaction between a query’s logical demands and a particular solver’s failure tendencies. We hypothesize a failure-to-control principle: once a solver’s failures are indexed by the reasoning operations that elicit them, its error history can be inverted into a query–solver-specific policy for selecting assistance before the same vulnerabilities recur. To test this hypothesis, we propose LOGICROUTE, a training-free framework for pre-response intervention routing. At its core is LOGICSIG, a sparse two-sided representation that separates what the query requires from where the solver tends to fail. Using offline zero-shot traces, LOGICROUTE constructs a solver-specific logic profile, organizes candidate interventions into general reasoning scaffolds and targeted repairs, and routes each query before the target model responds. Across five Qwen models spanning 1.7B–27B parameters on MATH500, AIME, GPQA, and MMLU-Pro, LOGICROUTE improves macro accuracy from 51.00% to 62.51%, outperforming semantic demonstration retrieval by 8.19 percentage points. Across the evaluated Qwen, Llama, and Mistral models, macro accuracy increases from 39.55% to 51.79%. Profile substitutions show that useful routes are solver-specific; route perturbations support the utility of logic-feature ordering; and demonstration-bank ablations show that assistance is compositional but not monotonic in context volume. Together, these findings suggest that failures in the past provide a reusable pre-response control signal for targeted assistance without parameter updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, In-Context Learning, Test-Time Reasoning, Failure Profiling, Dynamic Routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XN5yOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LogicRoute, a training-free framework that uses LogicSig to characterize query-specific reasoning requirements and model-specific failure patterns. It dynamically selects reasoning demonstrations before inference. Experiments across multiple LLMs and reasoning benchmarks demonstrate substantial accuracy improvements.

Critical strengths
  1. The solver-specific reasoning framework offers an interesting perspective on adaptive test-time assistance.
  2. Experiments cover four reasoning benchmarks and seven LLMs from different model families.
  3. The method achieves substantial accuracy improvements over zero-shot and demonstration retrieval baselines.
  4. Extensive ablations investigate model-specific profiles, routing strategies, and demonstration-bank configurations.
Critical weaknesses
  1. The individual contributions of query-side and failure-side logic signatures are not sufficiently isolated.
  2. The framework relies heavily on stronger LLMs for annotation and demonstration construction.
  3. End-to-end computational costs, including calibration and routing overhead, are not adequately evaluated.
  4. Statistical significance analysis is missing, and some evaluation subsets are relatively small.
Questions for authors
  1. Can the authors separately ablate query-side and failure-side logic signatures?
  2. How sensitive is the framework to the quality of diagnostic annotations and generated demonstrations?
  3. What are the end-to-end inference costs compared with the evaluated baselines?
  4. Can the authors provide statistical significance tests and comparisons with stronger adaptive demonstration selection methods?
2 comments

Thank you for your careful assessment and for the suggestions on ablations, construction sensitivity, and computational cost. We have added direct controls, a construction-model comparison, routing-token estimates, and evaluations on larger test sets.

Weakness #1 / Question #1. Contributions of the two signatures

Following your suggestion, we have added query-only (β = 0) and failure-only (α = 0) ablations, keeping the target solver, demonstration bank, feature extractor, and distance calculation fixed except for the ablated weight. Appendix C.5, Table 10, reports the results across five Qwen models and four benchmarks.

Feature-space routing configurationMacro accuracy (%)
Failure-side matching only, α = 053.74
Query-side matching only, β = 054.71
Joint matching, LogicRoute-Light55.93

Joint matching improves macro accuracy by 1.22 percentage points over query-only matching and 2.19 points over failure-only matching. With the profile-informed bank held fixed, these comparisons show how each matching term contributes to pre-response demonstration selection.

Weakness #2 / Question #2. Dependence on stronger LLMs and construction sensitivity

We agree that the role of stronger LLMs deserves explicit discussion. GPT-5.2 supplies the diagnostic annotations and default demonstrations during offline construction. Appendix C.4, Table 9, examines sensitivity at two stages. Across the four Qwen solvers in this comparison, replacing the test-time feature extractor with Qwen3.5-35B-A3B increases LogicRoute-Light's macro accuracy from 55.93% to 57.43%. The bank and distance weights are fixed in this comparison.

For offline construction, we replace GPT-5.2 with GPT-6.1-Sol to generate both the demonstration sets and routing instructions for LogicRoute-LLM. The frozen target solver performs routing and answer generation in both settings. The table below averages each solver's accuracy across the four benchmarks in Table 9.

Target solverGPT-5.2 construction, accuracy (%)GPT-6.1-Sol construction, accuracy (%)
Qwen3-14B64.2962.97
Qwen3-8B61.3160.39
Qwen3-4B56.9557.27
Qwen3-1.7B48.4549.09
Macro57.7557.43

Across the four Qwen solvers and four benchmarks, replacing GPT-5.2 with GPT-6.1-Sol reduces macro accuracy from 57.75% to 57.43%, a difference of 0.32 percentage points. Qwen3-4B and Qwen3-1.7B improve, and Qwen3-14B and Qwen3-8B decline. The sensitivity of pre-response routing to this bank-construction change varies across solvers.

Weakness #3 / Question #3. Calibration, routing, and inference costs

We agree that the computational cost needs clearer accounting. Appendix C.9, Table 14, adds routing-output estimates to the answer-token measurements. The estimates use each solver's tokenizer and are normalized to one routing call and one answer generation. Across the five Qwen solvers and four benchmarks, the comparison is as follows.

MethodMacro accuracy (%)Mean answer-output tokensRouting + one-answer output tokens
Zero-shot50.981,981.31,981.3
Fixed CoT48.081,789.91,789.9
DemRAG54.341,901.21,901.2
LogicRoute-Light55.931,820.11,909.0
LogicRoute-LLM62.501,811.71,904.5

Routing accounts for approximately 5% of LogicRoute's normalized output-token totals. Its routing counts are estimated by re-tokenizing reconstructed routing JSON.

Question #4. Adaptive selection baselines

Following your suggestion, we have added a bank-routing adaptation of Pattern-Aware CoT (PA-CoT) and a simple error-aware selector, Error-Bridge, using the same target solver and candidate bank as LogicRoute. PA-CoT selects a strategy set based on reasoning patterns extracted from an initial zero-shot rationale. Error-Bridge retrieves similar calibration questions that the solver answered incorrectly, then selects the strategy set whose examples best match those questions.

Sec. 4.1 describes both baselines. Table 1 reports their macro accuracies across five Qwen models and four benchmarks, with per-model results in Appendix C.2, Table 6.

MethodMacro accuracy (%)
Random ICL52.80±1.91
DemRAG54.34±1.51
PA-CoT55.26±1.51
Error-Bridge54.63±1.72
LogicRoute-LLM62.50±2.30

LogicRoute-LLM exceeds PA-CoT and Error-Bridge by 7.24 and 7.87 percentage points, respectively. In the cross-architecture comparison over Qwen3-4B, Llama3.1-8B-Instruct, and Mistral3-3B-Instruct, it reaches 51.77%, compared with 41.32% for PA-CoT and 41.76% for Error-Bridge (Table 2).

Weakness #4 / Question #4. Larger test sets

Thank you for suggesting a broader evaluation. We have expanded AIME from 30 to 60 questions by adding 30 AIME 2026 problems, and GPQA from 88 to 128 questions by adding 40 GPQA Extended questions after excluding all GPQA-Main questions. On the newly added GPQA questions alone, LogicRoute-LLM averages 52.08%, compared with 43.75% for DemRAG, the strongest baseline on average. On AIME 2026 alone, it averages 15.56% and ranks first or ties for first for three solvers.

Appendix C.8, Table 13, reports the following accuracies (%) on the full expanded test sets. For each solver and benchmark, the best-baseline value is the highest accuracy among the seven baselines.

Target solverAIME (60), best baselineAIME (60), LogicRoute-LLMGPQA (128), best baselineGPQA (128), LogicRoute-LLM
Qwen3-14B25.0030.0058.5958.59
Qwen3-8B20.0025.0045.3153.12
Qwen3-4B23.3325.0041.4149.22
Qwen3-1.7B15.0015.0031.2541.41
Llama3.1-8B-Instruct5.006.6739.8447.66
Mistral3-3B-Instruct20.0025.0063.2867.19

Tables 1–2 now include variability from three independent answer-generation runs on fixed evaluation splits. The central accuracy uses majority aggregation of the three correctness labels for each question. The SD values average the sample SDs of the three run-level accuracies across the corresponding model–benchmark pairs. For example, Table 1 reports 62.50% majority-aggregated accuracy with an average run-level SD of 2.30 percentage points for LogicRoute-LLM, and 50.98% with 2.86 points for Zero-shot.

Weakness #4 / Question #4. Statistical evidence

We also completed paired significance tests on the common support of 20 model–benchmark cells, weighting cells equally and paired questions equally within each cell. We use one-sided robust mean t-tests with Welch–Satterthwaite degrees of freedom, stratified by dataset and clustered by dataset/question index. Each question cluster includes all runs, reroutes, and model evaluations of that question. Holm correction jointly covers the 12 comparisons between two router configurations and six baselines.

The table reports paired gains in percentage points, per-comparison two-sided 95% question-clustered t intervals, and jointly Holm-adjusted p-values.

BaselineLogicRoute-Light gain [95% CI], ppHolm pLogicRoute-LLM gain [95% CI], ppHolm p
Zero-shot4.593 [2.211, 6.976]1.07e-49.806 [6.752, 12.860]2.36e-8
Fixed CoT7.440 [5.287, 9.594]7.21e-1012.653 [9.774, 15.532]3.15e-12
Fixed CoD12.997 [10.535, 15.459]4.03e-1918.210 [15.138, 21.282]4.61e-19
DemRAG6.396 [4.020, 8.772]8.48e-711.609 [9.234, 13.983]2.25e-14
Error-Bridge6.765 [4.594, 8.937]4.46e-811.978 [9.196, 14.760]1.88e-11
PA-CoT7.544 [5.320, 9.769]3.65e-912.757 [10.008, 15.506]7.78e-13

Both router configurations outperform all six baselines in the aggregate tests after joint Holm correction, with all adjusted p-values at or below 1.07e-4.

Thank you again for helping us make the comparisons more direct and clarify the scope of the evaluation.

Reviewer_YneJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies inference-time demonstration selection: choosing the most suitable few-shot demonstrations for a particular model and query without updating parameters. Offline, the target model first answers calibration questions zero-shot, and a strong diagnostic model reads each question and reasoning trace to annotate two kinds of logic signatures: which reasoning operations the question requires (query side) and where the model fails (failure side). These are aggregated into a "solver profile" of the model's accuracy under different logic patterns, which is used to build a demonstration bank of three general sets and three targeted repair sets, each with a routing description. At test time, the target model itself reads the question and the set descriptions to choose one set (LLM router), or a set is chosen by logic-feature distance (feature-space router), and the model then answers with the chosen demonstrations. On MATH500, AIME, GPQA and MMLU-Pro, the macro accuracy of five Qwen models rises from 51.00% zero-shot to 62.51%, 8.19 points above semantic demonstration retrieval (54.32%) and about 10 points above random selection from the same bank (52.80%), with similar gains across Qwen, Llama and Mistral models.

Critical strengths

The control is clean, and the gains are clear and consistent across models. Random ICL and LogicRoute draw from the same demonstration bank, so the difference between 52.80% and 62.51% macro accuracy comes only from how demonstrations are selected; LogicRoute is also 8.19 points above semantic demonstration retrieval. The direction of the results is consistent across five Qwen models as well as Llama and Mistral, without longer generations. This shows that selecting demonstrations for a specific model is effective, and it is the main argument in favor of acceptance.

Critical weaknesses
  1. The contribution of the failure-side information is not isolated, and the closest methods are not compared against.

(a) The central claim is that a model's failure history helps select demonstrations, but there is no ablation using only query-side information (β = 0); at test time, the LLM router sees only the question and the set descriptions, and the failure information enters only indirectly through the repair sets.

(b) The only demonstration-selection baselines are DPR semantic retrieval and random selection. Two closer lines of work that the paper itself cites are not compared: Pattern-Aware CoT, which selects demonstrations by reasoning pattern, and methods that retrieve demonstrations from the model's own errors (e.g., EdgeLM).

This determines my assessment of novelty and effectiveness: without this ablation and these comparisons, it is not possible to tell how much of the gain comes from the failure-side information and how much any reasonable demonstration-selection method would achieve.

  1. The statistical evidence is limited.

The test sets are small (e.g., 30 AIME and 88 GPQA questions), each result comes from a single data split with no variance or significance tests, and some ablation conclusions rest on differences of 1–3 points.

This affects how much I trust the ablation conclusions: the roughly 10-point gain over random selection is fairly convincing, but the mechanistic conclusions are not yet adequately supported.

Questions for authors
  1. Could you add an ablation that uses only query-side information (β = 0)?

  2. Could you compare, with the same bank and model, against Pattern-Aware CoT and a simple error-aware baseline that retrieves calibration questions similar to the test question that the target model answered incorrectly?

  3. Could you report the mean and standard deviation over multiple data splits or runs for the main results?

  4. Please report the values of α and β in the feature-space router and the versions of the GPT models used for diagnostic annotation and demonstration generation.

  5. Qwen3-14B, used to judge answers, is also one of the evaluated models; please state whether the judgments were spot-checked.

3 comments

Thank you for your careful assessment and for identifying the controls needed to assess the contribution of failure history. Your comments helped us add single-side routing ablations, closer demonstration-selection baselines, and evaluations on larger test sets.

Weakness #1(a) / Question #1. Contribution of failure-side information

We agree that this contribution needs a direct control. We have added query-only (β = 0) and failure-only (α = 0) ablations, keeping the target solver, demonstration bank, feature extractor, and distance calculation fixed except for the ablated weight. Appendix C.5, Table 10, reports the results across five Qwen models and four benchmarks, with each model–benchmark pair weighted equally.

Feature-space routing configurationMacro accuracy (%)
Failure-side matching only, α = 053.74
Query-side matching only, β = 054.71
Joint matching, LogicRoute-Light55.93

Joint matching improves macro accuracy by 1.22 percentage points over query-only matching and 2.19 points over failure-only matching. As you point out, in LogicRoute-LLM, failure history informs the repair sets and their descriptions. The LogicRoute-Light comparison isolates the contribution of each matching term when selecting from a fixed, failure-informed bank.

Weakness #1(b) / Question #2. Comparisons with closer selection methods

Following your suggestion, we have added a bank-routing adaptation of Pattern-Aware CoT (PA-CoT) and a simple error-aware selector, Error-Bridge, using the same target solver and candidate bank as LogicRoute. PA-CoT extracts reasoning patterns from an initial zero-shot rationale and uses them to select a strategy set. This preliminary generation adds a stage to its selection procedure. Error-Bridge retrieves calibration questions similar to the test query that the target model answered incorrectly, then selects the strategy set whose examples best match those retrieved questions.

Sec. 4.1 describes both baselines. Table 1 reports their macro accuracies across five Qwen models and four benchmarks, with per-model results in Appendix C.2, Table 6.

MethodMacro accuracy (%)
Random ICL52.80±1.91
DemRAG54.34±1.51
PA-CoT55.26±1.51
Error-Bridge54.63±1.72
LogicRoute-LLM62.50±2.30

LogicRoute-LLM exceeds PA-CoT and Error-Bridge by 7.24 and 7.87 percentage points, respectively. In the cross-architecture comparison over Qwen3-4B, Llama3.1-8B-Instruct, and Mistral3-3B-Instruct, it reaches 51.77%, compared with 41.32% for PA-CoT and 41.76% for Error-Bridge (Table 2). Your suggested controls help us evaluate our main contribution, selecting demonstrations for a frozen solver based on the query’s demands and the solver’s prior failures.

Weakness #2 / Question #3. Statistical evidence and test-set size

We appreciate your concern about the small test sets and the interpretation of small ablation differences. We have added three-run generation-variability summaries in Tables 1–2 and expanded the AIME and GPQA evaluations.

The value after ± is the equally weighted average of the sample SDs of the three run-level accuracies across the corresponding model–benchmark pairs. For example, Table 1 reports 62.50% accuracy with an average run-level SD of 2.30 percentage points for LogicRoute-LLM, and 50.98% with 2.86 points for Zero-shot.

For the larger evaluation, we added 30 AIME 2026 problems to the 30 AIME 2025 problems, and 40 GPQA Extended questions after excluding all GPQA-Main questions. The expanded test sets contain 60 AIME questions and 128 GPQA questions. Evaluation covers four Qwen models from 1.7B to 14B, Llama3.1-8B-Instruct, and Mistral3-3B-Instruct.

Appendix C.8, Table 13, reports the following full-set accuracies (%). For each solver and benchmark, the best-baseline value is the highest accuracy among the seven baselines.

Target solverAIME (60), best baselineAIME (60), LogicRoute-LLMGPQA (128), best baselineGPQA (128), LogicRoute-LLM
Qwen3-14B25.0030.0058.5958.59
Qwen3-8B20.0025.0045.3153.12
Qwen3-4B23.3325.0041.4149.22
Qwen3-1.7B15.0015.0031.2541.41
Llama3.1-8B-Instruct5.006.6739.8447.66
Mistral3-3B-Instruct20.0025.0063.2867.19

LogicRoute-LLM ranks first or ties for first for all six solvers on both expanded test sets. On the 40 newly added GPQA questions alone, it averages 52.08%, exceeding DemRAG, the strongest baseline on average, at 43.75% by 8.33 percentage points. On AIME 2026 alone, it averages 15.56% and ranks first or ties for first for three solvers.

Question #4. Routing weights and GPT versions

The feature-space router uses α = 0.45 and β = 0.20, as specified in Sec. 3.4.2. Diagnostic annotation and default demonstration generation use GPT-5.2. We have also replaced the generic “GPT models” wording in Appendix C.1 with this model version.

At test time, the default routers use the frozen target model itself for feature extraction in LogicRoute-Light and strategy selection in LogicRoute-LLM, as described in Sec. 3.4.

We have also added an ablation that replaces GPT-5.2 with GPT-6.1-Sol to generate both the demonstration sets and routing instructions (Appendix C.4, Table 9).

Question #5. Spot-checking Qwen3-14B's judgments

Yes. We manually spot-checked 3% of the answer judgments against the reference answers, with assistance from Qwen3.5-35B-A3B. The check identified cases in which overly long responses were incorrectly marked as correct, and we corrected the affected labels.

Within the reviewed sample, we found no clear evidence of preferential grading of Qwen3-14B's own outputs. We acknowledge the limited coverage of this check.

We have included the description of manual-review rate, model assistance, and label-correction procedure in Appendix C.1 alongside the evaluator description.

Thank you again for the suggestions that helped us make the comparisons more direct.

Thank you for the additional experiments. The new baselines and expanded evaluations make the gains of the full LogicRoute pipeline more convincing.

The method builds on familiar techniques for labeling reasoning patterns and selecting demonstrations. Its most distinctive idea is using the target model’s past failure types to construct and select corrective examples. The new ablation shows a 1.22-point improvement from failure-side matching in LogicRoute-Light, with the failure-informed bank held fixed. I would still like to see the strongest LogicRoute-LLM variant evaluated with and without failure history in bank construction, under the same budget.

Direct comparisons with closely related methods also remain limited. The PA-CoT adaptation is useful for comparing selection rules within a shared bank. An end-to-end comparison that lets PA-CoT build and select its own demonstrations under a comparable inference budget would better establish the practical advantage.

Given the limited evidence for the paper’s most distinctive contribution, I maintain my rating.

Reviewer_iW8eOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper identifies a limitation of the scalar view, which overlooks the interaction between a query’s logical demands and the failure tendencies of a particular solver. To address this limitation, the authors propose LOGICSIG, a framework that leverages offline solver-specific logic profiles to guide LLM reasoning trajectories.

Critical strengths
  1. The hypothesisi is interesting.

  2. The performance is promising.

Critical weaknesses
  1. The hypothesis that failures indexed by reasoning operations provide reusable information for selecting assistance is not justified and supported by authors.

  2. The paper also do not provide sufficient motivation for the proposed method.

  3. The proposed method is not clearly illustrated.

  4. The paper do not provide detailed experimental details.

Questions for authors
  1. How to choose the offline calibration set? Any more details?

  2. The paper mentioned use stronger diagnostic model for analysis. what stronger model you choose? And how to ensure the quality of the results.

  3. what are the prompts for each steps in the proposed method?

2 comments

Thank you for your assessment and concerns about the motivation, supporting evidence, and reproducibility. We address the four weaknesses and three questions individually below.

Weakness #1. Insufficient support for reusing failures to select assistance

We agree that this hypothesis calls for a controlled test. Section 4.4 and Figure 3 provide initial evidence through solver-transfer and routing comparisons. We have added single-side routing ablations that vary only the matching weight, keeping the target solver, demonstration bank, feature extractor, and distance calculation fixed. Appendix C.5, Table 10, reports results averaged equally across five Qwen solvers and four benchmarks.

Feature-space routing configurationMacro accuracy (%)
Failure-side matching only, α = 053.74
Query-side matching only, β = 054.71
Joint matching, LogicRoute-Light55.93

Joint matching improves macro accuracy by 1.22 percentage points over query-only matching and 2.19 points over failure-only matching. This controlled comparison provides empirical support for reusing calibration failures to select assistance for new queries. The result applies to routing within the fixed bank, which already incorporates failure information. We hope the added comparison addresses your concern about empirical support.

Weakness #2. Insufficient motivation

Thank you for highlighting the need for a clearer motivation. Section 1 explains that aggregate accuracy can obscure a solver's particular vulnerabilities and that the same demonstration can help one query–solver pair while distracting another. Diagnosing these vulnerabilities is important in its own right. Our main contribution is to use that diagnosis to select a demonstration set before a frozen solver answers a new query. Showing how diagnostic evidence can guide that choice is an essential part of a study of adaptive reasoning assistance, and the controlled comparison above provides evidence for this step.

Weakness #3. Unclear illustration of the method

We appreciate the opportunity to clarify the workflow. Figure 1 shows how offline calibration informs test-time selection, and Figure 2 illustrates how the signatures and demonstration bank are constructed. Sections 3.2–3.4 and Appendix D give the implementation and construction details. The workflow has three stages.

  1. Offline characterization. The frozen target solver generates zero-shot calibration traces. The diagnostic model annotates the operations required by each query and the failures observed in its trace.
  2. Demonstration construction. The default bank contains general strategy sets and targeted repair sets. Source failures are rewritten as correct demonstrations, audited, and paired with routing descriptions.
  3. Pre-response routing. For each new query, LogicRoute-LLM uses the target solver to select a set from its descriptions. LogicRoute-Light uses the target solver to extract logic features and applies a fixed distance rule. The selected complete set is then concatenated with the query for answer generation.

Weakness #4. Insufficient experimental details

We agree that the protocol should be explicit. Appendix C.1 specifies benchmark partitions, model versions, implementation, inference settings, generation-seed handling, and correctness-label aggregation. Target models remain frozen, with their built-in deep-thinking mode disabled.

Each test question receives three separately generated answers. Each answer is judged against the reference, and the question is counted as correct when at least two correctness labels are positive. Appendix C.2 provides per-model results; Appendices C.3–C.7 detail the ablations; and Appendix D describes annotation and demonstration construction.

Question #1. Selection of the offline calibration set

Thank you for requesting clarification. We use the benchmark partitions specified in Appendix C.1:

BenchmarkCalibration poolOriginal held-out test setSelection procedure
MATH500400100A 4:1 split, approximately preserving difficulty distribution
AIME10330Problems from 2021–2024 for calibration; AIME 2025 for testing
GPQA-Main36088Partition of the 448 questions
MMLU-Pro963241A 1,204-example subset obtained through stratified interval sampling across ten regularly spaced portions of the benchmark, followed by an approximately 4:1 split

The benchmark questions undergo no augmentation or manual filtering. Each calibration query receives a zero-shot response from the target solver. These records support diagnostic annotation, profiling, and demonstration construction. The original calibration partitions are retained for the expanded evaluations in Appendix C.8.

Question #2. Diagnostic model and quality assurance

Thank you for raising this reproducibility concern. GPT-5.2 supplies the diagnostic annotations and default demonstration generation. The quality checks operate at distinct stages:

  • Ontology assessment: After each update, a separate call to the same diagnostic model scores new or revised labels for grounding, clarity, distinctiveness, and generality on a 1–5 scale, and assesses query-side and failure-side coverage (Appendix D.1).
  • Demonstration auditing: A separate call checks completeness, reasoning correctness, grounded premises, and consistency with the reference answer. Failed solutions are regenerated using the audit feedback (Appendix D.2).
  • Benchmark answer grading: Qwen3-14B evaluates generated answers against references. We manually spot-checked 3% of the judgments, assisted by Qwen3.5-35B-A3B, and corrected identified errors (Appendix C.1).

We have also added a construction-model sensitivity experiment. Replacing GPT-5.2 with GPT-6.1-Sol for both the demonstration bank and strategy descriptions changes LogicRoute-LLM's macro accuracy from 57.75% to 57.43% across four Qwen solvers and four benchmarks (Appendix C.4, Table 9). Effects vary across solvers, so this supports limited sensitivity under the tested substitution.

Question #3. Prompts for each step

We agree that clear prompt documentation is important. Appendix D describes the offline prompt specifications, while Sections 3.2 and 3.4 explain the annotation and routing interfaces. Offline prompts guide ontology refinement, query-side and failure-side annotation, general strategy selection, profile-guided repair selection, and demonstration synthesis and auditing. The resulting strategies and demonstrations are then used to construct and refine the routing catalog.

At test time, LogicRoute-LLM receives the query and routing catalog and returns a strategy index with a brief rationale. LogicRoute-Light extracts query-side and failure-side features and applies the specified distance rule. The selected demonstration set is then concatenated with the query for answer generation. Profile aggregation and distance calculations do not require additional LLM decisions.

Due to space constraints, we do not reproduce every complete prompt template in the manuscript. Readers seeking further detail can examine the prompt templates and their implementation in the code.

Thank you again for your feedback. We would welcome any specific examples of passages or protocol details where further clarification would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.