acceptodds
Under review as a conference paper at ICLR 2027

Think Forward, Eliminate Backward: Training and Routing Complementary Reasoning Strategies in LLMs

Abstract

Large language models are trained almost exclusively on forward chain-of-thought (CoT): reasoning proceeds from premises toward a conclusion. Yet a second natural mode of human reasoning — elimination — proceeds by generating candidate answers, evaluating each against the problem's constraints, ruling out the incorrect ones, and converging on the survivor. Prior work has shown that LLMs prompted to reason by elimination perform poorly , but whether elimination can be trained into a model as a learned reasoning strategy, and whether the two strategies are complementary at the problem level, remains underexplored. We fine-tune Qwen3-8B with LoRA on teacher-generated elimination traces (generate candidates evaluate rule out converge) and forward CoT traces, and study the resulting strategy space. We find that (i) elimination reasoning can be successfully trained into an 8B model, (ii) the two strategies exhibit substantial complementarity — 5–29% of problems per dataset are solved by exactly one strategy — and (iii) a lightweight per-query router over TF-IDF and hand-crafted features yields a +2.4% gain over the best single strategy and exceeds the dataset-level oracle, at a modest 13% increase in token cost. Unlike model-level routing , our router selects between reasoning strategies on the same model, adding no deployment cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
chain-of-thought, elimination reasoning, reasoning strategies, model routing, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_y9gaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigated training process-of-elimination reasoning into Qwen3-8B via LoRA SFT on teacher-generated traces from Qwen3-32B, alongside standard forward CoT reasoning. The authors evaluated complementarity between the two strategies across 4 math and science benchmarks and trained a lightweight feature-based logistic regression router to select between forward and elimination reasoning per query.

Critical strengths

Experimental results across a few benchmarks and different configurations provide a structured comparison of strategy complementarity, token cost overhead, and self-consistency baselines.

Critical weaknesses
  1. This paper merely explored a single Qwen3-8B model, which narrows the paper's contribution.

  2. The router evaluation suffers from severe data leakage and methodological flaws: Section 5.3 trained and evaluated the router via 5-fold cross-validation directly on the held-out benchmark test sets across 1,428 samples pooled from all the model arms, meaning identical test prompts appear in both training and validation folds across arms.

  3. SFT on teacher-generated elimination traces fails to outperform the untrained base model on the hardest benchmarks: in Table 1, every SFT arm achieves lower elimination accuracy than the base model on both GPQA and MATH-500, and lowers the oracle union ceiling in Table 2, leading the authors to run their main routing evaluations in Tables 3, 4, and 5 on the untrained base model rather than the fine-tuned models.

  4. This submission never reports the training data sources, sample counts per benchmark, or how GPQA and MATH-500 were handled during trace generation without test leakage.

  5. Section 4 stated that the authors evaluate "the joint model's own self-routing via the strategy token", but in Section 3.3 the authors compare logistic regression with an MLP router, and neither result is reported anywhere in the paper. Additionally, Section 6.8 claims the router was trained only on base-model predictions, contradicting Section 5.3 and Section 6.2.

Questions for authors
  1. Which specific datasets and splits were employed to generate the Qwen3-32B teacher traces for LoRA SFT, how many traces were collected per benchmark, and why did SFT degrade elimination accuracy below the base model on GPQA and MATH-500?

  2. How did you prevent identical test problem texts from leaking across folds when pooling the 1,428 disjoint-success samples across all model arms during 5-fold cross-validation, and how is Section 6.8 reconciled with Section 5.3?

  3. What is the empirical performance of the promised self-routing baseline using the joint strategy-token model (joint_tok), and how does it compare to the external TF-IDF logistic regression router?

  4. Have the authors considered training a cost-aware router on the full problem distribution rather than discarding the 70% to 88% of problems where both strategies succeed, given that elimination traces consume 1.41x to 2.16x more tokens?

  5. It would be helpful to see how the router performs when replacing dataset-specific TF-IDF lexical cues with semantic embeddings or hidden-state representations on a strict train-test split.

Reviewer_yXm4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares forward chain-of-thought with elimination reasoning (generate candidates, evaluate, rule out, converge). Qwen3-32B produces gold-filtered traces of both kinds, and Qwen3-8B is fine-tuned with LoRA in four arms: forward-only, elimination-only, joint 50/50, and joint with a strategy token. Five arms, including base, are evaluated under both prompts on GSM8K, MATH-500, ARC-Challenge and GPQA-main. The paper reports that elimination can be trained, that 5-29% of problems are solved by exactly one strategy, and that a logistic-regression router over hand-crafted and TF-IDF features gains +2.4 points over the best single strategy and exceeds the dataset-level oracle at 13% extra token cost.

Critical strengths

The evaluation grid is complete and internally consistent. Five arms, two prompts and four datasets are reported (§4, l.186-187), and Table 2 agrees with Table 1 in every cell. Why this matters to my assessment: the basic measurements are usable even where the conclusions are not.

Critical weaknesses

W1. The headline routing result cannot be verified from the described protocol.

  • §5.3 (l.238-240) trains the router on "1,428 informative samples ... pooled across all arms and datasets", but §6.8 (l.426) says it "is trained on base-model predictions". By Table 2 the base arm has about 323 such problems, so both cannot hold.
  • CV is "5-fold stratified" (l.509). Whether folds are grouped by problem (the pool repeats problems across arms) and whether Table 3 uses out-of-fold decisions is not stated.
  • §4 (l.180-182) states GSM8K "test 1319" and ARC "test 1172". The Overall rows of Tables 3-4, the count 1,428, and the GSM8K McNemar p-values (§6.5) reproduce only with 500 problems per dataset (448 for GPQA). Sampling is not disclosed. Table 5 pairs full-set self-consistency counts (1080/1172, 1213/1319, l.280-281) with Table 3 columns, so those rows are not on the same problems.
  • Router minus dataset oracle is 1.27 points, about 25 problems (GSM8K 0, MATH-500 9, ARC 1, GPQA about 15), with no paired test, yet l.246-247 claims "value beyond simple dataset heuristics". Under the 500-problem reading and the five-arm pool, predicting the dataset majority gives 58.5% on the 1,428 pool versus 60.9% for the router (Table 7).

Why this matters to my assessment: contribution (iii) is the main empirical claim, and its margin is within what these protocol questions could move.

W2. Lack of novelty, and the figures and text overstate what the tables show.

  • The paper states that strategy-level routing "has recently been explored by Route to Reason (Pan et al., 2025), which adaptively selects among reasoning strategies" (§2.3, l.89-90), and cites elimination prompting (l.40-43) and reasoning distillation (l.80) as prior work. The router is logistic regression over TF-IDF and hand-crafted features (§3.3). The claimed difference is routing "on a single fine-tuned model" (l.91), but Table 3 is "on the base model" (l.240-241), and on GPQA every trained arm's routed accuracy implied by §6.8 (41.3 to 42.2) is below base routing (42.6).
  • Figure 2 is cited for the claim that the router "improves the accuracy-per-token Pareto frontier" (§5.3) and achieves "better accuracy than either alone" (§6.4). On GSM8K the router and forward both score 91.0 (Table 3), while forward is the cheapest point (248 tokens, Table 4), so the router point is dominated. The caption names a Pareto frontier but none is drawn, the oracle said to "define the upper bound" is not plotted, the legend lists diamond markers that do not appear, and the legend box covers the GPQA points and the MATH-500 elimination point.
  • The 13% cost increase in the abstract is computed from pooled means, 0.68×434+0.32×6100.68 \times 434 + 0.32 \times 610 (§5.4, l.261-263), not from the per-dataset points of Figure 2.
  • Figure 1 labels the forward trace "(candidates → answer)", although candidates belong to the elimination format (§3.1); the only solid arrow into the router comes from the SFT arms, while Algorithm 1 builds the router input from problem text alone; the output arrow runs through the router's text; and "Router v2" appears nowhere else in the paper. Figure 2 is placed after the reference list.
  • The text does not match the tables: the oracle gap is stated as "4–14 percentage points" (l.214-215), but Tables 1-2 give 2.2-13.4; Table 3 prints "+2.4" overall, but 64.8−62.5=2.364.8 - 62.5 = 2.3, as §5.3 itself computes; and §5.5 calls forward "dominant" on ARC, where base elimination is higher (90.6 vs 89.8, Table 3).
Questions for authors

could you please write your method clearly?

could you please rewrite figure 1?

1 comment

If authors do not reply anything, I would like to maintain the original score

Reviewer_ABkUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies forward and elimination reasoning in Qwen3-8B through supervised fine-tuning and lightweight per-query routing. However, the contribution of elimination-specific training remains unclear, and inconsistencies in the reported evaluation undermine confidence in the empirical conclusions.

Critical strengths

This paper studies forward and elimination reasoning in Qwen3-8B through supervised fine-tuning and lightweight per-query routing.

Critical weaknesses

Overall, this manuscript does not convincingly establish the incremental value of the proposed method, and the statistical reporting contains some inconsistency. The authors should carefully considerate these issues.

  1. The incremental benefit of elimination training is not established. On GSM8K, Table 1 reports the highest accuracy for forward-only training with forward prompting (91.4%), whereas the base-model router in Table 3 achieves 91.0%. Elimination training improves elimination-prompted accuracy from 80.4% to 85.8%, but forward-only training already achieves 85.0% under elimination prompting. The additional benefit attributable specifically to elimination training is therefore unclear. These results do not demonstrate that the routed system outperforms the strongest reported training configuration on GSM8K. They also do not adequately distinguish elimination-specific learning from improvements obtainable through forward distillation.

  2. The reported p-value is inconsistent with the tabulated results. For joint_tok on GSM8K, Tables 1–2 report forward accuracy of 89.4%, elimination accuracy of 86.0%, and union accuracy of 92.8%. Given the evaluation size of 1,319 problems stated in Section 4, standard rounding to one decimal place implies 1,179, 1,134, and 1,224 correct predictions, respectively. The paired discordant counts are therefore:

  • Forward correct, elimination incorrect: 1,224 − 1,134 = 90.
  • Elimination correct, forward incorrect: 1,224 − 1,179 = 45. The two-sided exact McNemar test specified in Section 6.5 gives p = 2P(X ≤ 45) ≈ 0.000134, where X ∼ Binomial(135, 0.5). This does not match the reported p = 0.024. This discrepancy raises concerns about the accuracy and internal consistency of the experimental reporting. Please verify the tabulated accuracies, evaluation denominator, paired contingency counts, and p-value, and identify which reported values require correction.
  1. Presentation and terminology need improvement. The manuscript needs more careful editing and layout control. On page 12, Figures 2-4 interrupt the reference list and precede the “Additional Figures” appendix heading. Supplementary figures should appear within the clearly designated appendix after the references. Figure labels and legends should also be enlarged for readability. The abstract would benefit from a more self-contained account of the contribution, with less reliance on citations. Terminology should distinguish training configurations from inference strategies more clearly. In particular, consistently using “training configuration” instead of “arm” would improve precision. These are secondary presentation concerns; my recommendation rests on the substantive issues above.
Questions for authors

Please refer to the weaknesses above.

1 comment

Since the authors have not given any responses, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.