- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigated training process-of-elimination reasoning into Qwen3-8B via LoRA SFT on teacher-generated traces from Qwen3-32B, alongside standard forward CoT reasoning. The authors evaluated complementarity between the two strategies across 4 math and science benchmarks and trained a lightweight feature-based logistic regression router to select between forward and elimination reasoning per query.
Critical strengths
Experimental results across a few benchmarks and different configurations provide a structured comparison of strategy complementarity, token cost overhead, and self-consistency baselines.
Critical weaknesses
-
This paper merely explored a single Qwen3-8B model, which narrows the paper's contribution.
-
The router evaluation suffers from severe data leakage and methodological flaws: Section 5.3 trained and evaluated the router via 5-fold cross-validation directly on the held-out benchmark test sets across 1,428 samples pooled from all the model arms, meaning identical test prompts appear in both training and validation folds across arms.
-
SFT on teacher-generated elimination traces fails to outperform the untrained base model on the hardest benchmarks: in Table 1, every SFT arm achieves lower elimination accuracy than the base model on both GPQA and MATH-500, and lowers the oracle union ceiling in Table 2, leading the authors to run their main routing evaluations in Tables 3, 4, and 5 on the untrained base model rather than the fine-tuned models.
-
This submission never reports the training data sources, sample counts per benchmark, or how GPQA and MATH-500 were handled during trace generation without test leakage.
-
Section 4 stated that the authors evaluate "the joint model's own self-routing via the strategy token", but in Section 3.3 the authors compare logistic regression with an MLP router, and neither result is reported anywhere in the paper. Additionally, Section 6.8 claims the router was trained only on base-model predictions, contradicting Section 5.3 and Section 6.2.
Questions for authors
-
Which specific datasets and splits were employed to generate the Qwen3-32B teacher traces for LoRA SFT, how many traces were collected per benchmark, and why did SFT degrade elimination accuracy below the base model on GPQA and MATH-500?
-
How did you prevent identical test problem texts from leaking across folds when pooling the 1,428 disjoint-success samples across all model arms during 5-fold cross-validation, and how is Section 6.8 reconciled with Section 5.3?
-
What is the empirical performance of the promised self-routing baseline using the joint strategy-token model (joint_tok), and how does it compare to the external TF-IDF logistic regression router?
-
Have the authors considered training a cost-aware router on the full problem distribution rather than discarding the 70% to 88% of problems where both strategies succeed, given that elimination traces consume 1.41x to 2.16x more tokens?
-
It would be helpful to see how the router performs when replacing dataset-specific TF-IDF lexical cues with semantic embeddings or hidden-state representations on a strict train-test split.
If authors do not reply anything, I would like to maintain the original score