acceptodds
Withdrawn from ICLR 2027

MicroMouse: Masking Harmful Supervision in Long CoT via Relative Potential

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long chain-of-thought (CoT) trajectories have become an important source of supervision for training large reasoning models, yet conventional supervised fine-tuning (SFT) typically optimizes all tokens in these trajectories indiscriminately. However, even successful long CoTs may contain harmful supervision, motivating the need for methods that can automatically identify and mask such signals during SFT. Inspired by the MicroMouse maze-solving process, we view a long CoT as a search trajectory from the input problem toward the target answer. This perspective motivates maintaining a goal-relative measure, analogous to a MicroMouse's spatial coordinates, to quantify how close the current reasoning state is to the target answer. Building on this perspective, we propose **MicroMouse**, an automated selective-SFT framework for masking harmful supervision in long CoTs. In MicroMouse we introduce a Relative Potential (REP) to measure how closely each reasoning state supports the gold answer. Unlike the perplexity of the gold answer alone, REP measures relative support by contrasting the gold answer with a matched foil. This contrast cancels likelihood variations shared by the pair, yielding a more stable signal of progress toward the target answer. By tracking the REP trajectory, MicroMouse identifies consecutive potential decreases as erroneous exploration and post-peak reasoning as redundant wandering, and masks these harmful supervision during SFT. Extensive experiments across multiple student model scales and reasoning benchmarks show that MicroMouse}consistently outperforms Full-CoT SFT and existing selective-SFT baselines, yielding average relative gains of **5.5%** in greedy accuracy and **2.6%** in Pass@1 relative to Full-CoT SFT. Meanwhile, MicroMouse improves reasoning efficiency by reducing average output length by **8.8%**, while identifying harmful supervision in only **1.7** seconds per long CoT on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Selective SFT, Chain-of-Thought, Reasoning Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WQJgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MicroMouse, a selective-SFT method for long reasoning traces. It splits each trace into segments and measures the relative support for the gold answer against a constructed incorrect answer, called a foil. Based on changes in this score, it masks losses for segments labeled as erroneous exploration or redundant wandering while keeping them in the context. Experiments report improved reasoning accuracy and shorter outputs compared with Full-CoT SFT and several selective-SFT baselines.

Critical strengths
  1. The paper addresses a useful problem: not every part of a successful reasoning trace provides equally valuable supervision. Keeping masked segments in the context also preserves the dependencies needed by later steps.
  2. The researchers categorize reasoning steps into Correct Progress, Necessary Exploration, Erroneous Exploration, and Redundant Wandering, which makes sense because sometimes in CoT, the model can spend some effort on exploring "what is wrong", and that is necessary.
  3. The scoring method is simple and relatively efficient. It uses answer likelihoods without requiring gradient-based attribution or a large model to annotate each step.
  4. The experiments cover three model sizes and seven reasoning benchmarks. Results under both greedy decoding and sampling provide useful evidence for the method’s effectiveness.
Critical weaknesses
  1. The maze analogy does not sufficiently justify the core classification rules. REP has not been shown to measure the actual distance to a complete solution, and useful exploration may involve several consecutive score decreases. Likewise, reaching the global REP peak does not establish that reasoning is complete or that all later steps are redundant.
  2. A single constructed foil may not reliably represent progress toward the correct answer. A numerically close answer is not necessarily a likely competing answer, and REP can increase even when the gold answer becomes less likely, provided the foil becomes less likely by a larger proportion. The paper does not establish whether its scores and masking decisions remain stable under different foil choices.
  3. The method’s sensitivity to the scoring model remains unclear. Replacing the default DeepSeek-R1-Distill-Qwen-1.5B scorer with Qwen3-4B-Instruct reduces greedy accuracy from 45.1 to 41.9 and Pass@1 from 50.9 to 49.0. Although some gains over Full-CoT SFT remain, testing one alternative scorer provides limited evidence of robustness across scoring models.
Questions for authors
  1. Can you provide independent step-level evidence that REP reflects actual reasoning progress? In particular, how reliably do consecutive decreases identify harmful exploration, and how often do steps after the REP peak still contain necessary reasoning?
  2. How sensitive are the masking decisions and downstream results to the choice of foil? Would the method remain effective with plausible model-generated wrong answers or multiple competing answers?
  3. What makes a model suitable for scoring, and why does the default 1.5B scorer perform better than the alternative 4B scorer? Can you evaluate more scorer families and sizes, and report how much their masking decisions agree?
Reviewer_S8wCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MicroMouse, which is a selective SFT framework that masks harmful reasoning steps in long CoT trajectories by tracking a contrastive Relative Potential between the gold answer and a minimally perturbed foil answer. Experiments across several Qwen-3 models and distinct reasoning benchmarks show improvements in reasoning accuracy and output length efficiency over full-trajectory SFT and baselines.

Critical strengths
  1. Long CoT step selection can be framed as trajectory tracking with a contrastive log-likelihood ratio between the gold answer and a foil, and is very intuitively structured and practically fast, taking only 1.7 seconds per trajectory.

  2. Experimental evidence from Qwen3-8B confirms the general performance improvement over all 7 benchmarks, as well as specific improvements on the harder benchmarks by retaining individual isolated drops but masking multiple consecutive drop samples.

Critical weaknesses
  1. The paper does not provide an ablation study that compares the proposed Relative Potential method against utilizing just gold-answer log-likelihood on the exact same trajectory classification rules, which provides an incomplete analysis of the contrastive element of the contribution.

  2. As Section 3.4 stated, replacing the default 1.5B R1-distilled scorer with Qwen3-4B-Instruct provides consistent results, however, Table 3 reflects that not only should Qwen3-4B-Instruct decrease greedy accuracy by nearly 3.2% and Pass@1 by 1.9%, but Qwen3-4B-Instruct also generates more text than the random masking approach while achieving similar or poorer performance.

  3. The foil construction presented in Appendix A.1 matches only integers or integers written as a simple fraction. As a result of only being able to match integers or simple fractions, this rule-based foil construction will not align with symbolic mathematics or the multiple-choice style of answers. It also does not provide a guarantee of identical lengths of subword tokens as given by Eq. (1).

  4. Performance gains do not scale consistently to Qwen3-14B, where MicroMouse underperforms full SFT on MATH500, AIME24, and OlympiadBench, with the positive overall average driven by unweighted averaging over small 30-problem contests.

Questions for authors
  1. How does MicroMouse perform when using the gold answer log-likelihood alone without subtracting the foil term under the exact same Stage 2 step categorization rules?

  2. Why does replacing the DeepSeek-R1-Distill-Qwen-1.5B scorer with Qwen3-4B-Instruct in Table 3 degrade accuracy and output length to the level of random masking?

  3. When the potential peak occurs before the final step, are the closing think delimiter and final boxed answer tokens also masked from the training loss, and why does keeping redundant steps in the conditioning prefix reduce generation length at inference time?

  4. How were foils constructed for LIMO problems whose ground-truth answers are symbolic expressions, radicals, or multiple-choice letters not supported by the regex in Appendix A.1?

Reviewer_U29UOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MicroMouse, an SFT method that masks the harmful parts of long chain-of-thought training data from the loss, aiming to improve the student model’s reasoning. Using a small scoring model, the method computes a score for each reasoning step (how much it prefers the correct answer over a slightly wrong one) and decides which steps to keep.

Critical strengths
  1. The motivation is reasonable.
  2. The presentation is rather clear.
Critical weaknesses
  1. The novelty is very limited. Selecting and pruning reasoning steps during training is already well studied. Although the authors include several step-pruning baselines, these are excluded from the main comparison in Table 1 and evaluated only in a narrow setting (Qwen3-8B in Fig. 2). Additionally, there is no strong evidence that the masked steps are actually harmful. Table 3 shows that masking only erroneous exploration performs worse than plain SFT (38.9 vs 41.0), which indicates that this mask is not useful. Adding it on top of post-peaking mask gains only 0.7, which indicates that most of the benefit comes from masking the REP peak. This contradicts with the paper's motivation that erroneous exploration is harmful supervision that should be masked. I am also not convinced that a small model (DeepSeek-R1-Distill-Qwen-1.5B) is sufficient to score reasoning steps. Finding wrong steps in CoT for competition-level math is difficult even for much larger models [1], and replacing the scorer with Qwen3-4B-Instruct performs about the same as random masking (41.9 vs. 41.7, Table 3). This is especially concerning since steps are labeled using a threshold, and no ablation study is conducted on this threshold.

  2. The reported gains in Pass@1 are less than 1 point for the 4B and 14B models (+0.7 and +0.9, Table 1), and some benchmarks get worse (e.g., Qwen3-14B on AIME24 under greedy decoding: 50.0 to 43.3). Several benchmarks used in the experiments are also very small (e.g., 30 questions for AIME24). The training uses only 817 LIMO examples from a single teacher, and only Qwen3 base models are tested. The experiments are therefore limited, and more experiments with larger training sets, different teachers, and different base models should be considered.

[1] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBench: Identifying Process Errors in Mathematical Reasoning. In ACL, 2025.

Questions for authors
  1. Can the authors conduct additional ablation studies on REP, comparing it with other metrics such as gold-answer perplexity under the same labeling rules (Sec. 2.1)?

  2. Does the choice of foil matter? Would a slightly wrong answer (e.g., 650 vs. 651) and a very wrong answer lead to different performance?

  3. The authors should add human verification to assess how accurate the step labels are, for example by checking whether the masked steps are actually erroneous or redundant.

  4. Since most of the gain comes from masking steps after the REP peak (Table 3), how is this different from keeping reasoning only up to the first correct answer

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.