- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RECOMATH evaluates whether language models can recover after receiving a reasoning prefix that contains an unmarked error. The benchmark combines 2,645 newly generated erroneous trajectories with 2,212 examples derived from ProcessBench. A controlled supervised fine-tuning study compares reflection, repair, and related trajectory constructions. Based on these experiments, the paper proposes Calibrated Reflection Training, which uses an 80:20 mixture of reflection and repair trajectories.
The task formulation is useful because it differs from clean-start solving and from settings where the location of the error is revealed. The experiments suggest that retaining the repair process can improve recovery. However, the reliability of the first-error labels, evaluator specification, contamination analysis, and selection of the 80:20 mixture require stronger validation.
Critical strengths
-
RECOMATH studies recovery from an erroneous reasoning prefix without revealing either the presence or location of the error. This is meaningfully different from clean-start solving and oracle-hinted correction.
-
The controlled training comparison uses the same data budget and training recipe across trajectory types. The construction pipeline also combines symbolic checks, model-based verification, and independent teacher solving.
Critical weaknesses
-
The benchmark depends on model-generated first-error labels, but their accuracy is not directly established. Choosing the wrong step can either leave an earlier error in the prefix or incorrectly mark a valid step, changing the task itself.
-
The Easy/Hard analysis is confounded by source dataset, error generator, error location, and writing style. The observed difference between reflection and repair cannot be attributed cleanly to mathematical difficulty.
-
The 80:20 mixture appears to be selected using the same benchmark suite on which CRT is reported. This makes it difficult to distinguish a general training principle from a mixture tuned to the evaluation set. The unspecified LLM evaluator adds further uncertainty to small performance differences.
Questions for authors
- How many first-error annotations were manually checked, and what counted as a correct localization?
- Was the 80:20 ratio fixed before inspecting the reported benchmark results?
- Which model and decision rule were used for final-answer evaluation?