acceptodds
Under review as a conference paper at ICLR 2027

Perfect Labels for Free? What Exact Answers and Exact Traces Teach a Small Language Model

Abstract

Exact verifiers (primality tests, SAT solvers, unit tests) label data without noise and supply the rewards of verifiable-reward post-training. Because every label is exact, a difference between two training regimes on the same inputs is a difference in the supervision format. We fine-tune Qwen2.5 models (0.5B to 7B; main size 1.5B, three seeds for the core answer and trace cells) with LoRA on exactly labeled decision tasks and compare supervision by the answer alone with a deterministic procedure trace followed by the answer. Exact answers teach shortcuts: on 4-digit primality the 1.5B answers (three seeds) follow "the last digit is 1, 3, 7 or 9" on 99.2-100.0% of test items, and on divisibility by 7 or 13 answer-only training stays at chance or constant at every size and configuration we ran (n<=540, including a logged sweep of four configurations, eight runs, none collapsing), while the same recipe learns divisibility by 3. Exact answers remove computation from the answers: at 3B and 7B the zero-shot model already divides by 7; answer-only fine-tuning drops it to chance in both seeds, and under a chain-of-thought prompt the answer-only model still answers at once at every size tested (at least 97.9% of items in three tokens or fewer) and stays at chance, while fine-tuning on its own verifier-filtered traces keeps the division (two seeds at each size); whether the capability survives is untested. Exact traces teach steps within a task: accuracy rises with the number of traces and with how often each table entry was supervised, a count-based prediction across tasks failed because steps differ in difficulty, and only the input's own trace, written before the answer, helps. A second model family (SmolLM2-1.7B-Instruct) shows the same direction on the three tasks run, but its timestamped prediction (S11) failed its 20-point threshold in one seed. Predictions were logged in timestamped commits before their cells ran; four of twelve failed (the cross-task dose prediction S9, the length prediction S5, the self-generated-trace prediction S10 and the second-family prediction S11), and all are reported.

Primary area:
foundation or frontier models, including LLMs
Keywords:
verifiable rewards, exact verifiers, supervision format, chain-of-thought fine-tuning, shortcut learning, process vs outcome supervision, preregistration, small language models, arithmetic reasoning, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dgiKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This feels like an AI-generated paper. I cannot even follow the abstract and the introduction. The writing has no logic and I have no idea what the research problem is.

Critical strengths

None

Critical weaknesses

The writing has no logic and I cannot follow.

Questions for authors

Can you elaborate what is the research problem?

Reviewer_4LjoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at whether perfectly correct labels actually teach small language models how to solve problems, or if they just learn shortcuts that happen to work on the training data.

  • They compare answer-only training against training on full reasoning traces.
  • They test this on primality, divisibility, and logic tasks using Qwen models from 0.5B to 7B.
  • The main finding is that answer-only training often learns shortcuts instead of the actual computation. On primality, the model basically learns to check the last digit.
  • On divisibility by 7, trace training gets to 88.3% accuracy at 1.5B, while answer-only stays around 50%.
  • More interestingly, answer-only fine-tuning can make models worse at tasks they could already solve. This happens with the 3B and 7B models.
  • They also try scrambled traces, traces from other problems, and putting answers before traces. These mostly don't work, which suggests the actual steps and their order matter. Overall, the paper makes the point that having perfect labels doesn't mean a model will learn the right underlying procedure.
Critical strengths

The controls are probably the strongest part of the paper. They don't just show that trace training works better and leave it at that. They actually try to figure out why. The scrambled traces stay around chance, and even giving the model a correct trace from a different number doesn't help. Putting the trace after the answer also doesn't work. I think this is a pretty important distinction because otherwise you could argue that the model just benefits from seeing more tokens or some generic reasoning-looking text. The controls aren't perfect, but they make a much stronger case that the steps need to be relevant to the actual input and appear before the answer. The per-step analysis adds to this too. Seeing lookup accuracy improve as individual table entries appear more often in training gives a pretty concrete picture of what the model is learning. The finding about losing existing reasoning ability is pretty compelling. The 3B model starts at 94.2% zero-shot accuracy on divisibility by 7 and drops to 52.3% after answer-only fine-tuning. The 7B model shows a similar drop. That's a pretty striking result since the model already knew how to do the task. I also liked the comparison with training on the model's own verifier-filtered traces, which largely preserves the original accuracy. That makes the result more useful than just showing that answer-only training fails from scratch. It suggests that the supervision format can change how a model uses capabilities it already has. The authors also try prompting the fine-tuned models to reason again, and they still give short answers instead. I think this is one of the more interesting contributions, even if it doesn't prove the underlying ability is actually gone.

Critical weaknesses

I'm still not completely convinced by the fairness of the answer-only vs. trace comparison. The trace models see 14–47x more supervised tokens under the shared training setup, which is a pretty big difference. I get that the authors include scrambled-trace controls and an answer-only hyperparameter sweep. The sweep is actually useful since answer-only still fails on divisibility by 7 even with more training. But I don't think it fully answers the question of whether a better optimized answer-only setup could close some of the gap. The scrambled traces also perform poorly on tasks where answer-only does well, so they aren't necessarily a clean way to isolate the benefit of meaningful traces. I think a more direct compute- or token-matched comparison on the main divisibility task would make the central claim a lot stronger. Right now, it's clear that the trace setup works better, but less clear exactly how much of that advantage comes from the supervision format itself. The scope of the experiments makes the broader takeaway a little harder to judge. Most of the results are on small, synthetic, binary decision tasks. The tasks are useful because the authors can check exactly what the model is doing, but they're still pretty far from the reasoning problems that motivate verifiable-reward training in practice. Most experiments also use one model family, and the second-family results are weaker. There are only one to three seeds per cell as well. Another thing is that trace training isn't consistently better. It actually hurts primality at 1.5B, and answer-only training does better on the logic validity task. I think those are important results, but they also make me wonder how much the conclusions depend on the specific task and the chosen trace format. There's also no actual RL experiment, despite the motivation around verifiable rewards. I would have liked a bit more evidence that these findings carry over beyond supervised fine-tuning on these particular tasks.

Questions for authors
  • Could you run a more directly token-matched comparison on divisibility by 7, ideally with some tuning for both training formats? I think this would help clarify how much of the improvement is specifically due to the reasoning traces rather than the additional supervised tokens.
  • Is there a way to test whether the original division capability is actually lost after answer-only fine-tuning, or whether the model just stops using it? The CoT prompting experiment is useful, but I don't think it fully distinguishes the two.
  • How much do you expect these results to carry over to actual verifiable-reward training or more complex reasoning tasks? I'm particularly curious because traces hurt on some of the tasks in this paper, and the experiments don't include RL.
Reviewer_bsqyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is an empirical study, where the authors compare training small language models with answer only v.s. answer with traces on some simple tasks such as primality testing, divisibility, and logical validity. The authors find that answer-only supervision can encourage shortcuts ("learn surface rules"). Trace supervision improves performance on some tasks as coverage of intermediate steps increases.

Critical strengths
  • A controlled study on the learning behavior of small language models on verifiable tasks is important, which may help us obtain a deeper understanding of supervised learning for LLMs.
Critical weaknesses
  • The paper is difficult to read and follow. Although the central question is straightforward, the manuscript introduces many terms, experimental variants, and notions without introduction or motivation. As I read through the paper, I feel confused how they connect to the main research question. The presentation frequently switches context, making it hard to identify the main takeaways.

The writing needs significant improvement. For example: “Trace supervision teaches the procedure within a task once its steps are supervised often enough: accuracy rises with training traces and with per-entry coverage, while a count-based prediction across tasks failed, exposing step difficulty as a confound of supervision frequency.” Terms such as “per-entry coverage” and “count-based prediction” are difficult to understand without first reading the detailed experiments.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.