acceptodds
Under review as a conference paper at ICLR 2027

Longer Answers, Heavier Questions: How Loss Weighting Shapes Comparisons of Synthetic Reasoning Data

Abstract

When a language model is fine-tuned on synthetic reasoning data with a token-mean loss, each question counts in proportion to the length of its target. We call this implicit question weighting. A recipe that sets how much reasoning each question receives therefore also sets how much each question counts in training. With targets of similar length the weights are close to uniform; in the mixed-depth corpus we study, direct-answer targets are 42% of the questions but receive under 6% of the loss weight. We measure how strongly a comparison between two recipes depends on this weighting. Both recipes come from one teacher pool on the same 11,853 questions: one gives every question a brief explanation, the other mixes direct answers with deeper reasoning. With the targets held fixed, replacing the token mean by an equal weight per example causes no detectable change for the fixed-depth corpus but costs the mixed-depth corpus 8.6 and 5.5 points of four-benchmark accuracy on Qwen3-4B-Base and OLMo-3-7B (three seeds per arm), so the lead of the fixed-depth corpus grows from 1.6 to 9.9 points on one student and from 3.1 to 8.3 on the other. Training the fixed-depth targets with the mixture’s own question weights does not reproduce this response, so the weights alone do not explain it. With each source dataset’s share of the loss held fixed, moving the direct-answer targets to their share of the examples costs more than equalizing the examples within the direct-answer and the reasoning groups. Comparisons of reasoning datasets should therefore report their loss objective and the loss share of their shortest targets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic reasoning data, supervised fine-tuning, loss weighting, loss normalization, reasoning distillation, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Uz3dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study how using the most common loss function, mean cross-entropy, could lead to imbalance when the training set contains examples of variable length. By averaging all token cross-entropies in a batch, the shorter examples get a lower weight when paired with longer examples. This is a known problem, which has already been tackled in RL training. Nevertheless, the current study controls the experimental variables to isolate the effect of the loss function in different scenarios. The experiments are focused on finetuning on reasoning traces, which are generated by a teacher model prompted for different reasoning depths. Different scenarios are tested to see how the mix of data influences the downstream performance, which is measured on several public math and reasoning datasets.

Critical strengths
  • The studied potential issue with the loss function is somehow overlooked in the literature. This study brings important empirical validation of the effect of training with classical mean cross-entropy compared to an equal-example loss function.
  • The experimental setup is well designed and covers a diverse set of conditions. Confidence intervals and significance tests are included for each experiment.
Critical weaknesses
  • The main issue with the paper is the clarity of the presentation. It is quite difficult to follow, mostly due to the high number of named recipes and dataset variations, which appear in multiple configurations. The structure, which mostly enumerates setups and experiments in an LLM-like fashion, lowers the cohesion and readability of the text. I would have liked more discussions and interpretations of the results instead of an enumeration of experiments.
Questions for authors
  • The paper would be more valuable if the authors included some recommendations and guidelines for best practices. The series of experiments is meant to isolate where the performance decrease comes from, but I couldn't find a complete explanation in the end.
  • As far as I understand, the mixed dataset with equal example loss obtains a lower performance because direct answers get a higher weight, so the model tends to answer with short or without reasoning with confidence some questions that would require longer reasoning. I am thinking that including some instruction at the beginning of the prompt used in finetuning could steer the model towards longer or shorter reasoning. Then, the evaluation could be done by forcing a higher reasoning effort (or on different levels). Is this a correct assessment, or am I missing something?
2 comments

We thank the reviewer for the careful reading. We answer the three points in order.

1. Presentation. Concern: the paper is difficult to follow because of the high number of named recipes and dataset variations.

Two corpora carry the main comparison: Fixed-T2 and Guided-Mix-T2sub, on the same 11,853 questions (Section 4). The interventions of Section 5 run on the full mixture Guided-Mix. Table 1 lists every experiment with what it holds fixed and what it changes. The other named recipes belong to the supporting results of Section 6 and to the registered recipe comparison of Appendix B.

2. Recommendations and explanation. Concern: the paper would be more valuable with recommendations and guidelines for best practices, and a complete explanation is hard to find in the end.

In the mixed-depth corpus the direct answers (T1) are 42.1% of the questions but receive 5.6% of the loss coefficients under the token mean. The equal-example objective gives them their example share. Moving only the T1 targets to their example share costs 7.74 points on Qwen3 and 5.22 on OLMo; equalizing the examples within the T1 and the T2-T5 groups costs less (Table 2). A new experiment agrees (see our reply to Reviewer uj6D): when every T1 target of Guided-Mix-T2sub is replaced by the T2 target of the same question, the cost of the switch falls from 7.91 and 5.46 points to 1.34 and 1.42. The paper recommends stating the loss normalization with every comparison of recipes, and reporting each group's share of the examples and of the loss (Sections 7 and 8).

3. The reviewer's assessment. Concern: with the equal-example loss the model answers directly some questions that would require longer reasoning, and an instruction or a forced higher reasoning effort at evaluation could steer it.

The first part agrees with what we measured (Appendix J). Under equal-example weighting, students of the mixture answer directly on 46% instead of 22% of MATH-500 questions on Qwen3, and their mean response length falls from 125 to 48 tokens. Students of the fixed-depth corpus never answer directly. We also ran the evaluation half of the suggestion: we evaluated the students of Guided-Mix-T2sub again on GSM8K and MATH-500 with the instruction "Give a brief solution showing the necessary steps before the final answer" added to the prompt. The instruction hardly changed their behaviour: on Qwen3 the students trained under the equal-example objective still answer directly on 41.5% of MATH-500 questions, against 44.7% with the unchanged prompt in the same run. Their accuracy changes by -1.24 to +1.42 points across the two students and the two benchmarks, with every interval including zero, and the gap between the objectives does not close. Because the students mostly kept answering directly, this evaluation does not show whether more reasoning would recover the accuracy. An instruction in the fine-tuning prompts would change the training rows, which the comparisons in the paper hold fixed; it was not tested.

Thank you for adding the new experiments and your answer. However, I will not change my current assessment because the paper still has limited impact. It shows that it is important to know the distribution of the data mix and that changing the loss function has a high influence in the performance, which is somehow expected. As also highlighted in your answer, the difference comes from steering the model towards shorter answers, which is a side effect of the equal-weight loss function, and could be mitigated by including instructions in the finetuning step.

Reviewer_uj6DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how loss normalization affects comparisons between synthetic reasoning-data recipes. Its main comparison uses a fixed brief-explanation recipe and a mixture of direct answers and explanations of different depths, trained on the same 11,853 questions. Switching from token-mean to equal-example weighting has little measured effect on the fixed-depth recipe but reduces the mixed recipe’s average accuracy by 8.64 points on Qwen3 and 5.46 points on OLMo. Further interventions investigate whether the effect comes from redistribution between datasets, between direct-answer and reasoning-target groups, or within those groups.

Critical strengths
  • The paper goes beyond observing that two loss objectives produce different scores. Matching questions and holding each recipe’s targets fixed makes the main comparison informative. The shared-question-weight control and the interventions on dataset and depth-group shares help narrow the explanation. In particular, the comparison between changing group shares and changing weights within groups provides more information than a simple short-versus-long comparison.

  • The authors report useful limitations and negative results. They acknowledge that the two normalizers are established, that the recipe ranking does not reverse, and that the guided allocation rule has no consistently demonstrated advantage over the matched random allocation. Reporting these boundaries makes the empirical results easier to assess.

Critical weaknesses
  • I find the additional insight and practical significance limited.

The central direction of the result is not particularly surprising to me. In a mixture containing many direct answers and fewer reasoning traces, equal-example weighting substantially increases the relative contribution of direct answers and decreases that of reasoning targets. Since reasoning traces can contain substantially more supervision, a performance loss under this redistribution is consistent with an existing training intuition. This does not imply that useful information is proportional to token length, or that token weighting must always be better.

The controlled magnitude of the effect is useful to document. However, I am less convinced that the paper establishes an important new conclusion beyond the need to inspect loss weighting when comparing data recipes. The ranking of the two main recipes stays the same; the gap becomes larger. The group-level analysis is the most informative part, but its implications for choosing a better training objective remain unclear. After reading the paper, I do not have a substantially clearer rule for deciding how to weight a new CoT corpus.

  • The evidence is specific to the constructed mixture, and the explanation remains incomplete.

The main evidence comes from one teacher-generated pool over four task datasets. The mixture is produced by a particular reliability threshold and shortest-correct selection procedure. The additional recipe comparisons largely reuse this pool. Two students provide useful replication across models, but do not establish that the phenomenon has similar importance across the diverse CoT corpora currently used for SFT.

The depth settings change generation policy and target content as well as length. The group interventions show that, under the tested redistributions, changing the balance between direct-answer and reasoning targets is more damaging than equalizing examples within those groups. They do not separately identify the effects of increasing direct-answer supervision and reducing reasoning supervision, nor do they establish which properties of the targets determine the sensitivity. The paper acknowledges these limitations, but they leave the explanation and its transfer to other corpora unresolved. The contribution is therefore more convincing as a controlled case study than as general guidance for synthetic reasoning-data design.

Questions for authors
  • What evidence supports extending the result beyond this teacher pool and selection rule? In particular, does the interaction persist on an independently constructed CoT corpus, or on a corpus containing varied reasoning traces without a large direct-answer component?

  • What training decision should a practitioner make differently after reading this paper? Can the proposed diagnostics predict which data mixtures will be sensitive to normalization, or guide a weighting choice beyond inspecting the loss shares after the fact?

  • How far can the explanatory claim be narrowed using the existing experiments? Can the authors distinguish the effects of reasoning content, answer style, and target length, or is the supported conclusion limited to the tested redistribution between the two generation-policy groups?

2 comments

We thank the reviewer for the careful reading. We answer the three points in order.

1. Other corpora. Concern: the evidence is specific to this teacher pool and selection rule, and a corpus containing varied reasoning traces without a large direct-answer component is not tested.

We ran the second case the reviewer names: we replaced every T1 target of Guided-Mix-T2sub by the T2 target of the same question (the same 11,853 questions, depths T2-T5, no direct answers) and trained this corpus under both objectives with three seeds on both students. Switching from the token mean to the equal-example objective costs it 1.34 points [0.35, 2.32] on Qwen3 (per seed 1.38 / 1.08 / 1.55) and 1.42 [0.28, 2.55] on OLMo (per seed 2.06 / 0.55 / 1.63) with sampled decoding, and 1.31 [0.31, 2.30] and 1.14 [0.13, 2.19] with greedy decoding. For Guided-Mix-T2sub the cost is 7.91 points [6.54, 9.28] on Qwen3 and 5.46 [4.19, 6.73] on OLMo (Table 8, Appendix F; on Qwen3 with the mixture arm retrained under the same attention kernel, 8.64 with the registered runs). The difference is 6.57 points [4.94, 8.21] on Qwen3 and 4.04 [2.35, 5.75] on OLMo. For the selection rule, on the full mixture the equal-example objective also lowers four-benchmark accuracy for the matched-random control, another policy on the same pool: by 9.67 points [8.37, 10.97] on Qwen3 and 6.36 [5.13, 7.61] on OLMo (Section 4). The paper states its scope as one teacher pool and two student families (Section 7).

2. Practical guidance. Concern: the paper gives no clearer rule for deciding how to weight a new CoT corpus, and it is unclear whether the diagnostics can predict which data mixtures will be sensitive to normalization.

The diagnostics need no training (Section 3): the example shares and the loss-coefficient shares of the important groups, here T1 against T2-T5 (Section 7). In the three corpora on these questions they mark the sensitive one before training. In Guided-Mix-T2sub the T1 targets are 42.1% of the questions but receive 5.6% of the loss coefficients on Qwen3, and the switch costs 7.91 and 5.46 points. Fixed-T2 and the new corpus have no T1 targets: for Fixed-T2 no penalty is detected (Section 4), and the new corpus loses 1.34 and 1.42 points, although its effective number of examples (4,349 on Qwen3) is closer to that of Guided-Mix-T2sub (2,917) than to that of Fixed-T2 (10,356). For the weighting itself, Table 2 orders two interventions: moving the T1 targets to their example share of the loss costs 7.74 points [6.47, 9.01] on Qwen3 and 5.22 [3.90, 6.53] on OLMo; equalizing the examples within the T1 and the T2-T5 groups costs less, 1.81 [0.74, 2.85] and 0.20 [-0.87, +1.27]. The paper recommends that comparisons of data-construction recipes state their loss normalization and the share of the loss that their shortest targets receive (Section 8).

3. Explanation. Concern: the interventions do not separately identify the effects of increasing direct-answer supervision and reducing reasoning supervision, nor which properties of the targets (reasoning content, answer style, target length) determine the sensitivity.

The new corpus and Guided-Mix-T2sub differ only in the targets of the 42.1% of questions that carry T1 targets in Guided-Mix-T2sub. In the new corpus the switch moves loss away from the deeper targets and none to direct answers. In both corpora the equal-example objective lowers the T3-T5 targets to 7% of the loss coefficients (from 32% in Guided-Mix-T2sub, Appendix F, and from 25% in the new corpus on Qwen3); only in Guided-Mix-T2sub it also raises the T1 targets, from 5.6% to 42.1%. The switch costs 1.34 and 1.42 points without this increase and 7.91 and 5.46 with it. For target length, equalizing the examples within the T1 and the T2-T5 groups costs less than moving the T1 targets to their example share, although it moves more weight away from the longest targets (longest tenth's share of the loss on Qwen3: from 45.73% to 22.89%, against 28.84%; Table 4, Appendix E). For answer style, students of the mixture answer directly more often under equal-example weighting (Qwen3: 46% instead of 22% of MATH-500 questions; Appendix J), and students of the new corpus never answer directly on the four benchmarks under either objective. Reasoning content and answer style are not separated within the T1 targets: T1 versus T2-T5 labels the generation policy, not the content of a target (Section 7).

Thank you for the additional experiments. Replacing the direct-answer targets with T2 targets is a useful control and strengthens the explanation of the large normalization effect in the original mixture. This addresses part of my concern. My remaining reservation is primarily about the significance of the contribution: the results provide a clearer controlled case study within one teacher pool, but I am still not convinced that they offer sufficiently new insight into synthetic reasoning-data design. I therefore retain my weak-reject rating, while acknowledging that the rebuttal strengthens the empirical evidence.

Reviewer_jDomOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies implicit question weighting in comparisons of synthetic reasoning data. Under the token-mean objective, each question’s mean token loss is weighted by its number of supervised tokens within an accumulation window. A data recipe therefore determines both supervision content and question weights. The authors compare FIXED-T2 and GUIDED-MIX-T2SUB on the same questions, holding target strings fixed within each recipe while changing between token-mean and equal-example objectives. A shared-weight control and interventions that fix dataset loss shares further examine how weighting affects the accuracy gap between the recipes.

Critical strengths
  1. The paper addresses a concrete methodological issue. Data recipes jointly change target content and length, while token-mean training also changes question weights. Quantifying how this coupling affects recipe comparisons helps interpret performance differences between synthetic reasoning datasets more reliably.
  2. The shared-weight control strengthens the interpretation of the main comparison. Applying the mixture’s question weights to FIXED-T2 targets within each accumulation window tests whether the weights alone explain the differing responses. Additional experiments hold dataset loss shares fixed and distinguish changes in group shares from changes in weights within each group, clarifying the effects of different redistributions.
  3. The main effects are substantial and supported across training seeds. The reported recipe-by-objective interactions are 8.31 and 5.23 percentage points on Qwen3-4B-Base and OLMo-3-7B, respectively, with consistent directions across three seeds. Seed-level intervals support the main interactions and the ordering of group intervention costs. The objective gap on the mixed corpus also persists after separate learning-rate selection.
  4. Negative findings and study boundaries are reported transparently. The reliability-guided allocation shows no consistent advantage over the matched-random control, and the registered non-inferiority hypothesis is not supported. The authors do not interpret nonsignificant results as equivalence. They also report quoted-instruction rescoring, bare-label scoring, and token-cap sensitivity, while distinguishing registered, pre-specified, post hoc, and descriptive analyses.
Critical weaknesses
  1. Some comparisons remain confounded by attention-kernel differences. Qwen3’s recipe-by-objective conditions use different kernels. Retraining the GUIDED-MIX-T2SUB token-mean condition retains an objective gap of 7.91 percentage points, but the original interaction of 8.31 percentage points lacks a fully matched estimate. Rerunning FIXED-T2 under token-mean training with the same kernel would address this concern. The OLMo main interaction uses matched kernels.
  2. Optimization robustness is only partially established. All four learning rates selected for GUIDED-MIX lie at the lower boundary of the search grid, and tuning does not cover the full recipe-by-objective matrix. A broader learning-rate search with consistent selection across conditions would strengthen the conclusions.
  3. Generality and independent confirmation remain limited. The main comparisons rely on a single teacher candidate pool, with prompted depth coupled to content, length, and format. Some follow-up hypotheses were developed after earlier results, although protocols were fixed before new runs. Three seeds support the main effects, while replication on independent teacher pools or mixed corpora would strengthen evidence for broader generalization.
Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the careful reading. We answer the three points in order.

1. Attention kernel. Concern: Qwen3's recipe-by-objective conditions use different kernels, so the interaction of 8.31 lacks a fully matched estimate.

We ran the experiment the reviewer suggested: we retrained the three Qwen3 Fixed-T2 token-mean runs with cuDNN attention disabled. With the retrained Guided-Mix-T2sub token-mean runs already in the paper, all four cells of the recipe-by-objective comparison now use the same kernel. The interaction is +7.82 points [6.18, 9.47] with sampled decoding (per seed 8.67 / 7.72 / 7.09) and +7.86 [6.27, 9.49] with greedy decoding. The submitted estimate is +8.31 [6.77, 9.87]. The retrained Fixed-T2 runs differ from the original ones by -0.24 points [-1.11, +0.63].

2. Learning rate. Concern: the selected learning rates lie at the lower boundary of the grid, and tuning does not cover the full recipe-by-objective matrix.

The check was run on the Guided-Mix corpus under both objectives; Table 16 (Appendix K) shows all twelve candidates. On the development set, the gap between the objectives is present at each of the three rates: 7.5 / 8.0 / 7.5 points on Qwen3 and 5.5 / 7.0 / 5.0 points on OLMo, at 1e-5 / 2e-5 / 4e-5 (sampled decoding, seed 0). At the selected rates, on the test sets with three seeds, the gap is +8.50 [7.18, 9.89] on Qwen3 and +6.35 [5.01, 7.72] on OLMo (Section 6).

3. Teacher pool. Concern: the main comparisons rely on a single teacher candidate pool.

The paper states its scope as one teacher pool and two student families (Section 7). Within this scope, the equal-example objective also lowers four-benchmark accuracy for the matched-random control, another policy on the same pool: by 9.67 points [8.37, 10.97] on Qwen3 and 6.36 [5.13, 7.61] on OLMo (Section 4).

Reviewer_jDomOpenReview ↗

Response to Authors

Thank you for your reply. I think most of my concerns have been addressed. I would like to maintain my original positive rating of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.