acceptodds
Under review as a conference paper at ICLR 2027

BREW: Progress-Aware Token Weighting for Open-Ended Writing

Abstract

Supervised fine-tuning (SFT) for open-ended writing learns from a single recorded realization even where several continuations are plausible. Under uniform SFT, every supervised token keeps weight one throughout training, so continued fitting concentrates probability on the recorded realization even after the token becomes predictable, a phenomenon we call single-reference overcommitment. Existing confidence-based weighting partly eases this problem by downweighting low-confidence tokens, but its weight can keep rising as the target becomes more predictable, leaving no signal to reduce weighting after substantial progress. We introduce BREW (Base-Relative Weighting), which compares each token's current loss with its loss under the frozen base model. The current loss measures how learnable the token is now, while the base-relative ratio measures progress from its own starting point. This keeps the cautious entry of confidence weighting while adding a progress-aware exit. In the primary setting, BREW performs best on every writing metric, improving WritingBench by 2.1 and LongBench-Write by 3.7 over the strongest baseline. Its WritingBench advantage holds across models and datasets, and the same configuration remains competitive on answer-determined mathematic tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
supervised fine-tuning, open-ended writing, post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YisGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BREW, a token-weighting method for supervised fine-tuning that attempts to mitigate what the authors call single-reference overcommitment. The central argument is that ordinary SFT continues reinforcing a particular observed token even after the model has already learned it sufficiently, potentially suppressing other perfectly reasonable continuations.

BREW modifies TALR by incorporating the ratio between a token's current loss and its loss under the frozen base model. The resulting weight initially favors learnable tokens but eventually decreases as their loss falls substantially relative to the reference. The method requires caching base-model token losses but no additional reference-model forward passes during training.

The empirical results are promising. On the primary Qwen3.5-9B/DeepWriting configuration, BREW improves WritingBench from 76.3 to 78.4 and LongBench-Write from 83.4 to 87.1 relative to TALR. The WritingBench improvement also generalizes across six model–dataset settings.

I find the empirical contribution considerably more convincing than the proposed explanation for why the method works. The authors have demonstrated that base-relative token weighting can improve benchmark performance. They have not, in my view, established that single-reference overcommitment is the mechanism responsible for those improvements.

Critical strengths
  1. A simple, technically motivated modification to an existing objective.

The method is straightforward to understand and implement. Using each token's frozen-base loss as a reference is a reasonable way to distinguish a token that was always easy from one that became easy during training. The rise-then-fall weighting behavior is analytically characterized, and the connection to TALR at γ = 0 makes the contribution reasonably clear.

  1. The empirical results are more extensive than is typical for a token-weighting paper.

The authors compare against SFT, DFT, ASFT, TALR, and InfoSFT. They also include additional Qwen architectures, a Llama model, a second writing corpus, and a mathematics transfer experiment. The consistent WritingBench improvements across these settings are difficult to dismiss as an isolated benchmark artifact.

  1. The ablations are useful.

Table 2 is particularly persuasive. Replacing the per-token frozen reference with a corpus-level constant reduces WritingBench from 78.4 to 76.6, while periodically refreshing the reference reduces it to 76.1. This supports the importance of retaining token-specific information from initialization.

I also appreciate the inclusion of the weight-floor analysis and training trajectories. They make the paper substantially more informative than a simple benchmark comparison.

Critical weaknesses
  1. The paper does not convincingly establish the phenomenon it claims to solve.

My central concern is the gap between the claimed mechanism and the evidence presented.

The authors argue that single-reference SFT suppresses plausible alternative continuations through excessive fitting. BREW supposedly prevents this by reducing token weights after sufficient learning progress.

But where is the systematic evidence that this is actually what improves the generated writing?

Figure 3 shows lower rates of near-zero training loss. That establishes that BREW produces fewer near-deterministic predictions of supervised tokens. It does not establish that the retained probability mass is assigned to semantically appropriate alternatives, or that the resulting model preserves meaningful creative diversity.

Table 3 provides two branching-point examples from the same prompt. These examples are suggestive, but they are not compelling evidence for a general distributional phenomenon.

I acknowledge that Appendix I includes judge-free diagnostics such as distinct-4, template mass, and anchored novelty. Those are useful additions. However, the diagnostics use one generation per prompt and do not directly measure the diversity of plausible responses to the same prompt. Lexical diversity, avoidance of repeated templates, and conditional generation diversity are not identical concepts.

An alternative explanation is that BREW acts as a form of adaptive regularization, reducing excessive fitting and improving generalization without specifically addressing ambiguity in open-ended supervision.

That explanation seems entirely consistent with the reported results.

I would like to see repeated sampling on held-out prompts, with measurements of conditional semantic diversity and human judgments of whether alternative outputs remain coherent and instruction-compliant. A controlled experiment using multiple valid references would be especially informative.

  1. The matched-effective-learning-rate control is not a convincing control for AdamW.

Section 4.3.1 argues that BREW's gains cannot be explained by a reduction in effective step size. The evidence is a uniform-SFT experiment with the learning rate reduced from 1.5 × 10⁻⁵ to 5.6 × 10⁻⁶, based on BREW's average token weight of 0.37.

I do not think this comparison establishes what the authors claim.

The experiments use AdamW. Multiplying gradients by a constant is not equivalent to multiplying AdamW's learning rate by that constant. In the idealized setting, Adam's moment normalization makes updates approximately invariant to uniform gradient rescaling, apart from effects involving ε, clipping, weight decay, and changing gradient statistics.

BREW's weights are not even uniform. They modify which token gradients contribute to the update, which changes both the gradient direction and the optimizer's moment estimates.

Consequently, the scalar mean token weight is not an appropriate proxy for the effective learning rate.

The poor performance of SFT at 5.6 × 10⁻⁶ therefore tells us relatively little about whether BREW benefits from gradient-scale effects, altered gradient composition, or the specific proposed progress mechanism.

A more informative experiment would compare batch-normalized weights, with the mean BREW weight rescaled to one, or otherwise control the gradient norm and AdamW update magnitude. I would also like to see the proportion of clipped updates across methods.

The frozen-reference ablations remain valuable, but this particular control is not adequate justification for the stronger causal conclusion.

  1. The adaptive temperature is a potential confound that has not been fully isolated.

The paper emphasizes the introduction of the base-relative progress term. However, the method also changes the statistic from which TALR's adaptive temperature is computed.

Appendix C reports that TALR's temperature decreases from 1.30 to 0.64 over the measured checkpoints, whereas BREW's increases from 2.24 to 2.97.

This is a substantial difference in the dynamics of the weighting objective.

The authors attribute the improved results primarily to the progress-aware exit behavior. But some fraction of that improvement could plausibly come from how the altered temperature affects all token weights, not merely from downweighting tokens that have made substantial progress.

Proposition 1 characterizes the weight profile for fixed temperature. The implemented system, however, uses a temperature that changes with the distribution of effective losses.

I would like an ablation that separates the direct contribution of the base-relative weighting term from the indirect contribution of the altered temperature trajectory. For example, how does BREW perform under a fixed temperature or an externally matched temperature schedule?

Without such a control, the paper has established that the overall objective works, but has not fully isolated the mechanism responsible.

  1. The evaluation needs stronger uncertainty reporting and better control of output length.

The main results are averaged over three runs, which is welcome. However, the primary tables do not report standard deviations, confidence intervals, or significance tests for the WritingBench improvements.

This matters particularly for the smaller transfer gains, including the 0.73-point improvement on Llama-3.1-8B-Instruct.

The external benchmark results also rely on LLM judgments. Appendix K documents the evaluation protocol and reports that each output is judged once. This is understandable operationally, but leaves the reliability of relatively small score differences unclear.

There is another issue: Figure 6 shows that BREW's average output length is approximately 2,221 tokens, compared with 3,854 for ordinary SFT. That is a substantial change in the distribution of generated responses.

A shorter response can be more coherent, avoid repetition, and receive a higher rubric score. Those are legitimate improvements, but they are not necessarily evidence of preserved creative alternatives.

The LongBench-Write length-compliance results provide some reassurance, particularly in the main setting, but do not eliminate this concern. Indeed, BREW does not achieve the best overall LongBench-Write score in both transfer settings, despite achieving the best judged-quality component.

I would like to see paired confidence intervals, some length-stratified comparisons, and preferably a limited human preference evaluation covering both writing quality and diversity.

Questions for authors

Q1. Can the authors provide a systematic evaluation of conditional response diversity using multiple generations per held-out prompt? How does BREW affect semantic diversity rather than merely lexical variation?

Q2. What proportion of BREW's gain remains when its token weights are batch-normalized or when AdamW update magnitudes are controlled? Why is scaling the SFT learning rate by the average token weight considered an appropriate control for an adaptive optimizer?

Q3. Can the authors disentangle the effect of the token-specific progress ratio from the altered adaptive-temperature trajectory?

Q4. What are the seed-level variances and paired confidence intervals for the main and transfer evaluations? Do the smaller improvements remain statistically reliable?

Q5. How much of the improvement persists after controlling for response length? Does BREW retain its advantage under human judgments of writing quality and creative diversity?

2 comments

We sincerely thank the reviewer for their constructive comments and recognition of our theoretical contributions. We address the concerns below.

Q1.

Can the authors provide a systematic evaluation of conditional response diversity using multiple generations per held-out prompt? How does BREW affect semantic diversity rather than merely lexical variation?

Response: We independently sampled five responses per method on 300 shared held-out prompts, using identical decoding for SFT, TALR, and BREW (4,500 responses). Within each prompt and method, semantic distance averages ten pairwise cosine distances between summary embeddings covering thesis, arguments, perspectives, and content. An independent LLM rates each five-response set's content and perspective diversity (1–5), excluding rewording alone, errors, and off-topic content. Critic-7B scores each response; we average the five scores per prompt. All metrics are then averaged across prompts.

MetricSFTTALRBREWBREW − TALR
Summary Semantic Distance0.2850.3050.335+0.030
Content Diversity (1–5)2.552.722.90+0.18
Perspective Diversity (1–5)2.202.352.60+0.25
WritingBench Quality (0–100)74.7076.8078.03+1.23

Human evaluation. Four annotators each compared BREW with both baselines on the same 30 prompts: 240 tasks, 120 judgments per comparison/criterion. Each task presents two anonymized sets of five full responses; positions and order are randomized. Quality-constrained diversity considers quality, relevance, and instruction following alongside meaningful variation.

CriterionBREW vs. TALR: W/T/LNon-tie win rateBREW vs. SFT: W/T/LNon-tie win rate
Content diversity34/64/2260.7%45/52/2366.2%
Perspective diversity33/62/2556.9%42/55/2364.6%
Quality-constrained diversity30/66/2455.6%40/57/2363.5%

W/T/L is from BREW's perspective; non-tie win rate = W/(W+L).

We will include these results and evaluation protocols in the revision.

Q2.

What proportion of BREW's gain remains when its token weights are batch-normalized or when AdamW update magnitudes are controlled? Why is scaling the SFT learning rate by the average token weight considered an appropriate control for an adaptive optimizer?

Response: During training, BREW-Norm divides valid token weights by their global-batch mean, setting it to one while preserving weight ratios. The original floor is applied first; normalized weights remain detached and are not capped at one.

MethodWBWB-AWB-BWB-CWB-DWB-EWB-F
SFT74.2772.1775.4471.4475.4675.3576.88
BREW78.4379.0578.9276.7378.6378.5279.12
BREW-Norm78.1979.1178.1776.0778.6878.5079.38

WB-A–F follow the paper's domain order.

Relative to the reported main means, Norm retains 94.2% of BREW's gain over SFT: 3.92/4.16. The table below reports pre-clipping gradients, not AdamW updates.

MethodMedian pre-clipping gradient normSteps with pre-clipping norm > 1.0
SFT0.6723.56%
BREW0.2371.38%
BREW-Norm0.5763.10%

AdamW uses moving averages of gradients and squared gradients, so multiplying SFT's learning rate by average token weight does not ensure matched parameter-update sizes. In the revised manuscript, we will label that control a reduced-learning-rate SFT ablation and add the normalization results and gradient diagnostics.

Q3.

Can the authors disentangle the effect of the token-specific progress ratio from the altered adaptive-temperature trajectory?

Response: We fix both methods' training-weight temperature at τ = 1.21 (Appendix B's initialization), with γ = 0 for TALR and 0.75 for BREW. Other training settings are matched on Qwen3.5-9B/DeepWriting.

MethodWBWB-AWB-BWB-CWB-DWB-EWB-F
TALR, fixed τ77.0178.2677.3574.7677.0477.6077.80
BREW, fixed τ78.2378.1978.4676.8679.1778.5478.45
BREW − TALR+1.22−0.07+1.12+2.10+2.13+0.94+0.66

Differences are computed before rounding.

With identical temperature at every step, BREW gains 1.22 points [95% prompt-paired CI: 0.76, 1.71]. Fixed BREW's 78.23 is numerically close to adaptive BREW's 78.43. Weights still change with token losses and relative progress at fixed temperature. Temperature determines how strongly effective loss reduces weight; adaptive BREW recalculates it as the median of per-sequence mean effective losses.

We will add this ablation and clarify both components' roles in the revision.

Q4.

What are the seed-level variances and paired confidence intervals for the main and transfer evaluations? Do the smaller improvements remain statistically reliable?

Response: All reported paired 95% CIs for WritingBench gains are above zero, including the smallest transfer gain: +0.73 points on Llama [0.22, 1.25].

Main setting: Qwen3.5-9B/DeepWriting. The following statistics summarize three independent training runs.

MethodWB (mean ± SD)Approx. seed variance95% CI of the mean
SFT74.27 ± 0.130.0169[73.93, 74.60]
DFT65.60 ± 0.260.0676[64.97, 66.24]
ASFT72.72 ± 0.170.0289[72.29, 73.14]
InfoSFT75.72 ± 0.210.0441[75.20, 76.24]
TALR76.27 ± 0.240.0576[75.68, 76.87]
BREW78.43 ± 0.160.0256[78.04, 78.82]

Variances approximate the squares of rounded SDs.

We pair BREW with same-seed TALR or SFT on each prompt and bootstrap the score differences. These CIs reflect prompt resampling, separately from run-to-run SDs.

ComparisonSeed 1: gain [95% CI]Seed 2: gain [95% CI]Seed 3: gain [95% CI]
BREW − TALR+2.44 [2.01, 2.87]+1.71 [1.28, 2.15]+2.32 [1.88, 2.75]
BREW − SFT+4.40 [3.41, 5.42]+3.93 [2.96, 4.95]+4.16 [3.16, 5.20]

Mean gains across runs are +2.16 over TALR and +4.16 over SFT.

Transfer. Method statistics use three runs; prompt-paired CIs compare BREW with the strongest baseline in each setting.

SettingStrongest baseline: mean ± SD; 95% mean CIBREW: mean ± SD; 95% mean CIMean gain95% prompt-paired CI
Qwen3-8B / DeepWritingInfoSFT: 74.07 ± 0.24; [73.48, 74.67]75.70 ± 0.18; [75.25, 76.15]+1.63[1.19, 2.07]
Qwen3.5-9B / LongWriterASFT: 69.73 ± 0.23; [69.17, 70.28]70.76 ± 0.14; [70.42, 71.10]+1.03[0.43, 1.59]
Llama-3.1-8B-Instruct / LongWriterTALR: 66.79 ± 0.16; [66.38, 67.20]67.52 ± 0.12; [67.22, 67.81]+0.73[0.22, 1.25]

Approximate BREW/baseline seed variances: 0.0324/0.0576, 0.0196/0.0529, and 0.0144/0.0256.

We will add seed-level variability and paired confidence intervals to the main and transfer results in the revised manuscript.

Q5.

How much of the improvement persists after controlling for response length? Does BREW retain its advantage under human judgments of writing quality and creative diversity?

Response: We estimate the equal-length score gap by linearly regressing per-prompt score differences on the log BREW-to-baseline character-length ratio, using one response per prompt and method. Matching directly selects same-prompt responses whose actual length differences are within the stated percentage of baseline length.

AnalysisBREW − TALR [95% CI]BREW − SFT [95% CI]
Unadjusted+2.16 [1.72, 2.61]+4.16 [3.17, 5.18]
Log-length adjusted+1.48 [1.00, 2.00]+0.61 [−0.26, 1.49]
Length difference ≤ 5%+0.34 [−0.45, 1.16]−0.27 [−1.31, 0.98]
Length difference ≤ 10%+0.82 [0.22, 1.41]+0.67 [−0.27, 1.68]
Length difference ≤ 20%+0.52 [0.08, 0.97]+0.52 [−0.18, 1.25]

Matched counts at 5%/10%/20%: TALR 128/246/476; SFT 96/183/365. CIs use paired prompt bootstrap; regression is refitted in each resample.

The adjusted TALR gap is 68.5% of the original. SFT's 5% estimate, −0.27, has a CI spanning zero and describes the selected subset with already-similar lengths.

Length groups. We sort prompts by their median character length across the three methods and split them into three similarly sized groups. Both comparisons use the same groups and original responses.

Response-length groupPromptsBREW − TALR [95% CI]BREW − SFT [95% CI]
Short: ≤ 3,537 characters333+1.69 [0.99, 2.39]−0.42 [−1.25, 0.43]
Medium: 3,538–8,728 characters332+2.31 [1.58, 3.07]+1.99 [0.59, 3.46]
Long: ≥ 8,729 characters333+2.47 [1.71, 3.24]+10.92 [8.61, 13.27]

Gains over SFT concentrate in medium and long groups. Its longest outputs include repetitive continuations (mean length 16,845, median 6,679 characters).

Q1's human results support quality-constrained diversity: 55.6%/63.5% non-tie wins against TALR/SFT. Individual-response writing quality is evaluated separately with Critic-7B.

We will include these length controls, length-stratified results, and human evaluation criteria in the revised manuscript.

Reviewer_t988OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes BREW for supervised fine-tuning on open-ended writing. Existing confidence-based token weighting schemes (DFT, TALR, InfoSFT) react only to how learnable a token is at the moment: they provide cautious entry by down-weighting tokens that remain difficult, but offer no exit signal for tokens that have already been learned. BREW addresses this by caching the frozen base model's per-token loss before training as a starting point, and during training measuring relative progress as the ratio of the current loss to this reference, which enters the weight as a penalty. The resulting weight rises and then falls, suppressed when a token is still too hard, maximal when it is half-learned, and receding once substantial progress relative to the base has been made, yielding a moving curriculum over training. On the main Qwen3.5-9B setting, BREW ranks first on all ten writing metrics, with substantial gains over the strongest baselines.

Critical strengths
  1. The method is simple and practical. BREW uses the ratio between the current loss and the frozen initial loss to reduce token weights after substantial learning progress, extending confidence-based weighting with a progress-aware mechanism. Ablations on the per-token reference, frozen reference, and weight floor support these design choices. The method requires only a one-time offline cache, making it practical to implement.
  2. Performance gains are consistent across settings. In the primary setting, BREW improves WritingBench and LongBench-Write by 2.1 and 3.7 points, respectively, over the strongest baselines. It also achieves the highest WritingBench score across six model–dataset combinations, supporting the consistency of its empirical benefits.
Critical weaknesses
  1. Evidence for the central mechanism remains indirect. A lower proportion of near-zero-loss tokens indicates less extreme fitting, but does not establish that the method specifically mitigates unnecessary single-reference overfitting in open-ended writing. The paper reports template mass, Distinct-4, and novelty diagnostics, but these are based on a single generation per prompt, and BREW does not clearly outperform the strongest baselines on Distinct-4 or anchored novelty. Thus, the results support reduced template usage but do not adequately establish greater diversity among plausible outputs for the same prompt.
  2. The statistical reliability of the gains is insufficiently characterized. The paper reports averages over three runs but provides no standard deviations or confidence intervals for the main results. Some gains in transfer settings are below one point, and evaluation relies substantially on LLM judges. It is therefore difficult to determine whether these smaller improvements consistently exceed training and evaluation variability.
Questions for authors
  1. Could the authors report standard deviations or confidence intervals for the main experimental results, along with paired score differences between BREW and the strongest baselines and confidence intervals for those differences? This would help establish whether the improvements are robust to training and evaluation variability.
  2. Could the authors supplement the current results with diversity evaluations using multiple generations per prompt, including human assessments of whether the outputs exhibit greater variation in content, perspectives, or expression while maintaining quality? The current lexical metrics based on a single generation per prompt do not directly establish this.
2 comments

We sincerely thank the reviewer for their constructive comments. We address the concerns below.

Q1.

Could the authors report standard deviations or confidence intervals for the main experimental results, along with paired score differences between BREW and the strongest baselines and confidence intervals for those differences? This would help establish whether the improvements are robust to training and evaluation variability.

Response: We report WritingBench mean ± SD and 95% CIs for the mean across three independent training runs.

Model / training datasetMethodWB, mean ± SD95% CI for the method mean
Qwen3.5-9B / DeepWritingTALR76.27 ± 0.24[75.68, 76.87]
BREW78.43 ± 0.16[78.04, 78.82]
Qwen3-8B / DeepWritingInfoSFT74.07 ± 0.24[73.48, 74.67]
BREW75.70 ± 0.18[75.25, 76.15]
Qwen3.5-9B / LongWriterASFT69.73 ± 0.23[69.17, 70.28]
BREW70.76 ± 0.14[70.42, 71.10]
Llama-3.1-8B-Instruct / LongWriterTALR66.79 ± 0.16[66.38, 67.20]
BREW67.52 ± 0.12[67.22, 67.81]

In the main setting, we compare BREW with the baseline trained using the same seed on each prompt, then average their score differences. Paired bootstrap CIs resample prompts while keeping recorded responses and scores fixed:

ComparisonSeed 1: difference [95% CI]Seed 2: difference [95% CI]Seed 3: difference [95% CI]
BREW − TALR+2.44 [2.01, 2.87]+1.71 [1.28, 2.15]+2.32 [1.88, 2.75]
BREW − SFT+4.40 [3.41, 5.42]+3.93 [2.96, 4.95]+4.16 [3.16, 5.20]

The three transfer comparisons use the strongest baseline in each setting:

Model / training datasetComparisonPaired WB difference95% paired CI
Qwen3-8B / DeepWritingBREW − InfoSFT+1.63[1.19, 2.07]
Qwen3.5-9B / LongWriterBREW − ASFT+1.03[0.43, 1.59]
Llama-3.1-8B-Instruct / LongWriterBREW − TALR+0.73[0.22, 1.25]

All reported main and transfer paired CIs lie above zero, including the smaller Llama gain.

We will add the run-level variability and paired confidence intervals to the revised main and transfer result tables.

Q2.

Could the authors supplement the current results with diversity evaluations using multiple generations per prompt, including human assessments of whether the outputs exhibit greater variation in content, perspectives, or expression while maintaining quality? The current lexical metrics based on a single generation per prompt do not directly establish this.

Response: SFT, TALR, and BREW each generated five independent responses to the same 300 held-out prompts under identical decoding settings: 300 × 3 × 5 = 4,500 responses.

Summary Semantic Distance averages the ten pairwise cosine distances between summary embeddings of each method's five responses. An independent LLM rates each five-response set for content and perspective diversity (1–5), excluding mere rewording, errors, and off-topic content. Content compares arguments, examples and solutions; perspective compares viewpoints and explanatory approaches. Critic-7B scores each response; the five quality scores are averaged per prompt. All four metrics are then averaged across prompts.

MetricSFTTALRBREWBREW − TALR
Summary Semantic Distance ↑0.2850.3050.335+0.030
Content Diversity, 1–5 ↑2.552.722.90+0.18
Perspective Diversity, 1–5 ↑2.202.352.60+0.25
WritingBench Quality, 0–100 ↑74.7076.8078.03+1.23

Human evaluation. All four annotators evaluated the same 30 prompts, each comparing BREW's five responses with TALR's and with SFT's: 240 blinded comparisons, with 120 judgments per baseline and criterion. Quality-constrained diversity asks which response set provides more meaningful variation while considering writing quality, relevance, and instruction following.

CriterionBREW vs. TALR: win / tie / lossNon-tie win rateBREW vs. SFT: win / tie / lossNon-tie win rate
Content diversity34 / 64 / 2260.7%45 / 52 / 2366.2%
Perspective diversity33 / 62 / 2556.9%42 / 55 / 2364.6%
Quality-constrained diversity30 / 66 / 2455.6%40 / 57 / 2363.5%

Counts are BREW wins/ties/losses; non-tie win rate = wins/(wins + losses). BREW leads all four automatic metrics and receives more human wins than losses on every criterion against both baselines.

We will include these results and evaluation protocols in the revised manuscript.

Thank you for response and extra experiments. The standard deviations and paired confidence intervals address my concerns about the statistical reliability of the results. In addition, the repeated generation experiments with the same prompt and the human evaluation provide more direct evidence that BREW preserves output diversity, although the improvements remain relatively limited. I will keep my score.

Reviewer_o2YYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the risk that single-reference supervised fine-tuning for open-ended writing may overemphasize one particular realization. It proposes BREW, a progress-aware token-weighting method that compares each token’s current loss with its loss under the frozen initial model, reducing its weight after substantial relative progress while retaining confidence-based weighting. The method is evaluated through writing benchmarks across multiple models and datasets, component ablations, and training-dynamics analyses.

Critical strengths
  1. The problem is practically relevant, and the method is simple.

  2. The writing improvements are reasonably consistent.

  3. The ablations address several relevant design choices.

Critical weaknesses
  1. This paper lacks important baselines that are directly relevant to the central motivation, including cited GEM, PriFT, and LP-SFT methods.

  2. There is a lack of explicit connection between the theoretical analysis and the implemented algorithm. Specifically, Proposition 1 characterizes the weight profile under a fixed temperature, whereas the actual temperature changes.

  3. A lower fraction of near-zero-loss training tokens does not actually mean that more plausible continuations are preserved.

Questions for authors

Please see weaknesses.

4 comments

We sincerely thank the reviewer for their constructive comments. We address the concerns below.

Q1. Related baselines

Although the experiments cover several token-weighting methods, they lack of comparisons with the cited GEM, PriFT, and LP-SFT methods.

Response: We added LP-SFT and PriFT for Qwen3.5-9B/DeepWriting. LP-SFT aims to preserve alternative-token distributions; PriFT determines weights using a frozen reference model.

MethodWBWB-AWB-BWB-CWB-DWB-EWB-F
LP-SFT75.276.576.473.973.675.775.5
PriFT74.775.174.573.774.076.775.4
BREW78.479.178.976.778.678.579.1

WB-A–F: Academic & Engineering; Finance & Business; Politics & Law; Literature & Arts; Education; Advertising & Marketing.

BREW gains 3.2 points over LP-SFT and 3.7 over PriFT, leading all six domains.

We will add these baseline results and the comparison with LP-SFT and PriFT to the revised manuscript.

Q2. Downweighting with changing temperature

Proposition 1 characterizes the weight profile under a fixed temperature, whereas the actual temperature changes with the batch state. Consequently, this result does not directly guarantee the claimed exit behavior during training.

Response: With adaptive temperatures recomputed at each checkpoint, 79.13% of all tracked token positions exhibit at least one post-peak weight decrease: their loss is already below γ\gamma and continues to decrease, while their weight also falls. Among positions with such loss reductions, 97.78% exhibit at least one accompanying weight decrease. This is the proposed exit behavior.

We evaluate the same 256 randomly sampled training records at checkpoints 30, 90, 210, 450, and 870, keeping batch grouping fixed. At each checkpoint, we recompute losses, temperature (the median of per-sequence mean effective losses), and weights, using γ=0.75\gamma=0.75 and wmin⁡=0.1w_{\min}=0.1. We compare each supervised position between consecutive evaluated checkpoints.

StatisticResult
Tracked supervised positions722,793
Positions with at least one post-peak learning interval584,988
Positions with at least one post-peak downweighting event571,974
Downweighted positions / all tracked positions79.13%
Downweighted positions / positions with post-peak learning97.78%
Median relative weight reduction at first observed event19.19%

Each position is counted once per condition. Relative weight reductions use the first observed post-peak downweighting event.

Proposition 1 holds temperature fixed. For 0<γ<ℓb,i0 < \gamma < \ell_{b,i}, the weight before applying the floor is largest at ℓi=γ\ell_i=\gamma, for any positive fixed temperature. When temperature changes, weight decreases exactly when

\quad\text{and}\quad w_{i,t_a} > w_{\min},$$ Here $u_i$ adjusts the current loss for progress relative to the frozen base. The comparison uses both the changing loss and temperature. We will add the dynamic condition, checkpoint evaluation protocol, and token-level results alongside Proposition 1 in the revised manuscript.

Q3. Diversity among plausible responses

A lower fraction of near-zero-loss training tokens does not imply that more plausible continuations are preserved.

Response: SFT, TALR, and BREW each independently generated five responses to each of 300 held-out prompts under identical decoding settings: 4,500 responses.

We summarize each response’s thesis, arguments, perspectives, and key content. Summary Semantic Distance averages the pairwise cosine distances between five summary embeddings within a prompt. An independent LLM gives each set 1–5 scores for content diversity (arguments, examples, solutions) and perspective diversity (viewpoints, explanations), excluding rewording, errors, and off-topic content. The original Critic-7B pipeline scores each response; we average the five quality scores per prompt. Each metric is then averaged across prompts.

MetricSFTTALRBREW
Summary Semantic Distance0.2850.3050.335
Content diversity (1–5)2.552.722.90
Perspective diversity (1–5)2.202.352.60
WritingBench quality (0–100)74.7076.8078.03

Human evaluation: Four annotators assessed the same 30 prompts. Each separately compared BREW’s five full responses with TALR’s and SFT’s: 240 tasks, 120 judgments per baseline and criterion. Model identities were hidden; set positions and within-set response order were randomized. Quality-constrained diversity compares content and viewpoint differences while considering writing quality, relevance, and instruction following.

BaselineCriterionBREW win/tie/lossNon-tie win rate
TALRContent34/64/2260.7%
TALRPerspective33/62/2556.9%
TALRQuality-constrained30/66/2455.6%
SFTContent45/52/2366.2%
SFTPerspective42/55/2364.6%
SFTQuality-constrained40/57/2363.5%

Non-tie win rate = wins/(wins + losses). BREW receives more wins than losses on every criterion against both baselines.

We will include the repeated-generation results, human evaluation, and evaluation protocol in the revised manuscript.

Reviewer_o2YYOpenReview ↗

Thanks for the response

I appreciate the new baselines added. For question 2, the results show that most tokens are downweighted at least once but do not solve the question directly of whether they stay downweighted. From my understanding, the gap between an occasional weight decrease and a stable “exit” remains, because as the temperature changes, their weights may rise again even while their losses keep falling. I think it is important to draw the detailed trajectory of the weight changes to better show the motivation and this setting and where the reweighting method actually contributes; it is the core part and claim of this paper. The analysis is still superficial and intuitive. I encourage the authors to dig deeper, as there are likely many interesting findings to uncover. Therefore, I prefer to maintain my score.

I am also curious about whether an open-ended writing setting has this problem or if there are other scenarios that can also use this method (it is fine that the authors only focus on this task, just curious).

AuthorsOpenReview ↗

Further Analysis of Token-Weight Dynamics and Adaptive Temperature

Thank you for your thoughtful follow-up and for encouraging us to investigate the token-weight dynamics more deeply. We appreciate your recognition of the additional baselines. Building on the cohort-level trajectories in Figures 3B and 5B and the temperature analysis in Figure 5C, we have extended our analysis to individual token trajectories.

1. Proposition 1 and adaptive temperature

The peak location characterized by Proposition 1 is independent of whether temperature τ\tau is fixed or adaptive.

Let ui=ℓi−γlog⁡(ℓi/ℓb,i)u_i=\ell_i-\gamma\log(\ell_i/\ell_{b,i}) and w~i=exp⁡(−ui/τ)\widetilde w_i=\exp(-u_i/\tau). Under the sequence-median temperature estimator, locally, where median selection remains unchanged, τ=aui+b\tau=a u_i+b, with a≥0a\geq0 and b>0b>0. Thus,

=-\frac{b}{\tau^2}\left(1-\frac{\gamma}{\ell_i}\right).$$ For $\ell_{b,i}>\gamma$, the derivative changes sign at $\ell_i=\gamma$, preserving the peak location under adaptive temperature. Temperature controls the sharpness of weighting: lower temperatures push weights toward the floor, while higher temperatures make them more uniform. Across training steps, both effective loss and temperature change: $$\log\frac{\widetilde w_{i,t+1}}{\widetilde w_{i,t}} =-\frac{u_{i,t+1}-u_{i,t}}{\tau_t} +u_{i,t+1}\left(\frac1{\tau_t}-\frac1{\tau_{t+1}}\right).$$ The two terms describe changes in token effective loss and temperature adjustment, respectively. Once token NLL falls below $\gamma$, further loss reduction increases effective loss and promotes downweighting, while temperature adjustment can partially offset this effect. This directly connects Proposition 1 to the implemented training dynamics. ### 2. Persistence of downweighting We use the same 256 DeepWriting records and 722,793 supervised token positions as in our previous response, tracking checkpoints 30/90/210/450/870. Among the 487,813 positions with observations after their first downweighting event, **40.27% continue decreasing at every subsequent observed checkpoint, 80.00% remain below their pre-decrease weight throughout, and 89.12% remain below it at step 870**. For tokens with $\ell_b>\gamma$, corresponding to the interior peak in Proposition 1, the persistence rate reaches **85.81%**. These findings extend our earlier observation that most tokens experience downweighting, showing that **the reduction persists for most tokens across subsequent training stages**. ### 3. Temperature-related weight rebounds We further quantify how temperature adjustment affects subsequent weight increases. Among 744,733 post-downweighting intervals where token NLL is below $\gamma$ and continues to decrease, only 61,575 intervals (8.27%) exhibit a weight increase due to temperature adjustment. For temperature-driven rebounds during continued NLL reduction, the **median net weight increase is 0.0209**, accompanied by a **median absolute NLL decrease of 0.00034**. The median NLL before these rebounds is already only **0.00227**. Thus, these rebounds typically occur at well-fitted positions, with relatively small absolute loss reductions and modest weight increases. ### 4. Open-ended writing and broader applicability Reduced output diversity has been documented directly in open-ended writing. Klypa and Cherednichenko (2026) show that standard SFT reduces generative diversity, including in creative-writing evaluations [1]. Karouzos et al. (2026) identify substantial diversity loss during the SFT stage in one post-training lineage and examine diversity collapse on WritingPrompts [2]. These findings support our focus on writing, where many continuations may be valid but SFT typically supervises only one realization. Our repeated-generation and human evaluations provide further evidence of diversity differences in this setting. BREW itself is not writing-specific, and our mathematics experiments explore its applicability beyond open-ended writing. Thank you again for your constructive suggestions. We plan to upload the revised manuscript, including detailed token-weight trajectory plots and accompanying analyses, within approximately two hours. ### References [1] Roman Klypa and Oleksandr Cherednichenko. *Diversity in Large Language Models under Supervised Fine-Tuning*. 2026. https://arxiv.org/abs/2605.00195 [2] Constantinos Karouzos, Xingwei Tan, and Nikolaos Aletras. *Where Does Output Diversity Collapse in Post-Training?* 2026. https://arxiv.org/abs/2604.16027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.