- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BREW, a token-weighting method for supervised fine-tuning that attempts to mitigate what the authors call single-reference overcommitment. The central argument is that ordinary SFT continues reinforcing a particular observed token even after the model has already learned it sufficiently, potentially suppressing other perfectly reasonable continuations.
BREW modifies TALR by incorporating the ratio between a token's current loss and its loss under the frozen base model. The resulting weight initially favors learnable tokens but eventually decreases as their loss falls substantially relative to the reference. The method requires caching base-model token losses but no additional reference-model forward passes during training.
The empirical results are promising. On the primary Qwen3.5-9B/DeepWriting configuration, BREW improves WritingBench from 76.3 to 78.4 and LongBench-Write from 83.4 to 87.1 relative to TALR. The WritingBench improvement also generalizes across six model–dataset settings.
I find the empirical contribution considerably more convincing than the proposed explanation for why the method works. The authors have demonstrated that base-relative token weighting can improve benchmark performance. They have not, in my view, established that single-reference overcommitment is the mechanism responsible for those improvements.
Critical strengths
- A simple, technically motivated modification to an existing objective.
The method is straightforward to understand and implement. Using each token's frozen-base loss as a reference is a reasonable way to distinguish a token that was always easy from one that became easy during training. The rise-then-fall weighting behavior is analytically characterized, and the connection to TALR at γ = 0 makes the contribution reasonably clear.
- The empirical results are more extensive than is typical for a token-weighting paper.
The authors compare against SFT, DFT, ASFT, TALR, and InfoSFT. They also include additional Qwen architectures, a Llama model, a second writing corpus, and a mathematics transfer experiment. The consistent WritingBench improvements across these settings are difficult to dismiss as an isolated benchmark artifact.
- The ablations are useful.
Table 2 is particularly persuasive. Replacing the per-token frozen reference with a corpus-level constant reduces WritingBench from 78.4 to 76.6, while periodically refreshing the reference reduces it to 76.1. This supports the importance of retaining token-specific information from initialization.
I also appreciate the inclusion of the weight-floor analysis and training trajectories. They make the paper substantially more informative than a simple benchmark comparison.
Critical weaknesses
- The paper does not convincingly establish the phenomenon it claims to solve.
My central concern is the gap between the claimed mechanism and the evidence presented.
The authors argue that single-reference SFT suppresses plausible alternative continuations through excessive fitting. BREW supposedly prevents this by reducing token weights after sufficient learning progress.
But where is the systematic evidence that this is actually what improves the generated writing?
Figure 3 shows lower rates of near-zero training loss. That establishes that BREW produces fewer near-deterministic predictions of supervised tokens. It does not establish that the retained probability mass is assigned to semantically appropriate alternatives, or that the resulting model preserves meaningful creative diversity.
Table 3 provides two branching-point examples from the same prompt. These examples are suggestive, but they are not compelling evidence for a general distributional phenomenon.
I acknowledge that Appendix I includes judge-free diagnostics such as distinct-4, template mass, and anchored novelty. Those are useful additions. However, the diagnostics use one generation per prompt and do not directly measure the diversity of plausible responses to the same prompt. Lexical diversity, avoidance of repeated templates, and conditional generation diversity are not identical concepts.
An alternative explanation is that BREW acts as a form of adaptive regularization, reducing excessive fitting and improving generalization without specifically addressing ambiguity in open-ended supervision.
That explanation seems entirely consistent with the reported results.
I would like to see repeated sampling on held-out prompts, with measurements of conditional semantic diversity and human judgments of whether alternative outputs remain coherent and instruction-compliant. A controlled experiment using multiple valid references would be especially informative.
- The matched-effective-learning-rate control is not a convincing control for AdamW.
Section 4.3.1 argues that BREW's gains cannot be explained by a reduction in effective step size. The evidence is a uniform-SFT experiment with the learning rate reduced from 1.5 × 10⁻⁵ to 5.6 × 10⁻⁶, based on BREW's average token weight of 0.37.
I do not think this comparison establishes what the authors claim.
The experiments use AdamW. Multiplying gradients by a constant is not equivalent to multiplying AdamW's learning rate by that constant. In the idealized setting, Adam's moment normalization makes updates approximately invariant to uniform gradient rescaling, apart from effects involving ε, clipping, weight decay, and changing gradient statistics.
BREW's weights are not even uniform. They modify which token gradients contribute to the update, which changes both the gradient direction and the optimizer's moment estimates.
Consequently, the scalar mean token weight is not an appropriate proxy for the effective learning rate.
The poor performance of SFT at 5.6 × 10⁻⁶ therefore tells us relatively little about whether BREW benefits from gradient-scale effects, altered gradient composition, or the specific proposed progress mechanism.
A more informative experiment would compare batch-normalized weights, with the mean BREW weight rescaled to one, or otherwise control the gradient norm and AdamW update magnitude. I would also like to see the proportion of clipped updates across methods.
The frozen-reference ablations remain valuable, but this particular control is not adequate justification for the stronger causal conclusion.
- The adaptive temperature is a potential confound that has not been fully isolated.
The paper emphasizes the introduction of the base-relative progress term. However, the method also changes the statistic from which TALR's adaptive temperature is computed.
Appendix C reports that TALR's temperature decreases from 1.30 to 0.64 over the measured checkpoints, whereas BREW's increases from 2.24 to 2.97.
This is a substantial difference in the dynamics of the weighting objective.
The authors attribute the improved results primarily to the progress-aware exit behavior. But some fraction of that improvement could plausibly come from how the altered temperature affects all token weights, not merely from downweighting tokens that have made substantial progress.
Proposition 1 characterizes the weight profile for fixed temperature. The implemented system, however, uses a temperature that changes with the distribution of effective losses.
I would like an ablation that separates the direct contribution of the base-relative weighting term from the indirect contribution of the altered temperature trajectory. For example, how does BREW perform under a fixed temperature or an externally matched temperature schedule?
Without such a control, the paper has established that the overall objective works, but has not fully isolated the mechanism responsible.
- The evaluation needs stronger uncertainty reporting and better control of output length.
The main results are averaged over three runs, which is welcome. However, the primary tables do not report standard deviations, confidence intervals, or significance tests for the WritingBench improvements.
This matters particularly for the smaller transfer gains, including the 0.73-point improvement on Llama-3.1-8B-Instruct.
The external benchmark results also rely on LLM judgments. Appendix K documents the evaluation protocol and reports that each output is judged once. This is understandable operationally, but leaves the reliability of relatively small score differences unclear.
There is another issue: Figure 6 shows that BREW's average output length is approximately 2,221 tokens, compared with 3,854 for ordinary SFT. That is a substantial change in the distribution of generated responses.
A shorter response can be more coherent, avoid repetition, and receive a higher rubric score. Those are legitimate improvements, but they are not necessarily evidence of preserved creative alternatives.
The LongBench-Write length-compliance results provide some reassurance, particularly in the main setting, but do not eliminate this concern. Indeed, BREW does not achieve the best overall LongBench-Write score in both transfer settings, despite achieving the best judged-quality component.
I would like to see paired confidence intervals, some length-stratified comparisons, and preferably a limited human preference evaluation covering both writing quality and diversity.
Questions for authors
Q1. Can the authors provide a systematic evaluation of conditional response diversity using multiple generations per held-out prompt? How does BREW affect semantic diversity rather than merely lexical variation?
Q2. What proportion of BREW's gain remains when its token weights are batch-normalized or when AdamW update magnitudes are controlled? Why is scaling the SFT learning rate by the average token weight considered an appropriate control for an adaptive optimizer?
Q3. Can the authors disentangle the effect of the token-specific progress ratio from the altered adaptive-temperature trajectory?
Q4. What are the seed-level variances and paired confidence intervals for the main and transfer evaluations? Do the smaller improvements remain statistically reliable?
Q5. How much of the improvement persists after controlling for response length? Does BREW retain its advantage under human judgments of writing quality and creative diversity?
2 comments
Q4.
What are the seed-level variances and paired confidence intervals for the main and transfer evaluations? Do the smaller improvements remain statistically reliable?
Response: All reported paired 95% CIs for WritingBench gains are above zero, including the smallest transfer gain: +0.73 points on Llama [0.22, 1.25].
Main setting: Qwen3.5-9B/DeepWriting. The following statistics summarize three independent training runs.
| Method | WB (mean ± SD) | Approx. seed variance | 95% CI of the mean |
|---|---|---|---|
| SFT | 74.27 ± 0.13 | 0.0169 | [73.93, 74.60] |
| DFT | 65.60 ± 0.26 | 0.0676 | [64.97, 66.24] |
| ASFT | 72.72 ± 0.17 | 0.0289 | [72.29, 73.14] |
| InfoSFT | 75.72 ± 0.21 | 0.0441 | [75.20, 76.24] |
| TALR | 76.27 ± 0.24 | 0.0576 | [75.68, 76.87] |
| BREW | 78.43 ± 0.16 | 0.0256 | [78.04, 78.82] |
Variances approximate the squares of rounded SDs.
We pair BREW with same-seed TALR or SFT on each prompt and bootstrap the score differences. These CIs reflect prompt resampling, separately from run-to-run SDs.
| Comparison | Seed 1: gain [95% CI] | Seed 2: gain [95% CI] | Seed 3: gain [95% CI] |
|---|---|---|---|
| BREW − TALR | +2.44 [2.01, 2.87] | +1.71 [1.28, 2.15] | +2.32 [1.88, 2.75] |
| BREW − SFT | +4.40 [3.41, 5.42] | +3.93 [2.96, 4.95] | +4.16 [3.16, 5.20] |
Mean gains across runs are +2.16 over TALR and +4.16 over SFT.
Transfer. Method statistics use three runs; prompt-paired CIs compare BREW with the strongest baseline in each setting.
| Setting | Strongest baseline: mean ± SD; 95% mean CI | BREW: mean ± SD; 95% mean CI | Mean gain | 95% prompt-paired CI |
|---|---|---|---|---|
| Qwen3-8B / DeepWriting | InfoSFT: 74.07 ± 0.24; [73.48, 74.67] | 75.70 ± 0.18; [75.25, 76.15] | +1.63 | [1.19, 2.07] |
| Qwen3.5-9B / LongWriter | ASFT: 69.73 ± 0.23; [69.17, 70.28] | 70.76 ± 0.14; [70.42, 71.10] | +1.03 | [0.43, 1.59] |
| Llama-3.1-8B-Instruct / LongWriter | TALR: 66.79 ± 0.16; [66.38, 67.20] | 67.52 ± 0.12; [67.22, 67.81] | +0.73 | [0.22, 1.25] |
Approximate BREW/baseline seed variances: 0.0324/0.0576, 0.0196/0.0529, and 0.0144/0.0256.
We will add seed-level variability and paired confidence intervals to the main and transfer results in the revised manuscript.
Q5.
How much of the improvement persists after controlling for response length? Does BREW retain its advantage under human judgments of writing quality and creative diversity?
Response: We estimate the equal-length score gap by linearly regressing per-prompt score differences on the log BREW-to-baseline character-length ratio, using one response per prompt and method. Matching directly selects same-prompt responses whose actual length differences are within the stated percentage of baseline length.
| Analysis | BREW − TALR [95% CI] | BREW − SFT [95% CI] |
|---|---|---|
| Unadjusted | +2.16 [1.72, 2.61] | +4.16 [3.17, 5.18] |
| Log-length adjusted | +1.48 [1.00, 2.00] | +0.61 [−0.26, 1.49] |
| Length difference ≤ 5% | +0.34 [−0.45, 1.16] | −0.27 [−1.31, 0.98] |
| Length difference ≤ 10% | +0.82 [0.22, 1.41] | +0.67 [−0.27, 1.68] |
| Length difference ≤ 20% | +0.52 [0.08, 0.97] | +0.52 [−0.18, 1.25] |
Matched counts at 5%/10%/20%: TALR 128/246/476; SFT 96/183/365. CIs use paired prompt bootstrap; regression is refitted in each resample.
The adjusted TALR gap is 68.5% of the original. SFT's 5% estimate, −0.27, has a CI spanning zero and describes the selected subset with already-similar lengths.
Length groups. We sort prompts by their median character length across the three methods and split them into three similarly sized groups. Both comparisons use the same groups and original responses.
| Response-length group | Prompts | BREW − TALR [95% CI] | BREW − SFT [95% CI] |
|---|---|---|---|
| Short: ≤ 3,537 characters | 333 | +1.69 [0.99, 2.39] | −0.42 [−1.25, 0.43] |
| Medium: 3,538–8,728 characters | 332 | +2.31 [1.58, 3.07] | +1.99 [0.59, 3.46] |
| Long: ≥ 8,729 characters | 333 | +2.47 [1.71, 3.24] | +10.92 [8.61, 13.27] |
Gains over SFT concentrate in medium and long groups. Its longest outputs include repetitive continuations (mean length 16,845, median 6,679 characters).
Q1's human results support quality-constrained diversity: 55.6%/63.5% non-tie wins against TALR/SFT. Individual-response writing quality is evaluated separately with Critic-7B.
We will include these length controls, length-stratified results, and human evaluation criteria in the revised manuscript.
We sincerely thank the reviewer for their constructive comments and recognition of our theoretical contributions. We address the concerns below.
Q1.
Response: We independently sampled five responses per method on 300 shared held-out prompts, using identical decoding for SFT, TALR, and BREW (4,500 responses). Within each prompt and method, semantic distance averages ten pairwise cosine distances between summary embeddings covering thesis, arguments, perspectives, and content. An independent LLM rates each five-response set's content and perspective diversity (1–5), excluding rewording alone, errors, and off-topic content. Critic-7B scores each response; we average the five scores per prompt. All metrics are then averaged across prompts.
Human evaluation. Four annotators each compared BREW with both baselines on the same 30 prompts: 240 tasks, 120 judgments per comparison/criterion. Each task presents two anonymized sets of five full responses; positions and order are randomized. Quality-constrained diversity considers quality, relevance, and instruction following alongside meaningful variation.
W/T/L is from BREW's perspective; non-tie win rate = W/(W+L).
We will include these results and evaluation protocols in the revision.
Q2.
Response: During training, BREW-Norm divides valid token weights by their global-batch mean, setting it to one while preserving weight ratios. The original floor is applied first; normalized weights remain detached and are not capped at one.
WB-A–F follow the paper's domain order.
Relative to the reported main means, Norm retains 94.2% of BREW's gain over SFT: 3.92/4.16. The table below reports pre-clipping gradients, not AdamW updates.
AdamW uses moving averages of gradients and squared gradients, so multiplying SFT's learning rate by average token weight does not ensure matched parameter-update sizes. In the revised manuscript, we will label that control a reduced-learning-rate SFT ablation and add the normalization results and gradient diagnostics.
Q3.
Response: We fix both methods' training-weight temperature at τ = 1.21 (Appendix B's initialization), with γ = 0 for TALR and 0.75 for BREW. Other training settings are matched on Qwen3.5-9B/DeepWriting.
Differences are computed before rounding.
With identical temperature at every step, BREW gains 1.22 points [95% prompt-paired CI: 0.76, 1.71]. Fixed BREW's 78.23 is numerically close to adaptive BREW's 78.43. Weights still change with token losses and relative progress at fixed temperature. Temperature determines how strongly effective loss reduces weight; adaptive BREW recalculates it as the median of per-sequence mean effective losses.
We will add this ablation and clarify both components' roles in the revision.