acceptodds
Under review as a conference paper at ICLR 2027

MIRROR: From Imitation to Internalization in LLM Personalization

Abstract

The demand for personalized LLMs is shifting from style imitation toward content quality. We investigate whether self-distillation can bridge this gap in existing fine-tuning paradigm. To address this limitation, we introduce MIRROR(Meta-personalization by Internalizing Reference-Revealed On-policy Reflections),a novel self-distillation framework that shifts LLM personalization from imitation toward preference internalization. First, we replace reference-token imitation with reference-revealed on-policy self-distillation, aligning the model’s next-token distributions along its own generation trajectories with those of its reference-conditioned self, thereby internalizing user preferences rather than reproducing reference wording. Second, we introduce MIRROR-F, a focal plug-in that augments on-policy distributional alignment with selective supervision over informative reference tokens, thereby strengthening content generation while preserving user-specific expression. Across three personalized generation benchmarks, two model scales, and complementary reference-based and LLM-based evaluations, MIRROR and MIRROR-F achieve leading overall personalization performance and superior text quality, while exhibiting less catastrophic forgetting than SFT-based baselines on three unseen personalized generation tasks. The gains are consistent across model scales and application scenarios, translating to improved performance in LLM personalization tasks. Code is available at https://anonymous.4open.science/r/MIRROR-F.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, LLM Personalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_365yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The task is conventional LLM personalization, with a shared model and in-context user history. The paper proposes MIRROR, which follows mostly standard on-policy self-distillation, where the users' Y serve as additional privileged content in the teacher's context. A symmetric JSD loss is used for distillation. The paper also proposes MIRROR-F, a variant that additionally adds a teacher-forced NLL term on a small set of reference tokens. The reference tokens are chosen by format rules and filtered to words that also appear in the input. The methods are evaluated primarily on LaMP with Qwen3 1.7B and 4B.

Critical strengths
  1. The formulation is simple and practical: on-policy self-distillation is conceptually a good fit for the personalization task.

  2. The evaluation covers out-of-distribution forgetting, with positive results.

Critical weaknesses
  1. It is unclear where the performance improvements come from. Although MIRROR is simple and principled, it seems to have limited performance gain without the additional token selector and NLL loss in MIRROR-F. But the token selector in MIRROR-F is a rather hard-engineered heuristic with little principled justification behind it. The absence of an ablation of α = 0 or 1 makes this more concerning.

  2. Experiments are not entirely well controlled. No confidence interval is included. And the tuning procedure for baselines is under-specified; several baselines score below Base in Table 1. Only one LLM family is used, at a rather small scale, and the judge is from the same family.

  3. The OOD experiment could be biased. The OOD experiment is evaluated with a metric that rewards the smallest deviation from Base, which systematically favors the self-distillation-based method like MIRROR.

Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Address the issues raised by Reviewer 3 and detail the revisions made to the paper.[Check Revised Paper for details]

We thank the reviewer for the encouraging assessment and the precise concerns. Each point gives the result table and where we revised.

W1 - Where do the gains come from? alpha=0/1 ablation; the heuristic selector. (Revised: single clean ablation, Table 3; selector analysis in Appendix A.5 and the new extensibility discussion in Appendix on focal token selection.) Built up from MIRROR (G-Eval / IC on Abstract):

VariantQwen3-1.7BQwen3-4BQwen2.5-3B
MIRROR4.610 / 0.4724.735 / 0.4954.527 / 0.526
teacher-forced NLL (SFT)3.627 / 0.4673.902 / 0.5023.697 / 0.483
+ selective anchor (MIRROR-F)4.545 / 0.4894.617 / 0.5064.537 / 0.543
+ uniform anchor4.490 / 0.4774.685 / 0.5004.565 / 0.523
alpha=1 (anchor only)1.101 / 0.4091.247 / 0.3941.145 / 0.416

Removing the frozen teacher or the on-policy term (alpha=1) each collapses the model, so both core components are necessary; the selective anchor adds IC at near-unchanged quality and beats uniform anchoring on IC in 9/9 settings. The selector is deliberately white-box and user-editable; we added a discussion of its industrial extensibility (domain-specific definitions of salient content) in the appendix.

W2 - Confidence intervals; baseline tuning; single family; same-family judge. (Revised: Appendix A.4, A.8; added Qwen2.5-3B and gemma-2-9b.) All trained methods share one tuning budget (Appendix A.4). We added a second model family and a second-family judge; conclusions hold (per-setting 95% intervals in Table 14; Spearman 0.86-1.00). A third backbone family beyond Qwen would require retraining all methods on all tasks and is time-prohibitive under our present compute budget; we will add it with more compute.

W3 - The OOD metric may favor self-distillation by rewarding small deviation from Base. (Revised: OOD now a table, Table 2; full per-metric Table 17.) Signed worst-case change vs. Base (smaller of ROUGE-1/METEOR, averaged over 3 OOD tasks; positive = better than Base):

Trained onSFTPerCENextQuillMIRROR-FMIRROR
1.7B Abstract-15.5-20.5-11.9-6.2-6.4
1.7B News-59.7-48.4-1.4-3.3-2.4
1.7B Paper-45.2-46.6-64.8-5.5-4.0
4B News-57.8-11.9-17.9-2.0-0.5
Qwen2.5-3B News-43.7-18.2-2.1+2.7-4.7

(Full 9x3-task grid in Table 2.) MIRROR/MIRROR-F never lose more than 10% on any of the nine settings, whereas every imitation baseline loses 48-65% in at least one - a large-margin gap, not a metric artifact; positive entries show the metric does not penalize improvements over Base.

We are grateful for the constructive review and happy to add any further clarification.

Reviewer_w71iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies personalized generation that preserves both user-specific expression and content quality. MIRROR aligns a student's next-token distributions with reference-conditioned teacher distributions along the student's own generated trajectories. MIRROR-F adds a selective likelihood loss on reference tokens that satisfy an informativeness heuristic and occur in the student context. Experiments cover three generation tasks, two model sizes, reference-based and model-based evaluations, and transfer to held-out review-generation tasks.

Critical strengths
  1. The method addresses a useful personalization problem. The evaluation considers both matching a user's writing and preserving content quality. Testing retention on held-out tasks makes the results more useful than measuring style similarity alone.

  2. The results support further interest in MIRROR-F. It improves ROUGE-1 over MIRROR in all six task–model combinations in Table 1. This consistent pattern is evidence that the added reference-token loss helps, even though the reason for that improvement needs a clearer test.

Critical weaknesses
  1. The teacher and training setup are unclear. Eq. (3) uses the current student as teacher, while Figure 2 and the code describe a frozen teacher. The scripts also allow differences in teacher size, input length, and training exposure. These defaults do not establish what produced the tables, but a different teacher or more context could improve performance, while less training could reduce forgetting. The actual configurations and comparisons under comparable conditions are needed.

  2. The added benefit and stability of selective anchoring need clearer evidence. OPSD v1 already studies reference-conditioned on-policy distillation, and SDFT reports reduced forgetting. MIRROR-F’s consistent gains are promising, but do not show whether selection helps beyond adding reference supervision. A matched comparison with uniform supervision on all reference tokens would clarify this. The main tables also lack uncertainty, and content quality does not improve in every setting: 4B news C&S is 3.125 versus 3.273 for Base. These limits should be reflected in the claims.

Questions for authors
  1. Clarification: Which teacher checkpoint and update rule produced each main result? Please reconcile Eq. (3), Figure 2, and the released implementation, and report the actual context limits, effective batch sizes, and examples or tokens processed for each method. Please also resolve the 150 versus 200/250 training-step descriptions.

  2. Validation of the claimed contribution: Does selective anchoring outperform uniform supervision on all reference tokens under matched conditions? If this comparison has already been conducted, please report the results and their uncertainty.

  3. Validation of the main results: Please provide the per-run results or uncertainty for the main performance and retention differences. If the actual teacher, input, or training conditions differed across methods, do the conclusions hold in a representative controlled comparison?

Resolving the configuration questions and showing a stable benefit from selective anchoring would move my recommendation toward acceptance.

4 comments
AuthorsOpenReview ↗

Address the issues raised by Reviewer 2 and detail the revisions made to the paper.[Check Revised Paper for details]

We thank the reviewer for the detailed, actionable feedback, which substantially improved the paper. Each point below gives the result table and where we revised.

W1 / Q1 - Reconcile Eq. (3), Figure 2, and the code; report actual configurations. (Revised: corrected Eq. (3); new Appendix A.4 with the budget table and the teacher-collapse Table 11.) All results use the Figure 2 teacher: a frozen same-size snapshot, loaded at start and never updated. Every trained method shares one budget:

Shared settingValue
Train examples / task800
Optimization steps250
Effective batch4
History+query budget13,500 tokens
Teacherfrozen snapshot, same size as student

The literal sg[pi] teacher collapses (Table 11), which is why it is not used; step counts are unified to 250 throughout.

W2 / Q2 - Selective vs. uniform anchoring under matched conditions, with uncertainty. (Revised: Table 3; Appendix A.7, Table 15.) At identical alpha and budget, uniform anchoring wins ROUGE-1 while selective anchoring (MIRROR-F) wins information completeness (IC) in every setting:

Abstract GenerationROUGE-1 (M / U / S)IC (M / U / S)
Qwen3-1.7B0.4044 / 0.4072 / 0.40570.472 / 0.477 / 0.489
Qwen3-4B0.4130 / 0.4165 / 0.41420.495 / 0.500 / 0.506
Qwen2.5-3B0.3690 / 0.3798 / 0.37820.526 / 0.523 / 0.543

(M=MIRROR, U=uniform, S=selective.) Selective anchoring has the highest IC in all 9/9 settings; the MIRROR-F vs. MIRROR IC gain is significant (e.g. +0.017 [+0.011, +0.023] on Qwen3-1.7B Abstract).

Q3 - Per-run uncertainty / controlled comparison. (Revised: Appendix A.8, Table 14.) Every method is trained once under one controlled setup; we report 95% paired-bootstrap intervals. G-Eval advantage of MIRROR over the strongest imitation baseline, under both judges:

G-Eval gain vs. best imitationmain judge (Qwen3-30B)second judge (gemma-2-9b)
Qwen3-1.7B Abstract+0.98 [+0.84, +1.13]+1.11 [+0.93, +1.29]
Qwen3-4B Abstract+0.83 [+0.72, +0.95]+0.72 [+0.57, +0.89]
Qwen2.5-3B Abstract+0.83 [+0.68, +0.98]+0.89 [+0.71, +1.05]
Qwen3-4B News+0.88 [+0.63, +1.11]+0.77 [+0.50, +1.03]

Across all nine settings the interval excludes zero in 8/9. Full multi-seed retraining across all methods and backbones exceeds our current compute budget within the discussion window; we will add multi-seed means and standard deviations once more compute is available.

We are happy to provide any further configuration details, and hope the revision moves the reviewer toward acceptance.

Thank you for the detailed response and revision. The clarified training setup and the new matched comparisons address the conditions in my review. However, the new results raise two further concerns.

  1. The Abstract Generation G-Eval scores in Table 13 differ from those in Tables 1/3 for what seem to be the same outputs. For example, MIRROR on Qwen3-1.7B scores 4.54 in one table and 4.610 in the other, while the News, Paper, and IC values match. Since the judge uses temperature 0, what causes this gap, and which numbers are correct? The gap is similar in size to many differences among the MIRROR variants.
  2. In Table 12, the JSD with the frozen teacher does not decrease during training. ROUGE-1 gains over Base are small, and ExPerT style exceeds Base in only two of six settings. I wonder whether the preserved quality partly comes from a small update rather than from the objective. Could you report how far the trained model moves from the base model, for example the KL divergence on held-out inputs?

Given these open questions, I lean toward keeping my score of 2 (weak rejection).

AuthorsOpenReview ↗

Address issues 2.1 raised by Reviewer 2 [Check Revised Paper for details]

We thank the reviewer for the careful reading and for both questions, which helped us tighten the evaluation protocol and characterize precisely what MIRROR changes in the model.

R2.1 — Abstract G-Eval of MIRROR in Table 13 (4.54) vs. Tables 1/3 (4.610)

The two values were computed on two different fixed-seed evaluation subsets of the Abstract test set: Table 1 draws its 100 examples from those with a non-empty user reference, since it also scores the references themselves, whereas the RQF run drew from the full test set. We have now evaluated every Abstract model in Tables 3, 13, 17, 20 and the α sweep on the same subset as Table 1, under both judges. Table 1 is unchanged, and MIRROR now reads 4.610 / 4.735 / 4.527 (Qwen3-1.7B / Qwen3-4B / Qwen2.5-3B) in every table. All conclusions are preserved:

Check (Abstract, main judge)On the shared subset
MIRROR in the RQF table (Qwen3-1.7B)4.610, identical to Table 1
MIRROR − RQF+0.59 to +0.85; every 95% CI excludes zero under both judges
Quality cost of the focality coefficient α ∈ [0.1, 0.7]within 0.18 G-Eval of MIRROR
Teacher-context ablation, ΔG-Eval vs. MIRRORwithin 0.06, all CIs contain zero

Section 4.1 now states that all tables share this evaluation subset.

AuthorsOpenReview ↗

Address issues 2.2 raised by Reviewer 2 [Check Revised Paper for details]

R2.2 — Is the preserved quality only a consequence of a small update?

No. We measured the distance directly (new Appendix A.16, Table 21): for 200 held-out inputs per setting and the exact generation prompt, the full-vocabulary sequence-level KL(π_θ ‖ π_0) along each trained model's own output, and the relative parameter displacement ‖θ − θ₀‖ / ‖θ₀‖ (bootstrap 95% intervals within ±13% of every entry). The analysis shows that MIRROR's gains are determined by the direction of the update — the user's own reference — and not by its size.

MethodKL, Qwen3-1.7B Abs / News / PaperKL, Qwen3-4B Abs / News / Paper‖Δθ‖ (×10⁻³)G-Eval Abs 1.7B / 4B
Base0 / 0 / 00 / 0 / 004.515 / 4.705
SFT305.1 / 62.4 / 41.8333.3 / 65.3 / 57.63.0–4.23.627 / 3.902
PerCE253.8 / 59.7 / 45.4294.4 / 61.0 / 60.92.9–4.23.415 / 3.595
NextQuill282.8 / 58.9 / 41.1311.5 / 62.1 / 55.33.2–4.53.530 / 3.638
OPSD117.1 / 10.8 / 6.354.4 / 13.3 / 5.82.2–3.44.543 / 4.755
SDFT244.2 / 34.9 / 33.5378.8 / 47.4 / 42.82.4–3.54.365 / 4.463
MIRROR23.0 / 7.1 / 4.921.1 / 11.2 / 8.92.1–3.14.610 / 4.735
MIRROR-F34.6 / 12.0 / 13.634.3 / 16.3 / 13.62.5–3.74.545 / 4.617
MIRROR, reference-free teacher (h, x)0.36 (Abs)0.36 (Abs)2.0 / 2.94.613 / 4.680
MIRROR, another user's reference (h, x, y′)11.8 (Abs)6.8 (Abs)2.1 / 2.94.550 / 4.683
  1. MIRROR stays closest to the base model in output space, by design. Imitation-based methods are 5.4–15.8× farther in KL (MIRROR is closest in five of six settings), and MIRROR agrees with the base model's top-1 token at 86–93% of positions, against 59–86% for SFT, PerCE and NextQuill. Because the teacher is π₀ itself conditioned on the reference, the target departs from π₀ only where the user's reference changes the prediction: MIRROR changes what the user's reference calls for, and leaves the rest of the model intact.
  2. The small output shift is not a short or weak run. All methods share the same 250 steps, data and schedule (Table 11), and their parameter displacements fall in the same range (2.0–4.5 × 10⁻³). The reference-free control moves its parameters as far as MIRROR (2.0 vs. 2.2 × 10⁻³ on Qwen3-1.7B Abstract), yet its outputs stay at the base model (KL 0.36). Parameter distance does not measure behavioral change; the output KL does, and MIRROR's output change is targeted.
  3. Direction, not size, determines the gain. A teacher given another user's reference moves the model by the same order of distance as MIRROR, yet yields no ROUGE-1 gain over the base model and at most a third of MIRROR's IC gain (Table 20; ROUGE-1 vs. MIRROR −0.0083, 95% CI [−0.0097, −0.0070]). Conversely, distance alone does not cost quality: on Abstract, OPSD moves 2.6–5.1× farther than MIRROR with comparable G-Eval, whereas the imitation methods, 11–16× farther, lose 0.8–1.1 points relative to the base model. The personalization gain is caused by the user's own reference, not by a small update.
  4. Distance-matched comparison. On Qwen3-1.7B Abstract, the coefficient α of MIRROR-F moves the model monotonically (KL 23.7 → 27.9 → 34.6 → 46.0 → 110.7 for α = 0 → 0.3 → 0.5 → 0.7 → 0.9) while IC rises from 0.472 to 0.525, a controllable coverage–distance trade-off. At equal distance, the direction decides quality: MIRROR-F at α = 0.9 and OPSD are equally far (110.7 vs. 117.1), but the reference-token anchor costs 0.40 more G-Eval; the anchor alone (α = 1, KL 223.4) and SDFT (244.2) are again equally far, with G-Eval 1.12 vs. 4.37.

On the frozen-teacher JSD in Table 12. This JSD compares a student that sees (h, x) with a teacher that additionally sees the user's reference y, so it has a floor: the part of the reference's effect that cannot be predicted from the history and the query, which no deployable student can remove. It is also measured on rollouts that change with the student. A flat curve is therefore expected and is not a progress measure; progress is visible in the reference-specific ROUGE-1/IC gains (Table 20) and in the distance analysis above. This explanation was added to Appendix A.4.

On the size of the personalization gains. MIRROR improves ROUGE-1 over the base model in all six settings (+0.004 to +0.021; up to +0.036 for MIRROR-F) while keeping quality and style at the base level. Imitation-based methods, by contrast, lower style below Base everywhere and lose 0.8–1.1 G-Eval points on Abstract; as item 3 shows, MIRROR’s gain comes from the user’s own reference.

Thank you very much for the guidance you provided regarding my paper. Our discussions have helped me better position my work and clarify its contributions. We hope our response goes some way toward addressing your concerns; if possible, we would appreciate it if you could reconsider your evaluation score.

Reviewer_wVVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MIRROR, an on-policy self-distillation framework for personalized text generation. It uses a teacher model conditioned on a user-provided reference response to align the student model’s output distribution along its own generation trajectories, allowing the student to learn user preferences rather than directly imitate the reference text. The authors further propose MIRROR-F, which selectively supervises informative tokens in the reference text to improve content quality. Experiments across three personalized generation tasks and two model sizes show improvements in personalization, text quality, and cross-task generalization.

Critical strengths
  1. The problem is practically relevant. The paper focus on an important challenge in personalized generation: how to learn users’ stylistic preferences without compromising the quality of generated content. This goal is closer to what real applications need than merely maximizing textual or stylistic similarity to reference responses.

  2. The method and experiments are reasonably well documented. The paper provides detailed training objectives and implementation procedures for both MIRROR and MIRROR-F, along with ablation studies. The appendix also clarifies the scope and limitations of the token selection rules, making it easier to understand the method’s practical implications.

Critical weaknesses
  1. The novelty of the method is not sufficiently established. MIRROR builds on the idea of constructing a teacher distribution from reference information and performing self-distillation along the student’s own generation trajectories. The proposed approach seems close to prior work on on-policy self-distillation, such as Zhao et al. (2026). The authors need to clarify what is genuinely different about MIRROR with those prior works. Using an existing paradigm for personalized generation does not make the method novel.

  2. The experiments do not directly test whether the proposed method is necessary. For instance, would filtering for high-quality reference samples, or supervising only the reference tokens relevant to content quality, can get similar improvement?

  3. The evidence for improved content quality remains insufficient. The paper claims that MIRROR improves the completeness, and professionalism of generated content. But this evaluation just relies on text similarity metrics and LLM-as-a-judge assessments. Overall text quality scores do not directly reflect improvements in factual accuracy, completeness, or domain expertise. The current experimental results therefore provide insufficient support for the paper’s central claims about the reliability and professional quality of generated content.

Questions for authors
  1. How does MIRROR differ fundamentally from prior on-policy self-distillation methods? What specific technical innovation does MIRROR introduce beyond applying reference-conditioned self-distillation to personalized generation?

  2. Can the authors demonstrate the necessity of the proposed training objectives through comparisons with simpler alternatives? For example, how would MIRROR-F compare with selective token-level supervision applied directly to the reference responses, without on-policy distributional alignment?

  3. How do the authors verify the claimed improvements in content quality and reliability? Since the current evaluation depends heavily on automatic metrics and a single LLM judge, could they provide more targeted assessments of factual accuracy, completeness, and domain expertise—or otherwise establish that the gains are genuine improvements in content quality, not merely better alignment with the reference or the judge model’s preferences?

3 comments
AuthorsOpenReview ↗

Address the issues raised by Reviewer 1 and detail the revisions made to the paper.[Check Revised Paper for details]

We sincerely thank the reviewer for the thorough and constructive review. Below, each point states the numerical evidence, the result table, and where we revised the paper.

W1 / Q1 - Novelty vs. prior on-policy self-distillation (OPSD, SDFT). (Revised: Introduction, contribution list, Related Work; Appendix A.4, Table 11.) We do not apply an existing paradigm unchanged: we carry on-policy self-distillation into a supervision regime that differs in kind, and redesign the learning objective for that regime. OPSD/SDFT rest on the assumption that the privileged signal is correct, is better than the student, and that its content should be absorbed. In personalization that assumption fails: the user reference is often of lower quality than the base model's own output (News: 3.08 vs. 4.40 in G-Eval), so it is not a target to be reproduced but hindsight information that reveals the user's preference. Three design choices follow. (i) The objective matches the behavioral shift the reference induces in a frozen copy of the model through a symmetric divergence, instead of maximizing the likelihood of the signal - this is what keeps the reference's defects out of the student, which imitation inherits. (ii) The teacher must be a frozen snapshot; a shared-parameter teacher collapses in this regime. (iii) Direct token-level supervision is restricted to grounded, information-bearing positions (MIRROR-F). Evidence for (ii):

Teacher on Abstract (Qwen3-1.7B)JSD@step40ROUGE-1BERTScore
Frozen snapshot (MIRROR)0.0690.40440.869
Shared-parameter sg[pi]~00.00030.694

W2 / Q2 - Would filtering high-quality references give the same gain? (Revised: new Appendix A.5, Table 12.) We score the training references with the judge, keep the top 50%, and retrain contextSFT (RQF) under the identical budget. Filtering helps but does not close the gap to MIRROR (G-Eval; main judge Qwen3-30B / second judge gemma-2-9b):

Abstract GenerationcontextSFTcontextSFT-RQFMIRROR
Qwen3-1.7B3.61 / 3.863.71 / 4.034.54 / 4.92
Qwen3-4B3.96 / 4.284.08 / 4.524.69 / 4.99
Qwen2.5-3B3.64 / 4.093.94 / 4.504.51 / 4.97

RQF improves contextSFT by at most +0.30 but still trails MIRROR by +0.57 to +0.83 (paired-bootstrap 95% intervals exclude zero). Token-level supervision without on-policy alignment is the alpha=1 ablation, which collapses G-Eval to ~1.1 (Table 3).

W3 / Q3 - Is the quality gain real, or just reference/judge alignment? (Revised: Figure 3; Appendix A.6, A.8.) Faithfulness is not reference-alignment. On News Headline Generation, G-Eval Consistency (main judge / second judge):

News ConsistencyReferenceBaseImitation (SFT/PerCE/NextQuill)MIRROR family
Qwen3-30B2.134.161.97-2.423.48-3.66
gemma-2-9b2.364.422.23-2.693.84-3.92

Imitation drops to the reference's own low faithfulness, whereas MIRROR preserves it; MIRROR's aggregate G-Eval is above the references on all three tasks. The second-family judge reproduces every main ranking (Spearman 0.86-1.00) and MIRROR's advantage in 8/9 settings (Appendix A.8, Table 14).

We hope these additions address your concerns and are glad to run any further analysis.

The authors have addressed a good part of my concerns, but my major concern regarding the methodological novelty still unresolved. The additional RQF baseline and α=1 ablation are useful experiments. They help clarify that the improvements cannot be simply attributed to reference filtering or direct token-level supervision, and provide stronger evidence for the proposed training objective.

However, it is still unclear what fundamentally distinguishes MIRROR from existing on-policy self-distillation methods. While the authors provide a reasonable motivation for adapting this paradigm to personalization, the methodological contribution beyond this adaptation and the combination of existing techniques remains insufficiently clear.

Overall, I appreciate the additional experiments, which have made the empirical results more convincing. Nevertheless, given that my main concern about methodological novelty has not been fully addressed, I would maintain my original rating.

AuthorsOpenReview ↗

Address the issue raised by Reviewer 1 and detail the revisions made to the paper.[Check Revised Paper for details]

We thank the reviewer for the opportunity to state the contribution more precisely.

What is new is the regime and what it requires. In personalization, the privileged signal is a user-written reference that is noisy and often worse than the model's own output (on News Headline Generation, references score 3.08 G-Eval against 4.40 for Qwen3-4B Base). It reveals a preference without being a correct target. OPSD and SDFT assume a correct privileged signal (verified reasoning traces, expert demonstrations). Whether self-distillation can extract a user's preference from such a signal without copying its defects is not addressed by prior work; MIRROR shows that it can, and identifies what this requires. Our contributions are:

  1. Framework. We cast personalization as reference-revealed preference internalization and show that on-policy self-distillation transfers to this regime: a noisy reference serves as privileged information without its defects being copied — on Abstract, MIRROR preserves base-level quality where every imitation-based method loses 0.8–1.1 G-Eval points.
  2. Evidence that the reference is used as user-specific hindsight, not as a target. Removing the reference, or replacing it with another user's reference, removes the personalization gain at unchanged quality (Table 20); another user's reference moves the model by the same order of distance without any ROUGE-1 gain (Table 21).
  3. Teacher choice is decisive in this regime. A teacher that shares the student's parameters collapses (Table 12: JSD 0.0605 → 0.0007 within 40 steps, ROUGE-1 0.0003), and the EMA teacher of SDFT moves the model more than 10× farther than MIRROR on Abstract and scores 0.25–0.27 G-Eval below it (Table 21).
  4. MIRROR-F, a selective, grounded anchor on information-bearing reference tokens, exposes a controllable coverage–quality frontier (Table 3, Table 21); a uniform anchor gives at most half of its IC gain.

MIRROR personalizes more than OPSD at comparable quality. Under the same reference-as-privileged-signal setup and budget, MIRROR has higher ROUGE-1 than OPSD in 5 of 6 settings and MIRROR-F in all six (Qwen3-1.7B Abs / News / Paper: 0.4044 / 0.1699 / 0.4269 vs. 0.3574 / 0.1612 / 0.4280; Qwen3-4B: 0.4130 / 0.1835 / 0.4547 vs. 0.3933 / 0.1764 / 0.4504). On Abstract, OPSD falls below the base model in ROUGE-1 (0.357 vs. 0.396 on 1.7B), while MIRROR stays above it (Appendix A.16).

Revisions. Related Work now positions MIRROR explicitly: it builds on the on-policy recipe of OPSD, claims no new objective beyond the symmetric bounded divergence, and contributes the transfer of this recipe to a noisy, preference-revealing signal together with the MIRROR-F anchor. Appendix A.16 reports the ROUGE-1 comparison above and documents the reimplementation of OPSD and SDFT, both run at MIRROR's settings (same data, steps, batch, rollout length, context budget, learning rate and LoRA rank).

Thank you very much for the guidance you provided regarding my paper. Our discussions have helped me better position my work and clarify its contributions. We hope our response goes some way toward addressing your concerns; if possible, We would appreciate it if you could reconsider your evaluation score(Final Rating).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.