acceptodds
Under review as a conference paper at ICLR 2027

Rubric-Aware On-Policy Self-Distillation for LLM Personalization

Abstract

LLM personalization aims to generate responses aligned with individual users' preferences and needs. User-specific rubrics make these expectations explicit, providing direct supervision on what a satisfactory answer should cover. Existing rubric-guided approaches, however, exploit such guidance only at a coarse granularity, either by using rubrics to supervise the prediction of relevant aspects for subsequent generation or by reducing aspect coverage to a single response-level reward for reinforcement learning. This leaves a gap between specifying what a personalized answer should contain and teaching the model how to generate it. To bridge this gap, we propose GRASP, a rubric-aware on-policy self-distillation framework for LLM personalization that turns user-specific rubric aspects into fine-grained, token-level supervision. Specifically, GRASP pairs a rubric-free student with a rubric-informed teacher that additionally receives the target user-specific rubrics. By aligning their next-token distributions along on-policy trajectories generated by the student, GRASP transfers the teacher's rubric-conditioned guidance into the student, translating user-specific semantic requirements into dense token-level supervision. Since rubric-informed teachers can still produce inadequate supervision, we further introduce Rubric-based Teacher Validation (RTV), which retains only instances where the teacher sufficiently covers the target aspects, improving both supervision quality and training efficiency. Experiments on the LaMP-QA benchmark for personalized question answering demonstrate that GRASP achieves state-of-the-art performance across multiple backbones, supporting the effectiveness of rubric-guided token-level supervision for personalization. To ensure reproducibility, our code is available at https://anonymous.4open.science/r/GRASP-10710.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM, Personalization, OPSD

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gWdnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRASP, a rubric-aware on-policy self-distillation framework for personalized question answering. The student receives a question and retrieved user history. A frozen teacher additionally receives user-specific rubric aspects and provides token-level supervision along student-generated responses. The method also introduces Rubric-based Teacher Validation (RTV), which filters training instances based on the performance of teacher-generated answers. Experiments on LaMP-QA across three domains and four backbones show higher average rubric-coverage scores than the selected baselines. Ablations examine RTV thresholds, teacher size, and rubric alignment. The appendix provides further details on the benchmark, training procedures, and related aspects.

Critical strengths
  1. GRASP offers a straightforward way to internalize rubric-based supervision into a model without requiring rubrics at inference time. It achieves state-of-the-art average rubric-coverage scores on LaMP-QA across four backbones. The paper also includes substantial ablations of its main components, strengthening the empirical support for the framework.

  2. The shuffled-rubric experiment and the rubric-completeness analysis in Appendix D are particularly insightful. They test whether the gains depend on correctly aligned and complete rubric information, rather than simply additional teacher context. Together, they provide strong evidence that GRASP learns from the semantic content of the rubrics.

Critical weaknesses

From a methodological perspective, it remains unclear whether the gains primarily come from on-policy trajectories, soft-label distillation, or RTV-based instance selection. Additional controlled experiments, such as matching the number of training instances across settings, would help clarify the contribution of each component.

Questions for authors
  1. RTV identifies instances where the teacher generates a good answer, but this does not necessarily mean that the teacher can provide reliable guidance along student-generated trajectories. Although the paper includes detailed RTV ablations, reporting the number of training instances retained at each threshold, together with comparisons of different thresholds that control training data size, would clarify RTV’s contribution.
  2. I hope to know more about the relationship between the benchmark’s training and test sets, especially the user overlap. This is important for assessing whether GRASP learns a transferable ability to infer users’ needs from their history and apply this ability to new topics.
  3. RTV retains instances with high-scoring teacher responses. These responses could also provide useful targets for SFT or off-policy distillation. Comparing these alternatives would help isolate the contributions of on-policy trajectories and soft-label supervision.
  4. Under capacity or information constraints, forward KL may favor overly averaged predictions. An ablation of forward KL, reverse KL, and potentially JS divergence would help justify the chosen objective.
  5. Does GRASP affect the model’s ability to answer general questions? Evaluating general capabilities before and after training would clarify whether personalization gains come at the cost of the model’s general performance.
  6. The method uses the top-10 retrieved history entries as context, but the paper does not establish whether the target rubric aspects can be inferred from this context. This distinction matters for interpreting the contribution: does teacher supervision teach the student to use user history, or mainly encourage more comprehensive answers that achieve higher rubric coverage?
  7. Similar rubric-conditioned self-distillation methods have been explored in other domains, including RuPI [1], RGSD [2], and RCSD [3]. In personalization, Aligning Language Models from User Interactions [4] also uses additional user context to construct token-level self-distillation signals. Please clarify which components GRASP shares with these methods and which adaptations or innovations specifically address personalized question answering.

If it is difficult to address all the questions within the limited rebuttal period, please prioritize Questions 1, 2, 3, and 5, as these are my main concerns.

[1] Rubrics as Privileged Information for Open-Ended Generation.

[2] Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers.

[3] Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation.

[4] Aligning Language Models from User Interactions.

5 comments

Dear Reviewer,

Thank you for the detailed and constructive feedback! We value the opportunity to address the weaknesses and questions you kindly raised and to further enhance the quality of our work.


R1: Additional RTV ablations (Q1)

Thank you for this suggestion. The retained sample counts in our threshold ablation are:

Backboneτ=0.2\tau=0.2τ=0.5\tau=0.5τ=0.8\tau=0.8τ=1.0\tau=1.0
Gemma2-9B22,72622,55013,6539,050
Qwen2.5-7B23,94223,90319,41315,011

We additionally matched the training dataset used in Gemma2-9B experiment with random selection:

SelectionA&EL&PDS&CAvg.
Random0.42660.52780.57040.5083
RTV0.43720.56060.57870.5255

This supports RTV's benefit beyond reducing data size.

R2: User overlap and generalization (Q2)

Thank you for raising this point. Our analysis identifies 1,383 test instances from seen users and 1,447 from unseen users. Performance on each partition is reported below:

BackboneSeenUnseen
Gemma2-9B0.52110.5297
Qwen2.5-7B0.53060.5308
Qwen2.5-14B0.55840.5640
Qwen3-4B0.61850.6141

These comparable results suggest that GRASP learns a transferable ability to generate personalized answers rather than relying solely on familiarity with training users.

R3: Training alternatives on RTV-selected data (Q3)

Thank you for this helpful suggestion. We compared two alternatives on the same RTV-selected instances: SFT learns from the retained teacher answers, while off-policy KD learns from teacher distributions along those answers.

BackboneSFTOff-policy KDGRASP
Gemma2-9B0.47520.46910.5255
Qwen2.5-7B0.47290.47440.5357
Qwen2.5-14B0.50130.49750.5657
Qwen3-4B0.55230.54430.6180

Both alternatives perform well on RTV-selected data, but remain substantially behind GRASP, supporting the benefit of on-policy distillation.

R4: Distillation objective (Q4)

Thank you for this suggestion. We evaluated alternative objectives on Gemma2-9B:

ObjectiveA&EL&PDS&CAvg.
Forward KL0.43720.56060.57870.5255
Reverse KL0.43150.54950.57380.5183
JSD0.43850.55730.57940.5250

All three objectives clearly outperform the strongest baseline (IAP, 0.5078), so GRASP is robust to the choice of divergence. Forward KL and JSD perform comparably, while reverse KL is slightly lower. We attribute this to mode-seeking: a rubric-informed teacher spreads probability over multiple valid ways to address the target aspects. Forward KL encourages the student to cover them, rather than producing overly averaged predictions, whereas reverse KL may collapse onto a subset. We therefore keep forward KL as the default.

R5: Impact on general capabilities (Q5)

Thank you for raising this concern. We evaluated all four backbones on MMLU-Pro, a benchmark of broad knowledge and reasoning, before and after GRASP training:

BackboneBeforeAfter
Gemma2-9B0.51540.5063
Qwen2.5-7B0.57320.5761
Qwen2.5-14B0.64320.6403
Qwen3-4B0.61200.6261

These results show no substantial degradation on MMLU-Pro, suggesting that GRASP strengthens personalization while largely preserving general capabilities.

R6: Contribution of user history for personalization (Q6)

Thank you for this insightful question. The top-10 history entries provide personalization context rather than directly specifying rubric aspects. Replacing them with other users' profiles across five random mappings reduces Gemma2-9B's score from 0.5255 to 0.4789 ±\pm 0.0016. This suggests that GRASP benefits from user-specific history, beyond simply generating more comprehensive answers.

R7: Relationship to prior methods (Q7)

Thank you for highlighting these relevant works. GRASP shares rubric-conditioned token-level self-distillation with RuPI, RGSD, and RCSD. Our contribution is its adaptation to history-conditioned personalized QA through user-specific rubric supervision and RTV-based teacher validation. Aligning Language Models from User Interactions derives supervision from subsequent user messages, limiting learning from interactions without follow-ups. GRASP instead learns from user-specific rubrics during training and personalizes from existing history at inference, supporting a broader range of personalization scenarios. We include more detailed discussion and comparison in the revision.

Thank you for the helpful rebuttal. They adress most of my questions. But I still have two brief questions about the reported numbers:

  1. For Qwen2.5-7B, the seen and unseen scores are 0.5306 and 0.5308 in R2, while Table 1 reports an overall score of 0.5357. If you could explain the difference, it will be better.
  2. The authors refer that IAP is the strongest baseline with a score of 0.5078 in R4, whereas Table 1 reports 0.3975. Could you clarify whether this is an updated result or a reporting error?
AuthorsOpenReview ↗

Clarification of Unclear Reporting

Dear Reviewer,

Thank you for your kind reply and follow-up questions! We apologize for the unclear reporting, and we provide our clarifications below:

  • Seen/unseen scores: We apologize for inadvertently evaluating wrong data in R2, which caused the discrepancy with the main results in Table 1. We have re-evaluated the correct model across three random seeds. The corrected results are provided below. All scores are macro-averaged across the three categories (as in Table 1). In the revised manuscript, we have updated Table 1 with the three-seed results and added the seen/unseen results to Table 17.
BackboneSeenUnseen
Gemma2-9B0.52180.5277
Qwen2.5-7B0.53680.5332
Qwen2.5-14B0.55810.5649
Qwen3-4B0.61760.6147
  • IAP results: The 0.5078 reported in R4 is an updated result. Specifically, we re-implemented IAP on the Gemma2-9B backbone using the swift framework, which achieved stronger performance than our earlier adaptation of the original repository. Details and updated results are provided in the revision.

We have updated our revised manuscript to reflect and clarify these results. Thank you again for pointing out these issues and helping us improve the quality of our paper. We welcome further discussion and appreciate your continued feedback.

Reviewer_zNSrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GRASP, a method for personalized question answering (PQA). The student model sees the question and the retrieved user profile. A frozen copy of the same model, the teacher, also sees privileged information: the user's own elaboration of the question and the user-specific rubric aspects. The student is trained with on-policy self-distillation (OPSD): it samples its own responses and minimizes the full-vocabulary forward KL between the teacher's and its own next-token distributions along those responses. Rubrics are therefore needed only at training time.

A second component, Rubric-based Teacher Validation (RTV), filters the training set offline. The teacher generates one response per instance, an LLM judge scores how well it covers the rubric, and only instances with full coverage (τ = 1) are kept.

The claimed contributions are:

  1. Identifying that existing rubric-guided methods (PlanPers, IAP) use rubrics only as coarse supervision.
  2. The first application of OPSD to personalization.
  3. The GRASP method together with RTV.

Validation is done on LaMP-QA (three categories) with four backbones (Gemma2-9B, Qwen2.5-7B/14B, Qwen3-4B), scored by a Qwen2.5-32B-Instruct rubric judge. GRASP has the best macro average on all four backbones, with relative gains over the strongest baseline of +13.9%, +10.7%, +7.6% and +0.7%. The analyses cover:

  • an RTV threshold sweep;
  • a larger-teacher comparison;
  • shuffled-rubric and single-aspect ablations;
  • a matched-prefix token-shift analysis showing that the student's distributions move toward the rubric-conditioned teacher.
Critical strengths

S1. A simple, deployable way to turn rubrics into training signal for personalization. GRASP needs no gold responses (Sec. 3.1), no external stronger teacher (Fig. 3 shows a self-teacher matches a larger one), and no extra inference cost: test-time input is identical to RAG (App. D.2). Training is cheap: one epoch, 217–470 steps, 6.7–11.9 h (Table 5). Rubric-style criteria are common in personalized systems such as e-commerce QA and personal assistants, where at inference the system has only the user's history. A recipe that compiles such criteria into the weights could therefore be broadly useful. This is the main reason I see value in the paper.

S2. Consistent gains, with ablations that link the gains to the privileged information. GRASP is best on average on all four backbones (Table 1). Two ablations show the gain comes from instance-aligned rubric content rather than from simply adding context to the teacher:

  • With shuffled rubrics, performance falls to roughly RAG level (Table 2).
  • With a single aspect instead of the full set, performance drops consistently (Table 6).

The matched-prefix analysis (Fig. 4) provides mechanistic evidence that the student internalizes the teacher's rubric-induced shifts. This makes the empirical claim credible, subject to the caveats below.

Critical weaknesses

W1. Limited technical novelty, and the claimed source of the gains is not isolated by the right comparisons. The GRASP objective (Eq. 13) is the generic OPSD objective (Eq. 4; Zhao et al., 2026), with the privileged context set to rubric aspects and the training set filtered. The forward-KL choice is standard and is not ablated. RTV is an offline, instance-level rejection filter, and it adds only about +0.002–0.013 over τ = 0 (Fig. 2). OPSD without RTV already beats all baselines on three of the four backbones. Fang et al. (2026), "Rubric-based On-Policy Distillation", is closely related; the paper cites it but does not discuss or compare against it.

Novelty therefore rests on the empirical claim that on-policy, token-level supervision is what makes rubrics effective (Table 4 ranks granularity #1–#4). The experiments do not isolate this claim:

  • There is no off-policy context-distillation baseline: SFT on the rubric-informed teacher's RTV-filtered outputs.
  • There is no plain rubric-reward RL baseline (GRPO/DAPO without IAP's intent component).
  • The rubric-informed teacher's own test score, the natural upper bound, is not reported.
  • The existing baselines are not matched on training. PlanPers trains only a 7B LoRA planner and keeps the backbone frozen, while GRASP fully fine-tunes it. The IAP reimplementation scores below RAG on Gemma2-9B (0.3975 vs. 0.4051) and on Qwen3-4B (0.5003 vs. 0.5180), which suggests under-tuning.
  • No seeds or variance are reported, so the Qwen3-4B margin (0.6180 vs. 0.6137) and the RTV gains cannot be separated from noise.

Without these comparisons, the gains could come from rubric-informed data in general rather than from GRASP's on-policy, token-level design.

W2. The method depends on oracle rubrics that the student often cannot infer. This leaves the loop open and risks teaching the student to assert things it cannot know. The paper assumes rubrics are provided with each training instance (Sec. 3.1). It does not discuss how they would be obtained when unavailable, for example inferred from past interactions or feedback, and it has no limitations section. Its own results show how sensitive GRASP is to rubric quality:

  • Misaligned rubrics reduce GRASP to RAG level (Table 2).
  • Partial rubrics hurt performance (Table 6).
  • Even with gold rubrics, the teacher fully covers them on only 37–81% of training instances (Fig. 5).

Moreover, the teacher's privileged context (prompt P2) includes the user's elaboration of the current question. Much of that information cannot be recovered from past questions. Minimizing forward KL toward a teacher whose behavior depends on such hidden information trains the student to reproduce confident, specific behavior that it cannot justify from its own inputs. Kim et al. (2026), "Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?" (arXiv:2603.24472), document exactly this failure mode. A teacher conditioned on rich context suppresses expressions of uncertainty; the student, lacking that context, inherits the condensed and overconfident behavior; and performance degrades out of distribution, by up to 40%.

The present paper shows signs of the same phenomenon:

  • In Fig. 4c, after training, the rubric-free student puts 0.87 probability on "Given your cat". The paper does not say whether that attribute appears in the retrieved profile.
  • In the case study (Fig. 7), the baseline's request for clarification is replaced by a confident answer containing a questionable factual claim.

Because evaluation measures only aspect coverage, without factuality, hallucinated user attributes or calibration, this risk is invisible in the reported numbers. Together with the single benchmark, this limits confidence that the gains carry over to realistic settings where rubrics must be inferred.

Questions for authors

Clarifications (answers could change my assessment):

  1. What does the rubric-informed teacher score on the test set, i.e., the upper bound?
  2. How does GRASP compare with (a) SFT on the teacher's RTV-filtered rubric-conditioned outputs (off-policy context distillation) and (b) GRPO/DAPO with the plain rubric reward, both trained under the same budget? This is the key test of the token-level, on-policy claim.
  3. Can you report mean ± std over ≥3 seeds for Table 1 and Fig. 2?
  4. How was IAP tuned, and why does it fall below RAG on two backbones? Can you reproduce the numbers reported in the original IAP paper?
  5. In Fig. 4c, does "cat" appear in the retrieved profile? More generally, how often do GRASP outputs assert user attributes that are not supported by the profile, compared with RAG?
  6. RTV uses the same judge model and scoring procedure as the test evaluation (App. C), and checkpoints are selected on validation score. Do the results hold with a different, non-Qwen judge and with a small human evaluation? Please also report response lengths and length-controlled scores.
  7. In the shuffled-rubric ablation, is the user elaboration ({details}) shuffled as well, or only the aspects?

Suggestions (would strengthen the paper; not required for my assessment):

  1. Close the loop by using inferred rubrics as the teacher's privileged context, for example the PlanPers planner's predicted aspects or a rubric-learning method such as Qiu et al. (2026), which the paper cites. Also test robustness to noisy rubrics.

  2. Evaluate on additional personalization benchmarks with implicit, evolving, or multi-turn preferences, for example LongLaMP, PrefEval or PersonaMem.

  3. Discuss Fang et al. (2026) and Kim et al. (2026), and add a limitations section.

References:

  • Jiang, B., Hao, Z., Cho, Y.-M., Li, B., Yuan, Y., Chen, S., Ungar, L., Taylor, C. J., & Roth, D. (2025). Know me, respond to me: Benchmarking LLMs for dynamic user profiling and personalized responses at scale. arXiv:2504.14225. [PersonaMem]

  • Kim, J., Luo, X., Kim, M., Lee, S., Kim, D., Jeon, J., Li, D., & Yang, Y. (2026). Why does self-distillation (sometimes) degrade the reasoning capability of LLMs? arXiv:2603.24472.

  • Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., et al. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300. [GRPO]

  • Zhao, S., Hong, M., Liu, Y., Hazarika, D., & Lin, K. (2025). Do LLMs recognize your preferences? Evaluating personalized preference following in LLMs. In ICLR 2025. [PrefEval]

3 comments
AuthorsOpenReview ↗

Response to Reviewer zNSr (1/2)

Dear Reviewer,

We sincerely thank you for your encouraging comments on our work. We value the opportunity to address the weaknesses and questions you kindly raised and to further enhance the quality of our work.


R1: Rubric-informed teacher performance (Q1, W1)

Thank you for this suggestion. We evaluated the rubric-informed teacher on all test instances, alongside the RAG and GRASP results. Since test-time rubrics are unavailable during deployment, the teacher serves as an oracle reference. We also report the fraction of the RAG-to-oracle gap closed by GRASP, i.e., (GRASP - RAG) / (Teacher - RAG):

BackboneRAGGRASPTeacher (with rubrics)Gap closed by GRASP
Gemma2-9B0.40510.52550.837527.8%
Qwen2.5-7B0.42770.53570.920021.9%
Qwen2.5-14B0.45000.56570.909825.2%
Qwen3-4B0.51800.61800.967122.3%

GRASP closes 21.9-27.8% of the RAG-to-teacher gap without accessing rubrics at inference time.

In addition, when given the same rubrics at test time, GRASP performs on par with the teacher (0.8357 / 0.9360 / 0.9308 / 0.9703 vs. 0.8375 / 0.9200 / 0.9098 / 0.9671), indicating that training preserves the model's ability to follow explicit rubric guidance.

R2: Controlled training comparisons (Q2, W1)

Thank you for this insightful question.

(a) Off-policy training. On the same RTV-selected instances, SFT (RTV) learns from the retained teacher answers, while off-policy KD learns from teacher distributions along those answers.

BackboneSFT (RTV)Off-policy KDGRASP
Gemma2-9B0.47520.46910.5255
Qwen2.5-7B0.47290.47440.5357
Qwen2.5-14B0.50130.49750.5657
Qwen3-4B0.55230.54430.6180

(b) GRPO. Gemma2-9B trained with GRPO and the plain rubric reward achieves a test score of 0.4684.

Together, these results support the effectiveness of GRASP's token-level, on-policy supervision.

R3: Multi-Seed Evaluation (Q3, W1)

Thank you for this suggestion. The results reported in Table 1 and Figure 2 use seed 0. We summarize results across 3 seeds below:

MethodGemma2-9BQwen2.5-7BQwen2.5-14BQwen3-4B
Non-Perso0.3483 ±\pm 0.00120.4212 ±\pm 0.00340.4420 ±\pm 0.00090.5011 ±\pm 0.0009
RAG0.4021 ±\pm 0.00370.4206 ±\pm 0.00630.4532 ±\pm 0.00280.5158 ±\pm 0.0025
SFT0.4046 ±\pm 0.01340.4312 ±\pm 0.01100.4600 ±\pm 0.00190.5218 ±\pm 0.0014
PlanPers0.4516 ±\pm 0.00850.4619 ±\pm 0.00200.5319 ±\pm 0.00530.6144 ±\pm 0.0012
IAP0.5078 ±\pm 0.00270.4831 ±\pm 0.00220.5541 ±\pm 0.00020.5510 ±\pm 0.0013
GRASP0.5254 ±\pm 0.00080.5360 ±\pm 0.00130.5630 ±\pm 0.00230.6177 ±\pm 0.0004
RTV threshold τ\tauGemma2-9BQwen2.5-7B
0.0 (without RTV)0.5112 ±\pm 0.00130.5279 ±\pm 0.0023
0.20.5117 ±\pm 0.00160.5315 ±\pm 0.0015
0.50.5120 ±\pm 0.00060.5333 ±\pm 0.0025
0.80.5157 ±\pm 0.00120.5335 ±\pm 0.0012
1.00.5254 ±\pm 0.00080.5360 ±\pm 0.0013

GRASP retains the highest mean score across all four backbones. On Gemma2-9B and Qwen2.5-7B, RTV with τ=1\tau=1 consistently outperforms other settings, supporting the benefit of validating teacher supervision.

R4: IAP Reproduction (Q4, W1)

Thank you for raising this concern. The original IAP paper does not report results on Gemma2-9B, Qwen2.5-14B, or Qwen3-4B. Our initial reproduction followed its public repository, but we identified some issues during training. In the rebuttal period, we subsequently reimplement IAP in the Swift framework. Following this setting, IAP achieves 0.5078 ±\pm 0.0027 on Gemma2-9B, 0.5541 ±\pm 0.0002 on Qwen2.5-14B, and 0.5510 ±\pm 0.0013 on Qwen3-4B. We will provide the reproduction details in the revision.

R5: Contextual grounding and unsupported attributes (Q5, W2)

Thank you for raising this point. Actually, "cat" comes from the question, not the profile. After distillation, GRASP assigns it a higher probability, consistent with better alignment with the question context.

We also use GPT-6-Luna to compare unsupported user-attribute claims in GRASP and RAG against the question and retrieved profile. GRASP is rated more severe in 4.52% of cases, less severe in 3.18%, and comparable in 0.64%; neither response is flagged in 91.66%. These findings suggest that GRASP only introduces minor user-attribute hallucinations. We will clarify these details in the revision.

AuthorsOpenReview ↗

Response to Reviewer zNSr (2/2)

R6: Additional evaluation and length control (Q6, W2)

Thank you for this helpful suggestion. With GPT-6-Luna as an additional judge, GRASP outperforms the strongest baseline across four backbones:

BackboneBest BaselineGRASP
Gemma2-9B0.34350.3600
Qwen2.5-7B0.32230.3533
Qwen2.5-14B0.37200.3833
Qwen3-4B0.38170.3855

We also report response lengths and length-controlled scores from the original judge, calibrated on the rubric-informed teacher's outputs:

BackboneGRASP avg. lengthBest baseline LCGRASP LC
Gemma2-9B1,384.40.51620.5351
Qwen2.5-7B1,677.30.49700.5394
Qwen2.5-14B1,749.60.56010.5678
Qwen3-4B3,104.30.60520.6143

GRASP achieves the best performance across all four backbones with both the additional judge and the LC-Score, suggesting that its gains reflect stronger personalization rather than longer responses.

R7: Shuffled rubric ablation (Q7, W2)

Thank you for this question. Actually, the teacher input in the shuffled-rubric ablation does not include the user elaboration, so this field is not shuffled. We shuffle only the rubric information across instances, keeping the question and retrieved user profile fixed. We will revise the prompt to match the actual setup.

R8: Inferred rubrics and robustness (Q8, W2)

Thank you for this suggestion. We trained GRASP on Gemma2-9B with PlanPers-predicted aspects as the teacher's privileged context, keeping the retained training instances and student inputs unchanged.

Teacher contextA&EL&PDS&CAvg.
Predicted rubrics0.35200.46050.53010.4475
Original rubrics0.43720.56060.57870.5255

Together with the performance drops under shuffled rubrics and the Single setting, these results highlight the importance of rubric quality, alignment, and completeness for effective supervision. We will clarify this limitation and include these results in the revision.

R9: Broader personalization benchmarks (Q9, W2)

Thank you for this suggestion. LaMP-QA is a suitable testbed for GRASP because it provides explicit user-specific rubrics. Due to the limited rebuttal period, we have not completed evaluations on additional benchmarks, but we will aim to include them in the revision.

R10: Related work and limitations (Q10, W2)

Thank you for this invaluable suggestion. Fang et al. derive rubrics from teacher-student responses and use rubric scores as GRPO rewards; GRASP instead uses rubrics as teacher context for token-level distribution matching. Kim et al. show that richer teacher context can suppress uncertainty expression and impair generalization. Our MMLU-Pro evaluation shows largely preserved general performance after training:

BackboneBeforeAfter
Gemma2-9B0.51540.5063
Qwen2.5-7B0.57320.5761
Qwen2.5-14B0.64320.6403
Qwen3-4B0.61200.6261

However, as reported in R5, GRASP can still generate unsupported claims about user attributes. We will add a limitation section to discuss it in detail.

AuthorsOpenReview ↗

Follow-up on Our Response and Revision

Dear Reviewer,

Thank you again for your detailed and constructive feedback, which has helped us improve our work. We have provided detailed responses and incorporated the following additions and clarifications into the revised manuscript:

  • Rubric-informed teacher performance (Appendix E.12).
  • Comparisons with off-policy training and rubric-reward GRPO (Section 5.3 and Appendix E.7).
  • Mean ±\pm standard deviation across three seeds for the main results and RTV threshold analysis (Table 1 and Appendix E.1, Table 6).
  • Updated IAP reproduction and implementation details (Table 1 and Appendix C).
  • Alternative-judge evaluation and response-length analysis (Appendices E.10–E.11).
  • Unsupported user-attribute claims and the evaluation prompt (Appendix E.14 and Appendix G, P4).
  • Predicted-rubric experiments and clarification of teacher inputs (Appendices E.5, E.8, and G).
  • General-capability evaluation on MMLU-Pro (Section 5.3, Figure 5), expanded related work (Section 2), and limitations (Appendix A).

We welcome further discussion and appreciate your continued feedback.

Best regards,

The Authors

Reviewer_Rh7GOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper applies on-policy self-distillation (OPSD) to personalized question answering on LaMP-QA. The teacher is a frozen copy of the backbone model prompted with the user’s rubric aspects and, according to Appendix F, also receives the user’s narrative and supporting evidence for each aspect. The student is a trainable copy of the same backbone that only sees the question and retrieved user profile, and is trained to minimize the full-vocabulary forward KL divergence to the teacher on its own sampled responses. Before training, Rubric-based Teacher Validation (RTV) filters out examples where the teacher’s response does not fully cover the rubric, based on scores from a Qwen2.5-32B judge.

GRASP achieves the best macro-average performance across four backbones: Gemma2-9B, Qwen2.5-7B, Qwen2.5-14B, and Qwen3-4B. The paper also includes ablations on the RTV threshold, teacher size, shuffled rubrics, and single versus full rubrics, along with a token-level analysis of how the output distribution shifts after training.

Critical strengths
  • The idea is simple and well matched to the problem. LaMP-QA provides rich per-instance specifications during training that are unavailable at inference time, making it a natural fit for the privileged-information setting that OPSD is designed to handle. The method also adds no inference-time cost, while training remains relatively modest.
  • The gains are substantial on three of the four backbones, improving over the strongest baseline by about 0.064 on Gemma2-9B, 0.052 on Qwen2.5-7B, and 0.040 on Qwen2.5-14B on average. The improvements also generalize across two different model families.
  • The shuffled-rubric control is especially useful, showing that misaligned rubrics reduce performance back to roughly the RAG baseline. The comparison between single and full rubrics is also informative, as is the larger-teacher experiment. In particular, the fact that a larger teacher does not improve performance is a useful negative result.
Critical weaknesses
  • The privileged information includes more than the rubric, but the paper does not isolate which part is actually responsible for the gains. Section 4.1 defines the privileged information as z=Exuz = E_{x_u}, which suggests that the teacher is conditioned on the rubric aspects. However, the actual teacher prompt in P2 contains substantially more information: the user’s own elaboration of the question, the rationale for each aspect, and verbatim evidence in the user’s own words. In particular, the narrative appears to be close to a fuller version of the original question. This makes the results in Table 6 difficult to interpret. If the Single setting still includes the narrative, then its large improvement over all baselines, for example 0.4956 on Gemma compared with 0.4612 for PlanPers, may be driven largely by access to the narrative rather than by the rubric itself.
  • It is unclear how much of the improvement actually comes from personalization. The rubrics in LaMP-QA are derived from the narrative associated with the current post rather than from the user profile. This leaves open the possibility that the student is learning task- or category-level priors without relying much on the retrieved profile. For example, it may learn that questions in a religion forum often benefit from scriptural references, even without any user-specific information.
  • The novelty is modest, and the claimed importance of supervision granularity is not well supported by the results. At a technical level, the method is essentially OPSD with the rubric used as privileged context, together with a threshold-based filtering step. This makes the distinction from prior work especially important. In particular, Fang et al. (2026), Rubric-based on-policy distillation, is only mentioned in a citation list despite appearing closely related based on its title. The paper should explain more clearly how GRASP differs from that work. The supervision-granularity argument in Table 4 is also not supported consistently by the reported results. The table suggests that finer-grained supervision should perform better, yet IAP (#2) underperforms PlanPers (#3) on three of four backbones and SFT (#4) on two. This weakens the claim that supervision granularity itself explains the observed performance differences.

Junfeng Fang, Zhepei Hong, Mao Zheng, Mingyang Song, Gengsheng Li, Houcheng Jiang, Dan Zhang, Haiyun Guo, Xiang Wang, and Tat-Seng Chua. Rubric-based on-policy distillation. arXiv preprint arXiv:2605.07396, 2026.

Questions for authors
  • What exactly is in the privileged context z? Does the Single setting in Table 6 still include the narrative and evidence? How do narrative-only and aspects-only teachers perform?
  • Figure 6 shows training KL starting around 0.45–0.6, while Figure 4(b) reports 1.10 on held-out data. Is the gap due to held-out rubrics having about 5 aspects versus about 3 in training (Table 3)? Does that train/test shift in aspect count affect the method?
  • Table 4: PlanPers is marked as “no training,” although its planner is LoRA-fine-tuned.
4 comments

Dear Reviewer,

We sincerely appreciate your invaluable comments on our work. We have carefully considered all your concerns and have provided detailed responses below.


R1: Teacher inputs and the contribution of rubric information (W1, Q1)

Thank you for pointing out this ambiguity. Our actual training setup provides the teacher with the question, retrieved user profile, and rubric entries, without separately including the full narrative. Appendix F presents a broader prompt template; we will revise it to match the actual setup.

We further conducted narrative ablations on Gemma2-9B to assess the contribution of narrative information as shown below:

Teacher contextAvg.
Single aspect0.4956
Narrative only0.4975
Full aspects + narrative0.5200
Full aspects (ours)0.5255

Single also uses only one rubric entry without the full narrative. Narrative-only performs comparably to Single but below Full aspects, while adding the narrative to Full aspects yields no improvement.

R2: Contribution of user history for personalization (W2)

Thank you for raising this concern. We evaluated GRASP with profiles from other users over five random mappings, obtaining 0.4789 ±\pm 0.0016, a substantial drop compared with the original profiles. This indicates that guidance from the correct user profile contributes to GRASP's personalization performance.

R3: Supervision Granularity and Contribution (W3)

Thank you for raising this concern. Fang et al. derive rubrics from teacher-student responses and use rubric scores as response-level rewards for GRPO. Instead, GRASP conditions the teacher on user-specific rubrics to provide token-level distributional supervision. As discussed in R1, our ablations support the contribution of rubric-informed guidance beyond access to the user narrative.

Our contribution is the introduction and empirical validation of rubric-aware on-policy self-distillation for LLM personalization, complemented by RTV to filter unreliable teacher supervision. We additionally evaluated SFT and off-policy KD on RTV-selected instances, comparing them with the GRASP results:

BackboneSFTOff-policy KDGRASP
Gemma2-9B0.47520.46910.5255
Qwen2.5-7B0.47290.47440.5357
Qwen2.5-14B0.50130.49750.5657
Qwen3-4B0.55230.54430.6180

GRASP outperforms both baselines across all four backbones. Since off-policy KD also provides token-level supervision, this comparison supports the importance of on-policy distillation beyond granularity alone.

We also revisited the IAP reproduction during rebuttal. Our updated IAP results rank second on Gemma-2-9B, Qwen2.5-7B, and Qwen2.5-14B, supporting the value of rubric-related supervision; reproduction details will be included in the revision.

Overall, our contribution extends beyond technical design by introducing token-level rubric-informed guidance to personalization, demonstrating the effectiveness of OPSD in this setting, and opening a direction for future research.

R4: KL measurement and comparability (Q2)

Thank you for pointing out this discrepancy. Figure 6 measures KL on training rollouts, whereas Figure 4(b) averages answer-token KL over a small held-out subset of 90 fixed responses. These measurements are not directly comparable, so the gap alone cannot be attributed to the train-test difference in aspect count.

To assess the impact of this shift, we stratified test instances by rubric aspect count. GRASP's gains over RAG are:

Aspect count# QuestionsGemma2-9BQwen2.5-7BQwen2.5-14BQwen3-4B
≤\le 3222+0.1521+0.1075+0.1145+0.0977
4847+0.1373+0.1157+0.1267+0.1033
51,065+0.1111+0.1026+0.1174+0.1000
≥\ge 6696+0.1015+0.0849+0.0884+0.0939

GRASP maintains positive gains across all groups, supporting generalization from training rubrics averaging 2.92 aspects to test rubrics averaging 4.91.

R5: Training requirements of PlanPers (Q3)

Thank you for raising this point. We intended to indicate that PlanPers keeps its backbone answer generator frozen and trains only the planner. We will clarify this distinction.

AuthorsOpenReview ↗

Follow-up on Our Response and Revision

Dear Reviewer,

Thank you again for your thoughtful feedback, which has helped us improve our work. We have responded to all your concerns and incorporated additional experiments and clarifications into the revised manuscript, particularly regarding teacher context (Appendix E.8), the contribution of user history (Section 5.3), and comparisons with off-policy training (Section 5.3).

When convenient, we would appreciate a brief follow-up on our responses. We are happy to discuss further.

Best regards,

The Authors

Thank you for the additional experiments. They have addressed my concern, I have raised the score.

Dear Reviewer,

Thank you for your recognition of our work. We sincerely appreciate your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response and Summary of Revisions

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their thoughtful feedback and constructive discussion. In this paper, we propose GRASP, a rubric-aware on-policy self-distillation framework that transfers user-specific rubric guidance into personalized generation. All reviewers recognized its simple and practical formulation, consistent improvements across four backbones, and informative ablations and mechanistic analyses connecting the gains to rubric supervision. We appreciate their insightful suggestions and summarize the additional experiments and manuscript revisions below.


  • Extended experimental results (zNSr): Added main results and RTV analysis across three seeds, together with updated IAP results and implementation details (Table 1; Appendix C; Appendix E.1, Table 6).
  • Additional training baselines and ablations (Rh7G, gWdn, zNSr): Added SFT and off-policy KD on the same RTV-selected data, plain rubric-reward GRPO, alternative distillation objectives, and RTV versus equal-size random selection (Section 5.3, Table 3; Appendices E.1, E.6, and E.7).
  • Further analysis of rubric-guided personalization (Rh7G, gWdn, zNSr): Examined teacher context, user history, rubric count, predicted rubrics, and rubric-informed upper-bound performance (Section 5.3; Appendices E.4, E.5, E.8, and E.12).
  • Broader evaluation perspectives (gWdn, zNSr): Added GPT-6-luna evaluation, response-length analysis, an audit of unsupported user-attribute claims, and an evaluation of general capabilities on MMLU-Pro (Section 5.3, Figure 5; Appendices E.10-E.11 and E.14; Appendix G, P4).
  • Expanded discussion (Rh7G, gWdn, zNSr): Enriched the related-work discussion, clarified teacher prompts, and discussed limitations (Section 2; Appendices A and G).

These additional experiments and clarifications provide further evidence for the effectiveness and applicability of our method. During the rebuttal discussion, reviewers Rh7G and gWdn acknowledged our revisions and raised their scores, bringing the current ratings to 3 / 3 / 3 (Weak acceptance). We sincerely thank the reviewers for their recognition of our work and constructive feedback, which have helped us improve the manuscript. We greatly appreciate the reviewers' and Area Chair's time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.