acceptodds
Under review as a conference paper at ICLR 2027

Hypotheses-Guided Self Distillation for Continual Personalization

Abstract

As people increasingly interact with LLM assistants in daily life, continually adapting to individual preferences has become essential for effective long-term interactions. However, user preferences are rarely stated in full, and instead emerge through heterogeneous, latent, and noisy signals, with existing methods relying on raw interaction histories or costly reward-based optimization to manage personalization. We introduce HypReflect, a reliable, scalable framework for continual personalization that infers explicit, uncertainty-aware preference hypotheses from diverse user signals, reflectively refines them over time, and incorporates them into the model via hypothesis-guided self-distillation. Experiments across three personalization settings: online personalization, multi-session interactions, and implicit behavioral signals, show that HypReflect outperforms a range of baselines, including raw-history and incremental-update methods. We further demonstrate strong generalization to unseen users and cross-domain settings, along with stability across context budgets, reusable hypotheses, and more focused personalization. These results suggest a step towards reliable and scalable continual personalization through explicit, revisable user preference hypotheses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
continual personalization, user modeling, preference hypotheses, self distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qqdPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HYPREFLECT maintains explicit preference hypotheses inferred from interaction chunks and refines them through a bounded reflection window. The refined hypotheses condition both the teacher and student during self-distillation, while the teacher additionally receives privileged information from subsequent user interactions. Experiments cover explicit feedback, multi-session personalization, and implicit behavioral preferences across three backbone models, with additional analyses of generalization, hypothesis reuse, and component contributions.

Critical strengths
  1. HYPREFLECT provides a well-motivated way to connect user modeling with response learning. Its bounded reflection window jointly revisits independently generated local hypotheses and an earlier consolidated state, allowing accumulated evidence to inform subsequent revisions. Conditioning both teacher and student on the refined hypotheses gives the method a clear interface between explicit user memory and self-distillation.

  2. The empirical evaluation covers complementary personalization signals across three datasets and three backbone models. Comparisons with incremental summaries and hypotheses, reflective prompting, and the reflection ablation help assess the contribution of the proposed components. The unseen-user, cross-domain, and hypothesis-reuse experiments further support the practical value of the learned user representations.

Critical weaknesses

1.The main limitation is the strength of the demonstrated advance over existing self-distillation and incremental hypothesis maintenance. While the proposed integration is useful, the added benefit of reflective refinement over the closest alternatives is uneven. For example, on Flight with Qwen3.5-4B, SD-IncHyp achieves 74.8% accuracy, compared with 74.0% for HYPREFLECT and 74.2% for HYPREFLECT+Sum (Table 1). The reflection ablation on HiCupid yields gains of 1.4 and 0.7 percentage points (Table 5), without reported uncertainty estimates. Although larger improvements elsewhere support the approach, the evidence for a substantial and robust contribution from the proposed refinement mechanism remains less convincing than the effectiveness of the complete system.

2.The uncertainty-aware aspect is not sufficiently validated. Each hypothesis receives an LLM-generated score from 1 to 5, with prompt-defined levels of evidential support (Section 3.1 and Figure 11). While this is a reasonable representation, the experiments do not establish whether these scores improve refinement or response quality beyond the hypothesis statements themselves. A comparison with unweighted hypotheses or uniform scores is missing, leaving unclear whether the reported benefits depend on uncertainty information. Thus, the framework explicitly represents confidence, but the practical contribution of this component remains unestablished.

Questions for authors
  1. Could the authors explain the explict situation when reflection offers a meaningful advantage over simpler hypothesis maintenance?

  2. How much do the confidence scores contribute independently of the hypothesis text? A comparison using the same hypotheses with omitted or uniform scores would help establish their role, particularly when the evidence is ambiguous or conflicting.

2 comments

We thank the reviewer for the thoughtful feedback and for highlighting how HypReflect connects explicit user modeling with response learning through self-distillation. We also appreciate the reviewer’s recognition of our evaluation across different personalization signals, including experiments on unseen users, cross-domain transfer, and hypothesis reuse. We address the weaknesses raised by the reviewer below and highlight how we intend to incorporate the changes in the revised paper.


  • When does reflection help?

As shown in the comparison between SD-IncHyp and HypReflect in Table 1, the reflection ablation in Table 5, and the qualitative examples in Figures 6 and 7, we find that reflection can help when preference evidence is spread across several chunks and the model needs to reconcile conflicting inferences or preserve specific preferences during consolidation. More specifically, on HiCuPID, reflection preserves specific preferences that direct consolidation obscures (Figure 6), and on Flight, it resolves conflicting inferences about price and duration (Figure 7). Empirically, adding reflection improves scores by 0.7–1.4 points on HiCuPID and 1.8–3.9 points on Flight (Table 5). HypReflect also outperforms SD-IncHyp, our baseline for simpler hypothesis maintenance in the continual update setting, in six of nine settings (Table 1).


  • Robustness of gains over SD

While Table 1 shows that HypReflect outperforms SD in eight of nine settings across datasets and model backbones, we agree with the reviewer on the value of a multi-seed comparison and report results from three random seeds below.

HelpSteer. We note that Table 1 already reports scores averaged over eight user profiles on evaluation set. For the three-seed experiment, we trained the Qwen3.5-4B assistant with a user simulator configured with the concise, casual beginner profile. The simulator provided feedback specific to each interaction, and we used the same evaluation set for all runs. The mean scores were 84.9 for SD and 87.6 for HypReflect, a gain of 2.7 percentage points. At each checkpoint (steps 0, 50, …, 250), we computed the sample standard deviation across the three seeds. Averaged across checkpoints, these standard deviations were 1.8 points for SD and 2.7 points for HypReflect.

HiCuPID and Flight. The three-seed results are broadly consistent with Table 1. HypReflect scores higher than SD on average for both datasets and both backbones, with modest gains on Flight. We also acknowledge that Flight is a tough setting for hypothesis extraction because preferences must be inferred from implicit choices alone, and the available context in the dataset often gives little insight into some synthetic choices (e.g., why a user would pick a 30-hour flight over an otherwise identical 8-hour flight). Here HypReflect is marginally better than SD. We will discuss this limitation in the appendix and include the table below, which reports the original scores alongside the mean ± sample standard deviation across seeds.

DatasetMethodQwen3.5-4B originalQwen3.5-4B mean ± std.Gemma4-4B originalGemma4-4B mean ± std.
HiCuPIDSD69.771.30 ± 1.1857.358.27 ± 0.84
HypReflect71.173.52 ± 1.6361.160.10 ± 0.95
FlightSD73.872.77 ± 1.0061.864.03 ± 1.94
HypReflect74.073.27 ± 0.6762.364.57 ± 1.97

  • Contribution of confidence scores

We thank the reviewer for suggesting this ablation. To further examine whether the confidence scores reflect meaningful differences in hypothesis quality, we evaluated each hypothesis individually by asking the model to predict the user's observed choice from that hypothesis alone. On Flight, where HypReflect generates up to three hypotheses per user, prediction accuracy increased with the assigned confidence score (w) across all three models, as shown in the table below.

Modelw = 1w = 2w = 3w = 4w = 5
Qwen3.5-4B41.4%45.5%47.2%48.3%53.3%
Qwen3.5-9B—41.5%46.3%51.4%55.7%
Gemma4-4B—40.4%41.0%49.5%51.5%

To assess the contribution of explicit confidence scores, we trained Qwen3.5-4B separately on Flight with and without them. The model trained with confidence scores achieved 73.1% accuracy, compared with 71.8% without confidence scores, a gain of 1.3 percentage points.

These results demonstrate that explicit confidence scores provide useful information beyond the hypothesis text, while making the model's assessment of each hypothesis more transparent and interpretable. We will include both analyses in the revised paper.

Thank you for the response. My overall assessment of the paper remains positive, and I will keep my original score.

Reviewer_i2GaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HYPREFLECT, a continual personalization framework which leverages self-distillation in which natural language user preference hypotheses provide helpful conditioning context for both teacher and student. HYPREFLECT is shown to engender performance gains compared to pertinent baselines across a suite of personalization benchmarks.

Critical strengths

The main strengths in my eyes of this work are 1) the proposed preference hypotheses approach to guide self-distillation, and 2) shown (contingent on being statistically clarified), generalization of HYPREFLECT across model families/scales. Using natural language preference hypotheses is an intuitive method + allows easy interpretation and revision. Using the hypotheses to guide self-distillation is a clever application. I also appreciate how the author's tested HYPREFLECT across different model families (qwen/gemma) and scales (4B and 9B), to show that HYPREFLECT is a (potentially) generalizable method.

Critical weaknesses

The biggest concerns for me are 1) lack of tests for evolving user preferences, and 2) lack of statistical uncertainty in benchmark metrics. If my understanding is correct, HYPREFLECT is framed as being an effective method to dynamically adapt to new user preferences, however the given benchmarks do not change user preferences dynamically. It would be nice to find some setting in which user preferences change and demonstrate HYPREFLECT works there as well. I am also concerned by the lack of statistical uncertainty measures. Without confidence measures in the form of things such as performance stds over different runs or paired t-tests, I am unsure of how to interpret the results.

Questions for authors

Any idea on how this method might scale beyond 9B?

3 comments
AuthorsOpenReview ↗

Response to Reviwer i2Ga (1/2)

Thank you for thoughtful feedback. We’re glad you found the preference hypotheses easy to interpret and revise, and saw value in using them to guide self-distillation. We also appreciate your comments on the breadth of our evaluation across model families and sizes.


Adaptation to changing user preferences

We thank the reviewer for highlighting the importance of evaluating adaptation to changing user preferences. Our original HelpSteer2 experiments capture a complementary form of dynamic adaptation: progressive preference revelation. The user simulator provides feedback based on its assigned profile and the current interaction, revealing only relevant aspects of the user's preferences over time. Thus, the model must incrementally refine its user representation as new information emerges, even when the underlying preferences remain unchanged. To directly address preference changes as suggested by the reviewer, we additionally conducted an experiment in which the user's profile completely changes during training. This evaluates whether HypReflect can adapt its learned representation to a new set of preferences.

Adapting to a profile change. On HelpSteer2, we trained HypReflect with Qwen3.5-4B for 100 steps using user simulator feedback profiled with detailed, professional, expert, then switched user simulator to the concise, casual, beginner profile and continued training for another 100 steps. Every 25 steps, we evaluated HypReflect against the base model on 100 randomly sampled examples using the active profile. Within 25 steps of the switch, average response length dropped from 2,006 to 605 tokens. HypReflect outperformed the base model at all four checkpoints under the new profile, averaging a 85% win rate.

Training step255075100125150175200
Evaluation profileDetailed, professional, expertConcise, casual, beginner
Mean response tokens1,6992,1082,0972,006605519475357
Win rate vs. base60.0%88.0%81.5%89.5%89.5%89.5%88.5%93.5%

Preference feedback in the original HelpSteer2 experiments. As mentioned previously, the user simulator, in the HelpSteer2 setting, provides feedback which gradually reveals the underlying user preferences over time. To characterize the feedback underlying the results reported in the paper, we used DeepSeek-V4-Flash to identify preference directions (e.g., “more formal”) in the simulator’s feedback and HypReflect’s hypotheses. The table shows how many preferences had surfaced by the end of each interval, how much of the feedback HypReflect’s hypotheses captured, and the corresponding win rates against the base model. Only 38–44% of the preferences surfaced by turn 5, and 79–85% had surfaced by turn 50. HypReflect’s hypotheses are able to continually cover at least 70% of the relevant preferences expressed in feedback across most intervals for both models.

These experiments demonstrate HypReflect's ability to adapt to both progressive preference revelation and preference evolution.

TurnsQwen3.5-4B revealedQwen3.5-4B coverageQwen3.5-4B win rateGemma4-4B revealedGemma4-4B coverageGemma4-4B win rate
1–538%81%—44%81%—
6–1050%81%—59%72%—
11–2568%62%—74%77%—
26–5079%77%94.4%85%79%91.4%
51–10090%71%96.6%91%77%93.3%
101–15095%69%93.7%97%78%94.7%
151–20098%75%95.3%99%77%96.5%
201–250100%70%95.3%100%77%94.6%
AuthorsOpenReview ↗

Response to Reviwer i2Ga (2/2)

Robustness of gains over SD

While Table 1 shows that HypReflect outperforms SD in eight of nine settings across datasets and model backbones, we agree with the reviewer on the value of a multi-seed comparison and report results from three random seeds below.

HelpSteer. We note that Table 1 already reports scores averaged over eight user profiles on evaluation set. In response to the reviewer’s request for multi-seed experiment, we trained the Qwen3.5-4B assistant with a user simulator configured with the concise, casual beginner profile with three random seeds. The simulator provided feedback specific to each interaction, and we used the same evaluation set for all runs. The mean scores were 84.9 for SD and 87.6 for HypReflect, a gain of 2.7 percentage points. At each checkpoint (steps 0, 50, …, 250), we computed the sample standard deviation across the three seeds. Averaged across checkpoints, these standard deviations were 1.8 points for SD and 2.7 points for HypReflect.

HiCuPID and Flight. The three-seed results are broadly consistent with Table 1. HypReflect scores higher than SD on average for both datasets and both backbones, with modest gains on Flight. We also acknowledge that Flight is a tough setting for hypothesis extraction because preferences must be inferred from implicit choices alone, and the available context in the dataset often gives little insight into some synthetic choices (e.g., why a user would pick a 30-hour flight over an otherwise identical 8-hour flight). Here HypReflect is marginally better than SD. We will discuss this limitation in the appendix and include the table below, which reports the original scores alongside the mean ± sample standard deviation across seeds.

DatasetMethodQwen3.5-4B originalQwen3.5-4B mean ± std.Gemma4-4B originalGemma4-4B mean ± std.
HiCuPIDSD69.771.30 ± 1.1857.358.27 ± 0.84
HypReflect71.173.52 ± 1.6361.160.10 ± 0.95
FlightSD73.872.77 ± 1.0061.864.03 ± 1.94
HypReflect74.073.27 ± 0.6762.364.57 ± 1.97

Scaling beyond 9B

We thank the reviewer for raising this point. We expect HypReflect to benefit further from larger backbones. In Table 1, HypReflect yields more stable gains with Qwen3.5-9B than with Qwen3.5-4B or Gemma4-4B on three datasets, reaching up to 3.1 points on HelpSteer2, 6.5 on HiCuPID, and 1.6 on Flight. A more capable backbone may infer better-supported preference hypotheses from user feedback, providing more informative targets for self-distillation. Shenfeld et al. [1] likewise found that the gains from on-policy self-distillation can grow as the model size increases.

[1] Self-Distillation Enables Continual Learning, Shenfeld et al. (2026)., ICML 2026.

Thank you for your thoughtful review! We have addressed each of the weaknesses and questions raised in your review. Please let us know if anything is unclear, we are open to learn about them.

Reviewer_7QFmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies continual personalization of large language models when user preferences emerge gradually through explicit feedback, profile information, and implicit behavioral choices. It builds on prior self-distillation approaches in which the same model serves as both student and teacher: the student responds using information available at the current interaction, while the teacher additionally observes the subsequent user turn as privileged hindsight. The student is trained to match the teacher through a reverse-KL distillation objective.

The proposed method, HYPREFLECT, adds to this setup (introduced by Buening et al.) by maintaining an explicit collection of beliefs about the user: each belief is an ordered pair (some natural language claim about user preferences, confidence between 1 and 5 for this claim), e.g., (the user prefers concise answers, 3). Their method maintains a limited, small number of such beliefs (e.g., capped at 3 or 5 in their experiments), and updated in each round through (other) LLMs, trained through a reverse-KL objective.

The interaction history is divided into bounded-size chunks, and an LLM generates local hypotheses from each chunk. To maintain these hypotheses over longer histories, HYPREFLECT periodically performs an LLM-based reflection over several local hypothesis sets, identifying which hypotheses should be preserved, merged, narrowed, reweighted, or removed. A subsequent refinement step produces an updated hypothesis set, while a fixed reflection window keeps the procedure bounded in context size. The refined hypotheses are supplied to both student and teacher during self-distillation, with the teacher additionally receiving the next user turn. The paper also considers HYPREFLECT+SUM, which maintains a textual summary alongside the preference hypotheses.

Experiments cover three settings: HelpSteer2, where synthetic user writing-style preferences are gradually revealed through feedback; HiCupid, where user information is distributed across multi-session conversations; and Flight Recommendation, where latent preferences must be inferred from repeated choices. The main baselines include plain self-distillation over raw history (SD), self-distillation with an incrementally maintained summary (SD-IncSum), and self-distillation with an incrementally maintained hypothesis set (SD-IncHyp). HYPREFLECT differs from SD-IncHyp by retaining multiple local hypothesis sets and jointly refining them rather than recursively updating only one running state.

The paper reports that HYPREFLECT outperforms plain SD in eight of nine main model/dataset settings, while HYPREFLECT+SUM does so in all nine. Additional experiments report that refined hypotheses can improve a frozen base model, that self-distillation provides further gains beyond inference-time hypothesis prompting, and that the reflection step improves over direct consolidation of hypotheses. The paper also evaluates unseen users, cross-domain transfer, varying context budgets, and reuse of the learned hypotheses by non-finetuned models.

Critical strengths

The paper addresses an important practical problem: continual personalization when user preferences change or are only partially revealed over time through mixed signals. The proposed framework is clear and easy to interpret. In particular, representing the user through explicit natural-language preference hypotheses provides a transparent intermediate state between raw interaction history and personalized generation, while the reflective refinement procedure gives the model a mechanism for revising this state as additional evidence arrives. The integration with self-distillation is also clean: the same refined user representation is supplied to both student and teacher, with the teacher additionally receiving the subsequent user turn as privileged information.

Critical weaknesses

The experiments do not present convincing evidence of improvement over the baseline. There are two main issues:

  1. The improvements over the baseline are modest, and not meaningful without error bars/standard deviations. Consider Table 1: the main methods in the paper (HypReflect and HypReflect+Sum) have an average improvement over the previous SD baseline of <2%< 2\% and ≃3%\simeq 3\% respectively. This is not significant, specially without any details of seeding, number of runs, or other details.

  2. Several claims in the experimental section are overstated. Examples:

  • Table 1: "HYPREFLECT further improves over SD-INCHYP and outperforms SD in eight of nine settings."

Technically true, but unclear if the improvements are within margin of error.

  • Table 1/Figure 3: "We find that incremental updates can propagate earlier errors and allow recent observations to overwrite valid prior information, resulting in unstable supervision over time. In contrast, reflective refinement maintains more reliable preference hypotheses as interaction histories grow."

Even if we concede that the paper's method is more stable, it does not imply anything about propagating earlier errors. Better performance does not explain the mechanism by which the improvement was achieved.

  • "Self-distillation teaches more effectively than GRPO."

This is essentially the result from the previous paper (Buening et al).

  • Section 5.2, Table 3: "HYPREFLECT+SUM improves over SD in all nine settings, often by substantial margins."

In 7 of 9 settings, the improvement is less than 2.5%.

  • Section 5.2, Table 3: "allowing the model to learn a cleaner mapping from user behavior to underlying preferences."

As before, making this inference from the data alone is unjustified.

A smaller issue is that technically, the paper is largely incremental over Buening et al. This would be ok if the experimental results were stronger.

Questions for authors
  1. What model is used to generate and refine the preference hypotheses? It is unclear whether hypothesis generation, reflection, and refinement are performed by the currently fine-tuned backbone model, a frozen copy of the base model, or some separate auxiliary LLM. Could the authors clarify this and specify whether the same model is used across all datasets and stages?

  2. Were the SD baseline and the proposed methods tuned independently? The appendix reports tuning learning rate and batch size “for each dataset,” but it is not clear whether SD, SD-IncHyp, HYPREFLECT, etc. each received an independent hyperparameter search with the same tuning budget. Could the authors clarify how hyperparameters were selected for each method?

  3. Why is HelpSteer2 omitted from Table 2?

  4. How were the subsets used in the ablations selected?

3 comments
AuthorsOpenReview ↗

Response to Reviwer 7QFm (1/2)

We thank the reviewer for the detailed feedback and for highlighting the clarity of our framework, the interpretability of our explicit preference hypotheses, and our clean integration with self-distillation.


Clarification on our contribution.

  • Developing a single method that works across three substantially different personalization settings is not trivial. Online feedback, multi-session interactions, and implicit behavioral choices provide different types of evidence for inferring preferences. We propose a way to construct a useful user preference representation from these different forms of evidence and show that it improves performance across all three settings.
  • We provide a way to effectively incorporate these representations into self-distillation by conditioning both the student and the teacher on the same hypotheses. We note that in our preliminary experiments, generating hypotheses with access to ground-truth answers in teacher prompt or providing them only to the teacher led to larger gaps between the teacher’s and student’s logits and less effective learning. Sharing the same hypotheses reduces the knowledge gap between the teacher and the student, making it easier for the student to learn from the teacher’s predictions. Our findings can offer guidance for future work on how to condition the teacher and student on explicit user representations when training personalized models through self-distillation.
  • Our evaluation demonstrates both generalization and the utility of the representation itself. Table 3 shows generalization to unseen users and transfer across domains, with HypReflect outperforming SD in most settings. Table 4 further shows that providing the constructed hypotheses to a frozen base model yields average improvements of 11.5 points on Flight and 10 points on HiCuPID, demonstrating that the representation provides useful preference information even without fine-tuning.

We believe that the contributions outlined above, along with our ablation studies on reflections and qualitative analyses on hypotheses refinement and final response, go beyond an incremental extension of the self-distillation method proposed by Buening et al. (2026).


Robustness of gains over SD

While Table 1 shows that HypReflect outperforms SD in eight of nine settings across datasets and model backbones, we agree with the reviewer on the value of a multi-seed comparison and report results from three random seeds below.

HelpSteer. We note that Table 1 already reports scores averaged over eight user profiles on evaluation set. In response to the reviewer’s request for multi-seed experiment, we trained the Qwen3.5-4B assistant with a user simulator configured with the concise, casual beginner profile with three random seeds. The simulator provided feedback specific to each interaction, and we used the same evaluation set for all runs. The mean scores were 84.9 for SD and 87.6 for HypReflect, a gain of 2.7 percentage points. At each checkpoint (steps 0, 50, …, 250), we computed the sample standard deviation across the three seeds. Averaged across checkpoints, these standard deviations were 1.8 points for SD and 2.7 points for HypReflect.

HiCuPID and Flight. The three-seed results are broadly consistent with Table 1. HypReflect scores higher than SD on average for both datasets and both backbones, with modest gains on Flight. We also acknowledge that Flight is a tough setting for hypothesis extraction because preferences must be inferred from implicit choices alone, and the available context in the dataset often gives little insight into some synthetic choices (e.g., why a user would pick a 30-hour flight over an otherwise identical 8-hour flight). Here HypReflect is marginally better than SD. We will discuss this limitation in the appendix and include the table below, which reports the original scores alongside the mean ± sample standard deviation across seeds.

DatasetMethodQwen3.5-4B originalQwen3.5-4B mean ± std.Gemma4-4B originalGemma4-4B mean ± std.
HiCuPIDSD69.771.30 ± 1.1857.358.27 ± 0.84
HypReflect71.173.52 ± 1.6361.160.10 ± 0.95
FlightSD73.872.77 ± 1.0061.864.03 ± 1.94
HypReflect74.073.27 ± 0.6762.364.57 ± 1.97

AuthorsOpenReview ↗

Response to Reviwer 7QFm (2/2)

Experimental details

We thank the reviewer for the detailed feedback. Appendix B.2 describes the hyperparameter search, training and evaluation details, but we will clarify the details below in the updated version.

  • Hypothesis model. In each experiment, we use the same target backbone (that is supposed to be trained) for all four stages of our method: hypothesis generation, reflection, refinement, and final response generation.
  • Hyperparameter tuning. We tune the SD baselines and the proposed methods independently. Appendix B.2 describes the hyperparameter search space and training procedure. For each experiment, we report the results of the best checkpoint from the corresponding search.
  • HelpSteer2 in Table 2. HelpSteer2 uses online updates, so extending the analysis in Table 2 to HelpSteer2 would require evaluating at least five checkpoints at regular training intervals. At each checkpoint, we evaluate 256 prompts for each of the eight personas, using two LLM judge calls per prompt and randomizing the response order to reduce position bias. This would require at least 20,480 additional judge calls. Given the time and resource costs, we limited Table 2 to HiCuPID and Flight.
  • Ablation subset selection. For the reflection window size analysis and LLM judge validation on HelpSteer2, we randomly sampled 100 examples from its evaluation set primarily to reduce LLM judging costs. We observed similar performance trends on the subset and the full HelpSteer2 evaluation set. We will clarify this in the updated version.

Revisions to the experimental claims

Thank you for pointing out where our experimental claims could be more precise. We will make the following revisions.

  • Table 1. We will revise the claim to read "HypReflect scores higher than SD in eight of the nine settings reported in Table 1."
  • Table 1 and Figure 3. We will revise the sentence to make it more rigorous, reporting only empirical observations and omitting our previous interpretations “We find that HypReflect can yield more stable performance over time than the incremental hypothesis-update baseline.”
  • Section 5.2, Table 3. We will remove “often by substantial margins” and state that "HypReflect+Sum scores higher than SD in all nine settings, with gains that vary across settings."
  • Section 5.2, interpretation. We will remove the claim that the method allows the model to "learn a cleaner mapping from user behavior to underlying preferences."

Thank you for your thoughtful review! We have addressed each of the weaknesses and questions raised in your review. Please let us know if anything is unclear, we are open to learn about them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.