acceptodds
Withdrawn from ICLR 2027

Signal Before Scale: User History and Inference Compute in LLM Personalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalizing a language model consumes two resources with unrelated cost structures: information about the user, and compute at inference time. Their marginal returns have been measured separately; their interaction has not, leaving open whether compute can buy back missing user signal. We cross a per-user signal axis against an inference-compute axis. The signal axis is the number k ∈ 0,...,32 of the user's own profile items retrieved into the prompt, manipulated within user rather than observed across users, removing the confound with activity level. The compute axis is selector-free: three model tiers and a chain-of-thought sampling budget over a frozen model. The design spans four LaMP tasks and 11,197 calls. On the three tasks where retrieved history moves the metric, eight items capture at least 96% of the total gain. At matched sample size, 39 of 71 signal contrasts have 95% bootstrap intervals excluding zero, against 10 of 82 for compute. On the citation-choice and rating-prediction tasks, the smallest model given eight of the user's own items outperforms the largest given none; on the latter it halves the error. The interaction is resolvable, and its sign depends on the task: model scale and user history substitute on rating prediction and complement on movie tagging, at four items and above. Any account of when inference compute pays for personalization must be task-conditional. A direct measurement explains why: at temperature 1.0, 38–67% of items return an identical answer on every sample, leaving self-consistency nothing to aggregate until history is in context. With no trained selector present, the shortfall lies in the candidate pool rather than in selection. Signal before scale: extra compute needs signal before it pays.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM personalization, test-time compute, inference-time scaling, retrieval-augmented generation, scaling laws, self-consistency, empirical study

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ywkrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the interaction between user context and test-time compute in personalization. Through experiments on LaMP, this paper demonstrates that user context generally provides more substantial and reliable performance gains and further shows that the interaction between user context and model scale varies depending on the task.

Critical strengths

None

Critical weaknesses
  • The experimental design needs to be revised:

    • First, regarding the experiments that increase the amount of user context, rather than simply increasing the value of k in BM25, the authors should first identify the oracle user contexts for each task and then gradually increase the number of oracle user contexts provided. Since the experiments were not conducted under such an oracle setting, and potentially irrelevant user contexts could be included, there is a fundamental flaw in the paper's assumption that "providing more user context" necessarily means providing more relevant information.
    • Furthermore, although the paper claims to investigate the effects of test-time compute scaling, the experiments simply compare different model tiers within the same model family. Changing the model itself does not constitute test-time compute scaling. To properly investigate test-time compute scaling, the authors should have varied parameters such as reasoning effort within the same model.
  • Use of a limited and outdated benchmark:

    • Although many recent personalization benchmarks have been introduced across diverse tasks with more realistic settings, this paper evaluates its approach only on LaMP, which has outdated and limited characteristics for evaluating the personalization capabilities of modern advanced systems (e.g., web agents, search agents, etc.). Of course, given the nature of this work, it may not be necessary to conduct experiments across a wide range of benchmarks. However, I believe that including experiments on at least some recent personalization benchmarks would further strengthen the value of this work.
Questions for authors

Refer to weakness

Reviewer_bXx6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how retrieved user history interacts with model tier and repeated sampling on four LaMP tasks. It varies history size for the same query, finding that history and model tier substitute for one another on rating prediction but complement one another on movie tagging. It also studies saturation, a shared error floor, sampling benefits, and a rule for deciding when to upgrade the model.

Critical strengths
  1. The within-query comparison provides a useful insight about personalization. Varying history for the same query avoids confounding history volume with differences between users. The task-dependent interaction explains when more history may replace a model upgrade and when the two may help together.

  2. The released records allow meaningful checks, and the main interactions survive them. After removing contaminated LaMP-3 queries, Haiku with eight history items still outperforms Opus without history: MAE difference −0.112, with a paired 95% interval of [−0.191, −0.034]. The LaMP-2 complementarity also survives corrected consolidation. This supports the core contribution despite the issues below.

Critical weaknesses
  1. Data problems affect the reported results. Ten of 99 LaMP-3 query reviews also appear in the user's history with their correct ratings. Removing them preserves the core comparison, but affected estimates and fits need updating. Separately, consolidation loses sampling records and Table 14 mixes experimental conditions. On corrected records with explicit, gold-independent tie rules, the confidence intervals for high-history voting gains include zero. The sampling conclusion therefore needs reanalysis.

  2. The broader statistical claims need to be narrowed. The shared-floor analysis treats dependent cross-validation folds as independent; a standard dependence correction removes the reported significance. The “4.5×” comparison also contrasts cumulative history additions with mostly single-step model upgrades, which lack a common unit. These analyses do not establish a shared irreducible error floor or a general exchange rate between history and compute.

Questions for authors
  1. Correction of existing results: After removing query-in-profile targets, which headline, saturation, and upgrade-policy conclusions remain? Please update the affected analyses and check the other tasks for the same form of contamination.

  2. Correction of existing results: Can you regenerate Table 11, Figure 3, and Table 14 with corrected experiment identities and an explicit tie rule? Do the high-history sampling gains remain statistically supported?

  3. Validation of the interpretation: Does the shared-floor result hold when uncertainty accounts for repeated-fold dependence? Please also correct the contrast flag, report adjacent and cumulative history contrasts separately, and remove the unsupported “4.5×” comparison.

  4. Clarification: What effective serving settings produced Grid A? Temperature is unrecorded and the internal wrapper is unavailable, so the greedy description cannot be verified. Variation across repeated outputs does not establish the actual temperature. Please substantiate or revise that description.

A corrected analysis that preserves the main interactions and either supports or withdraws the broader mechanism claims would move my recommendation to weak acceptance.

Reviewer_85VqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between user history and inference compute in LLM personalization. The authors study whether increasing inference compute can compensate for limited user information. They construct a two-dimensional experimental framework that varies the number of retrieved user history items and uses model tiers and sampling budgets to represent inference compute. The authors also find that user history and model scale can have substitutive or complementary effects depending on the task. They further examine answer diversity under repeated sampling and propose a simple compute allocation rule based on the amount of available user history.

Critical strengths
  1. The paper jointly studies user history and inference compute, showing task-dependent interactions between them.
  2. Within-user comparisons across history budgets control for user differences. Additional experiments cover model tiers, sampling, prompting, and retrieval methods.
  3. The analysis examines retrieval saturation, model-tier gaps, and sampling diversity, and reports exceptions and limits to the allocation rule.
Critical weaknesses
  1. The shared-error-floor claim lacks sufficient statistical support. The paired t-tests use 50 overlapping cross-validation folds without explaining dependence correction. Better prediction by a shared-floor model may reflect fewer parameters rather than a common irreducible error floor. Retrieval budgets stop at 32 items.
  2. Counts of significant contrasts do not directly compare user signal with compute. The reported 39/71 versus 10/82 counts depend on effect sizes, variance, and tested ranges. Contrasts share observations and baselines, with no multiple-testing correction.
  3. Model tiers conflate inference compute with training and other model differences. Sampling uses only 3–4 responses per instance, insufficient to establish diversity as the cause of small gains. Coverage is limited to one model family, four LaMP tasks, roughly 100 instances per main condition, and 39 for sampling.
Questions for authors
  1. How did you account for dependence between cross-validation folds? What distinguishes a shared error floor from a benefit of fitting fewer parameters?
  2. Would direct comparisons of effect sizes and uncertainty support the conclusion drawn from significant-contrast counts, given unequal ranges and shared observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.