acceptodds
Under review as a conference paper at ICLR 2027

Organizing LLM Inference for Aggregate Predictions from Behavioral Data

Abstract

When using a large language model (LLM) to simulate a user population, how should we combine individual behavioral histories to predict the fraction who take a particular action? We study this question in e-commerce, retrospectively predicting product-level conversion rates from browsing sessions without adapting the model to the task. We vary two design choices, each involving a bias–variance tradeoff. Summarizing a product's sessions before inference is economical but can discard predictive detail, adding bias, whereas judging sessions one at a time and averaging afterward keeps the detail but requires many calls to reduce variance. Evaluating products together in one prompt can refine the model's prior about what is typical in the setting, reducing bias, but also makes each judgment depend on which other products are sampled to share the prompt, adding noise. We evaluate both choices across five LLMs and two datasets. Our main finding is that judging individual sessions outperforms judging summaries in isolation, but once both are compared within a prompt, summaries lead: comparative summaries achieve the highest Spearman correlation with observed rates in 12 of 15 settings, with no alternative significantly better. Comparison thus helps summaries far more than session judgments, overcoming the advantage of behavioral detail. We interpret this disparity through a Bayesian model of the second tradeoff, which suggests that comparison helps more when the inputs in a prompt differ more across products, consistent with what we observe across four kinds of comparative context. Comparative summaries also require a small fraction of the inference cost of the session methods, and with the best model for each outcome they attain 66–89% of the correlation of the best supervised benchmark trained on thousands of labeled products.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Aggregate Prediction, Behavioral Data, Comparative Prompting, In-Context Learning, Bias–Variance Tradeoff, Zero-Shot Inference, E-Commerce

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_K4yfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to organize zero-shot LLM inference for product-level conversion ranking from browsing histories. It crosses aggregation before versus after inference with isolated versus cross-product prompting, yielding four procedures. The experiments demonstrate comparative full-history summaries attain the highest Spearman correlation among the four procedures in 12 of 15 settings. The author propose a Bayesian framework to interprets the results through prior refinement and context noise.

Critical strengths
  1. The paper tackles a practically relevant problem, offering consistent insights through a combination of empirical evaluation and theoretical analysis.
  2. The two-axis design clearly organizes the choices of evidence representation and comparative context, making the experimental question and comparisons easy to follow.
  3. The matched 80-session controls and standardization/repetition ablations help distinguish the effects of evidence depth, output normalization, and repeated judgments. These controls strengthen the evidence for the benefit of comparison itself.
Critical weaknesses
  1. The confidence intervals are computed from fixed predictions and do not capture changes caused by resampling sessions, regrouping products, or rerunning the models. The stability of the reported differences across independent runs therefore remains unclear.
  2. With the same 80 sessions, Batch summary-80 has similar average correlations to Batch session and trails isolated Session on Coveo. The 160-session extension covers only one model per dataset, leaving limited evidence about how the relative accuracy and cost change at larger session budgets.
  3. The evidence for prior refinement remains indirect. Table 31 relates comparison gains to the informativeness of a few context types, but does not directly test the mechanism by varying the comparison products while keeping the target input and prompt format fixed.
Questions for authors
  1. Could the authors report how stable the method differences are across independent session samples, product groupings, or fresh model calls?
  2. How does the accuracy–cost tradeoff change at larger session budgets, and does the ordering observed in the 160-session experiments hold across more models?
4 comments
AuthorsOpenReview ↗

Response to Reviewer K4yf (1/2)

Q1. Stability across session samples, groupings and fresh calls.

We reran the methods with each source of variation replaced by an independent draw:

  • Fresh calls: Summary and Batch summary rerun with identical prompts.
  • New groupings: Batch summary rerun three times, each with five new random groupings.
  • Independent session samples: Session and Batch session rerun on a new, independently drawn set of 80 sessions per product, and Batch session also on a third set.
  • Everything at once: a replicate in which every method uses fresh calls, new groupings and new sessions.

Groupings and fresh calls cover GPT, Sonnet, Gemini and GLM; session samples cover GPT, Gemini and GLM, on the products with at least 160 sessions (743 on Coveo, all 1,000 on YooChoose).

Mean over GPT, Gemini, GLM: original → all-new replicateCoveo cartCoveo purchaseYooChoose
Batch summary − Session0.13 → 0.090.13 → 0.120.13 → 0.16
Batch summary − Batch session0.16 → 0.110.16 → 0.120.07 → 0.06
Difference-in-differences0.32 → 0.290.32 → 0.310.18 → 0.19
  • Across all replicates, these differences vary with a standard deviation of 0.01–0.03, a third to a half of the bootstrap interval's half-width.
  • For every model, no replicate reverses a significant difference involving Batch summary or the difference-in-differences.
  • What does move is the small difference between Batch session and Session on Coveo, which starts close to zero (GPT: 0.00 originally, +0.05 on a new session sample).

The central conclusions were stable across the reruns we tested. We will add these replicates to the appendix.


Q2. Larger session budgets. Given the cost of session-level inference, we extended the larger-budget experiments in this rebuttal to three models per dataset: GPT, Gemini, and GLM. Coveo uses 743 products, with 147–160 distinct sessions judged per product (155 on average); YooChoose uses 160 for all 1,000 products.

Batch summary − Batch session, larger budgetGPTGeminiGLM
Coveo cart+0.10*+0.10*+0.12*
Coveo purchase+0.10*+0.11*+0.04
YooChoose−0.01+0.10*−0.03
  • The larger session budget raises Batch session's accuracy for every model, by 0.03 to 0.09.
  • On Coveo, Batch summary still leads in 5 of 6 comparisons; the sixth is not significant.
  • On YooChoose, where single sessions carry more information about the product (Appendix Q), Batch session has similar correlations to Batch summary for GPT and GLM (differences −0.01 and −0.03, neither significant); Gemini still trails.
  • Cost is what shapes the tradeoff: the larger-budget Batch session needs 4,000 calls per model on YooChoose and 2,880 on Coveo. Batch summary uses 125 calls for the original 1,000-product pool, with full-history summaries.

So Batch summary remains ahead on Coveo. On YooChoose, Batch session reaches similar correlations for two models, at 32 times the number of calls. We will add these results to Appendix N and state the tradeoff in the main text.

* 95% interval excludes 0.

AuthorsOpenReview ↗

Response to Reviewer K4yf (2/2)

W1. Intervals that condition on fixed predictions: You are right: our intervals treat each method's predictions as fixed. They capture which products were sampled, but not which sessions were drawn, how products were grouped, or which answer a model happened to return. We have now measured these three sources directly by rerunning the methods (details in our answer to Q1).


W2. Matched evidence and the 160-session extension: We agree with this reading of the matched-evidence result. Given the same 80 sessions, Batch summary-80 ranks about as well as Batch session on average (+0.002, −0.019, +0.015), and on Coveo it trails isolated Session. We do not claim that summaries extract more from the same evidence. Our claims are different:

  1. Comparison helps summaries far more than sessions, also at matched evidence. Comparing 80-session summaries adds 0.15 to 0.21, while comparing single sessions adds −0.04 to +0.06.
  2. A summary lets one comparative judgment draw on a product's whole history at fixed cost. At matched evidence, Batch summary-80 has mean correlations within 0.02 of Batch session at 12–18% of its inference cost; with full histories, Batch summary leads. For comparison, Session uses 80,000 calls per model and dataset, against 125 for Batch summary.

To address the narrow coverage of the 160-session extension, we extended it to three models per dataset (details in our answer to Q2).


W3. A direct test of the mechanism:

You asked us to vary the comparison products while keeping the target’s input and the prompt format fixed. We ran this experiment, together with a second test of what comparison supplies. To keep rebuttal inference costs manageable, we ran both experiments on four LLMs (GPT, Sonnet, Gemini, GLM), covering all 1,000 products per dataset and the purchase outcome.

1. Task variation: If the information comes from variation across products, the gain should weaken when products in a call are alike in what matters for the outcome. We grouped products into calls of 40 with similar predicted purchase rates, leaving each product’s input and the prompt format unchanged. These predictions came from a model trained only on products outside the evaluation pool. We compare unstandardized predictions in both conditions.

  • Random companions beat similar ones by 0.25 on Coveo and 0.30 on YooChoose, significant for every model.
  • Similar companions gave no detectable gain over predicting each product alone on Coveo (0.08 vs. 0.11), and reduced accuracy on YooChoose (0.16 vs. 0.30).

2. Prior refinement: If comparison helps because the other products tell the model what is typical in this store, giving that information directly should help even without other products in the prompt.

  • We predicted each product on its own and added a short description of the store-wide distribution of each input statistic, without outcome labels. Rank correlation rose from 0.11 to 0.31 on Coveo and from 0.30 to 0.47 on YooChoose. Averaging predictions over five comparative calls, without standardization, gives similar correlations (0.33 and 0.47; differences −0.02 and 0.00, not significant).

Information about what is typical in the store can therefore recover much of the gain from comparative prompting, even when each product is predicted on its own. This supports our view that other products help by providing context for interpreting the target’s summary.

Together, these results support the view that information about the store helps the model interpret a product’s summary, and that the choice of companions matters. They are consistent with prior refinement, but do not distinguish Bayesian updating from feature normalization or other ways of using context. We will add these experiments to the appendix, with add-to-cart results and the full conditions.

Dear Reviewer K4yf,

Thank you very much for your comments and feedback. As the discussion period ends tomorrow, we were wondering whether our responses have addressed your concerns and whether we can provide any further information or clarification that would help your assessment.

Best, The Authors

Thanks for your response. I will mantain my original positive score.

Reviewer_XLubOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how large language models should combine individual behavioral histories to predict aggregate outcomes, focusing on product-level conversion rates in e-commerce. Specifically, the authors examine two fundamental design choices: whether behavioral observations should be aggregated before or after LLM inference, and whether predictions should be generated independently or through comparative prompting. The paper studies four inference strategies:

  1. Summary: Aggregate a product's behavioral sessions into descriptive statistics and ask the LLM to predict the product-level conversion rate independently.
  2. Session: Predict conversion probabilities for individual sessions and aggregate the predictions afterward.
  3. Batch Summary: Present summaries of multiple products in a shared prompt, standardize predictions within each batch, and average across multiple groupings.
  4. Batch Session: Present individual sessions from different products in shared prompts and aggregate the resulting predictions. The authors evaluate these strategies using five LLMs across two public e-commerce datasets, Coveo and YooChoose, covering three prediction tasks: add-to-cart rates and purchase rates on Coveo, and purchase rates on YooChoose. The main findings are that session-level inference generally outperforms isolated summary-based inference, but comparative prompting disproportionately benefits summaries. Batch Summary achieves the highest Spearman rank correlation in 12 of 15 model–outcome settings, with no alternative significantly outperforming it. The authors further demonstrate that Batch Summary requires substantially fewer inference calls than session-level approaches. They propose a Bayesian interpretation in which comparative context refines the model's prior over the underlying environment, potentially reducing prediction bias while introducing context-dependent variance. Overall, the paper contributes a systematic empirical characterization of the interaction between evidence aggregation, comparative prompting, and inference efficiency in zero-shot aggregate prediction.
Critical strengths

S1. Clear and practically important research question. The paper investigates a fundamental design choice in LLM-based aggregate prediction: whether to summarize observations before inference or aggregate individual predictions afterward. This question is relevant beyond e-commerce, potentially extending to population-level prediction, market research, survey analysis, and other applications involving collections of individual observations. The additional consideration of comparative prompting makes the study particularly useful, as it demonstrates that the effectiveness of an aggregation strategy cannot be evaluated independently of the inference context. S2. Extensive statistical analysis and robustness checks. The paper reports paired bootstrap confidence intervals and provides detailed results for individual models and prediction tasks. The authors also examine alternative ranking conventions, Pearson correlation, unstandardized batch predictions, different session-sampling depths, and the effect of restoring traffic-count features. These analyses strengthen the credibility of the main empirical findings. The discussion of statistical limitations, including unadjusted multiple comparisons and dependence between products, is also appropriately transparent. S3. Strong practical contribution through inference-cost analysis. The inference-efficiency results are particularly valuable. For 1,000 products, isolated Session inference requires 80,000 model calls, Batch Session requires 2,000 calls, and Batch Summary requires only 125 calls. Despite this substantial reduction in calls, Batch Summary generally achieves better ranking performance. The paper also reports actual token usage and monetary inference costs across different models, making the findings directly relevant to practical LLM deployment.

Critical weaknesses

W1. The methodological novelty is relatively limited. The paper primarily studies combinations of established techniques: statistical summarization, independent prediction, comparative prompting, batching, standardization, and averaging. Although the systematic comparison yields interesting findings, the proposed methods themselves do not introduce a fundamentally new model architecture, learning algorithm, or inference mechanism. The main contribution is therefore empirical rather than methodological. The authors should more explicitly distinguish the novel scientific insight from existing work on batch prompting, contextual calibration, and listwise ranking. In particular, it would be helpful to clarify what new understanding the paper provides beyond the observation that comparative context improves ranking and that summary-based inference is computationally efficient. W2. The Bayesian prior-refinement explanation is not directly validated. Section 5.2 proposes that comparative prompting improves predictions by refining the LLM's implicit prior over the underlying environment. This is an interesting interpretation, but the empirical evidence is largely correlational. The experiments show that comparative gains increase with the informativeness of product-level context, but they do not directly establish that the LLM performs Bayesian updating or that prior refinement is the mechanism responsible for the improvements. Alternative explanations are possible. For example, comparative prompts may encourage relative ranking, normalize feature scales, or reduce model-specific response biases without meaningfully changing an implicit environmental prior. The decomposition in Equation 4 is mathematically valid as a conditional mean-plus-residual decomposition, but it does not by itself establish the proposed cognitive or computational mechanism. A stronger experimental test distinguishing prior refinement from alternative explanations would substantially strengthen the theoretical contribution.

Questions for authors

Q1 Calibration and absolute prediction accuracy The paper focuses on ranking quality, but many motivating applications require accurate estimates of conversion probabilities. How well calibrated are the raw predictions from the four methods? Could the authors report absolute prediction errors or calibration metrics, and investigate whether comparative prompting can improve ranking without sacrificing probability calibration? Q2. Generalization beyond e-commerce The paper's central claim concerns aggregate prediction from collections of behavioral observations, but both datasets are drawn from e-commerce. Could the authors discuss or experimentally investigate whether the same findings hold in other domains, such as survey response prediction, healthcare utilization, or population-level decision modeling? In particular, does comparative prompting remain beneficial when the natural aggregation units are not products? Q3. Practical comparison with lightweight supervised learning The supervised baselines achieve stronger results with relatively modest amounts of labeled data. Could the authors provide guidance on when zero-shot comparative LLM inference is preferable to training a lightweight supervised model? For example, what assumptions about label availability, distribution shift, or task adaptation would justify the additional LLM inference cost?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XLub (1/2)

Q1. Calibration and absolute error. We designed the evaluation around ranking because zero-shot procedures are not expected to know a store's base rates. But the raw outputs are probabilities, so we compared their mean bias and absolute error directly. We report purchase; add-to-cart is similar, and we will add it to the appendix.

Purchase: mean prediction ÷ observed rate (range over 5 models)CoveoYooChoose
Summary1.6–4.40.7–1.6
Session4.4–8.91.4–3.4
Batch session (raw outputs)4.2–9.21.1–3.7
Batch summary (raw outputs)3.0–7.61.1–3.5

No method is consistently well calibrated in level. All four overpredict purchases on Coveo, where purchases are rare (0.7% of sessions).

Comparative summaries have lower absolute error and mean bias than the session methods in most comparisons. Batch summary has a lower mean absolute error than Session in 9 of 10 model–dataset pairs, and lower than Batch session in all 10. Its mean predictions are also closer to the observed rate.

Relative to isolated summaries, comparison raises predicted levels. Summary has lower absolute error in 8 of 10 pairs, so ranking improves at some cost in level.

These results describe mean bias and absolute error; they do not establish calibration across the full range of predicted probabilities.


Q2. Generalization beyond e-commerce.

We agree that both datasets are e-commerce logs, and we do not claim the findings hold in other domains without testing them there; we will state this limitation in the revision. E-commerce is, however, one of the main application domains of LLM-based customer simulation in its own right, from shopping benchmarks to demand simulators and simulated A/B tests [Lu et al., 2026; Wang et al., 2026; Zhao et al., 2026, arXiv:2610.30932; Huang et al., 2026; Li et al., 2026]. Findings that hold across two independent e-commerce datasets are therefore directly relevant to that work.

We also chose e-commerce because its multi-product setting makes it possible to compare procedures robustly. The zero-shot procedures we study are not expected to be calibrated, but they may use the evidence well enough to produce predictions that can be calibrated downstream. A single population-level prediction gives one error, which mixes a miscalibrated overall level with how well the procedure uses the evidence, and offers no replication for comparing procedures. E-commerce instead provides many products, each with its own shoppers and observed outcome. We can therefore measure the rank correlation between predicted and observed rates across products and compare procedures on the same products with paired intervals.

The procedures themselves do not depend on the units being products: a unit is any group whose rate is predicted from its members' records. Testing them in another domain requires the same structure: many units, each with its own population and observed outcome. Courses with learner activity logs, or hospitals with patient records, would be natural settings to validate our findings further.


Q3. When is zero-shot comparative inference preferable to a lightweight supervised model?

For the retrospective task we evaluate, we would favor a lightweight supervised model when enough representative labeled products are available. The random forest reaches the best LLM method’s correlation with 15–25 labeled products for Coveo add-to-cart, 200–400 for Coveo purchase, and 100–200 for YooChoose purchase (Appendix O). With fewer labels, or none, zero-shot comparative inference becomes worth considering.

The broader case concerns the relevance of the available labels. Predicting responses to new prices or promotions is a goal of our research agenda, and historical intervention data may be unavailable or outdated. This involves the same design choices, but our retrospective experiments do not establish that LLMs predict interventions or distribution shifts reliably. Whether the findings carry over remains to be tested.

The additional inference cost is most defensible when obtaining relevant labels is expensive, slow, or infeasible. Comparative summaries keep that cost modest: 1,000 products require 125 calls per model and dataset, compared with 80,000 for isolated Session, and cost a few dollars at GPT-5.6 list prices (Appendix P). A trained lightweight model is cheaper to run. The practical question is whether suitable training data are available, or can be obtained in time.

AuthorsOpenReview ↗

Response to Reviewer XLub (2/2)

W1. On methodological novelty: To our knowledge, prior work on customer simulation has not (a) systematically compared aggregating observed behavioral records before LLM inference with aggregating predictions afterward or (b) studied how comparative prompting interacts with this choice. That is, the following question has not been answered: Can an LLM predict effectively from summary statistics, avoiding the inference cost of judging many individual sessions?

Summary-based inference is clearly cheaper, but that matters only if it does not lose too much accuracy by discarding behavioral detail. In isolation, summaries do lose accuracy: on the same 80 sessions, Session beats Summary-80 by 0.13–0.22. The surprising fact is not that comparative context improves ranking; it is that it improves ranking so much more for the summary method than the session method, to the extent that it overcomes the loss of behavioral detail.

Relative to the three literatures you mention:

  • Batch prompting packs several queries into one call, mainly for efficiency. We find that the accuracy effect of the same batching depends on the input representation: it helps summaries in all 15 settings, but sessions in only 3.
  • Contextual calibration addresses output bias. Our gains do not require within-call standardization: a single unstandardized comparative call already improves summaries in all 15 settings.
  • Listwise ranking asks the model to order items. Our new experiments show that much of the comparative gain can be recovered by predicting each product on its own, with store-level feature distributions as context (see below).

W2. Distinguishing prior refinement from the alternatives: Our Bayesian perspective proposes a hypothesis for when comparison should help. We ran two new sets of experiments to test this account against alternative explanations. To keep rebuttal inference costs manageable, we ran both experiments on four LLMs (GPT, Sonnet, Gemini, GLM), covering all 1,000 products per dataset and the purchase outcome.

1. Prior refinement: If comparison helps because the other products tell the model what is typical in this store, giving that information directly should help even without other products in the prompt.

  • We predicted each product on its own and added a short description of the store-wide distribution of each input statistic, without outcome labels. Rank correlation rose from 0.11 to 0.31 on Coveo and from 0.30 to 0.47 on YooChoose. Averaging predictions over five comparative calls, without standardization, gives similar correlations (0.33 and 0.47; differences −0.02 and 0.00, not significant).

Information about what is typical in the store can therefore recover much of the gain from comparative prompting, even when each product is predicted on its own. This supports our view that other products help by providing context for interpreting the target’s summary.

2. Task variation: If the information comes from variation across products, the gain should weaken when products in a call are alike in what matters for the outcome. We grouped products into calls of 40 with similar predicted purchase rates, leaving each product’s input and the prompt format unchanged. These predictions came from a model trained only on products outside the evaluation pool. We compare unstandardized predictions in both conditions.

  • Random companions beat similar ones by 0.25 on Coveo and 0.30 on YooChoose, significant for every model.
  • Similar companions gave no detectable gain over predicting each product alone on Coveo (0.08 vs. 0.11), and reduced accuracy on YooChoose (0.16 vs. 0.30).

Together, these results support the view that information about the store helps the model interpret a product’s summary, and that the choice of companions matters. They are consistent with prior refinement, but do not distinguish Bayesian updating from feature normalization or other ways of using context. We will add these experiments to the appendix, with add-to-cart results and the full conditions.

Dear Reviewer XLub,

Thank you very much for your comments and feedback. As the discussion period ends tomorrow, we were wondering whether our responses have addressed your concerns and whether we can provide any further information or clarification that would help your assessment.

Best, The Authors

Reviewer_5RSeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores how to use LLM to use browsing sessions to predict product-level conversion rates without any training on the task (zero-shot). Two design was explored: 1) when to aggregate, whether use aggregated statistics or session by session inspection, 2) point-wise or comparative prompts, either one product per prompt or multiple (k=40) product in one prompt. The paper evaluated five LLMs, and finds that: 1) isolation and per session beat aggregation, 2) comparison helps summary but mix on sessions, 3) comparison summary rank best.

Critical strengths
  1. Well designed experiments: summaries built from the same 80 sessions, separate the effect of input format from at the amount of history. Design choices were fixed on a separate product pool before testing. The experiments cover 5 models and 3 outcomes per each model.
  2. A practical results: more behavioral detail isn't better, summaries compared side-by-side beat session-level judgements. This is directly useful to anyone using LLM as population simulators. It also extends known results on list-wise and batch prompting into aggregate prediction.
Critical weaknesses
  1. The paper does not prove that using LLM is better or necessary.
  2. The design was only tuned on GPT-5.6 and did not change when apply to four other models.
  3. THe bayesian story was not proven quantitatively.
Questions for authors
  1. How does the batch summary compare with 1) averaging the standardized summary statistics or 2) the same LLM shown 10 to 20 labeled products in the prompt?
  2. Can we do ablation studies, like cutting certain events (like add-to-cart or purchase) or removing the viewing time or session length. Do we still get the same conclusion?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5RSe (1/2)

Q1. Comparison with (1) averaged standardized statistics and (2) labeled examples in the prompt.

(1) Averaged statistics. We tested a simple rule on the same inputs, without an LLM. We standardize each summary statistic across products, give it a sign, and average. We set the signs in two ways:

  • A priori, without labels. Price, number of competitors and competitor viewing time get −. Viewing time, arrival via search, search rank (better rank +), special offers and competitor price get +.
  • From labels. Each sign is that of the statistic's correlation with the outcome in the labeled training pool (8,410 Coveo / 2,530 YooChoose products).
SpearmanCoveo cartCoveo purchaseYooChoose
Batch summary (mean over 5 models)0.340.390.54
Average, a priori signs−0.090.06−0.02
Average, signs from labels0.550.440.46
  • With our a priori signs, the rule performs poorly. The intuitive sign for competitor prices is wrong in these data, because competitor prices track the product's own price level. Batch summary leads by 0.33–0.57*.
  • With labels:
    • Coveo cart: the rule leads by 0.21*; the paper already reports that a single statistic, the average price of the most-considered competing product, reaches 0.52.
    • Coveo purchase: the rule leads by 0.05, not significant.
    • YooChoose: Batch summary leads by 0.08*.

So in summary, the averaged-statistic rule (or the price-based rule we report in the paper) performs well with some supervision. One benefit of zero-shot Batch summary is that it requires no labeled examples.

(2) Labeled examples in the prompt. We added k = 10 or 20 labeled training products, with their observed rates, to every Batch summary call. We used two independent draws of examples and four models (GPT, Sonnet, Gemini, GLM).

Spearman (mean over 4 models)Coveo cartCoveo purchaseYooChoose
Zero-shot0.320.360.53
k = 100.540.440.55
k = 200.550.440.55
  • With ten examples, Batch summary is close to the label-signed rule on Coveo (0.54 vs. 0.55, 0.44 vs. 0.44) and exceeds it on YooChoose (0.55 vs. 0.46).
  • Gains are largest where zero-shot is weakest.
    • Coveo cart: +0.22*.
    • Weaker models on Coveo purchase: Sonnet +0.18*, GLM +0.12*.
    • The strongest models gain little: GPT −0.02 on Coveo purchase and +0.03* on YooChoose.
  • Twenty examples add little over ten.

Therefore, while a few labeled examples can substantially improve the comparative procedure (thanks for pointing to this comparison), our focus remains the zero-shot setting, where no labeled examples are supplied. We will add both comparisons to the appendix.


Q2. Ablations.

Add-to-cart and purchase events are already absent from every input. We mask the focal outcome fields in every prompted record and exclude the five summary columns derived from outcomes (Appendix E). The remaining concern is time. On Coveo, session length runs to the last logged event, including cart and purchase, and viewing time runs to the next event, so both can include time after the outcome (Appendix F). YooChoose sessions end at the first purchase, so the issue does not arise there.

To keep inference costs reasonable for this rebuttal, we ran a joint ablation on Coveo with four models (GPT, Sonnet, Gemini, GLM). We removed viewing time from summaries and session records and session length from session records, and reran all four methods. Session uses the paper's 80 sessions per product.

Spearman, mean over 4 models (cart / purchase)OriginalWithout time fields
Summary0.08 / 0.110.05 / 0.04
Session0.28 / 0.310.37 / 0.33
Batch summary0.32 / 0.360.36 / 0.34
Batch session0.23 / 0.260.34 / 0.32
  • The session methods improve without the time fields (Session +0.08* on cart, Batch session +0.11* on cart), so their performance does not depend on these time fields. Viewing time varies mostly within products: only 1.2% of its variance lies between products (Appendix Q). The improvement is consistent with these time fields adding noise to individual-session predictions.
  • The main interaction holds in all 8 model–outcome settings. Session beats Summary (8/8). Comparison improves summaries (8/8) and helps them more than sessions (8/8; difference-in-differences 0.19–0.47*, against 0.20–0.37* originally).
  • Batch summary's lead over the session methods narrows. Batch summary changes by +0.04 and −0.03 on average, neither significant. With the session methods improved, Batch summary’s mean correlations differ from theirs by only −0.01 to +0.02, none significant. Per model, it leads Session in 3 of 8 settings and trails in 1 (GLM, cart). It still uses 125 calls, against 2,000 for Batch session and 80,000 for Session.

We will add a full ablation study to the appendix.

* 95% paired bootstrap interval excludes 0.

AuthorsOpenReview ↗

Response to Reviewer 5RSe (2/2)

W1. The need for an LLM: The overarching goal of our research agenda is to enable LLM-based approaches in settings where relevant labeled data are scarce or difficult to obtain. The retrospective task considered here allows us to evaluate these approaches alongside supervised benchmarks and assess their relative performance. It is encouraging that the best zero-shot LLM methods in our study attain 89% (Coveo) and 85% (YooChoose) of the random forest’s Spearman correlation on the two purchase outcomes without task-specific training. The random forest requires about 200–400 labeled products for Coveo purchase and 100–200 for YooChoose purchase to match this performance.

An important potential application is predicting responses to interventions such as new prices or promotions using recent customer session data. This involves the same design choices about aggregation placement and comparative context. At the session level, one could ask: “Given this customer’s behavior and purchase outcome at the current price, would the customer have bought at price Y?” At the product level, the analogous question uses the product’s behavioral summary and current conversion rate. Relevant historical intervention data may be unavailable or reflect older behavior and market conditions. Zero-shot LLM inference offers a flexible way to condition directly on recent behavioral records and observed outcomes under the current offer. While our results do not establish accuracy under interventions, they provide a useful basis for obtaining retailer buy-in to test these methods in that setting.


W2. Design fixed on GPT-5.6. We fixed every design choice once, using GPT-5.6 on a separate Coveo discovery pool, before running any evaluation. We then applied the protocol unchanged to all five models and to YooChoose. This keeps tuning effort fixed and tests transfer without model-specific adaptation.

For this response, we also checked batch size and wording on the discovery pool for Sonnet, Gemini, GLM and Qwen. We tested 5 to 280 products per call and two paraphrases: 8 alternatives and 2 outcomes per model. Of the 64 differences from each model’s 40-product reference, 51 are not significant, and no alternative is better for every model. Gemini benefits from several alternatives, by up to 0.07, mostly on add-to-cart. We do not claim the chosen protocol is optimal for every model. We will add these checks to Appendix E.


W3. Validating the Bayesian mechanism: Our Bayesian perspective proposes a hypothesis for when comparison should help. We ran two new sets of experiments to test this account against alternative explanations. To keep rebuttal inference costs manageable, we ran both experiments on four LLMs (GPT, Sonnet, Gemini, GLM), covering all 1,000 products per dataset and the purchase outcome.

1. Prior refinement: If comparison helps because the other products tell the model what is typical in this store, giving that information directly should help even without other products in the prompt.

  • We predicted each product on its own and added a short description of the store-wide distribution of each input statistic, without outcome labels. Rank correlation rose from 0.11 to 0.31 on Coveo and from 0.30 to 0.47 on YooChoose. Averaging predictions over five comparative calls, without standardization, gives similar correlations (0.33 and 0.47; differences −0.02 and 0.00, not significant).

Information about what is typical in the store can therefore recover much of the gain from comparative prompting, even when each product is predicted on its own. This supports our view that other products help by providing context for interpreting the target’s summary.

2. Task variation: If the information comes from variation across products, the gain should weaken when products in a call are alike in what matters for the outcome. We grouped products into calls of 40 with similar predicted purchase rates, leaving each product’s input and the prompt format unchanged. These predictions came from a model trained only on products outside the evaluation pool. We compare unstandardized predictions in both conditions.

  • Random companions beat similar ones by 0.25 on Coveo and 0.30 on YooChoose, significant for every model.
  • Similar companions gave no detectable gain over predicting each product alone on Coveo (0.08 vs. 0.11), and reduced accuracy on YooChoose (0.16 vs. 0.30).

Together, these results support the view that information about the store helps the model interpret a product’s summary, and that the choice of companions matters. They are consistent with prior refinement, but do not distinguish Bayesian updating from feature normalization or other ways of using context. We will add these experiments to the appendix, with add-to-cart results and the full conditions.

Dear Reviewer 5RSe,

Thank you very much for your comments and feedback. As the discussion period ends tomorrow, we were wondering whether our responses have addressed your concerns and whether we can provide any further information or clarification that would help your assessment.

Best, The Authors

Reviewer_bi3SOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to organize LLM inference for aggregate predictions from individual behavioral records. Specifically, the authors study two design choices: (1) whether to aggregate behavioral records before LLM inference or aggregate individual predictions afterward, and (2) whether to perform inference independently or comparatively across multiple products. These choices result in four inference strategies, which are evaluated on two e-commerce datasets using five LLMs.

The main finding is that while session-level inference generally outperforms summary-based inference in isolation, comparative prompting substantially improves summary-based predictions, making comparative summaries the best-performing strategy in most experimental settings. The authors further propose a Bayesian perspective to interpret these observations through prior refinement and context-dependent noise.

Critical strengths
  1. Clear problem formulation and systematic experimental design.

  2. Comprehensive experiments and careful ablation studies.

  3. Interesting empirical observations with practical implications.

Critical weaknesses
  1. Limited novelty and insufficient generalizable insights. While the paper provides a systematic empirical investigation, its scientific contributions appear relatively limited. Both evidence aggregation and comparative prompting are established techniques, and the main contribution is the empirical observation of their interaction in a particular application. More importantly, the paper does not sufficiently establish generalizable principles beyond these experimental observations. For example, it remains unclear how the findings could guide the selection of inference strategies for unseen tasks, or how to predict when comparative prompting would be beneficial. The Bayesian interpretation provides a possible explanation for the observed results, but it remains largely conceptual. The proposed prior refinement mechanism is not directly validated, and alternative explanations, such as improved relative judgment or contextual calibration, are not sufficiently distinguished. Consequently, the theoretical analysis offers limited additional understanding beyond the empirical findings.

  2. The superiority of comparative summaries is partly attributable to access to more behavioral evidence. One of the paper's central findings is that comparative summary inference outperforms session-level inference. However, the main comparative summary method uses the complete behavioral history, whereas the session-based methods operate on a fixed sample of 80 sessions per product. The matched-evidence experiments show that when comparative summaries are constructed from the same 80 sessions, their performance becomes largely comparable to that of comparative session inference. This suggests that an important part of the advantage comes from the ability to incorporate more behavioral evidence efficiently, rather than an inherent superiority of the summary representation or inference strategy. Although the authors acknowledge this distinction and provide the relevant ablations, it weakens the significance of the main empirical finding as a general inference principle.

  3. Limited practical significance of the evaluated prediction setting. The paper motivates the problem through real-world applications such as pricing, promotion, and inventory decisions. However, the experiments focus on retrospectively reconstructing conversion rates from behavioral records collected during the same observation period. This differs substantially from predicting future outcomes or estimating responses to interventions, which are more relevant to the motivating applications. Furthermore, the supervised baselines achieve substantially better performance, and the training-size experiments indicate that relatively small numbers of labeled products can already yield performance comparable to the best LLM-based approach. Although the supervised baselines do not use exactly the same feature set, these results raise questions about the practical advantages of the proposed LLM inference strategies. Therefore, the practical scenarios in which these methods provide meaningful advantages over conventional prediction approaches remain insufficiently established.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bi3S (1/2)

W1. On Novelty: Evidence aggregation and comparative prompting are indeed established techniques. To our knowledge, however, prior work has not (a) systematically compared aggregating observed behavioral records before LLM inference with aggregating predictions afterward or (b) studied how comparative prompting interacts with this choice. That is, the following question has not been answered: Can an LLM predict effectively from summary statistics, avoiding the inference cost of judging many individual sessions?

Recent customer-simulation work predicts at the level of individuals: shopping benchmarks evaluate individual sessions [Lu et al., 2026; Wang et al., 2026; Zhao et al., 2026, arXiv:2610.30932], and demand and A/B-test simulators aggregate predictions for individual personas or agents [Huang et al., 2026; Li et al., 2026]. Our isolated-inference results confirm the value of retaining behavioral detail: on the same 80 sessions, Session exceeds Summary-80 in Spearman correlation by 0.190, 0.221, and 0.130 across the three outcomes, averaged over the five models.

Our surprising finding was that comparative context substantially changes this tradeoff. Comparison helps summaries more than sessions in 14 of 15 model–outcome settings (Appendix L). As a result, with matched evidence, Batch summary-80 achieves mean correlations within 0.02 of Batch session on every outcome (+0.002, −0.019, and +0.015), at only 12–18% of its monetary inference cost. Thus, in our experiments, comparative summaries provide a particularly favorable accuracy–cost tradeoff – a finding that we believe has significant practical value.


W1. On Generalizability: These findings replicate across two e-commerce datasets and five LLMs, supporting robustness within the evaluated e-commerce setting. We do not establish generalization beyond e-commerce or to future outcomes or responses to interventions. E-commerce is, however, a major application domain of LLM-based customer simulation in its own right, encompassing shopping benchmarks, demand simulators, and simulated A/B tests [Lu et al., 2026; Wang et al., 2026; Zhao et al., 2026; Huang et al., 2026; Li et al., 2026]. Findings that hold across two independent e-commerce datasets are therefore directly relevant to that work. We also chose e-commerce because its multi-product setting lets us compare procedures across many prediction targets, rather than assess them on a single aggregate outcome, where the sources of error may be difficult to disentangle.


W1. Validating the Bayesian mechanism: Our Bayesian perspective proposes a hypothesis for when comparison should help. We ran two new sets of experiments to test this account against alternative explanations. To keep rebuttal inference costs manageable, we ran both experiments on four LLMs (GPT, Sonnet, Gemini, GLM), covering all 1,000 products per dataset and the purchase outcome.

1. Prior refinement: If comparison helps because the other products tell the model what is typical in this store, giving that information directly should help even without other products in the prompt.

  • We predicted each product on its own and added a short description of the store-wide distribution of each input statistic, without outcome labels. Rank correlation rose from 0.11 to 0.31 on Coveo and from 0.30 to 0.47 on YooChoose. Averaging predictions over five comparative calls, without standardization, gives similar correlations (0.33 and 0.47; differences −0.02 and 0.00, not significant).

Information about what is typical in the store can therefore recover much of the gain from comparative prompting, even when each product is predicted on its own. This supports our view that other products help by providing context for interpreting the target’s summary.

2. Task variation: If the information comes from variation across products, the gain should weaken when products in a call are alike in what matters for the outcome. We grouped products into calls of 40 with similar predicted purchase rates, leaving each product’s input and the prompt format unchanged. These predictions came from a model trained only on products outside the evaluation pool. We compare unstandardized predictions in both conditions.

  • Random companions beat similar ones by 0.25 on Coveo and 0.30 on YooChoose, significant for every model.
  • Similar companions gave no detectable gain over predicting each product alone on Coveo (0.08 vs. 0.11), and reduced accuracy on YooChoose (0.16 vs. 0.30).

Together, these results support the view that information about the store helps the model interpret a product’s summary, and that the choice of companions matters. They are consistent with prior refinement, but do not distinguish Bayesian updating from feature normalization or other ways of using context. We will add these experiments to the appendix, with add-to-cart results and the full conditions.

AuthorsOpenReview ↗

Response to Reviewer bi3S (2/2)

W2. More behavioral evidence benefits Batch summary: Even at matched evidence, the fact that Batch summary-80 performs comparably to Batch session on average (+0.002, −0.019, and +0.015), at only 12–18% of its monetary inference cost, is surprising. Summarization discards behavioral detail that matters substantially in isolation, yet comparative prompting benefits summaries much more and closes this gap.

That batch summary can fold in more behavioral evidence without increasing inference cost (unlike session methods), pushing its performance further, is an additional benefit. Note that additional evidence helps substantially only in combination with comparison: full histories add almost nothing to isolated summaries (Summary − Summary-80: −0.004, 0.010, and −0.002), but improve comparative summaries by 0.108, 0.145, and 0.041. Thus, the finding goes beyond the benefit of more evidence: how inference is organized determines whether that evidence improves predictions.


W3. On the practical significance of the prediction setting: The overarching goal of our research agenda is to enable LLM-based approaches in settings where relevant labeled data are scarce or difficult to obtain. The retrospective task considered here allows us to evaluate these approaches alongside supervised benchmarks and assess their relative performance. It is encouraging that the best zero-shot LLM methods in our study attain 89% (Coveo) and 85% (YooChoose) of the random forest’s Spearman correlation on the two purchase outcomes without task-specific training. The random forest requires about 200–400 labeled products for Coveo purchase and 100–200 for YooChoose purchase to match this performance.

An important potential application is predicting responses to interventions such as new prices or promotions using recent customer session data. This involves the same design choices about aggregation placement and comparative context. At the session level, one could ask: “Given this customer’s behavior and purchase outcome at the current price, would the customer have bought at price Y?” At the product level, the analogous question uses the product’s behavioral summary and current conversion rate. Relevant historical intervention data may be unavailable or reflect older behavior and market conditions. Zero-shot LLM inference offers a flexible way to condition directly on recent behavioral records and observed outcomes under the current offer. While our results do not establish accuracy under interventions, they provide a useful basis for obtaining retailer buy-in to test these methods in that setting.

Dear Reviewer bi3S,

Thank you very much for your comments and feedback. As the discussion period ends tomorrow, we were wondering whether our responses have addressed your concerns and whether we can provide any further information or clarification that would help your assessment.

Best, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision posted

Summary of the revision

We thank the reviewers for their comments. The revised PDF adds the experiments and analyses described in our responses. Most new material is in Appendices R–V, which the appendix guide (Table 1) lists with the sections they support. Our responses below cite appendices by their letters in the original submission; in the revision, Appendices N, O, P and Q of the submission are Appendices M, N, O and P, and Appendix M of the submission (session depth curves) is Appendix W.

New experiments and analyses

  1. Direct tests of the mechanism (5RSe W3, XLub W2, K4yf W3; Section 6 and Appendix R).
    • Similar companions. With the comparative summaries and four models, calls of 40 products with similar predicted purchase rates leave about a fifth of the outcome-relevant variation within calls. Rank correlation is 0.25–0.30 lower than with random companions, significantly for every model and outcome, and is no higher than with isolated summaries (Section 6, item (1)).
    • Store-wide distributions instead of comparison. Each product is predicted on its own, after the 10th–90th percentiles of each summary statistic across the store's products. With four models, rank correlation rises from 0.08, 0.11 and 0.30 (isolated summaries) to 0.27, 0.31 and 0.47. This is level with five averaged comparative calls, at a similar cost, and has no context noise. Standardizing within calls, which needs several products per call, adds a further 0.05–0.06 in Batch summary.
    • Store-wide distributions with session records (new since our responses). With GPT-5.6 and 20 sessions per product, the gain for single-session judgments is 0.132 on YooChoose, above the 0.106 from comparing sessions, and 0.059 and 0.020 on Coveo, where comparing sessions gives no gain. An analysis of these judgments, with no further calls, is consistent with the task-variation account in Section 6: behavioral fields vary almost entirely within products, and the product-level part of the change follows the product’s price, which varies only across products (Appendix R, Table 38).
  2. Stability across reruns (K4yf W1, Q1; Section 3, Appendix S). Fresh calls, new groupings and independent session samples move the main differences by a standard deviation of 0.01–0.03. No significant difference involving Batch summary, and no difference-in-differences, changes sign.
  3. Larger session samples for three models (K4yf W2, Q2; Section 5.1, Appendix M). With about 160 sessions per product, Batch summary remains significantly higher in five of six comparisons on Coveo. On YooChoose, Batch session is level with it for GPT-5.6 and GLM5.3, at 32 times the number of calls.
  4. Removing the time fields (5RSe Q2; Section 3, Appendix T). Without viewing time and session length on Coveo, the session methods rank better. The interaction between representation and comparison holds in all eight model–outcome settings.
  5. Simple references on the same inputs (5RSe Q1; Section 5.1, Appendix U). An average of the standardized summary statistics without an LLM, with signs set a priori or from labels. Batch summary with 10 or 20 labeled products in each call.
  6. Levels and absolute error of the raw predictions (XLub Q1; Section 1, Appendix V). No method is calibrated in level. The raw Batch summary outputs have lower absolute error than Session in 14 of 15 model–outcome pairs and than Batch session in all 15.
  7. Design choices for the other four models (5RSe W2; Section 4, Appendix E). The batch-size and wording comparisons on the discovery pool, repeated for Qwen3.8, Gemini3.8, GLM5.3 and Sonnet5. 51 of 64 differences are not significant, and no alternative is better for every model.

Other changes

The revision also includes editorial changes made after the submission deadline. The results remain unchanged.

  • The Bayesian perspective is now a separate section (Section 6), and the conditional independence of the context is stated as an assumption.
  • The data section names the masked outcome fields and states what the intervals do not capture.
  • The introduction and conclusion are reworded.
  • To make room for the new results, three items were moved to the appendix without change: the same-product batching result (Appendix J), the Batch session depth figure (Appendix W), and the discussion of model-specific bias (Appendix P).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.