- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how to organize zero-shot LLM inference for product-level conversion ranking from browsing histories. It crosses aggregation before versus after inference with isolated versus cross-product prompting, yielding four procedures. The experiments demonstrate comparative full-history summaries attain the highest Spearman correlation among the four procedures in 12 of 15 settings. The author propose a Bayesian framework to interprets the results through prior refinement and context noise.
Critical strengths
- The paper tackles a practically relevant problem, offering consistent insights through a combination of empirical evaluation and theoretical analysis.
- The two-axis design clearly organizes the choices of evidence representation and comparative context, making the experimental question and comparisons easy to follow.
- The matched 80-session controls and standardization/repetition ablations help distinguish the effects of evidence depth, output normalization, and repeated judgments. These controls strengthen the evidence for the benefit of comparison itself.
Critical weaknesses
- The confidence intervals are computed from fixed predictions and do not capture changes caused by resampling sessions, regrouping products, or rerunning the models. The stability of the reported differences across independent runs therefore remains unclear.
- With the same 80 sessions, Batch summary-80 has similar average correlations to Batch session and trails isolated Session on Coveo. The 160-session extension covers only one model per dataset, leaving limited evidence about how the relative accuracy and cost change at larger session budgets.
- The evidence for prior refinement remains indirect. Table 31 relates comparison gains to the informativeness of a few context types, but does not directly test the mechanism by varying the comparison products while keeping the target input and prompt format fixed.
Questions for authors
- Could the authors report how stable the method differences are across independent session samples, product groupings, or fresh model calls?
- How does the accuracy–cost tradeoff change at larger session budgets, and does the ordering observed in the 160-session experiments hold across more models?
4 comments
Response to Reviewer K4yf (2/2)
W1. Intervals that condition on fixed predictions: You are right: our intervals treat each method's predictions as fixed. They capture which products were sampled, but not which sessions were drawn, how products were grouped, or which answer a model happened to return. We have now measured these three sources directly by rerunning the methods (details in our answer to Q1).
W2. Matched evidence and the 160-session extension: We agree with this reading of the matched-evidence result. Given the same 80 sessions, Batch summary-80 ranks about as well as Batch session on average (+0.002, −0.019, +0.015), and on Coveo it trails isolated Session. We do not claim that summaries extract more from the same evidence. Our claims are different:
- Comparison helps summaries far more than sessions, also at matched evidence. Comparing 80-session summaries adds 0.15 to 0.21, while comparing single sessions adds −0.04 to +0.06.
- A summary lets one comparative judgment draw on a product's whole history at fixed cost. At matched evidence, Batch summary-80 has mean correlations within 0.02 of Batch session at 12–18% of its inference cost; with full histories, Batch summary leads. For comparison, Session uses 80,000 calls per model and dataset, against 125 for Batch summary.
To address the narrow coverage of the 160-session extension, we extended it to three models per dataset (details in our answer to Q2).
W3. A direct test of the mechanism:
You asked us to vary the comparison products while keeping the target’s input and the prompt format fixed. We ran this experiment, together with a second test of what comparison supplies. To keep rebuttal inference costs manageable, we ran both experiments on four LLMs (GPT, Sonnet, Gemini, GLM), covering all 1,000 products per dataset and the purchase outcome.
1. Task variation: If the information comes from variation across products, the gain should weaken when products in a call are alike in what matters for the outcome. We grouped products into calls of 40 with similar predicted purchase rates, leaving each product’s input and the prompt format unchanged. These predictions came from a model trained only on products outside the evaluation pool. We compare unstandardized predictions in both conditions.
- Random companions beat similar ones by 0.25 on Coveo and 0.30 on YooChoose, significant for every model.
- Similar companions gave no detectable gain over predicting each product alone on Coveo (0.08 vs. 0.11), and reduced accuracy on YooChoose (0.16 vs. 0.30).
2. Prior refinement: If comparison helps because the other products tell the model what is typical in this store, giving that information directly should help even without other products in the prompt.
- We predicted each product on its own and added a short description of the store-wide distribution of each input statistic, without outcome labels. Rank correlation rose from 0.11 to 0.31 on Coveo and from 0.30 to 0.47 on YooChoose. Averaging predictions over five comparative calls, without standardization, gives similar correlations (0.33 and 0.47; differences −0.02 and 0.00, not significant).
Information about what is typical in the store can therefore recover much of the gain from comparative prompting, even when each product is predicted on its own. This supports our view that other products help by providing context for interpreting the target’s summary.
Together, these results support the view that information about the store helps the model interpret a product’s summary, and that the choice of companions matters. They are consistent with prior refinement, but do not distinguish Bayesian updating from feature normalization or other ways of using context. We will add these experiments to the appendix, with add-to-cart results and the full conditions.
Dear Reviewer K4yf,
Thank you very much for your comments and feedback. As the discussion period ends tomorrow, we were wondering whether our responses have addressed your concerns and whether we can provide any further information or clarification that would help your assessment.
Best, The Authors
Thanks for your response. I will mantain my original positive score.
Response to Reviewer K4yf (1/2)
Q1. Stability across session samples, groupings and fresh calls.
We reran the methods with each source of variation replaced by an independent draw:
Groupings and fresh calls cover GPT, Sonnet, Gemini and GLM; session samples cover GPT, Gemini and GLM, on the products with at least 160 sessions (743 on Coveo, all 1,000 on YooChoose).
The central conclusions were stable across the reruns we tested. We will add these replicates to the appendix.
Q2. Larger session budgets. Given the cost of session-level inference, we extended the larger-budget experiments in this rebuttal to three models per dataset: GPT, Gemini, and GLM. Coveo uses 743 products, with 147–160 distinct sessions judged per product (155 on average); YooChoose uses 160 for all 1,000 products.
So Batch summary remains ahead on Coveo. On YooChoose, Batch session reaches similar correlations for two models, at 32 times the number of calls. We will add these results to Appendix N and state the tradeoff in the main text.
* 95% interval excludes 0.