- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a benchmark for in-context experiential learning, in which a recommender agent serves the same simulated customer over several shopping episodes. It extends the chat-only BELA setup with a live web store. A persona agent (Claude Opus-4.7 with browser-use) answers questions and browses the catalog, which produces click logs. In each turn the recommender asks one question and updates a three-item sidebar. Log reliability has two levels: a clean regime with over 95% target-adjacent clicks, and a noisy regime with 30% target-adjacent and 70% random clicks. Metrics are the bag-of-words cosine between sidebar items and the target, and the number of turns until the target first appears. Each arm uses five personas over five episodes. The authors report that agents improve within a category, that clean logs help while noisy logs hurt unless a harness is added, and that no agent improves across episodes under default prompts.
Critical strengths
- The environment adds a behavioral channel and a second action (the sidebar) to experiential-learning evaluation, with adjustable log reliability, simulator checks. This is the main reason I see value in the paper: chat-only benchmarks cannot test whether an agent handles cheap but noisy evidence well, and this one can.
- Some diagnostics are well designed. Table 1 fixes the candidates and history and changes only the agent's own likelihood and certainty labels.
Critical weaknesses
- At the current scale, the across-episode and harness comparisons are not reliable. Each arm has five personas with no repeated runs reported, and some conditions use different persona sets. The main conclusions rest on these comparisons, so this weighs most in my rating.
- How informative each channel is comes from the simulator design rather than from data. The persona knows the target, its clicks are steered toward products that share the target's traits (App. E.2), the noisy regime mixes in 70% random clicks, and chat replies may not name the target or quote its attributes. The finding that clean logs help and noisy logs hurt is therefore largely built in, and the simulated logs are never compared with real browsing behavior.
Questions for authors
- What explains the gap between the unharnessed and summary arms in Fig. 7 and between Direct and full context in Fig. 8b ? Do all arms share the same personas, episode windows, and simulator settings?
- How many personas, categories, and runs enter each curve?
- Which model is used in Table 1?
We are grateful for the reviewer’s constructive review and we are glad that they see value in an environment that can test whether an agent handles cheap but noisy evidence well. We hope the general response addresses your concerns for experimental reliability, benchmark validity, and the relevance of the behavioral channels. We respond to your other points below:
We fully agree with the reviewers that the behavioral logs are not a perfect model of how a real customer responds to each recommendation. However, we note that such realism is not the goal of the benchmark. Instead of realism, we aim to support across-episode learning by verifying that a customer preference is persistent and learnable across episodes, which is verified by the specialized harness and ideal trajectory experiments. We will state the lack of realism as a limitation.
In Fig. 7, the summary harnesses help to denoise and extract meaningful information from noisy experiences. First, the extractor only keeps the durable traits of the repeat customer. As such, many random clicks of the noisy regime never enter the carried context, whereas the unharnessed agent re-reads every random click verbatim. Secondly, the consistent profile provides a prior that downweight random clicks within an episode.
In Fig. 8b, the direct harness carries the history of prior episodes compressed into a fixed schema of traits. Thus, the agent sees these traits at full salience with the synthesis already done. The full context harness replays the raw history verbatim, including all chats, logs, and recommendations. The traits are buried in it, and the agent must redo the synthesis at every turn, negatively affecting performance.
We thank the reviewer for raising these concerns, and will add this discussion to the main text.
Each curve in the submission covers five personas, five episodes per persona, and three categories per episode (75 categories per arm), from a single run. Error bars are standard errors over personas. The expanded experiments cover 15 personas across 3 disjoint five-episode windows (225 categories per arm). We will add a table with these numbers for every figure.
Table 1 evaluates GPT-5.4. We thank the reviewer for noting this, and will state it in the caption.