acceptodds
Under review as a conference paper at ICLR 2027

SIRA: Self-Improving Recommendation Agents under Demand Shifts

Abstract

Conversational agents increasingly sell products and services, and to be profitable, an agent must (i) find out what each customer wants, (ii) offer what the customer will buy, and (iii) complete the purchase before the customer loses patience. No benchmark measures profit in sales conversations that require all three. We introduce \env, a benchmark that scores the profit an airline keeps when an agent sells flight add-ons to simulated customers. Each customer has a hidden type that determines what they buy, so the agent must learn which questions reveal the type and what each type buys as demand shifts. Each turn, the agent makes one of three tool calls, to ask a question, offer an add-on, or complete the purchase. Unlike language-model customers, which can be persuaded, \env customers react only to these calls, making results reproducible and the best profit exactly computable. Neither prompting a language model nor fine-tuning it with standard recipes makes it a profitable seller. A hand-written sales strategy does better by asking for the facts each add-on requires and keeping a turn to complete the purchase, but ranks offers by revenue rather than profit. We propose \method (Self-Improving Recommendation Agent), which imitates this strategy, then corrects its objective with reinforcement learning on the profit of its own conversations. \method (i) earns 110.7% more profit than the prompted model, (ii) beats the strategy it imitates in every demand phase and by 14.5% overall, with the same test-time information, and (iii) learns which add-ons earn more, coming within 3.4% of the same strategy given product costs. Yet an agent that knows how customers are generated but not their price limits could capture at least 68% of the add-on profit of an agent with this privileged information, and \method captures 37%.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, conversational recommendation, demand shifts, planner distillation, imitation learning, reinforcement learning, agent benchmarks, economic decision-making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TpGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies conversational recommendation agents. First, it introduces an AEROSELL benchmark that scores the profit of each sales conversation with simulated travelers. Second, it develops a two-stage training framework SIRA that intimates a rule-based sales planner with reinforcement learning on profit-correction. SIRA earns much more profits than comparison methods.

Critical strengths
  • The paper is studying a meaningful evaluation task for conversational recommendation agents. The ultimate goal is profit, thus the agent needs to decide between asking a question, offering an add-on or completing the purchase. Conversations without purchase do not contribute to the goal. This is highly relevant in the commercial applications, yet existing benchmarks or models do not cover this.
  • The task is naturally suitable for reinforcement learning. The proposed SIRA implements this with imitation learning and profit reward guided self-improvement, obtaining 14.5% more profits. The method is simple and effective.
Critical weaknesses
  • Simulation of customer behaviors is relatively simple. It assumes a customer accepts an offer exactly when their willingness to pay exceeds the prices and wording never changes a sale. This is good for reproduction. However, in reality, human behaviors are much more complex. They tend to have an ambiguous willingness and can be influenced by the conversation contexts. A better way may be simulating customers with agents under some general action guidelines.
Questions for authors
  1. Could authors discuss whether it is possible to introduce some customers' willingness uncertainty and model that with conversations' pleasantness?
  2. Given the simple customer modeling, is SFT-phase necessary? Can we use RL to directly learn from the simulation?
1 comment
AuthorsOpenReview ↗

Author's response to Reviewer TpGT

We thank the reviewer for recognizing the relevance of the task and that SIRA is simple and effective. Both questions are now answered with new experiments.

W1/Q1: Customer simulation; uncertain willingness and pleasantness. Deterministic customers are a deliberate design choice and a core strength of AeroSell: every sale is reproducible, every agent meets the same customers in the same order, and the full-information optimum is exactly computable. Language-model customers can be talked into purchases and vary from run to run, so an agent could gain by persuading the simulator rather than by making better sales decisions (App. F). AeroSell nonetheless admits richer behaviour, and we tested it:

  • Uncertain willingness to pay (new). We multiplied each customer's willingness to pay for each add-on by e0.25ze^{0.25 z} with z∼N(0,1)z \sim \mathcal{N}(0,1), which is equivalent to probit acceptance, Pr⁡(accept)=Φ(ln⁡(w/L)/0.25)\Pr(\mathrm{accept}) = \Phi\big(\ln(w/L)/0.25\big) for willingness ww and price LL, with draws shared across agents. SIRA earns 107.76107.76 per customer; its change from the noise-free test set (−4.53-4.53, CI [−9.08,0.03][-9.08, 0.03]) is not significant. It still leads the planner by 9.599.59 (CI [2.37,16.80][2.37, 16.80]), the SFT parent by 48.0248.02, and the prompted model by 53.4853.48. SIRA's gains do not depend on exact price thresholds.
  • Pleasantness. The engine already tracks how pushy and how relevant a seller's offers are, and supports customers who leave after repeated or irrelevant offers, which makes deciding what not to offer part of the score (Sec. 6). Making acceptance depend on such conversation features, for example lowering it after each refused offer, is a small extension of this mechanism; we will describe it and how to enable it.
  • Agent customers. AeroSell's interface admits them directly: customers respond to tool calls, so an LLM customer with action guidelines can replace the rule engine while the engine keeps settling the profit. This is complementary; the deterministic engine remains the reproducible core.

Q2: Is the SFT phase necessary? Can RL learn directly from the simulation? Yes: with SIRA's budget, RL does not get started without it, and we now test this directly. We ran SIRA's RL stage from the base model with no Stage 1. Everything else was unchanged: the budget (32 groups of eight conversations), the reward r=M−50.40100+0.02,vr = \frac{M - 50.40}{100} + 0.02,v for profit MM and valid-response fraction vv (Eq. 4), the KL coefficient, the optimizer, the guards, and SIRA's training customers.

  • Few purchases, no learning trend. The policy completed 18 purchases in 240 training conversations (7.5%; 4, 6, 3, and 5 per quarter of training, no upward trend), and a third of its responses were invalid. On the same customer states, SIRA's runs completed 212–219.
  • Why. In 21 of its 30 groups no conversation earned any add-on profit, so the eight returns differed only through the validity term 0.02 v0.02\,v and the profit reward gave no learning signal.
  • Stability. The base model's sampled and trained token log-probabilities differ 2.7×2.7\times more than the SFT parent's (RMS 0.033 vs. 0.012), and the registered importance guard stopped the run after 30 of 32 groups, so it has no test score.

This matches our preliminary studies (App. E), where outcome-reward RL from the base model did not clearly beat the prompted model, and the mechanism in Sec. 5.2: imitation teaches purchase timing (the parent and SIRA complete purchases on the last turn in 51 of 52 and 824 of 824 cases), and RL then corrects what is offered. The revised paper reports this run in App. G (Table 7) and updates the limitation in Sec. 6.

With both questions answered by new experiments, we hope the reviewer will consider raising the score.

Reviewer_qBTuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces AeroSell, a benchmark for evaluating profit-oriented conversational sales agents under hidden customer preferences, limited interaction budgets, and demand shifts. The benchmark simulates airline add-on sales, where an agent must ask questions, recommend products, and complete purchases to maximize seller profit. The authors further propose SIRA, a two-stage framework that first imitates a rule-based planner through supervised fine-tuning and subsequently improves the policy through reinforcement learning using trajectory-level profit rewards. Experiments show that SIRA substantially outperforms the prompted baseline and the planner it imitates. The paper also provides behavioral analyses, comparisons with alternative training approaches, and a pre-registered replication study.

Critical strengths

The paper addresses an important practical challenge in conversational recommendation: maximizing actual seller profit rather than simply increasing recommendation acceptance or conversion rates. The distinction between revenue and profit provides a useful perspective for evaluating sales agents.

SIRA achieves substantial improvements over both the prompted model and the rule-based planner. The authors also provide restricted-decoding experiments and an independent replication study, which strengthen the credibility of the empirical results.

Critical weaknesses

The paper emphasizes self-improvement under demand shifts. However, SIRA's parameters remain fixed during evaluation, and the authors report that removing or replacing interaction histories has little effect on overall performance. Moreover, the test-time demand shifts follow the same mechanism as training, with different magnitudes. Therefore, the experiments primarily demonstrate robustness to previously encountered types of demand shifts rather than adaptation to new or unexpected changes. Additional experiments with unseen shift mechanisms would strengthen this claim.

The main training experiments focus on a single model family, Qwen3.5-9B, and the RL experiments refine only one selected SFT initialization. It remains unclear whether the reported improvements generalize to other model families, different initial policies, or more complex sales environments. The paper would benefit from broader evaluation to establish the general applicability of its conclusions.

Questions for authors

Can the authors clarify the role of demand shifts in the proposed self-improvement framework? Since the policy remains fixed during evaluation and historical interactions have limited influence on performance, what evidence supports the agent's ability to adapt to previously unseen demand changes rather than simply learning a robust policy during training?

The main experiments only evaluate SIRA using Qwen3.5-9B. Although the paper investigates the performance of prompted models at different scales, it does not examine whether the proposed training framework remains effective across different model sizes or architectures. The authors should extend the evaluation to additional LLM backbones to demonstrate the generalizability and robustness of SIRA.

Although the paper includes a Related Work section, the discussion of recent agent-based recommendation and reinforcement learning methods remains insufficient. In particular, the authors should more clearly distinguish SIRA from existing approaches that combine imitation learning with reinforcement learning. More comprehensive comparisons with closely related methods under consistent experimental settings would help establish the novelty and contributions of the proposed framework.

2 comments
AuthorsOpenReview ↗

Author's response to Reviewer qBTu

We thank the reviewer for recognizing the importance of profit over acceptance, and the credibility of the restricted-decoding control and the pre-registered replication. We ran the requested experiments; they support the paper's claims.

W1/Q1: The role of demand shifts; unseen shifts. "Self-improving" refers to Stage 2: the agent improves from the profit of conversations it generates itself, without new labels, as in self-training (STaR, ReST). It does not refer to test-time adaptation, and the paper says so: weights are fixed at test time, and our pre-registered ablation shows that removing SIRA's history changes its profit by only −2.66-2.66 (CI [−5.81,0.50][-5.81, 0.50]), so its gains "reflect robust selling under shift rather than online adaptation" (Sec. 6, App. F). The revised abstract and introduction now state this up front.

The substantive question is whether this robustness extends to shifts the agent never saw. New: under a shift mechanism never used in training (it re-partitions customers by a different fact and moves a different product family), SIRA earns 119.52119.52 per customer and leads the planner by 16.6116.61 (CI [12.54,20.68][12.54, 20.68], all 12 streams), by $17.96 in the shifted phase itself. It keeps 38.0% of the attainable add-on profit, as under the test shift (37.1%). The planner adapts online through recency-weighted estimates, yet SIRA, with fixed weights, beats it under the unseen shift. Demand shifts thus make AeroSell non-stationary, so online learners can be evaluated directly, and they test robustness, which SIRA passes for seen and unseen mechanisms.

W2/Q2: Other initial policies and model families.

  • Initial policies (new). The SFT parent was fixed on validation customers before any test result and is the weakest of the three SFT seeds on sale and completion rates, so SIRA's result does not rest on a favourable parent. We ran its RL stage, unchanged, from the two other seeds:
RL started fromBefore RLAfter RLGain (95% CI)
SFT parent (SIRA, 3 runs)60.8060.80112.28±7.48112.28 \pm 7.48+51.48+51.48 [43.90,59.06][43.90, 59.06]
SFT seed 1$60.7592.9692.96+32.21+32.21 [27.10,37.31][27.10, 37.31]
SFT seed 2$71.56103.32103.32+31.76+31.76 [21.82,41.70][21.82, 41.70]

RL improves every initializer by more than 3131, in all 12 streams. The final level varies, as it does across SIRA's own three runs (104.46–119.39104.46–119.39); the run from seed 2 is level with the weakest of them (−1.15-1.15, CI [−9.04,6.74][-9.04, 6.74]). From the base model, with no imitation stage, the same RL completed only 18 purchases in 240 training conversations (SFT initializers: 198–219).

  • Model size and family. The recipe has no model-specific component, and AeroSell evaluates any model. The failures SIRA fixes appear at every size of prompted Qwen3.5 from 2B to 27B (2B–9B stay within 1.321.32 of the ticket profit; 27B gains mainly by making fewer invalid calls), and the 9B SIRA beats the 3×3\times larger 27B by 33–3633–36 in every setting (prompted, restricted, repair; all 12 streams). Training another family does not fit the discussion period (SFT alone takes 11–19 GPU-hours per run); we will add a second model family in the final version.

Q3: Related work and comparisons under consistent settings. The revised Sec. 2 draws the distinction explicitly. DAgger and distillation converge to the expert, here a revenue-ranking planner; expert iteration needs search with the true customer model, which is privileged in AeroSell; ReST and ReST$^{\mathrm{EM}}$ fine-tune on filtered samples. SIRA corrects the teacher's objective with on-policy profit RL, which is why it surpasses its teacher by 14.5% with the same information. Comparisons under consistent settings are already extensive: DPO, three SFT seeds, and planner-advice prompting (Table 1); 17 recipes in App. E (imitation of logged dialogues, turn-level and action-value DPO, offline GRPO, DuCA, outcome-reward RL from the base model and from SFT); and the new runs above, all with the same benchmark, data, and evaluation.

Presentation. We will open Sec. 3 with an overview separating what is measured (profit per conversation), what is controlled (validity, information), and what is learned when.

We believe the new experiments answer each question, unseen shifts and other initializations in particular, and we would be grateful if the reviewer would reconsider the score.

i will keep my score.

Reviewer_37SBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AeroSell, a benchmark for evaluating profit-oriented conversational recommendation agents, and proposes SIRA (Self-Improving Recommendation Agent), a two-stage training framework that combines imitation learning with reinforcement learning to improve sales performance. AeroSell simulates airline add-on sales involving customers with hidden preferences, willingness-to-pay thresholds, limited patience, and changing demand patterns. Agents must decide when to ask questions, recommend products, and complete purchases while maximizing profit.

Critical strengths
  1. The paper investigates profit optimization in conversational recommendation, moving beyond conventional recommendation metrics such as accuracy, acceptance rate, and user satisfaction. The explicit consideration of information gathering, product selection, purchase completion, and demand shifts provides a useful perspective on commercial recommendation agents.
  2. A well-structured benchmark with measurable outcomes. AeroSell offers a controlled environment with hidden customer preferences, different product profit margins, constrained dialogue budgets, and deterministic purchase decisions.
  3. Strong empirical improvements over several baselines. SIRA substantially outperforms prompting, SFT, DPO, and the original rule-based planner.
Critical weaknesses
  1. Limited methodological novelty. The proposed SIRA framework combines supervised imitation of a rule-based planner with reinforcement learning from self-generated trajectories. Both components are well-established in existing work on policy distillation, expert iteration, and multi-turn RL.
  2. The benchmark’s design may exaggerate the benefits of learning valid actions. The prompted model produces invalid responses in approximately 90% of conversations, while the SFT parent produces invalid endings in approximately 70%. Furthermore, a single invalid response immediately terminates a conversation without an opportunity for correction.
  3. The evaluation leaves substantial room for improvement in recommendation quality. Despite strong improvements over the baselines, SIRA captures only 37.1% of the full-information reference’s add-on profit, while an agent with knowledge of the customer-generation process but without individual willingness-to-pay information captures approximately 68%. This substantial gap suggests that SIRA remains limited in its ability to infer customer preferences and make effective recommendations.
Questions for authors
  1. SIRA combines imitation learning from a rule-based planner with REINFORCE on self-generated trajectories. How does this framework differ fundamentally from existing approaches such as Expert Iteration, DAgger, ReST, and multi-turn reinforcement learning?
  2. The prompted and SFT baselines exhibit very high invalid-response rates, which substantially affect their profitability. Although constrained-decoding experiments are provided, could the authors compare SIRA against stronger agents equipped with structured action selection, explicit planning, or tool-call repair mechanisms?
1 comment
AuthorsOpenReview ↗

Author's response to Reviewer 37SB

We thank the reviewer for recognising the profit perspective, the benchmark's design, and SIRA's strong gains. Below we answer each weakness with evidence, including all the stronger agents requested in Q2.

W1/Q1: How does SIRA differ from Expert Iteration, DAgger, ReST, and multi-turn RL? We do not claim a new RL algorithm; our primary area is datasets and benchmarks. The point of SIRA is that it surpasses its teacher on the environment's objective, which the named families either cannot do in AeroSell or approach differently:

  • DAgger and distillation converge to the expert's policy, here a planner that ranks offers by revenue ($98.04). SIRA optimises profit and beats it by 14.2414.24 (CI [9.60,18.88][9.60, 18.88]), and by 17.3517.35 under restricted decoding on 360 untouched customers. This gain is unreachable for a method whose fixed point is the teacher.
  • Expert Iteration improves the expert by search with the true model. In AeroSell, search needs each customer's hidden willingness to pay, which is privileged; SIRA's teacher sees only public information.
  • STaR / ReST / ReST$^{\mathrm{EM}}$ fine-tune on filtered high-reward samples (positive-only, batch). SIRA uses on-policy REINFORCE with leave-one-out advantages, which also pushes probability away from below-average conversations, and a KL anchor to the imitation policy, with only 256 conversations per run.
  • Multi-turn RL (ArCHer, RAGEN, SWEET-RL) starts from a prompted policy. New: SIRA's own RL stage, run unchanged from the base model, completed 18 purchases in 240 training conversations (SIRA: 212–219 on the same customers), with no upward trend, and was stopped by its importance guard. Outcome- and process-reward RL in our preliminary studies (offline GRPO, DuCA; App. E) improved profit by at most 2%.

Imitation alone is bounded by its teacher, and RL alone does not get started. The combination, imitation for validity and purchase timing followed by profit RL that corrects the teacher's objective, is the design point; the revised Sec. 2 makes the comparison explicit.

W2/Q2: Does the benchmark exaggerate the benefit of valid actions? Stronger agents. We respectfully disagree. Production tool-calling systems reject malformed calls, so an invalid call is a real failure, and AeroSell measures it. Validity is also controlled in four ways:

  1. Structured action selection (restricted decoding). Every agent can only emit a legal call. SIRA still earns 58.9% more than the prompted model and 31.5% more than its SFT parent, in all 12 streams; the pre-registered replication on 360 untouched customers confirms every comparison.
  2. Explicit planning. The planner never makes an invalid call and completes every purchase. SIRA beats it in every phase, by 14.5% overall.
  3. Decomposition. Under restriction, SIRA's $27.19 lead over its parent splits into more completed purchases ($10.3) and better offers ($16.9): $63.26 of add-on profit per purchase versus $46.30, and 6.2% of offers to low-margin bundles versus the planner's 26.3%.
  4. Tool-call repair and a 3×3\times larger model (new, as requested). Prompted Qwen3.5-27B earns $76.10, $78.37 under restricted decoding, and $79.03 with repair (three retries with the validator's error message). Repair lifts the prompted 9B model to $73.40 and the SFT parent to $67.24, which still ends 50 of 100 conversations on an invalid response. SIRA (9B, $112.28) leads every one of these agents by $33–45 (every 95% CI above $27), in all 12 streams.

Most of SIRA's gain is in what it sells, not in producing valid calls.

W3: SIRA captures 37.1%, a generator-aware agent about 68%. This is headroom by design, not a weakness of SIRA. The $\approx$68% agent knows how customers are generated, the current demand phase, and each customer's patience, none of which any agent observes. Among agents with the same information, SIRA's 37.1% is the highest: it exceeds the planner (28.6%) and the SFT parent (6.2%), and comes close to the cost-aware planner, which is given product costs (39.5%). Under an unseen shift it keeps 38.0%. A benchmark whose best method sits at the bound would be of little use; AeroSell quantifies exactly what remains to be learned about customer types.

We believe these results resolve W1–W3 and both questions, and we would be grateful if the reviewer would reconsider the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.