acceptodds
Under review as a conference paper at ICLR 2027

Learn to Sell, Not What to Sell: Dualistic Self-Evolution (DSE) of LLM Sales Agents with Human Nature as Reward (HNR)

Abstract

Text-based selling is a long-horizon decision process in which customer behavior evolves across interactions and damaged trust is rarely restored. Delayed sales outcomes complicate autonomous improvement, so humans must still supply dense rewards and evaluation. Motivated by the distinction between transferable sales skills and business-specific knowledge, we introduce Dualistic Self-Evolution (DSE) with Human Nature as Reward (HNR), which separates a profession learned in the model weights from business knowledge, rules and tools carried by an external harness. The inner loop learns from open cross-category dialogues and 8 business-agnostic prototype tools through supervised fine-tuning followed by reinforcement learning. HNR is drawn from sales practice: it rewards selling with the grain of the customer (answer first, verify before claiming, move one step forward), penalizes the model's own counterproductive habits, and assigns zero episode reward to unsupported claims and other credibility violations. Outcome-gated turn-level scoring reduces the share of rollout groups with zero reward variance from roughly two thirds under outcome-only rewards to 3–5%. The outer loop updates the harness through reward-guided edits validated by real-conversation replay. The loops share no parameters and have separate reward agents and human arbiters: senior salespeople assess craft and quality-control staff assess factual and procedural correctness. With the target business excluded from training, frozen models transfer to a production used-car harness that maps the 8 prototypes onto 16 business tools. On replay of 600 real customer turns with full conversation histories, the trained 9B model improves strict response quality by 8.3 percentage points over its same-size base (13% relative; three seeds; paired 95% CI [+4.4, +12.0]), and the share of turns with a tool call rises from 57% to 80%; the 8B gain is smaller and statistically unresolved. On the out-of-domain Ψ-Bench, persuasion improves by 88% at 8B and 54% at 9B. At 9B the prototype-tool SFT carries the strict gain and reinforcement learning adds to it only at 8B; the gate's measurable contribution is gradient availability rather than replay score. Fine-tuning on business transcripts degrades replay quality and removes tool use, whereas the harness alone lifts untrained models above the deployed system and accommodates business changes without retraining.

Primary area:
reinforcement learning
Keywords:
LLM agents, self-evolving agents, dualistic self-evolution, human nature as reward, reinforcement learning, sales dialogue, persuasion, multi-turn RL, reward design, tool use, harness, domain transfer, human evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cdiaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dualistic Self-Evolution (DSE), a framework for developing transferable LLM-based sales agents by separating general sales skills learned in model parameters from business-specific knowledge, rules, and tools maintained in an external harness. The framework consists of two independent optimization loops: an inner loop that trains sales agents through supervised fine-tuning and reinforcement learning using a novel Human Nature as Reward (HNR) mechanism, which combines turn-level feedback, sales outcomes, and credibility constraints; and an outer loop that continuously improves the business harness through reward-guided modifications and real-conversation replay. Experiments on a production used-car sales platform and additional business domains demonstrate improved response quality, tool usage, and cross-domain persuasion. In particular, the trained 9B model achieves an 8.3-percentage-point improvement in strict response quality over its base model. The results suggest that separating transferable agent capabilities from domain-specific knowledge can improve adaptability while reducing the need for repeated model retraining.

Critical strengths
  1. The paper addresses an important challenge in developing LLM-based sales agents: learning transferable sales skills while adapting efficiently to changing business-specific knowledge and requirements.

  2. The paper includes extensive experiments across different model sizes, business domains, and evaluation benchmarks, supported by ablation studies and human evaluations.

  3. The proposed approach demonstrates improvements in response quality, tool usage, and cross-domain persuasion, suggesting practical benefits for building adaptable LLM agents.

Critical weaknesses
  1. The proposed framework primarily combines existing techniques, including SFT, GRPO, process rewards, and external harness optimization. The paper could better clarify its methodological contributions beyond integrating these established components.

  2. The experimental results show that most performance gains come from prototype-tool SFT rather than the proposed HNR-GPO algorithm. In particular, RL slightly reduces strict response quality for the 9B model (73.5% to 72.9%), raising questions about its practical effectiveness.

  3. Although HNR significantly reduces zero-variance rollout groups, this improvement does not consistently translate into better downstream performance. Alternative reward formulations achieve comparable results, making the necessity of the proposed reward mechanism unclear.

  4. Much of the evaluation relies on LLM judges and simulated customers, which may introduce biases. Human evaluation shows relatively low inter-annotator agreement and does not establish a statistically significant preference for the trained 9B model.

  5. The proposed outer-loop optimization still requires human participation for fault localization and harness modifications. Therefore, the framework does not yet demonstrate fully autonomous self-evolution.

Questions for authors

N/A

Reviewer_xkoZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Dualistic Self-Evolution (DSE), which trains transferable sales behavior in model weights while keeping business-specific knowledge and tools in an external harness. It combines prototype-tool SFT, process- and outcome-based RL, and human-assisted harness updates. Evaluation includes held-out production replay, synthetic businesses, benchmarks, and human assessments. The clearest finding is improved strict replay for the 9B model, largely attributable to SFT; the 8B gains and human craft preference remain inconclusive.

Critical strengths

1.Prototype-tool training improves a small model’s tool use on an unseen business harness, without training on that business’s conversations. The held-out evaluation and robustness checks make this a credible and practically useful transfer result. 2.The negative results help clarify the strengths and limits of the training recipe. The paper distinguishes the contributions of SFT and RL, identifies unsuccessful training choices, and acknowledges disagreement between simulated and human evaluation. These findings provide useful guidance beyond the headline improvement.

Critical weaknesses

1.The evidence does not isolate the contribution of outcome gating from dense process supervision. When all rollouts share the same outcome, the outcome multiplier cancels during advantage standardization, so process-score variation alone could explain the improved gradient availability. A matched process-only control is needed to support the gate-specific claim. 2.The replay improvement does not clearly establish better factual correctness. The evaluation rewards conformity with recorded tool-use decisions and supplies recorded tool outputs, making it difficult to distinguish improved compliance from fewer factual or procedural errors. The additional judge provides reassurance, but a direct human comparison separating correctness from tool-use compliance would better substantiate the main quality claim.

Questions for authors
  1. Does the implemented reward allow a turn to receive zero without triggering a veto? If so, which rule permits it? This would clarify the feasible process-score range in Lemma 1 and the clean episodes with P below 0.6 illustrated in Figure 3.
  2. What do the existing quality-control annotations show for the 9B trained-versus-base factual/procedural comparison, and how well does the retained replay judge agree with those labels? In the same replay, how are calls with tools or arguments absent from the production trace handled, and is claim grounding checked against only the evidence actually supplied to the candidate model? These details could materially change the interpretation of the strict-quality gain.
Reviewer_3hU9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  1. Paper summary: DSE trains transferable sales skills through SFT and RL while separately updating business knowledge, rules, and tools in an external harness.
  2. Main contributions: Skill–business separation, prototype-tool training for cross-business transfer, and HNR rewards combining process quality, sales outcomes, and credibility constraints.
  3. Validation: Evaluation uses 600 real customer turns, two synthetic businesses, external benchmarks, human ratings, and ablations. It tests offline quality and transfer, but not long-term real-world sales or trust.
Critical strengths
  1. Transparent reporting of mixed and negative results. The authors openly acknowledge several important limitations: reinforcement learning does not consistently improve upon supervised fine-tuning, simulator-based evaluations can disagree with human judgments, and the 8B model’s improvement over its base model on strict replay quality is not statistically significant. Reporting these results helps readers assess the scope and reliability of the findings.
Critical weaknesses
  1. Unclear presentation and an insufficient connection between the motivation and the experimental design. The paper repeatedly describes closely related distinctions using different terminology, including “profession versus business,” “character versus circumstance,” and “model weights versus harness.” Their relationships are not explained consistently, which creates unnecessary cognitive overhead. The presentation also blurs the distinction between DSE as an overall framework, HNR as a reward design, and HNR-GPO as an optimization procedure. More importantly, the motivation emphasizes long-horizon sales interactions, delayed commercial outcomes, and evolving customer trust. However, training uses simulated conversations with at most six agent turns, while the primary real-data evaluation assesses individual responses through replay with fixed conversation histories. The paper does not sufficiently explain or demonstrate how improvements under these settings address the long-horizon challenges that motivate the work, particularly delayed feedback and the effects of an agent’s actions on subsequent customer behavior.

  2. Insufficiently established methodological novelty. The inner loop updates model weights to acquire transferable skills, while the outer loop adapts a business-specific harness. Both components draw on established approaches to model training and external agent optimization. Similarly, HNR combines rubric-based process supervision, outcome-dependent reward weighting, and a credibility veto, while HNR-GPO largely retains a standard group-relative policy optimization objective. Although the paper cites relevant work, its comparisons do not sufficiently establish how the proposed mechanisms differ from existing approaches such as Co-Harness, rubric-based reinforcement learning, and gated reward methods. The combination may offer practical engineering value, but the paper needs to identify and validate the contribution of its specific design beyond that of the established components. In its current form, I am not convinced that the methodological novelty is sufficiently established for acceptance at ICLR.

  3. A mismatch between the central narrative and the strongest empirical evidence. The paper places substantial emphasis on HNR, HNR-GPO, and agent self-evolution, whereas its strongest results primarily support prototype-tool supervised fine-tuning. For the 9B model, the mean strict replay score is 73.5 after SFT and 72.9 after HNR-RL, providing no evidence of an additional improvement on this metric. Most of the gains on the external persuasion benchmark are also already present after SFT. HNR does substantially reduce the proportion of zero-variance rollout groups, which is a useful training-signal result. However, this does not by itself establish improved downstream performance or the necessity of the proposed reward composition. A crucial missing ablation is a process-only reward with the same credibility veto, which would help isolate the contribution of outcome gating from that of dense process supervision. The evidence for self-evolution is also limited. The authors explicitly describe an early-stage, human-assisted setting, and human involvement alone does not invalidate that framing. Nevertheless, failure localization and edit proposals in the outer loop still rely on humans. The experiments therefore do not clearly isolate the system’s capacity to generate effective updates autonomously or establish its advantage over a comparable human-assisted maintenance workflow.

Questions for authors
  1. Given that most reported gains are already achieved by prototype-tool SFT, what controlled evidence demonstrates an additional benefit attributable to HNR or the proposed self-evolution mechanism over SFT with a manually maintained harness, under comparable computational budgets and human involvement?
  2. I could understand the rationale for managing stable skills and changing business knowledge separately. However, how is the assumption J(θ,H)≈u(θ)+v(H)J(\theta,H)\approx u(\theta)+v(H) justified when interactions between the two may be substantial? For example, the effectiveness of learned skills may depend strongly on the business context, causing the same θ\theta to perform very differently under different harnesses HH. In that case, what justifies the conclusion that improvements from updating θ\theta will preserve their sign across harnesses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.