- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes IEL, an experience-learning framework for large language model-based customer service agents. Specifically, it aims to address the issue in interactive service environments where a single conversation may involve multiple different user intents, leading to a mismatch in the granularity of experience learning. IEL was evaluated using three execution models on ECom-Bench and a cross-category extension, and the comparison results show that IEL achieved the highest average pass^k scores, demonstrating its effectiveness.
Critical strengths
This paper deals with the practical issues of learning from multi-intent dialogues, where even a failed conversation can still have successfully handled parts for a single request. Specifically, it offers a reasonable way to preserve these useful parts by using cross-turn intent instances as the unit of experience. Overall, this gives the method a clear motivation and practical value for service agents.
Critical weaknesses
-
The construction and evaluation of ECom-Bench-XCat are not sufficiently documented. Appendix A reports task counts, user profiles, and environment contents, but does not explain whether the tasks were manually written, generated by a model, or adapted from existing templates. Besides, pass^k is appropriate for measuring overall task success, but it does not show which parts of a multi-intent task improve.
-
Although the paper discusses recent procedural memory approaches such as MemP, these closely related methods are not included in the experimental comparisons. The current baselines cover general experience reuse and workflow learning, but may not adequately represent recent advances in structured procedural memory. Consequently, it remains unclear whether IEL's improvements stem specifically from its proposed intent-level credit assignment mechanism or more generally from structured experience organization and retrieval.
Questions for authors
-
How were the XCat tasks and evaluation annotations created and validated? In particular, how were task goals translated into Action, Search, and Output checks, and how was task feasibility verified?
-
Could the authors explain why closely related procedural memory methods, particularly MemP, were not included in the experimental comparisons? Additionally, could the authors provide further experimental evidence or analysis to clarify whether the observed improvements are primarily attributable to intent-level credit assignment rather than structured experience organization and retrieval?
3 comments
Response to W1 (cont.): Finer-grained metrics beyond pass^k
W1 (cont.)
"... pass^k is appropriate for measuring overall task success, but it does not show which parts of a multi-intent task improve."
Response: Thanks for this insightful suggestion! We report finer-grained metrics:
- Per-check success: IEL and ReAct pass Search checks at almost the same rate (92.74% vs. 91.91%), so IEL's gain does not come from retrieving more information.
- Gain vs. number of goals: the pass^1 gain over ReAct is 0 (2 goals), +3.70 (3–4), and +6.25 points (6). The gain grows with the number of intents, where episode-level supervision is most ambiguous.
Response to W2 & Q2: MemP, and credit assignment vs. structure
W2 & Q2
"Could the authors explain why closely related procedural memory methods, particularly MemP, were not included ... whether the observed improvements are primarily attributable to intent-level credit assignment rather than structured experience organization and retrieval?"
Response: Thanks for this key question!
MemP. We added it (Table R2, in our response to Reviewer b93v, W1). IEL leads by 18.44 pass^1 and 32.24 pass^3 points. MemP indexes each episode by its opening request, so later intents rarely retrieve a memory (only 14.5% of test episodes do), and it discards partially failed episodes.
Credit assignment vs. structure. All variants below keep IEL's structure (intent reconstruction, taxonomy, intent-indexed retrieval, procedure format) and the same growth and evaluation protocol. They differ only in the outcome labels used for procedure induction:
- IEL-EpisodeCredit: the verifier is replaced by the oracle benchmark reward. Every intent record in a successful episode is labeled POSITIVE, and every record in a failed episode NEGATIVE.
- IEL-NoVerify: the verifier is removed, and every attributed intent record is treated as POSITIVE.
- IEL-NoNeg: the verifier is kept, but NEGATIVE records are dropped, so procedures are induced from POSITIVE records only and contain no error warnings.
Table R1: ECom-Bench, DS-V4.1-Flash, mean over three folds
| Variant | Organization | Outcome labels | pass^1 | pass^2 | pass^3 |
|---|---|---|---|---|---|
| IEL (full) | intent-level | intent-level verified | 66.88 | 58.03 | 52.94 |
| IEL-EpisodeCredit | intent-level | episode-level (oracle) | 56.60 | 45.91 | 39.62 |
| IEL-NoVerify | intent-level | all positive | 52.87 | 44.70 | 39.65 |
| IEL-NoNeg | intent-level | positives only | 59.69 | 50.84 | 45.10 |
| w/o Intent Decomposition | episode-level | episode-level | 50.22 | 38.85 | 31.92 |
| MemP | structured procedural | episode-level | 48.44 | 28.94 | 20.70 |
| ReAct (no memory) | – | – | 59.30 | 46.70 | 39.76 |
With the same structure and even oracle episode rewards, IEL-EpisodeCredit loses 13.32 pass^3 points and only matches ReAct; treating every record as positive (IEL-NoVerify) gives the same result. Starting from w/o Intent Decomposition (31.92 pass^3), intent-level organization adds 7.70 points (→ 39.62), and intent-level credit assignment adds the remaining 13.32 (→ 52.94). Dropping negative records (IEL-NoNeg) costs 7.84 pass^3 points, so verified failures also contribute.
We thank the reviewer again for the positive assessment and are happy to provide further details.
Response to W1 & Q1: How XCat tasks and checks were built and validated
W1 & Q1
Response: Thanks for this valuable question! We will add the following description to Appendix A. We adapted the service scenarios of ECom-Bench to clothing and food. An LLM drafted the user profiles and goal lists, and the authors revised every task by hand. Each task gives the simulated user a profile and two to six ordered goals (252 in total), all grounded in a fixed store state of 36 products, 20 customers, and 60 orders with their logistics, invoice, and after-sales records. We then annotated each goal with the tool calls it requires, following a written guideline and ECom-Bench's annotation schema and scorer. A goal that only asks for information is checked by a Search annotation: the required call, with its exact arguments, must appear among the agent's calls. A goal that changes the store state is checked by an Action annotation: the gold calls are replayed on a copy of the database, and the agent's final database must match the result exactly. An Output check requires the agent's replies to contain predefined key strings, matched as exact substrings. Every task and its checks were then reviewed against the guideline by an author who had not annotated that task, and disagreements were resolved by discussion. Finally, we replayed each task's gold calls in the environment and confirmed that every call succeeds and that the replayed state passes all checks, so every task is solvable. We also confirmed that every identifier a goal needs is either given to the simulated user or reachable through the tools.