- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentBake, a framework and benchmark for testing whether LLM agents use earlier user feedback in later tasks, in which the hidden user profile is only seen by the simulator and the judges. Compared to prior personalization benchmarks, which mainly evaluate text responses, AgentBake evaluates tool-using agents across different agent frameworks. Compared to execution benchmarks such as tau-bench, it also checks whether user preferences are followed, and judges task correctness separately from preference matching. The paper also builds a shared interface that connects personalization methods to various agent SDKs.
The main experiments run 12 personalization components and one baseline on the same calculation tasks across 7 agent SDKs. The results show that agents complete most tasks but fully match the preferred style in less than half of the responses. The paper also traces preferences from feedback to host input and finds that many are lost.
Critical strengths
-
The paper records what is extracted, stored, retrieved, and actually sent to the host, and separates this from the final behavior. It shows that many preferences are lost before the host sees them, which an evaluation based only on task success or on stored memory would not show.
-
The paper covers a setting existing benchmarks miss. Existing benchmarks mainly evaluate text responses, or check task outcomes but not user preferences. AgentBake tests tool-using agents that must learn preferences from feedback, and checks both task success and preference matching.
-
Careful experimental design. The 2 2 update/exploration design with identical starting states separates the effect of updating from exploration, task correctness is judged separately from preference matching, and unobservable or missing judgments are reported instead of counted as failures.
Critical weaknesses
-
The paper misses many details. Most preferences are lost at retrieval, but the paper does not say how retrieval works, e.g. what counts as "relevant". It is also not reported how much of the loss is due to no_memory, how much to relevance filtering, and how much to the budget. FABLE and its variant FABLE-Diag are not explained in detail, and each adapted method gets one or two lines. In addition, the paper states that the conversation dataset will not be publicly released, and it does not say whether the code will be released. All of this makes the paper hard to reproduce.
-
The adapted methods are quite far from the originals. For example, CIPHER no longer learns from user edits. A shared interface makes the comparison fairer, but the methods being compared are no longer the original methods.
-
The scope is narrower than the title and Figure 1 suggest. The main comparison uses only one type of calculation task. The title says "heterogeneous" agents, which sounds like many different kinds of agents, but in practice it mainly means 7 different SDKs. The underlying model is the same, and so are the tasks. Memory is unobservable in all judged calls under Astra (Table 3), and clarification is mostly not applicable.
Questions for authors
-
How does retrieval work? Can the authors provide more details such as relevance criterion?
-
What is the form of the extractor's confidence, and what threshold counts as "high-confidence"?
-
Could the authors provide more details about FABLE-Diag, and explain how exactly it differs from FABLE?
-
How the adapter is implemented for each SDK?
4 comments
Replies to Reviewer JxTW - Part 2
Reply 2/3 — Retrieval, confidence and reproducibility (W1, Q1, Q2)
Thank you for asking how relevant instructions are selected, how confidence affects updates, and what can be reproduced. We explain these mechanisms below and trace where saved instructions are lost before reaching the host. Card construction accounts for the largest measured loss.
What “relevant” means. The shared text path retrieves the current user's general rules and rules matching the task's declared type. It does not rank rules by semantic similarity to the current request. Eligible persistent instructions require valid fields, relevant feedback, confidence ≥0.70, nonempty text and a general or task-specific scope; task-completion, factuality and “other” targets are removed. General rules need two supporting records. For each scope/target, one rule survives ranking by confidence rounded to two decimals, support, explicitness and recency. The renderer deduplicates wording and packs whole rules within 512 tokens; the compiler allows eight instructions of at most 500 characters each. Selecting no_memory clears this list while the other selected settings remain active (revised manuscript, p. 16, lines 819–832).
Where instructions are lost. Of 2,241 newly accepted instructions in the update/exploration study, 232 have no later task. The remaining 2,009 reach the following furthest stages:
| Outcome before a later host call | Instructions |
|---|---|
| Removed during card construction | 1,333 |
| Excluded by task-type routing | 125 |
Retrieved only under no_memory | 365 |
| Entered a compiled instruction list | 186 |
Card losses comprise 1,309 rules outranked for the same target/scope and 24 general rules lacking a second supporting record. Deduplication, token budgets and compiler limits remove no additional instruction. The initial study has the same ordering: 919 card losses, 495 suppressed by no_memory, 162 routing losses and 619 compiled instructions among 2,195 with later opportunities. Each origin is counted once at its furthest later stage (Table 17; revised manuscript, p. 27, lines 1426–1455).
The omissions can remove useful content. Blind judgments find additional content in 59–82% of lost origin–target pairs, although agreement is only fair (κ 0.33–0.35) and answer-level effects were not tested. Routing excludes rules tagged as problem-solving/proof even though each judge finds 283/287 applicable to calculation (280 jointly). This supports information loss through selection and routing (revised manuscript, p. 27, lines 1454–1457; p. 28, lines 1461–1464).
Confidence and the update threshold. Confidence is a generated JSON field in [0,1]. The extractor learns profile-blind teacher labels. The inclusive 0.70 cutoff is an operating choice; all valid scores in the eight shared-extractor studies were at least 0.75, so it excluded no additional valid label. Valid feedback from a successful update-on call can be stored at any confidence. A numerical update additionally needs relevant, polarity-consistent feedback: positive with rating 4–5 or negative with rating 1–2. Mixed feedback can supply text instructions but no numerical update. The selected, executed action receives (rating − 3)/2, without confidence weighting (revised manuscript, p. 20, lines 1071–1078; p. 21, lines 1080–1087).
The source-learner experiments use separate retrieval: released CIPHER retrieves up to three contexts by BERT similarity, while PAHF retrieves up to ten entries from the current user's memory with DRAGON and summarizes them. These instructions bypass the shared card packing and memory filter (revised manuscript, p. 16, lines 833–839).
What can be reproduced. An anonymous package is prepared with derived ledgers for the two instruction-path studies, recomputation code and synthetic checks. These run without private conversations or model calls; rebuilding the histories requires the saved private inputs. Upon acceptance we will release benchmark and evaluation code, including interfaces, controllers, rule processing, SDK integrations, configurations and tests. Private conversations, hidden profiles and unreviewed checkpoint weights are excluded. The revision states this release scope rather than implying that the conversation dataset is public (revised manuscript, p. 36, lines 1906–1921).
Replies to Reviewer JxTW - Part 3
Reply 3/3 — FABLE-Diag and SDK implementation (Q3, Q4; W1)
Thank you for asking how FABLE-Diag differs from FABLE and how each SDK adapter is implemented. We clarify how diagonal precision changes both posterior means and uncertainty, then show how the adapters pass the selected memory, tool and style settings to each SDK.
FABLE versus FABLE-Diag. Both score each permitted memory/tool/style setting as b + φᵀθ, using fixed default utility b and the same full feature vector φ (265 coordinates for the controlled catalog). Both start with a zero-mean unit Gaussian prior, observation variance 0.25, and update from the residual y − b. FABLE updates full precision Λ ← Λ + φφᵀ/σ² and h ← h + φ(y − b)/σ², giving mean Λ⁻¹h. FABLE-Diag keeps diagonal precision d, giving mean h/d and covariance diag(d)⁻¹. With exploration off they use the mean; with exploration on they take one unit-scale Gaussian draw (revised manuscript, p. 17, lines 908–916; p. 18, lines 918–929).
All features remain in FABLE-Diag. For repeated observations of the same feature vector with k nonzero coordinates and residual r, the full model's residual prediction approaches r, whereas the diagonal prediction approaches kr.
Does heuristic promotion explain its behavior? Only the separate scripted configuration-choice diagnostic supplies numerical onboarding. FABLE-Diag can then add a weight-0.8 observation from its own prediction after eight supporting updates and a working sign probability of 0.9 (or 0.1 for the opposite sign), once per direction. Source FABLE's confidence-sequence writeback is different and leaves its posterior unchanged. In the host-execution studies there is no numerical onboarding, promotion never triggers, and its recorded sentences never enter the host's instructions (revised manuscript, p. 18, lines 931–939).
In the five-seed scripted diagnostic, disabling promotion changes each of five measured outcomes by at most 0.0005, with 95% intervals containing zero and lying within ±0.0013. FABLE-Diag still has higher configuration alignment but lower adequacy and reward, and higher regret and cost, than FABLE (Table 5). Promotion therefore does not explain that trade-off; the diagonal variant is not uniformly better (revised manuscript, p. 18, lines 941–955).
How each SDK receives a selected configuration. A shared compiler turns the selected memory/tool/style settings into a control block and a permitted tool catalog; the task request itself is unchanged. The control block contains one instruction per setting and, except under no_memory, up to eight stored preference instructions. Table 8 now gives the concrete entry points:
| SDK | Agent and field carrying control text |
|---|---|
| AutoGen | AssistantAgent, system_message |
| CrewAI | One Agent/Crew, Agent backstory |
| LangChain | create_agent, system_prompt |
| LangGraph | create_react_agent, prompt |
| LlamaIndex | FunctionAgent, system_prompt |
| PydanticAI | Agent, system_prompt |
| Strands | Agent, system_prompt |
LangChain's entry point runs on LangGraph. These are SDK integrations with distinct loop behavior. The controlled runs share Sonnet 4.6, temperature zero, 2,048 output tokens and a twelve-request relay limit, with stated exceptions for host/budget sensitivity tests. Tools enforce user/task-bound records, memory keys and commit permissions; memory deletion requires separate authorization. (revised manuscript, p. 20, lines 1026–1060).
Delivery is checked separately. In the accurate-instruction study, all 17,356 forwarded requests are checked for the supplied instruction. Six SDKs preserve the system text and function schemas; CrewAI embeds the text and parses textual actions. For 6,622 successful executions, the last forwarded request matches logged tool events in order, arguments and return text p. 20, lines 1062–1070).
Thank you for your detailed responses, which resolve some of my concerns. I am keeping my original rating because the remaining issues are not changed by the rebuttal.
- Reproducibility is still limited: the code is released only upon acceptance, and rebuilding the histories requires private inputs.
- Finally, most compared methods remain adaptations of the originals.
Replies to Reviewer JxTW - Part 1
Reply 1/3 — Source methods, learning and task scope (W2, W3)
We now separate the common-interface controller comparison from experiments retaining CIPHER's and PAHF's released learning routines. Both released learners improve style within eight feedback opportunities, under two judges and on two task families. This directly addresses whether our earlier adaptations removed the mechanisms needed to learn.
What the adaptations establish. You are right that they differ from the source systems. Scalar CIPHER uses selected-versus-default setting contrasts instead of learning from edits; these contrasts remain zero when initialized empty under the tested conditions. PUMA and PEToolLLaMA update configuration scores without training their source models. We now state this distinction in §4.2 and enumerate the retained mechanisms and departures for every method in Table 4. The matched comparison evaluates these implemented controllers (revised manuscript, p. 5, lines 231–242; p. 17, lines 864–906).
Learning with the released routines. The added calculation study retains CIPHER's edit inference, retrieval and aggregation, and PAHF's retrieval, questions and feedback updates. It bypasses the shared extractor and tests seven SDKs, four fixed synthetic users and ten replicate trajectories. Held-out evaluations disable learning and questions. For continued learning, both the updated state and a copy frozen after eight opportunities execute the held-out tasks afresh (p. 9, lines 448–454; p. 40, lines 2145–2159).
These are 97.5% intervals, adjusted over each study's two primary style contrasts; Astra is primary. Early gains concentrate in one profile per learner, and worst-case missing-outcome bounds remain positive. Style improvement does not ensure correctness: native task success falls from 97.8% to 93.3% for CIPHER and 69.8% to 62.5% for PAHF. Continued updating later raises PAHF's native success by 9.9 points [7.5, 12.1] (pointwise 95%), alongside its style gain; CIPHER shows no clear further style gain. (p. 41, lines 2169–2195).
Learning beyond calculation. Both learners train on the same eight support tickets in the same per-seed order, then face four held-out tickets (four users, five seeds; Sonnet 4.6/LangGraph). Astra full matches rise from 20/80 to 45/80 for CIPHER and 55/80 for PAHF; Sonnet's gains are +36.25 and +25.00 points. Correct first submissions remain 80/80 for CIPHER and 79–80/80 for PAHF. Further feedback gives no clear CIPHER gain; PAHF improves further only under Astra. (p. 42, lines 2233–2263).
The scope is now explicit. The title now specifies “Across Agent SDKs.” Figure 1 and §4.1 clarify that seven integrations share one host model; all-controller comparisons remain confined to calculation, and the added studies establish narrower instruction and learning effects beyond it (p. 4, lines 200–211). The SDK axis still matters: accurate instructions raise style matching over no instruction by 8.3 points under CrewAI but 35.5 under LangGraph (Sonnet; 26.3/40.4 under Astra) (p. 9, lines 443–444). On two other hosts, DeepSeek V3.2 shows no detected host × method interaction, whereas MiniMax M2.5 shows a fixed-state style interaction under both judges (pp. 45–46, lines 2428–2442).
To test persistent-memory operations absent from the calculation tasks, we added a cross-task test: recall succeeds on 54/56 tasks with memory versus 0/56 without. This directly demonstrates runtime memory capability, reported separately from the preference-learning experiments above. Clarification evaluation mainly covers the concise profile; the revised manuscript states this scope (p. 8, lines 395–398; p. 31, lines 1650–1664).