- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TipsWarm complements on-demand skill retrieval with a small persistent pool of reminders extracted from existing skills. LLM judgments govern tip admission and eviction upon skill events and assess tip utility after task completion, while lightweight scope matching and utility counters select tips for injection at each turn. The method is integrated into Hermes and evaluated on SkillsBench, AppWorld, and text-based ALFWorld, using a warmup phase followed by continued memory updates during testing. TipsWarm improves final success over HermesSkill across the reported settings, with a more pronounced gain in area under the learning curve than in final success on SkillsBench. Ablations examine utility attribution and screening, scope matching, a larger tip pool, and the augmentation of an existing skill bundle.
Critical strengths
-
The method addresses a concrete problem: useful skill guidance may arrive after an agent has committed to an ineffective approach. Its bounded reminder pool reuses existing skills with minimal changes to the underlying workflow. The experiment augmenting an existing skill bundle further supports its practicality.
-
The evaluation spans coding and interactive environments within a common agent framework and includes useful component ablations. The results suggest that utility attribution and screening contribute to the SkillsBench learning-curve gains, although improvements vary across configurations. The paper also reports mixed time and token-cost results, providing a more balanced assessment of efficiency.
Critical weaknesses
The benefit of early guidance could be better isolated from that of adding concise context. Comparisons with matched context budgets, initial-only reminders, and earlier skill retrieval would help clarify the advantage of the proposed scheduling mechanism.
The reliability of utility-based scheduling and its efficiency trade-offs remain to be clarified. The attribution prompt disfavors helpful labels for injected tips in failed episodes, which may not reflect their actual contributions. The joint ablation also leaves attribution’s individual effect unclear. Additionally, runtime is averaged over successful tasks only, and reported agent-token costs exclude internal operations.
Questions for authors
-
How much of the gain comes from early guidance rather than additional context? Comparisons with simpler reminder and early-retrieval baselines would help clarify this.
-
How sensitive are the results to the attribution rules for failed tasks and non-injected tips? A brief attribution audit and more complete runtime and token-cost reporting would help clarify the reliability and efficiency trade-offs.
2 comments
Thank you for the detailed responses and additional experiments. After considering the responses to other reviewers as well, I am willing to raise my rating to 3. However, several weaknesses remain, and my overall recommendation remains below acceptance. I leave the final decision to the AC.
Thanks very much for your valuable comments. Our responses and revisions are provided as follows. We hope that we have addressed all your concerns and welcome further discussion if necessary. We will be grateful if you would kindly raise the rating if you are satisfied.
W1/Q1. About performance of the early skill-retrieval strategy: In the revision, we added an ablation study of proactive early skill-retrieval strategies (in Section 5.3.Ablation Study->Proactive skill retrieval strategy). The results on the SkillsBench are shown in the following table.
As we know, HermesSkill retrieves full skills on demand during the conversation. The Early-Retrieval (HermesSkill-ER) strategy additionally prompts the agent to proactively retrieve a full task-relevant skill into the context at the beginning of the conversation. TipsWarm retrieves tips from the maintained warm-tip pool at every turn. Results show that merely retrieving a full skill early does not obviously improve accuracy when the agent can already retrieve skills on-demand (HermesSkill-ER vs. HermesSkill). In contrast, the warm-tip approach can improve accuracy further by injecting compact guidance that could help avoid ineffective exploration (TipsWarm vs. HermesSkill-ER and HermesSkill).
W2/Q2. About sensitivity of utility attribution rules: Thanks for your suggestion of a sensitivity study of the utility attribution rules. In the revision, we have added a sensitivity study of utility rules in Section 5.3 “Ablation Study->Sensitivity of utility rules”. Table 3 shows the accuracy of different combinations of two utility-attribution rules on SkillsBench. F and N can match the best default seed at 87.0% final success, while their AUCs are lower (0.491 and 0.440, against 0.533) than the default setting, which represents a better attribution for early guidance. As we observe from the trajectories, the default setting opens with a grader constraint and receives some harmful counts on failed tasks. F exempts a path-specific tip from harmful labels in failed tasks, and N leaves two unexposed evaluator-specific tips free of an irrelevant count. A combination of F and N can have both negative effects and drops the final success rate to 78.3% and AUC to 0.420. These early-guidance benefits of the default utility attribution rules can also be inferred from their lower token costs and fewer user message turns than the other variants. In the revision, for brevity, we include only the accuracy results in Table 3.
Table 3. Accuracy and cost results of utility-attribution sensitivity on SkillsBench. The default setting of TipsWarm prohibits helpful labels for injected tips in failed tasks and counts non-injected tips as irrelevant. F allows helpful labels in failed tasks when execution evidence supports a useful contribution, while retaining harmful and irrelevant labels when appropriate. N leaves the utility counters of non-injected tips unchanged. F+N combines both changes. F, N, and F+N each have one run.