- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BOOKMARKS, a search-based memory framework for role-playing agents that combines active grounding with checkpoint reuse. Given the current scene, the framework proposes relevant information sections, reuses semantically equivalent bookmarks or initializes new ones, and synchronizes their contents by processing only the storyline suffix after each stored checkpoint. Concept, state, and behavioral bookmarks provide entity descriptions, state-transition traces, and scene–action examples, respectively. Experiments across six narrative artifacts show improvements over retrieval-based and profiling baselines in next-action prediction, while accuracy on 284 human-authored reasoning questions increases from 39.44% to 45.42% over the strongest baseline. Additional analyses report benefits across model families and evaluation dimensions, with checkpoint reuse maintaining comparable prediction quality while reducing repeated updates and output-token costs.
Critical strengths
- BOOKMARKS combines task-dependent information selection with checkpoint reuse in a coherent framework, addressing both information loss in fixed profiles and redundant processing during active grounding.
- Consistent gains across narrative artifacts and model families, supported by human evaluation and human-authored reasoning questions, provide meaningful evidence of practical effectiveness.
Critical weaknesses
-
This paper employs GPT-4.1 as the Large Language Model (LLM) judge; however, relying on a single LLM judge may introduce bias. It is recommended to use a combination of models from different model families for evaluation.
-
Regarding human evaluation, the paper should clearly specify details concerning NAP validation and open-domain evaluation, including participant recruitment, informed consent, compensation arrangements, and relevant ethical review or exemption status. Furthermore, the paper does not disclose the number of human evaluators, the number of annotations per sample, or inter-annotator agreement.
Questions for authors
When grounding a target character, can BOOKMARKS retrieve events in the preceding storyline that the character has not witnessed or learned about? If so, how are such events handled in the reported role-playing evaluations?
3 comments
Dear reviewer, we fully respect your recommendation and suggestions on our work. We hope our response has addressed your concerns and will stay active until the final day of the rebuttal period. Please do not hesitate to raise any further questions or share any additional thoughts.
Thank you for the detailed response and additional clarifications. My concerns have been largely addressed. Since my original rating already reflected my positive assessment of the paper’s contributions and empirical results, I will maintain my score of Weak Accept.
Author's Response: Evaluation Reliability and Human Annotation & Character Knowledge and Grounding Scope
Thank you for your thoughtful review and constructive suggestions. We appreciate your recognition of the coherent design of BOOKMARKS and its consistent improvements across narrative artifacts and model families. We have added validation with a judge from another model family and clarified the human annotation protocol and character-specific grounding mechanism. We have uploaded a revised version and these additions and clarifications are marked in blue in the manuscript.
Evaluation Reliability and Human Annotation (W1, W2)
We additionally evaluated the same 200 cases per dimension (1000 in total) using Claude Sonnet 4, obtaining 92.1% agreement with GPT-4.1. This cross-family consistency complements the human validation reported in the paper and provides further evidence for the reliability of our evaluation.
Our human evaluation involved three judges familiar with the evaluated artifacts. For NAP validation, each case was assessed by one judge against the ground-truth reference, making the task relatively direct and limiting subjective interpretation. For open-domain evaluation, each sample was assessed by all three judges, with the final judgment determined by majority vote. The judges were blind to the source model in this evaluation. Participant recruitment, informed consent, compensation, and ethics review or exemption followed the applicable institutional procedures.
Appendix D now details both the cross-family judge comparison and the human annotation protocol, with a reference from the evaluation description in Section 4.
Character Knowledge and Grounding Scope (Q1)
BOOKMARKS can consult the complete preceding storyline when grounding a character. During state updates, the update rules instruct the model to skip events that the target character has neither witnessed nor learned about. Such events are therefore not automatically incorporated as the character’s observations. Incremental updating supports this distinction by following when information becomes available to the character and how it changes their state, preserving the character’s perspective as the story unfolds. We have made this character-specific update rule explicit in Section 3.4.
Thank you again for helping us make the evaluation and grounding protocol clearer. We hope you find the corresponding explanations in the revised manuscript helpful as you revisit these points. We fully respect the time and effort you have devoted to this review and will remain actively engaged throughout the entire discussion period. We would be glad to hear your further thoughts and work through any remaining questions with you.