- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors investigate the challenge of how long-term conversational agents should adapt their responses to the same standing instruction as its applicability changes with evolving conversational context. To this end, the paper introduces a benchmark specifically designed to evaluate Apply–Withhold–Reapply behavior in long-term conversation histories, incorporating an always-applicable control path for comparison with the reapplication path. The authors also design multiple evaluation metrics, including K, QF, QR, and RTS, to assess model performance across different stages. Experiments are conducted across multiple system configurations, including full-context baselines using large language models and existing agent memory systems. The results show that the strongest full-context baseline achieves the best overall performance, while the highest RTS among the evaluated persistent memory systems under Evidence Return is only 53.4%. These findings highlight the substantial room for improvement in existing memory systems' ability to handle dynamic instruction applicability and provide useful directions for future research.
Critical strengths
1. This paper investigates an important problem in long-term memory systems: the reapplication of standing instructions after temporary non-applicability. The authors construct a dedicated benchmark to evaluate Apply–Withhold–Reapply behavior in long-term conversation histories and introduce a set of evaluation metrics to assess the performance of memory systems. They also evaluate several existing memory-system baselines, with the experimental results highlighting substantial room for improvement in their ability to handle instruction reapplication.
2. By comparing two return formats, Evidence Return and Classification Return, the authors provide further insights into the challenges of instruction reapplication. The experimental results show that even when the current applicability of a historical instruction is explicitly stated, the model's final decision can still be affected by the preceding conversation history. Explicitly providing applicability information does not guarantee that the model can reliably reapply an instruction after a period of temporary non-applicability. This finding highlights an interesting challenge and provides a valuable direction for future research.
Critical weaknesses
1. The novelty of this paper appears somewhat limited. As shown in Table 1, several aspects of the proposed benchmark have already been explored in existing benchmarks, such as LongMemEval and ImplicitMemBench. In long-term conversational histories, the reapplication problem investigated in this paper is not uncommon, as several existing long-term memory benchmarks already cover, to varying degrees, tasks involving evolving events and changes in user states. Therefore, the incremental contribution of the proposed benchmark over existing work could be more clearly justified.
2. The proposed benchmark appears to consider only two fixed applicability trajectories: (1) Always-applicable Control (1→1→1) and (2) Reapplication Path (1→0→1), where SUPPRESS is triggered only at the middle stage of the Reapplication Path. However, in real-world long-term interactions, user states and task conditions may undergo multiple transitions over time, rather than following a fixed three-stage pattern. Therefore, the current task design may not adequately reflect the complexity of real-world scenarios, potentially limiting the coverage and generalizability of the benchmark.
Questions for authors
-
Could the authors further clarify the additional and substantive evaluation value that SIRRA provides compared to existing long-term memory benchmarks?
-
Have the authors considered incorporating more complex and diverse state-transition trajectories, rather than limiting the benchmark to the two fixed settings of Always-applicable Control and Reapplication Path?
-
The letter "f" in Figure 2 appears to be incorrectly rendered (e.g., in "profiles"). Please check and correct this formatting issue.
3 comments
Author Response (2/2): W2/Q2 and Q3
W2 / Q2: Coverage of applicability transitions
The original benchmark pairs a single withholding-and-return path (101) with an always-applicable control (111). As the reviewer points out, real interactions need not follow this pattern: a requirement may remain untriggered across several tasks, or its conditions may change again after the first return. We therefore tested five additional trajectories: 100 and 10000 for sustained non-applicability, 10111 for continued applicability after a return, 10101 for a second withholding-and-return sequence, and 11111 as a longer always-applicable control. Here, 1 means APPLY and 0 means SUPPRESS.
We evaluated all seven paths on the same 27 task families, including six new workflows, using full-context Qwen3.6-27B (approximately 89k–106k input tokens). Families were selected for clear decision rules, suitability for five successive tasks, and workflow diversity. The historical requirement is unchanged throughout each family; new task facts determine the correct decision at each checkpoint. We use fixed conversation histories, without feeding model responses into subsequent checkpoints. Invalid responses count as incorrect.
For example, a printing rule requires a wide-format plotter when sheet width exceeds 42 cm or the paper is textured. Five successive tickets—46 cm smooth, 42 cm smooth, 45 cm smooth, 40 cm smooth, and 39 cm textured—require 10101. Width triggers the first return; texture triggers the second.
| Trajectory | Final checkpoint correct | All checkpoints correct |
|---|---|---|
111 | 27/27 | 21/27 |
100 | 15/27 | 8/27 |
101 | 26/27 | 12/27 |
11111 | 23/27 | 17/27 |
10000 | 15/27 | 4/27 |
10111 | 22/27 | 10/27 |
10101 | 22/27 | 6/27 |
At checkpoint 3, 100 and 101 share the same preceding history and query but differ in the current task facts. Qwen3.6-27B correctly answers APPLY on 101 in 26/27 families, yet also answers APPLY on the matched 100 task in 11 of those families, where SUPPRESS is required. A correct return decision can therefore coexist with failure to distinguish when the same historical requirement warrants action.
The model’s correct initial withholding does not reliably persist across successive non-triggering tasks. On 10000, it correctly answers SUPPRESS at checkpoint 2 in 15 families, but in 10 of these it subsequently answers APPLY at least once while the condition remains unmet.
The 10111 and 10101 paths share the same final task facts and both achieve 22/27 correct final decisions, but complete-trajectory success is 10/27 and 6/27, respectively. Four families complete 10111 correctly but fail only at the fourth-position SUPPRESS decision on 10101. The additional transition exposes failures in withholding without a net decrease in final reapplication accuracy.
Together, these findings reveal a recurring tendency to over-apply still-valid standing instructions: The model resumes or continues to select their prescribed actions even when current task facts require withholding. This failure can follow initially correct restraint or remain hidden behind successful final reapplication.
We will report these trajectories and their stage-level findings in the revision.
Q3: Figure 2 rendering
We will correct the rendering of “f” in Figure 2, including “profiles,” and check the remaining figure text.
I thank the authors for their detailed response and additional experiments. While the rebuttal has alleviated some of my concerns, I still find the novelty and evaluation scope of the paper relatively limited. In particular, I believe the proposed benchmark remains a relatively incremental extension of existing evaluations of long-term memory and behavioral inhibition. Therefore, I am inclined to maintain my original rating.
Author Response (1/2): W1/Q1
We appreciate the reviewer's careful assessment and constructive feedback.
W1 / Q1: Evaluation value beyond existing benchmarks
LongMemEval (Wu et al., 2025) evaluates how conversational information, including updated user facts, supports later answers. ImplicitMemBench (Qin et al., 2026) examines whether acquired procedures and conditioned behaviors, including inhibition, continue to influence behavior after interference. Neither evaluates successive decisions in which the same historical requirement remains valid while its prescribed action alternates between warranted and unwarranted. Consequently, neither can determine whether successful reapplication coexists with inappropriate action during non-applicability. This distinction is absent from their evaluated tasks, not merely from their reported metrics.
SIRRA addresses a broader requirement of reliable memory use: a historical requirement must remain in force without triggering its prescribed action unconditionally, and withholding that action when unwarranted must not prevent its later reapplication. This challenge has practical consequences across different types of assistance:
SIRRA makes this distinction measurable through conditional standing instructions established in earlier conversations. Its paired Control and Reapplication histories share the historical requirement and final task, but differ in whether the intervening task warrants the prescribed action. By evaluating intermediate withholding as well as final reapplication, SIRRA tests whether the same still-valid requirement is followed appropriately under changing task conditions.
This distinction changes how memory systems are evaluated. In the Evidence Return setting, systems must determine applicability from current task evidence. SimpleMem and MemoryOS, using the same Qwen3.6-27B backbone on 479 families, achieve similar final reapplication accuracy (87.9% versus 86.6%). Yet MemoryOS substantially outperforms SimpleMem in intermediate withholding (69.1% versus 50.1%) and complete APPLY–SUPPRESS–APPLY trajectory accuracy (53.4% versus 38.8%). Return accuracy slightly favors SimpleMem, whereas the intervening decision and complete-trajectory results favor MemoryOS.