acceptodds
Under review as a conference paper at ICLR 2027

SIRRA: Matched Evaluation of Standing-Instruction Reapplication after Temporary Non-Applicability

Abstract

Long-term assistants often receive a standing instruction once, encounter a situation in which it should not trigger, and later receive new evidence that should trigger it again. We introduce SIRRA (Standing Instruction Reapplication and Return Assessment), a benchmark for this apply–withhold–reapply behavior in long conversational histories. Each case pairs a history in which the condition remains true with a matched history in which it temporarily becomes false; both histories share the instruction, the initial and final situations, and the decision question. We score the initial action, the temporary non-action, and the later return without repeating the instruction or stating whether it applies. Across 479 cases and nine system configurations, end-to-end success across all three decisions conceals different stage bottlenecks: retrieval systems often fail before the return, while memory systems vary in whether they lose initial application, temporary restraint, or reapplication. Among cases that get the first two decisions right, return-stage outcomes vary across configurations and return forms. Explicitly stating whether the final condition holds does not uniformly improve the return and can widen the gap between the matched histories. These results show why memory evaluations should separate an instruction's continuing validity from its current applicability and report initial use, appropriate non-use, and renewed use together.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Conversational Memory, Long-term Memory, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ciJwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate the challenge of how long-term conversational agents should adapt their responses to the same standing instruction as its applicability changes with evolving conversational context. To this end, the paper introduces a benchmark specifically designed to evaluate Apply–Withhold–Reapply behavior in long-term conversation histories, incorporating an always-applicable control path for comparison with the reapplication path. The authors also design multiple evaluation metrics, including K, QF, QR, and RTS, to assess model performance across different stages. Experiments are conducted across multiple system configurations, including full-context baselines using large language models and existing agent memory systems. The results show that the strongest full-context baseline achieves the best overall performance, while the highest RTS among the evaluated persistent memory systems under Evidence Return is only 53.4%. These findings highlight the substantial room for improvement in existing memory systems' ability to handle dynamic instruction applicability and provide useful directions for future research.

Critical strengths

1. This paper investigates an important problem in long-term memory systems: the reapplication of standing instructions after temporary non-applicability. The authors construct a dedicated benchmark to evaluate Apply–Withhold–Reapply behavior in long-term conversation histories and introduce a set of evaluation metrics to assess the performance of memory systems. They also evaluate several existing memory-system baselines, with the experimental results highlighting substantial room for improvement in their ability to handle instruction reapplication.

2. By comparing two return formats, Evidence Return and Classification Return, the authors provide further insights into the challenges of instruction reapplication. The experimental results show that even when the current applicability of a historical instruction is explicitly stated, the model's final decision can still be affected by the preceding conversation history. Explicitly providing applicability information does not guarantee that the model can reliably reapply an instruction after a period of temporary non-applicability. This finding highlights an interesting challenge and provides a valuable direction for future research.

Critical weaknesses

1. The novelty of this paper appears somewhat limited. As shown in Table 1, several aspects of the proposed benchmark have already been explored in existing benchmarks, such as LongMemEval and ImplicitMemBench. In long-term conversational histories, the reapplication problem investigated in this paper is not uncommon, as several existing long-term memory benchmarks already cover, to varying degrees, tasks involving evolving events and changes in user states. Therefore, the incremental contribution of the proposed benchmark over existing work could be more clearly justified.

2. The proposed benchmark appears to consider only two fixed applicability trajectories: (1) Always-applicable Control (1→1→1) and (2) Reapplication Path (1→0→1), where SUPPRESS is triggered only at the middle stage of the Reapplication Path. However, in real-world long-term interactions, user states and task conditions may undergo multiple transitions over time, rather than following a fixed three-stage pattern. Therefore, the current task design may not adequately reflect the complexity of real-world scenarios, potentially limiting the coverage and generalizability of the benchmark.

Questions for authors
  1. Could the authors further clarify the additional and substantive evaluation value that SIRRA provides compared to existing long-term memory benchmarks?

  2. Have the authors considered incorporating more complex and diverse state-transition trajectories, rather than limiting the benchmark to the two fixed settings of Always-applicable Control and Reapplication Path?

  3. The letter "f" in Figure 2 appears to be incorrectly rendered (e.g., in "profiles"). Please check and correct this formatting issue.

3 comments
AuthorsOpenReview ↗

Author Response (1/2): W1/Q1

We appreciate the reviewer's careful assessment and constructive feedback.

W1 / Q1: Evaluation value beyond existing benchmarks

LongMemEval (Wu et al., 2025) evaluates how conversational information, including updated user facts, supports later answers. ImplicitMemBench (Qin et al., 2026) examines whether acquired procedures and conditioned behaviors, including inhibition, continue to influence behavior after interference. Neither evaluates successive decisions in which the same historical requirement remains valid while its prescribed action alternates between warranted and unwarranted. Consequently, neither can determine whether successful reapplication coexists with inappropriate action during non-applicability. This distinction is absent from their evaluated tasks, not merely from their reported metrics.

SIRRA addresses a broader requirement of reliable memory use: a historical requirement must remain in force without triggering its prescribed action unconditionally, and withholding that action when unwarranted must not prevent its later reapplication. This challenge has practical consequences across different types of assistance:

  • Enterprise workflows: An assistant instructed to anonymize customer information in externally shared reports should anonymize the first report, retain identifying details during authorized internal troubleshooting, and anonymize the next external report. Unnecessary anonymization can hinder diagnosis, while failure to reapply it risks exposing private information.
  • Personalized travel recommendations: An assistant instructed to prioritize hiking-friendly mountain destinations for a user's vacations should apply this preference when planning the user's trip, avoid imposing it when planning for the user's parents, and apply it again for the user's next vacation. Over-application mispersonalizes others' recommendations, while failure to reapply undermines long-term personalization.
  • Everyday dietary assistance: An assistant instructed to exclude peanuts when planning meals for a family member with a peanut allergy should respect this requirement for that person, avoid unnecessarily restricting separate meals for non-allergic family members, and enforce it again when planning for the allergic person. The two failures carry different consequences: unnecessary restrictions and potential health risks.

SIRRA makes this distinction measurable through conditional standing instructions established in earlier conversations. Its paired Control and Reapplication histories share the historical requirement and final task, but differ in whether the intervening task warrants the prescribed action. By evaluating intermediate withholding as well as final reapplication, SIRRA tests whether the same still-valid requirement is followed appropriately under changing task conditions.

This distinction changes how memory systems are evaluated. In the Evidence Return setting, systems must determine applicability from current task evidence. SimpleMem and MemoryOS, using the same Qwen3.6-27B backbone on 479 families, achieve similar final reapplication accuracy (87.9% versus 86.6%). Yet MemoryOS substantially outperforms SimpleMem in intermediate withholding (69.1% versus 50.1%) and complete APPLY–SUPPRESS–APPLY trajectory accuracy (53.4% versus 38.8%). Return accuracy slightly favors SimpleMem, whereas the intervening decision and complete-trajectory results favor MemoryOS.

AuthorsOpenReview ↗

Author Response (2/2): W2/Q2 and Q3

W2 / Q2: Coverage of applicability transitions

The original benchmark pairs a single withholding-and-return path (101) with an always-applicable control (111). As the reviewer points out, real interactions need not follow this pattern: a requirement may remain untriggered across several tasks, or its conditions may change again after the first return. We therefore tested five additional trajectories: 100 and 10000 for sustained non-applicability, 10111 for continued applicability after a return, 10101 for a second withholding-and-return sequence, and 11111 as a longer always-applicable control. Here, 1 means APPLY and 0 means SUPPRESS.

We evaluated all seven paths on the same 27 task families, including six new workflows, using full-context Qwen3.6-27B (approximately 89k–106k input tokens). Families were selected for clear decision rules, suitability for five successive tasks, and workflow diversity. The historical requirement is unchanged throughout each family; new task facts determine the correct decision at each checkpoint. We use fixed conversation histories, without feeding model responses into subsequent checkpoints. Invalid responses count as incorrect.

For example, a printing rule requires a wide-format plotter when sheet width exceeds 42 cm or the paper is textured. Five successive tickets—46 cm smooth, 42 cm smooth, 45 cm smooth, 40 cm smooth, and 39 cm textured—require 10101. Width triggers the first return; texture triggers the second.

TrajectoryFinal checkpoint correctAll checkpoints correct
11127/2721/27
10015/278/27
10126/2712/27
1111123/2717/27
1000015/274/27
1011122/2710/27
1010122/276/27

At checkpoint 3, 100 and 101 share the same preceding history and query but differ in the current task facts. Qwen3.6-27B correctly answers APPLY on 101 in 26/27 families, yet also answers APPLY on the matched 100 task in 11 of those families, where SUPPRESS is required. A correct return decision can therefore coexist with failure to distinguish when the same historical requirement warrants action.

The model’s correct initial withholding does not reliably persist across successive non-triggering tasks. On 10000, it correctly answers SUPPRESS at checkpoint 2 in 15 families, but in 10 of these it subsequently answers APPLY at least once while the condition remains unmet.

The 10111 and 10101 paths share the same final task facts and both achieve 22/27 correct final decisions, but complete-trajectory success is 10/27 and 6/27, respectively. Four families complete 10111 correctly but fail only at the fourth-position SUPPRESS decision on 10101. The additional transition exposes failures in withholding without a net decrease in final reapplication accuracy.

Together, these findings reveal a recurring tendency to over-apply still-valid standing instructions: The model resumes or continues to select their prescribed actions even when current task facts require withholding. This failure can follow initially correct restraint or remain hidden behind successful final reapplication.

We will report these trajectories and their stage-level findings in the revision.

Q3: Figure 2 rendering

We will correct the rendering of “f” in Figure 2, including “profiles,” and check the remaining figure text.

I thank the authors for their detailed response and additional experiments. While the rebuttal has alleviated some of my concerns, I still find the novelty and evaluation scope of the paper relatively limited. In particular, I believe the proposed benchmark remains a relatively incremental extension of existing evaluations of long-term memory and behavioral inhibition. Therefore, I am inclined to maintain my original rating.

Reviewer_rK6SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the LLM's ability to re-apply a standing instruction after a period where it should not be triggered, arguing that scoring only the final decision hides where memory systems fail. The authors construct SIRRA, paired persona-grounded synthetic work conversations where the first and the last observation requires the action and the middle requires withholding, against an always-action control setup.Authors report that explicitly stating the applicability does not help the reapplication. While standing instructions have been explored in prior work (e.g., NLSI, which is missing in the paper), the primary distinction here is the apply-withhold-reapply evalution framework with a controlled counterpart.

I think the research topic is getting more relevant as assistants are increasingly delegated with recurring and long-context work (Meta Muse, OpenAI dots, Grokbot), and the paired construction allows for controled scientific evaluation.

However, what is measured does not fully match the claimed behavior. The model's own decisions are never fed back to the history, so "withhold" is only a change in observation, not an action taken. The evidence return should require inferring applicability, but the generated evidences can spell out the threshold comparison, so the distinction from the classification return is not so clear; we would need to see a baseline given only the last message and the query. With the classification "control" changing the outcome and the filler possibly differing between the matched histories, there remains a question of "controlled relative to what?" The benchmark has high potential with more variable controls nevertheless, such as middle turn position or dialogue length.

Critical strengths
  • A practically motivated behavior that existing memory benchmarks do not isolate.
  • Matched-history design is a promising template, hinting at where each memory framework loses the trajectory.
Critical weaknesses
  • The claimed behavior is not rigorously tested, with no variable controls such as middle turn position or dialogue length.
  • The gap is marginal for frontier models and all memory frameworks share one backbone, suggesting the issue is largely a retrieval weakness. This reduces the significance of the findings.
  • Hard to follow a bunch of proxy metrics, and the results reporting have little interpretation.
Questions for authors
  • L316: where does 86.6% come from?
  • Do you have the full-context Qwen3.6-27B result or the context doesn't fit?
  • How about the performance of a baseline with only the last message and the query as input?
  • I'm most interested in the change in results with different middle position, or the history length itself. I'd be nice to see such analysis.
5 comments
AuthorsOpenReview ↗

Author Response (1/3): W1/Q4 and W2

We thank the reviewer for the careful reading and constructive feedback.

W1 / Q4: Middle position and history length

We evaluated full-context Qwen3.6-27B on 27 matched conversation families to test the effects of Middle position and history length. Each begins with a standing instruction and a task requiring action. We constructed two histories: Control also requires action at Middle, while Reapplication requires withholding it; both end with the same task requiring action. The paired histories differ only in the Middle user message and scripted assistant reply, with filler held identical rather than allowed to differ as in the original benchmark. We placed Middle at approximately 25% or 75% of the interval between the initial and final tasks, and used short or long histories with median final-input lengths of approximately 53k or 107k tokens, respectively. At each checkpoint, we directly scored the model's APPLY or SUPPRESS decision against the fixed history, without feeding its response into later checkpoints.

ConfigurationMiddle: correct SUPPRESSControl: final APPLYReapplication: final APPLY
Short / Early22/2727/2727/27
Short / Late20/2727/2727/27
Long / Early25/2726/2727/27
Long / Late24/2726/2727/27

These are descriptive differences in a selected 27-family panel and do not establish a general position or history-length effect. All 108 Reapplication final decisions were correct, despite 16 erroneous APPLY decisions and one invalid output at Middle.

To further examine which applicability transitions produce incorrect decisions, we used a separate, extended 27-family panel. The initial task always required APPLY; we then varied whether the instruction required APPLY or SUPPRESS at Middle and at the final task. Within each Middle condition, the final observation changed while the preceding history and query remained fixed (1 = APPLY, 0 = SUPPRESS).

TrajectoryMiddle taskFinal taskFinal correct
111APPLYAPPLY27/27
101SUPPRESSAPPLY26/27
110APPLYSUPPRESS13/27
100SUPPRESSSUPPRESS15/27

When the final task required APPLY, the model made 53/54 correct decisions, with one invalid output. When it required SUPPRESS, the model incorrectly chose APPLY in 26/54 cases. In 11 families, this error occurred in both 110 and 100, showing that incorrect action was not confined to histories with prior non-applicability.

Together, these experiments reveal a more specific failure pattern: incorrect APPLY decisions occur when the current task requires SUPPRESS, even when the preceding Middle task required APPLY. A prior period of non-applicability was therefore not necessary for these observed failures, while the position and length controls showed no disadvantage at Reapplication Return. In these tested settings, the difficulty lies in correctly restricting when a standing instruction triggers action, rather than in resuming its prescribed action after a non-applicable task.

W2: Return gaps and retrieval

The Evidence Return gaps for GPT-5.4 and Gemini 3.1 Pro are 2.3 and 3.1 percentage points, respectively. These models also correctly withhold action at Middle in 463/479 (96.7%) and 467/479 (97.5%) cases, respectively, showing strong Middle performance in the original setting. Both Control and Reapplication Return require APPLY, so the gap compares final-decision accuracy between the two histories. SIRRA's additional value is in evaluating action, withholding, and reapplication together, which can distinguish memory configurations with similar Return accuracy.

Under the Evidence condition, SimpleMem slightly outperforms MemoryOS at Reapplication Return (421/479 vs. 415/479), yet MemoryOS substantially outperforms SimpleMem when all three decisions must be correct (256/479 vs. 186/479).

All memory frameworks use Qwen3.6-27B as their answer backbone. Retrieval may account for much of the performance difference between memory systems, and our comparisons do not isolate its contribution. However, the SimpleMem–MemoryOS results establish a distinct evaluation finding: similar success at final reapplication does not imply similar reliability in following the same standing instruction across tasks where its prescribed action is and is not warranted. Indeed, the system with slightly higher Return accuracy performs substantially worse across the complete trajectory. This difference matters for evaluating memory systems regardless of which components produce it.

AuthorsOpenReview ↗

Author Response (2/3): W3 and Q1–Q2

W3: Metrics and result interpretation

The primary outcomes are accuracy at each checkpoint and the proportion of families with all three Reapplication decisions correct (RTS). These show whether the model correctly chooses APPLY or SUPPRESS as task conditions change.

The other metrics summarize specific comparisons. K identifies families with correct Initial and Middle decisions on the Reapplication path; QF and QR count cases within this subset where only Control or only Reapplication Return is correct, respectively. RPG compares the two Return accuracies, PSC compares this gap across Evidence and Classification prompts, and PRG measures the change in Reapplication Return accuracy after adding supplementary information, relative to the unmodified replay.

For example, MemoryOS and SimpleMem have similar Reapplication Return accuracy, but correctly withhold action at Middle in 331/479 and 240/479 families, respectively. This exposes a substantial difference in restraint that the final scores obscure. We will organize the results around such behavioral comparisons and explain each derived metric alongside the question it answers.

Q1: Source of the 86.6% result

The 86.6% refers to Evidence Reapplication Return accuracy. Both MemoryOS and Gemini 3.1 Pro correctly answered 415/479 decisions (86.6%) at this checkpoint (see Section 4.1, “Stage-Specific Outcome Profiles”).

Q2: Full-context Qwen3.6-27B baseline

We added a full-context Qwen3.6-27B baseline by evaluating the model on all 479 families with complete checkpoint histories.

The table compares Reapplication results under Evidence Return on the same 479 families. All three configurations use Qwen3.6-27B as the answer model.

ConfigurationInitial: correct APPLYMiddle: correct SUPPRESSReturn: correct APPLYAll three decisions correct
Full context (new)458/479 (95.6%)420/479 (87.7%)459/479 (95.8%)396/479 (82.7%)
SimpleMem441/479 (92.1%)240/479 (50.1%)421/479 (87.9%)186/479 (38.8%)
MemoryOS389/479 (81.2%)331/479 (69.1%)415/479 (86.6%)256/479 (53.4%)

Even with complete checkpoint histories, full-context Qwen3.6-27B achieves 95.6% accuracy at Initial and 95.8% at Return, but only 87.7% at Middle. Of the 83 unsuccessful full-context trajectories, 59 included a failure at Middle. This shows that access to the complete history does not ensure conditional adherence: the answer model still struggles to withhold a prescribed action when the historical requirement remains valid but its condition is unmet. The two memory systems show substantially larger deficits at Middle (50.1% and 69.1%), despite achieving 87.9% and 86.6% at Return. For both memory systems, the gap to full context was largest at Middle.

In the revised paper, we will report these results in Section 4.1 (Stage-Specific Outcome Profiles) and add the full-context Qwen3.6-27B baseline to Table 2.

AuthorsOpenReview ↗

Author Response (3/3): Q3 Last-message baseline

Q3: Last-message baseline

We evaluated Qwen3.6-27B on 115 decision points from a selected 30-family panel, varying the available conversation history while keeping the query unchanged.

InputCorrect decisions
Last message + query84/115 (73.0%)
Opening user message + last message + query110/115 (95.7%)
Full history + query109/115 (94.8%)
Always-APPLY reference87/115 (75.7%)

Although the last-message baseline and always-APPLY achieved similar overall accuracy, their decisions differed: the former correctly selected SUPPRESS in 18/28 cases requiring withholding, compared with 0/28 for always-APPLY. Thus, the final message alone provides some information relevant to applicability, but does not achieve full-history accuracy.

Some Evidence observations already include threshold comparisons, while Evidence and Classification also differ in wording and accompanying requests. We will therefore treat their score differences as comparisons between input formats, not as an isolated effect of explicitly stating applicability.

Adding the complete opening user message increased accuracy from 84/115 to 110/115, closely matching full history (109/115). The opening message includes both the standing instruction and task background, so their separate contributions cannot be isolated. On this panel, providing the opening requirement and background alongside the current observation and query was sufficient to achieve accuracy comparable to full history; including the intervening dialogue yielded no further aggregate accuracy gain.

We will also add NLSI (Moghe et al., 2024) to Related Work. NLSI evaluates the applicability of standing instructions to current requests and their interpretation as API calls; SIRRA examines correct action, withholding, and reapplication under the same historical requirement across successive tasks.

Reviewer_vpufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes a new problem SIRRA (Standing Instruction Reapplication and Return Assessment), that the instruction will hold from a while untill meet the requirement to trigger it. It is a valuable research topic but kind of narrow.

Critical strengths
  1. Existing memory benchmarks cover long-horizon recall, implicit use, and memory-grounded action, but none separates an instruction's continuing validity from its current applicability. The protocol-coverage table makes this gap obvious.
  2. Each task family pairs two histories that share a standing instruction, a query, and the initial and return observations. Within a family, the always-applicable control and the reapplication path share the instruction, query, and initial/return observations, differing only in the middle observation. This makes "control return accuracy − reapplication return accuracy" an attributable contrast.
Critical weaknesses

1.The research topic is valuable but kind of narrow. 2. The construction data is limited. One non-applicability interval and one instruction per family; no revision, revocation, interacting instructions, or repeated switches. Label imbalance leaves restraint thinly assessed. No live interaction, no self-influencing history, no tool execution; this tests judgment more than agent behavior. 3. Nine configurations omit standard end-to-end long-context agents and commercial assistants; only one applicability-cue condition is tested.

Questions for authors

more frontier long-context agents and commercial assistants are valuable to compare.

4 comments
AuthorsOpenReview ↗

Author Response (1/3): W1

We appreciate the reviewer's careful assessment and constructive feedback.

W1: Scope of the research question

We respectfully disagree that the research question is narrow. SIRRA examines a fundamental challenge in memory-guided agent behavior: a historical requirement may remain in force while its prescribed action is warranted in some tasks and unwarranted in others. Remembering the requirement must not lead to executing its prescribed action unconditionally; withholding that action when its condition is unmet must not prevent the requirement from being followed again later. Reliable memory use requires maintaining both the historical requirement and its conditional relationship to the current task. This challenge arises across persistent user preferences, workflow requirements, and agent operating constraints:

  • Enterprise workflows: Anonymize customer information in an external report, retain identifiers for authorized internal troubleshooting, then anonymize the next external report. Unnecessary anonymization hinders diagnosis; failure to reapply risks privacy exposure.
  • Travel personalization: Prioritize hiking-friendly destinations for the user's vacation, not for a trip planned for their parents, then again for the user's next vacation. Over-application mispersonalizes others' recommendations; failure to reapply undermines personalization.
  • Dietary assistance: Exclude peanuts when planning meals for an allergic family member, not for non-allergic members, then again for the allergic person. Unnecessary restrictions limit choices; failure to reapply poses health risks.

By holding one historical requirement fixed while varying the current task facts, SIRRA provides a controlled instantiation of this broader memory-use challenge, rather than restricting the research question to any particular workflow.

Recent surveys emphasize that agent memory must support not only the retention and management of past information but also its reliable use in subsequent behavior (Luo et al., 2026; Tang et al., 2026). LongMemEval (Wu et al., 2025) evaluates long-term memory through question answering, while ImplicitMemBench (Qin et al., 2026) examines learned behavioral adaptation. More closely, NLSI (Moghe et al., 2024) evaluates which standing instructions apply to a given request, and MemPABench (Li et al., 2026) measures context-sensitive preference selection and behavioral realization. SIRRA's distinct contribution is to evaluate whether an agent can withhold the prescribed action of a still-valid historical requirement when its condition is unmet, yet reapply that action when the condition returns, without treating the requirement as unconditional or abandoned. In the paired design, Control keeps the prescribed action applicable throughout, whereas Reapplication makes it applicable, then inapplicable, then applicable again. Both histories share the standing instruction, query, and initial and final task observations, enabling comparison at the same final applicable task.

This distinction has measurable consequences. Under the Evidence condition, SimpleMem and MemoryOS, both using Qwen3.6-27B, achieve similar Reapplication Return accuracy: 421/479 (87.9%) and 415/479 (86.6%), respectively. Yet their accuracy at correctly withholding the prescribed action at Middle differs substantially: 240/479 (50.1%) versus 331/479 (69.1%). Their complete APPLY–SUPPRESS–APPLY trajectory accuracy also differs: 186/479 (38.8%) versus 256/479 (53.4%). Thus, similar success at reapplication can mask substantial differences in conditional adherence: systems that act correctly when a requirement applies may differ sharply in whether they withhold the same action when it does not.

AuthorsOpenReview ↗

Author Response (2/3): W2

W2: Task coverage and evaluation design

We extended the original three-stage setting to seven applicability trajectories across 27 selected task families, including six new workflows, using full-context Qwen3.6-27B with approximately 89k–106k input tokens. These trajectories include sustained non-applicability (100, 10000) and repeated switches (10101). Here, 1/0 indicates whether the prescribed action is warranted at a checkpoint; the historical requirement remains valid throughout. Each checkpoint is evaluated against a fixed conversation history; evaluated responses are not fed into subsequent checkpoints.

Counting shared checkpoint inputs once, the extended evaluation includes 243 APPLY and 135 SUPPRESS instances, with accuracy of 217/243 (89.3%) and 75/135 (55.6%), respectively. The 135 withholding decisions span positions 2–5. Entire-trajectory success requires every decision to be correct, and invalid responses count as incorrect.

TrajectoryFinal correctEntire trajectory correct
11127/2721/27
10015/278/27
10126/2712/27
1111123/2717/27
1000015/274/27
1011122/2710/27
1010122/276/27

First, 100 and 101 share the same preceding history and query and differ only in the current task facts at checkpoint 3. Although 101 achieves 26/27 correct final reapplications, 11 of those families also incorrectly answer APPLY at the corresponding final position of 100, where the action remains unwarranted. A correct reapplication decision can therefore coexist with failure to distinguish when the same historical requirement warrants action.

Second, under sustained non-applicability (10000), 15 families initially withhold correctly at checkpoint 2, but 10 of them subsequently answer APPLY when withholding is required. Correct restraint at one checkpoint does not establish that the requirement's conditional scope will be respected at later checkpoints.

Third, 10111 and 10101 share the same final task facts and attain identical final accuracy (22/27), yet their complete-trajectory success differs (10/27 versus 6/27). Four families complete 10111 correctly but fail only at the fourth-position SUPPRESS decision on 10101. Repeated changes in applicability expose additional failures in withholding that final-decision accuracy alone does not capture.

Together, these findings reveal a recurring tendency of Qwen3.6-27B to over-apply still-valid standing instructions: the model answers APPLY even when the current task facts require withholding the prescribed action. Successful reapplication and isolated correct withholding can therefore coexist with repeated failures to respect the conditional scope of the same historical requirement. The extended trajectories expose this behavioral failure across applicability conditions that the original three-stage setting does not fully cover.

Keeping the historical requirement unchanged is an intentional control for studying this behavior. Revision and revocation concern which requirement remains in force, while interacting instructions introduce questions of conflict and priority. Even when a requirement remains valid and encounters no competing instructions, an agent must still determine whether its prescribed action is warranted by the current task. By holding the requirement fixed and varying its applicability, SIRRA directly evaluates this capability without making changes in instruction validity or priority part of the explanation.

The decision to perform or withhold the prescribed action is itself the memory-conditioned behavior under evaluation. External tool execution evaluates a further aspect of agent behavior: whether the selected action is successfully carried out, which also depends on tool arguments, execution reliability, and environmental conditions. Such execution can be evaluated separately, but it is not necessary to observe or assess whether the historical requirement correctly governs the agent's action choice.

Fixed histories serve the same measurement objective by preventing earlier decisions from altering the later task evidence. The Control and Reapplication histories share the standing instruction, query, and initial and final task observations, preserving a matched Return comparison across different intervening applicability conditions. Self-influencing interactions could introduce additional action-dependent changes to subsequent observations, requiring further controls to distinguish their consequences from the memory-conditioned decisions being studied. SIRRA provides a controlled test of a specific requirement of reliable agent memory: a historical instruction must remain available and authoritative without triggering unwarranted action, and must still govern action when its applicability returns.

AuthorsOpenReview ↗

Author Response (3/3): W3

W3 / Q1: System coverage and applicability cues

System coverage. Our evaluation includes GPT-5.4 and Gemini 3.1 Pro as frontier commercial-model baselines with full-context access, alongside Qwen3.6-27B and dedicated memory systems. These model-level baselines are not evaluations of deployed commercial assistants. We additionally evaluated the upstream NanoMuse agent package through its programmatic interface, retaining native remember, recall, and forget operations. This adds a native memory-agent architecture to our comparison; evaluating further agent architectures and deployed commercial assistants remains future work.

Matched 479-family comparison. All four configurations below use Qwen3.6-27B on the same 479 families. Initial and Middle results are shared across Evidence (E) and Classification (C) Return formats; All denotes complete Reapplication trajectories. Each cell reports the number correct out of 479 (accuracy).

ConfigurationInitialMiddleE-ReturnE-AllC-ReturnC-All
Full context458 (95.6%)420 (87.7%)459 (95.8%)396 (82.7%)433 (90.4%)370 (77.2%)
SimpleMem441 (92.1%)240 (50.1%)421 (87.9%)186 (38.8%)357 (74.5%)141 (29.4%)
MemoryOS389 (81.2%)331 (69.1%)415 (86.6%)256 (53.4%)348 (72.7%)220 (45.9%)
NanoMuse432 (90.2%)294 (61.4%)405 (84.6%)207 (43.2%)326 (68.1%)164 (34.2%)

NanoMuse scores lower than SimpleMem at Return but higher on complete trajectories under both E and C. Its Middle withholding accuracy (61.4%) also exceeds SimpleMem's (50.1%), though it trails MemoryOS (69.1%). Thus, Return accuracy alone obscures meaningful differences in conditional adherence across the evaluated memory architectures.

Applicability-cue intervention. With full-context Qwen3.6-27B on a selected 30-family panel, 112 of 120 candidate checkpoints were eligible for a controlled intervention because their observations supported unambiguous applicability judgments. We compared a fact-preserving control (F0) with an applicability-judgment-augmented condition (F1). The control retained the preceding history, query, and current task facts while removing explicit applicability comparisons and some auxiliary wording. The augmented condition differed only by appending a correct judgment of whether the standing instruction's trigger condition was satisfied, without providing the APPLY/SUPPRESS answer.

Accuracy increased from 101/112 (90.2%) to 110/112 (98.2%): +8.0 percentage points (95% family-cluster bootstrap CI: +1.7 to +15.9). The cue corrected five unwarranted SUPPRESS answers to APPLY and four unwarranted APPLY answers to SUPPRESS; withholding improved from 24/28 to 28/28. One invalid answer became correct, while one correct APPLY became invalid. The explicit judgment improved both warranted action and warranted restraint despite the preceding history and current facts being available. This includes positive and negative applicability judgments, not merely cues favoring APPLY.

The original Evidence format primarily provides current task facts (sometimes with explicit condition comparisons), whereas Classification explicitly supplies the applicability classification. These formats also differ in wording and accompanying requests. Their E/C scores above are therefore descriptive format comparisons, not cue-only effects or directly equivalent to F0/F1.

Additional long-context stress test. On the same 27 families used in W2, we compared original histories (327–707 tokens) with filler-extended histories (approximately 89k–106k tokens), preserving task dialogue and queries. Across 378 requests per condition, accuracy fell from 327/378 (86.5%) to 292/378 (77.2%). APPLY fell from 228/243 (93.8%) to 217/243 (89.3%), while SUPPRESS declined more sharply, from 99/135 (73.3%) to 75/135 (55.6%). Under this long-context stress condition, withholding deteriorated substantially more than applying the requirement.

Area_Chair_VKR5OpenReview ↗

take part in the discussion

Dear reviewer vpuf,

pleae take part in the discussion. time is running out.

thank you your ac

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Summary for the Area Chair

Dear Area Chair,

SIRRA examines a fundamental challenge in memory-guided agent behavior: a historical requirement may remain in force while its prescribed action is warranted in some tasks and unwarranted in others. Reliable memory use requires maintaining both the historical requirement and its conditional relationship to the current task. Remembering the requirement must not lead to unconditional execution of its prescribed action; withholding that action when unwarranted must not prevent its later reapplication.

LongMemEval evaluates how conversational information supports later answers, while ImplicitMemBench examines procedural and conditioned behaviors after interference. NLSI evaluates standing-instruction applicability and its realization in API calls, while MemPABench examines context-sensitive preference selection, behavioral realization, and preference changes over time.

SIRRA makes the conditional use of the same continuously valid historical requirement across changing tasks the unit of evaluation. Its paired Control and Reapplication histories share the historical requirement and final task while varying the intervening applicability conditions. By jointly evaluating warranted action, appropriate withholding, and subsequent reapplication under the same still-valid requirement, SIRRA assesses whether the agent consistently respects the requirement's conditional relationship to each task. This evaluation also reveals failures in which correct final reapplication coexists with unwarranted action selection during non-applicability.

Our rebuttal provides three substantive empirical findings.

First, final reapplication accuracy and complete-trajectory accuracy can rank memory systems in opposite orders. On 479 families, SimpleMem slightly exceeds MemoryOS at Evidence Return (421/479 vs. 415/479), but trails it in Middle withholding (240/479 vs. 331/479) and complete-trajectory accuracy (186/479 vs. 256/479). The newly evaluated native NanoMuse agent shows the same contrast against SimpleMem: lower Return accuracy but higher complete-trajectory accuracy under both Evidence and Classification. A system that more accurately reapplies a standing instruction may nevertheless be less reliable at adhering to it across tasks where its prescribed action is and is not warranted.

Second, extended trajectories reveal a recurring tendency toward over-application of still-valid instructions. Across 27 selected families, Qwen3.6-27B repeatedly selects APPLY when current task facts require withholding. Under sustained non-applicability, 10 of 15 families with initially correct restraint subsequently produce an unwarranted APPLY. Under repeated switching, four families that complete 10111 correctly fail specifically at the renewed SUPPRESS checkpoint in 10101. These errors occur both during sustained non-applicability and when an action becomes unwarranted again after a period of reapplication.

Third, access to historical information does not guarantee appropriate memory-conditioned action selection. In a matched long-context stress test, SUPPRESS accuracy declined from 73.3% to 55.6%, compared with 93.8% to 89.3% for APPLY. In a separate controlled intervention, adding the correct applicability judgment while retaining the same preceding history and current facts improved accuracy from 101/112 to 110/112, correcting both erroneous APPLY and erroneous SUPPRESS decisions. The availability of historical requirements and current task facts does not ensure that their conditional implications are correctly translated into action choices.

These are completed rebuttal experiments, responding to the reviewers' requests for broader applicability transitions, position and history-length controls, additional full-context and native-agent baselines, last-message comparisons, and applicability-cue conditions. We also clarified the interpretation of derived metrics and input-format comparisons. We will incorporate these results and the corresponding protocol, metric, and related-work clarifications into the manuscript at the next permitted revision.

The central empirical finding is that correct reapplication can coexist with recurring unwarranted APPLY decisions under the same still-valid requirement, and this distinction changes comparative assessments of memory systems.

We thank the reviewers for their engagement and hope these results inform the Area Chair's assessment of SIRRA's contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.