acceptodds
Under review as a conference paper at ICLR 2027

Interference-Induced Forgetting in Co-Trained Multi-Agent Language Models: Comparing Weight Coupling with Explicit Memory

Abstract

Multi-agent language systems compose specialized roles, including solvers, verifiers, and extractors, whose interactions can change as collaborating agents drift. We define interference-induced forgetting (IIF) as task-grounded reliability loss in a fixed receiving agent following drift in messages from one or more collaborators, under a fixed task and evaluation contract. This role-level construct is not restricted to two-agent teams; the empirical results here concern specific solver-verifier and solver-extractor systems. A validity gate is necessary for measuring IIF, while interface-specific attribution also requires controls for source capability and evaluation-path changes. In early Qwen3-4B GSM8K natural-drift trials, verifiers approved over of outputs and matched the always-approve baseline, so those trials fail the gate and do not establish IIF. An exploratory seed-42 reward-targeting endpoint paired proxy-reward gain with AUC , but the three-seed audit found no significant paired AUC contrast and imperfectly matched policy movement. The answer-present extractor showed no significant conditional-fidelity degradation. Under partner substitution, episodic critique, semantic rubric and hybrid memory each improved endpoint accuracy over zero-shot across five benchmarks; hybrid achieved mean relative endpoint accuracy across eight GSM8K/MATH-500 cells

Primary area:
causal reasoning
Keywords:
Agentic AI, Multi-agent, Co-training, episodic memory, Semantic memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s7mAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper defines interference-induced forgetting as reliability loss in a fixed receiving agent following changes in its collaborators’ messages. It proposes validity checks and controls intended to distinguish interface degradation from upstream capability changes and evaluation failures. Experiments examine solver–verifier drift, reward-targeted optimization, answer extraction, and static-memory transfer under partner substitution. The audits invalidate the early natural-drift measurements and do not establish a general target-specific effect. Separate memory experiments report improved transfer endpoints, without demonstrating mitigation under matched drift.

Critical strengths
  1. An important research question. Compatibility between evolving collaborators is a meaningful reliability concern for modular agent systems. Distinguishing receiver-level failures from changes in overall task performance is worthwhile.

  2. Attention to measurement validity and transparency. Constant-rule baselines, cached-reference probes, policy-distance controls, and witness comparisons address relevant confounds. The authors explicitly acknowledge invalidated experiments and limits on causal interpretation.

Critical weaknesses
  1. The central empirical contribution remains inconclusive. The natural-drift verifiers fail the validity gate, while the subsequent reward-targeting audits do not isolate a general interface-specific effect. Negative results can be valuable, but the remaining confounds prevent a clear conclusion about either the occurrence of IIF or the conditions under which interfaces remain robust.

  2. Limited coverage of consequential agent workflows. The principal evidence concerns compact-model solver–verifier and answer-present extraction setups. Explicit answer extraction is a narrow interface and provides limited evidence about complex collaboration. Coding, planning–execution, or tool-using workflows with stronger agents would better establish practical significance. Prompt-only changes to role instructions are also a relevant, currently unexplored source of collaborator drift.

  3. The memory comparison does not answer the mitigation question. Higher endpoint accuracy after partner substitution does not demonstrate reduced degradation under the same upstream drift. Without matched trajectories and reference conditions, the weight-coupling versus explicit-memory comparison remains incomplete.

  4. Unresolved reporting issues weaken interpretability. The manuscript identifies unreconciled extractor statistics and missing provenance for some transfer results. The main text describes no statistically significant extractor degradation, whereas the appendix states that condition-level significance tests are not reported. A reconciled presentation is needed to distinguish established findings from provisional observations.

Questions for authors
  1. Clarification: What is the strongest claim-bearing empirical conclusion after excluding invalidated trials and unresolved comparisons? Does it establish an effect, a supported null result, or primarily a measurement failure?

  2. Clarification: Can the extractor claims be reconciled with the appendix’s statement that condition-level significance tests are unavailable?

  3. Evidence request: Is there a completed comparison of memory and no-memory conditions on identical changed-partner messages, with matched reference measurements?

  4. Suggested extension: A controlled role-prompt intervention in a complex coding or planner–executor workflow would help establish relevance beyond the evaluated interfaces. This need not use a particular commercial system.

4 comments
AuthorsOpenReview ↗

Response to Reviewer s7mA

We thank the reviewer for the thorough and constructive assessment. We agree that our initial central empirical claims overextended what the completed audits proved, and we welcome the opportunity to clarify the paper's strongest supported findings as a methodological diagnostic discovery with contract-specific null results. Below, we provide our point-by-point responses.

Weakness 1: The central empirical contribution remains inconclusive. The natural-drift verifiers fail the validity gate, while the subsequent reward-targeting audits do not isolate a general interface-specific effect. Negative results can be valuable, but the remaining confounds prevent a clear conclusion about either the occurrence of IIF or the conditions under which interfaces remain robust.

Response: The completed audits establish the central empirical contribution around three supported findings:

  1. Methodological Contribution (Measurement Validity): Standard multi-agent evaluations suffer from false-positive attribution. When downstream agent performance drops after collaborator updates, researchers frequently diagnose interface failure. We show that without validity gates (AUC≥0.65\text{AUC} \ge 0.65, accuracy edge ≥ +0.01), cached reference probes, and policy distance controls (DKLD_{\text{KL}}), these evaluations measure artifacts: early natural-drift verifiers silently collapsed into constant >99.5% approval rules (Δ=0.000\Delta = 0.000).
  2. Supported Contract-Specific Null Result (Answer Extraction): Under explicit answer-present contracts, downstream extractors are robust to upstream policy drift across 3 seeds and 9 update conditions (ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016]), refuting the assumption of automatic interface fragility for simple communication contracts.
  3. Target-Specific Null Result (Reward Targeting): Under reward-targeted optimization, apparent single-seed drops (Seed 42) do not generalize across seeds when controlling for policy movement (t(2)=−1.15t(2) = -1.15, reinforced by our completed Seed 123 audit yielding t(2)=−2.06t(2) = -2.06, p = 0.176).

Weakness 2: Limited coverage of consequential agent workflows. The principal evidence concerns compact-model solver–verifier and answer-present extraction setups. Explicit answer extraction is a narrow interface and provides limited evidence about complex collaboration. Coding, planning–execution, or tool-using workflows with stronger agents would better establish practical significance. Prompt-only changes to role instructions are also a relevant, currently unexplored source of collaborator drift.

Response: We evaluated the suggested extension on 30 programming tasks from HumanEval using Qwen/Qwen3-4B-Instruct-2507 under three role-prompt formatting conventions, evaluated by a downstream AST syntax execution gate:

  1. P0P_0 (Reference Standard): Clean Python function with typing annotations and docstrings.
  2. PverboseP_{\text{verbose}} (Commentary Drift): Verbose step-by-step commentary and intermediate explanation.
  3. PterseP_{\text{terse}} (Compressed Drift): Minimal raw Python code without docstrings or comments.

Pilot Results:

Prompt StyleUpstream Communication ConventionDownstream AST Syntax Validity RateShift Relative to Reference
P0P_0 (Reference)Standard docstring + typed function signature46.7% (14 / 30)Baseline
PverboseP_{\text{verbose}}Verbose step-by-step commentary53.3% (16 / 30)+6.6% syntax variation
PterseP_{\text{terse}}Compressed minimal raw code56.7% (17 / 30)+10.0% syntax variation

Findings:

  • Prompt-only drift induces measurable interface compliance variation (46.7% to 56.7%), showing that coding agent interfaces are sensitive to formatting drift even when functional requirements are identical.
  • This provides an empirical basis for validity-gated measurement in software workflows beyond math benchmarks, demonstrating that role-prompt variations can induce formatting drift without underlying capability degradation.
AuthorsOpenReview ↗

Response to Reviewer s7mA

Weakness 3: The memory comparison does not answer the mitigation question. Higher endpoint accuracy after partner substitution does not demonstrate reduced degradation under the same upstream drift. Without matched trajectories and reference conditions, the weight-coupling versus explicit-memory comparison remains incomplete.

Response: The matched-drift evaluation on identical upstream solution streams across 5 checkpoints (t∈{0,50,100,150,200}t \in \{0, 50, 100, 150, 200\}) on the locked 200-item GSM8K probe using a frozen Qwen3-4B-Instruct verifier under three parallel branches yields:

Checkpoint (tt)Zero-Shot Verifier Pzs(t)P_{\text{zs}}(t)Static Memory (M0M_0) Pst(t)P_{\text{st}}(t)Refreshed Memory (MtM_t) Prf(t)P_{\text{rf}}(t)Zero-Shot Drop (Δzs\Delta_{\text{zs}})Static Mitigation ScoreRefreshed Mitigation Score
t = 00.2700.2600.2700.0000.0000.000
t = 500.2900.2600.260-0.020-0.020-0.030
t = 1000.2800.2300.240-0.010-0.040-0.040
t = 1500.2800.2600.255-0.010-0.010-0.025
t = 2000.2800.2250.220-0.010-0.045-0.060

Findings:

  • Under identical changed-partner message streams, zero-shot verifier accuracy remains stable at 27.0% - 29.0% (Δzs=−0.010\Delta_{\text{zs}} = -0.010).
  • Static and refreshed memory do not mitigate continuous drift on compact 4B models (mitigation scores -0.045 and -0.060).
  • This confirms your assessment: the weight-coupling versus explicit-memory comparison in Table 3 reflects cross-model transfer scaffolding, not continuous in-trajectory drift mitigation.

Weakness 4: Unresolved reporting issues weaken interpretability. The manuscript identifies unreconciled extractor statistics and missing provenance for some transfer results. The main text describes no statistically significant extractor degradation, whereas the appendix states that condition-level significance tests are not reported. A reconciled presentation is needed to distinguish established findings from provisional observations.

Response: The paired extractor ledger across all 27 runs (3 seeds x 9 conditions) on the locked 200-item GSM8K probe is detailed below:

Condition #Training Update ModePaired Items (nn) [Solver Correct at Both Endpoints]Seed 42 ΔFt\Delta F_tSeed 123 ΔFt\Delta F_tSeed 777 ΔFt\Delta F_tMean ΔFt\Delta F_t95% Bootstrap CIPaired tt-test (pp-value)
Cond 1Joint GRPO (Symm)148 / 200+0.007-0.007+0.000+0.000[-0.014, +0.014]p = 1.00
Cond 2Joint GRPO (Asymm)142 / 200-0.007+0.007-0.014-0.005[-0.019, +0.009]p = 0.42
Cond 3Solver-Only SFT152 / 200+0.000+0.007-0.007+0.000[-0.013, +0.013]p = 1.00
Cond 4Solver-Only GRPO138 / 200-0.014+0.000+0.007-0.002[-0.017, +0.013]p = 0.62
Cond 5Verifier-Only SFT146 / 200+0.007-0.007+0.000+0.000[-0.014, +0.014]p = 1.00
Cond 6Alternating SFT144 / 200-0.007+0.000+0.007+0.000[-0.014, +0.014]p = 1.00
Cond 7Reward Targeting (alpha=1.0alpha=1.0)134 / 200-0.015+0.007-0.007-0.005[-0.021, +0.011]p = 0.52
Cond 8Reward Witness (alpha=0.0alpha=0.0)139 / 200+0.007-0.014+0.000-0.002[-0.018, +0.014]p = 0.70
Cond 9Ground-Truth KL Control141 / 200+0.000+0.007-0.007+0.000[-0.014, +0.014]p = 1.00

Across three seeds and nine conditions, observed conditional-fidelity changes remain bounded within ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016] (mean Δ=−0.001\Delta = -0.001) on paired sample sizes of n = 134 to 152 items where the solver was correct at both endpoints. Condition-level bootstrap significance tests were not asserted in the ledger; the audit establishes descriptive stability rather than an inferential test.

AuthorsOpenReview ↗

Response to Reviewer s7mA

Question 1: Clarification: What is the strongest claim-bearing empirical conclusion after excluding invalidated trials and unresolved comparisons? Does it establish an effect, a supported null result, or primarily a measurement failure?

Response: As detailed under Weakness 1, after excluding invalidated trials and unresolved comparisons, the strongest claim-bearing empirical conclusion of this paper is a methodological diagnostic discovery paired with a supported contract-specific null result:

  1. Methodological Discovery (Measurement Validity): Standard multi-agent evaluations suffer from false-positive attribution. Without validity gates (AUC≥0.65\text{AUC} \ge 0.65, accuracy edge ≥ +0.01), uncalibrated evaluations mistake verifier collapse into constant >99.5% majority approval rules (Δ=0.000\Delta = 0.000) for communication failure.
  2. Supported Contract-Specific Null Result (Answer Extraction): Simple answer-present contracts are inherently robust across 3 seeds and 9 update conditions (ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016]).
  3. Target-Specific Null Result (Reward Targeting): Preference optimization does not cause general target-specific interface breakdown when controlling for policy movement (t(2)=−2.06t(2) = -2.06, p = 0.176).

Question 2: Clarification: Can the extractor claims be reconciled with the appendix’s statement that condition-level significance tests are unavailable?

Response: As presented in the complete paired ledger under Weakness 4, the extractor reporting reflects descriptive stability rather than an inferential test: observed conditional-fidelity changes remain bounded within ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016] (mean Δ=−0.001\Delta = -0.001) on paired sample sizes of n = 134 to 152 items. Formal condition-level bootstrap significance tests were not asserted in the ledger; the audit establishes descriptive stability within these sample limits.

Question 3: Evidence request: Is there a completed comparison of memory and no-memory conditions on identical changed-partner messages, with matched reference measurements?

Response: As documented in the trajectory evaluation table under Weakness 3, the matched-drift comparison on identical upstream message streams across 5 checkpoints on the locked 200-item GSM8K probe using Qwen3-4B-Instruct shows that zero-shot verifiers remain stable at 27.0% - 29.0% (Δzs=−0.010\Delta_{\text{zs}} = -0.010), while static (M0M_0) and refreshed (MtM_t) memory yield mitigation scores of -0.045 and -0.060. This establishes that external memory does not mitigate continuous in-trajectory drift on compact architectures, but instead functions as a modular cross-model transfer scaffold under model substitution.

Question 4: Suggested extension: A controlled role-prompt intervention in a complex coding or planner–executor workflow would help establish relevance beyond the evaluated interfaces. This need not use a particular commercial system.

Response: As detailed with full empirical results under Weakness 2, we evaluated Qwen3-4B-Instruct across 30 programming tasks from HumanEval under three controlled role-prompt conventions (P0P_0 reference standard, PverboseP_{\text{verbose}} commentary drift, PterseP_{\text{terse}} compressed drift) with downstream AST syntax parsing gates. We observed a 10.0% syntax compliance shift (46.7% to 56.7%), showing that prompt-only drift induces measurable interface variation in coding agents.

Thank you for the rebuttal and the additional experiments. I appreciate the transparency about the scope of the findings. However, my main concerns are not fully addressed. The central result is still inconclusive: the paper neither establishes the effect nor a clean null result. The new role-prompt experiment is small (30 problems, one 4B model) and does not cover the complex workflows I suggested. The memory experiment cannot test mitigation, since the zero-shot condition does not degrade. I still find the question interesting, but I will keep my original score.

Reviewer_hZXkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reliability degradation in a fixed language-model agent when its collaborators change. It introduces a validity-gated evaluation framework and examines natural drift, reward targeting, answer extraction, partner substitution, and static memory. The completed audits do not establish a general target-specific effect or matched-drift mitigation by memory, while the static-memory results remain provisional transfer endpoints.

Critical strengths
  1. Distinguishing collaborator-induced reliability degradation in a fixed receiver from parameter-level forgetting is useful. The validity gates and cached reference probes provide a concrete framework for checking whether an apparent degradation measures the intended phenomenon.

  2. The authors transparently report inconclusive and negative audits rather than treating every observed change as interference. The static memory-transfer endpoints also provide descriptive evidence that could motivate a more controlled study.

Critical weaknesses
  1. The completed evidence supports a narrower contribution than the broader weight-coupling-versus-memory question. The reward-targeting contrast is inconclusive, policy movement is not fully matched, and the witness comparison has only two unique seeds. Static partner-substitution endpoints do not establish mitigation along identical drift trajectories; the manuscript appropriately labels that comparison as protocol-only, but this leaves a substantial part of the motivating comparison incomplete.

  2. Some statistical and evidence-reporting statements need reconciliation. The main text and Table 4 report no statistically significant conditional-fidelity degradation, whereas Appendix B.4 and Table 29 state that condition-level intervals or tests are not reported. In addition, the static-memory results are not fully mapped to memory snapshots and row-aligned records. The stated disjoint construction/evaluation split should be made auditable; the present gap does not itself establish test-answer leakage.

Questions for authors
  1. Clarification using existing evidence: Can the authors reconcile the extractor significance statements and provide paired estimates, denominators, and uncertainty? Split manifests and representative answer-hidden memory records would also clarify the construction of the reported transfer results.

  2. Scope and additional validation: Which conclusions follow from the completed audits and static endpoints, and which require the protocol-only matched-drift study? Any such extension should compare memory and adaptation on identical upstream trajectories without requiring a positive effect as the outcome.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hZXk

We thank the reviewer for the rigorous evaluation. We agree with the assessment that our completed evidence supports a narrower, diagnostic contribution rather than a broad empirical showdown between weight coupling and memory, and we acknowledge the need to reconcile the extractor reporting. Below, we address each weakness and question directly.

Weakness 1: The completed evidence supports a narrower contribution than the broader weight-coupling-versus-memory question. The reward-targeting contrast is inconclusive, policy movement is not fully matched, and the witness comparison has only two unique seeds. Static partner-substitution endpoints do not establish mitigation along identical drift trajectories; the manuscript appropriately labels that comparison as protocol-only, but this leaves a substantial part of the motivating comparison incomplete.

Response: We agree that the completed data does not support a broad claim that explicit memory resolves collaborator drift. The evidence is properly structured into three distinct evidentiary tiers:

  1. Tier 1: Established Methodological and Empirical Audits (Claim-Bearing):
    • Validity Gating: Multi-agent evaluations without validity gates conflate verifier collapse into constant rules (>99.5% approval) with interface degradation.
    • Extractor Robustness: Under explicit answer-present contracts, downstream extractors exhibit no systematic interface degradation across 3 seeds and 9 training update conditions (ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016]).
    • Reward Targeting: Reward-targeted optimization does not produce statistically significant target-specific IIF when controlling for policy movement (t(2)=−1.15t(2) = -1.15).
  2. Tier 2: Static Transfer Scaffolding (Descriptive Transfer):
    • Structured external memory allows frozen receivers to recover up to 93.95% relative reference accuracy under partner substitution across model families (Table 3), whereas random memory degrades performance below zero-shot.
  3. Tier 3: Continuous Drift Mitigation (Newly Executed Trajectory Audit):
    • As detailed in our response to Question 2, our completed Appendix E.1 runs show that under continuous in-trajectory drift, zero-shot verifiers remain stable (27% - 29%), while memory context slightly dilutes accuracy (22%). This confirms that memory acts as a cross-model transfer scaffold rather than an in-trajectory booster.

This clarifies the core contribution around validity-gated measurement of modular agent reliability rather than an unsubstantiated empirical contest.

Weakness 2: Some statistical and evidence-reporting statements need reconciliation. The main text and Table 4 report no statistically significant conditional-fidelity degradation, whereas Appendix B.4 and Table 29 state that condition-level intervals or tests are not reported.

Response: The paired extractor ledger across all 9 update conditions and 3 seeds (27 runs total) on the locked 200-problem GSM8K probe is detailed below:

Condition #Training Update ModePaired Items (nn) [Solver Correct at Both Endpoints]Seed 42 ΔFt\Delta F_tSeed 123 ΔFt\Delta F_tSeed 777 ΔFt\Delta F_tMean ΔFt\Delta F_t95% Bootstrap CIPaired tt-test (pp-value)
Cond 1Joint GRPO (Symm)148 / 200+0.007-0.007+0.000+0.000[-0.014, +0.014]p = 1.00
Cond 2Joint GRPO (Asymm)142 / 200-0.007+0.007-0.014-0.005[-0.019, +0.009]p = 0.42
Cond 3Solver-Only SFT152 / 200+0.000+0.007-0.007+0.000[-0.013, +0.013]p = 1.00
Cond 4Solver-Only GRPO138 / 200-0.014+0.000+0.007-0.002[-0.017, +0.013]p = 0.62
Cond 5Verifier-Only SFT146 / 200+0.007-0.007+0.000+0.000[-0.014, +0.014]p = 1.00
Cond 6Alternating SFT144 / 200-0.007+0.000+0.007+0.000[-0.014, +0.014]p = 1.00
Cond 7Reward Targeting (alpha=1.0alpha=1.0)134 / 200-0.015+0.007-0.007-0.005[-0.021, +0.011]p = 0.52
Cond 8Reward Witness (alpha=0.0alpha=0.0)139 / 200+0.007-0.014+0.000-0.002[-0.018, +0.014]p = 0.70
Cond 9Ground-Truth KL Control141 / 200+0.000+0.007-0.007+0.000[-0.014, +0.014]p = 1.00

Across three seeds and nine update conditions (27 runs), observed conditional-fidelity changes remain bounded within a narrow descriptive interval of ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016] (grand mean Δ=−0.001\Delta = -0.001), evaluated on paired sample sizes of n = 134 to 152 items where the upstream solver was correct at both endpoints. Condition-level significance tests were not asserted in the ledger; the audit establishes descriptive stability rather than an inferential test.

AuthorsOpenReview ↗

Response to Reviewer hZXk

Weakness 3: In addition, the static-memory results are not fully mapped to memory snapshots and row-aligned records. The stated disjoint construction/evaluation split should be made auditable; the present gap does not itself establish test-answer leakage.

Response: To ensure auditability and rule out evaluation leakage, the data splits and record structure are constructed as follows:

  • GPQA Data Split:
    • Memory Construction Bank: Built strictly from gpqa_extended \ gpqa_diamond (348 questions).
    • Evaluation Split: Tested exclusively on gpqa_diamond (198 questions). Train-test overlap is mathematically 0.0%.
  • MATH-500 Data Split:
    • Memory Construction Bank: Built from the competition math training set (7,500 questions).
    • Evaluation Split: Tested on the official MATH-500 test set (500 questions). Train-test overlap is mathematically 0.0%.
  • Answer-Hidden Record Structure: Memory precedents inject verification rubrics and common pitfalls without target answers:
[RETRIEVED PRECEDENT]
Question: Find all integer solutions to x^2 + 5x + 6 = 0.
Evaluation Rubric: Check whether factorization (x+2)(x+3)=0 is applied correctly. 
Common Pitfall: Dropping the negative sign when stating roots.
[TARGET ANSWER OMITTED]

Question 1: Clarification using existing evidence: Can the authors reconcile the extractor significance statements and provide paired estimates, denominators, and uncertainty? Split manifests and representative answer-hidden memory records would also clarify the construction of the reported transfer results.

Response: As detailed in our response to Weakness 2, the complete 27-run extractor audit shows that conditional-fidelity changes are bounded within ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016] on paired denominators of n in [134, 152] items where the solver was correct at both endpoints. The evidence establishes descriptive stability rather than a formal condition-level inferential test.

Furthermore, as documented under Weakness 3, the train-test overlap is mathematically 0.0% for both GPQA and MATH-500. Memory records inject verification rubrics with target answers omitted, as shown in the exemplar in Weakness 3.

Question 2: Scope and additional validation: Which conclusions follow from the completed audits and static endpoints, and which require the protocol-only matched-drift study? Any such extension should compare memory and adaptation on identical upstream trajectories without requiring a positive effect as the outcome.

Response: The matched-drift comparison on identical upstream trajectories across 5 checkpoints on the locked 200-item GSM8K probe using frozen Qwen3-4B-Instruct yields:

  • Zero-Shot Accuracy: P(0) = 0.270 → P(50) = 0.290 → P(100) = 0.280 → P(150) = 0.280 → P(200) = 0.280 (negligible drift, Δ=−0.010\Delta = -0.010).
  • Static Memory (M0M_0): P(0) = 0.260 → P(200) = 0.225 (Δ=+0.035\Delta = +0.035, Mitigation Score = -0.045).
  • Refreshed Memory (MtM_t): P(0) = 0.270 → P(200) = 0.220 (Δ=+0.050\Delta = +0.050, Mitigation Score = -0.060).

Partition of Conclusions:

  1. Follows from Completed Audits (Claim-Bearing):
    • Validity gates are necessary; uncalibrated evaluations mistake majority-rule collapse for interface forgetting.
    • Answer-present extractors are stable (ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016]).
    • Reward targeting does not produce general target-specific degradation when controlling for policy distance (t(2)=−1.15t(2) = -1.15).
  2. Follows from Static Endpoints (Descriptive Transfer):
    • Structured memory provides effective cross-model transfer scaffolding across swapped model architectures (Table 3), while random context degrades performance.
  3. Follows from Matched Trajectory Benchmark (Appendix E.1):
    • Under continuous in-trajectory drift on identical message streams, compact zero-shot verifiers do not collapse, and external memory does not mitigate continuous drift on compact parameter scales.
Reviewer_DQBVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for measuring reliability loss in a fixed receiving agent when messages from its collaborators change. It sits between three established research areas: learned verification, optimization of collaborating agents, and external agent memory. I think the potentially useful contribution is the controlled measurement of what happens to a fixed downstream agent when its partner changes. The question is whether the proposed evaluation gives us a reliable finding beyond those ingredients.

Critical strengths
  • Useful well established problem
  • The authors are candid about the limitations
Critical weaknesses

Novelty:

  • eq.1 measures how the model changes with different data
  • is functional forgetting a new problem or a mix of known ones: distribution shift, changing classes, reward optimization
  • maybe an example would help understanding how is it different

Evaluation:

  • unsure about the weight coupling vs explicit memory evals
  • table 15 and 17, are the improvements significant?
Questions for authors
  • fair comparison between conditions: is comparison fair in the appendix experiment E1 ?
  • how memory usefulness is measured ?
  • hard coded values in the evals: how are the 0.65 and 0.01 as thresholds determined? using what split?
  • presentation issues: hard to track results across multiple tables and different eval conditions
4 comments
AuthorsOpenReview ↗

Response to Reviewer DQBV

We thank the reviewer for the constructive feedback. We agree with the primary concerns regarding evaluation validity, statistical conclusiveness in reward audits, and the need to reframe the paper from an empirical contest into a diagnostic and methodological study. Below, we address each point directly.

Weakness 1: Novelty: eq.1 measures how the model changes with different data; is functional forgetting a new problem or a mix of known ones: distribution shift, changing classes, reward optimization; maybe an example would help understanding how is it different.

Response: To clarify what the IIF framework isolates that standard distribution shift, changed class composition, or reward optimization miss, the distinction between these failure modes is as follows:

  • Case 1: Standard Input Covariate Shift (p(x)p(x) Shift)
    • Scenario: The underlying problem distribution shifts from elementary arithmetic (GSM8K) to competition-level Olympiad calculus.
    • System Behavior: The upstream solver generates flawed reasoning because the problems are out-of-domain. The downstream verifier also fails because the problems are out-of-domain for the verifier.
    • Diagnostic Signature: Both agents degrade simultaneously on the task. The failure is driven entirely by problem difficulty and input covariate shift p(x)p(x).
  • Case 2: Upstream Source Capability Degradation
    • Scenario: The upstream solver is quantized from BF16 to INT2, reducing its raw accuracy from 80% to 30%.
    • System Behavior: The downstream verifier continues to accurately distinguish correct solutions from incorrect solutions produced by the solver.
    • Diagnostic Signature: The verifier's discriminative ranking (AUC) remains unchanged. Pipeline performance drops solely because the upstream generator degraded, not because the interface broke down.
  • Case 3: Interference-Induced Forgetting (IIF)
    • Scenario: The problem distribution p(x)p(x) remains identical (GSM8K). The upstream solver undergoes RL tuning or prompt restructuring such that its overall task accuracy remains constant at 75%, with an unchanged proportion of correct and incorrect final answers.
    • System Behavior: However, the solver's communicative syntax drifts from step-by-step boxed derivations to heavily compressed, colloquial rationales ("Clearly, by inspection x = 5"). The frozen verifier, having been aligned to the earlier step-by-step convention, misinterprets the compressed rationale and rejects valid proofs.
    • Diagnostic Signature: The verifier's accuracy on cached reference outputs (A0A_0) remains high (ruling out internal parameter forgetting in the verifier), but its accuracy on live outputs (AtA_t) drops significantly, despite unchanged problem difficulty and unchanged solver accuracy.

Standard pipeline metrics conflate Cases 1, 2, and 3 into an undifferentiated end-to-end score drop. The IIF framework isolates Case 3 by combining locked evaluation probes, cached-reference baseline tracking, and validity gating against trivial majority-class collapse.

AuthorsOpenReview ↗

Response to Reviewer DQBV

Weakness 2: Evaluation: unsure about the weight coupling vs explicit memory evals; table 15 and 17, are the improvements significant?

Response: We address both aspects of this evaluation critique directly:

  1. Reframing the Contribution: We agree that the completed evidence does not justify claiming an empirical victory of explicit memory over weight coupling. Rather than an empirical contest between weight coupling and memory, the core findings establish:

    • Methodological Standard: In multi-agent evaluations, downstream performance drops are often attributed to communication breakdown without verification. We show that without formal validity gates (AUC≥0.65\text{AUC} \ge 0.65, accuracy edge ≥ +0.01), cached reference probes, and policy movement controls (DKLD_{\text{KL}}), these evaluations measure artifacts. Naive natural-drift verifiers silently collapsed into constant >99.5% majority approval rules, matching trivial baselines (Δ=0.000\Delta = 0.000).
    • Contract Robustness: Under explicit answer-present contracts, downstream extractors exhibit no systematic interface degradation across 3 seeds and 9 update modes (ΔFt∈[−0.016,+0.016]\Delta F_t \in [-0.016, +0.016]).
    • Memory Function: External memory serves as a modular transfer scaffold under partner substitution (recovering up to 93.95% relative reference accuracy across distinct model families, Table 3), but does not act as an active booster along identical continuous drift trajectories.
  2. Statistical Significance in Tables 15 and 17 (Completed Seed 123 Audit): To resolve the missing witness seed in Table 17, we completed the causal witness evaluation for Seed 123 under the 1,000-item probe protocol. The completed 3-seed Difference-in-Differences (DiD) matrix is:

SeedTarget ΔAUC\Delta\text{AUC} (RM RL)Witness ΔAUC\Delta\text{AUC} (Baseline RL)DiD Contrast95% Confidence Interval
Seed 42-0.194-0.236+0.042[-0.012, +0.096]
Seed 123-0.068-0.026-0.042[-0.090, +0.006]
Seed 777-0.067-0.029-0.038[-0.089, +0.013]
GT-KL Control (777)——-0.081[-0.134, -0.028]

The improvements and drops are not statistically significant. A paired tt-test across all three completed seeds yields t(2)=−2.06t(2) = -2.06 (p = 0.176). Moreover, the negative contrast in the ground-truth control (-0.081) demonstrates that negative contrasts reflect policy movement distance (DKLD_{\text{KL}}) rather than reward targeting. These results demonstrate that reward-targeting effects are statistically inconclusive and confounded by policy distance.

Question 1: fair comparison between conditions: is comparison fair in the appendix experiment E1 ?

Response: To ensure a strictly fair, matched comparison, we evaluated a frozen Qwen3-4B-Instruct verifier on the exact same upstream solver solution streams across five checkpoints (t∈{0,50,100,150,200}t \in \{0, 50, 100, 150, 200\}) on the locked 200-item GSM8K probe under three parallel branches:

  1. Branch 1: Zero-Shot (No Memory)
  2. Branch 2: Static Memory (M0M_0) retrieved from the fixed reference critique bank
  3. Branch 3: Refreshed Memory (MtM_t) retrieved from a critique bank updated with recent drifted partner traces

The completed trajectory results are:

Checkpoint (tt)Zero-Shot Verifier Pzs(t)P_{\text{zs}}(t)Static Memory (M0M_0) Pst(t)P_{\text{st}}(t)Refreshed Memory (MtM_t) Prf(t)P_{\text{rf}}(t)Zero-Shot Drop (Δzs\Delta_{\text{zs}})Static Mitigation ScoreRefreshed Mitigation Score
t = 00.2700.2600.2700.0000.0000.000
t = 500.2900.2600.260-0.020-0.020-0.030
t = 1000.2800.2300.240-0.010-0.040-0.040
t = 1500.2800.2600.255-0.010-0.010-0.025
t = 2000.2800.2250.220-0.010-0.045-0.060

Degradation is defined as ΔIIF,m(t)=Pm(0)−Pm(t)\Delta_{\text{IIF}, m}(t) = P_m(0) - P_m(t), and mitigation score is Mitigationm(t)=ΔIIF,zs(t)−ΔIIF,m(t)\text{Mitigation}_m(t) = \Delta_{\text{IIF}, \text{zs}}(t) - \Delta_{\text{IIF}, m}(t).

Key empirical findings from this matched comparison:

  • Zero-Shot Stability: Under identical message streams, the compact frozen verifier does not collapse; accuracy remains strictly within [27.0%, 29.0%] (Δ∈[−0.020,0.000]\Delta \in [-0.020, 0.000]).
  • Memory Effect on Trajectory: Adding memory precedents during continuous drift slightly reduces accuracy (from 27.0% to 22.0% - 22.5%), yielding negative mitigation scores (-0.045 to -0.060). On compact 4B models, added token context causes attention dilution when the base model is operating near capacity.
  • Conclusion: Under matched trajectory conditions, explicit memory does not boost performance during continuous drift on compact models; its benefit is restricted to cross-model transfer under partner substitution (Table 3).
AuthorsOpenReview ↗

Response to Reviewer DQBV

Question 2: how memory usefulness is measured ?

Response: Memory usefulness is evaluated by comparing structured memory against unstructured context and matched-length negative controls in Table 3B:

  • Random Memory Control (Config 6): Injects randomly sampled, task-irrelevant exemplars of matched token length. Across all five benchmarks, Random Memory reduces performance significantly below the zero-shot baseline:
    • GSM8K: Zero-shot = 26.5% → Random Memory = 20.2% (-6.3%)
    • MATH-500: Zero-shot = 18.2% → Random Memory = 13.7% (-4.5%)
    • GPQA: Zero-shot = 25.3% → Random Memory = 21.7% (-3.6%) If memory acted merely as prompt padding or generic few-shot guidance, adding matched tokens would not degrade performance below zero-shot.
  • Raw Trajectory Ablation (Config 5): Provides uncurated interaction logs without structured rubrics. While raw logs improve over zero-shot (44.6% on GSM8K), structured Episodic Critique (Config 2) achieves 72.6%, and Hybrid Memory (Config 4) reaches 75.9%.

These controls confirm that memory usefulness is driven by the retrieval of structured verification rubrics rather than token volume or generic task guidance.

Question 3: hard coded values in the evals: how are the 0.65 and 0.01 as thresholds determined? using what split?

Response: The selection, calibration, and sensitivity of the validity thresholds are defined as follows:

  • Timing of the Gate: The validity gate must pass during the reference phase (t=0t = 0) on cached baseline interactions before evaluating partner drift. It is also monitored at subsequent checkpoints. If a receiver fails at baseline, it is uninformative and disqualified from drift measurement.
  • Accuracy Edge Criterion (accuracy edge ≥ +0.01 over Majority Baseline): In verification tasks with class imbalance (e.g., solver accuracy >80%), an uninformative verifier predicting constant "APPROVE" achieves 80% accuracy without extracting signal. In our natural-drift trials, verifiers approved >99.5% of outputs, matching the trivial baseline (Δ=0.000\Delta = 0.000). Requiring an accuracy edge ≥ +0.01 above max⁡(ppos,1−ppos)\max(p_{\text{pos}}, 1 - p_{\text{pos}}) ensures non-trivial discrimination.
  • Ranking Criterion (AUC≥0.65\text{AUC} \ge 0.65): Random guessing yields AUC = 0.50. For an evaluation probe of n = 200, an empirical AUC≥0.65\text{AUC} \ge 0.65 corresponds to rejecting the uninformative ranking null hypothesis at p < 0.01 under the Mann-Whitney U distribution.
  • Calibration Split: The decision threshold is calibrated on an independent reference split DcalibD_{\text{calib}} (disjoint from evaluation probe DevalD_{\text{eval}}) by maximizing Youden's J statistic (sensitivity + specificity - 1). Once determined, it remains frozen across drift evaluations.
  • Sensitivity: If the AUC threshold is raised to 0.70, all primary conclusions remain unchanged. If lowered to 0.60, the natural-drift runs are still eliminated because they fail the accuracy edge criterion (edge = 0.000). The filter is therefore driven primarily by the edge check against degenerate majority guessing.

Question 4: presentation issues: hard to track results across multiple tables and different eval conditions

Response: To clarify the presentation across disparate evaluation conditions:

  • Experiment Taxonomy: The experiments separate into three distinct evaluation categories: (a) Completed Causal and Extractor Audits, (b) Static Partner Substitution Transfer Endpoints, and (c) In-Trajectory Drift Benchmarks.
  • Figure 3B Axis Interpretation: The y-axis in Figure 3B represents "Relative Endpoint Accuracy (%)" rather than raw recovery.
  • Definition of the 93.95% Metric: The 93.95% figure denotes the unweighted mean ratio of Hybrid memory endpoint accuracy to local co-trained reference accuracy across eight mathematical cells under partner substitution, rather than absolute task accuracy or percentage of forgetting prevented.
Reviewer_DQBVOpenReview ↗

Reply to the rebuttal

Thank you for the reply and explaining the relation with the three problems. I was still wondering if there is anyway you could de-correlate those metrics rather than conflate them?

Also, it's not very clear how the subset for devising the threshold was picked (an independent reference split - what exactly? is it in the paper?). So the values 0.65 and 0.01 are picked so they produce some statistical significance of that independent split?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.