- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CIT-Mem, a framework for teaching policy how to use evidence through validated counterfactual interventions. It jointly trains invariance, sensitivity, and abstention while keeping the retrieval backend and reader frozen. Experiments across multiple models and benchmarks report improved answer quality, evidence efficiency, and intervention consistency.
Critical strengths
- The paper addresses a meaningful distinction between answering correctly and relying on appropriate evidence. This motivation is particularly relevant when conversational memories become outdated, contradictory, or insufficient.
- Jointly considering invariance, directional sensitivity, and abstention provides a coherent approach to memory-policy reliability. Keeping counterfactual processing offline also makes the framework appealing for deployment.
- The evaluation includes multiple readers, cross-dataset transfer, matched-data ablations, and an independent human audit. Together, these provide useful evidence that the benefits extend beyond simply adding synthetic training examples.
Critical weaknesses
- The paper motivates CIT-Mem through faithful evidence use, but IS and CDA are not defined clearly enough to establish which behavior is being verified. It is unclear whether they evaluate the memory policy, the reader’s answer, or both. There should be detailed formulations.
- In Section 3.1, the recoverability gate uses an empty to validate abstention targets, and Section 3.2 uses this condition to assign ABSTAIN supervision. However, Algorithm 1 only states “construct valid supports,” without explaining how evidence combinations are searched or when the search stops. Since this procedure determines the training labels, the authors should clarify how they distinguish genuinely insufficient evidence from failure to find a valid support within the candidate memories.
- In the related work, MemoryR1 and AgeMem provide similar approaches, while the experiments mainly compare with non-training methods.
Questions for authors
- How are IS and CDA computed, and do they check whether the answer follows the intended evidence change? (Refer to weakness 1)
- How is (S_K(x)) constructed? (Refer to weakness 2)
- If feasible, please compare with either Memory-R1 or AgeMem using at least one reader model. Otherwise, please explain why and provide alternative evidence supporting the claimed advantages over outcome-supervised memory policies. (Refer to weakness 3)
- What specific limitation of inference-time prompting does counterfactual training address, and which component is responsible for this improvement? Please discuss the trade-offs between the two approaches in flexibility and transfer to new domains, particularly whether the learned memory policy can be reused without retraining.
Questions 1- 3 correspond to one of the weaknesses above, so a single response to each pair is sufficient. Addressing some of these concerns could lead me to raise my rating.
Response to Reviewers
Response to Reviewer 3
R3.1. Directional training and the definition of IS/CDA
Sensitivity is trained toward a validated target, not an arbitrary difference: G_ans^(v) checks the declared changed-answer/abstention relation, and Eq. (9) raises the validated target's score with a margin over the hardest incompatible behavior:
L_sens = −s_θ(b_I^c | x^c) + λ_mar · [δ_c − Δs_θ]₊
As implemented, IS/CDA are behavior-level pair scores. For non-causal pairs D_nc, with E_θ the selected evidence (canonical/semantic-equivalence classes), d_θ the ANSWER/ABSTAIN decision, and P_i the twin-to-base evidence map:
IS_beh = (100 / N_nc) · Σᵢ 1[E_θ(xᵢ) = P_i·E_θ(xᵢ^nc)] · 1[d_θ(xᵢ) = d_θ(xᵢ^nc)]
Both evidence role and decision must remain stable. For causal pairs D_c, with b̂_θ the realized behavior and B_i^c the validated acceptable target behaviors:
CDA_beh = (100 / N_c) · Σᵢ 1[b̂_θ(xᵢ^c) ∈ B_i^c]
The validated set admits separately validated alternatives, but changes must point to validated targets. CDA reflects policy behavior (evidence and decision jointly), not the reader's answer.
Table R3-1: IS drops 93.6→82.1 without invariance; CDA drops 75.8→56.1 without sensitivity—descriptive gaps, not significance tests.
Table R3-1. Component ablation (Table 3; three-seed means).
R3.2. Support construction and independent validation
Section 3.1 validates over ≤ C exposed candidates: ≤ K records are a support if the validator recovers the target and fails after matched removal of its decisive members; distinct supports stay separate in Eq. (8); the recoverability gate rejects residual sufficient support for abstention twins. Annotations anchor candidates, not sufficiency or necessity.
Table R3-2: 92/100 auto-accepted twins pass all five checks (8 errors; κ = 0.78; blinded annotators)—validity for the audited subset, not search recall.
Table R3-2. Human audit (Appendix C.1). κ = 0.78; by criterion: structure .86, evidence .79, leakage .81, sufficiency .74, relation .72.
R3.3. Identifiability and outcome-supervised baselines
Eq. (2): f_φ(q, Z_θ1) = f_φ(q, Z_θ2) = y ⇏ Z_θ1 = Z_θ2 — equal answers do not identify the selected evidence; this is not an impossibility theorem for outcome-supervised methods. Memory-R1/AgeMem are discussed but not run as matched baselines—a limitation we acknowledge. Table R3-3 (32.5/43.4/49.5) assesses paired supervision and replay in our setting, not reproductions of those methods; our claim is an additional supervision mechanism for evidence dependence.
Table R3-3. Attribution (Table 10; three-seed means). The no-twins and answer-only rows report different IS/CDA; not merged; configurations will be clarified.
R3.4. Moving interventions from inference into training
Interventions are validated offline to train a reusable policy (backend/reader fixed); at deployment one controller pass selects ≤ K records and calls the reader at most once, with no construction or replay. Trade-offs: dependence on twin quality, support discovery, and validator behavior; offline costs; 6.4 vs 8.8 Epoch@95 is the epoch metric, not wall-clock. Appendix C.2: Long-MT-Bench+ F1 53.86/51.40/47.17 (GPT-4o-mini/Qwen3-8B/Llama 3.2 3B) with the LoCoMo-selected checkpoint—transfer evidence for this setting, not a no-retraining guarantee.