- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper distinguishes how useful a memory is from whether it is currently permitted for use. HARD removes memories labeled inadmissible before ranking the remaining candidates. SOFT keeps the candidates and subtracts a penalty from the scores of inadmissible memories. The theoretical analysis concerns fixed penalties on unnormalized utility scores under rescaling; the experiments use a normalized recency score. Both controllers receive the labels identifying which memories should no longer be used.
After excluding planned TIDE/LongMemEval evaluations that failed its protocol checks, the paper reports a replacement comparison using Qwen3-8B on Memora Remembering. The authors select a SOFT penalty from ten values using 15 development units, then evaluate on 185 held-out units. HARD achieves 4.04% balanced construct error, compared with 19.66% for SOFT. This error averages the missed-current-value rate and the forgotten-value leakage rate, based on whether the relevant value strings appear in the answer. The authors limit the result to the tested controllers, supplied labels, model, benchmark, and scoring procedure.
Critical strengths
The paper clearly separates a memory's usefulness from its current admissibility. This distinction provides a concrete question for evaluating how persistent memory systems handle updates, deletions, and revocation: whether information that should no longer be used can still influence an answer because it receives a high retrieval score.
Critical weaknesses
-
Insufficient theoretical and algorithmic novelty. Proposition 1 shows that a fixed additive penalty can be overcome by a sufficiently large positive rescaling of utility scores. Its proof reduces to choosing a scale for which . This elementary scale-dependence observation is not developed into a substantial theoretical advance, and it does not establish a performance advantage over the normalized SOFT controller used in the experiment, which already removes that particular scaling defect.
With inadmissibility labels supplied, HARD filters candidates and then ranks the remainder. A scalar rule with an appropriate admission threshold can reproduce its eligible set and ranking. The manuscript does not establish sufficient new algorithmic capability or theoretical insight beyond applying established hard-constraint and penalty-based control ideas to persistent memory. The distinction between usefulness and permission is relevant, but the formulation presented here does not provide sufficient novelty to support acceptance.
-
The empirical study does not establish a sufficiently substantive new finding. The completed performance evidence consists of one Qwen3-8B comparison on Memora Remembering, using 15 development units to select among ten penalties and 185 held-out units for evaluation. The balanced construct errors of 4.04% for HARD and 19.66% for SOFT require examination of their components: current-value recall differs by only 2.40 percentage points (91.92% versus 89.52%), whereas forgotten-value absence differs by 28.83 points (100.00% versus 71.17%). Most of the aggregate improvement therefore comes from avoiding specified forgotten values after the corresponding memories are removed. This score is directly coupled to the filtering operation and provides limited independent evidence of an improvement in answer quality.The comparison also lacks the controls needed to explain the observed gap. The provided PDF does not specify whether SOFT fills the available Top-K positions or rejects candidates below a threshold. If it fills them when permitted candidates are insufficient, forbidden memories can enter even when ranked below every permitted memory. The experiment does not separate this filling behavior from effects of prompt length, memory count, position, and distractor composition. The study also provides no completed cross-model or cross-benchmark validation and does not quantify uncertainty from development-set selection. The reported result supports a narrow comparison of the tested implementations, but does not supply a compelling empirical contribution that compensates for the limited theoretical and algorithmic novelty. I therefore recommend clear rejection; addressing these concerns requires substantial additional research and validation.
Questions for authors
-
Which memories enter the prompt? Does SOFT continue filling the available Top-K positions when fewer than K permitted candidates exist? Does it use a score threshold or another rejection rule? Please give the exact admission procedure and report, for each unit, the numbers of permitted and forbidden candidates, the numbers of each actually injected, and the resulting prompt length. If these details are in supplementary artifacts, please identify the exact files.
-
Comparison with a thresholded scalar rule. With , , and , the condition admits exactly the candidates and preserves their ranking by . This reproduces HARD's eligible set and ranking. The manuscript acknowledges that scalar thresholds can implement hard feasibility. Please clarify what the tested SOFT admission rule adds to this comparison, and distinguish increasing the penalty while still filling Top-K from using a threshold that rejects forbidden candidates.
-
Effect of context length and composition. HARD can supply fewer memories and a shorter prompt than SOFT. Can controlled comparisons quantify how much of the performance difference depends on the number, position, and length of the supplied memories or on interference from distractors? Please report these context differences alongside the results. This would help assess the explanation of the gap beyond the overall effect of the two tested controllers.
-
Stability of penalty selection. The complete DEV curve and the deterministic rule selecting are already reported. How stable are the selected penalty and the HARD–SOFT difference when the small development sample changes? Please assess selection uncertainty using a procedure that keeps related units grouped, such as alternative development splits or nested selection. The held-out confidence interval conditions on the selected penalty; it does not quantify uncertainty from choosing among ten values using 15 DEV units.
-
Practical meaning of the reported score. The paper explicitly limits its endpoint to current-value presence and forgotten-value absence. Can the authors provide a representative analysis of model outputs showing how changes in this score correspond to correct use of current information and avoidance of forgotten information? This would help assess the meaning of the improvement within the benchmark, beyond the direct benefit of removing memories labeled inadmissible from the input.
-
Scoring units without forgotten-value targets. How are units without materialized forgotten-value strings scored in the main 185-unit analysis? Please specify the empty-target convention and how current recall, forgetting absence, and balanced construct error are aggregated across units. The reported 167-unit sensitivity analysis, with HARD at 1.78% and SOFT at 19.08% error, is useful; this question concerns the definition of the main-cohort score.