- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper analysises 38 long-run (115 checkpoint) conservations, picking points for context consolidation, evaluating 15 models as context compactors, under 2
Critical strengths
- Fixed-history checkpoint replay. This is good. Holding histories, interaction blocks, etc constant across compressors is a good basis on top of which to study compression of memory. Checkpoints are author-set block boundaries, which is good.
- Fifteen compressors across several model families is a good population to study
- Schedule gap (∆c) is a useful, novel metric.
- The schedule penalty analysis is good: The fresh–rolling gap falls mainly on scope binding and current-state correctness, with almost none on verification calibration.
- Rolling memories are longer yet score lower. This rules out shorter memories as the cause of the schedule gap.
- Multiple robustness diagnostics are reported.
Critical weaknesses
- The second model and the arbitrator in the dual-model protocol are not named.
- gpt-5.5 reads the memory, and GPT compressors take most top places. So the score may be biased towards "same family". This is not controlled for.
- The top compressor also writes the longest memories. This is not well controlled for.
- The model(s) that produced the 38 DeepSWE v1.1 histories are not named. Compressors from the same family may have an advantage in summarizing text.
- The histories seem to concatenate unrelated sessions, hence with no cross-task dependencies. In this setting, an oracle model would drop facts that are no longer needed, but the compaction evaluator would say "facts have been forgotten".
- Re "'96.1–99.8% of sub-item scores fall on the rubric anchors 0, 3, or 5 rather than across partial scores" So the score is effectively tri-state which makes it a pretty blunt tool.
- At checkpoint 1 fresh and rolling are identical by construction.
- The 115 stages and 2,782 probes are within 38 histories. So the effective N is ~38.
- Given the above, the fact that no confidence intervals or significance tests reported makes me less confident in the paper results.
- Abstract framing is “hours to weeks” but I think each run is ~400K tokens, which is much lower.
- The fresh less rolling delta is expected. No method for closing the gap is tested or discussed as future work.
- The results are interesting, real and useful. But only the "everything in memory" approach is analysed. An edit-in-place MD (store) might avoid the rolling penalty entirely.
Questions for authors
- Which models generated the trajectories?
- Who are the second judge and the arbitrator?
- How is “not in memory” scored?
7 comments
Response to Reviewer tyVp (Part 2/6)
Q2: About the second judge and the arbiter (Concern 1 & Question 2).
The dual-model protocol's second model and the arbitrator are not named. Who are the second judge and the arbitrator?
Ans for Q2: Thank you for catching this ambiguity.
(1) Clarification. Main-table answers are produced by gpt-5.5 and scored by a single five-dimension judge, glm-5.2. Construction uses glm-5.2 and deepseek-v4-pro, with claude-sonnet-5 arbitration; the separate four-class claude-sonnet-5 diagnostic is not a second five-dimension score. We have corrected the description in the revision.
(2) Why a single judge does not affect the paired contrast. The judge receives only task_scope, question, state_reference, constraints, compressed_memory, and candidate_response — no mode or schedule field — so its strictness cannot be conditioned on the schedule, and any mode-independent bias cancels in the paired difference. A second judge therefore serves the calibration of absolute scores. We have prepared a 200-item stratified blind protocol (same fixed answers, no regenerated memories, per-dimension weighted κ and score offsets).
(3) Self-preference check via family exclusion. We recomputed saved scores after exclusions (history-equal rolling−fresh means, 10,000-draw history bootstrap; 95% intervals):
| Excluded compressors | Remaining | ΔQ [95% CI] | ΔCSC [95% CI] |
|---|---|---|---|
| All GLM-family compressors (judge family) | 12 | −0.311 [−0.402, −0.223] | −0.394 [−0.470, −0.322] |
| Exact construction/arbiter models: glm-5.2, deepseek-v4-pro, claude-sonnet-5 | 12 | −0.349 [−0.426, −0.274] | −0.465 [−0.546, −0.389] |
| All GLM, DeepSeek, and Claude families | 7 | −0.327 [−0.434, −0.214] | −0.399 [−0.477, −0.322] |
All 7 remaining compressors have negative probe-averaged Q and CSC differences. The contrast therefore does not depend on compressors sharing a family with the judge or construction models. The deepseek-v4-pro exception is reported separately: history-equal ΔQ +0.078 [−0.209, +0.344], ΔCSC +0.166 [−0.089, +0.434]. Both intervals include zero; as a construction model, it is excluded in rows 2–3 above.
Response to Reviewer tyVp (Part 3/6)
Q3: About how "not in memory" is scored and whether dropping ended-session facts is forgetting (Concern 5 & Question 3).
The histories concatenate unrelated sessions. An oracle would drop facts that are no longer needed; the evaluator would call that forgetting. How is "not in memory" scored?
Ans for Q3: Thank you for this insightful question. We answer with the exact prompt text, the actual score distribution, and our interpretation.
(1) Prompt and rubric text. The reader instruction is: "Do not invent facts, permissions, verification results, or unavailable evidence. If memory is insufficient, say exactly what is missing and what must be checked before acting." In the five-dimension judge rubric:
- NAV, 5 points: "When evidence is insufficient, proposing minimal evidence gathering or a safe stop can also earn 5 points."
- VC, 5 points: "When evidence is insufficient, it makes only supported claims and explicitly identifies the gaps."
- CSC, 0 points: "The current state is incorrect; a stale value is treated as current; mutually exclusive states are adopted simultaneously; or a critical state is fabricated."
Abstention thus earns high NAV/VC but not CSC, since CSC measures whether the current value is recovered; 93.5% of fixed refusals receive CSC 0. This rubric applies identically to both schedules.
(2) Actual scores. Over 83,460 scored answers (15 compressors × 2 modes), we group reader answers by a rule-based classifier:
| Category | Share | CSC | SB | NAV | VC | CSC=0 | VC≥3 |
|---|---|---|---|---|---|---|---|
| Fixed refusal | 54.7% | 0.20 | 1.48 | 2.98 | 3.43 | 93.5% | 85.1% |
| Refusal plus extra text | 2.0% | 1.40 | 2.92 | 3.78 | 3.97 | 52.8% | 96.5% |
| Other "not mentioned" wording | 2.2% | 2.72 | 3.43 | 3.78 | 3.79 | 20.6% | 95.6% |
| Substantive answer | 41.1% | 3.75 | 3.78 | 3.88 | 3.88 | 19.0% | 93.6% |
- Among CSC=0 cases, 84.6% are plain refusals and 12.9% substantive answers. Of the 60.5% CSC zero rate in this population, about 51 percentage points are plain refusals and about 8 are substantive answers assigned zero. The paper's 61.0% comes from a separate 170,896-judgment pool.
- The VC zero rate is low (10.4%) because 85% of refusals score 3 or more on VC.
- The refusal share is 61.4% under rolling and 52.1% under fresh; most of the lower CSC under rolling comes from more refusals.
- For ended-session facts, the refusal share is 60.2% under fresh and 73.7% under rolling, with CSC around 0.23 on refusals.
(3) Interpretation. We interpret the ended-slice CSC drop as reduced retention of completed-task details under a retrospective-query convention (see also Kyiq Q3). Since the same rubric applies to both schedules, abstention scoring affects the absolute level but not the sign of the paired difference.
Response to Reviewer tyVp (Part 1/6)
We sincerely thank the reviewer for the careful review and constructive comments. We appreciate your recognition of the strengths of our paper, including the fixed-history checkpoint replay with author-set block boundaries, the population of fifteen compressors across several families, the new schedule-gap metric Δc, the finding that the penalty falls on scope binding and current-state correctness rather than verification calibration, and the observation that rolling memories are longer yet score lower. Below, we address each of your questions and concerns in detail.
Q1: About the trajectory-generating models and same-family bias between reader and compressors (Concerns 2 & 4, Question 1).
Ans for Q1: Thank you for raising this important point. We answer along three dimensions:
(1) Trajectory models. The 38 histories come from recorded DeepSWE v1.1 trajectories. Session
source_modeltakes only two values: claude-opus-5 and gpt-5.6-sol. The 38 histories use 173 distinct sessions (228 slots), of which claude-opus-5 accounts for 87 and gpt-5.6-sol for 86. gpt-5.6-sol is also one of the 15 compressors.(2) Stratifying by trajectory source. We split probes by the source model of the session a fact belongs to:
The Spearman correlation of compressor rankings between the two strata is 0.957 (fresh) and 0.932 (rolling). For the difference-in-differences of GPT-family compressors on gpt-5.6-sol versus claude-opus-5 facts, fresh is +0.056 [−0.073, +0.206] and rolling is −0.054 [−0.152, +0.025]. Both intervals contain 0, so there is no evidence of same-family preference.
(3) Swapping the reader. We keep saved memories and probes fixed, re-answer with deepseek-v4.1-flash in place of gpt-5.5, and rejudge with the same judge. This covers 4 compressors × two modes × 115 checkpoints, comparing both readers on their 21,968 shared probes: