acceptodds
Under review as a conference paper at ICLR 2027

What Survives the Long Horizon Run? Benchmarking Task State in Coding Agents

Abstract

Long-horizon agents may run for hours to weeks, requiring automated context compaction to fit constrained LLM context windows. However, such compaction risks corrupting or obscuring critical task state: objective-relevant facts and their provenance. To address this risk, we introduce EndureBench to evaluate language agents' preservation of critical task state across compaction cycles. It builds histories from recorded coding sessions toward a 400K-token context target and compares 15 compressors at fixed checkpoints under two schedules: fresh, which independently compacts each raw history prefix once, and rolling, which recursively updates the previous memory with the next interaction block. The results are consistent across compressors: 14 of 15 lose quality under rolling, and the failures are outright misses rather than partial answers—current-state correctness scores 0 on 61.0% of judgments and scope binding on 38.1%, against 11.3% for verification calibration. Overall quality hides this failure: compacted memory may omit objective-relevant facts and their provenance and should be verified before the agent resumes rather than treated as authoritative. EndureBench thereby isolates state preservation as a distinct object of measurement for long-horizon agents.

Primary area:
datasets and benchmarks
Keywords:
coding agents, context compaction, task-state preservation, long-horizon evaluation, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tyVpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper analysises 38 long-run (115 checkpoint) conservations, picking points for context consolidation, evaluating 15 models as context compactors, under 2

Critical strengths
  • Fixed-history checkpoint replay. This is good. Holding histories, interaction blocks, etc constant across compressors is a good basis on top of which to study compression of memory. Checkpoints are author-set block boundaries, which is good.
  • Fifteen compressors across several model families is a good population to study
  • Schedule gap (∆c) is a useful, novel metric.
  • The schedule penalty analysis is good: The fresh–rolling gap falls mainly on scope binding and current-state correctness, with almost none on verification calibration.
  • Rolling memories are longer yet score lower. This rules out shorter memories as the cause of the schedule gap.
  • Multiple robustness diagnostics are reported.
Critical weaknesses
  • The second model and the arbitrator in the dual-model protocol are not named.
  • gpt-5.5 reads the memory, and GPT compressors take most top places. So the score may be biased towards "same family". This is not controlled for.
  • The top compressor also writes the longest memories. This is not well controlled for.
  • The model(s) that produced the 38 DeepSWE v1.1 histories are not named. Compressors from the same family may have an advantage in summarizing text.
  • The histories seem to concatenate unrelated sessions, hence with no cross-task dependencies. In this setting, an oracle model would drop facts that are no longer needed, but the compaction evaluator would say "facts have been forgotten".
  • Re "'96.1–99.8% of sub-item scores fall on the rubric anchors 0, 3, or 5 rather than across partial scores" So the score is effectively tri-state which makes it a pretty blunt tool.
  • At checkpoint 1 fresh and rolling are identical by construction.
  • The 115 stages and 2,782 probes are within 38 histories. So the effective N is ~38.
  • Given the above, the fact that no confidence intervals or significance tests reported makes me less confident in the paper results.
  • Abstract framing is “hours to weeks” but I think each run is ~400K tokens, which is much lower.
  • The fresh less rolling delta is expected. No method for closing the gap is tested or discussed as future work.
  • The results are interesting, real and useful. But only the "everything in memory" approach is analysed. An edit-in-place MD (store) might avoid the rolling penalty entirely.
Questions for authors
  • Which models generated the trajectories?
  • Who are the second judge and the arbitrator?
  • How is “not in memory” scored?
7 comments
AuthorsOpenReview ↗

Response to Reviewer tyVp (Part 1/6)

We sincerely thank the reviewer for the careful review and constructive comments. We appreciate your recognition of the strengths of our paper, including the fixed-history checkpoint replay with author-set block boundaries, the population of fifteen compressors across several families, the new schedule-gap metric Δc\Delta_c, the finding that the penalty falls on scope binding and current-state correctness rather than verification calibration, and the observation that rolling memories are longer yet score lower. Below, we address each of your questions and concerns in detail.

Q1: About the trajectory-generating models and same-family bias between reader and compressors (Concerns 2 & 4, Question 1).

Which models generated the trajectories? gpt-5.5 reads the memory, and GPT compressors take most of the top places; a compressor from the same family as the trajectory generator may also be better at summarizing that text.

Ans for Q1: Thank you for raising this important point. We answer along three dimensions:

(1) Trajectory models. The 38 histories come from recorded DeepSWE v1.1 trajectories. Session source_model takes only two values: claude-opus-5 and gpt-5.6-sol. The 38 histories use 173 distinct sessions (228 slots), of which claude-opus-5 accounts for 87 and gpt-5.6-sol for 86. gpt-5.6-sol is also one of the 15 compressors.

(2) Stratifying by trajectory source. We split probes by the source model of the session a fact belongs to:

Source modelΔQ (rolling − fresh)ΔCSCCompressors with ΔQ<0Cells with CSC lowest
claude-opus-5−0.358−0.44814/1530/30
gpt-5.6-sol−0.257−0.34213/1530/30

The Spearman correlation of compressor rankings between the two strata is 0.957 (fresh) and 0.932 (rolling). For the difference-in-differences of GPT-family compressors on gpt-5.6-sol versus claude-opus-5 facts, fresh is +0.056 [−0.073, +0.206] and rolling is −0.054 [−0.152, +0.025]. Both intervals contain 0, so there is no evidence of same-family preference.

(3) Swapping the reader. We keep saved memories and probes fixed, re-answer with deepseek-v4.1-flash in place of gpt-5.5, and rejudge with the same judge. This covers 4 compressors × two modes × 115 checkpoints, comparing both readers on their 21,968 shared probes:

CompressorModegpt-5.5 qualitydeepseek-v4.1-flash qualitygpt-5.5 CSCdeepseek-v4.1-flash CSC
gpt-5.6-lunafresh3.3372.9472.3611.927
gpt-5.6-lunarolling2.9862.6232.0151.621
gpt-5.6-terrafresh3.3242.8292.3641.991
gpt-5.6-terrarolling2.8332.5061.6071.417
glm-5.3fresh3.0802.5562.1271.607
glm-5.3rolling2.8682.3771.7241.335
claude-sonnet-5fresh2.7332.1111.4981.053
claude-sonnet-5rolling2.3852.0701.1520.871
  • Same structure under a non-GPT reader: rolling is worse than fresh, with a quality gap of −0.350 [−0.511, −0.183] (gpt-5.5) and −0.216 [−0.341, −0.085] (deepseek-v4.1-flash), and a CSC gap of −0.463 and −0.333. All intervals exclude 0.
  • In all 8 (compressor, mode) cells, CSC is the lowest dimension under both readers.
  • Spearman rank correlation of compressors (n=4): fresh 1.0, rolling 0.8.
  • Scope: four compressors; absolute scores are not comparable across readers (deepseek-v4.1-flash abstains on 69.8% of probes).
AuthorsOpenReview ↗

Response to Reviewer tyVp (Part 2/6)

Q2: About the second judge and the arbiter (Concern 1 & Question 2).

The dual-model protocol's second model and the arbitrator are not named. Who are the second judge and the arbitrator?

Ans for Q2: Thank you for catching this ambiguity.

(1) Clarification. Main-table answers are produced by gpt-5.5 and scored by a single five-dimension judge, glm-5.2. Construction uses glm-5.2 and deepseek-v4-pro, with claude-sonnet-5 arbitration; the separate four-class claude-sonnet-5 diagnostic is not a second five-dimension score. We have corrected the description in the revision.

(2) Why a single judge does not affect the paired contrast. The judge receives only task_scope, question, state_reference, constraints, compressed_memory, and candidate_response — no mode or schedule field — so its strictness cannot be conditioned on the schedule, and any mode-independent bias cancels in the paired difference. A second judge therefore serves the calibration of absolute scores. We have prepared a 200-item stratified blind protocol (same fixed answers, no regenerated memories, per-dimension weighted κ and score offsets).

(3) Self-preference check via family exclusion. We recomputed saved scores after exclusions (history-equal rolling−fresh means, 10,000-draw history bootstrap; 95% intervals):

Excluded compressorsRemainingΔQ [95% CI]ΔCSC [95% CI]
All GLM-family compressors (judge family)12−0.311 [−0.402, −0.223]−0.394 [−0.470, −0.322]
Exact construction/arbiter models: glm-5.2, deepseek-v4-pro, claude-sonnet-512−0.349 [−0.426, −0.274]−0.465 [−0.546, −0.389]
All GLM, DeepSeek, and Claude families7−0.327 [−0.434, −0.214]−0.399 [−0.477, −0.322]

All 7 remaining compressors have negative probe-averaged Q and CSC differences. The contrast therefore does not depend on compressors sharing a family with the judge or construction models. The deepseek-v4-pro exception is reported separately: history-equal ΔQ +0.078 [−0.209, +0.344], ΔCSC +0.166 [−0.089, +0.434]. Both intervals include zero; as a construction model, it is excluded in rows 2–3 above.

AuthorsOpenReview ↗

Response to Reviewer tyVp (Part 3/6)

Q3: About how "not in memory" is scored and whether dropping ended-session facts is forgetting (Concern 5 & Question 3).

The histories concatenate unrelated sessions. An oracle would drop facts that are no longer needed; the evaluator would call that forgetting. How is "not in memory" scored?

Ans for Q3: Thank you for this insightful question. We answer with the exact prompt text, the actual score distribution, and our interpretation.

(1) Prompt and rubric text. The reader instruction is: "Do not invent facts, permissions, verification results, or unavailable evidence. If memory is insufficient, say exactly what is missing and what must be checked before acting." In the five-dimension judge rubric:

  • NAV, 5 points: "When evidence is insufficient, proposing minimal evidence gathering or a safe stop can also earn 5 points."
  • VC, 5 points: "When evidence is insufficient, it makes only supported claims and explicitly identifies the gaps."
  • CSC, 0 points: "The current state is incorrect; a stale value is treated as current; mutually exclusive states are adopted simultaneously; or a critical state is fabricated."

Abstention thus earns high NAV/VC but not CSC, since CSC measures whether the current value is recovered; 93.5% of fixed refusals receive CSC 0. This rubric applies identically to both schedules.

(2) Actual scores. Over 83,460 scored answers (15 compressors × 2 modes), we group reader answers by a rule-based classifier:

CategoryShareCSCSBNAVVCCSC=0VC≥3
Fixed refusal54.7%0.201.482.983.4393.5%85.1%
Refusal plus extra text2.0%1.402.923.783.9752.8%96.5%
Other "not mentioned" wording2.2%2.723.433.783.7920.6%95.6%
Substantive answer41.1%3.753.783.883.8819.0%93.6%
  • Among CSC=0 cases, 84.6% are plain refusals and 12.9% substantive answers. Of the 60.5% CSC zero rate in this population, about 51 percentage points are plain refusals and about 8 are substantive answers assigned zero. The paper's 61.0% comes from a separate 170,896-judgment pool.
  • The VC zero rate is low (10.4%) because 85% of refusals score 3 or more on VC.
  • The refusal share is 61.4% under rolling and 52.1% under fresh; most of the lower CSC under rolling comes from more refusals.
  • For ended-session facts, the refusal share is 60.2% under fresh and 73.7% under rolling, with CSC around 0.23 on refusals.

(3) Interpretation. We interpret the ended-slice CSC drop as reduced retention of completed-task details under a retrospective-query convention (see also Kyiq Q3). Since the same rubric applies to both schedules, abstention scoring affects the absolute level but not the sign of the paired difference.

Reviewer_KyiqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces EndureBench to evaluate language agents’ preservation of critical task state across compaction cycles. It then uses the benchmark to compare fresh and rolling memory compressions across different compressors.

Critical strengths
  • The paper targets an important problem, better quality memory will result in better agent performance in long-running tasks.
Critical weaknesses
  • All the aspects of the benchmark are LLM generated, without human verification and the tasks are also not related. Therefore it is not a well designed benchmark for long-horizen tasks.
  • The paper ignores the additional cost of prompt caching when comparing fresh vs. rolling.
  • The sample size in the benchmark is small, given that it's LLM generated a larger sample size or leveraging various LLMs could help with potential biases that are introduced.
Questions for authors
  • Can you include the caching cost in your comparions?
  • Have you reviewed any of the generated questions manually? If yes, can you report your human verification process?
  • Did you have any samples where the tasks were relevant and did that make a difference in memory comparisons?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Kyiq (Part 1/3)

We sincerely thank the reviewer for their time and constructive comments, and for recognizing the importance of memory for long-running agents. Below, we address your questions on prompt-caching cost, human review of the generated questions, task-related samples, and sample size.

Q1: About including prompt-caching cost in the comparison (Concern 2 & Question 1).

The fresh-versus-rolling comparison ignores the extra cost of prompt caching. Can you include the caching cost in the comparisons?

Ans for Q1: Thank you for this practical suggestion. We estimated compression cost analytically using local-tokenizer counts of recorded calls.

(1) Cost model. For a history with K checkpoints, let IkI_k be the fresh input at checkpoint k (proportional to the prefix, about 150K, 300K, 450K), and let a cached read cost α times the input price:

  • Rolling: each call takes the previous memory plus a new block; the prefix changes every round, so the cache cannot be reused, and input cost is about ∑kIkroll\sum_k I_k^{\text{roll}}.
  • Fresh (no cache): input cost is ∑kIk\sum_k I_k.
  • Fresh (cached): the prefix H1:k−1H_{1:k-1} is billed at α and only the new part at full price, giving ∑k(Ik−Ik−1)+α∑kIk−1\sum_k (I_k - I_{k-1}) + \alpha \sum_k I_{k-1}.

(2) Measured tokens. Per history (averaged over 15 compressors), fresh input is 904,411 tokens (about 3.02 calls of 299K on average) and rolling input is 446,229 tokens; about 51.9% of fresh input is cacheable prefix. Memory produced per call ranges from 1.7K to 10.9K tokens for fresh and 1.9K to 12.4K for rolling.

(3) Cost ratio (fresh / rolling) under official list prices (collected November 2026); values above 1 mean fresh costs more:

CompressorNo cacheα=0.1α=0.25α=0.5α at break-even
gpt-6-astra1.651.0011.111.290.099
claude-opus-51.761.0291.151.360.065
claude-sonnet-51.841.0551.191.400.037
gpt-5.6-terra1.761.0481.171.360.040
gpt-5.6-luna1.781.0471.171.370.043
gpt-5.6-sol1.801.0481.171.380.043
glm-5.31.851.0431.181.400.052
glm-5.21.941.0651.211.450.034
glm-5.3-flash1.891.0611.201.430.033
kimi-k32.001.0921.241.500.009
minimax-m31.881.0431.181.420.053
hy4-preview2.001.0841.241.490.018
deepseek-v4-pro2.011.0941.251.500.007
deepseek-v4-flash1.881.0591.201.420.035
deepseek-v4.1-flash1.881.0611.201.430.033

(Cached columns assume 100% eligible-prefix hits within TTL and no cache-write premium.)

(4) Conclusions:

  1. Fresh is never cheaper at α ≥ 0.1. At α=0.1–0.25 with 100% eligible-prefix hits, fresh costs about 1.00–1.25× rolling; without caching, 1.65–2.01×.
  2. Cache misses widen the gap. Replacing the eligible-prefix price multiplier by 1−h(1−α)1-h(1-\alpha), where h is the token-weighted hit rate, gives fresh/rolling ratios of 1.13–1.28 at h=80% and 1.33–1.55 at h=50% for α=0.1. TTL expiry lowers h, and write premiums add further cost.
  3. Feasibility. Fresh is infeasible when the raw prefix exceeds the context window, whereas rolling remains feasible.

Costs cover compression calls only, using local token counts and list prices.

AuthorsOpenReview ↗

Response to Reviewer Kyiq (Part 2/3)

Q2: About human review of the generated questions (Concern 1 & Question 2).

Have you reviewed any of the generated questions manually? If yes, can you report the human verification process?

Ans for Q2: Thank you for the question. Probes were validated by automated checks (quotation matching, prefix-validity filtering, cross-model arbitration) rather than human review; our reply to jWJv Q2 reports the complete 80-item LLM audit and an exclusion sensitivity using its known flags. No comparative claim depends on this: probes, references, and checkpoints are frozen across both schedules, so whether the questions were human-reviewed affects the credibility of the 61.0% level, not the rolling-versus-fresh paired difference. Accordingly, the revision describes the work as a retrospective-recovery diagnostic rather than a long-horizon task benchmark.


Q3: About task-related samples and whether they change the conclusion (Concern 1 & Question 3).

Did you have any samples where the tasks were relevant, and did that change the memory comparison?

Ans for Q3: Thank you for this question. The release does not contain designed cross-session task dependencies; we report the closest available slice (active vs. ended sessions) and scope the claim accordingly.

(1) Session structure. The 38 histories contain 228 session slots: 4/5/6/7/8/9 sessions in 4/7/16/8/2/1 histories, respectively (mean 6, range 4–9). A block covers 2.64 sessions on average.

(2) Slicing method. We match each probe's evidence quotation against the raw messages before the checkpoint (90.5% of 5,507 quotations matched) and map it to its session. If the fact last appears in the session in progress at the checkpoint, it is active; if it last appears in an earlier, ended session, it is ended. The 106 probes (3.8%) whose quotation could not be found are listed separately.

SliceProbe pairs per modeCSC fresh→rollingQuality fresh→rollingCompressors with ΔQ<0Cells with CSC lowest
All2,7821.93→1.55 (−0.38)3.02→2.72 (−0.29)14/1530/30
anchor2,2841.85→1.38 (−0.46)2.99→2.66 (−0.34)14/1530/30
dynamic4982.30→2.31 (+0.01)3.13→3.03 (−0.09)10/1530/30
active7193.18→3.27 (+0.09)3.67→3.69 (+0.02)8/1523/30
ended1,9571.48→0.92 (−0.56)2.79→2.37 (−0.41)14/1530/30
active, checkpoint ≥22373.18→3.35 (+0.17)3.44→3.37 (−0.06)9/1512/30
ended, checkpoint ≥21,5671.38→0.68 (−0.70)2.68→2.17 (−0.51)14/1530/30

(Probe pairs are counts per compressor and per mode.)

(3) History-equal sensitivity (10,000 history bootstrap draws; 95% intervals):

SliceHistoriesΔQ [95% CI]ΔCSC [95% CI]
active38−0.016 [−0.111, +0.073]+0.048 [−0.041, +0.126]
ended38−0.425 [−0.521, −0.332]−0.567 [−0.657, −0.480]
active, checkpoint ≥234−0.116 [−0.221, −0.016]+0.130 [−0.017, +0.277]
ended, checkpoint ≥238−0.516 [−0.618, −0.413]−0.706 [−0.805, −0.609]

Takeaway: The rolling penalty is concentrated on ended-session facts; on active facts the CSC gap is indistinguishable from zero (482/719 active probes are from checkpoint 1). We therefore interpret the finding as reduced retention of completed-task details under a retrospective-query convention, in which a scoped reference remains valid until updated.

AuthorsOpenReview ↗

Response to Reviewer Kyiq (Part 3/3)

Q4: About sample size and the single generating model (Concern 3).

The sample is small, and because the items are LLM-generated, a larger sample or several generator models would be needed to dilute that bias.

Ans for Q4: Thank you for raising this point. We address sample size and the generating model separately.

(1) History-level inference. We do not treat the 2,782 probes as independent; the unit is the history (38). Within each history we average per (compressor, mode), then over the 15 compressors to obtain a paired fresh/rolling value per history, and run a 10,000-draw history bootstrap and a two-sided Wilcoxon signed-rank test:

ScopeMetricΔ (rolling − fresh) [95% CI]Wilcoxon pHistories with rolling lower
All checkpointsQuality−0.306 [−0.395, −0.221]6.6e-933/38
All checkpointsCSC−0.392 [−0.473, −0.313]3.6e-1137/38
All checkpointsSB−0.557 [−0.670, −0.448]5.1e-1137/38
Checkpoint ≥2Quality−0.466 [−0.567, −0.365]4.0e-1034/38
Checkpoint ≥2CSC−0.590 [−0.684, −0.499]7.3e-1238/38
Checkpoint ≥2SB−0.803 [−0.950, −0.662]1.5e-1137/38

History-equal differences are within 0.03 of the probe-averaged ones. Exclusion, session-slice, and budget analyses use a separate reproducible run (seed 20261110); point estimates agree. CSC is the lowest dimension within a history for 38/38 histories under fresh and 36/38 under rolling.

(2) Session-disjoint subsets. To check whether repeated source sessions drive the contrast, we evaluated session-disjoint subsets selected from metadata (not scores):

  • A fixed degree-then-ID greedy selection retains 19 histories, with history-equal Q and CSC gaps of −0.330 and −0.457.
  • Across 200 seeded random-order greedy selections (15–20 histories each), every Q and CSC difference remains negative: Q ranges from −0.403 to −0.149 and CSC from −0.510 to −0.275; excluding checkpoint 1, from −0.580 to −0.294 and −0.713 to −0.456.

The 228 session slots contain 173 distinct sessions, with 55 used twice across arms and no exact-session reuse within either arm; session overlap forms three components (33/3/2 histories), and the histories span 12 recorded task IDs. This check thus establishes robustness to repeated trajectories.

(3) Generating model. The construction model affects which facts are measured, not the paired contrast. If construction favors particular facts or model families, it changes the measured fact set (external validity); it does not act on rolling alone, because all 15 compressors and both modes share the same probes, references, and checkpoints. Only a construction-preference-by-schedule interaction could produce a paired gap, and reader swapping, source stratification, and family exclusion (tyVp Q1–Q2) show no such interaction. We therefore scope the claim to the released probe set.

We hope these responses resolve your concerns. If so, we would appreciate it if you would consider raising the score, and we are glad to discuss any further questions.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ending Soon

Dear Reviewer Kyiq,

Thank you again for your time and valuable comments on our submission. As the author–reviewer discussion period is coming to an end soon, we would like to gently follow up on our responses posted on 17 Nov.

In our rebuttal, we tried to address each of your concerns: (1) we added a prompt-caching cost model comparing fresh and rolling compaction; (2) we reported our verification pipeline together with an audit of zero-score cases and exclusion-based sensitivity analyses; and (3) we addressed sample size and potential construction bias through history-level statistics, session-disjoint subsets, reader swapping, and family-exclusion experiments.

We would greatly appreciate it if you could let us know whether these responses address your concerns, or whether any points remain unclear. We would be happy to provide further clarification before the discussion period closes.

Best regards, Authors of Submission #58427

Reviewer_jWJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EndureBench, a benchmark that evaluates task state preservation at fixed checkpoints. It compares two context compaction schedules: fresh, which compresses each raw prefix once, and rolling, which recursively compresses the previous memory plus the next block. A shared reader answers the probes from memory alone, and an LLM judge scores the answers on five dimensions. Experiments on 15 compressors show that current-state correctness and scope binding are the weakest dimensions.

Critical strengths
  1. The paper introduces a useful benchmark for finer-grained analysis of task state in coding agents. Fixing the histories, probes, and checkpoints makes memory the only variable, which provides a clean testbed for comparing different context compaction schedules.

  2. The analysis covers 15 models and shows that failures concentrate on current values and task scope. Treating state preservation as a separate measurement target reveals failures that are hard to spot through outcome metrics such as task pass rate.

Critical weaknesses
  1. The paper is hard to read. Some terms are used without definition, and no concrete example of a probe, reference, and answer is shown.

  2. In the proposed benchmark, it heavily relies on LLM labels without human validation. All facts, references, probes, and scores are produced by LLMs, and none are checked by humans. CSC scores 0 on 61.0% of judgments, but it is unclear how many of these zeros reflect reference errors rather than real memory loss. A small human-validated subset would help address this concern.

Questions for authors
  1. Is a fixed, identical checkpoint schedule a good design? In practice, when to compress is part of the compaction strategy: harnesses trigger compaction near the context limit or at sub-task boundaries. Here all compressors share ~150K-token blocks cut at message boundaries, which may split a session midway. Would a different trigger (e.g., compressing at task boundaries, or with smaller or larger blocks) change the main results, such as the fresh–rolling gap or the dimension ordering?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jWJv (Part 1/3)

We sincerely thank the reviewer for their time and insightful comments. We appreciate your recognition of the strengths of our paper, including the fixed-history design in which histories, probes, and checkpoints are held constant so that memory is the only variable, and the finding across 15 models that failures concentrate on current values and task scope, which task pass rate does not reveal. Below are our responses to your questions and concerns.

Q1: About undefined terms and the lack of a concrete example (Concern 1).

Terms are used before they are defined, and no concrete probe, reference, and answer is shown.

Ans for Q1: Thank you for this helpful suggestion. We now define all terms at first use in Section 2 and add a full worked example.

(1) Core terms:

TermMeaning
Task stateA fact keyed by (source scope, entity, attribute). The reference at a checkpoint gives the value currently valid for that key.
Stable anchor / dynamic chainAn anchor keeps one value across all scored checkpoints. A dynamic chain contains an explicit update.
Fresh (single_shot)At checkpoint k, compress the raw prefix H_{1:k} in one pass.
RollingAt checkpoint k, compress the previous memory M_{k-1} together with the new block B_k into M_k.
ReaderFixed to gpt-5.5. It sees only the memory and the question, not the raw history.
Judgeglm-5.2. It scores the answer against the checkpoint reference on five 0–5 dimensions: CSC, SB, PC, NAV, VC.

(2) Worked example (verbatim from the data): History cbv5-homo-4 joins six sessions into three blocks; the compressor is gpt-5.6-luna. The probe (anchor) asks: "What is the current file name of the newly introduced rule action pinning file in the project?" (originally in Chinese). The reference is rule_action_pinning.go at all three checkpoints. The evidence in the raw trajectory is message 390 (cat > rule_action_pinning.go <<'EOF') and message 558 (cat rule_action_pinning.go).

CheckpointModeReader answerCSC/SB/PC/NAV/VC
1freshThe newly introduced action-pinning rule file is rule_action_pinning.go.5/5/5/5/5
1rollingThe newly introduced rule file is rule_action_pinning.go.5/5/5/5/5
2fresh"No such information in the context"0/5/3/5/5
2rollingrule_action_pinning.go5/5/5/5/5
3freshrule_action_pinning.go5/5/3/5/5
3rolling"No such information in the context"0/0/5/5/5

Rolling memories 1 and 2 contain the file name. In memory 3, the task is reduced to "Added the action-pinning rule, configuration, CLI support, documentation, tests, and fixtures", and the file name is gone. The judge's note at checkpoint 3: "The compressed memory does not retain this filename. The candidate honestly says 'no information in context.' Correct fact is absent." CSC is 0 while NAV and VC stay at 5, because the reader correctly abstains; this is exactly how an overall score can hide a lost current value. Fresh loses the file name at checkpoint 2 and recovers it at checkpoint 3, since each checkpoint is compressed independently. This example is illustrative; population-level evidence is given in Q2 and Kyiq Q4.

AuthorsOpenReview ↗

Response to Reviewer jWJv (Part 3/3)

Q3: About the fixed ~150K checkpoint schedule and alternative compaction triggers (Question).

Is a fixed, identical checkpoint schedule a good design? Here all compressors share ~150K-token blocks cut at message boundaries, which may split a session midway. Would a different trigger (e.g., compressing at task boundaries, or with smaller or larger blocks) change the main results, such as the fresh–rolling gap or the dimension ordering?

Ans for Q3: Thank you for this insightful question. We answer in two parts.

(1) Existing checkpoints almost always cut inside a session. Aligning message ranges shows that 76 of the 77 non-final checkpoints fall in the middle of a session, and only 1 falls exactly on a session boundary (checkpoint 1 of cbv5-heter-6). A block covers 2.64 sessions on average, and the cut session is on average 39–43% complete at the cut. Existing checkpoints therefore do not admit an aligned-vs-cut comparison; we vary the trigger directly in (2).

(2) Alternative triggers and chunk sizes. We ran rolling with glm-5.3 and claude-sonnet-5 on all 38 histories. Evaluated checkpoints retain their original positions, probes, and references:

  • chunk75: split each block near its token midpoint at a legal turn boundary, adding an intermediate compression while retaining the original evaluation checkpoints.
  • chunk300: merge adjacent blocks and evaluate only at the merged block's end. The 37 three-block histories are evaluated at checkpoints 2 and 3; the four-block history at 2 and 4. Compression calls and evaluation checkpoints decrease from 115 to 76 per compressor (33.9% fewer).
  • session: compress at the union of session ends and original checkpoints, evaluating at the original checkpoints; coincident boundaries do not cause duplicate calls.

The reader is deepseek-v4.1-flash and the judge is unchanged. The analysis uses a fixed five-condition intersection (original fresh, original rolling, chunk75, chunk300, session) within each compressor: 1,634 probe/checkpoint keys over 37 histories for glm-5.3 and 1,776 over 38 for claude-sonnet-5. Brackets are 95% history-bootstrap intervals (10,000 draws, seed 20261111).

VariantCompressorΔQ (vs. original rolling)ΔCSC
chunk75glm-5.3−0.260 [−0.591, +0.080]−0.333 [−0.526, −0.151]
chunk75claude-sonnet-5−0.214 [−0.508, +0.075]−0.035 [−0.184, +0.098]
chunk300glm-5.3−0.171 [−0.503, +0.167]−0.195 [−0.389, −0.015]
chunk300claude-sonnet-5−0.024 [−0.415, +0.365]+0.167 [−0.067, +0.405]
sessionglm-5.3−0.255 [−0.586, +0.083]−0.246 [−0.478, −0.009]
sessionclaude-sonnet-5−0.112 [−0.461, +0.233]+0.210 [−0.013, +0.442]
  • A separate four-condition intersection excluding chunk300 retains original-checkpoint coverage (2,543/2,615 keys, 38 histories each). There, chunk75/session CSC differences are −0.400/−0.424 for glm-5.3 and +0.147/+0.255 for Claude (chunk75 [−0.064, +0.346]; session [+0.003, +0.510]).
  • Original rolling−fresh gaps on the five-condition intersection are Q −0.427, CSC −0.357 for glm-5.3, and Q −0.212, CSC −0.117 for Claude, permitting matched comparisons to saved fresh outputs at retained checkpoints.

Takeaway: None of the three variants improves overall quality over original rolling (all six ΔQ point estimates are negative); CSC effects are compressor-dependent (negative for glm-5.3, positive for claude-sonnet-5 under chunk300/session). Regenerating questions and references at new checkpoints changes the measured object itself and is left to future work. These results are added to the appendix of the revision.

Finally, we hope these responses address your concerns and appreciate your constructive feedback. If you find our response adequate, we would appreciate it if you would consider raising the score. We are glad to discuss any further comments and suggestions.

Thank you for the clarifications and the additional experiments. The audit and sensitivity analyses are helpful.

However, my main concern remains whether the probes and checkpoint references are reliable enough to interpret the high CSC zero rate. The audit is still LLM-based, and a human-validated subset, which I believe is feasible given the localized evidence for each reference, has not yet been completed. I am therefore keeping my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks to all Reviewers for Recognition and Constructive Comments (Global Comment, Part 1/2)

Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank all reviewers for their time and constructive comments. The reviewers have acknowledged the clean fixed-history design that isolates memory as the only variable, the diagnostic value of separating current-state and scope failures from overall quality, the new schedule-gap metric Δc\Delta_c, and the importance of memory for long-horizon agents. Below, we summarize the strengths highlighted by the reviewers:

[Clean Experimental Design]:

  • Reviewer jWJv: Fixing histories, probes, and checkpoints makes memory the only variable, giving a clean comparison of compaction schedules.
  • Reviewer tyVp: Fixed-history checkpoint replay is the right basis, and author-set block boundaries are a feature rather than a limitation.

[Informative Diagnostics & New Metric]:

  • Reviewer jWJv: Across 15 models, failures concentrate on current values and task scope, revealing what task pass rate does not.
  • Reviewer tyVp: The schedule gap Δc\Delta_c is a useful new metric; the penalty falls on scope binding and current-state correctness rather than verification calibration; rolling memories are longer yet score lower, so shorter memory does not explain the gap. "The results are interesting, real and useful."

[Problem Importance & Population]:

  • Reviewer Kyiq: The problem matters: better memory should directly improve long-running agents.
  • Reviewer tyVp: Fifteen compressors across several model families form a real population, supported by several robustness diagnostics.

During the rebuttal period, we have conducted extensive additional analyses, including history-level inference, an LLM-based audit, reader swapping, family exclusion, alternative compaction triggers, fixed summary budgets, keyed in-place editing, and a prompt-caching cost model. All numbers are derived from saved scores, completed runs, or reproducible offline analyses. New material in the revised PDF is highlighted. We first state the common framing, then give point-by-point responses.

[Overview: Three Layers of Claims]:

LayerClaimWhat it requiresStatus in this round
A. Paired contrastUnder fixed histories, probes, references, checkpoints, and judge, rolling is below fresh on CSC/SB and overall qualityOnly that both schedules are measured with the same rulerRobust: sign and magnitude survive history-level bootstrap/Wilcoxon, dropping checkpoint 1, reader swap, compressor-family exclusion, and removal of known underspecified keys
B. Within-measurement orderingCSC is the lowest dimension in all 30 cells; compressor ordering is not an artifact of the 0/3/5 scaleOnly saved scoresRobust: 30/30 cells; rank correlations 0.946–0.993 under threshold recoding
C. Construct-level readingThe 61.0% CSC zero rate means memory loss; dropping completed-task facts means unreasonable forgetting; the benchmark represents real long-horizon workHuman validation, task-relevant data, and a different generating modelScoped in the revision; no comparative claim depends on it

This separation follows directly from the paired design:

  • Layer-C defects are shared across the two schedules. Both use the same histories, probes, references, checkpoint schedule, and judge. A wrong reference, an ambiguous question, or a judge that is harsh on abstention shifts both arms in the same direction and cancels in the paired difference.
  • Neither reader nor judge sees the mode label. The reader prompt receives only the memory and the question; the judge prompt receives only task_scope, question, state_reference, constraints, compressed_memory, and candidate_response. Judge strictness therefore cannot be conditioned on the schedule. Even if the judge preferred longer or more polished text, rolling memories are longer for all 10 compressors with token records (1.02–1.36×), so such a preference would shrink rather than create the negative gap.
  • Only a schedule-dependent bias could move Layer A. We tested this four ways: trajectory-source stratification, a non-GPT reader, exclusion of judge/construction families, and dropping checkpoint 1. None reverses the sign.

We narrow Layer C to "retention of completed-task details under a retrospective-query convention" and state this convention in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.