acceptodds
Under review as a conference paper at ICLR 2027

Never Optimize Your Memory-Compression Harness Under Training–Inference Mismatch

Abstract

Production agent harnesses, including Claude Code and Qwen-Agent, compress an agent's context during rollout to make long-horizon interaction tractable. Training policies under such compression, however, introduces a fundamental conditioning problem: each compression-induced eviction branches the effective interaction history, making the object presented to the learning objective a tree rather than a single trajectory sequence. Existing learning pipelines typically linearize this tree in one of two ways: (1) retaining only the rightmost root-to-leaf path, which causes time-travel leakage, or (2) replaying the full depth-first traversal, which induces a train–inference mismatch. Thus, we introduce two exact conditioning-level corrections: (1) LogitTree, a segmented K-forward traversal of the trajectory tree, and (2) an equivalent packed Logit attention mask. LogitTree requires K+1 backward passes for every root-to-leaf, whereas the Logit formulation requires both a custom masked-attention kernel and white-box access to harness eviction records. To avoid these costs, we further propose SDCC (Self-Distillation for Conditioning Consistency), a training-friendly variational relaxation requiring only a single backward pass and multiple parallel forward passes. At each eviction junction, SDCC minimizes the forward KL divergence from a stop-gradient teacher distribution conditioned on the reconstructed pre-eviction prefix to the compressed-context student policy. By Pinsker's inequality, a residual per-junction KL divergence of ε_KL yields an O(√ε_KL) bound on the resulting train–deployment behavioral gap in total variation. In black-box harnesses, SDCC uses captured per-turn request payloads and requires no span-level eviction logs. Experiments across three white-box editors, two black-box harnesses, and seven web-search benchmarks reveal substantial conditioning drift under naive replay. LogitTree and Logit masking recover the no-compression numerical baseline, while SDCC substantially narrows the gap with a modest computational budget. On the WideSearch benchmark, our LogitTree method improves the Item F1 of Qwen3.7-Air (125B-A6B) from 67.32 to 74.03, surpassing all reference models in our evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory, Harness, Reinforcement Learning, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ef7YOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LogitTree addresses a mismatch between generation and training when an agent evicts or compresses its memory. Naive replay can remove evidence from earlier targets or restore evicted text to later ones. LogitTree replays each response under the live prefix available at generation time and uses disjoint loss masks so that every target contributes once. SDCC provides a cheaper alternative: a compressed-context pass is trained with forward KL toward the same model's detached live-prefix prediction. The student branch still receives gradients from this loss. Across three white-box and two black-box harnesses, the authors report better accuracy than naive compressed-context training, with SDCC recovering much of LogitTree's gain at lower cost.

Critical strengths

S1. The paper states a concrete conditioning requirement: each target should see exactly what it saw when it was generated, and should contribute to the loss once. Framed this way, the requirement excludes two easy mistakes in replay, namely stripping evidence from earlier targets and restoring evicted text to later ones.

S2. SDCC recovers most of LogitTree's reported accuracy gain at a much lower cost. On MemexRL, macro EM goes from 28.9 for Naive-Compressed to 45.9 with LogitTree and 43.1 with SDCC, at relative costs of 1.00, 4.20 and 1.55.

Critical weaknesses

W1. The single-copy graph is not generally equivalent to independent branch replay. A retained token can carry information from evicted text in its intermediate states and pass it to later targets. Matching direct visibility and token positions does not remove that indirect path. The appendix describes a viable block-diagonal implementation with branch replication and summed branch costs, but the main figure and exactness argument need to describe that executed computation consistently.

W2. Checkpoint selection uses the test suite. A disjoint development set produces a shortlist, but the reported checkpoint is whichever performs best on the full test suite. This makes the test results part of model selection. The black-box averages also differ: the text gives SDCC/LogitTree values of 37.5/35.9 and 36.9/35.0, while the main table gives 42.6/42.4 and 43.9/42.9.

Questions for authors

Q1. Which computation graph does the exactness statement describe? Please compare losses and gradients with independent per-branch replay on one batch containing an eviction. The main illustration and cost statement should then be aligned with the executed graph and its summed branch lengths.

Q2. Can you report results with checkpoints selected only on the development set and then evaluated on a locked test suite? For the two black-box harnesses, which runs and checkpoints produced each set of conflicting averages?

6 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1 Part (1/2)

We thank the reviewer for recognizing our formulation of generation-time conditioning consistency and the value of SDCC's accuracy–cost trade-off.

Your comments identify two aspects that need to be made explicit: the exactness argument must consistently describe the computation actually executed, and the evaluation protocol and black-box results must have clear provenance. We address these concerns below.

W1 / Q1: Which computation graph does the exactness claim describe? Do its losses and gradients agree with independent branch replay, and what is its computational cost?

Our exactness claim concerns the branch-replicated, block-diagonal attention construction implemented in Appendix H. This implementation computes hidden states separately for each branch and restricts attention within that branch at every layer, excluding the indirect cross-branch information path raised by the reviewer. We will align the main figure, proofs, and cost descriptions with this execution graph.

(1) Each branch performs the same computation as independent replay.

Each branch has its own copies of the input tokens, and its hidden states are computed from that branch's inputs. When the same token appears in different branches, its hidden states are computed separately under the respective contexts. A branch does not reuse hidden states from another branch that has attended to content evicted from its own context.

Block-diagonal causal attention prevents cross-branch access at every layer, so this isolation holds throughout the computation. Packed LogitMask and independent LogitTree replay use:

  • The same branch inputs and position IDs;
  • The same causal attention within each branch;
  • The same target assignments, loss weights, and normalization;
  • For RL objectives, the same old-policy probabilities, advantages, and other objective-specific quantities.

The loss mask assigns each original generated target to exactly one branch reproducing its generation-time context, so each target contributes to the task loss once. Replicated context tokens remain part of their respective branch computations.

Under the stated architectural assumptions, with no additional cross-branch coupling and with stochastic operations disabled or correspondingly matched, the hidden states of the two executions agree layer by layer. Let Tb\mathcal{T}_b denote the targets assigned to branch bb. Then :

Lpacked(θ)=1Z∑b∑twt ℓt(θ);t∈Tb\mathcal{L}_{\mathrm{packed}}(\theta)=\frac{1}{Z}\sum_b \sum_t w_t\,\ell_t(\theta); t\in\mathcal{T}_b

1Z∑b∑twt ℓt(θ)=Llogitmask(θ);t∈Tb\frac{1}{Z}\sum_b \sum_t w_t\,\ell_t(\theta) = \mathcal{L}_{\mathrm{logitmask}}(\theta); t\in\mathcal{T}_b

The target sets are disjoint and jointly cover all original training targets. Consequently, the two executions have identical target logits, scalar losses, and parameter gradients in exact arithmetic.

Figure 3(c) did not explicitly depict the branch copies and their independently computed hidden states, making it possible to interpret the diagram as a single-copy execution sharing hidden states across branches. We will redraw the figure and align Section 3, Proposition 3, Theorem 1, and Appendix H so that the exactness argument explicitly applies to the implemented branch-replicated computation graph.

AuthorsOpenReview ↗

Rebuttal for W1&Q1 Part (2/2)

(2) A historical same-model, same-trajectory comparison supports numerical agreement in the FP32 reference configuration.

We recovered the results of the historical experiment e01-20natural-r16; we did not rerun this experiment for the rebuttal. It fixes the same complete 36-layer Qwen3-4B model and compares the two executions using TP=8 and FP32 math SDPA on 20 trajectories containing natural AgentFold memory evictions. Each trajectory contains three branches, forming one packed group.

ExecutionComputation
Independent branch replayEvaluate each branch separately and aggregate losses and gradients using the same target assignments, weights, and normalization
Branch-replicated LogitMaskPack the same branches, isolate them with block-diagonal causal attention, and use the same target assignments and loss rules

The experiment covers 60 branches and 134,145 target tokens.

CheckHistorical result
Maximum absolute difference in target log-probabilities2.58446×10−42.58446 \times 10^{-4}
Mean absolute difference in target log-probabilities3.47022×10−63.47022 \times 10^{-6}
Maximum across trajectories of the absolute difference in normalized NLL loss1.57161×10−71.57161 \times 10^{-7}
Maximum across trajectories of the absolute difference in normalized controlled PPO loss5.49480×10−75.49480 \times 10^{-7}
Maximum across trajectories of the relative L2 error in NLL gradients4.00311×10−44.00311 \times 10^{-4}
Maximum across trajectories of the relative L2 error in controlled PPO gradients2.65254×10−42.65254 \times 10^{-4}
Trajectories passing the historical script's prespecified FP32 thresholds20/20

The mean log-probability difference is weighted over all target tokens. Gradient errors follow the historical script's aggregation across TP shards, including replicated parameters. The PPO results concern a controlled objective test. We will specify the thresholds and metric definitions in the revision.

These results provide a same-model, same-trajectory loss and gradient comparison with natural evictions, supporting numerical agreement for branch-replicated execution in the tested FP32 reference configuration. The layer-by-layer correspondence establishes mathematical equivalence, while this experiment quantifies numerical differences in that specific execution configuration.

(3) Cost is reported using actual branch lengths, including replicated context.

Let ℓb\ell_b be the input length of branch bb. The actual non-padding input size is

Npacked=∑bℓb.N_{\mathrm{packed}}=\sum_b\ell_b.

This quantity includes copies of shared context appearing in different branches. For the 20 trajectories above:

Input quantityTotalMean per trajectory
Number of branches603
Branch input tokens, including copies and excluding padding189,3689,468.4
Packed input positions, including padding198,6569,932.8
Independent-replay input positions, including padding214,01610,700.8

The padded position counts were reconstructed from the historical configuration.

One packed forward/backward describes how execution is organized; it does not eliminate computation for the branches or their context copies. If the attention backend skips cross-branch blocks, the quadratic attention term scales with ∑bℓb2\sum_b\ell_b^2. If it computes the full dense matrix before applying the mask, that term instead scales with Npacked2N_{\mathrm{packed}}^2.

We will align Figure 3(c), Table 6, and the associated cost derivations with this execution graph, using the sum of actual branch lengths to report input size.

AuthorsOpenReview ↗

Rebuttal for W2&Q2 Part (1/3)

W2 / Q2: Were checkpoints selected using only the development set? How do the Table 1 metrics correspond, and which runs produced the two sets of black-box results?

We confirm that the reported models were selected by validation performance; test scores were not used for checkpoint selection. The statement in Section 5.1 describing selection of the best test-performing checkpoint among fully evaluated checkpoints does not reflect the procedure actually followed. The reviewer's concern follows directly from this incorrect description. We will correct it and apologize for the confusion.

(1) The actual protocol selects checkpoints on validation data and reports their performance on a fixed test suite.

We compare candidate checkpoints on a fixed validation set disjoint from the test examples, select the checkpoint with the highest validation score, and report that model's performance on the fixed test suite. Valid Avg. EM is the equally weighted mean over the validation portions of the seven benchmarks.

To make the selection evidence concrete, the following table reports recovered validation records for 12 method/harness configurations. For each configuration, it lists up to five higher-scoring candidates among the recovered records, ordered by checkpoint number. The selected checkpoint is shown in bold.

MethodHarnessCandidate checkpoint → Valid Avg. EM
Naive-FullAgentFold132 → 27.56; 184 → 22.99; 205 → 14.61; 219 → 14.31; 223 → 14.91
Naive-CompressedTC-RAG39 → 35.29; 45 → 35.86; 49 → 36.23; 53 → 37.64; 55 → 39.24
Naive-CompressedAgentFold214 → 45.81; 218 → 46.63; 220 → 45.84; 228 → 45.80; 232 → 46.53
Naive-CompressedMemexRL67 → 25.10; 69 → 23.63; 71 → 25.84; 73 → 27.86; 75 → 29.64
LogitMask / 4D-MaskTC-RAG147 → 38.43; 159 → 38.09; 161 → 41.13; 163 → 40.09; 165 → 39.90
LogitMask / 4D-MaskAgentFold63 → 34.99; 65 → 33.61; 67 → 34.94; 69 → 33.94; 71 → 35.53
LogitMask / 4D-MaskMemexRL73 → 31.94; 80 → 31.56; 81 → 31.60; 131 → 31.61; 133 → 34.43
LogitTreeTC-RAG184 → 40.81; 186 → 41.00; 192 → 39.43; 194 → 39.91; 196 → 38.97
LogitTreeAgentFold96 → 34.43; 104 → 37.57; 112 → 39.77; 120 → 38.36; 128 → 39.66
SDCCTC-RAG168 → 44.83; 184 → 45.37; 192 → 46.44; 197 → 45.43; 199 → 47.36
SDCCAgentFold159 → 40.74; 166 → 44.06; 167 → 42.33; 169 → 43.07; 175 → 35.06
SDCCMemexRL134 → 44.33; 135 → 43.14; 138 → 44.34; 191 → 42.40; 199 → 41.76

These records document checkpoint selection; the different checkpoints within a configuration are model snapshots from different training steps.

Reviewer_bTASOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies agent RL under context compression. The authors argue that edits to the context cause different tokens to be generated under different histories; consequently, replaying either the final compressed trajectory or the full physical trace produces a train–inference conditioning mismatch. The paper proposes two exact approaches, multi-branch LogitTree and packed Logit attention mask, as well as the single-backward approximation SDCC. Experiments cover multiple editors, white-box and black-box harnesses, and web-search tasks.

Critical strengths
  1. The paper identifies an important problem: context compression changes token-level conditioning histories, not merely memory contents.
  2. The paper proposes both exact and approximate solutions, offering different trade-offs between conditioning consistency and computational efficiency.
  3. The paper evaluates the proposed methods across multiple editors and black-box harnesses, reporting both conditioning diagnostics and downstream task performance.
Critical weaknesses
  1. The proposed methods may differ from Naive-Compressed in both conditioning and the set of supervised tokens. The exact methods also supervise tokens that are later evicted, whereas these tokens may be absent from the final compressed trajectory. A target-matched ablation is needed to disentangle the benefits of correct conditioning from additional supervision.
  2. Table 1 uses different checkpoint-selection criteria for different metrics. Reward and logdiff are reported at the highest-reward iteration, whereas EM is selected from fully evaluated checkpoints. It is unclear whether these metrics correspond to the same checkpoint, potentially complicating their interpretation.
  3. The Claude Code and OpenCode average EM values in Section 5.2 are inconsistent with Table 1. Moreover, the experiments lack multi-seed results or confidence intervals, making it difficult to assess statistical reliability, particularly given the variance introduced by online search and RL rollouts.
  4. The WideSearch experiment compares an SFT base model with a model trained using RL and the proposed method, without matched RL baselines. Thus, the gains cannot be attributed specifically to conditioning correction. Furthermore, the comparability of external model results is unclear because matched agent settings are not established.
  5. Figure 3(c) presents the attention-mask approach without token replication, whereas Appendix H uses branch-replicated packing. Since retained tokens may indirectly carry information from evicted tokens across Transformer layers, the equivalence of the non-replicated formulation remains unclear. The authors should clarify this discrepancy and its computational implications.
Questions for authors
  1. For each harness, what fraction of rollout targets is absent from the final compressed trajectory? Could the authors provide a control that changes only target-loss support, but not conditioning, to quantify the benefit of supervising evicted tokens itself?
  2. For every Table 1 method, is the checkpoint used for reward/logdiff the same as the one used for Avg. EM? If not, please provide their correspondence. In addition, which Claude Code/OpenCode EM values are correct: those in the Section 5.2 text or those in Table 1?
  3. How many random seeds were run for each Table 1 cell? Could the authors provide per-seed Avg. EM, logdiff, and reward, or at least means and confidence intervals?
  4. Were the nine WideSearch reference models evaluated through the same Claude Code agent scaffold, search backend, and context budget? If not, what is the intended comparability between those results and the matched pair at the bottom of Table 2?
  5. What do the “15 LogitTree steps” in the text and the 40-step x-axis in Figure 4 respectively denote? Please also report total GPU-hours, rollouts per step, and total training tokens for this experiment.
12 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

We thank the reviewer for the detailed questions. We address them with matched-target ablations, three-seed evaluations, full-model loss and gradient checks, and additional Claude Code/OpenCode training runs.

The added evidence includes a 2.89-pp average gain from conditioning correction with matched targets, positive in all three seeds, numerical agreement between independent and packed branch replay on 20 natural eviction trajectories, and 48.0/46.3 macro-EM for SDCC on the simplified Claude Code/OpenCode configurations. We also provide the checkpoint mappings, shared evaluation settings, and training-resource accounting requested by the reviewer.


W1 / Q1: Do the improvements come from conditioning correction or additional supervision? What fraction of targets is missing from the final compressed trajectory?

To separate conditioning correction from target coverage, we added a matched-target ablation that evaluates both factors explicitly.

On Qwen3-4B with AgentFold, correcting conditioning while retaining the same task-loss targets improves macro-EM by 4.24, 2.85, and 1.57 pp across three seeds, averaging 2.89 pp. This provides evidence that conditioning correction can improve performance without adding original response targets to the task loss.

We compare four configurations:

ConfigurationContext used for the task lossTargets included in the task loss
A: Naive-CompressedContext in the final compressed trajectoryRetained targets matched to their original identities
B: LogitTree-retainedEach target’s generation-time contextThe same retained targets as A
Cₑ: Retained + explicitly evicted targetsEach target’s generation-time contextB’s targets plus explicitly evicted original targets
C: Full LogitTreeEach target’s generation-time contextAll original targets

For a given original rollout, A and B match token IDs, old log-probabilities, and the normalization denominator. B, Cₑ, and C use the same branch inputs and select task-loss targets through their loss masks. All configurations use the original number of generated response targets in that rollout as the normalization denominator.

All configurations use seeds 42, 43, and 44 and are evaluated after 400 optimizer updates. Each full evaluation covers 38,270 questions:

ConfigurationSeed 42Seed 43Seed 44Mean ± sample SD
A: Naive-Compressed29.4230.5539.6333.20 ± 5.60
B: LogitTree-retained33.6633.4041.2036.09 ± 4.43
Cₑ: Retained + explicitly evicted targets40.3842.2728.5337.06 ± 7.45
C: Full LogitTree39.8440.7731.8237.48 ± 4.92
ComparisonDifferences for seeds 42 / 43 / 44 (pp)Mean ± sample SD (pp)
B − A: Correct conditioning with matched targets+4.24 / +2.85 / +1.57+2.89 ± 1.34
Cₑ − B: Add explicitly evicted targets+6.72 / +8.87 / −12.67+0.97 ± 11.86
C − B: Restore all original targets+6.18 / +7.37 / −9.38+1.39 ± 9.35

The central result is that conditioning correction improves performance in all three seeds while preserving the original task-loss target set. The B − A contrast directly controls target coverage; Cₑ − B and C − B separately quantify the effect of restoring additional targets.

Full LogitTree reaches 37.48 macro-EM, compared with 33.20 for Naive-Compressed: an average improvement of 4.28 pp after 400 updates.

Table 1 supplies a complementary comparison with full target coverage: Naive-Full and full LogitTree both supervise every original response target exactly once, including subsequently removed targets. LogitTree improves EM by 5.2, 9.8, and 13.8 pp on TC-RAG, AgentFold, and MemexRL, respectively. The new matched-target ablation directly tests the conditioning contribution within this broader empirical picture.

For Q1, we count original response targets by identity, excluding prompts, tool-returned text, and replicated branch copies:

HarnessOriginal response targetsTargets retained in the final representationMissing-target fraction
TC-RAG13,271.111,064.816.63%
AgentFold9,247.65,743.537.89%
MemexRL12,997.410,937.315.85%
Claude Code31,574.826,450.616.23%
OpenCode33,171.227,126.918.22%

For SDCC, Eq. (10) makes the supervision structure explicit: the task loss remains on the retained compressed-stream targets, and the added teacher-to-student KL supplies distributional supervision. Setting λ=0\lambda=0 recovers Naive-Compressed. Thus, SDCC and Naive-Compressed share the hard-target task-loss support, while full LogitTree additionally restores the task-loss terms of evicted original targets.

AuthorsOpenReview ↗

Rebuttal for W2&Q2

W2 / Q2: Do reward, logdiff, and Avg. EM in Table 1 correspond to the same checkpoint? Which Claude Code/OpenCode numbers are correct?

Table 1 uses two reporting points: iteration K for reward/logdiff and the development-selected checkpoint N for test EM. The correct submission-era Claude Code/OpenCode EM values are those in Table 1.

  • The original table reports Reward Max at the training diagnostic iteration with the reported maximum training reward, together with logdiff at that iteration. Avg. EM is reported for the checkpoint selected using the development set.
  • The selection procedure uses a fixed development set of 1,325 questions, disjoint from all test benchmarks. Candidate checkpoints are compared on this development set, and the checkpoint with the highest development score is selected for test reporting. Test scores do not participate in this selection.

We will correct §5.1 to state this development-only selection procedure explicitly. For the recovered development-set checkpoint tables, please see our replies to Reviewer ef7Y, titled Rebuttal for W2&Q2 Part (1/3) and Rebuttal for W2&Q2 Part (2/3). These tables provide the checkpoint-selection records without duplicating them in this response as they are two huge.

Training rollout reward and development performance are measured on different data. The K → N mapping below makes the two reporting points explicit. The following table identifies the original diagnostic iteration K and the checkpoint N used for test EM. Reward Max and logdiff belong to K.

HarnessNaive-Full: K → NNaive-Compressed: K → NLogitMask: K → NLogitTree: K → NSDCC: K → N
TC-RAG108 → 168108 → 55169 → 16144 → 186113 → 199
AgentFold113 → 132130 → 21846 → 7122 → 11245 → 166
MemexRL180 → 53235 → 7516 → 13345 → 162113 → 138

For Claude Code and OpenCode, §5.2 retained results from earlier training runs, whereas Table 1 used the submission-era reruns:

HarnessMethodEarlier-run checkpointEarlier EM in §5.2Submission-run checkpointSubmission EM in Table 1
Claude CodeSDCC3837.55242.6
Claude CodeLogitTree4135.95742.4
OpenCodeSDCC3236.95443.9
OpenCodeLogitTree3635.06142.9

The submission-era results place SDCC 0.2 pp above LogitTree on Claude Code and 1.0 pp above it on OpenCode. We will update §5.2 and the associated discussion to use this same result version throughout.

The revised table headings and captions will label training diagnostics at K and test EM at the development-selected checkpoint N, preserving a direct mapping from each reported metric to its model state.

Additional runs with simplified Claude Code/OpenCode configurations

After submission and before rebuttal, we additionally reran training and evaluation with simplified harness configurations. For Claude Code, we removed the default system prompt and trimmed tool definitions while retaining the task instructions and necessary tools, reducing the prompt context to approximately 7K tokens. Within each harness, LogitTree and SDCC use the same initial checkpoint, prompts, tool set, and training/evaluation protocol. We report the equally weighted macro-EM over the seven benchmarks:

Simplified harnessMethodNQTriviaQAHotpotQA2WikiMuSiQueBamboogleFRAMESMacro-EM
Claude CodeLogitTree46.674.444.251.018.952.926.544.9
Claude CodeSDCC48.178.144.150.721.659.933.448.0
OpenCodeLogitTree46.873.543.651.218.956.426.945.3
OpenCodeSDCC48.378.242.746.020.857.031.246.3

Under these simplified configurations, SDCC reaches 48.0 macro-EM on Claude Code and 46.3 on OpenCode, exceeding LogitTree by 3.1 and 1.0 pp, respectively.

AuthorsOpenReview ↗

Rebuttal for W3&Q3

W3 / Q3: How many random seeds were used, and how reliable are the reported results?

Table 1 reports one training seed per configuration. We have expanded two key comparisons to three independent seeds and now report the individual scores, sample SDs, and confidence intervals.

We added three-seed experiments for two key comparisons:

  • Naive-Compressed versus SDCC on AgentFold.
  • LogitTree versus LogitMask on MemexRL.

Each method is trained independently with seeds 42, 43, and 44, using the same training settings and evaluation protocol within a harness. All additional runs are evaluated after 400 optimizer updates, with 38,270 questions per full evaluation.

The fixed-400-update evaluation complements Table 1’s development-selected checkpoints by comparing the methods at a common optimizer-update budget.

Harness / methodSeed 42 EM [95% CI]Seed 43 EM [95% CI]Seed 44 EM [95% CI]Across-seed mean ± sample SD
AgentFold / Naive-Compressed29.42 [27.08, 31.83]30.55 [28.08, 33.07]39.63 [37.01, 42.28]33.20 ± 5.60
AgentFold / SDCC34.27 [31.79, 36.88]32.00 [29.60, 34.55]44.05 [43.64, 44.66]36.77 ± 6.40
MemexRL / LogitTree45.49 [41.93, 49.15]47.75 [44.29, 51.38]44.46 [40.90, 48.24]45.90 ± 1.68
MemexRL / LogitMask44.65 [40.73, 48.72]44.94 [41.43, 48.56]42.38 [38.95, 45.99]43.99 ± 1.40
  • On AgentFold, SDCC outperforms Naive-Compressed in all three seeds, achieving 36.77 ± 6.40 versus 33.20 ± 5.60 macro-EM. The per-seed differences, SDCC − Naive-Compressed, are +4.85, +1.45, and +4.42 pp, with a mean gain of +3.57 pp and a sample SD of 1.85 pp.

  • On MemexRL, LogitTree outperforms LogitMask in all three seeds, achieving 45.90 ± 1.68 versus 43.99 ± 1.40 macro-EM. The per-seed differences, LogitTree − LogitMask, are +0.84, +2.81, and +2.08 pp, with a mean gain of +1.91 pp and a sample SD of 1.00 pp.


For AgentFold, we additionally report reevaluation reward and conditioning discrepancies on a fixed probe:

HarnessMethodFull-evaluation reward: mean ± sample SDFixed-probe mean absolute logdiff: mean ± sample SD
AgentFoldNaive-Compressed0.3362 ± 0.02960.8405 ± 0.0140
AgentFoldSDCC0.2985 ± 0.06160.5627 ± 0.0293
  • The fixed probe uses the same trajectories and target set across methods. For each evaluated checkpoint, the same model weights recompute target log-probabilities under the original generation-time context and under the method’s training-replay context.

  • On this common probe, SDCC reduces mean absolute logdiff from 0.8405 to 0.5627, a reduction of 0.2778, or approximately 33.1%. The probe measures conditioning discrepancy on shared targets, while the EM and reward results report end-task behavior.


Both comparisons show a consistent performance ordering across all three seeds at the same 400-update budget: SDCC improves over Naive-Compressed on AgentFold, and LogitTree improves over LogitMask on MemexRL. The AgentFold probe further shows that SDCC reduces conditioning discrepancy by 33.1% on shared trajectories and targets.

Reviewer_7KTmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of rollout and training trajectory mismatch due to context compression in agent harness. It discusses two exact solutions (logitree and logimask) and proposes a cost-efficient solution SDCC, where an additional loss function is added to RL training. The loss mimics self-distillation, where the predicted tokens of the compressed path try to regress to the original path.

Critical strengths
  1. The problem studied is practical. Context compression is universal for agent harness yet it introduces inconsistency in training and actual usage.

  2. The paper offers both exact solutions and an approximate solution (SDCC).

Critical weaknesses
  1. The intuition for why adding the additional SDCC loss is not super clear. Yes, it pulls the compressed path towards the original path, but this loss may be in different direction from the RL loss, and the original path's predictions may not always lead to a good reward. It is thus a tradeoff, deviating from the reward while trying to mimic the more "original" response, but it is not intuitively clear that it is a tradeoff worth making.

  2. The exact solutions are not completely new. As discussed by the paper, Slime already provides a method that is essentialy LogitTree.

  3. In experiments there is no comparison that is token or wall-clock time budget controlled with SDCC.

  4. The variance is very large: LogitTree and Logit mask, which should have identical gradients, but differ by 12.5 points on MemexRL, with no seeds or confidence intervals reported.

Questions for authors

See weaknesses.

9 comments
AuthorsOpenReview ↗

Rebuttal for W1

W1: Why introduce SDCC's consistency constraint, and how does it interact with the RL objective?

Thank you for raising this question. SDCC follows the idea of on-policy self-distillation (OPSD): the same current policy acts as teacher and student under different contexts. The teacher uses the context available when an action was generated, while the student learns from the compressed view. This provides a direct way to transfer the policy's predictions across context compression.


(1) Intuitive motivation: self-distillation from the generation-time context

Why use the generation-time context as the teacher's input? For a retained target token, let zpz_p denote the context actually visible when it was generated and xpx_p the context used to replay it after compression. The teacher has access to generation-time evidence that may be absent from the compressed view. This is the privileged information that SDCC uses for context distillation.

In Naive-Compressed, replaying the token under xpx_p can change its probability even when the model parameters remain unchanged. When compared with the recorded rollout probability, this change enters the GRPO importance ratio, mixing the effects of policy updates and context changes. SDCC keeps the reward-driven objective on the compressed view and adds token-level self-distillation targets wherever retained targets are replayed under changed contexts. Specifically, the stop-gradient teacher πsg(θ)(⋅∣zp)\pi_{\mathrm{sg}(\theta)}(\cdot\mid z_p) guides the student πθ(⋅∣xp)\pi_\theta(\cdot\mid x_p) through a teacher-to-student KL loss. This gives the compressed view direct supervision from the policy's predictions under the original decision context.

How does this work with reward-driven learning? GRPO uses rollout rewards to improve the policy's reasoning and action selection. SDCC helps the compressed view reproduce that same policy's predictions under the generation-time context. The teacher is recomputed from the current policy during training, so its targets evolve as the policy learns.

The weight λ\lambda controls the strength of this consistency constraint: λ=0\lambda=0 recovers Naive-Compressed, while larger values place more weight on matching the teacher. We assess whether this balance is useful through the controlled task comparison below.


(2) Empirical support for the consistency constraint

On Qwen3-4B + AgentFold, SDCC (λ=0.1\lambda=0.1) improves test macro-EM over Naive-Compressed (λ=0\lambda=0) in both training seeds under the same end-to-end training-time budget. Mean macro-EM increases from 33.20% to 45.25%, with a paired gain of 12.05 ± 1.68 percentage points (mean ± sample SD across seeds).

We also varied λ\lambda to measure its effect on prediction differences between the two contexts. We kept the other training settings fixed, selected checkpoints by validation performance, and evaluated the same fixed diagnostic samples:

λ\lambdaTeacher–student KL ↓logdiff ↓
00.0038130.839725
0.030.0030610.681205
0.10.0025840.554331
0.30.0024540.546367

Compared with λ=0\lambda=0, setting λ=0.1\lambda=0.1 reduces teacher–student KL by 32.2% and logdiff by 34.0%. At this weight, SDCC achieves both closer agreement between contexts and higher task performance under the common time budget.


(3) Lower policy-update cost than exact branch replay

In our additional WideSearch experiment with Qwen3.6-35B-A3B, a batch contains 128 main rollouts and 153 sub-agent invocations. LogitTree backpropagates through approximately 1,935 training sequences. SDCC uses 281 student views—one per main rollout and per sub-agent invocation—and evaluates the teacher branches without backpropagation.

With hardware, trajectories, and update counts fixed, policy-update time falls from 2 h 28 min 39 s to 24 min 47 s, an approximately 6.0× speedup. The timing includes all teacher forward passes and student forward and backward passes.

AuthorsOpenReview ↗

Rebuttal for W2

W2: Slime already provides related replay functionality. What does this work contribute?

Thank you for pointing out this connection. Our black-box implementation builds on Slime's example stack, including its TITO-style loss hook for scoring response segments against reconstructed rollout-time prefixes. The white-box implementation is build from trajectory views not from Slime's TITO view. We reuse this scaffold and its interfaces, and will make this overlap more explicit. Our contributions build on these existing capabilities in three ways.

  • (1) Formalizing two distinct failures caused by context editing. Naive-Compressed replays earlier actions under the final compressed view, applying later deletions or summaries retroactively (time-travel leakage). Naive-Full replays later actions with information already removed before those actions were generated (stale-context leakage). We explain both through the same requirement: each training target should be scored under the context actually visible when it was generated. Preserving sampled token IDs alone does not ensure this condition.

  • (2) Organizing training around trajectory trees and forests. A trajectory tree represents the context branches created by memory editing within one agent; a trajectory forest additionally represents agents with separate contexts. This framework specifies each target's agent, conditioning branch, and loss assignment, so that each target contributes once under its corresponding generation-time context.

    In a white-box harness, these structures can be built during execution from memory-editing events and sub-agent calls. In a black-box harness, captured requests and responses can reconstruct the observable branches afterward. Both follow the same conditioning and loss-assignment rules. LogitTree provides the exact-replay reference within this framework.

  • (3) Reducing the cost of exact branch replay through SDCC. Exact replay requires backpropagation through the relevant branches. SDCC instead evaluates those branches as stop-gradient teacher views and transfers their predictions to compressed student views through a consistency loss. Backpropagation is concentrated on the student views, reducing training cost. Captured request contexts can also supply teacher views without internal span-level eviction logs.

We will clarify Slime's role as an implementation foundation and emphasize our contributions in the problem analysis, the tree/forest training framework, and SDCC's efficient approximation.

AuthorsOpenReview ↗

Rebuttal for W3

W3: Controlled comparison under the same wall-clock training budget

Thank you for suggesting this comparison. The main experiments compare methods at the same number of optimizer updates. We additionally compared Naive-Compressed (λ=0\lambda=0) and SDCC (λ=0.1\lambda=0.1) on Qwen3-4B + AgentFold under the same end-to-end training-time budget. For each seed, both methods used the same initial model, training data, and resource configuration. The training set contains 79,175 examples.

The common budget was 12,337.63 seconds, approximately 3 h 25 min 38 s. We set this budget to the shortest elapsed time required to fully export the step-400 checkpoint among the four seed-42 runs with different λ\lambda values. We fixed the same budget before starting seed 43. For every run, we evaluated the last checkpoint fully exported within the budget.

ComparisonNaive-Compressed (λ=0\lambda=0)SDCC (λ=0.1\lambda=0.1)
HardwareOne node, 8×A100One node, 8×A100
Training seeds42, 4342, 43
Selected checkpoint steps (seed 42 / 43)248 / 280400 / 296
Test macro-EM (%) (seed 42 / 43)32.76 / 33.6443.62 / 46.87
Mean ± sample SD33.20 ± 0.6245.25 ± 2.30
GPU-hours to selected checkpoint (seed 42 / 43)26.48 / 26.9427.42 / 27.15
Archived rollout generation tokens (seed 42 / 43)6,977,030 / 7,919,39210,405,952 / 7,817,267
Student forward-input tokens (seed 42 / 43)6,462,976 / 7,738,88010,565,120 / 6,535,168
Teacher forward-input tokens (seed 42 / 43)0 / 0434,176 / 830,464

SDCC achieves higher macro-EM in both seeds under the same time budget, including its teacher computation. The mean paired gain is 12.05 percentage points, with an across-seed sample SD of 1.68 percentage points.

Paired bootstrap resampling of test questions, conditional on the trained models for each seed, gives 95% intervals of [7.92, 13.87] and [10.75, 15.82] percentage points for seeds 42 and 43, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.