acceptodds
Under review as a conference paper at ICLR 2027

LightCode: Distilling Execution Context for Coding Agents

Abstract

Large language model (LLM) agents tackle complex software tasks through extended sequences of reasoning, action, and observation. However, redundancy in incoming observations and accumulated histories inflates cumulative token costs and can impair reasoning, while indiscriminate compression risks discarding information needed later. To address this challenge, we propose LightCode, a framework for context-efficient coding agents that controls how incoming observations are represented and how accumulated histories are retained over successive interaction steps. For incoming observations, self-evolving observation compression uses execution experience to identify recurring redundancy and refine reusable compression programs, extending their coverage across formats while constraining updates to preserve required information. By integrating long-horizon history consolidation, LightCode maintains a compact task state while retaining historical details separately for selective retrieval. Evaluations across three benchmarks and four LLM backbones demonstrate consistent gains in token efficiency, with LightCode reducing token usage by 14-50% while maintaining or improving task performance. Source code is available at: https://anonymous.4open.science/r/lightcode-research-4664

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based Coding Agents, Agent Self-Evolution, LLM Context Management

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1p5VOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LightCode learns rules to shorten tool outputs and keeps older history as summaries with retrievable references. It aims to use fewer context tokens without hurting task success. The paper should explain why it calls this “distilling” as the method mainly compresses and organizes context.

Critical strengths
  1. Turning compression experience into executable programs is a practical design: deployed rules need no extra LLM call for each observation, and replay checks constrain updates.

  2. Token reductions transfer to an unseen coding benchmark across four backbones. Repeated evaluations and ablations provide useful evidence beyond a single successful example.

Critical weaknesses
  1. The method appears largely incremental, combining handcrafted compression rules, LLM-guided revisions, and summary-based retrieval. Its added value remains unclear without comparisons against simple masking/truncation and RTK with the same history management. Beyond DSH Basic, comparisons with native compaction in practical harnesses such as Claude Code and Codex would also help, with model and tool differences clearly documented.

  2. Information preservation is not directly validated. Retrieval accesses already compressed outputs, so it cannot recover initial compression errors. Designer-written checks cover recorded inputs only; average success rates do not show how often compression turns an otherwise successful task into a failure.

  3. What exact context thresholds, retained-history budgets, and output-length limits were used for each method? Which DSH Basic version and pruning plugins were enabled? These implementation details are very important.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1p5V(Part1)

Thank you for the careful assessment and for recognizing the practical value of executable compression and frozen-policy transfer. Your questions concern what is learned beyond the initial components, how information preservation is checked, and whether the evaluation can be reproduced. We address these concerns in five thematic parts below, using the paper's controlled comparisons, additional coding baselines, and concrete preservation evidence.

Response to Q1

1. The contribution concerns what execution experience adds to the initial system. The attribution question is what learning adds beyond the initial filters and history manager. LightCode turns execution experience into a reusable representation policy by revising both how an observation is transformed and the conditions under which that transformation is appropriate. Proposed changes are checked separately against accumulated original observations and explicit preservation requirements before adoption. The resulting knowledge persists as a frozen executable policy that can be used on new tasks without a model call for each observation compression.

Equation 1 explains why this is coupled with history consolidation: cumulative context cost depends on both representation length and repeated inclusion in subsequent requests. Observation compression addresses the former; consolidation separates the task state needed now from details that remain accessible for later retrieval. The research contribution lies in learning and reusing a constrained representation policy within this context-management framework.

2. Shared history management tests the value of the learned policy. To distinguish the contribution of policy revision from the choice of history manager, Section 4.3 compares the initial and evolved programs using the same history-consolidation procedure on LongCLI-Bench, with no further learning. Table 2 reports:

BackboneInitial-policy pass rateEvolved-policy pass rateFurther total-token reduction
DeepSeek V4 Flash32.5%42.5%14.95%
GLM-5.3 Flash40.0%40.0%24.75%
MiniMax-M327.5%30.0%27.17%
Qwen3.8 Flash42.5%47.5%11.98%

Pass rates are the paper's two-run averages over all 20 tasks; token reductions are calculated relative to the initial policy. The shared history mechanism is important to the interpretation: further savings occur on every backbone after that mechanism is held fixed. The GLM result makes this particularly clear, retaining the 40.0% pass rate while reducing total tokens by 24.75%. This supports the added value of the revised observation policy within the common context-management framework.

Complementary GLM trajectories on EvoAgentBench-SE explain what is reused. A rule learned at task 54 recognizes a SymPy report format and removes formatting overhead while preserving failure locations, tracebacks, and the result summary. With learning disabled, that frozen rule is subsequently used ten times across four unseen test tasks; eight of the ten added GLM rules activate during held-out evaluation. The learned artifact therefore captures an output format that recurs beyond its originating task. The controlled comparison quantifies the value of revision, while the trajectories connect a concrete learned capability to later deployment; these two forms of evidence jointly support reusable compression knowledge.

3. What we mean by distilling. The term refers to converting execution experience into reusable compression-program knowledge. The learned artifact is an executable policy, with no model-parameter update. We will state this meaning explicitly at its first occurrence.

AuthorsOpenReview ↗

Response to Reviewer 1p5V(Part2)

Response to Q2

1. We add the requested simple masking control and an evolved-guideline baseline. On EvoAgentBench-SE, we evaluate Observation Masking, the observation-age control from the Complexity Trap study (Lindenbauer et al., 2025). We also evaluate ACON with evolved guidelines: following its feedback-driven optimization, trajectories from training tasks drawn from train87 inform revisions to the observation- and history-compression instructions, and the selected guidelines are frozen before test56 (Kang et al., 2026).

BackboneMethodPass rate (%)Reported avg. total tokens/task†
DeepSeek V4 FlashLightCode (Table 1)85.12 ± 1.03877,739
DeepSeek V4 FlashACON (evolved guidelines)78.57 ± 1.791,037,741
DeepSeek V4 FlashObservation Masking77.38 ± 2.73986,422
GLM-5.3 FlashLightCode (Table 1)77.38 ± 3.72687,002
GLM-5.3 FlashACON (evolved guidelines)63.10 ± 2.73872,955
GLM-5.3 FlashObservation Masking60.12 ± 1.032,835,691

Pass rates are means ± sample standard deviations over three runs of all 56 tasks. LightCode retains the submitted Table 1 values. †Baseline token figures retain the existing measured means; DeepSeek ACON covers 47 tasks. They are not newly aggregated three-run costs.

LightCode's mean pass rates exceed masking by 7.74 and 17.26 percentage points, and evolved-guideline ACON by 6.55 and 14.28 points, on DeepSeek and GLM respectively. Its reported token means are also lower within the measurement scope above.

2. The added baselines test two concrete alternatives. Observation Masking asks whether simply removing older observations is sufficient to obtain a comparable result. Evolved-guideline ACON tests an adaptive approach that learns instructions for model-based compression. LightCode's higher mean pass rate on both backbones shows that its observed advantage persists against both alternatives, with lower reported token means within the scope stated above. This broadens the empirical comparison beyond native compaction. The attribution to individual mechanisms comes from the component ablations in Section 4.4 and the shared-history comparison in Q1; the additional baseline table establishes their combined performance against these alternatives.

Response to Q3

1. Preservation must be checked at the transformation stage. Your concern identifies the point at which information can become irrecoverable: a detail removed from the initial representation cannot be restored by later retrieval. LightCode therefore checks candidate transformations against original observations and specified retention requirements during learning. Retrieval then maintains later access to the representation that was retained.

The distinction between the learning archive and the agent's retrieval store matters here. The learning archive retains original observations, allowing a candidate's compliance to be checked against the evidence that preceded compression. Acceptance is therefore not based solely on what remains visible in a compressed view. Section 3.1.2 and Appendix A.3 specify this separation.

2. The decisive mechanism evidence is that preservation changes the acceptance decision. In a recorded GLM test-output case, an initial candidate removes text that resembles progress formatting but is protected as application output. It is rejected. A revised candidate narrows the transformation's applicability, preserving the ambiguous output while still shortening the unambiguous example. Both candidates achieve the same estimated replay saving, yet only the repaired candidate passes the preservation check. This shows why the learned object includes the conditions for applying a transformation: compression gain alone does not determine acceptance.

The paper provides broader evidence that this condition is active: the DeepSeek analysis in Figure 5b records 9 preservation-related rejections among 405 candidate checks. Appendix A.3 also prioritizes restoring newly required content over reducing length, permitting a repair to retain more information when necessary. The selected case explains how this constraint operates; the aggregate check record shows that preservation affects actual learning decisions beyond that example.

A second selected case addresses a different concern: whether shorter representations necessarily remove source content. A file-read representation decreases from 295 to 190 estimated tokens, a 35.59% reduction, while all 20 source lines and their indentation are reconstructed exactly. In this case, the saving comes from changing the representation while retaining reconstructable source content. This is a direct content-preservation check, independent of whether the final task happens to pass.

AuthorsOpenReview ↗

Response to Reviewer 1p5V(Part3)

------followed by Q3-------

3. These observations support a precise empirical claim. The acceptance case shows that specified content requirements can override a proposal's compression benefit; the reconstruction case verifies what remains available in one compressed representation. At the task level, the complete comparison in Q1 shows further token savings with matching or higher pass rates across four backbones under shared history management. The mechanism evidence explains the checks behind the method, while the benchmark comparison evaluates its overall cost and performance.

This evidence supports checked preservation on recorded inputs and an observed efficiency–performance benefit. As stated in Appendix A.2, it does not guarantee every unseen case. The frequency of tasks changing from success to failure is a separate paired-outcome statistic; the selected cases and aggregate pass rates do not supply that statistic.

Response to Q4

1. RTK is part of the acknowledged starting point; the evaluated policy is LightCode's. Section 3.1.1 cites RTK as prior work on rule-based output compression. LightCode draws on this design and reuses some RTK built-in filter definitions alongside additional initial rules, with attribution under Apache-2.0. LightCode supplies the DSH execution adapter, execution-driven revisions to transformations and their applicability conditions, and preservation-constrained acceptance. We will make this relationship explicit in the manuscript.

The initial/evolved comparison in Q1 holds history consolidation fixed and measures the value of adapting this common starting point. It is a controlled test of evolution's contribution, not a benchmark of the complete standalone RTK system with the same history manager; that separate comparison has not been run.

2. DSH Basic controls the harness-level differences raised in the review. DSH Basic uses native compaction in the same DSH harness. Section 4.1 keeps the agent loop, tool set, and evaluation protocol shared, allowing context-management methods to be compared within a common execution setting. This is the purpose of the DSH Basic reference.

Claude Code and Codex would offer complementary comparisons between complete systems, with model, tool, and harness differences documented as you suggest. They address a broader systems-comparison question than the controlled context-management comparison here. Our evidence supports the evaluated DSH setting and frozen-policy transfer; it does not establish a ranking against those untested harnesses.

Response to Q5

The documented MAB protocols pin DSH 47f94385, while the inspected coding run manifests record d347e703. Native DSH defaults to compaction at 80% of the configured context window, with a separate 16% recent-history budget. In coding, DSH Basic retains native compaction and tool-result pruning; LightCode uses Economy and LightShadow in their place. Benchmark-specific overrides and acting-model/summary generation limits are listed in the configuration supplement, which distinguishes saved profiles from run-manifest evidence. Summary generation and recent-history retention are separate budgets; provider-counted reasoning is included in generation limits.


Taken together, the controlled transfer comparison, added coding baselines, and preservation evidence address the central question of whether LightCode adds value beyond combining existing components. They show reusable gains from program evolution under shared history management, competitive task performance against both masking and evolved-guideline compression, and preservation requirements that affect actual acceptance decisions. We will make these distinctions and the configuration disclosure clearer in the revision. We hope you will take this evidence and the clarifications above into account in your final assessment of the paper's contribution. Thank you again for your careful consideration.

References

  1. Minki Kang et al. (2026). ACON: Optimizing Context Compression for Long-horizon LLM Agents. International Conference on Machine Learning (ICML).

  2. Tobias Lindenbauer et al. (2025). The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management. Fourth Deep Learning for Code Workshop, NeurIPS.

Reviewer_N1X6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into the problem of condensing context for coding agents. It presents LightCode, a context management framework that controls how observations are represented and how long past information is retained. It introduces self-evolving observation compression, which uses execution experience to iteratively revise a reusable compression program; a revision is accepted only if it shortens archived observations without violating preservation requirements. Further, history consolidation decouples task state from historical details, which are stored behind references for selective retrieval. Experiments show strength in saving tokens while maintaining or improving the pass rate in software coding tasks and memory tasks.

Critical strengths
  1. The compression policy is a deterministic program with no inference cost at deployment, and every revision must pass replay on all archived observations, making the evolution well-motivated and auditable.

  2. Token savings are consistent across four backbones on EvoAgentBench-SE and transfer to an unseen benchmark without re-evolution. The ablation shows that both components contribute to the savings.

Critical weaknesses
  1. Most of the method is described only at a high level, so it is not clear how key components actually work. The paper offers only schematic examples without a real compression rule, an accepted revision, a preservation requirement, or a consolidated task state. This makes some claims hard to assess.

  2. The token savings on the coding benchmarks are only compared with DSH Basic, Periodic Summary, and LLMLingua-2. ACON and Complexity Trap, the most related methods, are evaluated only on MemoryAgentBench. It therefore remains unclear whether LightCode saves more than existing context managers on coding tasks.

Questions for authors
  1. Can you provide some case-specific analysis on the evolution of the compression program? Fig. 5c shows which rules save the most tokens, but it is not clear what an accepted revision actually changes. For example, what does read-layout remove from file reads, and could you show one or two revisions with the rule and the observation before and after?

  2. The reported token counts cover only test-time execution (Appendix A.4). What is the cost of evolution itself (running the 87 learning tasks and the evolution agent, separately for each backbone)

  3. In Table 1, why does LightCode increase output tokens on MiniMax-M3 (16.9k -> 23.1k)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer N1X6 (Part1)

Thank you for the careful review and for recognizing the value of executable compression policies and their transfer across benchmarks. We address your questions below with concrete revision examples, rule-reuse statistics, supplementary learning-cost measurements, and additional coding comparisons.

Response to Q1

1. What the rule changes. The file-read example illustrates structural compression: repeated file metadata and line labels are replaced by a shared description, while the source content and an explicit representation of its indentation remain. All 20 original source lines were exactly reconstructed from this example. Its estimated length falls from 295 to 190 tokens. The complete before/after observation and reconstruction are included in the supplement.

2. How a revision is accepted. A proposed shortening must also satisfy the preservation requirements. A candidate learns to remove test-progress formatting, but a preservation example contains similar-looking text that could instead be output from the application being tested. The candidate is rejected because it removes that protected content. The repaired revision keeps ambiguous cases intact while retaining the compression benefit on the original unambiguous observation. This illustrates learning both a transformation and its conditions of use. The preserved candidate snapshots substantiate both decisions.

3. Whether the learned rules are reused. Section 4.6 reports that 21 of the 22 rules added by DeepSeek activate on unseen test tasks, and these additions account for 82% of the frozen policy's observation-compression token savings.

An additional GLM trace analysis provides evidence on another backbone. All 56 test tasks use the same final frozen policy with the Learner disabled. Eight of the ten added rules are applied during testing, with 586 recorded observation-compression applications. Representative rules are:

Added ruleTest-time applicationsUnseen test tasks
read-content: file-read output29556
unittest: unit-test output5513
sympy-test: SymPy test reports104

The SymPy rule is acquired at learning task 54. It removes report-formatting overhead while retaining failed-test locations, tracebacks and the result summary, and is subsequently reused on four unseen tasks. This connects a concrete later-stage transformation to deployment beyond its learning task. The reuse counts measure actual rule application; the paper's initial-versus- evolved comparisons separately assess end-to-end effects. Together, they support the acquisition of transferable output-structure handling from execution experience.

4. What consolidated history contains. The recorded state includes the task goal, established findings, next actions, exact user requirements, and references to earlier observations. We verified that the next model request uses this consolidated state together with recent interactions. The examples thus document how observations are transformed, how revisions are accepted, and how the resulting representations are used during continued execution.

Response to Q2

1. Self-evolution usage. The supplementary measurements below cover the selected Learner traces associated with 87 training tasks per model. Input counts uncached input tokens; output counts recorded model generation.

Learner TokenGLM 5.3 FlashMiniMax M3
Input31,506,88422,479,788
Output3,601,3296,289,277
Total35,108,21328,769,065

These recorded subtotals exclude cache reads and task execution; calls without reported usage remain uncounted. MiniMax aggregates learning branches covering 57 and 30 tasks.

2. Reuse of the learning investment. Once frozen, the executable policy serves subsequent tasks without rerunning the Learner or making an additional model call for each observation. History-consolidation calls remain included in deployment usage. Section 4.3 tests this reuse by transferring policies from EvoAgentBench-SE to LongCLI-Bench without further evolution.

3. Lifecycle comparison. With consistently accounted setup and deployment costs, consumption over N downstream tasks is C_setup + N * c_deploy. The average setup contribution is C_setup / N, so sustained deployment savings can outweigh the learning investment as the policy is reused. Evaluation-time efficiency establishes per-task savings; learning expenditure and the subsequent workload determine the lifecycle benefit.

AuthorsOpenReview ↗

Response to Reviewer N1X6 (Part2)

Response to Q3

1. Additional comparators. We add ACON with evolved compression guidelines and Observation Masking on EvoAgentBench-SE with DeepSeek V4 Flash and GLM-5.3 Flash. Following ACON's feedback-driven optimization, execution trajectories from train87 tasks guide revisions to observation- and history-compression instructions. The selected guidelines are frozen before test56 evaluation. Observation Masking is the evaluated Complexity Trap-style control: older observations are masked while recent interactions are retained.

2. Results. LightCode retains the submitted Table 1 values.

BackboneMethodPass rate (%)Reported avg. total tokens/task†
DeepSeek V4 FlashLightCode (Table 1)85.12 ± 1.03877,739
DeepSeek V4 FlashACON (evolved guidelines)78.57 ± 1.791,037,741
DeepSeek V4 FlashObservation Masking77.38 ± 2.73986,422
GLM-5.3 FlashLightCode (Table 1)77.38 ± 3.72687,002
GLM-5.3 FlashACON (evolved guidelines)63.10 ± 2.73872,955
GLM-5.3 FlashObservation Masking60.12 ± 1.032,835,691

Pass rates are means ± sample standard deviations over three runs of all 56 tasks. †Baseline token figures use the existing measured means; DeepSeek ACON covers 47 tasks. These token figures have not been re-aggregated across the additional repeats.

LightCode's mean pass-rate advantages over ACON are 6.55 and 14.28 percentage points, and over Observation Masking 7.74 and 17.26 points, for DeepSeek and GLM, respectively. Its reported token means are also lower within the measurement scope above.

3. Relation to the mechanism. Observation compression shortens information entering context, history consolidation bounds its repeated residence, and evolution retains execution experience as reusable transformations. The added comparisons cover learned model-based compression and observation masking; the component ablations separately assess the contributions of LightCode's mechanisms.

Response to Q4

1. Net token consumption. LightCode reduces the context repeatedly supplied to the agent; it does not constrain the agent's generated-token count to decrease. Table 1 includes this additional output in the total, and the input reduction more than offsets it:

MiniMax-M3, mean tokens per taskInputOutputTotal
DSH Basic1,691,44716,8991,708,346
LightCode1,300,06223,0551,323,117

Specifically, the additional 6,156 output tokens accompany a reduction of 391,385 input tokens per task, yielding 22.55% fewer total tokens. The efficiency result therefore already accounts for the increased generation.

2. Observed generation behavior. We additionally inspected a separate 56-task MiniMax frozen-policy run. Two completed model calls alone generated 108,134 output tokens, or 8.38% of that run's output. Both contained extensive repetition in their reasoning and were followed by successful tool execution. The separate trace audit identifies repetitive reasoning as a concrete source of high output during otherwise valid interactions. Table 1 independently establishes the aggregate result: total consumption falls after accounting for both input and output tokens.


These additions connect concrete program revisions and preservation checks to actual reuse on unseen tasks, and extend the coding evaluation to ACON with evolved guidelines and Observation Masking. Together with the component ablations and frozen-policy transfer results, they further support our central claim: LightCode turns execution experience into reusable compression knowledge, reducing deployment token consumption while maintaining competitive performance on the evaluated unseen tasks.

We will incorporate these examples, additional results, and learning-cost clarifications into the revised manuscript to make the mechanisms and their scope clearer. We hope you will take this evidence and the clarifications above into account in your final assessment of the paper's contribution. Thank you again for your careful consideration.

References

  1. Minki Kang et al. (2026). ACON: Optimizing Context Compression for Long-horizon LLM Agents. International Conference on Machine Learning (ICML).

  2. Tobias Lindenbauer et al. (2025). The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management. Fourth Deep Learning for Code Workshop, NeurIPS.

Dear Reviewer N1X6,

We sincerely appreciate the time and care you have devoted to reviewing our paper. As the discussion deadline is approaching, we would be very grateful if you could take another look at our rebuttal. If these additional results have addressed your concerns , we would sincerely appreciate it if you could reconsider and update your assessment.

If any concern remains, please let us know—we would be more than happy to respond.

Reviewer_xCdPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The contexts of LLM coding agents often include redundant tool outputs and stale history at every step, so cumulative token cost grows with both observation length and how long each observation stays in context. The authors model the cost of an agent's context as a function of the length of each observation and the duration for which it resides in the context. The authors propose a framework that minimizes this cost, namely LightCode\textit{LightCode}, featuring two main mechanisms: 1) observation compression and 2) history consolidation. Observation compression shortens each incoming tool output with an executable program that is evolved across learning tasks. History consolidation moves older observations to external storage behind references once the context exceeds a budget, summarizes the remaining history into a compact task state, and lets the agent retrieve stored details on demand. The compression program is evolved on 87 EvoAgentBench-SE training tasks and then frozen, and LightCode\textit{LightCode} is evaluated with four backbones on three benchmarks. Compared with the harness's native compaction, it reduces token usage by 14–38% on EvoAgentBench-SE and by 25–50% on LongCLI-Bench while maintaining or improving pass rates, and it achieves the highest accuracy at the lowest token cost on MemoryAgentBench. Ablations indicate that both mechanisms contribute to the savings and that evolving the program yields further reductions.

Critical strengths
  1. Past observations are kept retrievable behind references instead of being discarded. History consolidation (Ψ\Psi) externalizes observations before summarizing them (Sec. 3.2, Fig. 7), so details that an irreversible summary would lose remain accessible. The paper's own Periodic Summary baseline, which lacks this, lowers the pass rate on all four backbones, by up to 15.5 points (Table 1). On Fact-SH32K from MemoryAgentBench (Hu et al., 2026), LightCode reaches 98.0% with DeepSeek V4 Flash against 67.0% for the best baseline on that task, Complexity Trap (Lindenbauer et al., 2025), and leads on three of four backbones. This is consistent with reducing tokens without losing accuracy.

  2. The evolved compression policy runs as a deterministic program with no LLM calls. At deployment the policy adds no model tokens and negligible latency (Sec. 3.1.1, App. A.4), whereas ACON (Kang et al., 2026) calls an LLM compressor at every compression event and SWE-Pruner (Wang et al., 2026, arXiv 2601.16746) and CoACT (Chen et al., 2026, arXiv 2607.02911) need a separately trained compressor. Determinism also makes the replay-based acceptance check exact and cheap, since candidate and current policies are compared on the same recorded inputs Dk\mathcal{D}_k (Sec. 3.1.2, App. A.3).

Critical weaknesses
  1. The accuracy claims rest on evidence too thin to support them. EvoAgentBench-SE has 56 test tasks, so one task is 1.8 pass-rate points. The gains over DSH Basic in Table 1 (1.79, 0.60, 4.76, 0) are one, a third, under three, and zero tasks, and none exceeds two of the reported standard deviations. LongCLI-Bench has 40 trajectories, so the 7.5-point gain quoted in the abstract is three of them, and Table 2 reports no variance. The strongest baseline is also uncertain. App. A.4 calls ACON a fixed-guideline compressor, while Sec. 5.2 describes ACON as refining its guidelines from execution feedback (Kang et al., 2026). If its optimization was never run, LightCode's policy was given 87 learning tasks and the baseline none, and the guideline text, compressor model, and thresholds are not stated. A paired per-task comparison on the existing runs, variance for Table 2, and the ACON configuration would settle both points within a rebuttal.

  2. The benefit of self-evolution over a static program is not established. Evolution adds real token savings (a further 12% and 9% on EvoAgentBench-SE, 12 to 27% on LongCLI-Bench, Sec. 4.3 and 4.4), but its accuracy gains of 2.38 and 1.79 points sit against standard deviations of 1.03 and 3.72 (Fig. 3, Table 1). On LongCLI-Bench, π0\pi_0 falls 10 points below DSH Basic with DeepSeek V4 Flash before πk\pi_k recovers parity, and on GLM-5.3 Flash evolution changes nothing, so part of the gain is repair of the initial rules, as Sec. 4.4 acknowledges. RTK (Szymkowiak, 2026), cited in Sec. 3.1.1 as the motivation for the rule-program representation, is never run as a static baseline, and the one-time cost of evolution (an LLM agent over 87 tasks, 405 acceptance checks for DeepSeek V4 Flash, Sec. 4.6) is outside the token accounting of App. A.4. Without a strong static comparator, the central claim rests on a comparison with the authors' own untuned starting point.

Questions for authors
  1. For the ACON baseline, which guideline text, compressor model, and length thresholds were used, and was ACON's guideline optimization run on any of the three benchmarks?
  2. Please report Fπ0+ΨF_{\pi_0}+\Psi in Table 1 for all four backbones so that the saving from the hand-written rules, from consolidation, and from evolution can be read separately. Fig. 3 gives this for two backbones only.
  3. If intermediate policies were saved, please evaluate πk\pi_k for k = 0, 22, 44, 66, and 87 on the EvoAgentBench-SE test set with one backbone, reporting pass rate and tokens. This would show whether the policy keeps improving or is repaired once early on, and would clarify the t0 label on the 98.3K bar in Fig. 5c.
  4. How does π0\pi_0 relate to RTK (Szymkowiak, 2026)? Were its 81 rules derived from RTK, and can RTK be run in place of π0\pi_0 with the same consolidation on one backbone?
  5. How many preservation requirements were in P0P_0, how many were added during learning, at which tasks, and what failures triggered them? Were any added after observing behavior on test tasks?
  6. Would you define key terms before they are used? Compression program, history consolidation, and preservation constraint are used throughout the abstract and Sec. 1, but the program is defined only in Sec. 3.1.1, consolidation in Sec. 3.2, and the concrete form of a preservation requirement (a recorded observation paired with the content that must survive) only in App. A.2 and Fig. 6. Ψ\Psi is used in Sec. 4.4 before its definition in App. A.1. A reader reaches page 3 and the appendix before learning what the two mechanisms do.
7 comments
AuthorsOpenReview ↗

Response to Reviewer xCdP(Part1)

Thank you for the careful review and for recognizing the value of retrievable history and executable compression policies. We understand the central concern to be the contribution of evolution beyond the initial rules and history manager. Our response brings together three complementary forms of evidence: controlled initial/evolved comparisons, later-learned rules reused on unseen tasks, and additional comparisons with learned compression guidelines. These support an efficiency benefit that remains meaningful even when the observed pass rate is unchanged. We address your six numbered questions first, followed by the statistical interpretation and evolution cost raised in the weaknesses.

Response to Q1

1. ACON with evolved guidelines. We understand your concern to be whether LightCode's advantage persists when the comparator also improves its compression instructions from execution feedback. To examine this directly, we add ACON with evolved compression guidelines in the supplementary EvoAgentBench-SE coding experiments. Following ACON's feedback-driven optimization, trajectories from training tasks drawn from train87 guide revisions to observation- and history-compression instructions. The selected guidelines are frozen before test56 and used by ACON's LLM compressor during evaluation.

2. Comparison with LightCode. We report results with DeepSeek V4 Flash and GLM-5.3 Flash, alongside Observation Masking, the evaluated Complexity Trap-style control.

BackboneMethodPass rate (%)Reported avg. total tokens/task†
DeepSeek V4 FlashLightCode (Table 1)85.12 ± 1.03877,739
DeepSeek V4 FlashACON (evolved guidelines)78.57 ± 1.791,037,741
DeepSeek V4 FlashObservation Masking77.38 ± 2.73986,422
GLM-5.3 FlashLightCode (Table 1)77.38 ± 3.72687,002
GLM-5.3 FlashACON (evolved guidelines)63.10 ± 2.73872,955
GLM-5.3 FlashObservation Masking60.12 ± 1.032,835,691

Pass rates are means ± sample standard deviations over three runs of all 56 tasks. LightCode retains the submitted Table 1 values. †Baseline token figures retain the existing measured means; DeepSeek ACON covers 47 tasks. They are not newly aggregated three-run costs.

LightCode's mean pass-rate advantages over evolved-guideline ACON are 6.55 and 14.28 percentage points; the advantages over masking are 7.74 and 17.26 points. Its reported token means are also lower within the stated measurement scope. The higher mean pass rates thus persist against a comparator that has learned compression guidelines before evaluation, as well as against simple observation masking. Together with the shared-history comparisons and frozen transfer discussed below, this supports the value of turning execution experience into reusable compression-program knowledge.

Response to Q2

1. What the controlled comparison isolates. The relevant control for the contribution of evolution is the initial program with the same history consolidation. With Ψ unchanged, comparing π₀ with πₖ measures the increment associated with revising the observation program. The resulting savings therefore cannot be attributed simply to adding history consolidation.

2. Evidence across the four backbones. Figure 3 reports this comparison on EvoAgentBench-SE for DeepSeek and MiniMax, with a further 12.0% and 9.1% token reduction. Table 2 provides the four-backbone comparison on LongCLI-Bench, using the frozen policies without further evolution:

BackboneInitial program + consolidation: pass rateEvolved program + consolidation: pass rateFurther total-token reduction
DeepSeek V4 Flash32.5%42.5%14.95%
GLM-5.3 Flash40.0%40.0%24.75%
MiniMax-M327.5%30.0%27.17%
Qwen3.8 Flash42.5%47.5%11.98%

The reductions are calculated from Table 2 totals. Evolution reduces deployment tokens on every backbone while matching or improving the reported pass rate. GLM is particularly informative: its initial policy already achieves 40.0%, compared with DSH Basic's 32.5%; evolution maintains 40.0% while reducing tokens by a further 24.75%. Thus, even where the initial policy already performs well, evolution improves the observed cost–performance trade-off. Its value does not require a further increase in pass rate.

3. How this relates to the requested ablation. The four-model initial-rule rows requested for Table 1 remain unreported. The existing evidence nevertheless addresses the underlying attribution question through two complementary tests: the two-backbone ablation on EvoAgentBench-SE and the four-backbone frozen transfer on LongCLI-Bench. Both retain the same history manager within the initial/evolved comparison. We will present their scopes together so that the incremental contribution of evolution can be read separately from the benefit of consolidation.

AuthorsOpenReview ↗

Response to Reviewer xCdP(Part2)

Response to Q3

1. Whether evolution is only an early repair. The important distinction is whether learned changes remain useful beyond the experience that produced them. Repair is one legitimate component of evolution, alongside extending coverage to new formats. To examine your concern, we connect the timing of accepted changes to their subsequent use after freezing. The GLM trajectories show accepted revisions and new-rule additions throughout the sequence:

Learning tasksAccepted revisionsRevisions adding a new rule
1–22204
23–44121
45–66174
67–87181

2. Whether later changes matter after freezing. A SymPy test-report rule acquired at task 54 removes report-formatting overhead while retaining failure locations, tracebacks, and the result summary. It is subsequently applied ten times across four unseen test tasks. Overall, eight of ten added GLM rules are used during test56 with the Learner disabled. Section 4.6 provides complementary DeepSeek evidence: 21 of 22 new rules activate on unseen tasks and account for 82% of observation-compression savings.

The evidence therefore connects later acquisition to actual held-out use, beyond simply counting revisions. Together with the endpoint comparisons in Q2, it supports continued acquisition of transferable compression behavior. The requested intermediate-checkpoint pass rates and token totals have not been measured here, so these traces do not quantify each checkpoint's end-to-end contribution. They do establish that useful rule acquisition and reuse extend beyond an initial repair phase.

3. Figure 5c notation. The legend's t identifies the learning-task origin of a rule, while π₀ denotes the initial policy. The 98.3K bar reports a rule's attributed test-observation saving. We will make this distinction explicit in the caption so that a rule's origin is not read as the performance of the initial policy.

Response to Q4

1. Relationship to RTK. Section 3.1.1 cites RTK as prior work on rule-based output compression. LightCode draws on this design and reuses some RTK built-in filter definitions as part of the initial policy, alongside additional rules. The reused definitions are attributed to RTK under Apache-2.0.

LightCode implements its own DSH execution adapter, execution-driven revisions to rules and their applicability conditions, and preservation-constrained acceptance. The experiments execute LightCode's compression policy rather than the complete standalone RTK program. This distinguishes the existing filter knowledge used for initialization from LightCode's mechanisms for adapting that knowledge through execution experience. We will make this relationship explicit alongside the existing citation.

2. What the comparison establishes. Starting from that same program, the π₀/πₖ comparison asks whether execution-driven revision adds value while preserving the history manager. The reductions in Q2 show that it does in the evaluated settings. Running the full standalone RTK implementation with the same consolidation would additionally compare complete systems; that experiment is not included here. We therefore distinguish the supported benefit of evolving an existing program from a broader ranking of static implementations.

AuthorsOpenReview ↗

Response to Reviewer xCdP(Part3)

Response to Q5

1. Who supplies the requirements. Section 3.1.2 specifies that the system designers supply preservation requirements, which remain fixed within each candidate update and may be extended between learning tasks. Each requirement pairs an observation with content that must survive. The current and candidate programs face the same independently specified requirements, and the candidate cannot remove one to pass. This separates proposing a shorter representation from determining whether it is acceptable.

2. Available counts and concrete effect. The inspected GLM and MiniMax implementation snapshots each contain 16 preservation cases, covering source literals, application output, test failures, and output provenance. A recorded proposal that strips progress-like text is rejected because it also removes protected application output; the accepted revision narrows its applicability to preserve that content. This demonstrates a requirement changing the acceptance decision.

These are snapshot case counts. Figure 5b's nine preservation failures instead count rejected candidate checks. The available snapshots do not reconstruct the initial count and subsequent additions to Pₖ for every paper run.

3. Training and evaluation separation. The specified protocol performs learning on train87 and freezes the policy before held-out evaluation. The inspected acceptance objective uses training-observation replay and preservation checks, rather than test56 scores. This establishes how candidate selection is defined; establishing the complete history of manually added requirements would require the corresponding provenance records. We distinguish this protocol evidence from a complete historical audit.

Response to Q6

1. Definitions. We will introduce the terms together before the method overview:

  • A compression program maps a tool observation and its producing action to a shorter representation.
  • A preservation requirement specifies content that must survive on a recorded observation.
  • History consolidation, denoted Ψ, replaces older active history with a compact task state and references to stored observations.

2. Consistent notation. We will introduce Ψ at its first use and distinguish the initial policy π₀ from the rule-origin labels in Figure 5c. This makes the roles of observation transformation, constrained acceptance, and history management explicit before the experimental comparisons.

Response to Q7

1. Interpretation of the accuracy results. The efficiency conclusion does not depend on treating every small accuracy difference as statistically significant. Our central empirical result is a better observed cost–performance trade-off: lower deployment token consumption with competitive task performance. The four-backbone comparison in Q2 remains informative under this conservative interpretation of accuracy, including the GLM case where pass rate is unchanged. The additional three-run ACON and masking comparisons also show higher mean pass rates on both DeepSeek and GLM; we report these as observed differences rather than paired significance results.

2. Variance and the evaluation unit. EvoAgentBench-SE contains 56 test tasks. LongCLI contains 20 tasks evaluated twice; repeated trajectories from the same task are not independent task samples. The reported standard deviations describe run-to-run variation. The available aggregates do not recover paired success-to-failure transitions or the missing Table 2 variance, so we do not infer those statistics from the means. We will distinguish observed mean differences from significance claims in the revised presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.