- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LightCode learns rules to shorten tool outputs and keeps older history as summaries with retrievable references. It aims to use fewer context tokens without hurting task success. The paper should explain why it calls this “distilling” as the method mainly compresses and organizes context.
Critical strengths
-
Turning compression experience into executable programs is a practical design: deployed rules need no extra LLM call for each observation, and replay checks constrain updates.
-
Token reductions transfer to an unseen coding benchmark across four backbones. Repeated evaluations and ablations provide useful evidence beyond a single successful example.
Critical weaknesses
-
The method appears largely incremental, combining handcrafted compression rules, LLM-guided revisions, and summary-based retrieval. Its added value remains unclear without comparisons against simple masking/truncation and RTK with the same history management. Beyond DSH Basic, comparisons with native compaction in practical harnesses such as Claude Code and Codex would also help, with model and tool differences clearly documented.
-
Information preservation is not directly validated. Retrieval accesses already compressed outputs, so it cannot recover initial compression errors. Designer-written checks cover recorded inputs only; average success rates do not show how often compression turns an otherwise successful task into a failure.
-
What exact context thresholds, retained-history budgets, and output-length limits were used for each method? Which DSH Basic version and pruning plugins were enabled? These implementation details are very important.
Questions for authors
See Weaknesses.
5 comments
Response to Reviewer 1p5V(Part2)
Response to Q2
1. We add the requested simple masking control and an evolved-guideline baseline. On EvoAgentBench-SE, we evaluate Observation Masking, the observation-age control from the Complexity Trap study (Lindenbauer et al., 2025). We also evaluate ACON with evolved guidelines: following its feedback-driven optimization, trajectories from training tasks drawn from train87 inform revisions to the observation- and history-compression instructions, and the selected guidelines are frozen before test56 (Kang et al., 2026).
| Backbone | Method | Pass rate (%) | Reported avg. total tokens/task† |
|---|---|---|---|
| DeepSeek V4 Flash | LightCode (Table 1) | 85.12 ± 1.03 | 877,739 |
| DeepSeek V4 Flash | ACON (evolved guidelines) | 78.57 ± 1.79 | 1,037,741 |
| DeepSeek V4 Flash | Observation Masking | 77.38 ± 2.73 | 986,422 |
| GLM-5.3 Flash | LightCode (Table 1) | 77.38 ± 3.72 | 687,002 |
| GLM-5.3 Flash | ACON (evolved guidelines) | 63.10 ± 2.73 | 872,955 |
| GLM-5.3 Flash | Observation Masking | 60.12 ± 1.03 | 2,835,691 |
Pass rates are means ± sample standard deviations over three runs of all 56 tasks. LightCode retains the submitted Table 1 values. †Baseline token figures retain the existing measured means; DeepSeek ACON covers 47 tasks. They are not newly aggregated three-run costs.
LightCode's mean pass rates exceed masking by 7.74 and 17.26 percentage points, and evolved-guideline ACON by 6.55 and 14.28 points, on DeepSeek and GLM respectively. Its reported token means are also lower within the measurement scope above.
2. The added baselines test two concrete alternatives. Observation Masking asks whether simply removing older observations is sufficient to obtain a comparable result. Evolved-guideline ACON tests an adaptive approach that learns instructions for model-based compression. LightCode's higher mean pass rate on both backbones shows that its observed advantage persists against both alternatives, with lower reported token means within the scope stated above. This broadens the empirical comparison beyond native compaction. The attribution to individual mechanisms comes from the component ablations in Section 4.4 and the shared-history comparison in Q1; the additional baseline table establishes their combined performance against these alternatives.
Response to Q3
1. Preservation must be checked at the transformation stage. Your concern identifies the point at which information can become irrecoverable: a detail removed from the initial representation cannot be restored by later retrieval. LightCode therefore checks candidate transformations against original observations and specified retention requirements during learning. Retrieval then maintains later access to the representation that was retained.
The distinction between the learning archive and the agent's retrieval store matters here. The learning archive retains original observations, allowing a candidate's compliance to be checked against the evidence that preceded compression. Acceptance is therefore not based solely on what remains visible in a compressed view. Section 3.1.2 and Appendix A.3 specify this separation.
2. The decisive mechanism evidence is that preservation changes the acceptance decision. In a recorded GLM test-output case, an initial candidate removes text that resembles progress formatting but is protected as application output. It is rejected. A revised candidate narrows the transformation's applicability, preserving the ambiguous output while still shortening the unambiguous example. Both candidates achieve the same estimated replay saving, yet only the repaired candidate passes the preservation check. This shows why the learned object includes the conditions for applying a transformation: compression gain alone does not determine acceptance.
The paper provides broader evidence that this condition is active: the DeepSeek analysis in Figure 5b records 9 preservation-related rejections among 405 candidate checks. Appendix A.3 also prioritizes restoring newly required content over reducing length, permitting a repair to retain more information when necessary. The selected case explains how this constraint operates; the aggregate check record shows that preservation affects actual learning decisions beyond that example.
A second selected case addresses a different concern: whether shorter representations necessarily remove source content. A file-read representation decreases from 295 to 190 estimated tokens, a 35.59% reduction, while all 20 source lines and their indentation are reconstructed exactly. In this case, the saving comes from changing the representation while retaining reconstructable source content. This is a direct content-preservation check, independent of whether the final task happens to pass.
Response to Reviewer 1p5V(Part3)
------followed by Q3-------
3. These observations support a precise empirical claim. The acceptance case shows that specified content requirements can override a proposal's compression benefit; the reconstruction case verifies what remains available in one compressed representation. At the task level, the complete comparison in Q1 shows further token savings with matching or higher pass rates across four backbones under shared history management. The mechanism evidence explains the checks behind the method, while the benchmark comparison evaluates its overall cost and performance.
This evidence supports checked preservation on recorded inputs and an observed efficiency–performance benefit. As stated in Appendix A.2, it does not guarantee every unseen case. The frequency of tasks changing from success to failure is a separate paired-outcome statistic; the selected cases and aggregate pass rates do not supply that statistic.
Response to Q4
1. RTK is part of the acknowledged starting point; the evaluated policy is LightCode's. Section 3.1.1 cites RTK as prior work on rule-based output compression. LightCode draws on this design and reuses some RTK built-in filter definitions alongside additional initial rules, with attribution under Apache-2.0. LightCode supplies the DSH execution adapter, execution-driven revisions to transformations and their applicability conditions, and preservation-constrained acceptance. We will make this relationship explicit in the manuscript.
The initial/evolved comparison in Q1 holds history consolidation fixed and measures the value of adapting this common starting point. It is a controlled test of evolution's contribution, not a benchmark of the complete standalone RTK system with the same history manager; that separate comparison has not been run.
2. DSH Basic controls the harness-level differences raised in the review. DSH Basic uses native compaction in the same DSH harness. Section 4.1 keeps the agent loop, tool set, and evaluation protocol shared, allowing context-management methods to be compared within a common execution setting. This is the purpose of the DSH Basic reference.
Claude Code and Codex would offer complementary comparisons between complete systems, with model, tool, and harness differences documented as you suggest. They address a broader systems-comparison question than the controlled context-management comparison here. Our evidence supports the evaluated DSH setting and frozen-policy transfer; it does not establish a ranking against those untested harnesses.
Response to Q5
The documented MAB protocols pin DSH 47f94385, while the inspected coding run manifests record d347e703. Native DSH defaults to compaction at 80% of the configured context window, with a separate 16% recent-history budget. In coding, DSH Basic retains native compaction and tool-result pruning; LightCode uses Economy and LightShadow in their place. Benchmark-specific overrides and acting-model/summary generation limits are listed in the configuration supplement, which distinguishes saved profiles from run-manifest evidence. Summary generation and recent-history retention are separate budgets; provider-counted reasoning is included in generation limits.
Taken together, the controlled transfer comparison, added coding baselines, and preservation evidence address the central question of whether LightCode adds value beyond combining existing components. They show reusable gains from program evolution under shared history management, competitive task performance against both masking and evolved-guideline compression, and preservation requirements that affect actual acceptance decisions. We will make these distinctions and the configuration disclosure clearer in the revision. We hope you will take this evidence and the clarifications above into account in your final assessment of the paper's contribution. Thank you again for your careful consideration.
References
-
Minki Kang et al. (2026). ACON: Optimizing Context Compression for Long-horizon LLM Agents. International Conference on Machine Learning (ICML).
-
Tobias Lindenbauer et al. (2025). The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management. Fourth Deep Learning for Code Workshop, NeurIPS.
Response to Reviewer 1p5V(Part1)
Thank you for the careful assessment and for recognizing the practical value of executable compression and frozen-policy transfer. Your questions concern what is learned beyond the initial components, how information preservation is checked, and whether the evaluation can be reproduced. We address these concerns in five thematic parts below, using the paper's controlled comparisons, additional coding baselines, and concrete preservation evidence.
Response to Q1
1. The contribution concerns what execution experience adds to the initial system. The attribution question is what learning adds beyond the initial filters and history manager. LightCode turns execution experience into a reusable representation policy by revising both how an observation is transformed and the conditions under which that transformation is appropriate. Proposed changes are checked separately against accumulated original observations and explicit preservation requirements before adoption. The resulting knowledge persists as a frozen executable policy that can be used on new tasks without a model call for each observation compression.
Equation 1 explains why this is coupled with history consolidation: cumulative context cost depends on both representation length and repeated inclusion in subsequent requests. Observation compression addresses the former; consolidation separates the task state needed now from details that remain accessible for later retrieval. The research contribution lies in learning and reusing a constrained representation policy within this context-management framework.
2. Shared history management tests the value of the learned policy. To distinguish the contribution of policy revision from the choice of history manager, Section 4.3 compares the initial and evolved programs using the same history-consolidation procedure on LongCLI-Bench, with no further learning. Table 2 reports:
Pass rates are the paper's two-run averages over all 20 tasks; token reductions are calculated relative to the initial policy. The shared history mechanism is important to the interpretation: further savings occur on every backbone after that mechanism is held fixed. The GLM result makes this particularly clear, retaining the 40.0% pass rate while reducing total tokens by 24.75%. This supports the added value of the revised observation policy within the common context-management framework.
Complementary GLM trajectories on EvoAgentBench-SE explain what is reused. A rule learned at task 54 recognizes a SymPy report format and removes formatting overhead while preserving failure locations, tracebacks, and the result summary. With learning disabled, that frozen rule is subsequently used ten times across four unseen test tasks; eight of the ten added GLM rules activate during held-out evaluation. The learned artifact therefore captures an output format that recurs beyond its originating task. The controlled comparison quantifies the value of revision, while the trajectories connect a concrete learned capability to later deployment; these two forms of evidence jointly support reusable compression knowledge.
3. What we mean by distilling. The term refers to converting execution experience into reusable compression-program knowledge. The learned artifact is an executable policy, with no model-parameter update. We will state this meaning explicitly at its first occurrence.