acceptodds
Under review as a conference paper at ICLR 2027

Focused but Fragile: Diagnosing Tail-Rule Loss under Recursive Agent Skill Compression

Abstract

Compression can preserve an agent skill's routine while omitting the rule needed for an exception. We study this failure mode in an artifact-only maintenance regime, where each rewrite sees only its predecessor and cannot revisit the source evidence used to create the skill. We introduce TailSkills, a controlled benchmark of 208 task variants derived from 54 base tasks and 14 exception types. Each variant pairs an ordinary environment with an exception-triggering environment and is admitted only after reference-oracle execution and deterministic verification. The benchmark reveals a selective durability gap. Among the 56 variants with detectable registered markers, unit-weighted S4 retention is 48.7% for regular workflow units but only 15.7% for tail recovery units. A separate high-precision analysis over all 208 variants reproduces the artifact-level separation across four pipelines: at S4, regular retention ranges from 31.2% to 43.9%, whereas tail retention ranges from 2.1% to 15.2%. On a fixed set of 52 tail variants, S4 success is lower than S1 in three of four tested pipelines, with endpoint drops of 3.8–7.7 percentage points; the remaining pipeline follows a non-monotone trajectory and ends at its S1 level. Paired execution audits further show that artifact loss can become task failure in some executor–task settings, but does not produce a uniform downstream drop. These findings indicate that ordinary-task success is insufficient to assess the durability of compressed procedural skills: recovery rules require direct evaluation.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent skills, procedural memory, recursive compression, skill maintenance, robustness evaluation, exception handling, agent benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2hjQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the potential loss of long-tail anomaly recovery rules that may occur when Large Language Model (LLM) agents recursively compress external skills. The authors propose the TAILSKILLS Benchmark, which consists of 208 task variants built upon 54 base tasks and 14 categories of predefined anomalies. Through controlled environmental perturbations, reference solution execution, and deterministic verification, the study evaluates the retention of skill knowledge and actual task performance before and after compression. The experiments employed multi-round recursive compression and combined rule-labeling detection, semantic evaluation, cross-model validation, and paired execution tests to analyze the differences in the retention of anomaly recovery rules between conventional workflows and those incorporating such rules. The results show that, among 56 detectable variants, the retention rate of the conventional workflow after three rounds of compression was 48.7%, while that of long-tail recovery rules was only 15.7%; a similar trend was observed in the cross-model analysis across all 208 variants. Practical execution tests further indicate that the success rate of compressed anomaly tasks decreased for some models, though this effect was inconsistent. Overall, by constructing a specialized evaluation benchmark, this paper reveals the phenomenon of selective knowledge loss during the Skill compression process and emphasizes that relying solely on conventional task success rates is insufficient to measure the long-term reliability of an agent’s external procedural memory.

Critical strengths
  1. The research question exhibits a certain degree of novelty. This paper addresses the issue of selective knowledge loss that may occur in Agent Skills during the recursive compression process, pointing out that success rates in conventional tasks may mask a decline in the ability to recover from long-tail anomalies. Unlike existing research, which primarily focuses on the improvement of task performance or contextual compression efficiency achieved by Agent Skills, this paper treats the long-term reliability of external procedural memory as an independent research question, providing a new perspective on the maintenance and evaluation of Agent Skills.
  2. The benchmark design is systematic, and the experiments are persuasive. Based on 54 base tasks, the authors constructed 208 task variants covering 14 categories of anomalies. By pairing normal and anomalous environments, referencing Oracle executions, and employing deterministic verification, they ensured that anomalous tasks had clear and verifiable recovery paths. Additionally, the paper combines rule retention detection, semantic review, cross-model experiments, and analysis of actual task execution to verify, across multiple levels, that long-tail recovery rules are more prone to loss during the compression process, thereby enhancing the credibility of the core findings.
Critical weaknesses
  1. The research motivation lacks support from real-world scenarios, and the actual necessity of recursive compression has not yet been verified. This paper suggests that skill compression may lead to the loss of knowledge required to recover from long-tail anomalies; however, its core premise—that agents will continuously compress existing skills during long-term operation and skill evolution—lacks empirical support. In actual deployment, agents can easily retain their original skills over the long term and control context overhead through on-demand loading, modular storage, or retrieval mechanisms, without the need to repeatedly compress existing knowledge. Skill evolution may also manifest as the accumulation of experience, error correction, or rule updates, and does not necessarily entail a reduction in document length. The authors do not provide statistics on skill evolution trajectories or compression behavior in real-world agent systems; therefore, it remains unclear whether recursive compression is a common operation in the actual lifecycle of agents. This undermines the practical necessity and applied value of the problem addressed in the paper.
  2. The experiments employ artificially enforced compression and lack sufficient validation of real-world Skill maintenance mechanisms. The paper requires the LLM to rewrite the Skill for three consecutive rounds using preset compression instructions and length budgets, ultimately compressing it to approximately 34% of its original length. This setup resembles a human-designed stress test rather than an autonomous Skill maintenance process triggered by an Agent based on actual task requirements. Although the authors compared recursive compression, one-time compression, and compression methods that preserve original evidence, they did not further examine realistic and feasible maintenance strategies such as incremental updates, on-demand retrieval, and modular management. Furthermore, there is a lack of evidence demonstrating that the compression frequency and magnitude used are representative of real-world conditions. Therefore, the existing results primarily demonstrate the risk of selective knowledge loss under conditions of forced recursive compression; they are insufficient to indicate that this issue would occur universally during the long-term operation of real agents, thereby limiting the external validity of the study’s conclusions.
Questions for authors

Can the author provide empirical examples of autonomous compression or long-term evolution of skills in real-world agent systems, illustrating the frequency of recursive compression in actual deployments, its triggering conditions, and typical compression rates? Furthermore, given that real-world systems can maintain Skills through on-demand loading, incremental updates, or modular management, could the authors further explain why they chose “Artifact-Only Recursive Compression” as the primary research scenario, and how this setup relates to actual agent maintenance mechanisms? This evidence is crucial for assessing the practical significance of the issues revealed in the paper and the external validity of its experimental conclusions.

6 comments
AuthorsOpenReview ↗

Response to 2hjQ: Empirical evidence and experimental design

Thank you for the review. We examined public maintenance histories and Hermes Agent reports to assess the empirical basis of our protocol.

Section, figure, table, equation, and page numbers below refer to the original submission.

Response to Weakness 1: Evidence for skill compression

We examined 1,550 actively maintained skill and instruction files from 1,532 public repositories, each with at least five commits. A substantial reduction decreases a file’s character count by at least 30% in one commit.

ObservationResult
Files with at least one substantial reduction365/1,550 (23.5%)
Corresponding proportion among SKILL.md files26.7%
Median size remaining across 498 substantial reductions38% of the pre-edit size
Reduced files that subsequently grew again155/365 (42.5%)
Reduced files with more than one substantial reduction92/365 (25.2%)
Substantial-reduction commits with coding-agent co-author attribution126/498 (25.3%)

Agent-attributed commits include 23 substantial reductions of SKILL.md files, 21 attributed to Claude Code.

A second sample contains 2,853 skill files from 31 repositories and 6,417 substantive edits, sharing only one file with the first sample. Of these edits, 69.9% modify existing content, 15.9% shorten the file, and 14.2% purely append. These histories document repeated shortening; our experiment tests which instructions survive it.

Response to Weakness 2: Design and compression intensity

Each round receives the preceding artifact without new instructions (Section 3, Eq. (1), p. 3; Section 5.1, p. 5). This isolates compression and tracks the same source instructions through successive rewrites, holding additions and corrections fixed.

Our 30% per-round reduction budget (Section 5.1, p. 5) lies near the 75th percentile of observed shrink events; the cumulative endpoint corresponds to the most aggressive tenth of the observed histories. Under the same lexical matching criterion, tail retention is already lower than regular retention at S2 in all five pipelines, before the final, most compressed artifact.

Retaining originals, on-demand retrieval, and modular storage can preserve access to instructions. ONE tests direct compression to the target budget; SRC tests whether source access during rewriting helps retain instructions (Section 6.2, p. 7). These controls evaluate rewriting choices, rather than complete retrieval or modular-maintenance systems. TailSkills provides paired normal and exception environments, oracle admission, and deterministic checks (Section 4, pp. 4–5; Figure 2, p. 5) to evaluate recovery under such strategies, including those that retain or retrieve external instructions.

Response to the Question: Runtime examples, triggers, and artifact-only recursion

Runtime evidence. Hermes Agent can run a background self-improvement review after a conversation turn and update existing skills. It caps each skill at 100,000 characters. The review operates on the current skill and session.

One user report records two different skills being archived within an hour. One was a 53 KB package containing SKILL.md, 13 reference files, and four scripts; the reporter found none of seven distinctive markers in the remaining active skill and reference files. Another report describes background rewriting of a pinned, manually maintained rule skill after chat turns. Archival report; rewriting report.

Frequency, conditions, and magnitude. The table reports incidence, recurrence, and magnitude in the sampled histories. We also examined file length: among 737 files with both shortening and append events, shortening occurred at greater file lengths than appending in 523. Of reductions starting above 500 lines, 41.8% ended at or below that guideline. The conditions for background review do not specify how often skills are compressed; these sources do not establish a deployment-wide compression rate.

Artifact-only recursion. We test retention when each rewrite relies on the current artifact without revisiting the original source evidence. This retains Hermes’ dependence on the current skill while introducing no new content between rounds. ONE and SRC vary the compression path and source availability, respectively.

We will add the sampling criteria, event definitions, maintenance statistics, runtime sources, compression-intensity comparison, S2 results, and design rationale to the revision.

AuthorsOpenReview ↗

Follow-up on practical motivation and runtime evidence

Thank you for considering our response. We would appreciate your assessment of whether the maintenance-history analysis and runtime examples address your concern about the practical motivation for repeated skill rewriting.

The repository histories document repeated shortening of the same files, while the Hermes reports provide runtime examples of agents modifying persistent skills. Our protocol isolates the shortening operation by holding the source instructions fixed across rounds.

If any concern remains, we would appreciate clarification so that we can address it in the revision.

AuthorsOpenReview ↗

Follow-up: Revision posted, including the maintenance-history evidence (Appendix O)

Following up on our earlier note. The revised manuscript is now available on OpenReview, with changes marked in blue. The maintenance-history statistics, the runtime examples, and the comparison between our compression budgets and the observed histories are in Section 1 and Appendix O. We would appreciate your assessment of whether this evidence addresses your concern about practical motivation, and we are willing to clarify further.

Reviewer_aucbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

The paper brings an analysis on whether repeated compression of agent skills removes procedural knowledge required to handle exceptional situations while preserving instructions for common workflows. It also introduces TAILSKILLS, a benchmark with 208 task variants derived from 54 base tasks and 14 exception categories. The authors evaluate artifact retention across successive compression stages, compare regular workflow instructions against exception-handling procedures, and examine whether information loss translates into downstream execution failures.

The main finding is that recovery instructions exhibit substantially lower retention than regular workflow content according to the proposed artifact-level measurements. The authors further investigate semantic retention, different model pipelines, and the behavioral consequences of compression.

Critical strengths
  • The paper addresses an interesting reliability concern: procedural knowledge may appear sufficient for regular tasks, while failing to preserve instructions needed to handle exceptional situations.
  • TAILSKILLS is a potentially useful benchmark. It includes 208 task variants derived from 54 base tasks across 14 exception categories. The use of paired normal and exception-triggering environments, reference-oracle execution, and deterministic verification provides a systematic setting for investigating the preservation of recovery procedures.
Critical weaknesses
  • I am not fully convinced by the motivation for the repeated skill-rewriting process studied in this paper. While skill maintenance and compression are relevant to long-running agents, the experimental protocol repeatedly compresses an existing skill without incorporating new experiences, knowledge, or procedural updates between iterations. It is therefore unclear why a practical agent system would follow this particular process, especially when the original skill could potentially be retained or when direct compression to the target length is available.
  • the reported matched-control results primarily document generation completion rates, structural validity, and artifact lengths. They do not clearly provide the substantive recovery-rule retention comparisons needed to establish the specific effect of recursive inheritance or the benefit of source access. If recursive compression is no worse than one-shot compression at a comparable length, the central phenomenon may be better understood as compression-induced loss of exceptional instructions rather than a distinct problem arising from repeated skill maintenance.
Questions for authors
  1. Given that prior work has already established the preferential loss of important instructions under context compaction, what fundamentally new insight does the evaluation of recovery procedures provide? In particular, does the paper demonstrate a distinctive vulnerability of procedural knowledge, or does it primarily reproduce a known compression failure in a different type of artifact?

  2. Is there direct evidence that a recovery instruction removed during compression causes an execution failure? Can the authors identify cases where the executor demonstrably reads the skill and restoring the omitted instruction changes the execution outcome?

7 comments
AuthorsOpenReview ↗

Response to aucb (1/2): Motivation and REC/ONE/SRC controls

Thank you for the review. We analyzed public skill-maintenance histories and compared recovery-rule retention across REC, ONE, and SRC.

Section, figure, table, equation, and page numbers below refer to the original submission.

Response to Weakness 1: Why study repeated rewriting without new content?

We introduce no new instructions between rounds so that we can track the same source instructions through successive rewrites. Real maintenance combines additions, corrections, and shortening. Our protocol isolates shortening: each round receives the preceding artifact (Section 3, Eq. (1), p. 3), and we measure which source instructions remain.

We examined whether substantial and repeated shortening occurs in practice. Across 1,550 actively maintained skill and instruction files from 1,532 public repositories, each with at least five commits:

  • 365/1,550 files (23.5%) underwent at least one reduction of 30% or more in character count; the proportion was 26.7% for SKILL.md files.
  • Across 498 such reductions, the median file retained 38% of its pre-edit characters.
  • 155/365 affected files (42.5%) subsequently grew again, and 92/365 (25.2%) underwent further substantial reductions.
  • 126/498 reduction commits (25.3%) carried coding-agent co-author attribution, including 23 reductions of SKILL.md files.

Hermes users also report background rewriting of manually maintained skills and archival of skill packages, providing runtime examples of agents modifying persistent skills. Rewriting report; archival report.

We compared our compression budgets with the observed reductions: the 30% per-round reduction (Section 5.1, p. 5) lies near the 75th percentile of observed shrink events, and the cumulative endpoint corresponds to the most aggressive tenth of the observed histories. We therefore use repeated compression as a controlled stress test of an observed maintenance operation.

ONE tests direct compression to the target budget; SRC tests access to source material during rewriting (Section 3, Eqs. (2)–(3), p. 3; Section 6.2, p. 7). The next comparison measures retention under these alternatives.

Response to Weakness 2: Recovery-rule retention in REC, ONE, and SRC

We applied the Table 6 (p. 13) high-precision scorer to all three endpoints, including variants whose three chains passed quality checks and whose registered source units were scorable: 111 variants for DeepSeek and 141 for GLM. We paired conditions within each variant and averaged the per-variant scores within each model. Mean endpoint lengths across the conditions range from 51% to 56% of S1.

ModelEligible variantsREC retentionONE retentionSRC retentionVariants with no matched tail unit: REC / ONE / SRC
DeepSeek11111.3%14.5%17.5%71 / 71 / 66
GLM14116.1%15.5%19.6%86 / 89 / 77

The paired differences are:

ModelComparisonDifference, percentage pointsBootstrap intervalPaired-test p-value
DeepSeekREC − ONE−3.2[−8.0, +1.2]0.18
GLMREC − ONE+0.6[−2.7, +4.0]0.73
DeepSeekSRC − ONE+3.0[−1.7, +7.9]0.22
GLMSRC − ONE+4.1[+0.7, +7.9]0.026

REC and ONE show no statistically significant retention difference in either model; the intervals do not establish equivalence. SRC has higher mean retention than ONE in both models, with statistical support for GLM and an inconclusive difference for DeepSeek. Even with source access, the scorer matches no registered tail unit in 66/111 DeepSeek variants and 77/141 GLM variants. We will frame these findings as compression-induced textual loss across the evaluated paths, without claiming an additional recursion-specific penalty (Appendix A, Table 4, p. 12).

AuthorsOpenReview ↗

Response to aucb (2/2): Contribution, skill access, and recovery-package intervention

Response to Question 1: What does recovery-specific evaluation add?

TailSkills makes exception recovery a separately testable property of a maintained skill. Existing work, including ACE, studies detail loss and context collapse during iterative rewriting and develops methods for preserving evolving contexts. Our contribution is a task-grounded evaluation of which recovery instructions survive maintenance and how to test the corresponding recovery procedures. ACE.

The benchmark pairs normal and exception environments while holding task intent and the initial complete skill fixed. Its 208 variants span 54 base tasks and 14 exception types. Oracle admission verifies a prescribed recovery path, and deterministic checks evaluate the resulting output. This design tests recovery under a specific environmental change instead of relying on success in the normal environment as a proxy (Section 4, pp. 4–5; Figure 2, p. 5).

The artifact analysis adds a within-skill comparison. We applied the same near-verbatim matching criterion to regular and tail units and found lower tail retention in all five pipelines at all three depths. At S4, the regular-minus-tail gaps range from 6.8 to 15.0 percentage points, and the ordering is already present at S2.

Our comparison measures selective textual retention of regular and recovery instructions within the same skill. TailSkills complements this analysis with paired tasks that test the corresponding recovery procedures.

Response to Question 2: Skill access and the contribution of recovery instructions

Skill access. The Gemini CLI trajectories directly record the agent calling activate_skill and receiving the full skill body. Taking the latest valid run for each task–depth pair with a saved trajectory, we observed this in 291/316 runs (92.1%), including 87/97 at S4 (89.7%). These records demonstrate that executors accessed compressed skills during execution.

The OpenHands implementation we inspected supplies the full SKILL.md through AgentContext, without requiring a separate file-reading action. Re-examining the eight GLM trajectories also identified a logging limitation: they store only the static system message and omit tool-call arguments. The matching saved messages and absence of recorded file reads therefore do not establish that the skill was unavailable. We will correct this inference in Section 6.6 (pp. 8–9) and the related discussion (Section 7, p. 9; Appendix L, Tables 13–14, pp. 19–20).

Recovery-package intervention. We tested the contribution of the added recovery package by replacing the complete augmented skill with its original SkillsBench counterpart (Section 6.1, p. 7; Appendix D, Table 9, p. 15). On a subset selected for success with the complete skill, the complete skills passed 48/48 valid runs, while the replacements passed 0/48, under the same environment and verifier. This comparison demonstrates the importance of the added recovery package for solving these tasks.

The replacement removes the added recovery rules and helper logic together, so the intervention tests their contribution at the package level. The current experiments do not test the effect of restoring an individual instruction omitted during compression.

The revision will include the maintenance-history analysis, REC/ONE/SRC retention results, common-criterion regular–tail comparison, and skill-access evidence, and clarify the package-replacement experiment.

AuthorsOpenReview ↗

Follow-up on Q2: Restoring an Omitted Recovery Rule

Following up on Q2, we ran an experiment on sales-pivot to see whether adding back a recovery instruction omitted during compression changes how the agent handles the exception.

We ran 16 trials with each version of the compressed skill. The two versions differed by a single recovery instruction restored from the original skill. Both conditions used the same GLM executor, helper files, environment, and verifier. The request logs confirm that the instruction appeared in the model input in all 16 runs with the rule restored and was absent in all 16 control runs.

With the instruction restored, the agent used the specified recovery procedure in 13/16 runs, compared with 0/16 in the control group. The task passed in 11/16 runs with the instruction and 6/16 runs in the control group. We’ll include the per-run results in Appendix P of the revised manuscript.

Thank you for raising this question. We’d be happy to discuss the experiment further if there’s anything you’d like us to clarify.

Reviewer_NMUuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work introduces TAILSKILLS that studies whether repeated predecessor-only compression preserves exception-recovery procedures in agent skill files. It contains 208 oracle-admitted variants from 54 tasks and 14 exception types, with paired ordinary/exception environments and deterministic output checks (pp4–5, sec. 4). The paper reports S4 regular/tail retention of 48.7%/15.7% in a 56-variant registered-marker audit and the same ordering in a separate all-208 analysis across four pipelines (pp6–7; p13 Table6; p15 Table10). Fixed-52-variant execution endpoints decline in three pipelines and are unchanged in one. A separate semantic sample has high retention; paired execution effects are mixed, and fully traced failure cells show no skill access. These qualify the connection between measured artifact loss and operational failure (pp7–9, sec. 6; pp16,19–20).

Critical strengths
  1. The benchmark makes recovery durability a concrete evaluation target. Paired ordinary/exception settings hold task intent and the complete skill fixed; prescribed-path oracle admission and independent deterministic output checks help isolate a controlled, solvable recovery problem (pp4–5, sec. 4/Fig2). This is useful even without a new compression algorithm or uniform downstream degradation.

  2. The study separates evidence that is often conflated: registered markers, a broader content scorer, semantic adjudication, execution outcomes, and actual skill access. It reports regular-over-tail retention across four pipelines while preserving semantic and behavioral counterevidence (pp6–9; p15 Table10; pp19–20 M). That transparency makes the benchmark and its limitations informative; the ordering is reported evidence, not independently validated scorer accuracy.

Critical weaknesses
  1. The central retention measurements need reconciled accounting. The 56-variant marker audit reports 67.3% tail-absent, which cannot be an integer count out of 56 rounded to one decimal; likewise GLM’s 94.6% cannot be an integer share of 208 (p6 sec. 5.3; p13 Table6; pp7,15 Tables1,10). Under an exhaustive fixed regular/tail unit partition with common pooling, Total=(1−w)Regular+wTail requires a constant tail-unit share w. In the primary Table6, S2 permits w=8.33–9.00% and S3 permits 9.28–9.88%, allowing every displayed rate ±0.05 percentage points. These ranges do not overlap; Table10 also fails this interpretation (e.g., GLM S2/S3 imply 11.05%/6.78%). Total’s definition, exclusions, weighting, and exact counts are needed. This does not prove the component rates or regular–tail ordering false: an alternative aggregation or a Total/label-only correction could preserve the central result.

  2. The operational meaning of the broad retention score remains insufficiently validated in the inspected evidence. The separate semantic audit retains 19/23 aligned tail units, but its sampling universe differs from the observable audit; it is not a same-unit estimate of scorer accuracy or false negatives (pp3–4, Eq5; p8 sec. 6.4; pp19–20 M). Nor do outcome changes alone validate the scorer: paired effects are mixed, and the fully traced flip cells lack skill access (p8 sec. 6.5–6.6). This leaves a narrower reported artifact diagnostic, rather than established semantic recovery loss or skill-mediated failure. Such a diagnostic can still be valuable; I do not require uniform behavioral degradation, a recursion-specific effect, or a new broad execution study.

Questions for authors
  1. For Tables6 and10, please provide the exact source/eligible unit counts, variant denominators, exclusions, weighting, and definition of Total. How do these explain both the absence percentages and the incompatible fixed-partition weights? If only Total or a label needs correction, do the regular/tail rates remain unchanged on the stated populations? This distinguishes a reporting repair from a change in the central comparison (pp6–7,13,15).

  2. How do the 32 semantic items and 23 aligned tail units map to depth, condition, and the marker/all-208 scorer decisions? Please point to existing sampling/alignment rules and any same-unit validation against the blinded labels, or clarify the narrower interpretation supported without that calibration. This could change how much operational meaning I credit to the retention gap (pp3–4,8,19–20). These are clarification requests about current evidence, not future-work requirements.

4 comments
AuthorsOpenReview ↗

Response to NMUu (1/2): Table populations, definitions, and weighting

Thank you for identifying the inconsistencies in our retention tables. We recomputed both tables from the archived artifacts and compared regular and tail retention under a common matching criterion. We also clarify the semantic sample and its relation to the scorers.

Section, figure, table, equation, and page numbers below refer to the original submission.

Response to Weakness 1 / Question 1: Table populations and weighting

We reproduced the reported values and identified two reporting errors: the descriptions of Tables 6 (Appendix C, p. 13) and 10 (Appendix E, p. 15) were interchanged, and neither table uses the unit weighting stated in the captions. We apologize for these errors. The corrected analysis populations are:

TableDescription in the submissionActual analysisEligible variants
Table 656-variant marker auditHigh-precision unit analysis162 of 208
Table 10All-208 analysisMarker-based character analysis56

Table 6: counts and definitions. The original analysis excluded five variants without a paired baseline skill in the analysis snapshot and 41 without an extractable S1 tail unit. The remaining 162 variants contain 449 source tail units, with no missing artifacts at S2–S4. The tail-absent counts are 56/162, 87/162, and 109/162 at S2, S3, and S4. The reported S4 value of 67.3% (Section 5.3 and Figure 4, p. 6) therefore corresponds to 109/162.

We calculate each measure within a variant and then average across variants with equal weight. Let Lv,kL_{v,k} denote the word count of variant vv's skill at depth kk, and Hv,kH_{v,k} the source-word count of its S1 tail units that remain matched at that depth:

Totalv,k=Lv,kLv,1\mathrm{Total}_{v,k}=\frac{L_{v,k}}{L_{v,1}}

Tailv,k=Hv,kHv,1\mathrm{Tail}_{v,k}=\frac{H_{v,k}}{H_{v,1}}

Regularv,k=max⁡(Lv,k−Hv,k,0)Lv,1−Hv,1\mathrm{Regular}_{v,k}=\frac{\max(L_{v,k}-H_{v,k},0)}{L_{v,1}-H_{v,1}}

Recomputing these measures from the SKILL.md files reproduces all 162 per-variant records. We also recovered the five missing baselines from the pinned SkillsBench commit. Including these variants gives S4 regular/tail retention of 48.4%/15.4%, with 113/167 variants having no matched tail unit.

Table 10: counts and aggregation. Table 10 uses a marker-based character scorer on 56 variants with registered markers. We calculate Total, Regular, and Tail separately for each variant and report their unweighted means across the 56 variants at each depth, rather than ratios of pooled character counts. Rerunning the scorer reproduces all 48 reported values without further exclusions. The tail-absent counts are:

DepthGLMDeepSeekQwen3-MaxGPT-5.4
S217/5621/5617/5616/56
S349/5636/5628/5641/56
S453/5641/5640/5651/56

In particular, GLM's S4 tail-absent rate is 53/56 = 94.6%, rather than a proportion of 208.

Why the inferred pooled weights differ. Both tables average per-variant ratios. For Table 6, each variant has its own S1 tail share, wv=Hv,1/Lv,1w_v=H_{v,1}/L_{v,1}. We verified the following identity to floating-point precision for the analyzed variants:

Totalv,k=(1−wv)Regularv,k+wvTailv,k\mathrm{Total}_{v,k}=(1-w_v)\mathrm{Regular}_{v,k}+w_v\mathrm{Tail}_{v,k}

Because the tail share varies across variants, the table-level means need not satisfy this identity with a single pooled weight. This explains why the weights inferred from those means differ across depths.

AuthorsOpenReview ↗

Response to NMUu (2/2): Retention comparison and semantic audit

Response to Weakness 1 / Question 1 (continued): Common-criterion comparison

Regular and tail retention under a common criterion. Table 6 (Appendix C, p. 13) measures regular retention as residual text length and tail retention through near-verbatim matching. Its original 33.0-percentage-point S4 gap (Section 5.3, p. 6) therefore combines different criteria.

We addressed this mismatch by applying the same unit-matching criterion to regular and tail content. We extracted regular units from S1 blocks identical to the baseline and evaluated 138–139 eligible variants, depending on the pipeline. The S4 results are:

PipelineRegular retentionTail retentionRegular − tail, percentage points
Gemini27.5%15.3%+12.2
GLM13.3%6.5%+6.8
DeepSeek9.1%2.2%+6.9
Qwen3-Max18.9%10.1%+8.8
GPT-5.424.8%9.8%+15.0

Regular retention exceeds tail retention in all five pipelines at every evaluated depth. The 95% bootstrap intervals exclude zero in all 15 pipeline–depth comparisons. The common criterion preserves the regular-over-tail ordering and yields directly comparable retention estimates.

We also applied Table 6's high-precision scorer to all five pipelines. The table below lists each pipeline's eligible population, which differs from the population used in the common-criterion comparison:

PipelineS4 tail retentionVariants with no matched tail unit
Gemini15.7%109/162
GLM7.1%126/161
DeepSeek2.1%149/162
Qwen3-Max10.3%127/162
GPT-5.410.1%115/162

Response to Weakness 2 / Question 2: Semantic sample, annotation, and relation to the scorers

The semantic sample (Section 6.4, pp. 7–8; Appendix M, pp. 19–20) contains 32 items from DeepSeek and GLM, 16 each, spanning S2–S4. Each item specifies one tail unit and three regular or control units, frozen before annotation. The audit combined human and model-based assessment. Two human annotators worked independently; a third-party vote resolved disagreements.

Of the 32 tail units, 26 aligned with the target exception, 23 were adjudicable, and 19 were fully retained. The 19/23 result pools models and depths and describes semantic retention among the aligned, adjudicable tail units in this sample.

The semantic audit and the two textual scorers used separately defined units (Section 3, pp. 3–4). Their decisions were not aligned on the same clause and candidate artifact, so the 19/23 result describes the semantic sample rather than estimating scorer accuracy or false-negative rates.

Tables 6 and 10 measure marker-based or near-verbatim textual retention; the independent audit evaluates whether sampled clauses retain their meaning. A paraphrase can preserve a recovery instruction that lexical matching misses. The mixed execution results (Section 6.5 and Figure 6, p. 8) neither calibrate these scorers nor establish that an unmatched rule caused an execution failure.

We will correct the table descriptions, weighting labels, and corresponding abstract statements (p. 1); replace the mixed-criterion comparison with the common-criterion results; and clarify the semantic sample and annotation procedure in Section 6.4 and Appendix M.

AuthorsOpenReview ↗

Follow-up: Corrected tables and common-criterion comparison now in the revision

Following up on our responses to your review. The revised manuscript is now available on OpenReview, with all changes marked in blue. The corrected populations and weighting for Tables 6 and 10 are in Appendices C and E, and the common-criterion regular–tail comparison with bootstrap intervals is in Section 5.4, Figure 3, and Table 1. The semantic sample and its annotation procedure are clarified in Section 6.4 and Appendix M. We would appreciate your assessment of whether these revisions address the accounting concerns you raised, and we are happy to clarify anything further.

Thank you for correcting the table populations and explaining the weighting. Applying the same matching rule to regular and recovery instructions addresses my main concern. The semantic check also clarifies that the gap extends to meaning in three of the five pipelines. I’m raising Soundness from 2/3 to 3/3 and keeping my overall rating at 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their feedback and for recognizing the value of the paired normal and exception environments, oracle admission, and deterministic verification. We have posted individual replies. The main changes addressing concerns raised across the reviews are marked in blue in the revised manuscript. The references below use the revised numbering.

Retention measurement and interpretation (NMUu, aucb). We corrected the analysis populations and weighting reported for Tables 6 and 10 (Appendices C and E). We also recomputed the main comparison using the same near-verbatim matching criterion for regular and recovery instructions. Recovery instructions have lower textual retention across all five pipelines at S2–S4, with S4 gaps of 6.8–15.0 percentage points (§5.4, Figure 3, and Table 1). Appendix E gives the word weighting, per-variant aggregation, and bootstrap intervals for this comparison; the 95% intervals exclude zero in all 15 pipeline–depth comparisons. The separate semantic audit reports how often sampled instructions retain their meaning (§6.4 and Appendix M).

Practical motivation and compression controls (aucb, 2hjQ). In the sampled public maintenance histories, 365 of 1,550 files shortened by at least 30% in a single commit, and 92 of those files underwent further substantial reductions. Hermes Agent reports also document repeated background edits to a persistent skill. These observations motivate studying shortening under controlled conditions (§1 and Appendix O). The revision reports recovery-rule retention and realized endpoint lengths for REC, ONE, and SRC under a shared target budget. The REC–ONE intervals include zero for both tested models. SRC has higher mean retention than ONE in both models, with a positive interval for GLM (§6.2 and Appendices E and H).

Execution after restoring one omitted rule (aucb, NMUu). On sales-pivot, we restored one recovery instruction omitted during compression and ran 16 trials with each version of the compressed skill. Both conditions used the same GLM executor, helper files, environment, and verifier. The request logs confirm that the instruction appeared in the model input in all 16 runs with the rule restored and was absent in all 16 control runs. With the instruction restored, the agent used the specified recovery procedure in 13/16 runs, compared with 0/16 in the control group. The task passed in 11/16 runs with the instruction and 6/16 runs in the control group. §6.6 summarizes the experiment, and Appendix P gives the restored text, the request-log check, and the per-run results (Tables 17 and 18).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.