acceptodds
Under review as a conference paper at ICLR 2027

Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding

Abstract

Agentic coding READMEs like CLAUDE.md grow without bound in real repositories, stopping only when the repository retires or someone rewrites the file wholesale. We trace this to imperfect recall: appending an instruction is always cheap, but once an instruction's rationale is gone, deleting it without risking a correctness regression costs O(2^|D|) in a prompt of |D| instructions. We name the resulting divergence catastrophic remembering, the inverse of catastrophic forgetting around which continual learning is organized. First, we characterize this phenomenon across 247,694 instruction lifetimes in 1,867 repositories: agentic prompts grow without bound, more than tripling over their lifetime (+226%), gaining +4.9 net instructions every commit; further, the older an instruction gets, the less likely it is to be deleted (log-hazard -0.032/commit). Then, we show that prompt comments can halt the growth: inverting IFEval yields verifiable worlds whose optimal prompts are known, and there comments encoding latent reasoning remove 99.3% of excess instructions (+211.3% to +1.4%). Finally, applying the same inversion to WildIFEval, we show that prompt comments can improve real-world agentic instruction-following by up to 23.1%. If English is the new code, why don't we have comments yet?

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, context engineering, instruction following, agentic coding, prompt optimization, empirical study of LLM systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EoYrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines why context files like CLAUDE.md grow nearly monotonically. The authors hypothesize that the rationale behind each instruction is lost over time, so maintainers cannot safely delete instructions. They formalize this process as a survival model. The authors analyze instruction lifetimes across repositories and identify a growth–rewrite–regrowth pattern and a declining deletion hazard. In their experiments, they invert IFEval and show that adding comments helps keep the prompt near its optimal size. On WildIFEval with injected distractor instructions, such comments also improve instruction-following.

Critical strengths
  1. The question of why CLAUDE.md files keep growing is interesting and timely, and the paper looks at how individual instructions survive and die across almost 250K instruction lifetimes from about 1,900 repos.

  2. Inverting IFEval is a useful way to conduct experiments. It gives a reference prompt to compare against, which makes it easier to measure both how bloated a prompt is and how well it works.

Critical weaknesses
  1. It’s not clear the gains come from the comments. The treatment arm also receives explicit deletion guidance (Figure 9) that the control and placebo arms lack. The ablations in Table 11 all keep this guidance, so they cannot isolate its effect. A control with the guidance but no informative comments would separate the two.

  2. The repository evidence makes imperfect recall plausible but does not establish it as the root cause claimed in Section 3. The authors acknowledge the evidence is correlational, and rationale loss is never measured directly. Partial effort dilution also remains open: it is unclear whether the age effect persists after controlling for time-varying file size.

Questions for authors
  1. Besides writing comments, the treatment arm also seems to get explicit deletion rules (Figure 9) that the placebo arm doesn’t. Do you have results with the rules but no comments? If the gain mostly disappears, that would change my view of what’s driving it.

  2. Does the drop in deletion rate with age still hold after controlling for file size? This would help rule out the simpler explanation that instructions in longer files are just less likely to be deleted.

2 comments
AuthorsOpenReview ↗

Matched control: reading comments carries 2/3; age survives size

Thank you for a careful review, and especially for naming the control that would change your view; we ran it, an even stronger control, the file-size re-fit and other analyses to get to your deeper questions.

Causally isolating rationale loss (Q1, W1). We agree that varying the prompt inadvertently introduced a confound. A new control, control::write-only, keeps Fig. 9's rules but never shows the maintainer its own comments; reading them still strips 86% of its remaining excess size (0.208 → 0.029; 64 IFEval worlds, T=15). However, we found this comparison awkward: Fig. 9's add/delete rules act on what the comments record, so without readable comments they are ill-defined.

We therefore built a new ablation ladder in which every arm shares one system prompt and only the comment channel varies (64 worlds, one seed; fresh seeds at camera-ready):

ArmPromptMaintainer writesMaintainer seesSize vs. |D*| ↓Size vs. |Dⱽ| ↓Satisfaction ↑
control::nonesharednothingnothing1.3751.5860.424
control::write-onlysharedits own commentsnothing1.2111.5270.453
control::noise-onlysharedits own commentscomments from unrelated worlds1.0231.1050.378
practical::commit (ours)sharedits own commentsits own comments0.8781.0730.443

Bold: best in its column and beats the runner-up at equal satisfaction (p < 0.05, two-sided, paired over worlds). Dⱽ drops the 17.5% of IFEval's instructions today's executor follows by default (leave-one-out on D*).

With the system prompt held fixed, reading informative comments carries two-thirds of the gain: practical::commit ends 0.333 smaller than control::write-only (z −2.34) and 0.497 smaller than control::none (z −3.47), at indistinguishable satisfaction. Comments from unrelated worlds also shrink the prompt but cost satisfaction (−0.065 vs. ours, z −2.12), so comment content matters beyond volume. We will add these arms to Table 11 and replace the fixed-string placebo, which bundled comment content with the rules.

Observationally isolating rationale loss (Q2, W2). Beyond the requested file-size control, we add five analyses:

  • The drop in deletion with age holds net of file size. Regressed against instructions, the age slope is −0.059 per commit [−0.071, −0.047] and negative in every band. Size lowers hazard too (partial effort dilution), but the strong age residual (z −9.8) is consistent with fading rationale.
Log-hazard termAge onlyAge + file size
Age (per commit)−0.056 [−0.070, −0.042]−0.059 [−0.071, −0.047]
Log file size (per e-fold)–−0.171 [−0.302, −0.040]
Log file size × age–+0.022 [+0.001, +0.043]
File size (directives)Age slope [95% CI]DeletionsFilesRepositories
1–31−0.139 [−0.177, −0.101]2,733706653
32–127−0.078 [−0.096, −0.061]15,301985867
128+−0.047 [−0.064, −0.030]19,835348321
  • Real prompts rarely record a rationale. Under 1% of 247,694 real instructions state a reason ("because", "so that", …) and a hand-checked 1.5% the failure they fix. So §4's comment-free prompts are the real-world case, consistent with rationale living only in authors' memory.
  • When instructions encode rationale, files churn less. Net of size and age, files that record reasons are wholesale-rewritten 0.67× as often (log-hazard −0.401 [−0.562, −0.239]), consistent with rationale enabling targeted pruning.
  • Rewrites emerge organically. Our §4 worlds never induce rewrites, yet at T=51 control maintainer agents rewrite 0.41 times per world (mostly after growth, as in the corpus), against 0.14 for ours (Δ +0.27 [+0.11, +0.45]), consistent with lost rationale driving the growth–rewrite cycle.
  • Maintainers holding the rationale prune disproportionately. Authors delete their own instructions 1.3× as often as other maintainers (log-hazard +0.288), even controlling for proximity; neither file size nor fading relevance predicts this, consistent with the author still knowing the latent rationale.
  • Relevance drift biases against us. Drift predicts more deletion and novel content with age. Deletion falls (first point); each age doubling cuts novel additions by 5.9 points [−6.7, −5.1] and raises restatements of the file's instructions by 1.7 [+1.0, +2.5], consistent with forgetting, then relearning.

That said, you're right that §3 alone can't establish rationale loss as the root cause. Our aim there was to characterize the empirics, rule out common rivals and point to lost rationale; §4 carries the causal claim. We will soften §3 to say exactly that.

Thank you again: naming the control you wanted made the paper's central claim sharper and easier to test. We hope the matched ladder and new observational analyses address your main concerns — we'd be glad to run anything else that would help.

Thank you for the additional experiments and analyses. Overall, I think this paper addresses a narrow but interesting research question and provides a thoughtful analysis.

The new ablations largely address my first concern, and the additional analysis controlling for file size addresses my second concern. I also appreciate the authors' clarification regarding the causal claims about imperfect recall.

I encourage the authors to incorporate these additional results and clarifications into the revised version. I am inclined to raise my rating to weak accept.

Reviewer_kgwLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why agentic prompt files (e.g., CLAUDE.md) grow unboundedly and proposes prompt comments carrying each instruction's latent reasoning to halt growth and recover instruction-following.

An observational study of 1,867 GitHub repositories (247,694 instruction lifetimes) finds prompts grow +226% in instruction count, with 67.8% of instruction deaths occurring in wholesale rewrites and deletion hazard decaying geometrically with age. The authors call this "catastrophic remembering" and reject rival mechanisms (staleness, fragility, effort dilution) via hazard-family fits.

The interventional half inverts IFEval and WildIFEval into worlds with known minimum cover and hidden constraints. On IFEval (184 worlds, T=51), comments reduce excess size from +211.3% to +1.4% at satisfaction parity (removing 99.3% of the ratchet). On WildIFEval (64 worlds), 16 distractors cost 24.1pp of satisfaction and comments recover 11.6pp. A cross-vendor judge reproduces the contrasts.

Critical strengths
  1. A new, methodologically rigorous diagnosis of agentic prompt growth. Prior work stops at aggregate line diffs. This paper goes to the instruction level: 247,694 instruction lifetimes across 1,867 repositories, a matcher validated at 1.000 precision / 0.933 recall, and a hazard-family comparison showing geometric decay (Rw2=0.815R_w^2 = 0.815) dominates five rivals, with staleness, fragility, and effort dilution rejected via explicit covariates.

    The "catastrophic remembering" framing—rationale decays faster than the instruction—is crisp and falsifiable. This is consequential because the central diagnosis is a new empirical object, not a restatement, and is likely to shape how the community thinks about prompt maintenance regardless of whether the proposed fix generalizes.

  2. A clean mechanism proof with ablations that do real work. Inverting IFEval/WildIFEval makes minimum cover observable, turning "does a maintainer converge?" into a measurable quantity.

    Comment-shaped-noise placebo, field-by-field ablations (dropping recurrence count loses 37%; narrating without outcomes ends at +70.0%, worse than control), and cross-vendor judge replication (κ=0.639\kappa = 0.639, DID covering zero) rule out form, token count, and judge idiosyncrasy.

    The five reported failed protocols show the authors tested obvious alternatives. This is consequential because it makes the mechanism credible even to a reader skeptical of external validity: under controlled confounds, preserving rationale stops the ratchet.

Critical weaknesses
  1. The diagnosis and the proposed fix are evaluated in misaligned settings. The observational study shows that 67.8% of instruction deaths occur in wholesale rewrites, that growth resumes faster afterward, that low-frequency instructions may never be tested, and that tasks drift.

    The Section 4.2 worlds have no rewrites, every instruction receives high-frequency feedback, and the task is stationary; the feedback is also artificially impoverished to maximize the apparent value of comments.

    This is consequential because the fix is tested exactly where the dominant real-world growth mechanisms are absent, leaving open whether comments address the primary cause of observed expansion.

  2. The headline metric and its "parity satisfaction" claim rest on unverified assumptions. Minimal cover is taken by construction and not verified, so if instructions and verifiers are not one-to-one, the +211.3% to +1.4% reduction may not represent true convergence.

    Moreover, the treatment empties 12.1% of prompts (control: 2.4%), and on non-emptied worlds satisfaction is 39.9% vs. 41.3% (placebo) and 40.6% (control)—a consistent negative direction.

    This is consequential because the central claim of size reduction without correctness cost is weaker than presented.

Questions for authors
  1. The denominator of "excess size" and the "parity satisfaction" claim. Section 4.1 states that D∗D^* is taken as minimal by construction and not verified. If the original IFEval instructions and verifiers are not strictly one-to-one, how would the reported reduction from +211.3% to +1.4% (Table 1) change under a verified minimal cover? Furthermore, Section 4.2 notes that the protocol over-prunes, and Table 1 shows the treatment ends with fewer instructions than the minimum cover on average. Can you clarify whether the "at parity satisfaction" claim holds when prompts are emptied or over-pruned, or whether it is driven by the aggregate?

  2. Generalizability of format-based constraints vs. real-world semantic instructions. The evaluated constraints in IFEval and WildIFEval are purely format-based (e.g., specific word counts, paragraph numbers). Real agentic prompts largely encode semantic or behavioral rules (e.g., "always use existing utility functions," "do not modify test files"). Such rules lack immediate, mechanically verifiable feedback every round. Since your protocol relies on recording precise recurrence counts and outcomes (Figure 6), does it remain effective when the failure signal is delayed, ambiguous, or semantically attributed? If the mechanism only works for cheap-to-verify format constraints, it may not address the growth in real repositories where behavioral rules dominate.

2 comments
AuthorsOpenReview ↗

Coding tasks, verified cover, disaggregated performance

Thank you for a careful, exacting review; it led us to several new analyses, an illuminating decomposition and a stronger headline.

Aligned settings (Q2, W1). We tested prompt comments on coding tasks with semantic feedback and on real users' prompts (32 worlds, one seed):

SuiteArmSize vs. |D*| ↓Satisfaction ↑
CodeIFcontrol::none1.020.344
CodeIFpractical::commit (ours)0.670.357
WildIFEvalcontrol::none1.390.427
WildIFEvalpractical::commit (ours)1.140.478

Bold: better than control::none at p < 0.05 (paired over worlds).

CodeIF (Yan et al., 2025) gives each Python task about 13 hidden conventions and the maintainer sees only a judge's complaint ("the response relied on a different control structure instead of the requested one") — exactly the ambiguous, semantic signal you describe. WildIFEval's 27,199 real-user constraints are 50% content (include/avoid, focus, editing), 16% style (tone, persona) and only 22% form (format, length; Lior et al., 2025, §3.1, pp. 4–5).

We also add three analyses of representativeness:

  • Real-world rewrites are largely a response to growth. Files grow 2.8% per commit up to wholesale rewrites. Rewrite commits cite length six times as often as failure (5.0% vs. 0.8% of 1,018 commits). Afterward, 60% of re-adds restore a rewrite's cut, so growth resumes partly by relearning.
  • Rewrites emerge in our worlds. Although we never induce them, agentic wholesale rewrites emerge naturally. At T=51, control rewrites 0.41 times per world against 0.14 for ours (Δ +0.27 [+0.11, +0.45]), mostly after growth. Comments prevent rewrites by stopping the growth that triggers them.
  • Control and feedback are realistic. Only 1.5% of 247,694 real instructions state the failure they fix, so comment-free control is the real-world case. Further, many WildIFEval rules are inherently semantic (e.g. "the story needs to be suited to a nine-year-old"). Finally, richer feedback doesn't erase the effect: when CodeIF's complaints name the rule, comments still shrink the prompt (Δ −0.27, satisfaction within 0.03; 32 worlds).

That said, you're right that our feedback is dense and our task stationary. That is where comments should matter least: a wrong deletion fails the next round, and drift only adds irrelevant growth, so our result is a lower bound. We'll describe these as untested limitations in §6.

Verified cover (Q1a, W2). Thank you for pushing us to check the IFEval-provided reference prompt. Although instructions and verifiers are 1:1 by construction in IFEval, today's agentic executors follow 17.5% of them by default (leave-one-out on the reference prompt). Removing those gives us a verified cover Dⱽ (64 worlds, one seed):

ArmSize vs. |D*| ↓Size vs. |Dⱽ| ↓Satisfaction ↑
D* (reference prompt)1.0001.2900.870
Dⱽ (verified cover)0.8311.0000.872
control::none1.3751.5860.424
practical::commit (ours)0.8781.0730.443

Dⱽ works just as well as D* (0.872 vs. 0.870), and practical::commit ends 7% above it against 59% for control::none. For Table 1 itself (T=51), excess moves from +209.6% → +1.2% against D* to +263.7% → +30.9% against Dⱽ: comments remove 88% of control's excess rather than 99%.

Over-pruning and emptied prompts (Q1b, W3). Against Dⱽ, practical::commit ends above the minimum on average, so most of what looked like over-pruning was excess in D*. Decomposed by end state (Table 1's T=51 run, 184 worlds; size vs. |D*|):

WorldsWhat happensShareSize, oursSize, controlSatisfaction, oursSatisfaction, control
parity-sizeat |D*|43.5%1.002.460.3710.379
smallerbelow |D*|31.0%0.541.540.5030.500
largerabove |D*|16.3%2.547.520.5670.528
emptiedempty9.2%0.003.710.3330.373
↳ over-prunedours deleted a learned target4.3%0.006.560.3330.375
↳ unlearnableno arm learned it3.8%0.001.140.4050.405
↳ learned elsewhereonly other arms learned it1.1%0.001.250.0830.250

Ours matches control's satisfaction in every non-empty row at a fraction of its size. At T=15, the kept-world gaps are −0.7 and −1.4 points, both intervals spanning zero; at T=51 they vanish. You're also right that no arm reaches D*'s satisfaction, oracle included. We will describe §4.2's result as convergence in size, with satisfaction matched across arms and bounded by the limits of current agentic learning, and add this table to the appendix.

We found your points quite incisive; we hope the expanded tables and analyses answer your questions.

The additional experiments partially address my concerns, but generalization to realistic feedback conditions remains untested. Please narrow the conclusions accordingly. I will raise my score to 3.

Reviewer_QxZFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper attributes instruction accumulation in context files for coding agents to the loss of their original rationale. It studies 247,694 instruction lifetimes across 1,867 repositories and proposes retaining rationale comments. Controlled experiments show shorter prompts with preserved or improved instruction following.

Critical strengths
  1. The repository study offers a substantial empirical basis. Tracking individual instructions, alternative hazard models, and threshold checks make the analysis informative even though the mechanism remains uncertain.

  2. Table 1 shows a meaningful compression result: 2.2 versus 6.6 instructions, with 44.0% measured satisfaction in both arms. Placebo and content ablations, a replication over fewer rounds with three seeds, and a second judge provide useful supporting evidence. Reporting failed protocols and cases of excessive pruning is also helpful.

Critical weaknesses
  1. I am not convinced that the evaluation establishes the optimum defined in Eq. (1). Section 4.1 assumes that the original IFEval instructions form a minimal cover and explicitly leaves this unverified. Matching their instruction count therefore does not establish optimality. Table 1 shows a useful compression result at equal measured satisfaction, but the “99.3%” claim in Section 4.2 measures a reduction in excess size relative to the reference prompt. I would like to see the reference prompt’s satisfaction under the same executor and verifiers before interpreting this as convergence to an optimal prompt.

  2. I am not convinced that Sections 3 and 4 establish lost rationale as the cause of prompt growth. In Eq. (3), deletion hazard depends on both rationale recoverability and whether an instruction is excess; the observational fit does not separate them. The intervention also combines rationale access with retention/deletion guidance, so a matched control is still needed despite the useful content ablations. Section 5 also omits ACE (Zhang et al., 2025) and Dynamic Cheatsheet (Suzgun et al., 2025). I would want to understand what recording a rationale for each instruction adds over these existing approaches to maintaining textual experience, using a comparison that accounts for their different settings. The experiments on instruction following do not yet establish a benefit on actual coding tasks.

Questions for authors
  1. Section 4.1 assumes the reference instructions form a minimal cover. What supports treating proximity to their count as reaching the optimum in Eq. (1), and how does the reference prompt perform under the same executor and verifiers?

  2. What evidence distinguishes lost rationale from changes in instruction relevance in the repository study? In the intervention, does the benefit persist when retention/deletion guidance and available feedback are held constant, leaving rationale access as the only difference?

  3. What does recording a rationale for each instruction add over existing approaches to maintaining context such as ACE or Dynamic Cheatsheet? An adapted comparison using the same feedback, models, and total token/call budget, reporting both satisfaction and context size, would materially affect my assessment.

3 comments
AuthorsOpenReview ↗

ACE, Dynamic Cheatsheet adapted: ours ends smallest, including coding tasks

Thank you for a thoughtful, precise review.

Coding tasks and per-instruction rationale vs. ACE and Dynamic Cheatsheet (Q3, W2). We adapted ACE and Dynamic Cheatsheet and compared them with ours (32 worlds, one seed; fresh seeds in the camera-ready) on WildIFEval and a new coding suite (CodeIF):

SuiteArmSize |D_T|/|D*| ↓Satisfaction ↑Tokens/round
CodeIFcontrol::none1.020.3442.9K
CodeIFDynamic Cheatsheet2.010.3328.3K
CodeIFACE1.300.28310.9K
CodeIFpractical::commit (ours)0.670.3575.7K
WildIFEvalcontrol::none1.390.4272.4K
WildIFEvalDynamic Cheatsheet4.060.4886.2K
WildIFEvalACE1.960.4098.7K
WildIFEvalpractical::commit (ours)1.140.4784.7K

Bold (both tables): best in its column, and better than the runner-up at equal satisfaction at p < 0.05 (two-sided, paired over worlds).

practical::commit ends with the smallest prompt on both suites, coding tasks included, at equal or better satisfaction and with fewer maintainer tokens than either; ACE ends above even control::none, and Dynamic Cheatsheet's list lengthens every round. Their primary application suggests an explanation: both methods maintain advisory knowledge the model filters at use time, so surplus is cheap (ACE scores 78.6 vs. 78.3 on FiNER at 10K vs. 100K tokens, their Table 21); in contrast, our directives conjunctively bind every output under instruction-following, so surplus is costly. Neither records why an entry was added, so neither can tell a necessary entry from an excess one.

Isolating rationale access (Q2, W2). Every arm below shares one system prompt and feedback; only the comment channel varies (one seed, 64 worlds):

ArmMaintainer writesMaintainer seesSize vs. |D*| ↓Size vs. |Dⱽ| ↓Satisfaction ↑
D* (reference prompt)––1.0001.2900.870
Dⱽ (verified cover)––0.8311.0000.872
control::nonenothingnothing1.3751.5860.424
control::write-onlyits own commentsnothing1.2111.5270.453
control::noise-onlyits own commentscomments from unrelated worlds1.0231.1050.378
practical::commit (ours)its own commentsits own comments0.8781.0730.443

Writing comments, without ever reading them, already shrinks the prompt by 0.164 (z −1.06), about a third of the full effect, which we did not expect. Reading them gives the other two-thirds: practical::commit ends 0.333 smaller than control::write-only (z −2.34) at approximately the same satisfaction. Comments from unrelated worlds also shrink the prompt but lower satisfaction (z −2.12), showing that read content also matters.

We also add three new analyses to better distinguish lost rationale from changes in instruction relevance:

  • Age, net of file size. Deletion becomes rarer with age (−0.059 per commit [−0.071, −0.047]); fading relevance due to drift would of course make it more common.
  • Authorship. Authors delete their own instructions 1.3× more than other maintainers; in the same edit and section, they delete their own 1.2× more than others'. This is consistent with authorship driving retained rationale and further driving deletion (controlling for ownership).
  • Prevalence. ~1% of real instructions state a reason, so §4's comment-free prompts are the real-world case; net of size and age, files that record them are wholesale-rewritten 0.67× as often, consistent with rationale enabling targeted pruning.

That said, you're right that §3 alone can't definitively isolate rationale and we appreciate you pressing on it. Our aim there was to characterize the empirics, rule out commonly suspected rivals and point to lost rationale; we will soften its language to say exactly that.

Verified cover (Q1, W1). Thank you for pushing us to check the IFEval-provided reference prompt; it strengthens the result.

We took IFEval's own instructions as D* and tested satisfaction on a leave-one-out basis: dropping one usually breaks its requirement (satisfaction −0.664), but 17.5% are ones that today's executor agent follows by default. Removing those gives the verified cover Dⱽ, which works just as well as D* (0.872 vs. 0.870, table above).

D* is thus a worthwhile target and practical::commit matches the other agentic arms' satisfaction at the smallest size. However, no arm reaches D* satisfaction, not even an oracle arm seeing the full history, so we will describe §4.2's result as convergence in size and satisfaction bounded by the limits of current agentic learning capabilities.

We hope the adapted baselines and the matched ladder answer your questions. We found them quite insightful and even beyond ICLR we believe your feedback materially strengthens the manuscript — thank you.

Thanks, this is very helpful; I plan to increase my rating to 3.

Two quick clarifications:

  1. Can you confirm that control::write−onlycontrol::write-only and practical::commitpractical::commit use exactly the same retention/deletion guidance, with the only difference being whether the maintainer reads its own comments? If so, that goes a long way toward answering my Q2.

  2. Also, the table shows 0.453 vs. 0.443 satisfaction, so I would describe this as a size reduction at similar satisfaction, not improved instruction following. And with z=−1.06z = -1.06, I would be careful about the “about a third of the full effect” claim.

AuthorsOpenReview ↗

Same guidance, one difference; wording corrected

Thank you; both points are right, and the revision adopts them.

  1. Yes, exactly. WriteOnly and PromptComments (formerly control::write-only, practical::commit) share the system prompt and add/delete instructions character for character, and both comment every edit; only PromptComments reads its comments back. At three seeds (64 worlds, T=15), not reading them forfeits 53.6% of the size gain (z = 2.29) at similar satisfaction (41.9% vs. 43.0%).
  2. Agreed on both. §4.2 now claims a smaller prompt at similar satisfaction, with no arm near the reference prompt's 85.6%. We claim better instruction-following only twice: in real repositories, comment-enabled pruning reaches 44.2% against 37.6% uncommented (p = 0.003) and 37.5% for the actual rewrites (p = 0.001; 680 pairs, paired sign-flip); and the strongest maintainer gains 8.3 ± 7.7 points (one seed; Table 2). Writing alone is z = 1.82 against None at three seeds, so we no longer report its share.

We're grateful for how closely you've read the paper. If anything else stands between it and a stronger recommendation, we'd be glad to address it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New results: comments beat real rewrites on GitHub; paper rewritten for clarity; three-seed comment-access ladder

We thank all three reviewers. Two results are new since our individual replies; the rest are now in the revised PDF, which includes several changes to improve readability.

To make changes easy to find, we've highlighted them in a different color per reviewer and named them in the margin: red for EoYr, yellow for kgwL and blue for QxZF. Text that answers two reviewers mixes their colors, e.g. green answers kgwL and QxZF.

1. Prompt comments improve instruction-following on real GitHub repositories (kgwL, QxZF; §4.4). Files pruned by an agentic maintainer with prompt comments satisfy 6.7 points more of real coding instructions than the maintainers' own rewrites (p = 0.001), and 7.2 points more than rewrites by human maintainers (p = 0.031):

PromptPruned bySize ↓Satisfaction ↑
Original filenobody (before rewrite)111.442.1%
Real rewriterepository maintainers27.437.5%
Noneagent, no comments104.737.6%
PromptComments (ours)agent, with comments92.444.2%

We took real context files that their maintainers later mass-rewrote and drew 680 task–instruction pairs from the commits after each rewrite. We rebuilt every instruction's comment from history strictly before the rewrite (no lookahead), and an agentic maintainer pruned each file once, with or without those comments. Two reasoning judges (gpt-5.6-luna, gemini-3.8-flash) scored every version on all pairs.

The real rewrites cut far more instructions but lost 4.5 points of instruction-following against the original prompt; we think the wrongly cut instructions come back as failures and drive regrowth.

2. Presentation (EoYr, kgwL). We rewrote the paper to be easier to read:

  • §4 proof builds up real-world complexity. Inverted IFEval, where the best prompt is known (§4.1–4.2) → real-world prompts and coding conventions (§4.3, Table 3) → real repositories with mass rewrites (§4.4, Table 4).
  • §3.4 rules out observational rivals in groups. Staleness and relevance drift predict the wrong sign; content fragility and effort dilution have the correct sign but show too small an effect. Additional analyses add color.
  • Appendix: checkable, organized claims. One figure now holds every prompt (Fig. 8, formerly Figs. 8–10), so readers can see the arms differ only in their comments. New subsections cover recorded latent reasoning (A.5), the verified cover (B.3), results by final prompt size (B.7), unprompted rewrites (B.9), and the real-repository experiment (C).
  • Simpler, cleaner naming. Arms are named for what they do (None, ShuffledNoise, PromptComments). Table 1 lists what each writes and reads. Agentic prompts, latent reasoning and scoring mechanics keep one name throughout.

3. Comment-access ladder at three seeds (EoYr, QxZF; §4.2). Every arm shares one system prompt and the same feedback; only the comment channel varies (64 IFEval worlds, 3 seeds, T=15). These numbers supersede the one-seed ones in our replies:

ArmSystem promptMaintainer writesMaintainer readsExcess vs. D_V ↓Satisfaction ↑
Nonesharednothingnothing+56.0% ± 23.340.2%
WriteOnlyshared + comment formatown commentsnothing+35.9% ± 21.441.9%
ShuffledNoiseshared + comment formatown commentsanother world's comments−0.2% ± 10.837.8%
PromptComments (ours)shared + comment formatown commentsown comments+12.7% ± 15.843.0%

The shared prompt carries the same add/delete guidance for every arm and none of Fig. 9's comment-specific deletion rules.

Maintainers that write comments but never read them forfeit 53.6% of the gain (z = 2.29). Another world's comments shrink the prompt as far, but cost 5.1 points of satisfaction (z = 3.05): the content of the comments, not their presence, keeps the right instructions.

Also in the revision, as reported in our individual replies:

  • Verified cover (§4.1): 17.3% of IFEval's reference instructions are followed unprompted; the verified cover without them satisfies 85.4% against the reference's 85.6%.
  • Coding tasks and baselines (§4.3): on CodeIF and WildIFEval, comments end smallest (−32.7% and +14.5% excess) against ACE-style (+29.9%, +95.7%) and Dynamic Cheatsheet (+101.0%, +305.7%) at similar satisfaction.
  • Repository study (Appendices A–B): the age slope survives file size (−0.059 ± 0.012 per commit); authors delete their own instructions 1.33× as often [1.09, 1.63]; files that record a rationale are rewritten 0.67× as often [0.57, 0.79]; and uncommented maintainers rewrite 0.41 times per world against 0.14 with comments.
  • Framing (§3, §4.2, Limitations): revised per the reviewers' corrections.

Whatever happens at ICLR, your feedback has been tremendously helpful and has made this a much stronger paper. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.