- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines why context files like CLAUDE.md grow nearly monotonically. The authors hypothesize that the rationale behind each instruction is lost over time, so maintainers cannot safely delete instructions. They formalize this process as a survival model. The authors analyze instruction lifetimes across repositories and identify a growth–rewrite–regrowth pattern and a declining deletion hazard. In their experiments, they invert IFEval and show that adding comments helps keep the prompt near its optimal size. On WildIFEval with injected distractor instructions, such comments also improve instruction-following.
Critical strengths
-
The question of why CLAUDE.md files keep growing is interesting and timely, and the paper looks at how individual instructions survive and die across almost 250K instruction lifetimes from about 1,900 repos.
-
Inverting IFEval is a useful way to conduct experiments. It gives a reference prompt to compare against, which makes it easier to measure both how bloated a prompt is and how well it works.
Critical weaknesses
-
It’s not clear the gains come from the comments. The treatment arm also receives explicit deletion guidance (Figure 9) that the control and placebo arms lack. The ablations in Table 11 all keep this guidance, so they cannot isolate its effect. A control with the guidance but no informative comments would separate the two.
-
The repository evidence makes imperfect recall plausible but does not establish it as the root cause claimed in Section 3. The authors acknowledge the evidence is correlational, and rationale loss is never measured directly. Partial effort dilution also remains open: it is unclear whether the age effect persists after controlling for time-varying file size.
Questions for authors
-
Besides writing comments, the treatment arm also seems to get explicit deletion rules (Figure 9) that the placebo arm doesn’t. Do you have results with the rules but no comments? If the gain mostly disappears, that would change my view of what’s driving it.
-
Does the drop in deletion rate with age still hold after controlling for file size? This would help rule out the simpler explanation that instructions in longer files are just less likely to be deleted.
2 comments
Thank you for the additional experiments and analyses. Overall, I think this paper addresses a narrow but interesting research question and provides a thoughtful analysis.
The new ablations largely address my first concern, and the additional analysis controlling for file size addresses my second concern. I also appreciate the authors' clarification regarding the causal claims about imperfect recall.
I encourage the authors to incorporate these additional results and clarifications into the revised version. I am inclined to raise my rating to weak accept.
Matched control: reading comments carries 2/3; age survives size
Thank you for a careful review, and especially for naming the control that would change your view; we ran it, an even stronger control, the file-size re-fit and other analyses to get to your deeper questions.
Causally isolating rationale loss (Q1, W1). We agree that varying the prompt inadvertently introduced a confound. A new control,
control::write-only, keeps Fig. 9's rules but never shows the maintainer its own comments; reading them still strips 86% of its remaining excess size (0.208 → 0.029; 64 IFEval worlds, T=15). However, we found this comparison awkward: Fig. 9's add/delete rules act on what the comments record, so without readable comments they are ill-defined.We therefore built a new ablation ladder in which every arm shares one system prompt and only the comment channel varies (64 worlds, one seed; fresh seeds at camera-ready):
control::nonecontrol::write-onlycontrol::noise-onlypractical::commit(ours)Bold: best in its column and beats the runner-up at equal satisfaction (p < 0.05, two-sided, paired over worlds). Dⱽ drops the 17.5% of IFEval's instructions today's executor follows by default (leave-one-out on D*).
With the system prompt held fixed, reading informative comments carries two-thirds of the gain:
practical::commitends 0.333 smaller thancontrol::write-only(z −2.34) and 0.497 smaller thancontrol::none(z −3.47), at indistinguishable satisfaction. Comments from unrelated worlds also shrink the prompt but cost satisfaction (−0.065 vs. ours, z −2.12), so comment content matters beyond volume. We will add these arms to Table 11 and replace the fixed-string placebo, which bundled comment content with the rules.Observationally isolating rationale loss (Q2, W2). Beyond the requested file-size control, we add five analyses:
controlmaintainer agents rewrite 0.41 times per world (mostly after growth, as in the corpus), against 0.14 for ours (Δ +0.27 [+0.11, +0.45]), consistent with lost rationale driving the growth–rewrite cycle.That said, you're right that §3 alone can't establish rationale loss as the root cause. Our aim there was to characterize the empirics, rule out common rivals and point to lost rationale; §4 carries the causal claim. We will soften §3 to say exactly that.
Thank you again: naming the control you wanted made the paper's central claim sharper and easier to test. We hope the matched ladder and new observational analyses address your main concerns — we'd be glad to run anything else that would help.