- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks into the question about whether LLMs know which corrections go to persistent memory. Extensive experiments are conducted to search for the answers. And there are some helpful conclusions, e.g., a valid correction does not determine the scope of the rule an assistant will save, changing the user tone can substantially change the rule, explicitly eliciting a recurrence judgment can increase rule writing immediately after a one-off judgement, and so on.
Critical strengths
- The task and topic itself is timely, interesting and meaningful.
- The experiments are extensive.
- The explicit codebase-convention arm is a useful positive control.
- The paper separates pre-specified from exploratory tests, stating the limitations honestly.
Critical weaknesses
- The protocol tests only two synthetic error types, this cannot fully represent the wider types of real-world bugs.
- The experiments are mostly limited to single-function python code generation tasks from HumanEval, many other scenarios are not considered, e.g., non-coding conversations, complex engineering projects, etc.
- Only commercial models are considered, such as Claude and GPT family models, it can be also valuable to access the open-sourced models.
- There are many other factors or parameters which might also influence the performance, such as whether open reasoning, the temperature, those details are not clearly demonstrated or controlled.
Questions for authors
See above.
Thank you for the review. Two of the four weaknesses concern information that was in the submission but easy to miss; The other two are real limits, which we state.
Weakness 3 (only commercial models). The paper evaluates eight open-weight models next to five Claude and two GPT-6 models: Qwen3-14B, gpt-oss-20b, Gemma-3-12B, Gemma-2-9B, DeepSeek-R1-14B, Llama-3.1-8B, Qwen2.5-7B and Qwen2.5-Coder-7B (Table 2, lower block; Appendix F). The abstract did not say so; it now names the three groups. The open-weight models differ from one another as much as the commercial ones do. The two Gemma models write a rule after most typo corrections (37 and 30 of 40 sessions), DeepSeek-R1-14B in 23 of 39, and the other five rarely. Asking for a judgment first raises typo rules in six of the seven open-weight models on which we ran it, significantly in four (Table 15).
Weakness 4 (reasoning and temperature). The settings were in Section 3; a new Table 7 lists the identifier, sampling and reasoning setting of every model. Claude Sonnet 5, Opus 5, Opus 5.5 and the GPT-6 models do not accept sampling parameters and run at the API defaults; all other models are decoded greedily. The main contrast does not follow these settings. Sonnet 4.5 and Haiku 4.5 share the same settings (greedy, no thinking) and write a typo rule in 40 and 3 of 40 sessions. Among the models run at API defaults with reasoning, Sonnet 5 and Opus 5 write it in 40 sessions and GPT-6 Sol and Luna in 0 and 1. We did not vary reasoning or temperature within a model, and the paper says so.
Weaknesses 1 and 2 (two synthetic error types, single-function tasks). We agree, and the limitations say so. The design is deliberate. An injected misspelling that raises NameError under the official tests gives every model the same verifiable correction, and it is the clearest case of an error a user would call a one-off. Holding the first answer fixed removes first-turn differences between models. The cost is coverage: the paper describes SCOPE as a controlled coding setting, not a sample of real use, and makes no claim about other error types or multi-file tasks.