- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates Default Capture, a failure mode where language models replace externally observed values with default values stored in parametric memory. A controlled real-name vs. made-up-name comparison is introduced to isolate this effect across models and input settings. The paper further proposes Name-Conditioned Self-Distillation (NCSD) to mitigate Default Capture while preserving default-value knowledge. Experiments demonstrate the prevalence of this issue, particularly in tool-output settings, and the effectiveness of NCSD.
Critical strengths
-
Interesting and practically relevant failure mode. The paper identifies a meaningful failure mode where parametric knowledge overrides observed values, particularly relevant to coding agents that rely on environment observations.
-
Careful experimental design and analysis. The real-name vs. made-up-name comparison effectively isolates prior-knowledge interference. Experiments across six models, 48 services, and multiple settings provide convincing empirical evidence.
Critical weaknesses
- Limited generalization beyond default-value copying.
The experiments primarily focus on service ports with well-known defaults. It remains unclear whether Default Capture generalizes to more complex coding scenarios involving API parameters, version dependencies, or dynamic configurations.
- Limited validation in realistic coding agent workflows.
Most experiments involve generating a single value in prefilled code rather than realistic agent workflows. Although Appendix U includes agent-loop experiments, only two models are evaluated, with substantially different results. The impact on end-to-end task success remains unclear.
- Limited methodological novelty of NCSD.
NCSD closely resembles existing consistency training or context distillation approaches. Its main modifications involve constructing teacher targets using made-up names and applying KL anchoring, making its standalone methodological contribution relatively limited.
Questions for authors
-
Generalization. Does Default Capture extend beyond service ports to more complex coding scenarios, such as API versions or dynamic configurations? Does it require a widely known default value?
-
Realistic agent workflows. Does Default Capture remain significant when coding agents can verify and correct configurations through execution feedback? Can NCSD improve end-to-end task success rates?
-
Methodological novelty and knowledge preservation. What distinguishes NCSD from existing consistency training or context distillation methods? Does it genuinely improve the model's ability to balance contextual observations and parametric knowledge, rather than simply increasing reliance on context?