- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies why LLMs struggle with context learning and identifies specification acquisition as an important bottleneck, where the models fail to recover task-specific rules and constraints from context. The authors propose PSCI, a training-free pipeline that extracts specifications before generation and uses them for verification and repair. PSCI improves GPT-5.1's strict pass rate on CL-Bench from 22.55% to 28.14%, and consistent improves on two other models, including Qwen3.5-27B and Gemini 3 Pro.
Critical strengths
- Valuable empirical insights: the distinction between content and specification acquisition provides a useful perspective on context learning failures. The finding that models often recover the necessary information but fail to follow context-specific requirements is interesting.
- Thorough empirical evaluation: the paper analyzes over 31K rubric items, evaluates 12 existing methods and includes extensive ablations and human validation.
- Simple and effective mitigation method: PSCI is simple and intuitive and the authors have demonstrated consistent improvement with different models.
Critical weaknesses
- The main contribution appears to lie in the empirical findings rather than the proposed method. PSCI is a relatively straightforward combination of specification extraction, generation, verification, and repair. The overall architecture closely resembles existing generate–verify–repair frameworks. Some relevant work such as LLM-Modulo [1] is not discussed in the paper.
- Insufficient analysis of specification discovery failures. Although PSCI achieves 94.4% specification coverage, only 62.5% is strongly covered, while 31.9% is partially covered. The paper provides limited analysis of which types of specifications are missed or incompletely extracted, why these failures occur, and how they could be addressed.
- Unclear attribution of remaining failures. The authors attribute 83% of remaining rubric failures to specification enforcement, but it is unclear whether this includes partially covered specifications. For example, a contract may identify a required warning but miss its exact format, leading to failure even with correct enforcement. A finer-grained analysis distinguishing incomplete specification discovery from genuine enforcement failures would better support this conclusion.
[1] Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in llm-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).
Questions for authors
- Given that 71.6% of specifications are explicitly stated in the context, could a simpler specification-focused retrieval or extraction baseline achieve similar improvements to PSCI?
4 comments
Rebuttal Part 2/2
W3. Unclear attribution of remaining failures. The authors attribute 83% of remaining rubric failures to specification enforcement, but it is unclear whether this includes partially covered specifications. For example, a contract may identify a required warning but miss its exact format, leading to failure even with correct enforcement. A finer-grained analysis distinguishing incomplete specification discovery from genuine enforcement failures would better support this conclusion.
R3. This is a valid point, and we thank the reviewer for the concrete example. The 83% figure includes partially covered items, so we have added the finer-grained analysis you suggested to the updated draft (Sec. 6.2; Table 14, Appendix M.3). Of the 5,965 rubric items still failed by PSCI:
| Cause of the remaining failure | Share |
|---|---|
| Enforcement (the contract states the obligation, but the answer does not satisfy it) | about 58% |
| Discovery (the obligation is missing or only partly captured in the contract) | about 40% |
| Contradiction | 2% |
Here, partially covered items are split between the two causes by comparing their pass rate with that of strongly covered items (details in Appendix M.3). Enforcement remains the largest bottleneck, with incomplete discovery of exact scope and format second, consistent with R2. We have revised Sec. 6.2 accordingly.
Q1. Given that 71.6% of specifications are explicitly stated in the context, could a simpler specification-focused retrieval or extraction baseline achieve similar improvements to PSCI?
A1. Thank you for the practical suggestion. We have now conducted additional experiments on both, including a new specification-retrieval baseline (Sec. 5.2, Table 6; Appendix D and Appendix L in the updated draft). 1) Extraction: the first two stages of PSCI alone (contract induction and contract-guided answering, without checking and repair; "Baseline+SI") give 26.09 on CL-Bench (+3.54 pp, 63% of PSCI's gain). Specification extraction, the core step of PSCI, already captures most of the effect, which supports the diagnosis, and checking and repair add the rest. 2) Retrieval (new): this baseline selects context sentences with obligation cues ("must", "only", "never", "format", "unless", …), ranks them by cue density and query overlap, and appends them to the prompt:
| Strict pass (%) | CL-Bench, Claude Haiku 5.5 (200 tasks) | Life, Claude Haiku 5.5 | Life, Gemini 3.1 Pro | Life, Gemini 4 Argon |
|---|---|---|---|---|
| Direct baseline | 24.5 | 9.5 | 13.2 | 19.3 |
| Specification retrieval (new) | 27.5 | 10.1 | 14.0 | 19.3 |
| PSCI | 30.0 | 12.9 | 17.6 | 26.3 |
As shown in the table, retrieval recovers about half of the gain on CL-Bench, where rules are mostly stated explicitly, but almost none on CL-bench Life (+0.6 / +0.8 / +0.0 pp vs. +3.4 / +4.4 / +7.0 pp for PSCI; pooled p < 0.001), where obligations are spread across chats, notes, and logs. This highlights that being explicit in the context is not sufficient: retrieval surfaces rule text, but the model still has to turn scattered rules into obligations for this request and check its answer against them. Obligations implied by patterns and examples (about a quarter of specification items; R2) cannot be surfaced lexically at all.
We thank you again for these suggestions, which made our analysis of discovery and enforcement more precise. All new analyses and results are in the updated draft, and we sincerely hope they address your concerns.
Follow-up
Dear Reviewer 8XDB,
We would like to sincerely thank you for your efforts and comments in reviewing our submission. As we approach the end of the discussion period, we would like to ask you to kindly confirm that you have reviewed our rebuttal and let us know if there are any remaining concerns regarding our work. As mentioned in our previous response, in the updated draft (new content in blue) we have cited and discussed LLM-Modulo (Sec. 2, Appendix F.3), added the analysis of which specifications the contract misses (Sec. 6.1, Table 13) and a finer attribution of the remaining failures (Sec. 6.2, Table 14), and added the specification extraction and retrieval baselines you suggested (Sec. 5.2, Table 6). We hope that these responses effectively address your questions. If you have any further concerns or questions, please do not hesitate to let us know.
Thank you again for your help and time in reviewing our submission!
Authors
Thank you for the thorough rebuttal and additional experiments. I particularly appreciate the new analyses of specification discovery and the refined attribution of remaining failures, which directly address my concerns. The additional baselines and benchmark results also strengthen the empirical contributions.
I continue to find the empirical insights valuable, although I believe the methodological novelty remains relatively limited. Given that my original score of 3 already reflects a positive assessment of the work, I will maintain my original rating.
Rebuttal Part 1/2
We genuinely appreciate your positive and insightful review, and we are grateful that you found the distinction between content and specification acquisition "a useful perspective on context learning failures", our evaluation "thorough", and PSCI "simple and intuitive" with "consistent improvement". We have uploaded an updated draft (new content in blue), and below we address your concerns and question in detail.
R1. Thank you for the feedback and for pointing out LLM-Modulo [1]. We have now cited it and discussed it carefully in the updated draft (Sec. 2 and Appendix F.3). We agree with the reviewer that our main contribution is the empirical diagnosis; PSCI is deliberately minimal (Sec. 4), so that its gains validate the diagnosis rather than engineering.
We would like to respectfully note that PSCI differs from existing generate–verify–repair frameworks in one key aspect. LLM-Modulo pairs LLM generation with external, sound critics derived from a domain model. In context learning there is no external verifier: the "domain model" exists only inside the prompt, and most of its obligations are never stated in the query (76.7%, Sec. 3.4). The test specification itself must therefore first be induced from the context, and our ablations show that this induction step, not the loop, carries the gain: the same check-and-repair loop with no contract (A3, 24.32), a generic contract (A12, 24.00), or another task's contract (A10, 22.00) falls far short of PSCI (28.14; Table 4).
R2. This is a good suggestion. We have added this analysis to the updated draft (Sec. 6.1, new paragraph "What discovery misses"; Table 13, Appendix M.2). We break down contract coverage of specification items by how the obligation is conveyed in the context:
As shown in the table, 1) which types and why: discovery fails mostly on obligations that are demonstrated rather than stated. Under partial coverage, the contract captures the intent but not the exact scope or form, e.g., "retry exactly once on timeout, then escalate to the Lab Tech" or "use exactly the statuses PASS / Fail / Required Adjustment". 2) How to address them: the results point to a concrete direction, an explicit example → rule abstraction step during discovery (Appendix M.2).
(continued in Part 2/2)
[1] Kambhampati et al., LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks, ICML 2024.