acceptodds
Under review as a conference paper at ICLR 2027

Agentic Context Learning with Self-Discovered Specification

Abstract

Context learning is an emerging inference-time task where LLMs learn and apply novel, task-specific knowledge from complex contexts. Unlike demonstration-based in-context learning, which emphasizes inferring task mappings from examples, context learning requires acquiring genuinely new knowledge that may be absent from pre-training—a capability that remains challenging even for frontier models today. In this work, we conduct a comprehensive empirical study to understand why models struggle with this capability. A natural hypothesis is that failures stem from content access; however, across twelve retrieval, reflection, and verification baselines on CL-Bench, we find limited gains over direct full-context prompting. Instead, our analysis reveals a key insight: recovering relevant content is insufficient; models must also infer local specifications—domain-specific formats, local rules, and completeness conditions—that are often unstated in the query but recoverable from the context. Our rubric-level analysis finds that specification acquisition is evaluated more than twice as often as content acquisition. Most specifications are unstated in the user query, yet nearly all are traceable to the context, indicating learnable obligations rather than hidden requirements. To validate this diagnosis, we introduce PSCI (private specification-contract induction), a deliberately simple intervention that extracts local specifications and enforces them through adversarial checking and repair. PSCI improves GPT-5.1 from 22.55% to a state-of-the-art 28.14% task success on CL-Bench, with consistent gains on Qwen3.5-27B and Gemini 3 Pro. Extensive ablations further isolate the role of task-specific specifications. Together, our findings suggest that effective context learning hinges not only on content acquisition but also on specification acquisition.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context Learning, Large Language Models, Long-Context Reasoning, CL-Bench, Agentic Reasoning, Agentic Context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8XDBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why LLMs struggle with context learning and identifies specification acquisition as an important bottleneck, where the models fail to recover task-specific rules and constraints from context. The authors propose PSCI, a training-free pipeline that extracts specifications before generation and uses them for verification and repair. PSCI improves GPT-5.1's strict pass rate on CL-Bench from 22.55% to 28.14%, and consistent improves on two other models, including Qwen3.5-27B and Gemini 3 Pro.

Critical strengths
  • Valuable empirical insights: the distinction between content and specification acquisition provides a useful perspective on context learning failures. The finding that models often recover the necessary information but fail to follow context-specific requirements is interesting.
  • Thorough empirical evaluation: the paper analyzes over 31K rubric items, evaluates 12 existing methods and includes extensive ablations and human validation.
  • Simple and effective mitigation method: PSCI is simple and intuitive and the authors have demonstrated consistent improvement with different models.
Critical weaknesses
  • The main contribution appears to lie in the empirical findings rather than the proposed method. PSCI is a relatively straightforward combination of specification extraction, generation, verification, and repair. The overall architecture closely resembles existing generate–verify–repair frameworks. Some relevant work such as LLM-Modulo [1] is not discussed in the paper.
  • Insufficient analysis of specification discovery failures. Although PSCI achieves 94.4% specification coverage, only 62.5% is strongly covered, while 31.9% is partially covered. The paper provides limited analysis of which types of specifications are missed or incompletely extracted, why these failures occur, and how they could be addressed.
  • Unclear attribution of remaining failures. The authors attribute 83% of remaining rubric failures to specification enforcement, but it is unclear whether this includes partially covered specifications. For example, a contract may identify a required warning but miss its exact format, leading to failure even with correct enforcement. A finer-grained analysis distinguishing incomplete specification discovery from genuine enforcement failures would better support this conclusion.

[1] Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in llm-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).

Questions for authors
  • Given that 71.6% of specifications are explicitly stated in the context, could a simpler specification-focused retrieval or extraction baseline achieve similar improvements to PSCI?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

We genuinely appreciate your positive and insightful review, and we are grateful that you found the distinction between content and specification acquisition "a useful perspective on context learning failures", our evaluation "thorough", and PSCI "simple and intuitive" with "consistent improvement". We have uploaded an updated draft (new content in blue), and below we address your concerns and question in detail.

W1. The main contribution appears to lie in the empirical findings rather than the proposed method. PSCI is a relatively straightforward combination of specification extraction, generation, verification, and repair. The overall architecture closely resembles existing generate–verify–repair frameworks. Some relevant work such as LLM-Modulo [1] is not discussed in the paper.

R1. Thank you for the feedback and for pointing out LLM-Modulo [1]. We have now cited it and discussed it carefully in the updated draft (Sec. 2 and Appendix F.3). We agree with the reviewer that our main contribution is the empirical diagnosis; PSCI is deliberately minimal (Sec. 4), so that its gains validate the diagnosis rather than engineering.

We would like to respectfully note that PSCI differs from existing generate–verify–repair frameworks in one key aspect. LLM-Modulo pairs LLM generation with external, sound critics derived from a domain model. In context learning there is no external verifier: the "domain model" exists only inside the prompt, and most of its obligations are never stated in the query (76.7%, Sec. 3.4). The test specification itself must therefore first be induced from the context, and our ablations show that this induction step, not the loop, carries the gain: the same check-and-repair loop with no contract (A3, 24.32), a generic contract (A12, 24.00), or another task's contract (A10, 22.00) falls far short of PSCI (28.14; Table 4).

W2. Insufficient analysis of specification discovery failures. Although PSCI achieves 94.4% specification coverage, only 62.5% is strongly covered, while 31.9% is partially covered. The paper provides limited analysis of which types of specifications are missed or incompletely extracted, why these failures occur, and how they could be addressed.

R2. This is a good suggestion. We have added this analysis to the updated draft (Sec. 6.1, new paragraph "What discovery misses"; Table 13, Appendix M.2). We break down contract coverage of specification items by how the obligation is conveyed in the context:

How the specification appears in contextStrongPartialMissed
Explicitly declared rule73.3%23.6%2.6%
Implied by a validation pattern or schema40.1%51.5%8.0%
Implied by a worked example26.4%64.4%9.3%
Implied by repeated convention16.0%71.7%12.3%

As shown in the table, 1) which types and why: discovery fails mostly on obligations that are demonstrated rather than stated. Under partial coverage, the contract captures the intent but not the exact scope or form, e.g., "retry exactly once on timeout, then escalate to the Lab Tech" or "use exactly the statuses PASS / Fail / Required Adjustment". 2) How to address them: the results point to a concrete direction, an explicit example → rule abstraction step during discovery (Appendix M.2).

(continued in Part 2/2)

[1] Kambhampati et al., LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks, ICML 2024.

AuthorsOpenReview ↗

Rebuttal Part 2/2

W3. Unclear attribution of remaining failures. The authors attribute 83% of remaining rubric failures to specification enforcement, but it is unclear whether this includes partially covered specifications. For example, a contract may identify a required warning but miss its exact format, leading to failure even with correct enforcement. A finer-grained analysis distinguishing incomplete specification discovery from genuine enforcement failures would better support this conclusion.

R3. This is a valid point, and we thank the reviewer for the concrete example. The 83% figure includes partially covered items, so we have added the finer-grained analysis you suggested to the updated draft (Sec. 6.2; Table 14, Appendix M.3). Of the 5,965 rubric items still failed by PSCI:

Cause of the remaining failureShare
Enforcement (the contract states the obligation, but the answer does not satisfy it)about 58%
Discovery (the obligation is missing or only partly captured in the contract)about 40%
Contradiction2%

Here, partially covered items are split between the two causes by comparing their pass rate with that of strongly covered items (details in Appendix M.3). Enforcement remains the largest bottleneck, with incomplete discovery of exact scope and format second, consistent with R2. We have revised Sec. 6.2 accordingly.

Q1. Given that 71.6% of specifications are explicitly stated in the context, could a simpler specification-focused retrieval or extraction baseline achieve similar improvements to PSCI?

A1. Thank you for the practical suggestion. We have now conducted additional experiments on both, including a new specification-retrieval baseline (Sec. 5.2, Table 6; Appendix D and Appendix L in the updated draft). 1) Extraction: the first two stages of PSCI alone (contract induction and contract-guided answering, without checking and repair; "Baseline+SI") give 26.09 on CL-Bench (+3.54 pp, 63% of PSCI's gain). Specification extraction, the core step of PSCI, already captures most of the effect, which supports the diagnosis, and checking and repair add the rest. 2) Retrieval (new): this baseline selects context sentences with obligation cues ("must", "only", "never", "format", "unless", …), ranks them by cue density and query overlap, and appends them to the prompt:

Strict pass (%)CL-Bench, Claude Haiku 5.5 (200 tasks)Life, Claude Haiku 5.5Life, Gemini 3.1 ProLife, Gemini 4 Argon
Direct baseline24.59.513.219.3
Specification retrieval (new)27.510.114.019.3
PSCI30.012.917.626.3

As shown in the table, retrieval recovers about half of the gain on CL-Bench, where rules are mostly stated explicitly, but almost none on CL-bench Life (+0.6 / +0.8 / +0.0 pp vs. +3.4 / +4.4 / +7.0 pp for PSCI; pooled p < 0.001), where obligations are spread across chats, notes, and logs. This highlights that being explicit in the context is not sufficient: retrieval surfaces rule text, but the model still has to turn scattered rules into obligations for this request and check its answer against them. Obligations implied by patterns and examples (about a quarter of specification items; R2) cannot be surfaced lexically at all.

We thank you again for these suggestions, which made our analysis of discovery and enforcement more precise. All new analyses and results are in the updated draft, and we sincerely hope they address your concerns.

Dear Reviewer 8XDB,

We would like to sincerely thank you for your efforts and comments in reviewing our submission. As we approach the end of the discussion period, we would like to ask you to kindly confirm that you have reviewed our rebuttal and let us know if there are any remaining concerns regarding our work. As mentioned in our previous response, in the updated draft (new content in blue) we have cited and discussed LLM-Modulo (Sec. 2, Appendix F.3), added the analysis of which specifications the contract misses (Sec. 6.1, Table 13) and a finer attribution of the remaining failures (Sec. 6.2, Table 14), and added the specification extraction and retrieval baselines you suggested (Sec. 5.2, Table 6). We hope that these responses effectively address your questions. If you have any further concerns or questions, please do not hesitate to let us know.

Thank you again for your help and time in reviewing our submission!

Authors

Thank you for the thorough rebuttal and additional experiments. I particularly appreciate the new analyses of specification discovery and the refined attribution of remaining failures, which directly address my concerns. The additional baselines and benchmark results also strengthen the empirical contributions.

I continue to find the empirical insights valuable, although I believe the methodological novelty remains relatively limited. Given that my original score of 3 already reflects a positive assessment of the work, I will maintain my original rating.

Reviewer_bePTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why language models struggle with context learning on CL-Bench. It argues that retrieving answer-relevant content is insufficient: models must also acquire local specifications—requirements recoverable from the context but often unstated in the query. The authors introduce PSCI, which induces a private specification contract before generation and uses it to guide answer generation, checking, and repair. On CL-Bench, PSCI raises GPT-5.1 strict task success from 22.55% to 28.14%; results on two other models and ablations support the value of context-grounded specification discovery and enforcement.

Critical strengths
  • A useful diagnosis of context-learning failures. The paper shows that many unmet CL-Bench requirements are local specifications: they are unstated in the query but recoverable from the context. This helps explain why retrieving answer-relevant content alone may be insufficient.

  • Strong empirical support on CL-Bench. PSCI improves GPT-5.1 strict task success from 22.55% to 28.14%, with gains on two other models. Ablations of contract quality, generation order, and checking provide evidence for the role of specification acquisition beyond the overall performance gain.

Critical weaknesses
  • The source of the gains and the method’s framing could be clearer. The ablations support the value of the specification contract, but PSCI also changes when specifications are induced, how answers are generated, and how they are checked. Clarifying the contribution of each component—and what “agentic” specifically refers to—would help readers assess the paper’s central contribution.

  • Some experimental reporting needs clarification. TICK/STICK achieves 23.00% strict pass in Table 3 but 24.00% in Table 6. The paper also uses 31,592 CL-Bench rubric items, compared with 31,607 reported in the original benchmark paper. If these differences reflect dataset versions or reporting conventions, stating so would make the results easier to verify.

Questions for authors
  • Clarification — method framing: What does “agentic” refer to in PSCI beyond its fixed induction–generation–checking–repair pipeline? This would help clarify the scope of the paper’s methodological contribution.

  • Clarification — source of the gains: How do the experiments distinguish the benefit of a context-grounded specification contract from that of a pre-generation, task-specific checklist? In particular, how closely do the adapted checklist/rubric baselines retain the mechanisms of the cited methods? This affects how I interpret the comparative results.

  • Clarification — reporting: Could the authors reconcile the TICK/STICK strict-pass rates in Tables 3 and 6 (23.00% vs. 24.00%) and explain why the analysis uses 31,592 rubric items rather than the 31,607 reported by CL-Bench? The answer would help establish the evaluation and data-counting protocol.

5 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

Thank you for your detailed review and your constructive, actionable feedback. We are encouraged that you found our work "a useful diagnosis of context-learning failures" with "strong empirical support on CL-Bench". We have uploaded an updated draft (new content in blue) and address your concerns individually in detail below.

W1. The source of the gains and the method's framing could be clearer. The ablations support the value of the specification contract, but PSCI also changes when specifications are induced, how answers are generated, and how they are checked. Clarifying the contribution of each component ...

R1.1 Thank you for the thoughtful question. Our ablations (Table 4) vary one factor at a time. Following your suggestion, we regroup them along the three axes you name, and we have added this view to the updated draft (Table 12, Appendix M.1; referenced from Sec. 5.3). All numbers are strict pass with GPT-5.1 on the full benchmark:

Factor varied (others fixed)VariantStrict pass
—Direct baseline22.55
What is in the contractshuffled / generic / zero-shot / PSCI22.00 / 24.00 / 26.67 / 28.14
When it is inducedanswer-first (A6) / contract-first23.67 / 28.14
How it is enforcednone (A1) / no contract (A3) / no checker (A4) / PSCI26.09 / 24.32 / 26.30 / 28.14
Extra compute onlyspec-free critique (A13)22.67

As shown in the table, each component contributes. 1) Contract induction plus contract-guided generation gives +3.54 pp (63% of the gain), and explicit checking and repair add +2.05 pp. 2) The contract helps only when it is grounded in the current context: a generic checklist (24.00) or another task's contract (22.00) gives little or nothing in the same pipeline with the same compute. 3) The same contract induced after generation loses 4.47 pp. The gain therefore comes from what goes into the contract and when it is used, not from having a checklist step or extra compute.

W1. ... and what "agentic" specifically refers to—would help readers assess the paper's central contribution.

R1.2 This is a good point, and we thank the reviewer for raising it. "Agentic" refers to the model setting its own success criteria (it discovers the specification of the task) and auditing its answer against them, instead of receiving criteria from a user or an external verifier. This happens within a fixed four-stage scaffold (induce → generate → check → repair). We now state this meaning explicitly in the updated draft (Sec. 4). To avoid any misreading, we will also simplify the title in the camera-ready version (Context Learning with Self-Discovered Specifications), as title changes are not permitted during the discussion period. The contribution is unchanged: the diagnosis that specification acquisition is a key bottleneck, plus a deliberately minimal intervention that validates it.

(continued in Part 2/2)

AuthorsOpenReview ↗

Rebuttal Part 2/2

W2. Some experimental reporting needs clarification. TICK/STICK achieves 23.00% strict pass in Table 3 but 24.00% in Table 6. The paper also uses 31,592 CL-Bench rubric items, compared with 31,607 reported in the original benchmark paper. If these differences reflect dataset versions or reporting conventions, stating so would make the results easier to verify.

R2. Thank you for the careful reading. We have fixed both in the updated draft. 1) TICK/STICK: the correct value is 23.00 (Table 3); the 24.00 was a typo in the compute table (Table 7 in the updated draft). 2) 31,592 vs. 31,607 rubric items: our rubric-level analysis (Sec. 3) leaves out 15 ambiguous items (0.05%), while all experiments follow the official protocol and are scored on the full 31,607 items. We now state this in Appendix B.2.

Q1. Clarification — method framing: What does "agentic" refer to in PSCI beyond its fixed induction–generation–checking–repair pipeline? This would help clarify the scope of the paper's methodological contribution.

A1. Please refer to R1.2.

Q2. Clarification — source of the gains: How do the experiments distinguish the benefit of a context-grounded specification contract from that of a pre-generation, task-specific checklist? In particular, how closely do the adapted checklist/rubric baselines retain the mechanisms of the cited methods? This affects how I interpret the comparative results.

A2. For the first part, please refer to R1.1: in the same pipeline with the same compute, a context-grounded contract (28.14) is far better than a generic checklist (24.00) or another task's contract (22.00).

For the second part, the checklist baselines keep the core mechanism of the cited methods: Agentic Rubrics derives a task-specific checklist before answering (20.33), and TICK/STICK generates task-derived checks that guide the answer (23.00). As for all methods, the only adaptation is that the checklist is derived from the task's own CL-Bench context. We now state this in the updated draft (Appendix D).

In addition, we added two comparisons with recent methods that need no reimplementation by us, since they use the other authors' own numbers (Sec. 5.2, Appendix K):

Comparison on CL-Bench (strict pass, %)Other methodPSCI
Ctx2Skill, the authors' released GPT-5.1 outputs (1,867 matched tasks)25.8728.01
CCA, as reported in its paper (Kimi K2.5)21.423.3

As shown in the table, PSCI is ahead in both comparisons (p = 0.041 against Ctx2Skill, with gains in all four task families).

Q3. Clarification — reporting: Could the authors reconcile the TICK/STICK strict-pass rates in Tables 3 and 6 (23.00% vs. 24.00%) and explain why the analysis uses 31,592 rubric items rather than the 31,607 reported by CL-Bench? The answer would help establish the evaluation and data-counting protocol.

A3. Please refer to R2.

Finally, we kindly note that the updated draft also adds a second benchmark, CL-bench Life, with the PSCI prompts unchanged (Sec. 5.2, Table 6):

CL-bench Life (strict pass, %)Claude Haiku 5.5Gemini 3.1 ProGemini 4 ArgonClaude Sonnet 5.5
Direct baseline9.513.219.314.1
PSCI12.917.626.319.2

PSCI improves all four models (+3.4 to +7.0 pp; pooled p < 0.001). We have incorporated all of the above into the updated draft and sincerely hope these clarifications and new results could resolve the reviewer's concerns.

Dear Reviewer bePT,

We would like to sincerely thank you for your efforts and comments in reviewing our submission. As we approach the end of the discussion period, we would like to ask you to kindly confirm that you have reviewed our rebuttal and let us know if there are any remaining concerns regarding our work. As mentioned in our previous response, in the updated draft (new content in blue) we have added a grouped view of the ablations that separates the contribution of each component (Table 12, Appendix M.1), stated what "agentic" refers to (Sec. 4), clarified how the adapted checklist and rubric baselines relate to the cited methods, and fixed the reporting details you pointed out. We hope that these responses effectively address your questions. If you have any further concerns or questions, please do not hesitate to let us know.

Thank you again for your help and time in reviewing our submission!

Authors

Reviewer_HgNROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why LLMs struggle with context learning and identifies specification acquisition as a major bottleneck. The authors propose Private Specification-Contract Induction (PSCI), an inference-time framework that extracts context-specific requirements before generating answers, followed by verification and repair. On CL-Bench, PSCI improves GPT-5.1's task success rate from 22.55% to 28.14%, with improvements across additional models.

Critical strengths
  1. The paper provides a systematic analysis of 31,592 evaluation rubrics, identifying specification acquisition as a major source of failure.
  2. Extensive experiments involving 12 baselines, 17 ablations, and three LLMs support the effectiveness of the proposed approach.
  3. The method is relatively simple, interpretable, and requires no additional model training.
Critical weaknesses
  1. Limited methodological novelty: Extracting context-specific requirements and using them to guide generation has similarities to prior work, particularly Ctx2Skill (Si et al., arXiv 2026). The paper does not sufficiently distinguish PSCI from this closely related approach or the more recent Compile, Don't Memorize (Qi et al., EMNLP findings 2026), which similarly extracts structured requirements and uses verification.
  2. Limited benchmark coverage: All experiments use CL-Bench. Since the proposed method directly targets failure patterns identified in this benchmark, its generalizability remains uncertain.
  3. Costly Inference: PSCI consumes five times more tokens for 20-30% relative performance improvement.
Questions for authors
  1. How does PSCI's specification-contract induction fundamentally differ from the context-specific skill extraction proposed by Si et al. (2026)?
  2. How does PSCI differ from the context compilation and verification framework proposed by Qi et al. (2026), particularly regarding the intermediate representation and error correction?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

We would like to thank you for your thoughtful feedback, which has greatly strengthened our work. We are encouraged that you found our analysis "systematic", our experiments "extensive", and our method "relatively simple, interpretable" and training-free. We have uploaded an updated draft (new content in blue) and address your concerns point by point below.

W1. Limited methodological novelty: Extracting context-specific requirements and using them to guide generation has similarities to prior work, particularly Ctx2Skill (Si et al., arXiv 2026).

R1.1 Thank you for pointing out these two important and insightful works. We have carefully read both papers, and we have now cited both and discussed them carefully in the updated draft: in the related work (Sec. 2), in the main results (Sec. 5.2), and in a dedicated appendix (Appendix K). We also want to kindly note that, as of today, Ctx2Skill is still an arXiv preprint, and CCA first appeared on arXiv on Sep 1, 2026, about two weeks before the submission deadline. The ICLR 2027 reviewer guidelines do not require authors to compare to these works.

Even so, we agree with the reviewer that a careful comparison is valuable, so we (i) clarify the differences among the three works and (ii) compare directly with Ctx2Skill, which released its GPT-5.1 outputs. For CCA, which reports results on open-weight models, we additionally ran PSCI on Kimi K2.5, its strongest reported setting, and compare in R1.2.

We would like to respectfully note that while all three works make context knowledge explicit before answering, our focus is different (diagnosis vs. method). Ctx2Skill shows that reusable skills can be distilled from a context through self-play, and CCA shows the power of compiling a context into a verifiable structure. Our paper asks an upstream question: why do models fail at context learning? Our rubric-level diagnosis (Sec. 3) finds that the missing piece is often the specification of the current task rather than its content, and PSCI is the minimal intervention that tests this diagnosis.

Differences (Table 10, Appendix K.2).

Ctx2SkillCCAPSCI (ours)
Unitper context (reused across its tasks)per contextper task (query-conditioned)
How obtainedoffline multi-agent self-play over synthetic probe taskscompile prose into typed slots plus executable Python verifiersone induction step at inference, no offline phase
Focusmethodmethoddiagnosis (Sec. 3) plus a minimal method to validate it

The task-level difference matters: 76.7% of specifications are not inferable from the query (Sec. 3.4), and contracts not grounded in the current task do not help (generic 24.00, shuffled 22.00 vs. PSCI 28.14; Table 4).

Head-to-head results (Sec. 5.2; Table 8, Appendix K.1). We compare against Ctx2Skill's own released outputs on the 1,867 CL-Bench tasks shared by both releases, with the same backbone (GPT-5.1):

Method (GPT-5.1)Strict passDKRRSAPTEEDS
Direct baseline22.5023.6323.5221.4818.13
Ctx2Skill (released outputs)25.8727.7425.1326.9019.17
PSCI28.0131.2525.4929.5020.73

As shown in the table, PSCI outperforms Ctx2Skill (28.01 vs. 25.87, +2.14 pp, p = 0.041), with gains in all four task families. PSCI also needs no offline phase, while Ctx2Skill runs self-play for each context before answering (5 iterations × 5 probe tasks in its released configuration).

(continued in Part 2/3)

AuthorsOpenReview ↗

Rebuttal Part 2/3

R1.1 (continued). To further confirm this result, we also re-graded all three systems together with three additional judges on a fixed random 300-task subset (Table 9, Appendix K.1):

Judge (same 300 tasks)BaselineCtx2SkillPSCI
GPT-4.134.040.343.0
Gemini 4 Argon26.428.831.8
Gemini 3.8 Flash23.428.831.4
Majority of the three26.429.432.4

As shown in the table, the ranking PSCI > Ctx2Skill > baseline holds under every judge, and the trend is consistent with the full-scale result above. Since per-task contracts and per-context skills act at different stages, combining them is a promising direction, which we now note in the conclusion (Sec. 7).

W1. ... The paper does not sufficiently distinguish PSCI from this closely related approach or the more recent Compile, Don't Memorize (Qi et al., EMNLP findings 2026), which similarly extracts structured requirements and uses verification.

R1.2 Thank you for the thoughtful question. CCA and PSCI differ in both the intermediate representation and the error correction. CCA compiles each context into a typed slot structure and catches errors with executable verifiers, which applies to obligations that can be compiled into code. PSCI induces a list of natural-language obligations for the current query (Sec. 4.1); a separate checker audits the answer item by item against it, and repair edits only the violated items (Sec. 4.2). This also covers obligations that are hard to compile into code (tone and role, source-only discipline, exclusions, completeness), and PSCI improves every specification subtype in Table 1. We have added this discussion to Appendix K.2.

To make the comparison concrete, we additionally ran PSCI on Kimi K2.5, the model on which CCA reports its largest gain, with the same benchmark (all 1,899 CL-Bench tasks) and the same judge (GPT-5.1):

Kimi K2.5DirectWith methodGain
CCA (as reported in [2])15.421.4+6.0
PSCI (ours, 3 runs)16.1 ± 0.423.3 ± 0.8+7.2

As shown in the table, PSCI is slightly ahead of CCA on its strongest model (23.3 vs. 21.4). PSCI's gain is also consistent across models: +5.3 to +7.2 pp on all four models we evaluate on the full benchmark (GPT-5.1, Gemini 3 Pro, Qwen3.5-27B, Kimi K2.5).

W2. Limited benchmark coverage: All experiments use CL-Bench. Since the proposed method directly targets failure patterns identified in this benchmark, its generalizability remains uncertain.

R2. We agree this is an important point, and we thank the reviewer for the suggestion. We have now conducted additional experiments on a second benchmark with a fourth model family (Sec. 5.2, Table 6; Appendix L in the updated draft). CL-bench Life [3] uses real-life contexts (group chats, notes, activity logs) and is constructed differently from CL-Bench. We evaluate four models with the PSCI prompts unchanged (Appendix I): Claude Haiku 5.5, Gemini 3.1 Pro, and Gemini 4 Argon on all 388 Life tasks that fit in 100K tokens (of 405), and Claude Sonnet 5.5 on a random 200-task subset. Each answer is graded with the official prompt by a majority of three judges.

CL-bench Life (strict pass, %)Claude Haiku 5.5Gemini 3.1 ProGemini 4 ArgonClaude Sonnet 5.5
Direct baseline9.513.219.314.1
PSCI12.917.626.319.2
Gain (relative)+3.4 (+36%)+4.4 (+33%)+7.0 (+36%)+5.1 (+36%)

As shown in the table, PSCI improves all four models by +3.4 to +7.0 pp (+33% to +36% relative; pooled p < 0.001) on a benchmark it was not developed on, and the trend is consistent under each of the three judges individually (Table 11). Claude Haiku 5.5, a new model family, also improves on CL-Bench itself (200-task subset: 24.5 → 30.0). Together with Kimi K2.5 (R1.2), PSCI now improves five model families across two benchmarks, highlighting that the benefit of making specifications explicit is not specific to CL-Bench. We have also corrected our statement that CL-Bench is the only context-learning benchmark (Sec. 2, Sec. 3, Appendix A).

(continued in Part 3/3)

AuthorsOpenReview ↗

Rebuttal Part 3/3

W3. Costly Inference: PSCI consumes five times more tokens for 20-30% relative performance improvement.

R3. This is a valid point, and we agree that cost is a trade-off for inference-time methods such as PSCI (Table 7 in the updated draft; Table 6 in the original submission). We kindly note three points. 1) The gain is not explained by extra tokens: methods with similar or larger token budgets stay well below PSCI (28.14), e.g., majority vote 24.12 (94.5k tokens), self-reflection 23.02 (94.5k), and Agentic Rubrics 20.33 (84.8k); GPT-5.1 at high reasoning effort (23.7) also stays below PSCI at medium (Table 3). 2) A cheaper version of PSCI already exists: using only the first two stages (contract induction and contract-guided answering, without checking and repair; "Baseline+SI" in Table 7) costs about 3× the baseline's tokens and already gives +3.54 pp, more than every baseline. With a small model the absolute cost is also low ($0.037 per task for Claude Haiku 5.5 on CL-bench Life). 3) The extra compute buys fully correct answers: PSCI produces 24.8% more fully correct answers than the baseline (22.55 → 28.14). Following the reviewer's comment, we have expanded this discussion in the updated draft (Appendix A). Overall, this gain cannot be obtained by simply spending more tokens, and a cheaper version is available, so we respectfully believe the extra cost is justified in this case.

Q1. How does PSCI's specification-contract induction fundamentally differ from the context-specific skill extraction proposed by Si et al. (2026)?

A1. Please refer to R1.1. In short, Ctx2Skill learns reusable per-context skills offline through self-play, while PSCI induces a per-task, query-conditioned obligation contract at inference time and uses it as a shared control object for generation, checking, and repair (Sec. 4).

Q2. How does PSCI differ from the context compilation and verification framework proposed by Qi et al. (2026), particularly regarding the intermediate representation and error correction?

A2. Please refer to R1.2. In short, CCA uses a typed slot structure with executable verifiers, while PSCI uses natural-language obligations for the current query, checked item by item and repaired only where violated.

We have incorporated these results into the updated draft and sincerely hope these new sets of experiments could resolve the reviewer's concerns.

References

[1] Si et al., From Context to Skills: Can Language Models Learn from Context Skillfully?, arXiv 2026.

[2] Qi et al., Compile, Don't Memorize: A Context Compilation Architecture for In-Context Learning, Findings of EMNLP 2026.

[3] Dou et al., CL-bench Life: Can Language Models Learn from Real-Life Context?, arXiv 2026.

Dear Reviewer HgNR,

We would like to sincerely thank you for your efforts and comments in reviewing our submission. As we approach the end of the discussion period, we would like to ask you to kindly confirm that you have reviewed our rebuttal and let us know if there are any remaining concerns regarding our work. As mentioned in our previous response, in the updated draft (new content in blue) we have cited, discussed and compared with Ctx2Skill and CCA and clarified how PSCI differs from each (Sec. 2, Sec. 5.2, Appendix K), added a second benchmark, CL-bench Life, on which PSCI improves all four models tested with the prompts unchanged (Table 6, Appendix L), and clarified the cost trade-off (Table 7). We hope that these responses effectively address your questions. If you have any further concerns or questions, please do not hesitate to let us know.

Thank you again for your help and time in reviewing our submission!

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Summary of changes in the updated draft

We thank all reviewers for their thoughtful and constructive feedback. We have uploaded an updated draft (new content in blue). The main additions are:

  1. Comparison with recent methods (Sec. 2, Sec. 5.2, Appendix K). We now cite and discuss Ctx2Skill and CCA. PSCI outperforms Ctx2Skill on its authors' own released outputs (28.01% vs. 25.87%) and is slightly ahead of CCA's reported result on its strongest model, Kimi K2.5 (23.3% vs. 21.4%).
  2. A second benchmark (Sec. 5.2, Table 6; Appendix L). On CL-bench Life, with the prompts unchanged, PSCI improves all four models tested (+3.4 to +7.0 pp). PSCI now improves five model families across two benchmarks.
  3. Where the gains come from (Appendix M.1; Appendix D and L). A grouped view of the ablations shows that the gain depends on what goes into the contract, when it is used, and how it is enforced, not on extra compute. A new specification-retrieval baseline recovers almost none of the gain on CL-bench Life.
  4. Finer analysis of discovery and enforcement (Sec. 6.1–6.2; Appendix M.2–M.3). We show which specifications the contract misses and give a finer attribution of the remaining failures.
  5. Related work and clarity (Sec. 2, Sec. 4, Appendix B.2, Appendix F.3). We discuss LLM-Modulo, state what "agentic" refers to, and clarify the reporting details raised by the reviewers.

We address each reviewer's points in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.