acceptodds
Under review as a conference paper at ICLR 2027

RECAP: From Requirement Recovery to Reliable Execution in Context Learning

Abstract

Large language models are increasingly used in long-context settings where the context defines temporary knowledge, rules, procedures, and constraints. Many failures are near misses: models capture the main task but miss some low-salience requirements. Yet improving requirement recall alone can produce redundant and fragmented representations that are difficult to execute, making context learning a joint problem of broad requirement recovery and executable representation. We study whether such representations can be discovered from empirical feedback rather than designed entirely by hand. To this end, we perform feedback-driven search over a structured space of executable inference pipelines, treating each edit as a falsifiable hypothesis and retaining regressions, ineffective edits, and non-transferring changes as constraints for subsequent search. Within this structured search space, feedback-driven evaluation supports broad requirement enumeration followed by consolidation into a compact representation before execution. This design yields RECAP, which constructs such a representation at inference time and conditions generation on it. For weaker models, we further introduce RECAP-RL, which learns to execute self-constructed requirement representations using rubric-level rewards. On CL-bench, RECAP improves Claude Opus-4.8 from 21.09% to 27.04% strict accuracy and Qwen3.5-9B from 11.74% to 15.60%, while RECAP-RL further raises Qwen3.5-9B to 17.96%. The approach also transfers to complex instruction following and multimodal context learning. These results suggest a general recipe for context learning: discover an effective requirement representation, construct it explicitly, and learn to execute it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Context-Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wiQuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the challenge of reliable context learning in large language models, where models must recover and execute task-specific requirements embedded in long contexts. The authors identify a trade-off between requirement coverage and executability: extensive requirement extraction may improve recall but introduce redundancy and fragmentation that hinder subsequent generation. To address this challenge, the authors propose RECAP, a framework derived from feedback-driven search over structured inference pipelines. RECAP consists of four stages: multi-view requirement enumeration, compact consolidation, coverage-oriented answering, and guarded additive patching. The framework aims to construct comprehensive yet executable requirement representations at inference time. Furthermore, the authors introduce RECAP-RL, which employs reinforcement learning with rubric-level rewards to improve smaller models' ability to execute self-constructed requirement representations. The proposed methods are evaluated on CL-bench across six backbone models, with additional evaluations on AgentIF, MultiChallenge, and the multimodal MCL-bench. On CL-bench, RECAP improves the strict accuracy of Claude Opus-4.8 from 21.09% to 27.04% and Qwen3.5-9B from 11.74% to 15.60%, while combining RECAP with RECAP-RL further improves Qwen3.5-9B to 17.96%.

Critical strengths
  1. The problem formulation is relatively clear and well motivated. The paper explicitly distinguishes requirement recovery from requirement execution in context learning and identifies the trade-off between comprehensive requirement coverage and the redundancy introduced by excessive enumeration. RECAP combines multi-view requirement enumeration with compact consolidation to preserve essential requirements while reducing the execution burden on the model.
  2. The experimental design is reasonably systematic and evaluates the proposed method across different model scales and task settings. The paper evaluates six backbone models on CL-bench and further investigates generalization performance on AgentIF, MultiChallenge, and MCL-bench. In addition to comparisons with existing methods, the authors conduct component ablation studies, budget-matched baseline experiments, and controlled experiments on representation-conditioned reinforcement learning.
Critical weaknesses
  1. On the methodological novelty compared with existing approaches. RECAP shares a similar overall inference structure with PSCI, as both methods involve specification extraction, conditioned generation, and verification or repair. Although RECAP introduces multi-view requirement enumeration and compact consolidation, the extent to which these components provide advantages beyond existing specification-based approaches remains insufficiently established. In particular, Table 7 shows that PSCI with consolidation achieves comparable strict accuracy to full RECAP (27.90% versus 27.78%), although RECAP obtains higher rubric adherence (83.31% versus 82.57%). Furthermore, while the paper introduces feedback-driven pipeline discovery and RECAP-RL, their independent methodological contributions relative to existing approaches could be more clearly distinguished.

  2. On the performance improvements and computational overhead. Although RECAP consistently improves strict accuracy over direct inference, its improvements over the stronger PSCI baseline are relatively modest. For example, RECAP achieves 27.04% versus 26.34% on Claude Opus-4.8 and 15.60% versus 14.56% on Qwen3.5-9B, while achieving slightly lower strict accuracy on GPT-5.1 (27.78% versus 28.14%). The paper provides budget-matched comparisons in Table 4, but these primarily control the number of model calls rather than actual computational cost. Since RECAP involves multiple stages of requirement enumeration, consolidation, generation, and verification, the absence of detailed input and output token consumption, end-to-end inference latency, and computational cost makes it difficult to assess whether the observed improvements justify the additional inference overhead, particularly compared with PSCI and PSCI with consolidation.

  3. On the statistical reliability of the reported improvements. Although Section 4.1 states that the CL-bench results are averaged over three independent runs, the main experimental tables, particularly Tables 1 and 7, do not report standard deviations or confidence intervals for the reported performance differences. The paper considers variability during feedback-driven pipeline discovery and provides confidence intervals for requirement coverage in Figure 6, but these analyses do not directly establish the statistical reliability of the main performance comparisons. Given that several improvements over PSCI are below one percentage point, reporting run-level variability and confidence intervals for the differences between methods would provide stronger evidence of the robustness of the reported results.

Questions for authors
  1. On the methodological differences between RECAP and PSCI. Although RECAP introduces multi-view requirement enumeration and compact consolidation, Table 7 shows that PSCI with consolidation achieves comparable strict accuracy to full RECAP (27.90% versus 27.78%), with a relatively small difference in rubric adherence (82.57% versus 83.31%). Could the authors further clarify the key technical advantages of the proposed two-stage requirement construction over simpler extensions of PSCI? In particular, what evidence establishes the independent contributions of multi-view enumeration and consolidation, and how do the feedback-driven discovery process and RECAP-RL contribute to the overall methodological novelty?

  2. On the computational efficiency of RECAP. Although Table 4 provides budget-matched baselines based on the number of model calls, this does not fully reflect the actual inference overhead. Could the authors provide a more detailed comparison of total input and output token consumption, end-to-end inference latency, and computational cost per task for RECAP, PSCI, and PSCI with consolidation under comparable evaluation settings? Such an analysis would help determine whether the additional inference stages provide a favorable trade-off between performance and computational efficiency.

  3. On the statistical reliability of the main results. Section 4.1 states that the CL-bench results are averaged over three independent runs, but the main experimental tables do not provide corresponding uncertainty estimates. Could the authors report the mean, standard deviation, and 95% confidence intervals for the principal comparisons, particularly those involving PSCI and PSCI with consolidation? Where paired evaluation results are available, confidence intervals for the performance differences would be especially useful for assessing whether the observed improvements are consistent across runs.

  4. On the consistency of experimental results with the original PSCI study. Although both papers evaluate on CL-bench with 1,899 tasks, RECAP reports 31,607 rubric items, whereas the original PSCI paper reports 31,592. Moreover, the GPT-5.1 Direct baseline differs between the two studies: PSCI reports 22.55% strict accuracy and 76.01% rubric adherence, while RECAP reports 22.18% and 82.05%, respectively. The difference in rubric adherence is particularly notable. Meanwhile, Table 1 of RECAP appears to adopt the original PSCI result of 28.14% strict accuracy, as indicated by the dagger symbol. Could the authors clarify the reasons for these discrepancies, including potential differences in dataset versions, evaluation protocols, grading configurations, or inference settings? Furthermore, were the PSCI results reproduced under the same experimental conditions as RECAP, or were they directly taken from the original publication?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wiQu — Part 1/3

We thank the reviewer and jointly address novelty, cost, statistical reliability, and protocol consistency with new controlled analyses.

W1 & Q1. Novelty beyond PSCI + consolidation

We do not claim novelty for the extract–condition–repair skeleton. PSCI already reports that 83% of residual failed rubrics were in its contract but not executed (Sec. 6.2). RECAP addresses how to recover broad coverage while constructing an executable, compact requirement representation. Our central finding concerns the interaction between multi-view recovery and consolidation, and the resulting representation is designed for both reliable execution and effective learning.

New matched-protocol, cross-backbone comparison (three-run means; strict / rubric; now main-text Table 2):

BackbonePSCIPSCI + consolidationRECAP
GPT-5.128.05 / 81.0227.90 / 82.5727.78 / 83.31
Claude Opus-4.826.34 / 83.3326.07 / 83.0827.04 / 83.78
Qwen3.5-4B11.40 / 71.1411.23 / 70.4711.80 / 70.61
Qwen3.5-9B14.56 / 74.9914.17 / 74.6315.60 / 75.53

On GPT-5.1, PSCI + consolidation slightly exceeds RECAP in strict accuracy (27.90 vs. 27.78), while RECAP gains +0.74 rubric. On the other three backbones, PSCI + consolidation reduces both metrics relative to PSCI, whereas full RECAP improves on PSCI + consolidation. Thus, consolidation appears more useful after broad recovery than after a single pass.

New common-skeleton 2×2 experiment. On 197 Claude Opus-4.8 tasks, we crossed single-pass vs. four-view extraction with consolidation on/off, holding the answer prompt fixed without auditing or patching. GPT-5.5 assessed representations against official rubrics blind to the arm; strict accuracy uses the 191 tasks graded in all arms.

ArmMean reqsGT coverageRedundancyCoverage retainedAnswer rubric (macro)Strict
A: single-pass17.5767.61%5.53%—74.14%15.71%
B: single + consolidation12.1266.02%2.56%93.80%74.23%15.71%
C: multi-view60.9373.47%63.48%—75.52%15.18%
D: multi-view + consolidation14.0168.90%2.41%90.60%74.26%17.28%

Multi-view raises ground-truth coverage by +5.86 pp (C−A; 95% paired CI [3.95, 7.85]) and still by +2.88 pp after consolidation (D−B). Consolidation cuts 60.93 → 14.01 requirements (−77%) and redundancy by 61.1 pp while retaining 90.6% of covered rubrics. Their combination (D) yields the highest observed strict accuracy (17.28%), although C retains higher macro answer-rubric accuracy (75.52% vs. 74.26%).

New matched PSCI-conditioned RL (strict, %; same data, reward, GRPO, and 69 steps; only requirement construction changed; App. E.9, Table 24):

TrainingInferenceQwen3.5-4BQwen3.5-9B
BasePSCI11.4014.56
BaseRECAP11.8015.60
PSCI-conditioned RLPSCI11.7115.40
PSCI-conditioned RLRECAP11.4616.20
RECAP-RLRECAP14.1717.96

PSCI-conditioned RL gains only +0.31/+0.84 pp over base PSCI, versus +2.37/+2.36 pp for RECAP-RL over base RECAP: requirements missed by one-pass extraction never enter the conditioning representation, so the policy learns to execute only what was extracted. Under RECAP inference, it trails RECAP-RL by 2.71/1.76 pp, is slightly below base RECAP on Qwen3.5-4B (−0.34 pp), and does not exceed No-representation RL on Qwen3.5-9B (16.20 vs. 16.69). Requirement construction must therefore be designed for learnability, not recovery alone.

Discovery, learning, and transfer. Feedback-driven validation selects the design (Table 5(b); Fig. 7); our claim is the discovered recover-then-consolidate principle, not the search procedure. Under RECAP inference, RECAP-RL also beats No-representation RL (17.96 vs. 16.69 strict; Table 11), and self-constructed training requirements outperform GPT-5.5-generated ones (17.96 vs. 15.97; Table 6), suggesting representation–executor alignment. RECAP improves complex instruction following (54.7 vs. 52.5 PSCI, Claude Opus-4.8; Table 4), and text-only RECAP-RL transfers to MCL-bench (11.89 → 21.79 strict, Qwen3.5-9B; Table 3).

In the revision: contribution in Sec. 1; matched comparison in Table 2; 2×2 experiment in Sec. 4.2 and App. E.4 (Table 17).

AuthorsOpenReview ↗

Response to Reviewer wiQu — Part 2/3

W2 & Q2. Measured tokens, latency, and cost trade-off

We measured per-task calls, input/output tokens (including reasoning tokens), and elapsed time, including method-internal judges but not final grading; shared-context Ctx2Skill construction is amortized across its tasks.

GPT-5.1 (medium); 51 tasks

MethodCallsInput tokensOutput tokensTotal tokensLatency p50 / p95 (s)
Direct1.0013,2995,02918,32948 / 140
Ctx2Skill159.471,548,726312,8491,861,5755,738 / 7,463
PSCI5.00101,16126,466127,626294 / 606
PSCI + consolidation6.00109,99426,590136,585352 / 634
RECAP7.43121,76926,714148,483393 / 754

Claude Opus-4.8; 74 tasks

MethodCallsInput tokensOutput tokensTotal tokensLatency p50 / p95 (s)
Direct1.0016,4181,93918,35731 / 121
Ctx2Skill158.121,813,983153,4771,967,4604,668 / 5,973
PSCI5.00121,77413,824135,598245 / 519
PSCI + consolidation6.00133,14513,296146,441266 / 585
RECAP7.16142,5318,451150,981200 / 683

RECAP uses 1.16× / 1.11× PSCI tokens and 1.09× / 1.03× PSCI + consolidation tokens (GPT-5.1 / Opus-4.8), mainly due to repeated context inputs. Compared with Ctx2Skill, RECAP uses 12–13× fewer tokens while achieving higher strict / rubric accuracy by +4.60 / +1.85 pp on GPT-5.1 and +4.33 / +1.48 pp on Opus-4.8. Median latency vs. PSCI varies by backbone: 393 vs. 294 s (GPT-5.1) and 200 vs. 245 s (Opus-4.8, with fewer output tokens).

For tight budgets, Table 5(a) (now also Table 20) offers 1 view + answer (2 calls; 25.03 strict / 83.13 rubric) and 2 views + answer (3 calls; 26.71 / 83.69) on GPT-5.1. Full RECAP prioritizes strict accuracy at greater inference cost.

In the revision: Tables 19–20 (App. E.6), summarized in Sec. 4.4(c); cost is also discussed in Limitations (App. F).

W3 & Q3. Uncertainty of performance differences

Three-run mean ± SD. The submitted GPT-5.1 PSCI result was quoted from the original paper. We reproduced PSCI using its published prompts under the same inference and grading protocol as RECAP and PSCI + consolidation. Strict accuracy uses validly evaluated tasks; rubric adherence is micro-averaged over valid items. We report mean ± sample SD over three runs:

BackboneMethodStrictRubric
GPT-5.1Direct22.18 ± 0.4382.05 ± 0.37
PSCI (reproduced)28.05 ± 0.5581.02 ± 0.82
PSCI + consolidation27.90 ± 0.4682.57 ± 0.52
RECAP27.78 ± 0.8183.31 ± 0.84
Claude Opus-4.8Direct21.09 ± 0.1681.71 ± 0.12
PSCI26.34 ± 0.4583.33 ± 0.13
PSCI + consolidation26.07 ± 0.3683.08 ± 0.31
RECAP27.04 ± 0.1283.78 ± 0.21
Qwen3.5-4BDirect10.70 ± 0.1970.49 ± 0.28
PSCI11.40 ± 0.1471.14 ± 0.45
PSCI + consolidation11.23 ± 0.1270.47 ± 0.48
RECAP11.80 ± 0.4370.61 ± 0.55
Qwen3.5-9BDirect11.74 ± 0.1071.91 ± 0.12
PSCI14.56 ± 0.7874.99 ± 0.40
PSCI + consolidation14.17 ± 0.5674.63 ± 0.27
RECAP15.60 ± 0.5375.53 ± 0.46

Task-paired bootstrap 95% CIs (pp). Δ is the difference between three-run means. CIs resample tasks, with per-task scores averaged over three runs and only tasks valid under both methods paired; GPT-5.1 uses our PSCI reproduction. For strict accuracy: Claude Opus-4.8 RECAP − Direct +5.95 [4.45, 7.41]; Qwen3.5-9B RECAP − Direct +3.86 [2.64, 5.06]; Qwen3.5-4B RECAP-RL + RECAP − RECAP +2.37 [1.08, 3.62]. Comparisons with PSCI:

ComparisonStrict Δ [95% CI]Rubric Δ [95% CI]
Claude Opus-4.8: RECAP − PSCI+0.70 [−0.85, +2.25]+0.45 [−0.12, +1.01]
GPT-5.1: RECAP − PSCI−0.27 [−1.84, +1.30]+2.29 [+1.73, +2.87]
Qwen3.5-4B: RECAP − PSCI+0.40 [−0.83, +1.62]−0.53 [−1.23, +0.19]
Qwen3.5-9B: RECAP − PSCI+1.04 [−0.27, +2.33]+0.54 [−0.12, +1.20]
Qwen3.5-4B: RECAP-RL + RECAP − PSCI+2.77 [+1.62, +3.93]+5.03 [+4.34, +5.71]
Qwen3.5-9B: RECAP-RL + RECAP − PSCI+3.40 [+2.04, +4.76]+4.81 [+4.16, +5.44]

Inference-only RECAP is comparable to or modestly better than PSCI: strict Δ is positive on three of four backbones, but no strict CI excludes zero; only the GPT-5.1 rubric CI does. The complete RECAP + RECAP-RL framework improves over PSCI by +2.77/+3.40 pp strict and +5.03/+4.81 pp rubric on Qwen3.5-4B/9B, with all CIs excluding zero.

In the revision: Tables 14–15 (App. E.2); PSCI comparisons in main-text Table 2 and Sec. 4.2.

AuthorsOpenReview ↗

Response to Reviewer wiQu — Part 3/3

Q4. PSCI protocol and dataset consistency

Rubrics: official CL-bench release has 500 contexts, 1,899 tasks, 31,607 rubric items; PSCI reports 31,592 items (15 fewer), without enough information to identify the cause.

Published versus reproduced PSCI (GPT-5.1, three runs): PSCI reports 28.14 ± 0.50 strict / 81.13 ± 0.90 rubric; our reproduction achieves 28.05 ± 0.55 / 81.02 ± 0.82, differing by only −0.09 / −0.11 points. Table 1 quotes the published PSCI GPT-5.1 and GPT-4.1 Ctx2Skill results (†); Tables 2 and 14 use our reproduction.

GPT-5.1 Direct baseline: our 22.18 ± 0.43 strict / 82.05 ± 0.37 rubric differs from PSCI's 22.55 ± 0.60 / 76.01 ± 1.20. We use released system/user messages without additional prompts, GPT-5.1 at medium effort, a 32,768-token cap, and up to three retries. The official CL-bench verifier (gpt-5.1-2025-11-13, low effort) gives binary judgments for each rubric item, counting items without an explicit “yes” as unmet. Rubric adherence is micro-averaged.

In the revision: App. E.3 documents these protocol details, referenced in Sec. 4.1.

Dear Reviewer wiQu,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_dsDvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a central challenge in context learning: language models often fail to recover all relevant requirements from long contexts. The authors propose RECAP, an inference-time framework that constructs a compact requirement representation through multi-view enumeration, consolidation, coverage-oriented answer generation, and guarded additive patching. The pipeline is developed using feedback-driven search over structured design choices, with candidate modifications evaluated across decoding seeds and retained based on variance-aware acceptance criteria and held-out validation.

The paper also introduces RECAP-RL, which trains smaller models to generate answers conditioned on self-constructed requirement representations. Training uses a GRPO-style objective with rubric-level rewards and token-entropy-based advantage reweighting.

The authors evaluate RECAP on CL-bench, comprising 1,899 tasks and 31,607 rubric items, across six model backbones. RECAP improves strict accuracy over direct inference across all six models, including gains from 21.09% to 27.04% for Claude Opus-4.8 and from 22.18% to 27.78% for GPT-5.1, although PSCI achieves slightly higher strict accuracy on the latter. RECAP-RL provides additional improvements when combined with the RECAP inference pipeline. The validation includes component ablations, inference-budget-matched baselines, analyses of requirement redundancy and coverage, and transfer evaluations on complex instruction-following and multimodal context-learning benchmarks.

Critical strengths
  1. Empirical improvements across model scales and task settings: RECAP improves strict accuracy over direct inference across all six backbones on CL-bench, including gains of 5.95 percentage points for Claude Opus-4.8 and 5.60 points for GPT-5.1. The paper also evaluates transfer to complex instruction-following and multimodal context-learning benchmarks, with additional gains from RECAP-RL on smaller models. This breadth of evaluation suggests that the approach addresses a recurring limitation in context learning.

  2. Ablations and inference-budget-controlled comparisons: The authors investigate the contributions of requirement enumeration, consolidation, and answer refinement, including unsuccessful pipeline variants. They also compare RECAP against additional-compute baselines, such as repeated self-refinement and best-of-eight direct generation. Analyses of requirement redundancy and coverage further illuminate the trade-off between recovering requirements comprehensively and keeping them manageable during execution. These experiments strengthen the paper because they provide evidence that the improvements are not due to additional inference calls and offer useful empirical insight into the pipeline's design choices.

Critical weaknesses
  1. Limited methodological novelty: RECAP combines established techniques such as requirement extraction, consolidation, prompted answer generation, and refinement. While the feedback-driven search provides a systematic way to select these components, the resulting method is largely incremental. RECAP-RL evaluation does not cleanly isolate the benefits of representation-conditioned training from general improvements due to rubric-based RL. These limitations weaken the paper's central mechanistic claims and make it difficult to assess the contribution beyond an effective pipeline design.

  2. Insufficient uncertainty reporting for the empirical comparisons: Although CL-bench results are averaged over three independent runs, the paper does not report per-run variability or confidence intervals for its main results. This is consequential because several improvements are modest, and some comparisons with competing methods are particularly close (e.g., RECAP achieves 27.78% strict accuracy versus PSCI's 28.14% on GPT-5.1). Without uncertainty estimates, it is difficult to determine which differences are statistically reliable. At the minimum reporting variability across independently trained RL checkpoints would substantially strengthen the empirical conclusions.

Questions for authors
  1. Isolating the contribution of representation-conditioned RL: The comparison between RECAP-RL + Direct and RECAP-RL + RECAP does not fully isolate the benefit of learning to execute requirement representations, since Direct inference removes the explicit representation used during training. Have the authors evaluated a matched rubric-based RL baseline trained without requirement representations, with both checkpoints evaluated using the same RECAP inference pipeline? This would help determine whether the gains arise specifically from representation-conditioned learning or from more general improvements due to RL.

  2. Statistical reliability of the reported improvements: Section 4.1 states that CL-bench results are averaged over three independent runs, but the main tables do not report uncertainty estimates. Can the authors provide per-run results, standard deviations, or paired bootstrap confidence intervals for the main comparisons? For RECAP-RL, were multiple independently trained checkpoints evaluated, or do the three runs reflect repeated inference using a single checkpoint? These details are important for assessing the reliability of smaller improvements.

  3. Clarification of the development/evaluation separation. The RECAP pipeline is selected through feedback-driven search using rubric-level evaluation signals. Can the authors clarify whether the contexts, tasks, and rubrics used during pipeline search and selection are strictly disjoint from those used in the final CL-bench evaluation? This would help assess the extent to which the reported improvements reflect generalizable pipeline design rather than adaptation to the evaluation benchmark.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dsDv — Part 1/2

We thank the reviewer and address each weakness alongside its corresponding question.

W1 & Q1. Novelty and representation-conditioned RL

We agree that the generic extract–condition–repair skeleton is not new. Our contribution concerns how requirement representations are discovered, constructed, and learned to be executed, rather than combining familiar inference steps: feedback-driven search discovers the recover-then-consolidate principle from empirical evidence rather than hand design (Sec. 3.1).

Representation design. PSCI reports that 83% of residual failed rubric items were covered by its contract but not executed (Sec. 6.2). RECAP addresses this recovery–executability gap through multi-view recovery followed by consolidation. On GPT-5.1, consolidation reduces near-duplicates from 49% to 12% while retaining 94.2% of previously covered rubrics (Figs. 4–6). Multi-view without consolidation lowers PSCI rubric adherence from 81.13 to 80.42 (Table 8). We claim the combined representation principle, not novelty for either operation alone.

Learning contribution: matched control. Table 11 (App. D) includes the requested No-representation RL control: identical data, reward judge, rubric-mean reward, and GRPO settings, but no self-constructed requirements during training. Both checkpoints use matched inference modes:

TrainingDirect strict / rubricRECAP strict / rubric
Base (no RL)11.74 / 71.9115.60 / 75.53
No-representation RL13.01 / 74.3016.69 / 78.00
RECAP-RL14.38 / 76.4717.96 / 79.80

With matched RECAP inference, representation-conditioned RL improves on generic rubric RL by +1.27 strict / +1.80 rubric points; under matched Direct inference, by +1.37 / +2.17. Thus, the benefit is not solely due to test-time scaffolding. Three independent training seeds confirm the trend (W2 & Q2).

Further empirical insights. Using the same rubric-mean reward and RECAP inference, self-constructed training requirements outperform GPT-5.5-generated ones (17.96 vs. 15.97 strict, Qwen3.5-9B; Table 6), suggesting representation–executor alignment. RECAP also improves complex instruction following (53.0 vs. 50.6 PSCI on GPT-5.1; Table 4). Notably, text-only RECAP-RL transfers to multimodal context learning on MCL-bench without multimodal training data, raising Qwen3.5-9B strict from 11.89 to 21.79 (base Direct vs. RECAP-RL + RECAP; Table 3), with gains under Direct inference as well.

In the revision: the contribution statement is revised in Sec. 1, and the matched No-representation RL control is moved into the main text (Sec. 4.5, with three-seed results in Table 16).

AuthorsOpenReview ↗

Response to Reviewer dsDv — Part 2/2

W2 & Q2. Statistical reliability and independent checkpoints

Three-run variability. We retain GPT-5.1 PSCI's published three-run result (28.14 strict / 81.13 rubric); other entries are measured under our evaluation protocol. For each run, strict accuracy is computed over validly evaluated tasks, while rubric adherence is micro-averaged over validly evaluated rubric items. We report mean ± sample SD across three runs for both metrics:

BackboneMethodStrict (mean ± SD)Rubric (mean ± SD)
GPT-5.1Direct22.18 ± 0.4382.05 ± 0.37
PSCI †28.14 ± 0.5081.13 ± 0.90
PSCI + consolidation27.90 ± 0.4682.57 ± 0.52
RECAP27.78 ± 0.8183.31 ± 0.84
Claude Opus-4.8Direct21.09 ± 0.1681.71 ± 0.12
PSCI26.34 ± 0.4583.33 ± 0.13
PSCI + consolidation26.07 ± 0.3683.08 ± 0.31
RECAP27.04 ± 0.1283.78 ± 0.21
Qwen3.5-4BDirect10.70 ± 0.1970.49 ± 0.28
PSCI11.40 ± 0.1471.14 ± 0.45
PSCI + consolidation11.23 ± 0.1270.47 ± 0.48
RECAP11.80 ± 0.4370.61 ± 0.55
Qwen3.5-9BDirect11.74 ± 0.1071.91 ± 0.12
PSCI14.56 ± 0.7874.99 ± 0.40
PSCI + consolidation14.17 ± 0.5674.63 ± 0.27
RECAP15.60 ± 0.5375.53 ± 0.46

† GPT-5.1 PSCI is quoted from the original PSCI paper (three-run mean ± SD); all other rows use our evaluation protocol. Our reproduction under the same protocol gives 28.05 ± 0.55 / 81.02 ± 0.82 (Table 14).

Task-paired bootstrap 95% CIs for strict-accuracy improvements (percentage points):

ComparisonPaired Δ strict95% CI
Claude Opus-4.8, RECAP − Direct+5.95[+4.45, +7.41]
Qwen3.5-9B, RECAP − Direct+3.86[+2.64, +5.06]
Qwen3.5-4B, RECAP-RL + RECAP − RECAP+2.37[+1.08, +3.62]

Δ is the difference between the three-run means above; pairing uses only tasks with valid evaluations under both compared methods.

Independent RL checkpoints. The original paper's three RECAP-RL runs repeated inference from one checkpoint. We additionally trained two independent checkpoints per RL configuration (same data, judge, hyperparameters), each evaluated over three inference runs:

TrainingSeedDirect strict / rubricRECAP strict / rubric
No-representation RL113.01 / 74.3016.69 / 78.00
213.43 / 74.8817.08 / 78.52
312.71 / 73.9516.32 / 77.61
Mean ± SD13.05 ± 0.36 / 74.38 ± 0.4716.70 ± 0.38 / 78.04 ± 0.46
RECAP-RL114.38 / 76.4717.96 / 79.80
213.92 / 75.8817.41 / 79.27
314.71 / 76.9518.24 / 80.31
Mean ± SD14.34 ± 0.40 / 76.43 ± 0.5417.87 ± 0.42 / 79.79 ± 0.52

Across three independent training seeds, RECAP-RL exceeds No-representation RL by +1.17 strict / +1.75 rubric under RECAP inference and +1.29 / +2.05 under Direct. Every RECAP-RL seed has higher strict accuracy than every control seed in either mode.

In the revision: mean ± SD results are in Table 14 (App. E.2) and, for PSCI-related comparisons, in the new main-text Table 2; paired CIs are in Table 15; independent RL checkpoints are in Table 16 and summarized in Sec. 4.5.

Q3. Discovery/training versus evaluation data

The stages share broad task categories, not CL-bench instances:

StageDataFeedback
Discovery400 GPT-5.5-synthesized development + 400 context-disjoint validation tasksGPT-5.1 candidate execution; synthetic rubric scores; acceptance requires validation transfer
RECAP-RL training9,410 synthetic tasks over 3,737 contexts, 129,782 rubrics; 192-task validation splitGPT-5.1-generated data; Qwen3.8-27B reward judge; rubrics not supplied as policy targets
Final evaluationOfficial CL-bench: 1,899 tasks, 31,607 rubricsGPT-5.1 evaluation verifier; not used for discovery or RL updates

No official CL-bench contexts, tasks, rubrics, or answers enter discovery or RL training. The pipeline is frozen after GPT-5.1 discovery and transferred without backbone-specific tuning.

In the revision: this table is Table 13 (App. E.1), referenced in Sec. 4.1. We have also foregrounded the matched RL control (Sec. 4.5) and distinguished methodological contributions from empirical findings (Sec. 1).

Reviewer_dsDvOpenReview ↗

Rebuttal response

Thanks for the detailed rebuttal and additional experiments. The matched generic RL baseline and results across three independent training seeds address my concern about isolating the contribution of representation-conditioned training. The clarification of data separation also addresses my question about potential overlap between pipeline discovery and final evaluation.

My main remaining empirical concern is the statistical reliability of RECAP's incremental improvements over PSCI. The reported task-paired bootstrap confidence intervals focus on comparisons against Direct inference or the effect of RECAP-RL, rather than RECAP versus PSCI. In the matched-protocol comparison (revised Table 2), RECAP improves strict accuracy over PSCI by 0.70 percentage points on Claude Opus-4.8, decreases it by 0.26 points on GPT-5.1, and improves it by 0.40 and 1.04 points on Qwen3.5-4B and Qwen3.5-9B, respectively. The corresponding changes in rubric adherence are +0.45, +2.18, −0.53, and +0.54 percentage points, respectively. Given the relatively small differences on several backbones and the reported run-to-run variability, task-paired confidence intervals for RECAP versus PSCI would establish the reliability of these incremental improvements.

The broader results in Tables 1, 3, and 4 provide meaningful evidence of effectiveness across backbones and transfer settings, particularly for smaller models. These results strengthen the empirical contribution of the framework. Regarding methodological novelty, I appreciate the authors' clarification of the distinction between multi-view requirement recovery and consolidation. However, RECAP still builds on the extract–condition–repair framework established by PSCI. The primary distinction is the explicit separation of requirement recovery and compression. While this is a reasonable and empirically useful design choice, the extent of the new algorithmic contribution is incremental. The feedback-driven search provides a systematic approach to pipeline design, but operates over a predefined space of established components.

I maintain my borderline-positive recommendation, with some reservations about methodological novelty and the statistical reliability of the incremental improvements over PSCI.

Reviewer_rnM6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies context learning: given a source context cc and a task xx, a model must produce an answer yy, and each task carries a rubric set where every rubric item states a necessary condition for a correct answer. The paper formulates context learning as a joint problem of requirement recovery and representation executability, and uses feedback-driven search over a structured space of executable context-learning pipelines as a discovery instrument. The central empirical finding is that broad multi-view requirement enumeration helps recover low-salience requirements, but the resulting set becomes redundant and fragmented; consolidating this over-complete set into a compact, executable representation is the most effective intervention.

Based on this, the authors propose RECAP: multi-view requirement enumeration, compact consolidation, coverage-oriented answering, and guarded additive patching only when residual coverage can be improved. For weaker models, they further propose RECAP-RL, which lets the model construct requirement representations offline and trains answering conditioned on these self-constructed representations with rubric-level rewards and GRPO. Experiments are conducted on CL-bench, with transfer to AgentIF, MultiChallenge, and MCL-bench; RECAP is evaluated on six backbones, and RECAP-RL is trained and evaluated on three open-weight backbones.

Critical strengths
  1. The paper articulates a practical tension between requirement recovery and execution burden in long contexts, and uses feedback-driven search to provide empirical evidence rather than hand-designing the pipeline directly.
  2. The four-stage RECAP design is reasonable, and treating consolidation as a first-class design choice rather than simple pruning is well motivated.
  3. The experiments cover multiple backbones, multiple task families, and cross-modal transfer; the ablations are relatively thorough, and the appendix provides prompts, search algorithm details, and training details.
  4. RECAP-RL trains on self-constructed requirements rather than distilling teacher-generated answers or reasoning traces, which is a meaningful direction.
Critical weaknesses
  1. Limited methodological novelty.
    The core finding—enumerate broadly, then consolidate—is a useful empirical conclusion, but the method itself is a combination of multi-view requirement extraction, merging, conditioned generation, and patching. PSCI already adopts an extract–condition–repair skeleton; RECAP mainly separates recovery from compression and makes compression a first-class design choice. This factorized design is reasonable, but it is hard to argue that it constitutes a new algorithmic idea. Feedback-driven search is a meta-level contribution, but the search space, evaluation backbone, and discovered conclusions are centered on CL-bench and GPT-5.1; the paper does not sufficiently discuss whether a pipeline discovered with GPT-5.1 remains the same design when transferred to other backbones, or whether the search process itself overfits to the discovery backbone.

  2. The trade-off between practical gains and cost is not sufficiently clear.
    At test time, RECAP requires multi-view enumeration, consolidation, answering, auditing, and optional patching, roughly 7–9 backbone calls. The appendix uses budget-matched self-refine and best-of-8 controls, which help show that the gains are not merely from more computation. However, the paper does not sufficiently discuss whether the strict-accuracy improvement (e.g., GPT-5.1 from 22.18 to 27.78) is worth the corresponding latency and cost in real deployments. RECAP-RL also requires GPT-5.1 to synthesize training data, Qwen3.8-27B as the rubric judge, and GRPO training; these dependencies imply a non-trivial barrier to practical use.

  3. The nature of the gains requires a more careful interpretation.
    The qualitative case studies show that many strict-accuracy flips come from missing purely formatting or low-salience obligations, such as square brackets versus parentheses, conversational filler before JSON, or an unset swing parameter. These examples illustrate the fragility of all-or-nothing grading well, but they also raise the question of how much of RECAP’s gain reflects “better satisfying the rubric checks” rather than a substantive improvement in context understanding or reasoning. The appendix acknowledges this, but the main text could state the “context learning capability improvement” claim more cautiously. In addition, both evaluation and training rely on LLM judges (GPT-5.1 / GPT-5.5 / Qwen3.8-27B); the judges’ biases and agreement with human annotations are not systematically validated, which also affects the interpretation of the gains.

Questions for authors
  • Was the pipeline discovered by feedback-driven search on GPT-5.1 re-searched when applied to other backbones? If not, how can the authors rule out overfitting of the discovered prompts and number of views to GPT-5.1?

  • Can the authors provide a clearer discussion of the trade-off between RECAP’s additional call cost and strict-accuracy gains? In latency-sensitive or cost-constrained deployments, which stages of RECAP are necessary and which can be omitted?

  • Many flips in the qualitative cases come from formatting or low-salience obligations. Can the authors estimate what fraction of the strict-accuracy improvement comes from such “rubric compliance” improvements rather than improvements in core content understanding?

  • The quality of the rubric judge directly affects both the RECAP-RL training signal and evaluation. Have the authors validated the judge’s agreement with human ratings? If the judge has systematic biases, could RECAP-RL amplify them?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rnM6 — Part 1/3

We thank the reviewer and jointly address the related weaknesses and questions with new structural-search, resource, rubric-flip, and judge-reliability analyses.

W1 & Q1. Novelty and backbone-specific search

We do not claim the extract–condition–repair skeleton as novel. PSCI reports that 83% of residual failed rubrics were already in its contract but not executed (Sec. 6.2). RECAP addresses this recovery–executability tension through feedback-driven discovery of broad recovery followed by consolidation. On GPT-5.1, near-duplicates fall from 49% to 12%, retaining 94.2% of previously covered rubrics (Figs. 4–6, Sec. 4.6).

Learning and transfer. Beyond inference design, RECAP-RL learns to execute self-constructed requirements, outperforming matched No-representation RL under identical RECAP inference (+1.27 strict; Table 11; three-seed results in Table 16). Self-constructed training requirements also outperform GPT-5.5-generated ones (17.96 vs. 15.97 strict, Qwen3.5-9B; Table 6), suggesting representation–executor alignment. Transfer extends to complex instruction following (54.7 vs. 52.5 PSCI on Claude Opus-4.8; Table 4) and MCL-bench: text-only RECAP-RL, without multimodal RL training, raises Qwen3.5-9B strict from 11.89 to 21.79 (Table 3). Direct-inference gains further show transfer beyond extra test-time calls.

Representation for learning. The representation also serves as an RL training interface. We add a matched PSCI-conditioned RL control (strict, %; same data, reward, GRPO, and 69 steps; only requirement construction changed; App. E.9, Table 24):

TrainingInferenceQwen3.5-4BQwen3.5-9B
BasePSCI11.4014.56
BaseRECAP11.8015.60
PSCI-conditioned RLPSCI11.7115.40
PSCI-conditioned RLRECAP11.4616.20
RECAP-RLRECAP14.1717.96

PSCI-conditioned RL gains only +0.31/+0.84 pp over base PSCI, versus +2.37/+2.36 pp for RECAP-RL over base RECAP: requirements missed by one-pass extraction never enter the conditioning representation, so the policy learns to execute only what was extracted. Under RECAP inference, it trails RECAP-RL by 2.71/1.76 pp, is slightly below base RECAP on Qwen3.5-4B (−0.34 pp), and does not exceed No-representation RL on Qwen3.5-9B (16.20 vs. 16.69, Table 11). The contribution therefore lies in designing the representation for both execution and learning, not in any single operation.

Frozen pipeline transfer. We initially searched once on GPT-5.1, using 400 synthetic development + 400 context-disjoint validation examples, then froze the stage prompts, four views, consolidation target, and single guarded patch. The unchanged pipeline improves strict accuracy on all six backbones:

BackboneDirect strictRECAP strictStrict gain (pp)Rubric gain (pp)
GPT-5.1 (discovery)22.1827.78+5.60+1.26
Claude Opus-4.821.0927.04+5.95+2.07
Qwen3.5-9B11.7415.60+3.86+3.62
GPT-4.114.0117.19+3.18+1.38
Qwen3.5-4B10.7011.80+1.10+0.12
Llama-3.2-3B3.013.50+0.49−1.19

Task-paired 95% CIs exclude zero on two non-discovery backbones: Claude Opus-4.8 +5.95 pp [4.45, 7.41]; Qwen3.5-9B +3.86 pp [2.64, 5.06] (Table 15).

New Qwen3.5-9B structural re-search. With stage prompts and audit/patch fixed, we tested 12 configurations (1–4 views × {no consolidation, target 15, target 9}). Selection used synthetic development/validation data, not CL-bench. The selected 3-view + target-9 variant scored 15.82 strict / 75.71 rubric, versus 15.60 / 75.53 for the frozen GPT-5.1 design. The top three development configurations all used ≥3 views with consolidation. The same design preference emerges on a second backbone.

In the revision: the novelty statement now reads that our contribution is how the requirement representation is discovered, constructed, and learned (Sec. 1). The frozen-pipeline transfer (Table 18) and the Qwen3.5-9B structural re-search are reported in App. E.5 and summarized in Sec. 3.1.

AuthorsOpenReview ↗

Response to Reviewer rnM6 — Part 2/3

W2 & Q2. Measured overhead versus PSCI and cheaper variants

We measure Direct, PSCI, PSCI + consolidation, RECAP, and Ctx2Skill on common subsets. The resource figures below are our measurements under matched conditions. Costs include internal calls, input/output tokens (including reasoning tokens), and latency, excluding final benchmark grading. Ctx2Skill construction is amortized across tasks sharing a context.

Backbone / sampleMethodCalls/taskInput / output tokensTotal tokensp50 / p95 latency (s)
GPT-5.1; 51 tasksDirect1.0013,299 / 5,02918,32948 / 140
PSCI5.00101,161 / 26,466127,626294 / 606
PSCI + consolidation6.00109,994 / 26,590136,585352 / 634
RECAP7.43121,769 / 26,714148,483393 / 754
Ctx2Skill159.471,548,726 / 312,8491,861,5755,738 / 7,463
Claude Opus-4.8; 74 tasksDirect1.0016,418 / 1,93918,35731 / 121
PSCI5.00121,774 / 13,824135,598245 / 519
PSCI + consolidation6.00133,145 / 13,296146,441266 / 585
RECAP7.16142,531 / 8,451150,981200 / 683
Ctx2Skill158.121,813,983 / 153,4771,967,4604,668 / 5,973

Cost versus accuracy. RECAP uses 1.16× / 1.11× PSCI tokens and 1.09× / 1.03× PSCI + consolidation tokens (GPT-5.1 / Opus-4.8), mainly due to repeated context inputs. Compared with Ctx2Skill, RECAP uses 12–13× fewer tokens while achieving higher strict / rubric accuracy by +4.60 / +1.85 pp on GPT-5.1 and +4.33 / +1.48 pp on Opus-4.8. Despite higher token usage, median latency varies by backbone: 393 vs. 294 s on GPT-5.1 and 200 vs. 245 s on Opus-4.8 (RECAP vs. PSCI). The lower Opus-4.8 median may partly reflect fewer output tokens.

On GPT-5.1, RECAP uses 8.1× Direct tokens for a +5.60 pp strict-accuracy gain (22.18 → 27.78). This trade-off favors accuracy-critical tasks, where missing a single requirement can invalidate the answer. The token measurements above use the reported subsets, while accuracy gains come from the full benchmark (Table 1).

Which stages can be omitted? Table 5(a) provides lower-call operating points (now Table 20):

GPT-5.1 configurationCallsStrictRubric
Direct122.1882.05
1 view + answer225.0383.13
2 views + answer326.7183.69
3 views + answer (no consolidation)426.0879.25
Full RECAP7–9 (7.43 measured)27.7883.31

Requirement recovery drives the early gains: 1 view + answer captures 51% of the full strict gain with 2 calls, while 2 views + answer captures 81% with 3 calls. Adding a third view without consolidation lowers strict / rubric accuracy from 26.71 / 83.69 to 26.08 / 79.25, highlighting the importance of consolidation when recovery is broadened. For budget-constrained settings, 2 views + answer omits consolidation, audit, and patch while retaining 81% of the full strict-accuracy gain. Full RECAP achieves higher strict accuracy (27.78 vs. 26.71) at additional inference cost.

RECAP-RL additionally requires a one-time offline training run (69 GRPO steps) with open-weight Qwen3.8-27B as reward judge. GPT-5.1 synthesizes training data offline but is not used for RL judging or inference.

In the revision: the measured cost (Table 19) and operating points (Table 20) are in App. E.6 and summarized in Sec. 4.4(c); the cost trade-off is also discussed in the new Limitations section (App. F).

AuthorsOpenReview ↗

Response to Reviewer rnM6 — Part 3/3

W3 & Q3. Are strict gains merely formatting compliance?

We quantitatively classified rubric items from 1,899 CL-bench tasks as form/content, generic/context-dependent, and mechanically checkable/not, using GPT-5.4. Only 9.6% of corpus items were labeled form and 8.9% generic. We then examined Direct-fail → RECAP-pass tasks and the specific items missed by Direct (the decisive items):

Flip attributionClaude Opus-4.8GPT-5.1Qwen3.5-9B
Flipped tasks207225169
Require ≥1 content item88.4%91.1%94.7%
Require ≥1 context-dependent item99.0%98.7%99.4%
Require ≥1 non-mechanically-checkable item36.7%36.4%48.5%

Of 1,444 decisive items pooled across three backbones:

GenericContext-dependent
Form5 (0.3%)153 (10.6%)
Content65 (4.5%)1,221 (84.6%)

Takeaway. Only 0.3% of decisive items are both generic and form-only; 88.4–94.7% of flipped tasks require content and 98.7–99.4% require context-dependent obligations. Mechanically checkable items may still be substantive (e.g., GRID_OVERLOAD rules or session deltas). The formatting examples are not representative.

Label reliability. Claude Opus-4.8 re-labeled 300 stratified tasks / 5,028 items. Agreement was 86.8% (κ 0.720) for objectivity, 92.7% (0.667) for form/content, and 93.8% (0.578) for genericity. This cross-model agreement supports the rubric classification.

W3 & Q4. Judge reliability and reward bias

The CL-bench authors previously evaluated the GPT-5.1 verifier, reporting >90% human consistency on 100 sampled solutions and >90% raw agreement with alternative verifiers. To validate the separate MultiChallenge evaluation judge and RECAP-RL reward judge, we conducted blind expert audits: two annotators per item and a third adjudicator.

SampleResponses / rubric itemsHuman–human agreement / κJudge–human agreement / κ
MultiChallenge, GPT-5.5 evaluator816 / 81689.71% / 0.77993.14% / 0.853
RECAP-RL reward, Qwen3.8-27B279 / 3,83294.44% / 0.77294.28% / 0.785

Three additional blinded model auditors (GPT-6.1-sol, Claude Opus-5, Claude Opus-4.8) agree with the original judges at 89.7–95.6% (κ 0.776–0.908) on MultiChallenge and 93.1–94.8% (κ 0.742–0.813) on the reward sample.

Reward-bias check. The training judge Qwen3.8-27B is not used to evaluate CL-bench or MCL-bench. Qwen3.5-9B's RECAP-RL gains persist under GPT-5.1 CL-bench grading (+2.64 strict Direct; +2.36 RECAP) and GPT-5.5 MCL-bench grading (+9.90). General-capability scores change by roughly one point or less (Fig. 11). Together, these results argue against judge-specific reward exploitation.

In the revision: the flip attribution and label reliability are in App. E.7 (Tables 21–22) and summarized in Sec. 4.6, where we also state the claim more cautiously; the judge audits and reward-bias check are in App. E.8 (Table 23) and Sec. 4.1; limits of the rubric-based evaluation are discussed in App. F.

Dear Reviewer rnM6,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response [2/2]

B.2 Reviewer-Specific Concerns & Solutions (Q4–Q8)

Q4 (rnM6 W1/Q1): Does the GPT-5.1-discovered pipeline overfit to its discovery backbone, and would searching on another backbone recover the same principle?

  • Frozen transfer (Table 18, App. E.5): Frozen after GPT-5.1 discovery (synthetic data only), the unchanged pipeline (4 views, consolidation, guarded patch) improves strict accuracy on all 6 backbones (+5.95 on Opus-4.8, +3.86 on Qwen3.5-9B, +3.18 on GPT-4.1).
  • Structural re-search on Qwen3.5-9B (Sec. 3.1, App. E.5): Testing 12 configurations (1–4 views × {no consolidation, target 15, target 9}) on Qwen3.5-9B using synthetic dev/val splits only (never CL-bench), the top-3 configs all select ≥3 views with consolidation, and the chosen 3-view + consolidation design scores 15.82 strict / 75.71 rubric on CL-bench vs. 15.60 / 75.53 for the frozen GPT-5.1 design—confirming that recover-then-consolidate re-emerges on a second backbone.

Q5 (rnM6 W3/Q3): Do strict gains merely reflect superficial formatting compliance rather than content understanding?

  • Rubric classification & flip audit (Sec. 4.6, App. E.7, Tables 21–22, App. F): We classified all CL-bench rubrics (form/content, generic/context-dependent, mechanically checkable) with GPT-5.4 and verified labels on 300 tasks / 5,028 items with Opus-4.8 (86.8%–93.8% agreement, κ = 0.578–0.720).
  • Substantive content drives flips: Across 1,444 decisive items missed by Direct on 601 Direct-fail → RECAP-pass flipped tasks (Opus-4.8, GPT-5.1, Qwen3.5-9B), 84.6% are context-dependent content, 88.4%–94.7% of flipped tasks require ≥1 content item, 98.7%–99.4% require ≥1 context-dependent obligation, and only 0.3% (5/1,444) are generic formatting. Claims are stated more cautiously in Sec. 4.6 and App. F.

Q6 (rnM6 W3/Q4): Are the LLM judges reliable vs. humans, and could RECAP-RL exploit reward-judge bias?

  • Blind human audits (Sec. 4.1, App. E.8, Table 23): Two annotators + an adjudicator audited 816 MultiChallenge items and 3,832 RECAP-RL reward judgments. Judge–human agreement is 93.14% (κ = 0.853) for MultiChallenge (GPT-5.5) and 94.28% (κ = 0.785) for RL rewards (Qwen3.8-27B)—matching or exceeding human–human κ (0.779 and 0.772) and backed by 3 external model auditors (89.7%–95.6%).
  • Judge decoupling (App. E.8): The RL reward judge (Qwen3.8-27B) is never used at test time; RECAP-RL gains persist under disjoint evaluators on CL-bench (GPT-5.1, +2.64 Direct / +2.36 RECAP strict) and MCL-bench (GPT-5.5, +9.90 strict), while general benchmarks change by ≤1 point (Fig. 11).

Q7 (dsDv Q3): Are discovery and RL training data strictly disjoint from CL-bench evaluation?

  • 100% disjoint (Sec. 4.1, App. E.1, Table 13): Discovery (400 dev + 400 context-disjoint val tasks, GPT-5.5-synthesized) and RECAP-RL (9,410 synthetic tasks over 3,737 contexts, 129,782 rubrics) use purely synthetic data; no CL-bench context, task, rubric, or answer is used before final evaluation.

Q8 (wiQu Q4): Is our protocol consistent with the PSCI study (rubric count, reproduced vs. quoted PSCI, Direct baseline)?

  • Reproduced vs. published PSCI (Sec. 4.1, App. E.3, Table 14): Re-running PSCI's prompts on GPT-5.1 under our exact grading pipeline yields 28.05 ± 0.55 strict / 81.02 ± 0.82 rubric, matching published PSCI (28.14 / 81.13) within 0.09 / 0.11 points.
  • Dataset & Direct setup (App. E.3): We use the official CL-bench release (1,899 tasks, 31,607 rubrics vs. PSCI's 31,592, an unexplained 15-item gap in PSCI) and document our full setup (official messages without wrapper prompts, GPT-5.1 medium effort, 32k token cap, official gpt-5.1-2025-11-13 binary verifier with micro-averaging).

B.3 Per-Reviewer Checklist & Discussion Status

  • Reviewer dsDv — Replied (borderline-positive recommendation): Explicitly confirmed that our No-representation RL control (Q1), 3 independent RL seeds (Q3), and data-separation audit (Q7) resolved their initial concerns (W1/Q1, W2/Q2, Q3). Raised a follow-up on statistical reliability and methodological novelty vs. PSCI, which we resolved via paired CIs vs. PSCI (+2.77 / +3.40 strict for RECAP-RL + RECAP, Q3 / Table 15) and PSCI-conditioned RL (+2.71 / +1.76 strict advantage for RECAP-RL, Q1 / Table 24).
  • Reviewer rnM6 — No follow-up received: All four concerns resolved: W1/Q1 (novelty & backbone overfitting) → Q1 & Q4; W2/Q2 (cost & stage omission) → Q2; W3/Q3 (formatting vs. content flips) → Q5; W3/Q4 (judge reliability & reward bias) → Q6.
  • Reviewer wiQu — No follow-up received: All four concerns resolved: W1/Q1 (novelty vs. PSCI + consolidation) → Q1; W2/Q2 (token/latency overhead) → Q2; W3/Q3 (statistical reliability) → Q3; Q4 (PSCI protocol consistency) → Q8.

We sincerely thank the AC and all reviewers for their time and constructive feedback.

AuthorsOpenReview ↗

General Response [1/2]

We thank the AC and reviewers and uploaded a revised paper (updates in blue). In this rebuttal, [1/2] covers (A) contributions, results, and strengths, and (B.1) common concerns (Q1–Q3); [2/2] covers (B.2) specific concerns (Q4–Q8) and (B.3) discussion status.

A. Contributions, Main Results & Strengths

Core contributions. Reliable context learning requires not only recovering requirements comprehensively, but also representing them in a form that models can execute and learn from (separating recovery from execution and coverage–redundancy tension; rnM6, wiQu). We address this through three steps:

  • Discovery: feedback-driven search identifies the recover-then-consolidate principle (rnM6, dsDv, wiQu).
  • Construction: RECAP recovers requirements from multiple views, consolidates them into a compact representation (first-class choice vs. pruning; rnM6, dsDv, wiQu), answers conditioned on it, and applies a guarded patch that adds missed requirements only if coverage does not drop.
  • Learning: RECAP-RL trains models to execute their own constructed requirements (not teacher distillation; rnM6, wiQu).

Main results (praised for broad gains, thorough ablations, budget-matched self-refine/best-of-8, clarity, and reproducibility; rnM6, dsDv, wiQu):

  • Context learning (CL-bench, main benchmark): RECAP improves strict accuracy on all six backbones (e.g., +5.95 on Opus-4.8), and RECAP-RL + RECAP outperforms PSCI by +2.77 / +3.40 on Qwen3.5-4B / 9B.
  • Transfer to complex instruction following (AgentIF, MultiChallenge): RECAP raises the mean score from 43.2 to 54.7 on Opus-4.8.
  • Transfer to multimodal context learning (MCL-bench): RECAP-RL, trained only on text, lifts strict accuracy from 11.89 to 21.79 on Qwen3.5-9B.
  • Analysis: we analyze why context learning and complex instruction following share a bottleneck: both require recovering and covering dense requirements, implicit and distributed in the former, explicit but numerous in the latter.

Relation to PSCI. Both share an extract–condition–repair skeleton. PSCI prompts a single-pass specification; our representation is discovered, constructed, and learnable.

B.1 Common Concerns & Solutions (Q1–Q3)

Q1 (rnM6 W1, dsDv W1/Q1, wiQu W1/Q1): Is RECAP just "PSCI + consolidation," and does RECAP-RL add value beyond generic RL or PSCI-RL? Our contribution is how the representation is discovered, constructed, and learned (Sec. 1):

  • Consolidating 1-pass PSCI hurts on 3/4 backbones (Table 2, Sec. 4.2): Trimming a 1-pass list without adding missed rules lowers both metrics on Opus-4.8 and Qwen3.5-4B/9B, where RECAP beats PSCI + consolidation on both. On GPT-5.1, all three are within 1 SD on strict (28.05/27.90/27.78) and RECAP leads on rubric (83.31 vs. 81.02/82.57).
  • 2×2 ablation isolates multi-view × consolidation synergy (Table 17, App. E.4): On 197 Opus-4.8 tasks (1-pass vs. 4-view × consolidation off/on, no audit/patch): 4-view adds +5.86 coverage [3.95, 7.85] but inflates items to 60.93 (63.5% redundancy), lowering strict (15.18 vs. 15.71); consolidation cuts items by 77% (to 14.01, 2.4% redundancy), keeping 90.6% coverage and giving the best strict (17.28).
  • RECAP-RL beats No-representation RL (Tables 11, 16, Sec. 4.5) & PSCI-conditioned RL (Table 24, App. E.9): Across 3 seeds on Qwen3.5-9B (same data, judge, reward, 69 GRPO steps), RECAP-RL beats No-representation RL by +1.17/+1.29 strict under RECAP/Direct (every seed beats every control seed), and self-constructed requirements beat GPT-5.5 (17.96 vs. 15.97, Table 6). Training GRPO on PSCI's 1-pass extractions (PSCI-conditioned RL) adds only +0.31/+0.84 over base PSCI and trails RECAP-RL by 2.71/1.76 strict on Qwen3.5-4B/9B.

Q2 (rnM6 W2/Q2, wiQu W2/Q2): What is RECAP's cost vs. PSCI, and which stages can be dropped?

  • Measured cost (Table 19, App. E.6): On GPT-5.1/Opus-4.8, RECAP uses 7.43/7.16 calls, 1.16×/1.11× PSCI tokens (1.09×/1.03× PSCI + consolidation), and 12–13× fewer than Ctx2Skill with +4.60/+4.33 higher strict (Opus-4.8 p50 latency is lower: 200s vs. 245s).
  • Cheaper variant & RL cost (Table 20, Sec. 4.4c): 2 views + answer (3 calls, no consolidation/audit/patch) keeps 81% of the strict gain over Direct (26.71 vs. 27.78). RECAP-RL is one offline run (69 GRPO steps, Qwen3.8-27B judge; App. F).

Q3 (dsDv W2/Q2, wiQu W3/Q3): Are the gains statistically reliable across runs and RL seeds?

  • 3-run Mean ± SD (Tables 2, 14), 3 RL seeds (Table 16), & paired 95% CIs (Table 15, App. E.2): RECAP − Direct is +5.95 [4.45, 7.41] (Opus-4.8) and +3.86 [2.64, 5.06] (Qwen3.5-9B); RECAP-RL adds +2.37 [1.08, 3.62] (Qwen3.5-4B). Vs. PSCI, inference-only RECAP is comparable (strict CIs include 0; GPT-5.1 rubric Δ +2.29 [1.73, 2.87]), while RECAP-RL + RECAP beats PSCI by +2.77 [1.62, 3.93] / +3.40 [2.04, 4.76] strict on Qwen3.5-4B/9B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.