- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the challenge of reliable context learning in large language models, where models must recover and execute task-specific requirements embedded in long contexts. The authors identify a trade-off between requirement coverage and executability: extensive requirement extraction may improve recall but introduce redundancy and fragmentation that hinder subsequent generation. To address this challenge, the authors propose RECAP, a framework derived from feedback-driven search over structured inference pipelines. RECAP consists of four stages: multi-view requirement enumeration, compact consolidation, coverage-oriented answering, and guarded additive patching. The framework aims to construct comprehensive yet executable requirement representations at inference time. Furthermore, the authors introduce RECAP-RL, which employs reinforcement learning with rubric-level rewards to improve smaller models' ability to execute self-constructed requirement representations. The proposed methods are evaluated on CL-bench across six backbone models, with additional evaluations on AgentIF, MultiChallenge, and the multimodal MCL-bench. On CL-bench, RECAP improves the strict accuracy of Claude Opus-4.8 from 21.09% to 27.04% and Qwen3.5-9B from 11.74% to 15.60%, while combining RECAP with RECAP-RL further improves Qwen3.5-9B to 17.96%.
Critical strengths
- The problem formulation is relatively clear and well motivated. The paper explicitly distinguishes requirement recovery from requirement execution in context learning and identifies the trade-off between comprehensive requirement coverage and the redundancy introduced by excessive enumeration. RECAP combines multi-view requirement enumeration with compact consolidation to preserve essential requirements while reducing the execution burden on the model.
- The experimental design is reasonably systematic and evaluates the proposed method across different model scales and task settings. The paper evaluates six backbone models on CL-bench and further investigates generalization performance on AgentIF, MultiChallenge, and MCL-bench. In addition to comparisons with existing methods, the authors conduct component ablation studies, budget-matched baseline experiments, and controlled experiments on representation-conditioned reinforcement learning.
Critical weaknesses
-
On the methodological novelty compared with existing approaches. RECAP shares a similar overall inference structure with PSCI, as both methods involve specification extraction, conditioned generation, and verification or repair. Although RECAP introduces multi-view requirement enumeration and compact consolidation, the extent to which these components provide advantages beyond existing specification-based approaches remains insufficiently established. In particular, Table 7 shows that PSCI with consolidation achieves comparable strict accuracy to full RECAP (27.90% versus 27.78%), although RECAP obtains higher rubric adherence (83.31% versus 82.57%). Furthermore, while the paper introduces feedback-driven pipeline discovery and RECAP-RL, their independent methodological contributions relative to existing approaches could be more clearly distinguished.
-
On the performance improvements and computational overhead. Although RECAP consistently improves strict accuracy over direct inference, its improvements over the stronger PSCI baseline are relatively modest. For example, RECAP achieves 27.04% versus 26.34% on Claude Opus-4.8 and 15.60% versus 14.56% on Qwen3.5-9B, while achieving slightly lower strict accuracy on GPT-5.1 (27.78% versus 28.14%). The paper provides budget-matched comparisons in Table 4, but these primarily control the number of model calls rather than actual computational cost. Since RECAP involves multiple stages of requirement enumeration, consolidation, generation, and verification, the absence of detailed input and output token consumption, end-to-end inference latency, and computational cost makes it difficult to assess whether the observed improvements justify the additional inference overhead, particularly compared with PSCI and PSCI with consolidation.
-
On the statistical reliability of the reported improvements. Although Section 4.1 states that the CL-bench results are averaged over three independent runs, the main experimental tables, particularly Tables 1 and 7, do not report standard deviations or confidence intervals for the reported performance differences. The paper considers variability during feedback-driven pipeline discovery and provides confidence intervals for requirement coverage in Figure 6, but these analyses do not directly establish the statistical reliability of the main performance comparisons. Given that several improvements over PSCI are below one percentage point, reporting run-level variability and confidence intervals for the differences between methods would provide stronger evidence of the robustness of the reported results.
Questions for authors
-
On the methodological differences between RECAP and PSCI. Although RECAP introduces multi-view requirement enumeration and compact consolidation, Table 7 shows that PSCI with consolidation achieves comparable strict accuracy to full RECAP (27.90% versus 27.78%), with a relatively small difference in rubric adherence (82.57% versus 83.31%). Could the authors further clarify the key technical advantages of the proposed two-stage requirement construction over simpler extensions of PSCI? In particular, what evidence establishes the independent contributions of multi-view enumeration and consolidation, and how do the feedback-driven discovery process and RECAP-RL contribute to the overall methodological novelty?
-
On the computational efficiency of RECAP. Although Table 4 provides budget-matched baselines based on the number of model calls, this does not fully reflect the actual inference overhead. Could the authors provide a more detailed comparison of total input and output token consumption, end-to-end inference latency, and computational cost per task for RECAP, PSCI, and PSCI with consolidation under comparable evaluation settings? Such an analysis would help determine whether the additional inference stages provide a favorable trade-off between performance and computational efficiency.
-
On the statistical reliability of the main results. Section 4.1 states that the CL-bench results are averaged over three independent runs, but the main experimental tables do not provide corresponding uncertainty estimates. Could the authors report the mean, standard deviation, and 95% confidence intervals for the principal comparisons, particularly those involving PSCI and PSCI with consolidation? Where paired evaluation results are available, confidence intervals for the performance differences would be especially useful for assessing whether the observed improvements are consistent across runs.
-
On the consistency of experimental results with the original PSCI study. Although both papers evaluate on CL-bench with 1,899 tasks, RECAP reports 31,607 rubric items, whereas the original PSCI paper reports 31,592. Moreover, the GPT-5.1 Direct baseline differs between the two studies: PSCI reports 22.55% strict accuracy and 76.01% rubric adherence, while RECAP reports 22.18% and 82.05%, respectively. The difference in rubric adherence is particularly notable. Meanwhile, Table 1 of RECAP appears to adopt the original PSCI result of 28.14% strict accuracy, as indicated by the dagger symbol. Could the authors clarify the reasons for these discrepancies, including potential differences in dataset versions, evaluation protocols, grading configurations, or inference settings? Furthermore, were the PSCI results reproduced under the same experimental conditions as RECAP, or were they directly taken from the original publication?
4 comments
Response to Reviewer wiQu — Part 2/3
W2 & Q2. Measured tokens, latency, and cost trade-off
We measured per-task calls, input/output tokens (including reasoning tokens), and elapsed time, including method-internal judges but not final grading; shared-context Ctx2Skill construction is amortized across its tasks.
GPT-5.1 (medium); 51 tasks
| Method | Calls | Input tokens | Output tokens | Total tokens | Latency p50 / p95 (s) |
|---|---|---|---|---|---|
| Direct | 1.00 | 13,299 | 5,029 | 18,329 | 48 / 140 |
| Ctx2Skill | 159.47 | 1,548,726 | 312,849 | 1,861,575 | 5,738 / 7,463 |
| PSCI | 5.00 | 101,161 | 26,466 | 127,626 | 294 / 606 |
| PSCI + consolidation | 6.00 | 109,994 | 26,590 | 136,585 | 352 / 634 |
| RECAP | 7.43 | 121,769 | 26,714 | 148,483 | 393 / 754 |
Claude Opus-4.8; 74 tasks
| Method | Calls | Input tokens | Output tokens | Total tokens | Latency p50 / p95 (s) |
|---|---|---|---|---|---|
| Direct | 1.00 | 16,418 | 1,939 | 18,357 | 31 / 121 |
| Ctx2Skill | 158.12 | 1,813,983 | 153,477 | 1,967,460 | 4,668 / 5,973 |
| PSCI | 5.00 | 121,774 | 13,824 | 135,598 | 245 / 519 |
| PSCI + consolidation | 6.00 | 133,145 | 13,296 | 146,441 | 266 / 585 |
| RECAP | 7.16 | 142,531 | 8,451 | 150,981 | 200 / 683 |
RECAP uses 1.16× / 1.11× PSCI tokens and 1.09× / 1.03× PSCI + consolidation tokens (GPT-5.1 / Opus-4.8), mainly due to repeated context inputs. Compared with Ctx2Skill, RECAP uses 12–13× fewer tokens while achieving higher strict / rubric accuracy by +4.60 / +1.85 pp on GPT-5.1 and +4.33 / +1.48 pp on Opus-4.8. Median latency vs. PSCI varies by backbone: 393 vs. 294 s (GPT-5.1) and 200 vs. 245 s (Opus-4.8, with fewer output tokens).
For tight budgets, Table 5(a) (now also Table 20) offers 1 view + answer (2 calls; 25.03 strict / 83.13 rubric) and 2 views + answer (3 calls; 26.71 / 83.69) on GPT-5.1. Full RECAP prioritizes strict accuracy at greater inference cost.
In the revision: Tables 19–20 (App. E.6), summarized in Sec. 4.4(c); cost is also discussed in Limitations (App. F).
W3 & Q3. Uncertainty of performance differences
Three-run mean ± SD. The submitted GPT-5.1 PSCI result was quoted from the original paper. We reproduced PSCI using its published prompts under the same inference and grading protocol as RECAP and PSCI + consolidation. Strict accuracy uses validly evaluated tasks; rubric adherence is micro-averaged over valid items. We report mean ± sample SD over three runs:
| Backbone | Method | Strict | Rubric |
|---|---|---|---|
| GPT-5.1 | Direct | 22.18 ± 0.43 | 82.05 ± 0.37 |
| PSCI (reproduced) | 28.05 ± 0.55 | 81.02 ± 0.82 | |
| PSCI + consolidation | 27.90 ± 0.46 | 82.57 ± 0.52 | |
| RECAP | 27.78 ± 0.81 | 83.31 ± 0.84 | |
| Claude Opus-4.8 | Direct | 21.09 ± 0.16 | 81.71 ± 0.12 |
| PSCI | 26.34 ± 0.45 | 83.33 ± 0.13 | |
| PSCI + consolidation | 26.07 ± 0.36 | 83.08 ± 0.31 | |
| RECAP | 27.04 ± 0.12 | 83.78 ± 0.21 | |
| Qwen3.5-4B | Direct | 10.70 ± 0.19 | 70.49 ± 0.28 |
| PSCI | 11.40 ± 0.14 | 71.14 ± 0.45 | |
| PSCI + consolidation | 11.23 ± 0.12 | 70.47 ± 0.48 | |
| RECAP | 11.80 ± 0.43 | 70.61 ± 0.55 | |
| Qwen3.5-9B | Direct | 11.74 ± 0.10 | 71.91 ± 0.12 |
| PSCI | 14.56 ± 0.78 | 74.99 ± 0.40 | |
| PSCI + consolidation | 14.17 ± 0.56 | 74.63 ± 0.27 | |
| RECAP | 15.60 ± 0.53 | 75.53 ± 0.46 |
Task-paired bootstrap 95% CIs (pp). Δ is the difference between three-run means. CIs resample tasks, with per-task scores averaged over three runs and only tasks valid under both methods paired; GPT-5.1 uses our PSCI reproduction. For strict accuracy: Claude Opus-4.8 RECAP − Direct +5.95 [4.45, 7.41]; Qwen3.5-9B RECAP − Direct +3.86 [2.64, 5.06]; Qwen3.5-4B RECAP-RL + RECAP − RECAP +2.37 [1.08, 3.62]. Comparisons with PSCI:
| Comparison | Strict Δ [95% CI] | Rubric Δ [95% CI] |
|---|---|---|
| Claude Opus-4.8: RECAP − PSCI | +0.70 [−0.85, +2.25] | +0.45 [−0.12, +1.01] |
| GPT-5.1: RECAP − PSCI | −0.27 [−1.84, +1.30] | +2.29 [+1.73, +2.87] |
| Qwen3.5-4B: RECAP − PSCI | +0.40 [−0.83, +1.62] | −0.53 [−1.23, +0.19] |
| Qwen3.5-9B: RECAP − PSCI | +1.04 [−0.27, +2.33] | +0.54 [−0.12, +1.20] |
| Qwen3.5-4B: RECAP-RL + RECAP − PSCI | +2.77 [+1.62, +3.93] | +5.03 [+4.34, +5.71] |
| Qwen3.5-9B: RECAP-RL + RECAP − PSCI | +3.40 [+2.04, +4.76] | +4.81 [+4.16, +5.44] |
Inference-only RECAP is comparable to or modestly better than PSCI: strict Δ is positive on three of four backbones, but no strict CI excludes zero; only the GPT-5.1 rubric CI does. The complete RECAP + RECAP-RL framework improves over PSCI by +2.77/+3.40 pp strict and +5.03/+4.81 pp rubric on Qwen3.5-4B/9B, with all CIs excluding zero.
In the revision: Tables 14–15 (App. E.2); PSCI comparisons in main-text Table 2 and Sec. 4.2.
Response to Reviewer wiQu — Part 3/3
Q4. PSCI protocol and dataset consistency
Rubrics: official CL-bench release has 500 contexts, 1,899 tasks, 31,607 rubric items; PSCI reports 31,592 items (15 fewer), without enough information to identify the cause.
Published versus reproduced PSCI (GPT-5.1, three runs): PSCI reports 28.14 ± 0.50 strict / 81.13 ± 0.90 rubric; our reproduction achieves 28.05 ± 0.55 / 81.02 ± 0.82, differing by only −0.09 / −0.11 points. Table 1 quotes the published PSCI GPT-5.1 and GPT-4.1 Ctx2Skill results (†); Tables 2 and 14 use our reproduction.
GPT-5.1 Direct baseline: our 22.18 ± 0.43 strict / 82.05 ± 0.37 rubric differs from PSCI's 22.55 ± 0.60 / 76.01 ± 1.20. We use released system/user messages without additional prompts, GPT-5.1 at medium effort, a 32,768-token cap, and up to three retries. The official CL-bench verifier (gpt-5.1-2025-11-13, low effort) gives binary judgments for each rubric item, counting items without an explicit “yes” as unmet. Rubric adherence is micro-averaged.
In the revision: App. E.3 documents these protocol details, referenced in Sec. 4.1.
Dear Reviewer wiQu,
Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Response to Reviewer wiQu — Part 1/3
We thank the reviewer and jointly address novelty, cost, statistical reliability, and protocol consistency with new controlled analyses.
W1 & Q1. Novelty beyond PSCI + consolidation
We do not claim novelty for the extract–condition–repair skeleton. PSCI already reports that 83% of residual failed rubrics were in its contract but not executed (Sec. 6.2). RECAP addresses how to recover broad coverage while constructing an executable, compact requirement representation. Our central finding concerns the interaction between multi-view recovery and consolidation, and the resulting representation is designed for both reliable execution and effective learning.
New matched-protocol, cross-backbone comparison (three-run means; strict / rubric; now main-text Table 2):
On GPT-5.1, PSCI + consolidation slightly exceeds RECAP in strict accuracy (27.90 vs. 27.78), while RECAP gains +0.74 rubric. On the other three backbones, PSCI + consolidation reduces both metrics relative to PSCI, whereas full RECAP improves on PSCI + consolidation. Thus, consolidation appears more useful after broad recovery than after a single pass.
New common-skeleton 2×2 experiment. On 197 Claude Opus-4.8 tasks, we crossed single-pass vs. four-view extraction with consolidation on/off, holding the answer prompt fixed without auditing or patching. GPT-5.5 assessed representations against official rubrics blind to the arm; strict accuracy uses the 191 tasks graded in all arms.
Multi-view raises ground-truth coverage by +5.86 pp (C−A; 95% paired CI [3.95, 7.85]) and still by +2.88 pp after consolidation (D−B). Consolidation cuts 60.93 → 14.01 requirements (−77%) and redundancy by 61.1 pp while retaining 90.6% of covered rubrics. Their combination (D) yields the highest observed strict accuracy (17.28%), although C retains higher macro answer-rubric accuracy (75.52% vs. 74.26%).
New matched PSCI-conditioned RL (strict, %; same data, reward, GRPO, and 69 steps; only requirement construction changed; App. E.9, Table 24):
PSCI-conditioned RL gains only +0.31/+0.84 pp over base PSCI, versus +2.37/+2.36 pp for RECAP-RL over base RECAP: requirements missed by one-pass extraction never enter the conditioning representation, so the policy learns to execute only what was extracted. Under RECAP inference, it trails RECAP-RL by 2.71/1.76 pp, is slightly below base RECAP on Qwen3.5-4B (−0.34 pp), and does not exceed No-representation RL on Qwen3.5-9B (16.20 vs. 16.69). Requirement construction must therefore be designed for learnability, not recovery alone.
Discovery, learning, and transfer. Feedback-driven validation selects the design (Table 5(b); Fig. 7); our claim is the discovered recover-then-consolidate principle, not the search procedure. Under RECAP inference, RECAP-RL also beats No-representation RL (17.96 vs. 16.69 strict; Table 11), and self-constructed training requirements outperform GPT-5.5-generated ones (17.96 vs. 15.97; Table 6), suggesting representation–executor alignment. RECAP improves complex instruction following (54.7 vs. 52.5 PSCI, Claude Opus-4.8; Table 4), and text-only RECAP-RL transfers to MCL-bench (11.89 → 21.79 strict, Qwen3.5-9B; Table 3).
In the revision: contribution in Sec. 1; matched comparison in Table 2; 2×2 experiment in Sec. 4.2 and App. E.4 (Table 17).