acceptodds
Under review as a conference paper at ICLR 2027

CapSynth: Capability-Decomposed Synthesis for Long-Context Multi-Constraint Instruction Following

Abstract

Complex real-world tasks increasingly require language models to acquire new knowledge from long and noisy contexts while simultaneously satisfying large numbers of mutually constraining requirements. Existing work on instruction following leaves a systematic gap along this dimension, which produces constraints that are largely orthogonal to task content, attachable to arbitrary questions with- out any understanding of the material. Benchmarks whose constraint density approaches realistic settings rely on expert manual construction, which does not scale. We present CapSynth, an automated synthesis framework targeting long- context, reasoning-driven multi-constraint instruction-following tasks. CapSynth begins from a decomposition of the capability space into atomic capability items organized under several capability families, and synthesis is organized in reverse: we first determine the capability combination a question should cover, then gen- erate context materials and scoring criteria matched to that combination, so that the information density of the material serves the target capabilities. Generated instances pass through layered verification, including execution-based checks and grounding filters that help reduce inconsistencies between the material and scoring rubrics. We evaluate CapSynth on two long-context multi-constraint benchmarks. On a base model without instruction tuning, ten epochs of training produce strictly monotonic improvement on CL-bench. On the strongest backbone, the trained model surpasses its instruct version via reinforcement learning. Both results show that CapSynth data synthesis is high quality.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Instruction Following, long context

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_M8CcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CapSynth, a capability-driven framework for synthesizing long-context, multi-constraint instruction-following data. It generates tasks and scoring rubrics from decomposed capabilities, followed by verification and difficulty filtering. Experiments demonstrate improvements on CL-bench and CL-bench Life through RFT and GRPO training.

Critical strengths
  • Structured capability-guided synthesis. CapSynth constructs content-grounded, multi-constraint tasks rather than simply stacking surface-level constraints, providing a scalable approach to complex instruction-following data generation.

  • Promising empirical improvements. The method achieves substantial gains across different backbones. The capability ablation further supports the effectiveness of capability-guided synthesis.

Critical weaknesses
  • Unverified compositional complexity of synthesized tasks. CapSynth assumes that sampling multiple atomic capabilities produces genuinely entangled constraints. However, combining capability labels does not necessarily create interdependent reasoning requirements. The paper lacks direct evidence that synthesized tasks require meaningful cross-constraint reasoning rather than independent constraint satisfaction.

  • Insufficient isolation of the capability decomposition contribution. The ablation compares capability-guided synthesis against synthesis without capability guidance, but does not establish whether the fine-grained 185-item hierarchy is necessary. Simpler capability lists or complexity-matched synthesis strategies might provide similar benefits, leaving the specific contribution of the proposed decomposition unclear.

  • Limited evidence of compositional generalization. Improvements on CL-bench and CL-bench Life demonstrate training effectiveness but do not establish whether models acquire transferable compositional capabilities. The paper lacks systematic evaluation on unseen capability combinations, varying constraint densities, or increasingly complex reasoning dependencies.

Questions for authors
  1. How do the authors verify that synthesized tasks contain genuinely interdependent constraints rather than independently satisfiable requirements? Could they provide quantitative analyses of constraint dependencies or reasoning complexity?

  2. Why is the proposed fine-grained capability hierarchy necessary? Could the authors compare it with a simpler flat capability list or alternative capability sampling strategies under matched data and training budgets?

  3. Could the authors evaluate compositional generalization by training on certain capability combinations and testing on unseen combinations? This would help establish whether CapSynth improves transferable capabilities rather than task-specific adaptation.

  4. How does performance scale with context length, constraint density, and reasoning complexity? Could the authors provide stratified evaluations to determine whether the improvements persist on the most challenging tasks?

3 comments

Thanks for your review. We have addressed the concerns as follows:

  1. Unverified compositional complexity of synthesized tasks.

If a task's rubrics were independently satisfiable, then across repeated rollouts each rubric's outcome would be statistically independent of the others, and the all-pass rate would equal ∏i_i pip_i. We measure within-task rubric coupling as the mean pairwise Pearson φ between rubric outcome vectors across rollouts, and compare it against a permutation null that shuffles each rubric's outcomes across rollouts independently.

Task settasksobserved all-pass∏i_i pip_i(independence)dependence ratiomean pairwise φpermutation null
CL-bench (expert)1,89924.47%22.61%1.08+0.233−0.002
CL-bench Life (expert)40511.85%10.44%1.14+0.246+0.001
CapSynth v6 pool (ours)19,88316.30%9.10%1.79+0.459−0.0002

Based on the results, we can see that the synthesized tasks are more entangled than the expert benchmarks, so we can conclude that the synthesized tasks have compositional complexity.

  1. Insufficient isolation of the capability decomposition contribution.

We want to clarify that our motivation is to find a scalable way to automatically generate long-context, reasoning-driven instruction-following tasks. Most academic and industrial teams have tried to deal with it but failed.

We find that capability decomposition is a good way to generate such problems. We believe that's enough to prove the effectiveness of our method.

We do not deny that using easier or more complex capabilities may also improve the performance of the model. But this is not the focus of this paper. It can be studied in the future.

  1. Limited evidence of compositional generalization.

Generalize to unseen capability combinations.

The distribution of CL-Bench's capability combinations is not the distribution of the synthesized tasks (at the capability level, it is similar, but at the combination level, it is not). So the improvement on CL-Bench can prove such generalization.

This method probably can not generalize to unseen capabilities, but we believe the capability set is finite, and can be enumerated. So if the capability set is not comprehensive, we suggest just expanding the capability set.

Generalize to different numbers of rubrics.

On the fully auditable Qwen3.6 chain, we compare the final result under different numbers of rubrics:

#rubricstasksbase+CapSynthΔ [95% CI]per-rubric satisfaction
≤34119.51%26.83%+7.32 [−1.22, +17.07]47.4% → 59.1%
4–510616.51%19.81%+3.30 [−2.12, +9.20]49.4% → 61.2%
6–718812.63%16.89%+4.26 [0.00, +8.38]53.8% → 67.1%
8–1034410.61%20.49%+9.88 [+7.05, +12.79]56.0% → 75.5%
11–1443413.02%22.00%+8.99 [+6.45, +11.58]61.6% → 76.8%
≥1578616.54%29.90%+13.36 [+11.32, +15.39]72.6% → 87.5%

This result can prove that CapSynth can improve the model's performance with different numbers of rubrics.

  1. How do the authors verify that synthesized tasks contain genuinely interdependent constraints rather than independently satisfiable requirements? Could they provide quantitative analyses of constraint dependencies or reasoning complexity?

When the pipeline just makes the plan, it will consider the capabilities' dependencies and design related tasks. It may have independent requirements; for example, if you need to submit a report about 5 different topics, each topic has one 100-word sentence summary; the length and the topic are not independent, but the 5 topics are independent. We do not consider this kind of dependency to be a bad pattern.

  1. Why is the proposed fine-grained capability hierarchy necessary? Could the authors compare it with a simpler flat capability list or alternative capability sampling strategies under matched data and training budgets?

As we mentioned in the response to question 2, we want to find a scalable way to automatically generate long-context, reasoning-driven instruction-following tasks. And we find that CapSynth (using capability decomposition and combining them) can generate such tasks. Maybe a simpler flat capability list can also generate such tasks, but it is not the focus of this paper.

  1. Could the authors evaluate compositional generalization by training on certain capability combinations and testing on unseen combinations? This would help establish whether CapSynth improves transferable capabilities rather than task-specific adaptation.

As we mentioned in the response to question 3, the results on CL-bench and CL-bench Life can prove such generalization.

  1. How does performance scale with context length, constraint density, and reasoning complexity? Could the authors provide stratified evaluations to determine whether the improvements persist on the most challenging tasks?

We compare the performance of the model with different context lengths and different numbers of rubrics. The results are shown in the following table:

Δ pass@1ctx shortctx midctx long
few rubrics+0.00 (n=30)+8.09 (n=204)+3.37 (n=304)
mid rubrics+16.67 (n=120)+11.56 (n=225)+6.41 (n=230)
many rubrics+15.79 (n=483)+11.64 (n=204)+5.05 (n=99)

This result proves that CapSynth can improve the model's performance with different context lengths and different numbers of rubrics, especially on tasks with many rubrics.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer M8Cc,

Thank you for your comments.

As the rebuttal period is ending soon, we would greatly appreciate it if you could take a final look at our responses and see whether they address your concerns.  Thank you for your time and consideration. 

The Authors

Reviewer_ZqJyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets instruction-following tasks that combine two properties under-served by existing benchmarks and synthesis pipelines: (i) the constraints can only be satisfied by first acquiring new knowledge from a long, noisy context, and (ii) the number of constraints per instance is large and the constraints are mutually entangled rather than independently attachable. The proposed framework, CapSynth, inverts the usual synthesis order. It starts from a three-level capability codebook (13 families / 47 groups / 185 atomic items, with per-family sampling probabilities, per-family caps, and 14 registered mutual-exclusion pairs), samples a specification σ_t (capability set, presentation form, length band, number of user turns, target rubric count, conditional/pressure flags), and only then produces the reference material (either retrieved from a local corpus or synthesized), a task design plan, and finally the problem together with self-contained rubrics. Three verification layers follow (execution-based repair of numerical rubrics, a grounding filter, a rule-base filter), plus a difficulty-calibration step. Training uses rejection-sampled RFT followed by GRPO with a rubric-level partial-credit reward and a full-pass bonus (β = 0.5). Reported results on CL-bench / CL-bench Life: Qwen3.6-27B 14.34 → 23.34 (RFT) → 24.47 (RFT+RL); Qwen3.8-27B 24.94 → 33.86 (RL); a Qwen3-30B-A3B base model improves monotonically from 6.53 to 15.81 over ten epochs; IFBench moves 79.5 → 80.7.

Critical strengths
  1. The codebook appears to be fairly reusable, but its source is not clearly specified.
  2. The overall motivation is reasonably clear, and the design of the method section largely aligns with that motivation.
Critical weaknesses
  1. The evaluation lacks baselines and covers too few benchmarks, leaving the method without a fair comparison; the overall design therefore risks overfitting to the CL benchmark.
  2. The pipeline is long yet insufficiently ablated, and the dataset is too short on average ("mean prompt 47,025 characters"); the proposed way of constructing the materials is also unlikely to generalize to much longer contexts.
Questions for authors
  1. How were the 185 items in the codebook obtained?
  2. How is an appropriate real document selected for a given set of items, given that some real documents may not be well suited to the selected items?
  3. Given multiple rubrics, is a single question generated in the end, or multiple questions? How do you ensure the generated question(s) cover all rubrics?
  4. The upper bound on the length of the synthetic data for this task remains unexplored.
6 comments

Thanks for your review. We have addressed the concerns as follows:

  1. Insufficient isolation of the capability decomposition contribution.

Capsynth mostly focuses on long-context, reasoning-driven multi-constraint instruction-following tasks. Beyond CL-bench, we also prove that CapSynth can improve the model's performance on IFBench. This proves that CapSynth does not only focus on CL-bench, but also can improve the model's performance on other instruction-following tasks.

We also use this in our internal midtraining experiments; it can improve model's instruction-following performance while does not harm the performance on other tasks.

  1. The proposed way of constructing the materials is also unlikely to generalize to much longer contexts.

The average length of the synthetic data mostly depends on the predefined settings in Figure 2 (draw the framing). Since we can iteratively generate the context (generate it chunk by chunk), the length of the synthetic data is not limited.

So the dataset length is not the limitation of CapSynth, but a predefined setting. CapSynth can generate tasks with much longer context.

  1. How were the 185 items in the codebook obtained?

We selected thousands of internal long-context, reasoning-driven instruction-following tasks as the codebook, using LLMs to note what capabilities are needed for the tasks (humans will also give some suggestions). Then we cluster the tree-structured capabilities into 185 items.

  1. How is an appropriate real document selected for a given set of items, given that some real documents may not be well suited to the selected items?

We just randomly select some real documents from a large corpus. If the document is not well suited to the selected items, we will use the synthetic data pipeline to generate the context instead. Synthesized-material tasks did not use the real document, so they are not bound by the real document.

We do have a small pre-experiment to evaluate the performance of the Synthesized-material tasks and the Retrieved-material tasks. The results are quite similar. (But since we have already generated some Retrieved-material tasks, we just use them in the paper)

  1. Given multiple rubrics, is a single question generated in the end, or multiple questions?

It's a single question. But we think it should be considered as a "task," not a "question". For example, in the introduction, we present a task like "A legal assistant may be asked to review a thirty-page contract, verify whether six rights the client asserts are supported by specific clauses, identify disputed language, and produce a formal written opinion without citing precedents outside the contract text." in this example:

Question is:

  • "review a thirty-page contract" and give the review report.

Rubrics may include:

  • read what "six rights" means in the context and check it one by one (to judge if some rights are supported by the contract or not)

  • "Identify disputed language" will also be a rubric.

  • produce a formal written opinion without citing precedents outside the contract text.

Such context will be in the materials, so there will be a lot of rubrics with a single question.

  1. How do you ensure the generated question(s) cover all rubrics?

Each rubric will link the raw context inside the contexts (materials, question, etc). We will just check if such context is in the materials or question; if it needs to be calculated, we will calculate it via Python. So the whole context covers all rubrics.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer ZqJy,

Thank you for your comments.

As the rebuttal period is ending soon, we would greatly appreciate it if you could take a final look at our responses and see whether they address your concerns.  Thank you for your time and consideration. 

The Authors

I have read the authors' reply in detail. I remain concerned about the reproducibility of the method, as well as its behavior on additional benchmarks and at longer lengths. The reply does not add any new information on these points.

Reviewer_YUiHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CapSynth, a data synthesis framework designed for long-context, multi-constraint instruction-following tasks. Unlike prior methods that primarily focus on relatively independent constraints such as format and length, CapSynth generates complex tasks that require reasoning over specific materials while simultaneously satisfying multiple interdependent conditions. The work constructs a hierarchical capability system comprising 13 major capability categories, 47 capability groups, and 185 atomic capability items, which is further applied to Rejection Sampling Fine-Tuning (RFT) and GRPO-based reinforcement learning. The paper reports improvements on CL-bench and CL-bench Life, including capability-guided ablation studies, baseline training curves, IFBench checks, and comparisons between alternative judge protocols and GPT-5.1 judgments.

Critical strengths
  1. Provides ablation results supporting the core design. CapSynth achieves strong results on CL-bench and CL-bench Life. Specifically, Qwen3.8-27B improves pass@1 by 8.92 percentage points on CL-bench. In Table 2, RFT with capability decomposition achieves 23.34%, compared to 17.07% without it. This difference provides direct evidence for the effectiveness of capability-guided data synthesis. The authors also include supplementary analyses on evaluation consistency and conduct preliminary IFBench checks on post-training instruction-following capabilities, which enhance the credibility of the results.
  2. Emphasizes interdependencies among constraints rather than simple accumulation. Instead of treating complex instruction-following as merely increasing the number of constraints, the paper focuses on the dependencies between different constraints and their connections to reference materials. This framing more closely reflects the characteristics of complex tasks in real-world applications (e.g., a lawyer extracting and arguing information from dozens of pages of materials), thereby providing clearer motivation for the design of capability combinations and task construction.
Critical weaknesses
  1. Insufficient validation of correctness and capability coverage for automatically generated training supervision. CapSynth relies on model-generated scoring rubrics as training rewards but lacks independent validation of their correctness and capability coverage. Appendix C further indicates that the numerical verification prompts were not actually used, making it unclear whether the relevant rubrics were adequately verified. Consequently, the reliability of the synthesized data remains uncertain.
  2. The independent contribution of capability decomposition has not been sufficiently validated. Although Table 2 shows that capability decomposition yields significant improvements, the experiments control for data volume and training configuration. It remains unclear whether the two sets of data are comparable in terms of task difficulty, length, and the number of scoring rubrics. Additionally, the paper lacks sufficient comparisons with related data synthesis methods, making it difficult to accurately determine how much of the performance gain is attributable to capability decomposition itself.
Questions for authors
  1. Lack of fine-grained experiments targeting the core research objective. Could you provide further analysis of CapSynth's improvements across different types of complex constraint tasks? The current experiments mainly report overall pass rates. Could you conduct a fine-grained analysis by constraint count, context length, or the degree of dependency among constraints to demonstrate whether CapSynth shows more pronounced advantages on tasks requiring multi-condition joint reasoning?

  2. What is the independent advantage of CapSynth compared to existing data synthesis methods? The existing experiments primarily compare different baselines and internal ablations. Could you further elaborate on the key differences between CapSynth and methods such as RECAST, DecIF, and LongCrafter? Where feasible, could you provide experimental comparisons under unified conditions to clarify CapSynth's independent contributions?

  3. How is the correctness of automatically generated scoring rubrics and supervision signals ensured? Did the authors conduct independent manual spot checks on the generated tasks and scoring rubrics? Could you provide statistics on the accuracy of the rubrics, the actual coverage of target capabilities, and the proportion of programmatic verification for numerical rubrics? In particular, given that the verification prompts mentioned in Appendix C were not actually used, what impact did this have on data quality?

2 comments

Thanks for your review. We have addressed the concerns as follows:

  1. Insufficient validation of correctness and capability coverage for automatically generated training supervision.

All rubrics will be double check its capability coverage and assure its reference is in the reference context. If the capability is about num/ml, the checker will also write Python code to verify the number is correct. This is not in the original problem generation process, apologies for the confusion. As a result, at least the rubrics cover the capability.

Since we need to use rejection sampling to filter the wrong answer, the correct answer passes all rubrics, so it "covers all capabilities"

  1. The independent contribution of capability decomposition has not been sufficiently validated.

It's hard to tell the task difficulty between differnet settings, but we can tell the average length, and the number of scoring rubrics is comparable since we just use a similar prompt (remove cap part) and also "draw the framing" (Figure 2), so such metrics are quite similar.

  1. Lack of fine-grained experiments targeting the core research objective.

On the fully auditable Qwen3.6 chain, we compare the final result under different numbers of rubrics:

#rubricstasksbase+CapSynthΔ [95% CI]per-rubric satisfaction
≤34119.51%26.83%+7.32 [−1.22, +17.07]47.4% → 59.1%
4–510616.51%19.81%+3.30 [−2.12, +9.20]49.4% → 61.2%
6–718812.63%16.89%+4.26 [0.00, +8.38]53.8% → 67.1%
8–1034410.61%20.49%+9.88 [+7.05, +12.79]56.0% → 75.5%
11–1443413.02%22.00%+8.99 [+6.45, +11.58]61.6% → 76.8%
≥1578616.54%29.90%+13.36 [+11.32, +15.39]72.6% → 87.5%

We also compare the final result under different numbers of rubrics x different context length:

Δ pass@1ctx shortctx midctx long
few rubrics+0.00 (n=30)+8.09 (n=204)+3.37 (n=304)
mid rubrics+16.67 (n=120)+11.56 (n=225)+6.41 (n=230)
many rubrics+15.79 (n=483)+11.64 (n=204)+5.05 (n=99)

These results can prove that CapSynth can improve the model's performance with different context lengths and different numbers of rubrics.

  1. What is the independent advantage of CapSynth compared to existing data synthesis methods?

CapSynth can automatically synthesize long-context, reasoning-driven, multi-constraint instruction-following tasks. None of these methods can do it at scale. Most of the methods only deal with single-constraint, short-context problems. And most of the time the context is orthogonal to the constraint.

  1. How is the correctness of automatically generated scoring rubrics and supervision signals ensured?

As we mentioned in the response to question 1, all rubrics will be double check its capability coverage and assure its reference is in the reference context. We have also double-checked its correctness when we constructed this pipeline, and revised the pipeline if there were any issues. So the authors checked about 100 final tasks used in the training, and all of them are correct.

About 75% of the questions will be verified by numerical verification.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer YUiH,

Thank you for your comments.

As the rebuttal period is ending soon, we would greatly appreciate it if you could take a final look at our responses and see whether they address your concerns.  Thank you for your time and consideration. 

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.