- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CapSynth, a capability-driven framework for synthesizing long-context, multi-constraint instruction-following data. It generates tasks and scoring rubrics from decomposed capabilities, followed by verification and difficulty filtering. Experiments demonstrate improvements on CL-bench and CL-bench Life through RFT and GRPO training.
Critical strengths
-
Structured capability-guided synthesis. CapSynth constructs content-grounded, multi-constraint tasks rather than simply stacking surface-level constraints, providing a scalable approach to complex instruction-following data generation.
-
Promising empirical improvements. The method achieves substantial gains across different backbones. The capability ablation further supports the effectiveness of capability-guided synthesis.
Critical weaknesses
-
Unverified compositional complexity of synthesized tasks. CapSynth assumes that sampling multiple atomic capabilities produces genuinely entangled constraints. However, combining capability labels does not necessarily create interdependent reasoning requirements. The paper lacks direct evidence that synthesized tasks require meaningful cross-constraint reasoning rather than independent constraint satisfaction.
-
Insufficient isolation of the capability decomposition contribution. The ablation compares capability-guided synthesis against synthesis without capability guidance, but does not establish whether the fine-grained 185-item hierarchy is necessary. Simpler capability lists or complexity-matched synthesis strategies might provide similar benefits, leaving the specific contribution of the proposed decomposition unclear.
-
Limited evidence of compositional generalization. Improvements on CL-bench and CL-bench Life demonstrate training effectiveness but do not establish whether models acquire transferable compositional capabilities. The paper lacks systematic evaluation on unseen capability combinations, varying constraint densities, or increasingly complex reasoning dependencies.
Questions for authors
-
How do the authors verify that synthesized tasks contain genuinely interdependent constraints rather than independently satisfiable requirements? Could they provide quantitative analyses of constraint dependencies or reasoning complexity?
-
Why is the proposed fine-grained capability hierarchy necessary? Could the authors compare it with a simpler flat capability list or alternative capability sampling strategies under matched data and training budgets?
-
Could the authors evaluate compositional generalization by training on certain capability combinations and testing on unseen combinations? This would help establish whether CapSynth improves transferable capabilities rather than task-specific adaptation.
-
How does performance scale with context length, constraint density, and reasoning complexity? Could the authors provide stratified evaluations to determine whether the improvements persist on the most challenging tasks?
3 comments
- Why is the proposed fine-grained capability hierarchy necessary? Could the authors compare it with a simpler flat capability list or alternative capability sampling strategies under matched data and training budgets?
As we mentioned in the response to question 2, we want to find a scalable way to automatically generate long-context, reasoning-driven instruction-following tasks. And we find that CapSynth (using capability decomposition and combining them) can generate such tasks. Maybe a simpler flat capability list can also generate such tasks, but it is not the focus of this paper.
- Could the authors evaluate compositional generalization by training on certain capability combinations and testing on unseen combinations? This would help establish whether CapSynth improves transferable capabilities rather than task-specific adaptation.
As we mentioned in the response to question 3, the results on CL-bench and CL-bench Life can prove such generalization.
- How does performance scale with context length, constraint density, and reasoning complexity? Could the authors provide stratified evaluations to determine whether the improvements persist on the most challenging tasks?
We compare the performance of the model with different context lengths and different numbers of rubrics. The results are shown in the following table:
| Δ pass@1 | ctx short | ctx mid | ctx long |
|---|---|---|---|
| few rubrics | +0.00 (n=30) | +8.09 (n=204) | +3.37 (n=304) |
| mid rubrics | +16.67 (n=120) | +11.56 (n=225) | +6.41 (n=230) |
| many rubrics | +15.79 (n=483) | +11.64 (n=204) | +5.05 (n=99) |
This result proves that CapSynth can improve the model's performance with different context lengths and different numbers of rubrics, especially on tasks with many rubrics.
A Kind Reminder for Reading the Response
Dear Reviewer M8Cc,
Thank you for your comments.
As the rebuttal period is ending soon, we would greatly appreciate it if you could take a final look at our responses and see whether they address your concerns. Thank you for your time and consideration.
The Authors
Thanks for your review. We have addressed the concerns as follows:
If a task's rubrics were independently satisfiable, then across repeated rollouts each rubric's outcome would be statistically independent of the others, and the all-pass rate would equal ∏i pi. We measure within-task rubric coupling as the mean pairwise Pearson φ between rubric outcome vectors across rollouts, and compare it against a permutation null that shuffles each rubric's outcomes across rollouts independently.
Based on the results, we can see that the synthesized tasks are more entangled than the expert benchmarks, so we can conclude that the synthesized tasks have compositional complexity.
We want to clarify that our motivation is to find a scalable way to automatically generate long-context, reasoning-driven instruction-following tasks. Most academic and industrial teams have tried to deal with it but failed.
We find that capability decomposition is a good way to generate such problems. We believe that's enough to prove the effectiveness of our method.
We do not deny that using easier or more complex capabilities may also improve the performance of the model. But this is not the focus of this paper. It can be studied in the future.
The distribution of CL-Bench's capability combinations is not the distribution of the synthesized tasks (at the capability level, it is similar, but at the combination level, it is not). So the improvement on CL-Bench can prove such generalization.
This method probably can not generalize to unseen capabilities, but we believe the capability set is finite, and can be enumerated. So if the capability set is not comprehensive, we suggest just expanding the capability set.
On the fully auditable Qwen3.6 chain, we compare the final result under different numbers of rubrics:
This result can prove that CapSynth can improve the model's performance with different numbers of rubrics.
When the pipeline just makes the plan, it will consider the capabilities' dependencies and design related tasks. It may have independent requirements; for example, if you need to submit a report about 5 different topics, each topic has one 100-word sentence summary; the length and the topic are not independent, but the 5 topics are independent. We do not consider this kind of dependency to be a bad pattern.