- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CoDepth, a framework for generating coding benchmarks by programmatically composing existing atomic tasks without language-model generation or validation. It represents tasks as nodes in a graph, with edges indicating which tasks can be combined, and constructs higher-level tasks by selecting connected groups. Domain-specific admission checks validate the resulting tasks and their reference solutions. The framework is instantiated in four coding domains: multi-bug repair, proof completion, complexity analysis, and clone equivalence.
The authors evaluate GPT-5.6-Sol, Kimi-K3, and Qwen3.5-9B on the generated benchmarks and observe that performance generally declines as composition depth increases. They analyze several failure patterns, including cases where constituent bugs can be solved individually but not jointly, and argue that composition introduces challenges beyond atomic task difficulty. Finally, rejection-sampling fine-tuning experiments on multi-bug repair and clone equivalence show that the generated tasks can also provide useful training data for coding agents.
Critical strengths
-
The paper addresses an important challenge in constructing challenging coding benchmarks. Composing existing tasks provides a practical way to reuse code, reference solutions, and verification infrastructure, while varying task complexity without additional language-model generation.
-
The approach is instantiated across four distinct coding domains, demonstrating its applicability beyond a single task type. The evaluation reveals performance gaps that are less visible on atomic tasks, and the fine-tuning experiments provide additional evidence that the generated tasks can support training.
Critical weaknesses
-
The evaluation does not sufficiently separate composition-specific difficulty from increased workload under fixed budgets. Higher-depth tasks require more reasoning and execution, but the budgets do not scale with depth. This is particularly relevant because the paper reports that Qwen’s higher-depth complexity-analysis failures are almost entirely timeouts, and that the discussed GPT verification failures reach the final repair round. A systematic breakdown of budget-related failures and a budget-sensitivity analysis are missing. The existing failure analyses identify useful patterns, but do not establish their underlying causes. In particular, solving constituent bugs individually but failing jointly does not by itself demonstrate interference, since individual evaluations provide a separate budget for each bug. This limits the evidence supporting the paper’s attribution of performance degradation to compositional reasoning and interference.
-
The model coverage is limited for drawing broad conclusions about frontier models. The experiments include two frontier models and one small open-weight model, with one model per family. The small model’s budget-related failures offer limited additional evidence about stronger models, while the changing model rankings in proof completion suggest that sensitivity to composition varies across models. Evaluation on additional strong model families and a larger open-weight model would strengthen the generality of the findings.
Questions for authors
-
Could you report, for each model and composition level, the fraction of failures associated with token, turn, time, or cost limits, alongside failures that complete with an incorrect answer? For the multi-bug cases classified as interference errors, is there evidence distinguishing incomplete repairs or budget exhaustion from edits that actually disrupt another repair?
-
How much of the performance decline remains when the evaluation budget increases with composition depth, particularly at N=2 and N=3? This would help assess whether the observed gap is largely explained by additional workload or persists with more resources.
-
Have you evaluated additional strong model families or a larger open-weight model? If so, do they exhibit similar depth-related degradation and failure patterns?
4 comments
Q3: More model families and a larger open-weight model
Additional model families. Yes. We extended the proof-completion evaluation to a larger open-weight model, DeepSeek-V4-Pro-0813, and to an additional model family, GLM-5.3 Flash. Both were run at composition levels N = 1 and N = 3, using the original 100-task bank for each level. At N = 1 the model proves a single function. At N = 3 it must establish the contracts of a three-function chain and the assertions in its driver. We keep the paper's AutoVerus protocol unchanged: the same prompts; the generation, refinement and repair phases; up to 10 repair rounds; a 16,000-token limit per call (4,096 for repair calls); and an 80-minute limit per task.
Both models show the same depth-related degradation as the models in the paper. From N = 1 to N = 3, DeepSeek-V4-Pro-0813 drops from 91.0% to 33.0% and GLM-5.3 Flash from 84.0% to 27.0%, losses of 58 and 57 points. Even the larger open-weight model loses about two thirds of its N = 1 success rate.
| Model | N = 1 | N = 3 |
|---|---|---|
| DeepSeek-V4-Pro-0813 | 91.0 | 33.0 |
| GLM-5.3 Flash | 84.0 | 27.0 |
Proof success rate (%), 100 tasks per level.
Thank you again for your thoughtful review. We've posted responses addressing each of the points you raised and wanted to check in as the discussion period continues. If anything in our replies is unclear or if you have follow-up questions, we'd be very glad to answer them. Thank you for your time and engagement.
Thank you for the detailed responses and additional experiments. The stop-reason analysis, concrete trajectory examples, and results on additional model families substantially address my main concerns.
It remains somewhat unclear how much of the performance decline comes from having more work to do versus interactions between the combined tasks. I encourage the authors to clarify this limitation and incorporate the new analyses into the paper. Overall, I find the task-generation approach useful, and the additional evidence strengthens its contribution. I will raise my score.
We thank the reviewer for focusing on what drives the depth effect. We answer from the multi-bug runs, where every run's stop reason was logged, and from the agents' own trajectories.
W1 / Q1: Failures by budget limit, and evidence of interference
Stop reasons. The budget is 100 turns, $1.50 and 1500 s at every level. There is no separate token limit; tokens are bounded through cost.
No logged GPT-5.6-Sol run was stopped by a budget limit. For Kimi-K3, a substantial decline remains in both summaries. Under (A), accuracy among runs that finish within budget falls from 94.3% at L1 to 57.0% at L3. Under (B), counting every budget-limited failure as solved while holding all other outcomes fixed, accuracy falls from 95.0% to 67.3%. The L1-to-L3 drop is 27.7 percentage points under this scenario, compared with 38.5 points in the observed results.
On Qwen3.5-9B: we do not use it as evidence about composition, because it is a small 9B model and we expected it to struggle on many of these tasks regardless of depth. Its results confirm this: most of its failures are turn or time limits at every level, including L1, so its depth curve mainly reflects limited capacity. We include it because it is the base model for our RFT experiments (§5) and serves as a small open-weight reference point. Our conclusions about composition rest on the two frontier models.
What interference means here. By interference we mean every way in which the presence of the other constituents makes a bug harder to repair than it is alone, not only edits that regress or conflict with another repair. We reconstructed 82 failed Kimi-K3 runs and tracked each bug's tests across the agent's own test runs. The trajectories show each form of interference:
This also answers the reviewer's question of how to tell interference apart from incomplete repairs and budget exhaustion:
We will add this trajectory analysis to §4.2, alongside the interference shares.
Q2: Depth-scaled budget
We could not complete a depth-scaled budget experiment within the rebuttal window. No logged GPT-5.6-Sol run was stopped by a turn, cost, or wall-clock limit. For Kimi-K3, counting every budget-limited failure as solved while holding all other outcomes fixed reduces the N=1 to N=3 drop from 38.5 to 27.7 percentage points. This scenario shows what would happen if only the runs stopped by a limit improved. We will rerun the agents with budgets that scale with depth and report the results in the revision.
We also see fixed budgets as part of the benchmark design. A useful benchmark has to bound cost, and an unbounded budget measures persistence rather than capability. Composed tasks are meant to be harder within the same cost envelope.