acceptodds
Under review as a conference paper at ICLR 2027

CoDepth: Model-Free Generation of Compositional Coding Benchmarks

Abstract

High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typically rely on costly model generation and validation. In this work, we show that new coding problems can be created at almost no cost by composing existing ones. We formulate problem generation as code composition: verified atomic problems are combined into level- tasks, and a task is kept only if every atomic problem remains unsolved and a joint reference solution passes the same check used at evaluation time. The entire task-construction process is programmatic and requires no language-model generation or validation. We instantiate this approach in four coding settings: multi-bug repair, proof completion, complexity analysis, and clone equivalence. The generated tasks are challenging for frontier models, which all degrade as compositional depth increases. The generated tasks are also useful as training data: rejection-sampling fine-tuning on them improves Qwen3.5-9B at every composition level.

Primary area:
datasets and benchmarks
Keywords:
benchmark, llm evaluation, code generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U3XxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CoDepth, a framework for generating coding benchmarks by programmatically composing existing atomic tasks without language-model generation or validation. It represents tasks as nodes in a graph, with edges indicating which tasks can be combined, and constructs higher-level tasks by selecting connected groups. Domain-specific admission checks validate the resulting tasks and their reference solutions. The framework is instantiated in four coding domains: multi-bug repair, proof completion, complexity analysis, and clone equivalence.

The authors evaluate GPT-5.6-Sol, Kimi-K3, and Qwen3.5-9B on the generated benchmarks and observe that performance generally declines as composition depth increases. They analyze several failure patterns, including cases where constituent bugs can be solved individually but not jointly, and argue that composition introduces challenges beyond atomic task difficulty. Finally, rejection-sampling fine-tuning experiments on multi-bug repair and clone equivalence show that the generated tasks can also provide useful training data for coding agents.

Critical strengths
  1. The paper addresses an important challenge in constructing challenging coding benchmarks. Composing existing tasks provides a practical way to reuse code, reference solutions, and verification infrastructure, while varying task complexity without additional language-model generation.

  2. The approach is instantiated across four distinct coding domains, demonstrating its applicability beyond a single task type. The evaluation reveals performance gaps that are less visible on atomic tasks, and the fine-tuning experiments provide additional evidence that the generated tasks can support training.

Critical weaknesses
  1. The evaluation does not sufficiently separate composition-specific difficulty from increased workload under fixed budgets. Higher-depth tasks require more reasoning and execution, but the budgets do not scale with depth. This is particularly relevant because the paper reports that Qwen’s higher-depth complexity-analysis failures are almost entirely timeouts, and that the discussed GPT verification failures reach the final repair round. A systematic breakdown of budget-related failures and a budget-sensitivity analysis are missing. The existing failure analyses identify useful patterns, but do not establish their underlying causes. In particular, solving constituent bugs individually but failing jointly does not by itself demonstrate interference, since individual evaluations provide a separate budget for each bug. This limits the evidence supporting the paper’s attribution of performance degradation to compositional reasoning and interference.

  2. The model coverage is limited for drawing broad conclusions about frontier models. The experiments include two frontier models and one small open-weight model, with one model per family. The small model’s budget-related failures offer limited additional evidence about stronger models, while the changing model rankings in proof completion suggest that sensitivity to composition varies across models. Evaluation on additional strong model families and a larger open-weight model would strengthen the generality of the findings.

Questions for authors
  1. Could you report, for each model and composition level, the fraction of failures associated with token, turn, time, or cost limits, alongside failures that complete with an incorrect answer? For the multi-bug cases classified as interference errors, is there evidence distinguishing incomplete repairs or budget exhaustion from edits that actually disrupt another repair?

  2. How much of the performance decline remains when the evaluation budget increases with composition depth, particularly at N=2 and N=3? This would help assess whether the observed gap is largely explained by additional workload or persists with more resources.

  3. Have you evaluated additional strong model families or a larger open-weight model? If so, do they exhibit similar depth-related degradation and failure patterns?

4 comments

We thank the reviewer for focusing on what drives the depth effect. We answer from the multi-bug runs, where every run's stop reason was logged, and from the agents' own trajectories.

W1 / Q1: Failures by budget limit, and evidence of interference

Stop reasons. The budget is 100 turns, $1.50 and 1500 s at every level. There is no separate token limit; tokens are bounded through cost.

ModelLevelAccuracyFailures by budget(A) Accuracy on runs not stopped by the budget(B) Budget-limited failures counted as solved
GPT-5.6-SolL1100%none100%100%
L286.5%none86.5%86.5%
L378.6%none78.6%78.6%
Kimi-K3L182.0%turn 18, cost 894.3%95.0%
L261.5%turn 25, cost 873.7%78.0%
L343.5%turn 32, cost 6, wall clock 257.0%67.3%

No logged GPT-5.6-Sol run was stopped by a budget limit. For Kimi-K3, a substantial decline remains in both summaries. Under (A), accuracy among runs that finish within budget falls from 94.3% at L1 to 57.0% at L3. Under (B), counting every budget-limited failure as solved while holding all other outcomes fixed, accuracy falls from 95.0% to 67.3%. The L1-to-L3 drop is 27.7 percentage points under this scenario, compared with 38.5 points in the observed results.

On Qwen3.5-9B: we do not use it as evidence about composition, because it is a small 9B model and we expected it to struggle on many of these tasks regardless of depth. Its results confirm this: most of its failures are turn or time limits at every level, including L1, so its depth curve mainly reflects limited capacity. We include it because it is the base model for our RFT experiments (§5) and serves as a small open-weight reference point. Our conclusions about composition rest on the two frontier models.

What interference means here. By interference we mean every way in which the presence of the other constituents makes a bug harder to repair than it is alone, not only edits that regress or conflict with another repair. We reconstructed 82 failed Kimi-K3 runs and tracked each bug's tests across the agent's own test runs. The trajectories show each form of interference:

Form of interferenceEvidence
Edits break another repairIn 5 of the 82 runs, a bug's tests passed at some point, later edits broke them, and they still fail at the end. In robotpy/cxxheaderparser (L3), all three bugs pass at step 79; later edits break two of them, and repairing one leaves the third failing.
The combined failing signal hides a root causeIn 72 of the 82 runs, some bug's tests never pass at any point in the episode. The agent never isolates this root cause from the merged failing tests; in 10 of these runs it fully fixes another bug in the same task along the way.
One fix depends on anotherFrom the construction audit (response to AmCS, Q2): 154 fixes apply only on top of another bug's fix, and in 14 bug slots the target test cannot even run until another bug is fixed.

This also answers the reviewer's question of how to tell interference apart from incomplete repairs and budget exhaustion:

  • Budget exhaustion is separated by the stop-reason analysis above.
  • Edits that disrupt another repair are observed directly in the trajectories.
  • Incomplete repairs. A bug solved in isolation but left unresolved in the composed task provides evidence of a difficulty associated with composition. The matched control described in our response to AmCS, Q4 would help distinguish interference from the effect of doing more work in a single run.

We will add this trajectory analysis to §4.2, alongside the interference shares.

Q2: Depth-scaled budget

We could not complete a depth-scaled budget experiment within the rebuttal window. No logged GPT-5.6-Sol run was stopped by a turn, cost, or wall-clock limit. For Kimi-K3, counting every budget-limited failure as solved while holding all other outcomes fixed reduces the N=1 to N=3 drop from 38.5 to 27.7 percentage points. This scenario shows what would happen if only the runs stopped by a limit improved. We will rerun the agents with budgets that scale with depth and report the results in the revision.

We also see fixed budgets as part of the benchmark design. A useful benchmark has to bound cost, and an unbounded budget measures persistence rather than capability. Composed tasks are meant to be harder within the same cost envelope.

Q3: More model families and a larger open-weight model

Additional model families. Yes. We extended the proof-completion evaluation to a larger open-weight model, DeepSeek-V4-Pro-0813, and to an additional model family, GLM-5.3 Flash. Both were run at composition levels N = 1 and N = 3, using the original 100-task bank for each level. At N = 1 the model proves a single function. At N = 3 it must establish the contracts of a three-function chain and the assertions in its driver. We keep the paper's AutoVerus protocol unchanged: the same prompts; the generation, refinement and repair phases; up to 10 repair rounds; a 16,000-token limit per call (4,096 for repair calls); and an 80-minute limit per task.

Both models show the same depth-related degradation as the models in the paper. From N = 1 to N = 3, DeepSeek-V4-Pro-0813 drops from 91.0% to 33.0% and GLM-5.3 Flash from 84.0% to 27.0%, losses of 58 and 57 points. Even the larger open-weight model loses about two thirds of its N = 1 success rate.

ModelN = 1N = 3
DeepSeek-V4-Pro-081391.033.0
GLM-5.3 Flash84.027.0

Proof success rate (%), 100 tasks per level.

Thank you again for your thoughtful review. We've posted responses addressing each of the points you raised and wanted to check in as the discussion period continues. If anything in our replies is unclear or if you have follow-up questions, we'd be very glad to answer them. Thank you for your time and engagement.

Thank you for the detailed responses and additional experiments. The stop-reason analysis, concrete trajectory examples, and results on additional model families substantially address my main concerns.

It remains somewhat unclear how much of the performance decline comes from having more work to do versus interactions between the combined tasks. I encourage the authors to clarify this limitation and incorporate the new analyses into the paper. Overall, I find the task-generation approach useful, and the additional evidence strengthens its contribution. I will raise my score.

Reviewer_AmCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CoDepth builds harder coding tasks by composing existing verified ones. Atoms are nodes in a graph. A level-N task combines N connected atoms. The framework aims to admit a task only if each atom stays unsolved and a joint reference passes the verifier, with domain-specific implementations of these checks. The authors build over 1,700 tasks in four domains with no LLM calls. Frontier accuracy generally drops with N. For example, GPT-5.6-Sol falls from 94% to 35% on proof completion from N=1 to N=3. RFT on the generated tasks improves Qwen3.5-9B at every evaluated level in multi-bug repair and clone equivalence.

Critical strengths
  1. The task-construction process does not rely on a large language model and is programmatic, quick and cheap.
  2. In general, I like the idea and motivation of the paper, which is combining multiple problems into a bigger and harder problem.
Critical weaknesses
  1. Turning a pool of atomic problems into a difficulty knob is an appealing idea. But although the paper discusses some prior work on composition, it omits several closely related approaches:

[1] Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM. COLM 2024. https://arxiv.org/abs/2403.19114.

[2] DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. ICLR 2024. https://arxiv.org/abs/2309.17167.

[3] DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph. NeurIPS 2024. https://arxiv.org/abs/2406.17271.

[4] HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task. Findings of ACL 2025. https://aclanthology.org/2025.findings-acl.686/.

EvoEval includes a Combine setting; DyVal uses graph-based generation to control task complexity; DARG modifies reasoning graphs to generate tasks with controlled complexity; and HumanEval Pro and MBPP Pro require models to use a base solution to solve a more complex problem. These approaches are not identical to CoDepth, and some rely on LLMs. However, they are directly relevant to its contribution. The authors should discuss them and clarify what is new beyond applying composition and graph-based difficulty control to these coding domains.

  1. The authors have demonstrated the effectiveness of their methods on several benchmarks. However, I have particular concerns about one of the most important tasks, i.e., multi-bug repair.

Multi-bug repair is the most convincing domain. But I am not sure the construction is as safe as the paper suggests. Defects4J [5] isolates each target fix, so the buggy and fixed versions differ only by that fix. This does not mean that each version contains only one fault, but it provides a controlled relationship between the target fault, its tests and its repair.

CoDepth instead goes back to the base commit of the bug with the earliest fixing date and transplants later test patches and gold patches onto it. The code in between can drift substantially. The admission checks filter out some incompatible combinations, but it is unclear whether they establish that each transplanted test still exposes the original fault.

Further, this idea also has close prior work. An et al. [6] transplanted fault-revealing tests across Defects4J versions to find versions with coexisting faults. They found that 311 of 326 versions contain two to 24 faults. This is particularly relevant prior work for this domain, and it is not discussed.

I have more concrete concerns:

  • The method orders bugs by fix date and assumes that the selected base commit precedes every fix. Date ordering alone does not establish Git ancestry. More importantly, even a commit that precedes all fixes need not contain all the bugs: some may have been introduced later. How does the construction check this beyond observing test failures?

  • A failing test does not prove that the original bug is present. Each bug's tests were written for a potentially later version of the code. Moved back to the selected base commit, a test may fail because an API or behavior it relies on is unavailable, rather than because of the original bug. The joint reference passing all target tests does not by itself establish the cause of each initial failure. Useful diagnostic checks would apply each bug's fix separately and apply all fixes except that bug's fix, then inspect which tests pass or fail. These checks need not succeed for every valid task, since faults and fixes may interact, but reporting their results and auditing the exceptions would make the construction more convincing.

  • The reference solution may introduce regressions. Each fix was written for its original version of the code, and passing the transplanted FAIL_TO_PASS tests does not establish that other functionality is preserved. Appendix B.1 explicitly acknowledges that PASS_TO_PASS tests are not checked and that the metric measures targeted bug fixing rather than full-patch acceptability. Nevertheless, this limits the claim that admission establishes a correct joint repair. Could the authors check whether tests that pass at the selected base commit still pass after applying the joint reference?

[5] Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. ISSTA 2014. https://doi.org/10.1145/2610384.2628055.

[6] Searching for Multi-Fault Programs in Defects4J. SSBSE 2021. https://arxiv.org/abs/2108.04455.

  1. My bigger concern is whether the composed tasks measure abilities that matter in practice. Type compatibility allows candidate chains to be formed, but matching types does not establish that a combined task is representative of real programming work. In App. A.4, a function that takes the last byte of each string is chained with a function that checks for the letter Z. In App. A.3, a padding function is chained with a substring check. In App. A.2, an integer is passed on to be used as a threshold. These examples demonstrate data flow between functions, but do not establish that the generated distribution reflects realistic combinations of programming problems.

The paper's own motivation is that real problems arrive together and interact. Multi-bug repair provides the clearest connection to this motivation, but its demonstrated depth is limited. The evaluation stops at N=3, and only 15 groups are available at N=4 across 270 repositories (App. B.1). This is a limitation of the current task pool rather than proof that the method cannot scale, but higher-depth construction has not been demonstrated in this domain. The domains that reach N=10 use synthetic function chains whose practical representativeness remains unclear. The paper also does not show how the approach extends to tasks such as feature implementation or refactoring, where suitable atoms and composition rules may be harder to define.

I suggest the authors show that the composed tasks measure something that matters in practice. For example, does the model ranking agree with harder real-world benchmarks? Does RFT on composed tasks improve results on an external benchmark such as SWE-bench Verified? The current RFT results show gains on held-out tasks from the generated distribution, but do not establish this broader transfer.

Questions for authors
  1. Beyond the works listed in Weakness 1, especially DyVal [2], and the multi-fault construction of An et al. [6], what exactly is new in CoDepth? Please discuss these works and state the technical differences.

  2. Did you check Git ancestry, or only fix dates? How did you distinguish failures caused by the original bugs from failures caused by transplanting tests across versions? What proportion of admitted groups pass the single-fix and leave-one-out checks described in Weakness 2, and what explains the exceptions? Does the joint reference preserve the tests that already pass at the selected base commit?

  3. The reference label is derived from the labels of the parts and their input/output-size growth, but the derivation and its validation are not sufficiently described. CodeComplex labels depend on input characteristics and constraints; how are these preserved after extraction and composition? How is the output-size growth of each step obtained? In App. A.2, what input-size measure and assumptions about a, b, n and m justify the label “quadratic”? An independently checked sample, with explicit derivations, would help assess label reliability.

  4. Section 4.2 compares against an independent-success baseline and identifies joint failures when all constituent bugs are solved separately. These analyses are useful, but they do not separate interference from increased workload, context length or fixed budgets. How do models score on a matched control without interaction, using the same atoms and comparable budgets? A budget-scaling analysis would also help, especially since Section 4.4 attributes most small-model failures to timeouts.

  5. Section 2.4 requires every atomic problem to remain unsolved, but Appendix B.2 states that proof completion checks failure of the whole program rather than each component independently. How do you ensure that a level-N task actually contains N components requiring proof work? Whole-program verification failure alone does not establish this.

14 comments

We thank the reviewer for a detailed and constructive review. The concerns about multi-bug construction were specific enough to test directly. We ran every diagnostic the reviewer proposed, on all 368 L2/L3 multi-bug tasks (904 bug slots), with no LLM calls. The results are below; we will consider adding them to the paper.

First, one clarification of what a level means. A level-N task asks the agent to make the union of the constituent bugs' FAIL_TO_PASS tests pass. The prompt does not reveal N, and the agent may fix the code in one edit or many. N is the number of atomic problems combined and serves as a difficulty knob. It is not a claim that the task contains exactly N separable faults. Task validity rests on the admission checks: every bug's tests fail at the base commit c⋆c_\star, and the joint reference makes all of them pass. The diagnostics below test whether those checks admit the right tasks.

W1 / Q1: Related work and what is new

We thank the reviewer for the references and will discuss all of them in the related work section.

  • EvoEval (Combine) uses an LLM to merge two HumanEval problems into one new problem, and the merged problem is checked by an LLM or by hand.
  • HumanEval Pro / MBPP Pro use an LLM to write a harder problem that must call the solution of a base problem. The tests are generated and then manually checked. Depth is fixed at 2.
  • DyVal synthesises reasoning problems (arithmetic, logic, reachability, and so on) from generated DAGs whose depth and width control complexity. The ground truth is computed from the graph.
  • DARG extracts a reasoning graph from an existing benchmark item, perturbs it to change complexity, and uses an LLM to turn the graph back into text, with code-assisted LLM verification.
  • An et al. (SSBSE 2021) transplant fault-revealing tests across Defects4J versions to find versions in which several real faults coexist, for fault-localisation research.

CoDepth differs from these works in three ways:

  1. The graph is over whole problems, not inside one problem. In DyVal and DARG, the graph is the internal reasoning structure of a single problem, and complexity changes when that structure is grown or perturbed. In CoDepth, each node is an entire verified problem, and an edge says that two problems can be combined (in multi-bug repair, two bugs whose fixes touch the same files). A level-N task is a connected group of N nodes, found by walking this graph. Depth is therefore the number of real problems combined, not the size of one synthetic problem.
  2. It composes existing verified problems; it does not generate new ones. Each atom keeps its original reference solution and verifier: real repository tests, the Verus verifier, CodeComplex labels or clone labels. The composed task's reference and verifier are derived from the atoms'. No LLM is used to generate or to validate tasks. EvoEval, HumanEval Pro and DARG rely on an LLM at one of these steps. DyVal is model-free, but it synthesises new problems from primitives rather than reusing real, already-verified ones.
  3. Depth is a controlled knob, enforced by admission checks. An et al. search for faults that already coexist in one version. We instead construct tasks with a chosen N and admit a task only if every constituent still fails at the base and a joint reference passes. The result is a graded benchmark (N=1–3 for multi-bug repair, up to N=10 in other domains) rather than a census.
  4. One framework spans four coding domains, including formal verification and complexity analysis, and its tasks support training as well as evaluation.

W2 / Q2: Multi-bug construction

We first clarify that ancestry is not a requirement for a valid task. Construction only needs a base commit c⋆c_\star at which the two admission gates hold: every bug's tests fail at c⋆c_\star (G1), and the joint reference makes all of them pass (G2). Ordering by date is just a convenient way to pick a c⋆c_\star that is likely to pass these gates, and whether c⋆c_\star is also an ancestor of the later bugs' commits does not change whether a task is admitted. We nonetheless checked ancestry for the reviewer's reference, using the GitHub compare API from c⋆c_\star to the base commit of every later bug in each task (536 pairs).

ResultPairs
c⋆c_\star is an ancestor511
Identical commits14
Diverged branches (one task)2
Repository no longer public on GitHub, so it can't be checked (6 tasks, 2 repositories)9

That gives 361 of 368 tasks (98%) with verified ancestry. Each fix commit descends from its own base, so in these tasks c⋆c_\star precedes every fix in history, not just by date. In practice, then, the date-ordered base is almost always a true ancestor as well. We will state in §3.1 that ordering uses dates, that validity rests on the two gates, and report this ancestry check alongside them.

Is each bug present at c⋆c_\star, and does the transplanted test expose it? For every bug, we ran its FAIL_TO_PASS tests twice:

  1. at its own original base commit, in its own environment, with only its own test patch (by definition the bug is present there);
  2. at c⋆c_\star, inside the composed task.

We then compared the per-test outcomes and failure messages. We excluded the one modin task whose sandbox cannot start Ray (3 slots), leaving 901 of the 904 slots.

Failure at c⋆c_\star compared with failure at the bug's own baseBug slots
Same kind of failure, identical message783 / 901 (87%)
Same kind of failure, same exception type21 (2%)
Same kind of failure, different message80 (9%)
Different kind of failure (explained below)17 (1.9%)

In 98% of cases a test fails at c⋆c_\star for the same kind of reason it fails where the bug is known to exist, and 87% fail with the identical message. This also addresses the concern that a bug could have been introduced after c⋆c_\star: it would show up as a different failure at c⋆c_\star.

The 17 slots whose failure kind changes are not artifacts of transplanting:

  • 15 slots: the test cannot run, or errors, at c⋆c_\star, because code it uses does not exist there yet. In 14 of them, that code comes from another fix in the same task: applying that other bug's fix alone makes the test run and fail on an assertion, as it does at the bug's own base. In the remaining one, the bug's own fix adds the API (a feature-style instance). In every case the joint reference makes the test pass (G2).
  • 2 slots: the reverse. The test runs and fails at c⋆c_\star but cannot run at the bug's own base, because of an environment difference in the original image.

The first group consists of dependent bugs: one bug's failure can only be reached once another bug is repaired. This is one of the couplings composition is meant to create, not a flaw in it. As noted at the start of this response, a task does not ask the agent to fix the bugs in a given order, to fix them one at a time, or to identify how many there are. It asks the agent to make all target tests pass. A dependency of this kind adds difficulty while the task remains valid, because the admission gates still hold.

In 46 slots the tests are already not runnable at the bug's own base, because of a collection error or a missing test. These are feature-style instances in SWE-rebench itself, not artifacts of transplanting.

Single-fix and leave-one-fix-out. At c⋆c_\star, with all test patches applied:

CheckResult
No fix applied: every bug's tests fail (G1, rerun)904 / 904
Joint reference: all tests pass (G2, two independent reruns)368 / 368
Only the bug's own fix applied (the three rows below partition all 904 bugs):
its own tests pass648 / 904 (72%)
the fix applies, but its own tests do not all pass102 / 904 (11%)
the fix does not apply without an earlier fix (textual dependency)154 / 904 (17%)
Only another bug's fix applied: this bug's tests pass anyway2 / 904
L3 leave-one-out: the left-out bug's tests still fail415 / 417 graded

Explanation of the exceptions:

  • Not sufficient alone (102). In 98 of these, another bug's test patch edits the same test file, and in 78 the two bugs share a FAIL_TO_PASS test. The test therefore checks both behaviours and passes only when both fixes are in place.
  • Does not apply alone (154). The later fix was written on top of the earlier one in the repository's history.
  • G2 exception (1). In one modin task, Ray cannot start in the sandbox (an environment issue). We will remove it.
  • Degenerate tasks (3). In these, one bug's fix also resolves another bug. They are the tasks flagged by the "another bug's fix" row (2 bugs) and the leave-one-out row (2 bugs); one task appears in both. We will remove them.

Separability need not hold for a valid task. These dependencies are the interactions that composition is meant to create, and an isolated benchmark cannot produce them.

Does the joint reference regress tests that already pass? At c⋆c_\star, with all test patches applied, we ran the union of all constituent bugs' PASS_TO_PASS tests, with and without the joint reference. This is the same set SWE-bench and SWE-rebench use for their "resolved" criterion.

  • 62,617 tests across 347 tasks (21 tasks list none).
  • 59,414 pass at c⋆c_\star.
  • None of them fails after applying the joint reference.

The remaining 3,203 do not exist or do not pass at c⋆c_\star, so they cannot regress. We will add this as an admission check and update App. B.1.

Relation to Defects4J. Defects4J isolates each fix so that the buggy and fixed versions differ by one fault. CoDepth deliberately combines faults whose fixes touch shared files, to raise difficulty. The checks above show what this costs and what it buys:

  • Cost: in 98% of cases, a fault's tests fail at c⋆c_\star for the same reason as at the fault's own version, and the joint reference introduces no regressions.
  • Gain: dependencies between fixes, which an isolated benchmark cannot produce.
Reviewer_4Ff2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CoDepth, a model-free framework that composes existing coding problems into more challenging tasks without LLM-based generation or validation. It covers four domains: multi-bug repair, proof completion, complexity analysis, and clone equivalence. Experiments show that model performance generally declines as compositional depth increases, while rejection-sampling fine-tuning on the generated tasks improves Qwen3.5-9B, demonstrating their usefulness for both evaluation and training.

Critical strengths
  1. Simple and practical approach. CoDepth generates challenging coding tasks by composing existing verified problems without LLM-based generation or validation, offering a cost-effective and scalable framework.

  2. Comprehensive evaluation. Experiments across four coding domains and multiple models demonstrate that the generated tasks become increasingly challenging with compositional depth, revealing limitations of current frontier models.

  3. Effective training data. Rejection-sampling fine-tuning on the generated tasks consistently improves Qwen3.5-9B across evaluated composition levels, demonstrating their usefulness beyond benchmarking.

Critical weaknesses
  1. Missing budget-matched training baseline. The curriculum introduces higher-level tasks alongside additional rounds of trajectory collection and fine-tuning. An iterative L1-only baseline with matched rollout and training budgets would better establish the additional value of composed tasks beyond continued training on atomic problems.

  2. Unclear availability of benchmark artifacts. I could not find a clear statement in the manuscript about access to, or plans to release, the generated datasets, task-generation code, and evaluation harness. Given the paper’s emphasis on reusable benchmark construction, the authors should clarify artifact availability and release plans, including fixed data splits and environment configurations, to support reproducibility and community adoption.

Questions for authors

see the weakness section

2 comments

We thank the reviewer for the positive assessment and for the two concrete suggestions.

W1: Budget-matched L1-only training baseline

We agree that a budget-matched L1-only baseline is needed to separate the effect of training on deeper compositions from the effect of additional training. We could not complete this ablation within the rebuttal window. We will run the same three-stage training schedule using only L1 tasks, match the rollout and optimizer-update budgets to RFT-L1+L2+L3, and report the number of accepted trajectories. We will evaluate both runs at L1, L2, and L3 on the same held-out evaluation split and report the comparison in the revision.

W2: Availability of artifacts

Our dataset is publicly available on Hugging Face: https://huggingface.co/CodeWalk. The code is available via this anonymous link: https://anonymous.4open.science/r/artifact-4045afe5135f/README.md.

Thank you again for your thoughtful review. We've posted responses addressing each of the points you raised and wanted to check in as the discussion period continues. If anything in our replies is unclear or if you have follow-up questions, we'd be very glad to answer them. Thank you for your time and engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.