acceptodds
Withdrawn from ICLR 2027

Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mining-based code corpora are abundant but uncontrolled: a snippet's semantics, its surface messiness, and its difficulty are whatever the wild happened to contain, and any public sample may already sit in a model's training set. We present Spaghetti Architect, an anti-optimization transpiler that mints code datasets with the control mining cannot supply: it maps a clean, language-agnostic JSON intermediate representation to deliberately redundant yet execution-verified programs in five languages, compiles, runs, and checks every program against a reference oracle, labels each instance along two orthogonal difficulty axes (intrinsic problem size and incidental presentation), and mints scored instances fresh from a private held-out seed. The control pays off in a finding: the generator annotates its own output, and re-rendering the identical corpus without those annotations shows they inflate every model's refactoring score, but the weakest model in our four-model panel an order of magnitude more than the strongest (semantic-equivalence against , the difference replicating at same-week under pre-registration). Differential inflation compresses the capability ladder: annotated baselines resolve one of three adjacent pairs, unannotated all three (two in the replication). The annotations also flatten the second difficulty axis: the incidental-messiness knob, inert as released, degrades both tasks for effectively every model once they are removed. Falsifying the annotations converts the help into damage and scrambles the model ranking itself. The unannotated corpus is the one to evaluate on. Re-minting preserves difficulty: dev-split scores match freshly re-minted private-test scores within though every prompt hash differs, and the intrinsic scale knob collapses the aggregation accuracy of even the strongest tested model to zero under direct answering. The artifact is open source and dependency-free.

Primary area:
datasets and benchmarks
Keywords:
code datasets, benchmark contamination, data contamination, evaluation methodology, code refactoring, code comprehension, synthetic data generation, large language models for code

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FSX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Spaghetti Architect, a deterministic generator for synthetic code benchmarks. Starting from a simple language-independent JSON IR, it generates semantically equivalent but increasingly messy programs in five programming languages. The generator also supports two controlled difficulty dimensions: intrinsic problem size and incidental code messiness. The paper studies three possible uses of the generated data: code refactoring, code comprehension through output prediction, and LLM-as-a-judge evaluation. Four open models are evaluated on the first two tasks. A particularly interesting result is that the comments automatically inserted by the generator leak useful information to the model: removing these comments hurts weaker models much more than stronger models. The authors repeat the main refactoring experiment in a pre-registered same-week replication.

My main concern is that the paper sometimes makes the benchmark sound more mature and general than the current evidence supports. In particular, the generated programs are small and semantically narrow, the central "messiness" ordering has not yet been validated by humans, and the contamination defense mainly protects against some forms of direct memorization rather than contamination in a broader sense.

Critical strengths
  • Excellent reproducibility. Generation is deterministic, raw model completions are committed, the reported results can be re-derived offline, and the analysis setup is frozen and documented. This is stronger artifact practice than I usually see in benchmark submissions. The paper states that the public split can be regenerated byte-for-byte and that published numbers are reproduced from the committed evaluation records.
  • The paper is unusually honest about its own defects. For example, light and standard produce identical source code, so the released six-level knob really has only five distinct levels. For many examples, other levels also collapse. The authors disclose this rather than hiding it.
  • The annotation ablation is the strongest result in the paper. The same program is rendered with and without comments, so the intervention is clean. The result that annotations help weaker models more than stronger ones is interesting, and the refactoring effect survives the same-week pre-registered replication.
  • The paper does more validation than many benchmark papers. The authors include bootstrap uncertainty, a power analysis for the freshness comparison, a fixture-hardcoding audit for refactoring, and explicit analysis of serving drift.
  • The distinction between intrinsic and incidental difficulty is useful. The width experiment is a nice example: increasing aggregation width strongly hurts output prediction, while refactoring on the same programs remains much more stable.
  • Documentation quality is high. The datasheet, machine-readable metadata, canary, deterministic generator, and explicit extension contract make this a potentially useful research artifact even if the current dataset is limited.
Critical weaknesses
  1. The benchmark is still quite small and narrow - The current public dataset has only 100 items derived from 50 base IRs, and the underlying language is limited to four primitive operations. Programs are also single-file. This is acceptable for a controlled diagnostic benchmark, but it limits what can be claimed about general code understanding, refactoring, or maintainability. The paper itself acknowledges that this is a controlled stimulus rather than a realistic software benchmark.

I would therefore recommend consistently presenting Spaghetti Architect as a controlled measurement instrument, rather than as a broad coding benchmark. There is also a sample-size issue inside individual families. Three of the seven families are based on only one base program, which makes those per-family numbers illustrative rather than meaningful estimates.

  1. The central "messiness" label has not been validated by humans - This is probably the biggest validity issue. The benchmark assumes that moving from clean toward max produces progressively worse code. The paper validates this against cyclomatic complexity, cognitive complexity, maintainability index, and a surface readability metric. However, some of these metrics are mechanically related to the transformations being injected. The paper correctly acknowledges this: the generator adds constructs that the complexity metrics themselves count.

The less circular readability signal is much weaker, around ρ = 0.39, and there is currently no human study confirming that developers actually perceive the levels in the intended order. This matters because the by-construction quality ordering is central to the benchmark. I would strongly encourage the authors to run even a modest human-ranking study before camera-ready. Without that, I would describe the label as a synthetic transformation-strength ordering, not yet a validated maintainability or readability ordering. 3. The contamination claims should be more carefully scoped The private re-minting protocol is useful, but I think terms such as "contamination-resistant" and especially statements suggesting that freshness is "certified" can be read too strongly. Tier A changes literals but keeps public structures. Tier B changes operation-chain structure. Tier C moves to larger/deeper programs. This clearly protects against some forms of exact or near-exact memorization. But the generator itself, the public split, the task definition, and the transformation mechanisms are public. A future model may therefore know the distribution and templates extremely well even if the exact instance is new. The paper does acknowledge in Limitations that contamination is mitigated, not eliminated, and that the tiers establish freshness rather than novelty.
I think this more careful language should also appear in the abstract, introduction, and contribution claims. It would help to explicitly map the tiers to different contamination types, for example:

  • exact/verbatim contamination;
  • literal-level contamination;
  • structural/template contamination;
  • distributional contamination. Then state clearly which ones each tier addresses and which ones remain possible.
  1. The freshness experiment needs a positive control - The dev and privately re-minted test scores are very close, which the paper interprets as evidence that fresh minting preserves difficulty. That is useful, but it does not yet show that the protocol can actually detect contamination when contamination is present. The paper's own power analysis shows that the comprehension experiment has limited sensitivity: even an uplift of 0.10 may not reach 80% power.

A strong addition would be a deliberately contaminated model or setting. For example, fine-tune one of the smaller models on the public dev data and show that the dev-vs-held-out comparison detects the expected advantage. That would provide a known-positive reference for the contamination methodology.

  1. The default release format conflicts with the recommended evaluation format - This is a confusing design choice. The paper's main experimental result is that annotations leak useful information and distort model comparisons. The authors therefore recommend evaluation on the unannotated rendering. However, the released/default rendering is the annotated one, while the unannotated version requires an explicit configuration choice. The reproducibility statement confirms that the unannotated rendering is the intended standard evaluation configuration, while the annotated version remains the release/documentation format. This seems likely to cause accidental misuse.
Questions for authors
  1. Do you plan to run any human evaluation of the messiness ordering before camera-ready? Even a small pairwise-ranking study would significantly strengthen the construct-validity claim.
  2. Why is the annotated rendering the default release format when the paper explicitly recommends the unannotated rendering for evaluation? Could you reverse the default?
  3. Have you considered a positive contamination control, for example fine-tuning one model on the dev split and checking whether the freshness protocol detects the resulting dev/test gap?
  4. Can you explicitly state which types of contamination each tier addresses? In particular, what remains vulnerable to template or distributional contamination once the generator itself is public?
  5. Do you have experimental results for the Tier B/C successor mechanism, rather than only the current Tier-A-style literal re-minting comparison?
  6. What is the intended role of the 100-item public dataset for training? Is the frozen split itself supposed to support training, or should training data always be generated dynamically from the generator?
Reviewer_qa85OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Spaghetti Architect, a generator that turns a small JSON IR with four operation types into deliberately messy but oracle-verified programs in five languages. Each instance is labelled on an intrinsic size axis and an incidental messiness axis. Scored splits are re-minted from a private seed for contamination resistance. Validation uses static metrics (complexity, MI, a readability proxy) and a four-model open LLM ladder on refactoring and comprehension. The main empirical claim is that the generator's own comments inflate weak models' refactoring scores far more than strong ones, so the authors recomend evaluating on the unannotated corpus.

Critical strengths
  1. The generator varies messiness at fixed semantics, with execution-verified equivalence and a known clean reference. Mined corpora cant provide this, and it is the main reason I see value in the artifact.
  2. The paper audits itself unusually carefully. It uses clustered bootstrap intervals, a pre-registered same-week replication, and a hardcoding audit, and it reports honestly what did not replicate.
Critical weaknesses
  1. The headline finding is largely self-inflicted. The intent comment states the clean form, so the annotation effect mostly shows that models copy a leaked answer, and the remedy is simply to drop the comments. The effect also weakens under closer inspection: the refactoring gap shrinks from -0.156 to -0.096 in the replication, the comprehension version does not replicate, the log-odds interaction is null, and the rescue rates point to a composition effect rather than differential reliance.
  2. Scope and validity are thin for a benchmark contribution. There are only four primitives and only single-file programs, and the dev split has 100 items, three of whose families are a single program each. Two rungs are inert, 36 items have only two distinct rungs, and the messiness order has no human calibration (the readability proxy correlates at only 0.39). Comprehension accuracy dosen't seperate any adjacent model pair. Together with very dense writing that leans heavily on the appendix, this makes it hard to see the instrument as ready.
Questions for authors
  1. On the unannotated corpus, does the negative messiness slope also hold on the Tier B and Tier C held-out shapes, and does it separate adjacent models there?
  2. Do the ablation and knob-slope results still hold under the opt-in untied rendering spec, where all six rungs are distinct?
Reviewer_exCpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents SPAGHETTI ARCHITECT, a tool for generating controlled code benchmarks. It transforms a language-agnostic IR into deliberately redundant programs, verifies their behavior against a reference oracle, and varies intrinsic problem size and incidental code messiness independently. The authors also use a private seed to generate held-out instances. The authors find that annotations embedded in the generated code inflate refactoring scores, with a larger effect on the weakest model. Removing the annotations improves the benchmark’s ability to distinguish model capabilities and reveals performance degradation as incidental messiness increases.

Critical strengths
  1. The studied problem is important and significant.

  2. The evaluation covers model from Llama to Deepseek.

Critical weaknesses
  1. The paper is difficult to follow, and several key components are not sufficiently explained. For example, Figure 1 introduces a JSON intermediate representation (IR), but it is unclear whether the IR is manually defined or derived from a seed dataset. The data-driven anti-pattern database is also not adequately described. In addition, the roles of the validator and reference oracle, and how they interact, need to be clarified.

  2. The paper states that each task instance is assigned a difficulty score, but it does not explain how this score is calculated or validated. The authors should clarify what the score measures and why it can be treated as ground-truth difficulty.

  3. The evaluation setup is difficult to follow. The research questions are not clearly stated, and it is unclear how the experimental tasks, models, metrics, and baselines map to those questions. The authors should present the evaluation protocol more systematically and explain what each experiment is intended to establish.

  4. The paper’s contamination-resistant mechanism appears to rely on dynamically generating or refreshing benchmark instances, an approach explored in prior work. The authors should explain how their mechanism differs from existing methods and qualify any claims of being “contamination-free.”

  5. The related-work section overlooks several relevant studies on dynamic code benchmark generation and evolution, including:

    [1] DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination

    [2] Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation

    [3] Compilable, Contamination-Aware Benchmarks for Evaluating LLMs on Software Systems Tasks

    [4] PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models

    [5] DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

    [6] EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations

    The authors should discuss these works and clarify how SPAGHETTI ARCHITECT differs from them.

Questions for authors
  1. Can you explain more details about the design?

  2. What is the advantage of this work compared with existing work?

3.What is the key research question the evaluation aim to answer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.