- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Spaghetti Architect, a deterministic generator for synthetic code benchmarks. Starting from a simple language-independent JSON IR, it generates semantically equivalent but increasingly messy programs in five programming languages. The generator also supports two controlled difficulty dimensions: intrinsic problem size and incidental code messiness. The paper studies three possible uses of the generated data: code refactoring, code comprehension through output prediction, and LLM-as-a-judge evaluation. Four open models are evaluated on the first two tasks. A particularly interesting result is that the comments automatically inserted by the generator leak useful information to the model: removing these comments hurts weaker models much more than stronger models. The authors repeat the main refactoring experiment in a pre-registered same-week replication.
My main concern is that the paper sometimes makes the benchmark sound more mature and general than the current evidence supports. In particular, the generated programs are small and semantically narrow, the central "messiness" ordering has not yet been validated by humans, and the contamination defense mainly protects against some forms of direct memorization rather than contamination in a broader sense.
Critical strengths
- Excellent reproducibility. Generation is deterministic, raw model completions are committed, the reported results can be re-derived offline, and the analysis setup is frozen and documented. This is stronger artifact practice than I usually see in benchmark submissions. The paper states that the public split can be regenerated byte-for-byte and that published numbers are reproduced from the committed evaluation records.
- The paper is unusually honest about its own defects. For example, light and standard produce identical source code, so the released six-level knob really has only five distinct levels. For many examples, other levels also collapse. The authors disclose this rather than hiding it.
- The annotation ablation is the strongest result in the paper. The same program is rendered with and without comments, so the intervention is clean. The result that annotations help weaker models more than stronger ones is interesting, and the refactoring effect survives the same-week pre-registered replication.
- The paper does more validation than many benchmark papers. The authors include bootstrap uncertainty, a power analysis for the freshness comparison, a fixture-hardcoding audit for refactoring, and explicit analysis of serving drift.
- The distinction between intrinsic and incidental difficulty is useful. The width experiment is a nice example: increasing aggregation width strongly hurts output prediction, while refactoring on the same programs remains much more stable.
- Documentation quality is high. The datasheet, machine-readable metadata, canary, deterministic generator, and explicit extension contract make this a potentially useful research artifact even if the current dataset is limited.
Critical weaknesses
- The benchmark is still quite small and narrow - The current public dataset has only 100 items derived from 50 base IRs, and the underlying language is limited to four primitive operations. Programs are also single-file. This is acceptable for a controlled diagnostic benchmark, but it limits what can be claimed about general code understanding, refactoring, or maintainability. The paper itself acknowledges that this is a controlled stimulus rather than a realistic software benchmark.
I would therefore recommend consistently presenting Spaghetti Architect as a controlled measurement instrument, rather than as a broad coding benchmark. There is also a sample-size issue inside individual families. Three of the seven families are based on only one base program, which makes those per-family numbers illustrative rather than meaningful estimates.
- The central "messiness" label has not been validated by humans - This is probably the biggest validity issue. The benchmark assumes that moving from clean toward max produces progressively worse code. The paper validates this against cyclomatic complexity, cognitive complexity, maintainability index, and a surface readability metric. However, some of these metrics are mechanically related to the transformations being injected. The paper correctly acknowledges this: the generator adds constructs that the complexity metrics themselves count.
The less circular readability signal is much weaker, around ρ = 0.39, and there is currently no human study confirming that developers actually perceive the levels in the intended order. This matters because the by-construction quality ordering is central to the benchmark.
I would strongly encourage the authors to run even a modest human-ranking study before camera-ready. Without that, I would describe the label as a synthetic transformation-strength ordering, not yet a validated maintainability or readability ordering.
3. The contamination claims should be more carefully scoped
The private re-minting protocol is useful, but I think terms such as "contamination-resistant" and especially statements suggesting that freshness is "certified" can be read too strongly. Tier A changes literals but keeps public structures. Tier B changes operation-chain structure. Tier C moves to larger/deeper programs. This clearly protects against some forms of exact or near-exact memorization.
But the generator itself, the public split, the task definition, and the transformation mechanisms are public. A future model may therefore know the distribution and templates extremely well even if the exact instance is new.
The paper does acknowledge in Limitations that contamination is mitigated, not eliminated, and that the tiers establish freshness rather than novelty.
I think this more careful language should also appear in the abstract, introduction, and contribution claims.
It would help to explicitly map the tiers to different contamination types, for example:
- exact/verbatim contamination;
- literal-level contamination;
- structural/template contamination;
- distributional contamination. Then state clearly which ones each tier addresses and which ones remain possible.
- The freshness experiment needs a positive control - The dev and privately re-minted test scores are very close, which the paper interprets as evidence that fresh minting preserves difficulty. That is useful, but it does not yet show that the protocol can actually detect contamination when contamination is present. The paper's own power analysis shows that the comprehension experiment has limited sensitivity: even an uplift of 0.10 may not reach 80% power.
A strong addition would be a deliberately contaminated model or setting. For example, fine-tune one of the smaller models on the public dev data and show that the dev-vs-held-out comparison detects the expected advantage. That would provide a known-positive reference for the contamination methodology.
- The default release format conflicts with the recommended evaluation format - This is a confusing design choice. The paper's main experimental result is that annotations leak useful information and distort model comparisons. The authors therefore recommend evaluation on the unannotated rendering. However, the released/default rendering is the annotated one, while the unannotated version requires an explicit configuration choice. The reproducibility statement confirms that the unannotated rendering is the intended standard evaluation configuration, while the annotated version remains the release/documentation format. This seems likely to cause accidental misuse.
Questions for authors
- Do you plan to run any human evaluation of the messiness ordering before camera-ready? Even a small pairwise-ranking study would significantly strengthen the construct-validity claim.
- Why is the annotated rendering the default release format when the paper explicitly recommends the unannotated rendering for evaluation? Could you reverse the default?
- Have you considered a positive contamination control, for example fine-tuning one model on the dev split and checking whether the freshness protocol detects the resulting dev/test gap?
- Can you explicitly state which types of contamination each tier addresses? In particular, what remains vulnerable to template or distributional contamination once the generator itself is public?
- Do you have experimental results for the Tier B/C successor mechanism, rather than only the current Tier-A-style literal re-minting comparison?
- What is the intended role of the 100-item public dataset for training? Is the frozen split itself supposed to support training, or should training data always be generated dynamically from the generator?