Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator
Abstract
Mining-based code corpora are abundant but uncontrolled: a snippet's semantics, its surface messiness, and its difficulty are whatever the wild happened to contain, and any public sample may already sit in a model's training set. We present Spaghetti Architect, an anti-optimization transpiler that mints code datasets with the control mining cannot supply: it maps a clean, language-agnostic JSON intermediate representation to deliberately redundant yet execution-verified programs in five languages, compiles, runs, and checks every program against a reference oracle, labels each instance along two orthogonal difficulty axes (intrinsic problem size and incidental presentation), and mints scored instances fresh from a private held-out seed. The control pays off in a finding: the generator annotates its own output, and re-rendering the identical corpus without those annotations shows they inflate every model's refactoring score, but the weakest model in our four-model panel an order of magnitude more than the strongest (semantic-equivalence against , the difference replicating at same-week under pre-registration). Differential inflation compresses the capability ladder: annotated baselines resolve one of three adjacent pairs, unannotated all three (two in the replication). The annotations also flatten the second difficulty axis: the incidental-messiness knob, inert as released, degrades both tasks for effectively every model once they are removed. Falsifying the annotations converts the help into damage and scrambles the model ranking itself. The unannotated corpus is the one to evaluate on. Re-minting preserves difficulty: dev-split scores match freshly re-minted private-test scores within though every prompt hash differs, and the intrinsic scale knob collapses the aggregation accuracy of even the strongest tested model to zero under direct answering. The artifact is open source and dependency-free.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.