Executable Trust: Query-Blind Validation of Rule Proposals for Linguistics Olympiad Reasoning
Abstract
Linguistics Olympiad problems require solvers to induce latent rules from sparse evidence in unfamiliar languages. Yet final-answer accuracy is fundamentally insufficient: fluent but invalid rules routinely reach the same prediction, making it impossible to distinguish supported analyses from plausible confabulations. We introduce Executable Trust, a generator-agnostic framework that compiles rule proposals from language models or human experts into typed executable hypotheses and validates them before query labels are revealed. Four gates-leave-one-context reconstruction, counterexample sensitivity, perturbation stability, and support calibration-jointly produce an auditable admit, reject, or abstain decision with a replayable evidence trace. Across six rule families spanning numeral systems, morphology, allomorphy, agreement, role alignment, and conditioned forms, the protocol decisively admits three supported families (Yidiny morphology, Sulka numeral-classifier structure, and Zoque conditioned allomorphy) and rejects three under-specified or unstable alternatives (Kayapo, Wambaya, and Benabena), with every decision backed by reconstruction scores, perturbation variance, and counterexample counts. On 32 query-blind typed queries, a strict selective policy answers 20 and is correct on all 20-yielding 62.5% coverage at 100% answered precision (95% Wilson interval: 83.9-100.0%). The 12 abstentions are not failures but calibrated refusals, each traced to reconstruction or stability gates that flagged insufficient evidence. An expert-authored derivation language of 21 rules executes all hidden-context analyses correctly under systematic source-side ciphers across 5 cipher strengths and rejects 68 of 71 corrupted controls (95.8% rejection rate); the three survivors are behaviorally equivalent rule-order permutations. Executable validation constitutes a falsifiable trust layer between linguistic hypothesis generation and answer production, forcing abstention and representation failures into the open rather than allowing them to hide behind answer accuracy.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.