acceptodds
Under review as a conference paper at ICLR 2027

Executable Trust: Query-Blind Validation of Rule Proposals for Linguistics Olympiad Reasoning

Abstract

Linguistics Olympiad problems require solvers to induce latent rules from sparse evidence in unfamiliar languages. Yet final-answer accuracy is fundamentally insufficient: fluent but invalid rules routinely reach the same prediction, making it impossible to distinguish supported analyses from plausible confabulations. We introduce Executable Trust, a generator-agnostic framework that compiles rule proposals from language models or human experts into typed executable hypotheses and validates them before query labels are revealed. Four gates-leave-one-context reconstruction, counterexample sensitivity, perturbation stability, and support calibration-jointly produce an auditable admit, reject, or abstain decision with a replayable evidence trace. Across six rule families spanning numeral systems, morphology, allomorphy, agreement, role alignment, and conditioned forms, the protocol decisively admits three supported families (Yidiny morphology, Sulka numeral-classifier structure, and Zoque conditioned allomorphy) and rejects three under-specified or unstable alternatives (Kayapo, Wambaya, and Benabena), with every decision backed by reconstruction scores, perturbation variance, and counterexample counts. On 32 query-blind typed queries, a strict selective policy answers 20 and is correct on all 20-yielding 62.5% coverage at 100% answered precision (95% Wilson interval: 83.9-100.0%). The 12 abstentions are not failures but calibrated refusals, each traced to reconstruction or stability gates that flagged insufficient evidence. An expert-authored derivation language of 21 rules executes all hidden-context analyses correctly under systematic source-side ciphers across 5 cipher strengths and rejects 68 of 71 corrupted controls (95.8% rejection rate); the three survivors are behaviorally equivalent rule-order permutations. Executable validation constitutes a falsifiable trust layer between linguistic hypothesis generation and answer production, forcing abstention and representation failures into the open rather than allowing them to hide behind answer accuracy.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.