acceptodds
Under review as a conference paper at ICLR 2027

Executable Trust: Query-Blind Validation of Rule Proposals for Linguistics Olympiad Reasoning

Abstract

Linguistics Olympiad problems require solvers to induce latent rules from sparse evidence in unfamiliar languages. Yet final-answer accuracy is fundamentally insufficient: fluent but invalid rules routinely reach the same prediction, making it impossible to distinguish supported analyses from plausible confabulations. We introduce Executable Trust, a generator-agnostic framework that compiles rule proposals from language models or human experts into typed executable hypotheses and validates them before query labels are revealed. Four gates-leave-one-context reconstruction, counterexample sensitivity, perturbation stability, and support calibration-jointly produce an auditable admit, reject, or abstain decision with a replayable evidence trace. Across six rule families spanning numeral systems, morphology, allomorphy, agreement, role alignment, and conditioned forms, the protocol decisively admits three supported families (Yidiny morphology, Sulka numeral-classifier structure, and Zoque conditioned allomorphy) and rejects three under-specified or unstable alternatives (Kayapo, Wambaya, and Benabena), with every decision backed by reconstruction scores, perturbation variance, and counterexample counts. On 32 query-blind typed queries, a strict selective policy answers 20 and is correct on all 20-yielding 62.5% coverage at 100% answered precision (95% Wilson interval: 83.9-100.0%). The 12 abstentions are not failures but calibrated refusals, each traced to reconstruction or stability gates that flagged insufficient evidence. An expert-authored derivation language of 21 rules executes all hidden-context analyses correctly under systematic source-side ciphers across 5 cipher strengths and rejects 68 of 71 corrupted controls (95.8% rejection rate); the three survivors are behaviorally equivalent rule-order permutations. Executable validation constitutes a falsifiable trust layer between linguistic hypothesis generation and answer production, forcing abstention and representation failures into the open rather than allowing them to hide behind answer accuracy.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
linguistic reasoning, executable validation, abstention, neurosymbolic AI, rule induction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NhMSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Executable Trust, a framework for validating linguistic rule proposals before using them to answer Linguistics Olympiad queries. Model-generated or expert-authored analyses are compiled into typed executable hypotheses and assessed through context reconstruction, counterexample sensitivity, perturbation stability, and query-specific support. The framework distinguishes proposal rejection from abstention on insufficiently supported queries. The authors report pilot studies and a larger confirmation involving eight generators, 57,600 proposals, and 2,400 queries, together with expert controls, adversarial tests, and human audits.

Critical strengths
  1. The paper addresses a meaningful distinction between producing a correct answer and proposing a supported linguistic analysis. Separating proposal validation from query-specific abstention offers a useful perspective on evaluating linguistic reasoning beyond final-answer accuracy.

  2. The evaluation explicitly reports coverage alongside answered precision and distinguishes proposal counts from unique query outcomes. The pilot failure analyses, corrupted controls, and human audits provide complementary evidence about validator behavior and its limitations.

Critical weaknesses
  1. Insufficient specification of the core method and evaluation. Sections 3.1–3.2 do not adequately specify how a proposal (zz) is compiled into an executable hypothesis (hh), or how the confirmation study computes query-level support (sqrs_{qr}) and determines the relevant-run set (RqR_q). The paper also lacks a complete worked example connecting a real puzzle, a proposed rule, executable checks, and the final decision. The supplement’s statement that latent correctness is “sampled or assigned” additionally requires clarification of how per-run correctness is obtained from executed predictions and reference answers. These missing details make the central procedure and its reported reliability difficult to assess or reproduce.

  2. Unclear incremental value over existing executable hypothesis validation. Hypothesis Search [1] and iterative hypothesis refinement [2] already combine LLM-generated rules with executable checking and selection, while DSL-based rule induction has been studied directly on Linguistics Olympiad problems [3]. These works do not implement the same complete trust contract, but their relationship to the proposed framework is not discussed. The pilot selective-answering and perturbation analyses are useful; however, the large-scale confirmation lacks comparisons with simpler execution- or reconstruction-based selectors and controlled gate ablations at comparable coverage. Consequently, the additional reliability benefit of the proposed validation design remains unclear.

Minor presentation comment: Section 6, “Discussion,” contains no text and is immediately followed by Section 7, “Conclusion.”

References:

[1] Ruocheng Wang et al. Hypothesis Search: Inductive Reasoning with Language Models. ICLR 2024. https://arxiv.org/abs/2309.05660

[2] Linlu Qiu et al. Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement. ICLR 2024. https://arxiv.org/abs/2310.08559

[3] Saujas Vaduguru et al. Sample-efficient Linguistic Generalizations through Program Synthesis: Experiments with Phonology Problems. SIGMORPHON 2021. https://aclanthology.org/2021.sigmorphon-1.7/

Questions for authors
  1. Clarification: Could you provide the operational specification of the compiler and confirmation-stage support calculation, including (sqrs_{qr}) and (RqR_q), together with one complete example from an actual model proposal to the final decision?

  2. Clarification: What does “latent correctness is sampled or assigned” mean? Please explain how model outputs, executor predictions, reference answers, and the correctness labels used in Tables 6–8 are connected.

  3. Assessment-relevant validation: Can you provide a comparison against simple execution-fit or reconstruction-only selection on the same proposal pool, at matched coverage, and controlled results showing the incremental contribution of the additional gates?

Reviewer_kzUjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes "Executable Trust", a framework that compiles rule proposals for Linguistics Olympiad puzzles, from LMs or humans, into typed executable programs drawn from a hand-built library of 12 rule families. Four gates that never see query labels (leave-one-row-out reconstruction, counterexample sensitivity, perturbation stability including source-side ciphers, and support calibration) yield an admit/reject/abstain decision.

Critical strengths
  1. The framing is sensible. Separating reject (the proposal fails an executable check) from abstain (admitted, but unsupported for this query) and reporting risk–coverage with all failures in the denominator is better practice than final-answer accuracy alone.
  2. The evaluation design shows good statistical intent: cluster-level bootstrap, worst-family reporting, and openly reported negative results (Benabena selection failure, cipher collapse in Table 2). This only carries weight if the underlying data can be trusted (see W1).
Critical weaknesses

W1. The central evidence is unverifiable and hard to believe, and there are no baselines.

  • Supp. says "latent correctness is sampled or assigned independently of answer status for every query–run." The main text never explains how correctness is obtained. If it comes from executing proposals against gold answers, the paper should say so and release per-query traces. If any part is sampled or simulated, Tables 6–13 carry no evidential weight.
  • The implied numbers are surprising. At the 32B tier, the highest-support run is correct on 2,033/2,400 (~85%) of queries even at tau = 0.44 (Supp. Table 5). However, §5.2 reports that direct prompting, CoT, and a shallow verifier solve 0/50 pilot items. The paper does not reconcile these.
  • There is no comparison with the ungated accuracy of each generator on the same 2,400 queries, or with cheap selective-prediction baselines (self-consistency agreement, model confidence, LLM-judge at matched coverage). By Table 8, ungated precision is ~86%, and the frozen point reaches 93.3% only by dropping 44% of queries; the permutation null mean is 85.7%. Without baselines, one cannot tell whether executable validation adds anything over trivial confidence signals.

W2. What is validated is a hand-written library, and the headline pilot claims contradict the paper's own gate semantics.

  • The admit/reject verdicts in Table 1 concern the authors' own per-language prototypes (e.g., "Kayapo prototype"). They say little about LM or expert proposals, so "rejects three under-specified alternatives" is somewhat circular. How many real Olympiad problems the 12-family library can express is never quantified.
Questions for authors
  1. How exactly is "latent correctness" obtained (Supp. §1: "sampled or assigned")? Are labels from actually executing each proposal against gold answers? Can you provide per-query traces?
  2. Who wrote the 2,500 expert derivations (vs. 21 in the pilot), and how were the 2,500 corrupted controls generated?
  3. How are tier-level numbers derived from generator-level ones? The 7B tier answers 885 queries, fewer than either of its generators (1,098 and 907), and the 14B tier's precision (90.80%) exceeds both of its generators' (90.50%/85.48%).
Reviewer_pX7JOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Executable Trust, a validation layer for Linguistics Olympiad reasoning. Rule proposals from LLMs or experts are compiled into typed executable hypotheses and passed through four gates: leave-one-out reconstruction, counterexample sensitivity, perturbation stability, and support calibration. The result is an admit, reject, or abstain decision with a replayable trace.

Overall, the paper seems to propose a potential problem but contradicts itself in many places with dense text and missing details.

Critical strengths
  • Separating "proposed a valid analysis" from "reached a correct output" is a real gap in LO evaluation. Making the hypothesis the unit of trust, rather than the answer seems to be useful
  • Reject (proposal-level), abstain (query-level), and admit (enables, but does not guarantee) are cleanly separated and seem sufficient for this useful case.
  • Negative results are reported (although rarely analysed or used) and statistical tests are reported to ensure the validity of the experiment
Critical weaknesses

The work has several consistency issues and concerns.

  • Could not tell how correctness is computed in the large study. The paper calls it "latent correctness," and the supplement says it is "sampled or assigned." If outcomes are not measured against gold answers from real model outputs, the main results do not hold. This needs a clear answer and released traces.
  • The abstract describes only the pilot, which the paper itself calls a development diagnostic, and never mentions the main study.
  • The gates are not applied as described. Yidiny and Zoque collapse under the cipher test, which should reject them, yet they stay admitted and are called "decisively" supported.
  • Key details are missing: the data source, what a "cluster" is, how problems are assigned to families, how LLM output is parsed, the prompts, and who wrote the 2,500 expert derivations.
  • The models are older open models only, there is no comparison with standard LO benchmarks, and the Discussion section is empty.
Questions for authors
  • How exactly is correctness computed for each run? Is anything simulated?
  • How are tier-level answers formed from the two generators in each tier?
  • What makes a proposal "incorrect," and how does that relate to query-level precision?
  • Where do the 300 problems come from, what is a cluster, and why does each problem have exactly 8 queries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.