acceptodds
Under review as a conference paper at ICLR 2027

Language-Conditioned Executable World Hypotheses for UAVs: Causal Evaluation and Certified Resolution

Abstract

Language-conditioned UAV systems can emit syntactically valid, executable actions whose behavior, referents, or evidential support are nevertheless wrong. Existing evaluations usually isolate interface validity, semantic accuracy, or trajectory error, obscuring whether a model’s complete decision is both justified and executable. We formalize the executable world hypothesis, a contract-bound claim that couples an intended operation with typed grounding, epistemic support, and deterministic rollout, and define Executable-World-Hypothesis Correctness (EWHC) as its end-to-end criterion. We instantiate this problem in AeroWorld32K, a benchmark independent of model family with 4,096 executable UAV scenes and 32,768 causally paired queries spanning supported and open world conditions. We further introduce Certificate-guided Epistemic Resolution for Executable States (CERES), a deterministic resolver that requires no training and audits one frozen model completion against a typed public evidence representation, certifying a uniquely grounded and supported interpretation or compiling an explicit abstention to the protocol defined safe hold. Across ten frozen languagemodel configurations and six deterministic comparators, unassisted models remain limited particularly at the open world support boundary, and a deterministic symbolic resolver outperforms every unassisted language model. Giving five representative models the same public information in a typed representation closes a mean 9.44% of their model-wise Direct-to-CERES gaps. CERES yields an average absolute EWHC gain of 35.07% across all model configurations, repairs 114,908 Direct errors, and causes no observed EWHC harm, harm on supported cases, or unsafe determinate output on the complete evaluation population. Preregistered closed loop simulations further show that small ADE/FDE can coexist with an incorrect executable world hypothesis. These results establish evidence grounded resolution as a practical complement to language conditioned action generation under open world uncertainty.

Primary area:
causal reasoning
Keywords:
UAV autonomy, executable world hypotheses, causal evaluation, evidence-grounded resolution, auditable abstention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LxxXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines EWHC, an “executable world hypothesis” and a score. It checks whether a model output has the right behavior, the right typed grounding, enough evidence, and a correct compiled rollout. The authors build AeroWorld-32K, a UAV benchmark with 4,096 scenes and 32,768 queries. They also propose CERES, a deterministic resolver that either certifies one grounded interpretation or outputs a safe hold. They test it on ten frozen LLMs and six baselines.

Critical strengths

The problem framing is clear and useful. EWHC separates schema compliance from real grounding and execution. This is a good point for language-conditioned embodied systems.

The experiments are broad and consistent. CERES improves every model without extra training or model calls, also, the safe-hold outputs are recorded and can be checked.

Critical weaknesses

The test set is not from real flights. CERES uses rules written by hand. AeroWorld-32K comes from UAV worlds that can be run. CERES depends on fixed contracts and an evidence graph.

The safety claims only cover the synthetic cases. The paper does not show much real-world safety.

Questions for authors

How many CERES rules are written by hand? How hard is it to move them to a new UAV environment or a new operational protocol?

How are the expected-unknown labels and support boundaries defined?

Do the authors plan to test on real flight data or a higher-fidelity simulator beyond the four PyBullet cases?

2 comments
AuthorsOpenReview ↗

Fixed Contracts, Rules, and Transfer

W1: Synthetic worlds and hand-written rules

Yes: this is a controlled synthetic, rule-friendly benchmark with hand-written resolution and fixed contracts, not independent semantic generalization. Labels and parsing use distinct calls but co-designed template language, geometry and predicates. Perfect within-contract agreement is a co-design ceiling.

EWHC requires semantic exactness plus 30-second replay with ADE/FDE at most 5 m; supported requests require expected behavior/arguments without fallback, and expected-unknown requests the expected reason and unknown-abstention status. DE_full counts erroneous determinate releases over all requests.

Direct is model output without the resolver layer. For the matched sample (pilot queries across model configurations), CERES vs Direct EWHC is 295/496 vs 107/496 on original text and 270/496 vs 113/496 on rewritten text; erroneous releases are 0/496 vs 91/496 and 9/496 vs 87/496, respectively. For the disjoint sample, EWHC is 1137/1968 vs 526/1968 (original) and 840/1968 vs 503/1968 (rewritten); erroneous releases are 0/1968 vs 309/1968 and 76/1968 vs 296/1968. Filtering benefits come with lower supported coverage: 145/344 vs 146/344 (original) and 129/344 vs 153/344 (rewritten) in the matched sample; 667/1488 vs 668/1488 and 482/1488 vs 637/1488 in the disjoint sample.

Q1: How many rules, and how difficult is transfer?

There is no single defined semantic-rule total; transfer engineering effort and cross-domain modification scope are not run as measurements. The hand-written layer contains six supported-behavior keys, six entity-reference argument types and four selector families: identity/name matching, current-position distance extrema, coordinate-axis extrema and relative direction. The contract stores four refusal reasons; two are used as expected-unknown labels. Using top-level rejecting if clauses as the counting unit, _graph_evidence has three and _outcome_status five; after eligibility checks, the final canonical-lane acceptance expression has two conjuncts.

The interface checks identity, schema and typing; recovery filters/orders typed public entities and substitutes bindings while preserving the proposed high-level behavior; release checks evidence and compiled outcomes or abstains. Interface acceptance is not entity correctness. Symbolic integration demonstrates a common interface, not arbitrary-source portability or equal performance; other integrations are not run. The authored protocol fixtures pass 56/64, with nearest variants 0/8 because their surface form falls outside the frozen grammar.

Within-contract mechanism evidence

Policy A (A), symbolic proposals through the existing recovery/release layer, has EWHC 315,530/327,680 and DE_full 0/327,680 on aligned symbolic records under zero disturbance. Verification-only L2 keeps 197,760/197,760 originally correct supported releases and withholds 35,850/35,850 erroneous ones, without modifying retained actions; its EWHC remains 267,610/327,680, equal to original symbolic. This is a retained-set identity under co-designed predicates, not independent semantic-discrimination validation. L2-to-A adds 35,850/245,760 correct supported releases by recovering bindings, plus 12,070/81,920 expected-unknown successes from cause policy: 35,850+12,070=47,920 additional correct instances over 327,680 aligned requests. The distinction is coverage/filtering, not necessity of LLM proposals for sealed zero errors.

AuthorsOpenReview ↗

Support Boundaries, Safety Scope, and Validation Direction

Q2: Expected-unknown labels and support boundaries

The support boundary is the fixed behavior set and its required reference keys. The formal supported constructor selects behavior/arguments, renders the message and stores the directive resolution as its label. unsupported_behavior denotes a directive outside that set. Formal insufficient_information cases supply an intended supported non-hold behavior with empty arguments, leaving required references missing.

In the sealed aligned population, expected-unknown requests are 81,920/327,680. A predicts unsupported_behavior for 20,480/81,920 and insufficient_information for 61,440/81,920. On the same 81,920 expected-unknown requests, cause exactness is 64,452/81,920 for resolver control (referent recovery, evidence checks and abstention) and 78,751/81,920 for CERES on native-model proposals, and 81,920/81,920 for A on symbolic proposals (cross-source comparison). The last is construction-label agreement, not general unknown recognition. Reasons depend on proposal bindings and are not one-to-one first-rejection explanations; per-check pass vectors are field absent. Tied extrema yield insufficient information, but 19,604/73,080 of CERES's 73,080/245,760 withheld sealed supported requests depend on non-tied final-graph extrema. Complete tie-rejection events are field absent. Semantic potential does not mean additional completed tasks. Policy B's supported semantic ceiling is 245,760/245,760.

W2: Safety evidence

The measured contribution is a reduction in erroneous determinate releases, with the coverage cost reported explicitly. On rewritten text, CERES has 9/496 erroneous releases versus Direct's 87/496 in the matched sample, and 76/1968 versus 296/1968 in the preregistered disjoint sample. As reported in Fixed Contracts, Rules, and Transfer, CERES also releases fewer supported requests in both rewritten populations. These comparisons quantify a decision-level filtering trade-off, not an unrestricted benefit from abstention.

The matched sample's 9/496 erroneous releases are traced to the shared extraction front-end; resolver control and CERES release the same entity in those cases. The 76/1968 disjoint-sample erroneous releases were not traced individually, so that attribution is not extended to them. The contribution is therefore an explicit accounting of retained errors and coverage, rather than a safety inference from a zero count.

Q3: Real flight and higher fidelity

Our proposed direction extends the paper's distinction between semantic correctness, execution outcomes and coverage to higher-fidelity simulation and real-flight evaluation. Beyond the four existing PyBullet cases, both extensions are not run. Higher-fidelity simulation would assess semantic correctness separately from execution success; real-flight evaluation would assess evidence-based decision validity separately from physical execution outcomes, with coverage reported separately in each setting. This is a proposed research direction, not completed validation or a fixed implementation commitment: the protocol, resources, and schedule remain to be established.

Contribution and remaining limits

The contribution is measuring errors hidden by aggregate compliance and exposing coverage/filtering trade-offs. Original symbolic releases 35,850/245,760 contract-valid, behavior-correct but entity-incorrect supported actions; 500/35,850 pass both 5 m geometry thresholds. Among native-model proposals subsequently repaired for supported bindings, 33,580/39,055 were contract-valid—not an execution count; a separately identified replay result for that subset is field absent. Across different sealed sources, A releases 233,610/245,760 supported aligned-symbolic requests and CERES 172,680/245,760 supported native-model requests, each with DE_full 0/327,680. On native model proposals, CERES raises EWHC from 136,523/327,680 to 251,431/327,680, improving all ten evaluated configurations (10/10), with 0/327,680 correct-to-incorrect transitions.

Reviewer_Be1KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how well language models make decisions for UAVs based on operator instructions. The authors argue that getting the action format right or following a similar trajectory is not enough if the model chooses the wrong action. For example, a model might select a landing site even when the available evidence does not support either option. To address this, they introduce EWHC, a metric that checks whether the predicted action, grounded entities, supporting evidence, and resulting behavior all match the reference. They also create AeroWorld-32K, a synthetic benchmark with paraphrased instructions, controlled changes to the world state, and cases where the model should abstain. Their proposed method, CERES, checks an LLM's output against available evidence and either accepts a uniquely supported action or defaults to a safe hold. Across ten small open models, CERES improves scores considerably. However, a simpler rule-based resolver also outperforms all the LLMs used without additional checks.

Critical strengths
  • The paper makes a good point about the limits of trajectory-based evaluation. An agent can follow a trajectory close to the reference and still make the wrong decision. The paired experiments make this clear: some models change their answers for over 99% of intervention pairs but get both cases right only around 30–37% of the time. Simply changing an answer when the world changes does not mean the model is responding correctly.

  • The benchmark also highlights how poorly these models handle cases where the evidence is insufficient. Most models rarely abstain when they should, and performance on unknown requests is much worse than on supported ones. This is a useful finding for UAV control, where acting on an unsupported guess can be more problematic than refusing to act.

Critical weaknesses
  • Most of the gain comes from the strict adapter. For Gemma-3-12B, the score improves from 56.33 to 96.76 with the strict control, while CERES only brings it to 99.16. Similar trends appear across models. This makes it unclear how much the evidence-based resolver actually adds. The paper should focus more on the gains over the strict control.

  • The benchmark may favour CERES. The instructions are template-generated, and the resolver uses rules that closely match the benchmark setup. The symbolic resolver alone scores 81.67%, but there are no tests on human-written instructions or data from an independent generator. The experiments also use only small open models. Finally, changing EWHC's distance threshold from 0 to 20 metres does not affect the results, which raises questions about whether the trajectory component is needed.

Questions for authors

Can you report the gains from the strict adapter and CERES separately?

  • Have you tested CERES on human-written instructions or data from another generator?

  • Why does CERES withhold 29.74% of supported requests despite reporting no harm? Were these cases already answered incorrectly by the original LLM?

  • How would CERES compare with larger models, especially on unknown requests?

  • How does CERES compare with a simple rule that abstains when multiple compatible candidates exist?

  • Are the zero Direct scores for Llama-3.2-1B and Phi-4-mini mainly due to output-format issues?

4 comments
AuthorsOpenReview ↗

Component Attribution, Benchmark Dependence, and Coverage

W1 / Q1: Gains over the strict control

CERES adds 14,299/327,680 EWHC-correct sealed native-model instances beyond resolver control, all on expected-unknown requests; supported repairs at this stage are 0/245,760. Our “strict adapter” label denoted resolver control (recovery, evidence checks and abstention), not the shared interface adapter alone. Direct is model output; CERES adds cause discrimination.

For Gemma-3-12B, resolver-control EWHC is 31,707/32,768 versus CERES 32,493/32,768: 786/32,768 additional correct instances, entirely from unknown-cause exactness, 7,295/8,192→8,081/8,192. Over sealed model records, resolver control contributes 100,609/114,908 repairs and CERES 14,299/114,908. Supported substitutions 39,055/245,760 plus unknown-cause repairs 61,554/81,920 close the control count: 39,055+61,554=100,609. Adding CERES's cause repairs gives 61,554+14,299=75,853 unknown repairs over 81,920 expected-unknown requests.

Within the subsequently repaired supported-binding set, 33,580/39,055 proposals were contract-valid—not an execution count; a separately identified replay result for that subset is field absent. Policy A (A) applies the same recovery/release layer to symbolic proposals aligned to the same queries across ten configurations; its replay is not this count.

In the matched (pilot-query) and disjoint (preregistered, pilot-disjoint) samples, CERES and resolver control have identical supported-release, determinate-release and DE_full (erroneous releases/all records) aggregates. Their EWHC differences, original then rewritten, are 44/496 and 49/496 (matched), 103/1968 and 87/1968 (disjoint), each equal to the unknown-reason exact-match difference.

W2 / Q2: Benchmark dependence and geometry

Yes: the co-designed benchmark favours CERES. Human-written instructions and a non-isomorphic generator are not run; language, geometry and predicates are co-designed despite separate label/parsing calls. Under rewriting, CERES's erroneous releases rise from 0/496 to 9/496 (matched) and from 0/1968 to 76/1968 (preregistered); the former are traced to the shared extraction front-end, the latter are not traced (table in Q3).

EWHC requires semantic exactness and 30-second zero-disturbance replay with ADE/FDE at most 5 m. Supported outputs need expected behavior/arguments without fallback; expected-unknown outputs need the expected reason and unknown-abstention status. For A, 0/233,610 semantically correct aligned-symbolic releases fail geometry. Conversely, 500/35,850 entity-incorrect original symbolic releases pass both thresholds. Geometry alone admits some binding errors, but its independent contribution here is not established; this does not make trajectory evaluation uninformative elsewhere.

Q3: Withheld supported requests and “no harm”

“No harm” denotes 0/327,680 correct-to-incorrect transitions on sealed native-model records, not zero coverage cost. None of the 73,080/245,760 withheld supported requests was EWHC-correct under Direct or resolver control. Within those 73,080 requests, original Direct outputs were invalid 61,992/73,080, incorrect determinate 9,548/73,080 and explicit abstentions 1,540/73,080. CERES refusal categories were adapter rejection 58,469/73,080, initial-graph failure 13,071/73,080 and originally unknown proposals 1,540/73,080: 58,469+13,071+1,540=73,080. Failure under EWHC does not mean every withheld request was a wrong-entity release.

Final-graph matches for withheld requests describe semantic potential, not completed tasks. Policy B's sealed supported semantic ceiling is 245,760/245,760; execution/replay not run.

Q4: Larger models

Larger-model evaluation is not run, so these counts do not establish a scale trend. In sealed native-model records, Direct unknown-cause exactness is 2,898/81,920 overall and at most 1,320/8,192 for a configuration (Phi-4), versus CERES 78,751/81,920. Qwen3-14B and Phi-4 are the largest represented configurations.

AuthorsOpenReview ↗

Coverage Table, Abstention Controls, and Collapsed Configurations

Q3 (continued)

Matched: pilot queries across configurations. Disjoint: preregistered pilot-disjoint queries. Cells: CERES vs Direct, % (n/N); coverage uses supported records, DE_full all records, precision each condition's own determinate releases.

Sample / textSupported coverageDE_fullRelease precision
Matched / original42.15% (145/344) vs 42.44% (146/344)0.00% (0/496) vs 18.35% (91/496)100.00% (145/145) vs 53.57% (105/196)
Matched / rewritten37.50% (129/344) vs 44.48% (153/344)1.81% (9/496) vs 17.54% (87/496)93.02% (120/129) vs 56.28% (112/199)
Disjoint / original44.83% (667/1488) vs 44.89% (668/1488)0.00% (0/1968) vs 15.70% (309/1968)100.00% (667/667) vs 62.86% (523/832)
Disjoint / rewritten32.39% (482/1488) vs 42.81% (637/1488)3.86% (76/1968) vs 15.04% (296/1968)84.23% (406/482) vs 62.67% (497/793)

CERES releases fewer supported requests in every row: by 1/344 and 1/1488 on original text, and by 24/344 and 155/1488 on rewritten text. Sealed 0/327,680 correct-to-incorrect transitions and nonzero rewritten erroneous releases concern different populations and quantities; not pooled.

Q5: Abstain when multiple compatible candidates exist

We measured both raw-type compatibility (L1) and compatibility after frozen evidence/predicate checks (L2). L2 verifies without replacing parameters. On the same sealed native-model proposals, L2 gives supported releases 133,625/245,760 and EWHC 202,583/327,680; CERES gives 172,680/245,760 and 251,431/327,680, respectively, both with DE_full 0/327,680. The EWHC gain is 48,848/327,680: 39,055/245,760 supported plus 9,793/81,920 expected-unknown; 39,055+9,793=48,848. The supported gain equals the recorded referent-repair count.

L1 produces 5,927/327,680 erroneous predicted holds over ten native configurations: hold has no entity slots, so singleton checking misses its behavior correctness. This is distinct from symbolic alignment, where all 40,960/40,960 L1 supported releases are hold.

On the aligned symbolic source, L2 retains 197,760/197,760 originally correct supported releases and withholds 35,850/35,850 erroneous ones without altering retained actions. EWHC stays 267,610/327,680, equal to original symbolic. This retained-set identity is contract-internal under co-designed predicates, not independent semantic-discrimination validation. L2-to-A adds 35,850/245,760 correct supported releases through recovery and 12,070/81,920 unknown successes through cause policy; the latter is not substitution.

Q6: Are zero Direct scores mainly format failures?

For JSON extraction, yes for Phi-4-mini: 32,768/32,768 recorded rejections; for Llama-3.2-1B only 136/32,768 are extraction failures. Both have interface acceptance 0/32,768; successful JSON extraction does not establish schema/decision-contract compliance.

Llama's first recorded diagnoses are final_decision_fields 19,451/32,768, unknown_arguments 13,170/32,768, evidence_conflict 11/32,768 and top_level_type 136/32,768. The extraction failures split into multiple objects 92/32,768 and no object 44/32,768. The checks respectively compare the decision field set with the required fields, reject nonempty arguments for an unknown behavior, and compare evidence-message IDs with expected evidence. Sequential checking means the unknown-arguments records passed preceding field-set, schema-version, subject, evidence, confidence and type checks.

Phi-4-mini has no-top-level-JSON-object rejection in 32,768/32,768 records. For each configuration, Direct EWHC is 0/32,768, CERES EWHC is 8,192/32,768 and supported releases are 0/24,576. CERES correctness comes entirely from expected-unknown abstention; zero erroneous releases are vacuous: no determinate release in this population. Token-level generation-cause logs are field absent. Both configurations remain in aggregate CERES EWHC 251,431/327,680.

I thank the authors for the detailed evaluation and responses in correspondence to the questions/weakness raised. These has resolved the concerns which I had previously had mostly around direct scores and questions around benchmark. I hence decided to increase my score. I hope the authors update the draft based on the above discussion appropriately before the rebuttal phase ends.

Thank you for the discussion and for asking us to carry these clarifications into the paper. We have uploaded the revised manuscript, with changes marked in blue.

Reviewer_Pvi3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors tackle the problem of valid and executable but wrong UAV actions. In the context of language conditioned drones, they receive commands in natural language or a restricted natural language and they must turn this into an executable action but the actions might be wrong. The authors tackle this problem by forcing the models to produce a structured output so that drones cannot bind to a wrong entity or perform an action that the current evidence cannot support as being valid. The authors also build a “deterministic resolver” that certifies a particular action or trajectory.

Critical strengths
  1. The motivation for the paper which is that a command can be valid and executable and still be wrong is a very valid failure mode and so a solution is urgent.
Critical weaknesses
  1. From the results, the strict adapter is responsible for most of the 35% increase and CERES adds about a 7% increase to it. I believe this makes the design more complicated than it needs to be and that a more simplified framework could achieve better results.
  2. A rules-based resolver with no language model is able to score 81% which is higher than any language model. I think that this is a failure of the benchmark rather than the models because the benchmark appears fully solvable by CERES. This makes me wonder if CERES can handle real operator language where the ambiguity and the stochastic nature of the environment might lead to some wrong entity binding. This might be a very big ask but could you test out some human-written messages?
Questions for authors
  1. How well does the rules-based resolver do when it’s combined with the CERES’s abstention step with no language model? If it matches the language model, what value is adding the LLM then?
2 comments
AuthorsOpenReview ↗

Rule-Based Proposals, Recovery, and the Scope of Our Claims

Q1: Rule proposals with recovery and abstention

Without an LLM, Policy A (A) applies the existing recovery/release layer to symbolic proposals and reaches EWHC 315,530/327,680 with DE_full 0/327,680. LLM proposals are therefore not necessary for zero erroneous releases under this sealed synthetic contract and zero disturbance. This is a cross-source comparison with CERES on native model proposals, 251,431/327,680, not a same-source ablation or an advantage on every request.

EWHC requires original-scorer semantic exactness and 30-second zero-disturbance replay with ADE and FDE at most 5 m. Supported requests require expected behavior/arguments without fallback; expected-unknown requests require the expected reason and unknown-abstention status. DE_full counts erroneous determinate releases over all requests.

The closest measured control to abstention without recovery is L2: frozen verification without parameter substitution. The ladder was designed after the main study, with its policies, scoring/replay settings and comparison rules registered before its execution.

Symbolic conditionSupported coverageDE_fullFull EWHC
Original symbolic95.06% (233,610/245,760)10.94% (35,850/327,680)81.67% (267,610/327,680)
L1: raw-type abstention16.67% (40,960/245,760)0.00% (0/327,680)33.82% (110,810/327,680)
L2: verification only80.47% (197,760/245,760)0.00% (0/327,680)81.67% (267,610/327,680)
A: full recovery95.06% (233,610/245,760)0.00% (0/327,680)96.29% (315,530/327,680)

These counts align the same symbolic queries across ten configurations; they are not independent replications or the native-model population. Original symbolic is an output condition; the other rows apply policies to that source.

L2 retains all 197,760/197,760 originally correct supported releases and withholds all 35,850/35,850 erroneous supported releases, without changing retained actions or turning original unknowns into releases. This is a retained-set identity, not merely equal totals: contract-internal under co-designed predicates, not independent semantic-discrimination validation.

L2-to-A adds 35,850/245,760 correct supported releases by replacing withheld incorrect bindings, plus 12,070/81,920 expected-unknown successes from cause policy: 35,850+12,070=47,920 additional EWHC-correct instances over 327,680 aligned requests. The unknown gain is not parameter substitution.

The comparison is not uniform. Among ten model configurations, 3/10 exceed A's unique-query EWHC 31,553/32,768: Gemma-3-12B 32,493/32,768, Phi-4 31,854/32,768 and Qwen3-14B 32,238/32,768. Gemma correctly releases 24,412/24,576 supported requests versus A's 23,361/24,576. Aggregate release sets are nonnested: only-A 69,172/245,760 and only-CERES 8,242/245,760. These are configuration-specific benefits, not LLM necessity or a controlled scale effect.

L1's 40,960/40,960 supported symbolic releases are entity-slot-free hold. Its singleton test ignores hold behavior correctness. On ten native model configurations, each with 32,768 records, L1 releases 5,927/327,680 erroneous predicted holds; L2 releases 0/327,680 erroneous actions. This differs from symbolic alignment.

AuthorsOpenReview ↗

Complexity, Language Scope, and Contribution

W1: Is the architecture unnecessarily complex?

The measured components have distinct contributions, but architectural minimality is not established. Our “strict adapter” label meant resolver control—referent recovery, evidence checks and abstention—not the shared interface adapter alone; Direct is model output, and CERES additionally changes unknown-cause discrimination.

On the same aligned symbolic source, verification adds 156,800/245,760 supported releases over raw-type abstention; recovery adds 35,850/245,760 correct supported releases over verification by correcting withheld bindings, each with DE_full 0/327,680. These increments measure coverage and filtering, not equivalence to arbitrary simpler systems.

On sealed native model proposals, resolver control supplies 100,609/114,908 repairs; CERES's cause layer supplies 14,299/114,908. Resolver control's supported substitutions are 39,055/245,760; expected-unknown cause repairs are 61,554/81,920. The count decomposition is 39,055+61,554=100,609. CERES's additional repairs cannot be credited to referent recovery.

Within the subsequently repaired supported-binding set, 33,580/39,055 original model proposals were contract-valid. This is not an execution count; a separately identified replay result for that subset is field absent. Recovery preserves the proposed high-level behavior while replacing its binding with the uniquely supported typed referent.

W2: Rule-friendly benchmark and human-written language

No: human-written instruction evaluation is not run. A non-isomorphic generator and stochastic-environment evaluation are not run. The benchmark is rule-friendly: labels and parsing use separate calls but co-designed template language, geometry and predicates. Their agreement is a co-design ceiling, not independent semantic generalization.

In the matched sample (pilot queries across model configurations), CERES EWHC falls from 295/496 original-text records to 270/496 rewritten-text records; erroneous releases rise from 0/496 to 9/496, also for resolver control. In the disjoint sample, CERES EWHC falls from 1137/1968 to 840/1968; erroneous releases rise from 0/1968 to 76/1968, also for resolver control. On rewritten text, Direct has EWHC 113/496 and 503/1968, with erroneous releases 87/496 and 296/1968. These populations are separate.

All 9/9 traced instances (three model configurations × the same three queries) are the matched sample's 9/496 rewritten erroneous releases, attributed to text-side surface-form dependence; the other three traced categories each have 0/9. Control and CERES release the same entity in these cases. Our post-freeze review rates meaning and referent as preserved for the three traced queries; the same review rates 29/313 reviewed pairs not double-yes. The traced-query finding is limited to those queries, is not independent validation, and does not adjust any experimental score.

Retained contribution

The contribution is to measure hidden semantic errors and the coverage/filtering effects of reconstructible criteria within the stated contract. The separately implemented original symbolic baseline releases 35,850/245,760 contract-valid, behavior-correct but entity-incorrect supported actions; 500/35,850 pass both 5 m geometry thresholds. Geometry alone therefore admits some binding errors.

Across different sealed proposal sources, A releases 233,610/245,760 supported aligned-symbolic requests with DE_full 0/327,680; CERES releases 172,680/245,760 supported native-model requests with DE_full 0/327,680. These are source-dependent coverage/filtering results, not a source-independent zero-error property. CERES raises native-model EWHC from 136,523/327,680 to 251,431/327,680, improving all ten evaluated configurations (10/10), with 0/327,680 correct-to-incorrect transitions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Public Response: Clarifying Component Contributions and the Tradeoff between Recovery and Abstention

EWHC requires semantic exactness under the original scorer and replay for 30 seconds with zero disturbance, with ADE and FDE ≤5\le 5 m. Supported requests require the expected behavior/arguments without fallback; requests whose expected outcome is unknown require an unknown output with the expected reason and abstention status. The full determinate error metric (DE_full) counts determinate releases failing EWHC, over all requests.

Results on new samples

The matched sample, matched to the pilot query set, has 496 records of model and query combinations per text condition; the disjoint sample, using query IDs separate from the pilot, has 1968 such records per text condition.

Sample / textConditionFull EWHCDE_fullRelease precisionSupported coverage
Matched / originalCERES59.48% (295/496)0.00% (0/496)100.00% (145/145)42.15% (145/344)
Matched / originalDirect21.57% (107/496)18.35% (91/496)53.57% (105/196)42.44% (146/344)
Matched / rewrittenCERES54.44% (270/496)1.81% (9/496)93.02% (120/129)37.50% (129/344)
Matched / rewrittenDirect22.78% (113/496)17.54% (87/496)56.28% (112/199)44.48% (153/344)
Disjoint / originalCERES57.77% (1137/1968)0.00% (0/1968)100.00% (667/667)44.83% (667/1488)
Disjoint / originalDirect26.73% (526/1968)15.70% (309/1968)62.86% (523/832)44.89% (668/1488)
Disjoint / rewrittenCERES42.68% (840/1968)3.86% (76/1968)84.23% (406/482)32.39% (482/1488)
Disjoint / rewrittenDirect25.56% (503/1968)15.04% (296/1968)62.67% (497/793)42.81% (637/1488)

Each row uses its own sample and text condition. Full EWHC and DE_full use all records of model and query combinations; precision uses determinate releases; supported coverage uses requests with expected behavior other than unknown (344 matched; 1488 disjoint). These populations are not pooled with each other, the pilot, or the sealed full population.

Under rewriting, CERES releases fewer supported requests than Direct: 129/344 versus 153/344 in the matched sample, and 482/1488 versus 637/1488 in the disjoint sample.

What is attributable to CERES beyond control?

Our “strict adapter” label denoted resolver control, which includes referent recovery, evidence checks and abstention, rather than the shared interface adapter alone. Direct is model output; CERES additionally changes discrimination among unknown causes.

In both samples and both text conditions, CERES and control have identical aggregate counts of supported releases and determinate releases, and identical DE_full. Their numerator differences in full EWHC equal their numerator differences in exact matches of the unknown reason: matched original 44/496 and rewritten 49/496; disjoint original 103/1968 and rewritten 87/1968. These fractions use denominators for the full sample, not the unknown subset. This comparison does not isolate the contribution of referent recovery; that component is supported by comparisons on the sealed full population.

In the sealed population of native model proposals, resolver control supplies 100,609/114,908 repairs and CERES supplies 14,299/114,908, all latter repairs on requests whose expected outcome is unknown. Full EWHC rises from Direct 136,523/327,680 to CERES 251,431/327,680, improving all ten evaluated configurations (10/10). The 0/327,680 transitions from correct to incorrect describe regression, not zero coverage cost.

Reproduction and scope

Archived final outputs reproduce for 496/496 previously completed records of model, text and query combinations; checked fields from the original scoring also match for 496/496. This is reproduction, not independent semantic validation. Recovery deviations (a revised model file for one configuration, a change to the logging flag checked by a diagnostic probe for output neutrality, and a change to the power cap during one model's inference) are registered with the results.

AuthorsOpenReview ↗

Public Response 2: Mechanism, Author Review, and Remaining Scope

Sensitivity of the text front end

Tracing assigns 9/9 erroneous release instances (three model configurations × the same three queries) to dependence on textual surface form. Each other traced category is 0/9. Control and CERES release the same entity in the traced cases; attribution concerns their shared extraction front end, not a difference in CERES's release decision. These 9 traced instances are the 9/496 erroneous releases on rewritten text in the matched sample. The 76/1968 erroneous releases in the preregistered disjoint sample were not traced individually; their attribution is not run, and the attribution to the front end above is not extended to them.

In the separate comparator population, each source has 63 requests per text condition. After the same layer, original→rewritten DE_full is: keyword_nearest 0/63→3/63; symbolic_belief_tracker 0/63→3/63; rule_parser 0/63→2/63; random_valid 0/63→1/63; always_unknown 0/63→0/63; state_only 0/63→0/63. always_unknown has supported releases 0/44 in each condition: vacuous: no determinate release in this population. state_only has 9/44 original and 8/44 rewritten supported releases; no additional erroneous release was observed in these conditions.

Retained contribution and limits

The contribution is measuring errors hidden by aggregate compliance and exposing coverage/filtering tradeoffs. The separate original symbolic baseline releases 35,850/245,760 supported actions that are valid under the contract and correct in behavior but incorrect in entity binding; 500/35,850 pass both 5 m geometry thresholds. Geometry alone therefore admits some binding errors. Among sealed native model proposals subsequently repaired for supported bindings, 33,580/39,055 were contract-valid; this is not an execution count. A separately identified replay result for that subset is field absent.

L2 performs verification without recovery. Its unchanged retained set on aligned symbolic queries establishes an identity within the contract under jointly designed predicates, not independent validation of semantic discrimination. Policy A with recovery reaches 315,530/327,680 EWHC. Against CERES on native model proposals, supported release sets are nonnested: only A 69,172/245,760 and only CERES 8,242/245,760 (comparison across proposal sources). Policy B's sealed supported semantic ceiling is 245,760/245,760; execution/replay not run.

We thank the reviewers for their questions and suggestions. We have uploaded a revised manuscript with changes marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.