acceptodds
Under review as a conference paper at ICLR 2027

Verified Display for Language Interfaces to Optimisation-Based Control

Abstract

Language interfaces can answer questions about recorded runs of optimisation-based controllers. Verifying the structured claims a language model proposes does not ensure that its final prose contains only those claims. Across six models, answers whose claims passed verification still carry a scorer-flagged addition in 21 to 28 percent of paired cases. We introduce a verified display layer. The model proposes typed claims, deterministic checks evaluate them against the recorded solve, and a closed renderer turns only accepted claims into text that decodes back to those claims. In a paired experiment that changes only the displayed text, rendering removes every scorer-flagged addition and introduces no reverse transition; the preregistered split reproduces this result. Human adjudication finds a smaller but persistent rate of false statements in model-written prose. With tool calling, the layer trades coverage for lower displayed risk: useful coverage rises in dispatch and falls in HVAC, and the confidence interval of the pooled change includes zero. The guarantee covers declared predicates over the recorded committed solve, rather than controller correctness, plant behaviour, or causal explanation.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
verified text generation, faithfulness, hallucination, constrained decoding, optimisation-based control, model predictive control, KKT multipliers, explainable optimisation, selective prediction, neurosymbolic verification, LLM interfaces, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hf6dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies factual errors in language interfaces for querying recorded optimisation-based controller runs. It argues that verifying structured claims does not ensure that the accompanying model-written prose remains faithful to those claims. The proposed interface combines typed claims, deterministic record-based verification, and a closed renderer whose output can be decoded back into the accepted claims.

Experiments across three control domains compare model-written and rendered answers while holding accepted claims fixed, and evaluate the complete interface against ordinary tool calling. Rendering eliminates all observed scorer-flagged errors in the paired comparison, while the complete system exhibits domain-dependent trade-offs between reliability and useful coverage.

Critical strengths
  1. The paired experiment isolates the effect of the final display stage.

Holding accepted claims, verification outcomes, and retry behaviour fixed while comparing model-written and rendered text provides direct evidence that verified structured claims do not necessarily imply faithful final prose. This design makes the effect of the display mechanism easier to interpret than a comparison between two complete systems.

  1. The scope of the system’s guarantee is explicitly defined.

The paper distinguishes consistency with a recorded solve from controller correctness and causal explanation, and checks the implementation through an independent verifier and renderer–decoder consistency tests. This allows readers to understand what the system guarantees and which questions remain outside that guarantee.

Critical weaknesses
  1. The incremental contribution and necessity of the proposed machinery are not sufficiently established or clearly presented.

Existing work, including PCN, SymGen, and Think Fast, Talk Smart, already investigates binding displayed content to structured evidence and separating deterministic computation from language generation. This paper further restricts the entire answer to a closed grammar, but does not sufficiently establish what additional benefits this restriction and the control-specific verification machinery provide over a simpler deterministic question-answering pipeline.

This difficulty is also reflected in the figures and organisation of the main text. Figure 1 primarily presents an error example and aggregate paired results, while Figure 2 summarises the central verification mechanism as “five checks.” Neither figure explains concretely how the checks process a claim, when they cause refusal, or which components constitute the main technical contribution. Complete examples and important analyses are dispersed across the appendices. The main text lacks a worked example connecting the input, recorded evidence, verification decision, and final display, as well as a clear visual account of the trade-off between risk, coverage, and answer completeness. These presentation choices make the contribution and its necessity harder to assess.

A particularly relevant missing comparison is a non-oracle pipeline in which the language model translates the question into a structured query, code directly retrieves or computes the requested results from the record, and fixed templates produce the answer. This avoids asking the model to propose numerical values that subsequently require verification. The symbolic renderer supplied with gold claims is an upper bound and does not substitute for this baseline. Consequently, the current experiments do not sufficiently distinguish benefits attributable to the proposed verification machinery from those obtained by adopting deterministic, restricted output.

  1. The evaluation does not sufficiently distinguish more constrained output from more reliable and useful answers.

The main results rely on errors flagged by automatic scorers. In the small human audit, the prose scorer has precision 0.409 and recall 0.562, while rendered text is assessed through a structural scorer. The human audit and calibrated-judge analyses provide useful supplementary evidence, but do not fully resolve uncertainty about the magnitude of the improvement. Conformance to a predefined grammar and the absence of automatic flags remain different evaluation targets from factual correctness and completeness under an independent semantic assessment.

The matched-claims experiment controls answer availability, so its result cannot simply be attributed to increased refusal. However, refusal and information retention remain central to the comparison between the complete system and ordinary tool calling. On the primary model, pooled useful coverage improves by only 2.1 percentage points, with a 95% confidence interval that includes zero. The confirmatory experiment likewise does not establish a significant overall improvement, while useful coverage decreases in HVAC. The paper therefore demonstrates how to constrain displayed content more convincingly than it demonstrates a robust improvement in overall question-answering utility while preserving necessary information. This limits the demonstrated practical significance of the method.

Questions for authors
  1. Clarification and key evidence concerning the technical contribution:

Compared with a non-oracle pipeline consisting of natural-language-to-structured-query translation, direct record retrieval or computation, and fixed-template rendering, which failure modes are specifically addressed by the additional mechanisms proposed here? Please identify the key technical differences and provide a comparison that isolates their contribution under the same language model, accessible records, and claim semantics. This comparison would directly affect my assessment of the methodological novelty and whether the additional complexity is justified.

  1. Key evidence concerning practical utility:

The paired display experiment establishes that rendering reduces scorer-flagged errors when accepted claims are fixed. However, the complete system’s overall useful-coverage improvement over ordinary tool calling remains statistically inconclusive. Could the two complete systems be compared using a common semantic evaluation criterion that is independent of the output grammar, assessing factual errors, retention of information required by the question, and unnecessary refusals on answerable questions? The existing human audit and calibration analyses are useful, but more direct evidence is needed to establish whether fewer errors are accompanied by preservation of necessary information and ultimately a higher proportion of correct, complete answers. This would materially affect my assessment of the paper’s practical value.

2 comments

Thank you for the two concrete suggestions; both made the evaluation stronger. We ran the query–execute–render (QER) baseline and a common semantic evaluation. Intervals are 95% trajectory-bootstrap intervals (2,000 resamples within domain).

W1 - Non-oracle query–execute–render baseline

In QER, Llama 3.1 8B writes queries without values. Code runs them against the record, applies the status policy and renders the result.

(a) Full system: On the 1,456 test questions, QER's useful coverage is 0.776, against 0.620 for direct proposal (+0.156 [0.125, 0.185]). It is 0.171 [0.139, 0.202] above ordinary tool calling and higher on all five other open models (+0.185 to +0.339). Part of the gain is expressivity: QER answers 136 of the 344 relational questions usefully, and direct proposal answers none. QER has three scorer flags (risk 0.002); replay inspection classified all three as scorer errors. The two systems also differ in inputs, claim language, renderer version and attempts (one against up to three).

(b) Controlled comparison: We fixed the protocol before generation. Both arms received the same question, record snapshot and entity list (not the gold claims). They also used the same non-relational claim types, renderer, single attempt and structured-output prompt. On the 1,112 questions that need no relational claim, useful coverage is 0.902 for QER and 0.493 for direct proposal (+0.410 [0.376, 0.444]). Over all questions the difference is +0.314 [0.289, 0.340]. This compares two ways of obtaining facts, not query syntax alone.

(c) Display within QER: With QER's facts held fixed, closed rendering raises the judge-assessed correct-complete rate by 0.188 [0.164, 0.213] over model-written prose (exploratory, see W4).

So on this benchmark, retrieving facts by code works better than having the model propose values. This improves the acquisition stage but does not remove the display failure, which is the paper's main result. The status check is still needed for derived labels: it withholds 67 of 704 status mentions, although we have not ablated it within QER. We will present QER with the status checks and closed renderer as the recommended configuration (line 339). We will add the three comparisons to Table 3 (line 339) and Section 5.2 (line 314).

W2 - PCN, SymGen and Think Fast, Talk Smart

We test adaptations of these display policies, but not in the original systems. A proof-carrying-numbers-style policy hides unverified numbers but keeps the prose. Its scorer risk therefore stays at 0.095, because false status, comparison or action statements contain no number to hide. For our SymGen-style adaptation, the span detector flags unsupported content in 110 of 429 assessed answers (0.256). Our display test within QER (W1, part c) holds code-selected facts fixed and compares closed rendering with model-written prose. This tests the choice made in Think Fast, Talk Smart, letting a model phrase code-selected facts, without reproducing that system. We will make this explicit in Section 2 (lines 138–145).

W3 - Figures

Figure 2 (line 120) will become a worked example showing the question, record fields, checks, retry and display. It will include a rejected value and a status question that ends in refusal. A new figure in Section 5.3 (line 337) will show coverage, risk and correct-complete answers, with benchmark and judge results labelled separately.

W4 - Common semantic evaluation

We judged all systems with one rubric and one judge, Qwen3-32B. The judge sees the question, the record and the displayed answer, but not the benchmark labels. It rates meaning rather than our grammar and identifies the required facts itself. Agreement with human labels is modest (κ = 0.385 for factual errors, on items that also informed judge selection), so we treat these results as exploratory.

QER here is the full system from W1(a). Differences are against ordinary tool calling, on resolved pairs (both judgments parseable and certain), macro-averaged over domains.

Difference from ordinary tool callingResolved / all pairsFalse-statement rateRequired-fact retentionCorrect-completeCorrect-complete bounds
Tool calling + verified display1,330 / 1,456−0.186 [−0.213, −0.158]−0.023 [−0.042, −0.006]+0.048 [+0.023, +0.074][−0.007, +0.086]
Direct-proposal verified display1,330 / 1,456−0.169 [−0.201, −0.136]−0.079 [−0.099, −0.060]+0.008 [−0.018, +0.034][−0.037, +0.054]
QER1,322 / 1,456−0.180 [−0.208, −0.152]+0.039 [+0.017, +0.058]+0.143 [+0.115, +0.172][+0.096, +0.190]

A correct-complete answer states every required fact correctly, says nothing false or unsupported, and is not a refusal, so correct refusals get no credit. The last column sets unresolved judgments to their worst and best values; it is not a confidence interval.

The original pipelines cut false statements but lose some required facts, and their correct-complete gains are small. Part of the drop comes from refusals: on answerable questions they refuse 261 and 136 times, against none for tool calling. QER lowers false statements, keeps more required facts and raises correct-complete by 0.143. Restricting to pairs where the judge's answerability decisions agree gives +0.082, +0.057 and +0.165. With the same rubric, Gemma 2 27B found only one of seven factual errors (κ = 0.075). This is well below the κ = 0.524 it reached with the paper's original prompt (line 445).

We will add these results to Table 3 (line 339) and Section 5.3 (lines 359–368), and the calibration to Section 5.6 (line 445). They will be labelled exploratory and kept separate from benchmark useful coverage.

Reviewer_WfZaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies LLM interfaces that answer questions about recorded solves of optimisation-based controllers. It shows that verifying the structured claims an LLM proposes does not stop its final prose from adding statements that were never checked. As a fix, the paper proposes a "verified display" layer. Experiments are conducted on HVAC, vehicle control, and economic dispatch scenarios.

Critical strengths
  1. The paired intervention (same generations and accepted claims, only the display changes) cleanly isolates post-verification prose as its own failure point. The effect replicates across six open models, two proposal pipelines, three control domains, hosted models and two transfer domains. This matters because many verify-then-explain pipelines implicitly assume that verifying an intermediate object makes the final answer trustworthy.
Critical weaknesses
  1. Methodological novelty is limited. The pipeline combines well-established parts (LLM semantic parsing, deterministic checks, template NLG), and nothing is learned or adapted. The benchmark is generated from author templates, with gold labels derived from the same predicates the verifier uses.

  2. The paper is quite hard to understand (at least for me), and I wonder whether this topic fits the ICLR community.

Questions for authors

See the weakness above.

2 comments

We thank you for the review. Below we explain the task in plain terms, say what we consider new, report a simpler baseline we added, and describe how Section 3 will change.

The task is to answer questions about a recorded controller run. For example, the record may establish a generator's output or a constraint's status at a particular time. Our concern is that a model can report these facts correctly in its structured claims, then add an incorrect statement when writing the answer shown to the user.

W1 - Novelty

We do not claim a new learning algorithm or template technique. We study the gap between checking an intermediate representation and displaying a faithful answer. The method defines which statements the recorded solve supports and uses a renderer that cannot add statements outside the accepted claims.

We also tested whether the finding persists when the model does not propose numerical values. In the new query–execute–render baseline, the model selects what to query and code obtains the facts from the record. Holding those executed facts fixed, rendered answers have a correct-complete rate 0.188 [0.164, 0.213] higher than model-written answers under a common model judge (95% interval in brackets). This result is exploratory, but it suggests that the problem is not limited to checking model-proposed values. We will add QER as a stronger acquisition baseline and clarify that the display-stage contribution is acquisition-agnostic: it applies whether facts are proposed and verified or retrieved deterministically (line 339).

W2 - Template-generated benchmark and shared predicates

We agree that the benchmark shares declared semantics with the verifier and scorer. It therefore tests agreement with that specification, not whether the specification captures every fact an operator might need.

The paired display experiment holds accepted claims fixed, so the benchmark's question-to-claim mapping does not determine the difference between the two displays. However, the automatic flags still depend on our semantics and scorers. The blinded human audit and model-judge evaluation provide additional assessments against the record, but they do not remove this limitation. In particular, the new judge has only modest factuality agreement with human labels (κ = 0.385).

There is also a difference between the benchmark labels and the deployed policy that we need to state more clearly. The verifier applies a regularity check that the gold labels do not. This explains part of the HVAC coverage loss. Of the 109 HVAC questions that ordinary tool calling answers usefully but its verified-display version does not, 45 involve a status the check withholds. This is a conservative policy difference, not independent validation of the labels.

SEC filings and BOPTEST broaden the tested domains, but neither establishes transfer to independently authored operator questions. We will make these limits explicit beside Table 3 (line 339) and in Section 6 (line 466). We will also correct the statements at lines 212 and 1131 that imply identical verifier and benchmark predicates.

W3 - Presentation

We will begin Section 3 (line 150) with the task in plain language and replace Figure 2 (line 120) with a worked example.

The example asks for generator G6's output at 2001-03-27 16:00. The first proposal gives 0 MW and is rejected because the record gives 20.874 MW. A retry proposes the supported value. The rendered answer reports 20.874 MW, whereas the model-written answer accompanying the same accepted claims still says, "Generator G6 is not dispatched, as its output is 0 MW."

The figure will show the evidence, the rejection, the retry and the two displays. It will also include a case where the record cannot support the requested status and the system refuses.

W4 - Relevance to ICLR

We submitted the paper under ICLR's neurosymbolic and hybrid AI area. The central question is how verified symbolic evidence can pass through a language-model interface without losing its guarantee at the final generation stage. This question also arises in tool-augmented and retrieval-augmented systems, although our experiments test the recorded-data settings described in the paper.

Control logs are useful for this study because they support explicit checks of values and constraint statuses. We will explain that motivation earlier in the introduction (line 031) rather than assuming familiarity with optimisation-based control.

Thank you for the responses.

Overall, the rebuttal does not resolve my core concerns; instead, it deepens three underlying issues. I acknowledge that the additional results establish that even when facts are retrieved directly by code, generating final answers in free-form prose still introduces unverified content, confirming the role of closed-grammar rendering, and that the response more clearly distinguishes the proposed display policy from related work. However, my primary concerns remain unaddressed: limited methodological novelty and the reliance on an author-templated benchmark rather than independently authored operator questions are merely acknowledged; the newly introduced independent LLM judge exhibits only modest agreement with human annotations, limiting its evidentiary weight; and none of the promised structural or textual revisions have been incorporated into an updated manuscript for me to verify. More importantly, in my view, the additional experiments undermine the paper's original framing in three ways. First, the simpler query-execute-render (QER) baseline substantially outperforms the paper's central claim-proposal, verification, and retry pipeline in overall performance, prompting a shift to QER as the recommended configuration and thereby diminishing the necessity of the original verification architecture. Second, running the requested ablations revealed that Table 4 in the submitted manuscript compared separately prompted arms, invalidating the paper's claim that rendering unchecked claims increases risk. Third, in the single-attempt ablation on three frontier models, enabling verification eliminates scorer-flagged risk only at the expense of higher refusal rates and lower useful coverage across all three models, while the subsequent recovery with three attempts relies substantially on retry feedback supplying recorded values rather than on verification filtering alone.

To sum up, I would keep my initial rating as weak reject.

Reviewer_vBAhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliability of displayed answers in question answering over logs from optimisation-based controllers. A language model proposes typed claims, a program verifies them, and fixed templates render the accepted claims into the final answer. The aim is to prevent unverified additions when the model verbalises verified facts.

Critical strengths

The authors distinguish between verified claims and reliable final answers. Their paired experiments keep the accepted claim sets fixed and vary only the displayed text, showing that free generation can still introduce errors after verification. The closed renderer directly addresses this failure mode.

The evaluation covers multiple open-weight models and frontier models, including Opus, and is supplemented by a human audit, a Gemma judge, and extensions to SEC financial filings and BOPTEST. These provide evidence for the benefits of the display mechanism and strengthen the credibility of the results.

Critical weaknesses
  • The approach primarily combines established components: structured claims, verification rules, and template generation. The verification rules for control applications are useful, but the additional technical contribution over existing controlled language and template generation systems is not sufficiently clear.

  • Both display conditions in Appendix C.4 use the same intermediate verification and accepted claims, so the comparison isolates the benefit of template rendering. For frontier models such as Opus, it remains unclear how many additional errors intermediate verification filters out and how much it increases refusals and reduces answer coverage.

Questions for authors
  • What is the main technical advance over existing controlled language and template generation methods? What capabilities do the verification semantics for control applications provide that existing methods cannot directly offer?

  • Could the authors provide a 2 × 2 ablation on frontier models such as Opus, with intermediate verification enabled or disabled and template rendering enabled or disabled, analogous to Table 4? Please report risk, coverage, adequacy, and useful coverage to separate the contributions of verification and rendering.

2 comments

Thank you for the review. We ran the verification × rendering ablation you asked for on three frontier models. Doing so exposed a problem with Table 4, which we correct below. We also state more precisely what we claim beyond templates.

W1 - Contribution beyond templates and controlled language

Templates and controlled language can implement our interface. We do not claim a new template-generation method. The contribution is to define which statements the recorded optimisation result supports and ensure that the displayed answer contains only accepted statements. We also measure what happens when a model writes the final answer instead.

The paired experiment tests the last point directly. Both displays use the same accepted claims. Across 12,687 pairs, 3,067 model-written answers are flagged by the scorer while their rendered counterparts are not. No pair changes in the opposite direction. This result concerns the final display; it does not independently validate the accepted claims.

The control-specific checks determine when a status statement can be reported. Our "binding" label requires a converged, feasible solve, a constraint at its limit, and a scale-normalised multiplier above the declared threshold. We also require LICQ, which is sufficient for multiplier uniqueness at the fixed recorded solution. This distinguishes a multiplier-qualified status from merely being at a constraint limit.

Of 704 status mentions in the 1,456 test questions, the policy withholds 67. In 55, the regularity check does not establish a unique multiplier. In 44 of these, the constraint is at its limit with a multiplier above threshold, so it would otherwise be labelled binding. In 12, the constraint has a multiplier above threshold but is away from its limit. The policy also applies the check to slack labels. This is conservative, since a strictly inactive constraint has zero multiplier in every KKT vector, and on this benchmark it never withholds a slack label. We will add these examples to the contribution statement and the binding definition (lines 048 and 087).

W2 - Verification × rendering on frontier models

We ran the requested comparison. While doing so, we found that the submitted Table 4 used separately prompted arms and therefore did not isolate the effect of rendering. We will replace it (line 378) and remove the corresponding interpretation (line 392).

For each model, the four one-attempt conditions below use the same stored first generation for each question. When verification is disabled, parsing still normalises the schema, aliases, timestamps and unit spellings, but no verification gate runs. Unchecked rendering uses the proposed values without substituting recorded values. It refuses only when no claim can be parsed.

The three-attempt rows evaluate the deployed retry procedure separately. Coverage and useful coverage use all 1,456 questions. Risk and adequacy condition on answering. Risk is the automatic scorer's flag rate, and 0.000 means zero observed flags.

ModelVerificationAttemptsRenderingCoverageRiskAdequacyUseful coverage
Opus 5off1off1.0000.6200.5080.180
Opus 5off1on0.9340.1960.5460.403
Opus 5on1off0.6110.5670.6420.178
Opus 5on1on0.6110.0000.6420.375
Opus 5on3off0.9250.5340.7130.320
Opus 5on3on0.9250.0000.7130.630
Sonnet 5off1off1.0000.3580.4010.227
Sonnet 5off1on0.7770.1130.4910.400
Sonnet 5on1off0.5830.3830.5780.275
Sonnet 5on1on0.5830.0000.5780.396
Sonnet 5on3off0.7940.3880.6120.359
Sonnet 5on3on0.7940.0000.6120.545
GPT-5.5off1off1.0000.2070.3720.274
GPT-5.5off1on0.7550.1110.4750.367
GPT-5.5on1off0.5810.2020.5420.296
GPT-5.5on1on0.5810.0000.5420.356
GPT-5.5on3off0.8300.1850.6280.424
GPT-5.5on3on0.8300.0000.6280.512

The comparison shows both the benefit and the cost of verification:

  • Verification alone does not consistently lower the flag rate of model-written answers. At one attempt, the changes are −0.053 for Opus 5, +0.025 for Sonnet 5 and −0.005 for GPT-5.5. These rates apply to different answered subsets.
  • Rendering without verification leaves 266, 128 and 122 flagged answers, respectively. In the paired one-attempt comparison, enabling verification refuses every one of these cases; with verification enabled, no flagged answers remain among those displayed (risk 0.000).
  • This comes at a coverage cost. Relative to unchecked rendering, one-attempt verification reduces coverage by 0.323, 0.194 and 0.174, and changes useful coverage by −0.028, −0.004 and −0.011.
  • With three attempts, the coverage differences relative to unchecked rendering are −0.009 [−0.030, +0.014], +0.017 [−0.001, +0.035] and +0.075 [+0.054, +0.095]. Useful coverage increases by 0.227, 0.144 and 0.145. Retry feedback can supply recorded values, so these gains include correction after a failed proposal, not just filtering.

Intervals are 95% trajectory-bootstrap intervals from 2,000 resamples within domain. The prose scorer sometimes attaches a value to the wrong quantity, as discussed in Appendix C.4; these flag rates should not be read as human-adjudicated error rates.

For Llama 3.1 8B, unchecked rendering in the matched comparison reduces risk from 0.473 to 0.282, leaving 360 flagged answers. Thus, the increase from 0.395 to 0.524 in the submitted Table 4 cannot be attributed to rendering alone. This correction does not affect the matched accepted-claims comparison in Table 2. We will replace Table 4 (line 378) and report the coverage costs in Section 5.4 and Appendix C.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.