- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies factual errors in language interfaces for querying recorded optimisation-based controller runs. It argues that verifying structured claims does not ensure that the accompanying model-written prose remains faithful to those claims. The proposed interface combines typed claims, deterministic record-based verification, and a closed renderer whose output can be decoded back into the accepted claims.
Experiments across three control domains compare model-written and rendered answers while holding accepted claims fixed, and evaluate the complete interface against ordinary tool calling. Rendering eliminates all observed scorer-flagged errors in the paired comparison, while the complete system exhibits domain-dependent trade-offs between reliability and useful coverage.
Critical strengths
- The paired experiment isolates the effect of the final display stage.
Holding accepted claims, verification outcomes, and retry behaviour fixed while comparing model-written and rendered text provides direct evidence that verified structured claims do not necessarily imply faithful final prose. This design makes the effect of the display mechanism easier to interpret than a comparison between two complete systems.
- The scope of the system’s guarantee is explicitly defined.
The paper distinguishes consistency with a recorded solve from controller correctness and causal explanation, and checks the implementation through an independent verifier and renderer–decoder consistency tests. This allows readers to understand what the system guarantees and which questions remain outside that guarantee.
Critical weaknesses
- The incremental contribution and necessity of the proposed machinery are not sufficiently established or clearly presented.
Existing work, including PCN, SymGen, and Think Fast, Talk Smart, already investigates binding displayed content to structured evidence and separating deterministic computation from language generation. This paper further restricts the entire answer to a closed grammar, but does not sufficiently establish what additional benefits this restriction and the control-specific verification machinery provide over a simpler deterministic question-answering pipeline.
This difficulty is also reflected in the figures and organisation of the main text. Figure 1 primarily presents an error example and aggregate paired results, while Figure 2 summarises the central verification mechanism as “five checks.” Neither figure explains concretely how the checks process a claim, when they cause refusal, or which components constitute the main technical contribution. Complete examples and important analyses are dispersed across the appendices. The main text lacks a worked example connecting the input, recorded evidence, verification decision, and final display, as well as a clear visual account of the trade-off between risk, coverage, and answer completeness. These presentation choices make the contribution and its necessity harder to assess.
A particularly relevant missing comparison is a non-oracle pipeline in which the language model translates the question into a structured query, code directly retrieves or computes the requested results from the record, and fixed templates produce the answer. This avoids asking the model to propose numerical values that subsequently require verification. The symbolic renderer supplied with gold claims is an upper bound and does not substitute for this baseline. Consequently, the current experiments do not sufficiently distinguish benefits attributable to the proposed verification machinery from those obtained by adopting deterministic, restricted output.
- The evaluation does not sufficiently distinguish more constrained output from more reliable and useful answers.
The main results rely on errors flagged by automatic scorers. In the small human audit, the prose scorer has precision 0.409 and recall 0.562, while rendered text is assessed through a structural scorer. The human audit and calibrated-judge analyses provide useful supplementary evidence, but do not fully resolve uncertainty about the magnitude of the improvement. Conformance to a predefined grammar and the absence of automatic flags remain different evaluation targets from factual correctness and completeness under an independent semantic assessment.
The matched-claims experiment controls answer availability, so its result cannot simply be attributed to increased refusal. However, refusal and information retention remain central to the comparison between the complete system and ordinary tool calling. On the primary model, pooled useful coverage improves by only 2.1 percentage points, with a 95% confidence interval that includes zero. The confirmatory experiment likewise does not establish a significant overall improvement, while useful coverage decreases in HVAC. The paper therefore demonstrates how to constrain displayed content more convincingly than it demonstrates a robust improvement in overall question-answering utility while preserving necessary information. This limits the demonstrated practical significance of the method.
Questions for authors
- Clarification and key evidence concerning the technical contribution:
Compared with a non-oracle pipeline consisting of natural-language-to-structured-query translation, direct record retrieval or computation, and fixed-template rendering, which failure modes are specifically addressed by the additional mechanisms proposed here? Please identify the key technical differences and provide a comparison that isolates their contribution under the same language model, accessible records, and claim semantics. This comparison would directly affect my assessment of the methodological novelty and whether the additional complexity is justified.
- Key evidence concerning practical utility:
The paired display experiment establishes that rendering reduces scorer-flagged errors when accepted claims are fixed. However, the complete system’s overall useful-coverage improvement over ordinary tool calling remains statistically inconclusive. Could the two complete systems be compared using a common semantic evaluation criterion that is independent of the output grammar, assessing factual errors, retention of information required by the question, and unnecessary refusals on answerable questions? The existing human audit and calibration analyses are useful, but more direct evidence is needed to establish whether fewer errors are accompanied by preservation of necessary information and ultimately a higher proportion of correct, complete answers. This would materially affect my assessment of the paper’s practical value.
2 comments
W4 - Common semantic evaluation
We judged all systems with one rubric and one judge, Qwen3-32B. The judge sees the question, the record and the displayed answer, but not the benchmark labels. It rates meaning rather than our grammar and identifies the required facts itself. Agreement with human labels is modest (κ = 0.385 for factual errors, on items that also informed judge selection), so we treat these results as exploratory.
QER here is the full system from W1(a). Differences are against ordinary tool calling, on resolved pairs (both judgments parseable and certain), macro-averaged over domains.
| Difference from ordinary tool calling | Resolved / all pairs | False-statement rate | Required-fact retention | Correct-complete | Correct-complete bounds |
|---|---|---|---|---|---|
| Tool calling + verified display | 1,330 / 1,456 | −0.186 [−0.213, −0.158] | −0.023 [−0.042, −0.006] | +0.048 [+0.023, +0.074] | [−0.007, +0.086] |
| Direct-proposal verified display | 1,330 / 1,456 | −0.169 [−0.201, −0.136] | −0.079 [−0.099, −0.060] | +0.008 [−0.018, +0.034] | [−0.037, +0.054] |
| QER | 1,322 / 1,456 | −0.180 [−0.208, −0.152] | +0.039 [+0.017, +0.058] | +0.143 [+0.115, +0.172] | [+0.096, +0.190] |
A correct-complete answer states every required fact correctly, says nothing false or unsupported, and is not a refusal, so correct refusals get no credit. The last column sets unresolved judgments to their worst and best values; it is not a confidence interval.
The original pipelines cut false statements but lose some required facts, and their correct-complete gains are small. Part of the drop comes from refusals: on answerable questions they refuse 261 and 136 times, against none for tool calling. QER lowers false statements, keeps more required facts and raises correct-complete by 0.143. Restricting to pairs where the judge's answerability decisions agree gives +0.082, +0.057 and +0.165. With the same rubric, Gemma 2 27B found only one of seven factual errors (κ = 0.075). This is well below the κ = 0.524 it reached with the paper's original prompt (line 445).
We will add these results to Table 3 (line 339) and Section 5.3 (lines 359–368), and the calibration to Section 5.6 (line 445). They will be labelled exploratory and kept separate from benchmark useful coverage.
Thank you for the two concrete suggestions; both made the evaluation stronger. We ran the query–execute–render (QER) baseline and a common semantic evaluation. Intervals are 95% trajectory-bootstrap intervals (2,000 resamples within domain).
W1 - Non-oracle query–execute–render baseline
In QER, Llama 3.1 8B writes queries without values. Code runs them against the record, applies the status policy and renders the result.
(a) Full system: On the 1,456 test questions, QER's useful coverage is 0.776, against 0.620 for direct proposal (+0.156 [0.125, 0.185]). It is 0.171 [0.139, 0.202] above ordinary tool calling and higher on all five other open models (+0.185 to +0.339). Part of the gain is expressivity: QER answers 136 of the 344 relational questions usefully, and direct proposal answers none. QER has three scorer flags (risk 0.002); replay inspection classified all three as scorer errors. The two systems also differ in inputs, claim language, renderer version and attempts (one against up to three).
(b) Controlled comparison: We fixed the protocol before generation. Both arms received the same question, record snapshot and entity list (not the gold claims). They also used the same non-relational claim types, renderer, single attempt and structured-output prompt. On the 1,112 questions that need no relational claim, useful coverage is 0.902 for QER and 0.493 for direct proposal (+0.410 [0.376, 0.444]). Over all questions the difference is +0.314 [0.289, 0.340]. This compares two ways of obtaining facts, not query syntax alone.
(c) Display within QER: With QER's facts held fixed, closed rendering raises the judge-assessed correct-complete rate by 0.188 [0.164, 0.213] over model-written prose (exploratory, see W4).
So on this benchmark, retrieving facts by code works better than having the model propose values. This improves the acquisition stage but does not remove the display failure, which is the paper's main result. The status check is still needed for derived labels: it withholds 67 of 704 status mentions, although we have not ablated it within QER. We will present QER with the status checks and closed renderer as the recommended configuration (line 339). We will add the three comparisons to Table 3 (line 339) and Section 5.2 (line 314).
W2 - PCN, SymGen and Think Fast, Talk Smart
We test adaptations of these display policies, but not in the original systems. A proof-carrying-numbers-style policy hides unverified numbers but keeps the prose. Its scorer risk therefore stays at 0.095, because false status, comparison or action statements contain no number to hide. For our SymGen-style adaptation, the span detector flags unsupported content in 110 of 429 assessed answers (0.256). Our display test within QER (W1, part c) holds code-selected facts fixed and compares closed rendering with model-written prose. This tests the choice made in Think Fast, Talk Smart, letting a model phrase code-selected facts, without reproducing that system. We will make this explicit in Section 2 (lines 138–145).
W3 - Figures
Figure 2 (line 120) will become a worked example showing the question, record fields, checks, retry and display. It will include a rejected value and a status question that ends in refusal. A new figure in Section 5.3 (line 337) will show coverage, risk and correct-complete answers, with benchmark and judge results labelled separately.