acceptodds
Under review as a conference paper at ICLR 2027

Check, Don't Trust: Evidence Contracts for Small Language Agents in GPU-Cluster Operations

Abstract

Operators of shared GPU clusters often did not write the scheduler or the jobs they are diagnosing, and a small, locally hosted language model is an attractive interface for their questions. But an ungrounded model can state numbers no metric supports, and an agent allowed to act can shut down a node that is still working. We study ClusterMind, an assistant built on 1–1.5B models in which a single retrieval pass produces one frozen, typed evidence set that is shown verbatim to the model, the operator and a deterministic checker, and in which state-changing actions pass a busy-node precondition re-evaluated at confirmation time. To evaluate it without synthetic metrics we build PAI-Diag from the Alibaba PAI GPU trace: 320 questions over 40 real cluster snapshots with computed gold answers, a 600-item controlled answer set, and trace-driven action scenarios. The prototype’s original grounding check certifies most wrong answers. A claim-level checker removes fabricated numbers but still delivers answers that are true of the evidence and wrong for the question; adding superlative, set and question-target claims raises the accuracy of delivered answers on held-out Llama-3.2-1B outputs from 38.5% to 98.5% at 24.5% coverage, with no model call and about 65 μs per answer. A 7B LLM judge given the same evidence is a weaker gate: at matched coverage its delivered answers are 75.4% correct, and it costs 10.7 s per answer on CPU. On actions, 47.0% of busy machines in the trace show ≤10% GPU utilisation, so a utilisation-based guard would permit shutting down nearly half of them, and 6.3% of idle candidates become busy within 30 minutes. The guarantees are narrow and we report where they fail, most notably when retrieval omits the relevant node

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
small language models, grounding verification, hallucination detection, deterministic verification, LLM agents, agent safety, AIOps, GPU cluster operations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DvmAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ClusterMind, an assistant that uses small local language models to answer GPU-cluster questions and propose operational actions. A frozen telemetry snapshot is shared by the model, operator, and deterministic checker. The checker validates numerical claims, node states, comparisons, and question targets, while disruptive actions require confirmation and a fresh state check. Evaluation on a benchmark derived from the Alibaba PAI trace shows higher accuracy among accepted answers, with reduced coverage. The approach offers useful lessons about verification, but the necessity of generative answering and the generality of its predefined checking rules remain insufficiently demonstrated in the evaluation.

Critical strengths
  1. Clearly demonstrates that individually correct facts can still produce an incorrect answer to the question.

  2. Sharing one evidence snapshot makes verification consistent and auditable, with low checking overhead.

  3. Reports accuracy alongside coverage and openly examines important failure cases.

Critical weaknesses
  1. Establish the model’s additional value over simpler answering methods. The evaluated questions mainly require lookups, comparisons, filtering, and state checks. The checker already implements several operations needed to compute their answers. It is therefore unclear why generating an answer and checking it afterward is preferable to executing a structured query and formatting the result. A model could still provide a natural-language interface by translating questions into validated query parameters. Please compare against both deterministic answering and this language-to-query alternative, measuring accuracy, coverage, latency, and operator effort. Such comparisons would establish whether free-text generation provides a practical benefit.

  2. Evaluate generalization beyond the predefined checking rules. Testing held-out Llama outputs provides useful evidence across models, but the questions still use the same task types, templates, and telemetry format. It does not demonstrate robustness to new metrics, unfamiliar phrasing, or questions combining several operations. Furthermore, qualitative statements outside the recognized vocabulary can remain unchecked even when an answer receives a supported verdict. Please evaluate independently collected operator questions and hold out question types, paraphrases, and evidence formats during checker development. Reporting unsupported claims that pass verification and the effort needed to extend the rules would clarify the approach’s practical scope.

Questions for authors
  1. What benefit does free-text answer generation provide over deterministic query execution, optionally using a model only to interpret the user’s intent?

  2. How accurately does the checker handle unseen questions and evidence formats, and how much manual rule development is required to support them?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DvmA (part 1/2)

Thank you for a precise review. Your two requests define the two new subsections of the revised paper (§6.4 and Appendices I–N). Below we answer each, including results that go against our original framing.

W1 / Q1. What does free-text generation add over deterministic query execution, optionally with a model only to interpret intent?

We ran the comparison you asked for: deterministic answering, a language-to-query answerer, free text, and their combination. The table shows PAI-Diag, 320 questions, seed 0: correct & delivered per 100 / delivered accuracy / delivered errors per 100 / median latency.

AnswererQwen2.5-1.5BLlama-3.2-1B
Deterministic (keyword intent → query)100 / 100 / 0 / ~0 s(model-free)
Free text + v2 (paper)55.6 / 100.0 / 0.0 / 3.3 s18.8 / 98.4 / 0.3 / 4.0 s
Model → validated query, ungated73.8 / 86.4 / 11.6 / 6.1 s86.2 / 86.2 / 13.8 / 5.3 s
Model → query + checker (v3)73.1 / 99.2 / 0.681.9 / 100.0 / 0.0
Hybrid (query if it passes, else checked free text)87.8 / 99.3 / 0.6 / 6.3 s85.3 / 100.0 / 0.0 / 5.4 s

Our conclusions:

  1. For questions inside the schema, query execution is better than free text. The deterministic answerer is perfect on PAI-Diag by construction. We now recommend query-first answering.
  2. Validated query parameters are not enough. Every query here is schema-valid; constrained decoding guarantees that. Yet the model still writes the wrong query for 11.6–13.8% of questions, most often a value lookup where the question asks for an extremum, and those answers are delivered with no signal. Checking the answer against the evidence and the question removes them. So the contract remains useful in the design you propose; it moves from checking prose to checking the rendered query result.
  3. Free text is a fallback, not the main path. The hybrid adds 14 (Qwen) correct answers per 100 over query alone. On the held-out types, the deterministic parser declines 29.7% of questions; this is optimistic, since we wrote both. Model-written queries do not transfer cleanly either. Qwen's answer 73.8 per 100 correctly but deliver 13.8 errors; Llama's answer 29.7 with 58.4 errors. v3 removes these errors but withholds most correct query answers, because it parses only some of the formatter's phrasings. With Qwen2.5-7B, queries do transfer: 87.5 correct per 100 on held-out types, against 71.9 for the rule parser on the same questions. This is the strongest argument for your design, and it argues for checking the typed query rather than its rendering, which we now list as future work. No query language addresses "why" or "what should I do" questions, and these make up 27 of the 84 questions our colleagues wrote, plus 2 predictions. On the 44 answerable colleague questions, the keyword rule parser answers only 61.4% correctly, while Qwen2.5-7B writing queries answers 93.2%.

Operator effort. We use two measures: the share of questions on which the operator must look up the answer themselves (100 − correct & delivered), and how many wrong answers reach them silently. The hybrid behind the checker minimises both among the model-based options.

W2 / Q2. Generalisation beyond the predefined rules; unsupported claims that pass; effort to extend.

Held-out question types. We wrote 10 new types after the paper's checker was frozen:

  • 8 scored: maxima of other metrics, counts, a threshold filter, a pairwise comparison, a cluster total, and a busy-and-failed conjunction (combining operations).
  • 2 open: explain and recommend.

Each type has two paraphrases.

The frozen v2 does not generalise. Its delivered accuracy on the new types is 45.2% (Qwen) and 21.9% (Llama), because counts, totals, thresholds and comparisons are outside its claim language. You anticipated this.

Effort to extend. v3 adds six claim types in 221 lines. It was developed only on Qwen's answers to the new types and frozen (hash recorded) before the held-out data were inspected.

  • On held-out Llama answers, delivered accuracy rises from 21.9% to 77.8%. All six remaining delivered "errors" state both compared values correctly but give no explicit yes/no. On a second held-out model, Qwen2.5-7B, v3 raises delivered accuracy on the new types from 56.6% to 97.1% (coverage 42.8%). By hand, only one of its 4 delivered "errors" is genuine: "3.3 GB … the same as Node-1 … 0.0 GB", because "same" is not a comparative claim.
  • On PAI-Diag, v3 changes coverage by at most 0.9 points, with no new delivered error.

The code was drafted with an AI coding assistant, so we report lines of code and claim types rather than hours as the effort measure.

(continued in part 2/2)

AuthorsOpenReview ↗

Response to Reviewer DvmA (part 2/2)

Evidence formats. We rendered the same records as JSON and as a Markdown table instead of sentences (PAI-Diag, seed 0).

  • Checker. Its delivered errors stay at most 2.8 per 100 by the strict scorer. By hand, at most 0.6 per 100 are genuine: 4 of 16 flagged answers. The other 12 are correct answers in unaccepted forms.
  • The 4 genuine misses show lexicon gaps: "is not currently running any instances", "did not fail recently".
  • Models. Rendering changes the models more than the checker. Llama's ungated accuracy rises from 32.8% to 44.7% with a table, while Qwen's falls from 56.6% to 45.6%.

Independently collected operator questions. Three colleagues who are not authors wrote 84 questions about three snapshots. We froze the gold answers before running any system; the full table is in the general response. Of the 84 questions, 44 are answerable from the table, 11 ask for data it lacks, and 29 are open.

On the answerable questions:

  • v3 keeps free-text delivered accuracy at 88.9% (Qwen), 77.8% (Llama) and 95.7% (7B).
  • The keyword rule parser answers only 61.4% correctly and delivers 20.5 wrong answers per 100.
  • Qwen2.5-7B writing queries answers 93.2% correctly.

Two weaknesses appear that our templates had hidden:

  • Typos switch off question-aware rules. "How many node have…" does not trigger the count rule.
  • The checker cannot recognise questions about missing data. It verifies answers, not questions, so it delivers true but irrelevant or fabricated answers to them, and it withholds correct refusals.

The revised Limitations call for an explicit out-of-scope check on the question.

Unsupported claims that pass verification. We audited every open-ended answer that v3 delivered (73 answers):

  • 2/44 (Qwen) and 9/29 (Llama) contained a false factual sentence from which no claim was extracted. Causes: pronoun subjects ("It has the highest GPU utilization (113.2%)"), comparisons with "the other nodes", number words ("two instances running" for 22), and one negation-scope error.
  • 22/73 contained a judgement that the evidence cannot settle ("relatively high", "nearing the end of its life cycle").
  • On "Should I shut down X?", Qwen never gave an explicit recommendation.

We added a sentence-coverage rule (v3s, 76 lines): every sentence with a factual cue must yield a supported claim of the matching kind. It withholds all 11 answers with false sentences and none of those we labelled fully true, at a cost of ≤1.5 coverage points on PAI-Diag. Because v3s was written after this audit, its held-out test is the colleague set.

Held-out test of v3s (colleague set).

  • Answerable questions. v3s raises free-text delivered accuracy from 88.9% to 100% (Qwen) and from 77.8% to 87.5% (Llama).
  • Open questions. Of the 20 answers v3 delivered, 2 contain an unchecked false sentence. v3s withholds one of them. It misses the other, whose false statement is phrased like a refusal ("The telemetry does not mention any running instances on Node-1"), and it withholds one true answer.
  • Shut-down recommendations. Two delivered recommendations propose shutting down a node that runs 24 instances. The checker cannot judge recommendations, but the action guard would block the shutdown.

Recommendations and judgements remain unchecked by design, and the revised Limitations say this explicitly.

Thank you again. These experiments changed how we present the contribution. ClusterMind is now presented as a verification layer that also gates query-based and agent answers, together with an honest measurement of what extending it costs.

We would be happy to address any further questions or concerns the reviewer may have.

Reviewer_WjooOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors aim to solve the problem of diagnosing issues about shared GPU clusters with limited resources. The authors provide two main contributions. The first is a retrieval system that pairs TF-IDF retrieval, LLM reader, and a rule-based checker. The main novelty here seems to be the rule-based checker, which rejects ungrounded responses. The second contribution is the PAI-Diag benchmark, which is generated programmatically from the PAI trace dataset. The authors evaluated two models, Qwen2.5-1.5B and Llama-3.2-1B, and showed that inclusion of the rule-based checker significantly increases the true positive rate of responses compared to the setup without the rule-based checker.

Critical strengths
  1. The problem is novel. Being able to debug large GPU clusters is important given that failures and underutilization may be costly.
  2. The authors consider the intersection of cloud operations and hallucination detection, which is novel.
  3. The problem formulation in Section 3 provides a useful framework.
Critical weaknesses
  1. The authors discuss tool-using agents in the related works of Section 2, but do no directly compare their ClusterMind method to these baselines in their empirical evaluation. For example, a natural baseline would be Qwen2.5-1.5B and Llama-3.2-1B paired with an agent harness.
  2. The paper demonstrates the claim-level checker significantly improving the accuracy of delivered answers (e.g., 38.5% to 98.5% at 24.5% coverage), but the paper did not make clear what is the overall accuracy when considering both delivered and refrained answers.
Questions for authors
  1. Did the authors consider benchmarking Qwen2.5-1.5B and Llama-3.2-1B paired with an agent harness on PAI-Diag? For example, a common model-agnostic agent harness is Terminus-2 via the Harbor Framework. The related works mention tool-calling agents as a key motivation, so it would seem natural to include these results.
  2. See W2 in "Critical Weaknesses"
  3. The authors consider the resource constrained setting where the LLM must be run on CPU only. I would imagine another setting is running some sort of coding agent (e.g., Claude Code, Codex) with access to LLM via API. Of course, the latter setting would require API access, which is not free. Can the authors clarify the pros and cons of each setting? This would help me better understand why the resource-constrained setting considered is so important to study.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Wjoo (part 1/2)

Thank you for the review and the concrete suggestions. We address the two weaknesses and three questions below. Section and table numbers refer to the revised PDF.

First, a correction that affects how Table 1 could be read. In the submitted PDF, Table 1 showed the rows of the ablation table because of a LaTeX \input error. The numbers quoted in §6.2 were correct, but the table you saw was not. The table is restored in the revision. We apologise for this.

W2 / Q2. Overall accuracy when withheld answers are counted.

Table 1 now reports, per 100 questions: answers that are correct and delivered, delivered errors, correct answers withheld, and decision accuracy (the share of questions on which the gate either delivers a correct answer or withholds a wrong one). For held-out Llama-3.2-1B (3 seeds):

GateCoverageCorrect & deliveredDelivered errorsCorrect withheldDecision acc.
none100.038.561.50.038.5
v0 (prototype)82.136.245.82.351.9
v1s38.925.912.912.674.5
v224.524.20.314.485.3

Seen end to end, the checker does not make the 1B model answer more questions correctly: it answers 24.2 of 100 instead of 38.5. What it does is turn 61.5 silent errors into 0.3, so the operator knows which 24.5 answers to trust and which 75.5 to look up. About two thirds of the 14.4 withheld "correct" answers are abstentions that go on to print values for a node that does not exist; the task scorer accepts them, but they contain a fabricated claim.

At equal coverage, the 7B LLM judge's delivered answers are 75.4% correct, against 98.4% for v2 (Llama seed 0; §6.3).

The new query-first configuration also raises the end-to-end number substantially. The hybrid answers 85.3 of 100 Llama questions correctly, with 0 delivered errors (§6.4).

W1 / Q1. Agent-harness baseline.

We added one (§6.4, Appendix M). We did not use Terminus-2/Harbor. Terminus-2 drives a model through a tmux terminal for Terminal-Bench-style tasks, which assumes shell competence that 1B models lack and would mostly measure command syntax.

Instead we built a model-agnostic ReAct-style tool-calling harness:

  • Tools: list_nodes, get_node, top_nodes, find_nodes, final.
  • Decoding: schema-constrained, so every step parses.
  • Budget: up to 5 steps.
  • Models: the same Qwen2.5-1.5B and Llama-3.2-1B, on the same snapshots.

The checker is applied to the agent's final answer, with E = exactly the records the agent's tool calls returned. This is the natural evidence contract for an agent.

Results on PAI-Diag (320 questions, seed 0; per 100 questions):

Finished in 5 stepsCorrect & deliveredDelivered acc.Delivered errorsMedian latency
Qwen2.5-1.5B agent, ungated56%32.858.323.419.2 s
Qwen2.5-1.5B agent + v330.992.52.5 (1 genuine)19.2 s
Llama-3.2-1B agent, ungated13%10.380.52.528.8 s
Llama-3.2-1B agent + v39.187.91.2 (0 genuine)28.8 s
For reference: Qwen single-pass free text + v255.6100.00.03.3 s
For reference: Qwen hybrid (query → free text) + v387.899.30.66.3 s

We inspected every delivered "error" by hand. "Genuine" means the answer is actually wrong; the others are correct answers that our strict scorer rejects because they lack a number or unit, e.g. "Node-4 did not fail recently" or "Node-5 has 0 GPU memory in use". The one genuine error is "Node-5 did not fail recently" for a node with 82 recent failures. Negative failure statements are outside the claim lexicon, and the revised Limitations now say so.

Interpretation:

  • At 1–1.5B, the agent harness hurts. The models often fail to finish within the step budget: Qwen loops on the same tool call, and Llama rarely emits final. Answering is 6–7× slower than a single pass, and the agent answers fewer questions correctly than either single-pass free text or the query route.
  • Giving the model the full snapshot works better at this scale. The small context of an 8-node snapshot makes this cheap, and a tool loop adds failure points without adding information.
  • The evidence contract carries over to agents unchanged. With E defined as the records the tools returned, v3 cuts the agent's delivered errors from 23.4 to 2.5 per 100 (1 genuine).

(continued in part 2/2)

AuthorsOpenReview ↗

Response to Reviewer Wjoo (part 2/2)

On the held-out question types (first 20 snapshots, 160 scored questions):

  • Qwen agent. It finishes 44 of 160 and answers 16.9 per 100 correctly, against 30.0 for a single pass. Behind v3 it delivers 70.4% correct.
  • Llama agent. It never finishes within 5 steps.
  • All 8 delivered errors of the Qwen agent have one cause: the agent's own evidence is incomplete. In five cases it answered "how many machines are idle?" with "no machines are running anything" after its tools had returned no records. In three it named the maximum after seeing one node.

The checker verified these claims against exactly the evidence the agent gathered, so they passed. This is the same failure we report for the prototype's TF-IDF retrieval (§7). For count, set and extremum claims, an agent's evidence must be complete, or the checker must use the full snapshot. The revised paper states this explicitly.

Q3. Pros and cons of a coding agent (Claude Code, Codex) with API access versus a local CPU model.

This is a fair question, and we added a paragraph to the Introduction and Limitations. In short:

API-based agent: pros

  • Much stronger reasoning and tool use.
  • Can write and run its own queries.
  • No local hardware.

API-based agent: cons

  • Cluster telemetry and job metadata leave the organisation, which many HPC and university clusters' data policies forbid or restrict.
  • Recurring per-query cost.
  • Dependence on network and vendor availability during exactly the incidents in which operators need answers.
  • An agent with shell access to a production cluster is a larger attack surface: prompt injection via job names and logs.

Local small model: pros

  • Data stays on site.
  • Fixed cost; runs on a spare CPU node.

Local small model: cons

  • Much weaker. Our results quantify how much weaker, and why a verification layer is necessary.

The evidence contract applies to both. A large API agent also benefits from a deterministic gate on its claims and a hard precondition on its actions. The precondition does not depend on model quality: in our trace analysis, 47% of busy machines would pass a utilisation-based guard regardless of who proposed the shutdown. We study the CPU-only setting because it is the constrained end of the design space, where verification matters most, not because we think API agents are inappropriate where policy allows them.

We would be happy to address any further questions or concerns the reviewer may have.

Reviewer_SXXFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work tackles the problem of inaccurate results when using a small language model (1-1.5B) to perform cluster management. It solves it using a deterministic checker that verifies the output of the LLM which it extracts using regex rules.

Critical strengths

The setup is interesting, where the cluster could respond to questions in natural language. This (if implemented correctly) could be a nicer interface to things like "squeue" or "kubectl".

Critical weaknesses
  • The LLM seems redundant, since the gold-answer generator already computes all the answers from the typed state.
    • If the scorer can already detect "intent" through keyword matching on the question, why can't it answer the question directly?
    • Even if a response is checked, the process of LLM output could also potentially hallucinate.
  • The checker has a high cost of generalization, and is itself heuristic.
    • If one wants to extend the capabilities of the system, they must first compose a number of rules / templates for the checker.
    • The correctness of the checker is also only justified on the eight evaluated question types. A rule could conceivably bind a number or state word to the wrong metric. This may become more prevalent once enough rules are added.
  • The checker rules are revised after observing model outputs and evaluation results on the same eight question templates. Although the checker is frozen before the Llama runs, the held-out split is by model only, and no result tests the rules on new question types or phrasings.
Questions for authors
  • Qwen2.5-7B with the v2 checker already delivers 89.1% of answers at 100% accuracy on the same CPU (section 6.2). What is the case for the 1-1.5B models beyond latency, and how does coverage change with intermediate model sizes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer SXXF (part 1/2)

Thank you for the review. Your three concerns are, in our view, the right ones, and the new experiments partly confirm them. Section and table numbers refer to the revised PDF. The general response gives the full numbers.

W1. "The LLM seems redundant, since the gold-answer generator already computes all the answers. If the scorer can detect intent by keywords, why can't it answer directly?"

On PAI-Diag, it can, and we now say so. We added a keyword intent parser that maps each question to a typed query and executes it. It answers all 320 PAI-Diag questions correctly. This is by construction: PAI-Diag's gold answers are computed the same way.

We also added an LM-to-query answerer: the same 1–1.5B model writes a schema-constrained JSON query, and code executes and formats it. It answers more questions correctly than free text does (73.8 vs 56.6 per 100 for Qwen; 86.2 vs 32.8 for Llama). For questions inside a fixed schema, query-first answering is the better design, and the revised paper recommends it (§6.4).

What this changes is the role of the contract, not its usefulness: • Queries are also unverified model output. Without a gate, LM-to-query delivered 11.6 (Qwen) and 13.8 (Llama) wrong answers per 100 questions. Most were queries for the wrong thing, such as one node's value when the question asks for the maximum. The same evidence/checker contract removes them: Llama's query answers go from 86.2% to 100.0% delivered accuracy with v3. The best system overall is the hybrid behind the checker: 87.8 (Qwen) and 85.3 (Llama) correct answers per 100, at ≥99.3% delivered accuracy. • Keyword rules stop where the schema stops. On the held-out question types (W3 below), the rule parser declines 29.7% of questions. This figure is optimistic, because we wrote both the parser and the types. It cannot answer "why" or "what should I do" questions at all. The colleague questions are its real test, and it fails it: on 44 answerable questions written by non-authors, it answers 61.4% correctly and delivers 20.5 wrong answers per 100. It answers all 11 questions about missing data with irrelevant facts.

You also note that a checked answer could still contain unchecked content. We agree and quantify it. Among open-ended answers v3 delivered, 2/44 (Qwen) and 9/29 (Llama) contained a false sentence it did not check. A sentence-coverage rule (v3s) withholds all of them. Recommendations themselves remain unchecked (Appendix L).

W2. "The checker has a high cost of generalization and is itself heuristic … a rule could bind a number or state word to the wrong metric."

Agreed. We measured both points. • Cost. Supporting 8 new question types took 221 lines and six new claim types: counts, threshold and conjunction sets, comparisons, totals, and a rule that extremum questions need a superlative claim (Appendix K). • Mis-binding. It happens, and the paper reports it: 2 of v2's 5 false positives on the manually labelled set are metric mis-bindings. While building v3 we found and fixed one more ("Failures are concentrated on Node-2, which had 22.6 GB" was bound to memory). The audit also found a negation-scope error. • Phrasing dependence. v3 withholds correct query answers written in forms it does not parse ("> 100", "(0.0 GB vs 4.1 GB)").

The revised Limitations state all of this. The checker is cheap to run and auditable, but it is not free to extend.

(continued in part 2/2)

AuthorsOpenReview ↗

Response to Reviewer SXXF (part 2/2)

W3. "The checker rules are revised after observing outputs on the same eight templates … no result tests the rules on new question types or phrasings."

This was a real gap; the submission's only held-out axis was the model. We added three:

  1. Held-out question types (8 scored + 2 open, two paraphrases each), written after the paper's checker was frozen.
    • The frozen v2 does not transfer: delivered accuracy is 45.2% (Qwen) and 21.9% (Llama).
    • v3 was developed on Qwen's answers only, then frozen with a recorded hash. On held-out Llama answers it reaches 77.8% delivered accuracy. The remaining six "errors" give both compared values correctly but no explicit yes/no. On a second held-out model, Qwen2.5-7B, v3 raises delivered accuracy on the new types from 56.6% to 97.1% (coverage 42.8%). By hand, only one of its 4 delivered "errors" is genuine: "3.3 GB … the same as Node-1 … 0.0 GB", because "same" is not a comparative claim.
  2. Evidence formats. We rendered the same records as JSON and as a Markdown table instead of sentences (PAI-Diag, seed 0).
  • Checker. Its delivered errors stay at most 2.8 per 100 by the strict scorer. By hand, at most 0.6 per 100 are genuine: 4 of 16 flagged answers. The other 12 are correct answers in unaccepted forms.
  • The 4 genuine misses show lexicon gaps: "is not currently running any instances", "did not fail recently".
  • Models. Rendering changes the models more than the checker. Llama's ungated accuracy rises from 32.8% to 44.7% with a table, while Qwen's falls from 56.6% to 45.6%.
  1. Independently written operator questions (84 questions from 3 colleagues who are not authors; gold frozen before any run). On the 44 answerable questions, v3 delivers free-text answers that are 88.9% (Qwen), 77.8% (Llama) and 95.7% (7B) correct. Your concern about phrasing is confirmed in two ways. A typo ("How many node have…") switches off the count rule. The checker also cannot recognise questions about data the telemetry lacks, so it delivers irrelevant or fabricated answers to them. Details are in the general response.

Q1. "Qwen2.5-7B with v2 already delivers 89.1% at 100% accuracy. What is the case for 1–1.5B models beyond latency, and how does coverage change with intermediate sizes?"

The size curve on the same CPU (Qwen2.5 0.5B / 1.5B / 3B / 7B; coverage, delivered accuracy, latency; v2 and v3):

Qwen2.5 (Q4_K_M)File sizeUngated acc.v2 coverageCorrect & deliveredDelivered acc.Delivered errorsMedian latency
0.5B0.5 GB40.936.635.095.71.61.6 s
1.5B1.1 GB56.655.655.6100.00.03.3 s
3B2.1 GB85.982.880.397.02.55.9 s
7B4.7 GB92.589.189.1100.00.013.0 s

All runs use PAI-Diag with the full snapshot, seed 0, on the same 2-vCPU machine. Rates are per 100 questions.

Coverage rises steeply between 1.5B and 3B. We inspected every delivered "error" at 0.5B and 3B by hand.

  • 0.5B: of 5, 3 are scorer false negatives (correct idle answers) and 2 are genuine ("Node-7 failed in the last hour, with 2 instances" for a node with no failures; the verb form "failed" is outside the state lexicon).
  • 3B: of 8, 7 are scorer false negatives (correct yes/no answers given as a record echo, which our strict scorer does not accept) and 1 lists two nodes' utilisation without naming the maximum.

Results with v3 are within 5 coverage points of v2 at every size (Appendix M).

Query-first answering behind v3 at each size (correct & delivered per 100 / delivered accuracy / latency):

  • 0.5B: 52.2 / 97.1 / 5.0 s
  • 1.5B: 87.8 / 99.3 / 6.3 s
  • 3B: 86.5 / 94.2 / 11.4 s. All of its "errors" are yes/no questions answered by a query that compares a node with itself: true, but with no yes or no.

Our position after these results is more modest than in the submission:

  • If a 7B model fits the latency and memory budget, use it. The checker still matters there: it removes the 7B model's 7.5 delivered errors per 100 (§6.2).
  • The case for 1–1.5B models is the deployment budget, not answer quality: about 1/4 of the latency and 1/4–1/5 of the memory on a 2-vCPU, 7 GB machine.
  • With query-first answering, the small models reach 85–88 correct answers per 100 at ≥99.3% delivered accuracy (above), comparable to the 7B model's free-text 89.1/100.0, at about half its latency (query 6.1 s vs 13.0 s; the hybrid adds the free-text call only when the query fails). The 3B model is a middle point: ungated it is 85.9% correct, but behind v2 it delivers 2.5 apparent errors per 100, of which 2.2 are scorer artefacts.

We have revised the introduction so that it no longer suggests small models are preferable in general. We would be happy to address any further questions or concerns the reviewer may have.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers (part 1/3)

We thank the reviewers for careful and constructive reviews. They converge on three questions: (i) whether generating free text and checking it is preferable to executing a structured query (SXXF, DvmA); (ii) whether the hand-written checker generalises beyond the eight PAI-Diag question types it was developed on (SXXF, DvmA); and (iii) how the system compares with agent harnesses and larger models, and how well it does overall when withheld answers are counted (Wjoo, SXXF). We ran a few new experiments where necessary and revised the paper. We believe the revised manuscript address the reviewers' concerns and improve the rigor, clarity, and reproducibility of our work. We would be happy to continue the discussion if there are any more questions.

Correction. In the submitted PDF, Table 1 was filled with the rows of the ablation table (Appendix Table 8 in the revision) because of a LaTeX \input error. The text of Section 6 quoted the correct numbers, and the correct table is now restored. We apologise for the confusion this caused.

Revised PDF. New or substantially changed text is shown in blue. Contents that were only shortened or minor changes are not marked.

New experiments (all on the same CPU, same snapshots; code and logs in the supplementary material)

E0. End-to-end accuracy with withheld answers counted (Wjoo W2). Table 1 now also reports the share of all questions answered correctly and delivered, and decision accuracy: the share of questions on which the gate either delivers a correct answer or withholds a wrong one. For held-out Llama-3.2-1B, v2 answers 24.2/100 questions correctly, delivers 0.3 errors and withholds 14.4 correct answers. Its decision accuracy is 85.3%, against 38.5% without a gate and 51.9% with the prototype check. At the same coverage, the 7B LLM judge's delivered answers are 75.4% correct, against 98.4% for v2.

E1. Free-text generation vs. language-to-query vs. deterministic answering (SXXF W1, DvmA W1/Q1). We added three answerers that share the evidence contract:

  • Rule parser. A keyword intent parser mapped to a typed query.
  • LM-to-query. The same small model emits a schema-constrained JSON query (op, metric, node, condition), which is executed deterministically and formatted.
  • Hybrid. The query answer is used if it passes the checker; otherwise the free-text answer is used, gated by the checker.

Results on PAI-Diag (320 questions; correct & delivered / delivered accuracy / delivered errors per 100):

AnswererQwen2.5-1.5BLlama-3.2-1B
Free text + v2 (paper)55.6 / 100.0 / 0.018.8 / 98.4 / 0.3
LM-to-query, ungated73.8 / 86.4 / 11.686.2 / 86.2 / 13.8
LM-to-query + v273.1 / 99.2 / 0.683.4 / 88.4 / 10.9
LM-to-query + v3 (frozen, see E2)73.1 / 99.2 / 0.681.9 / 100.0 / 0.0
Hybrid + v387.8 / 99.3 / 0.685.3 / 100.0 / 0.0
Rule parser100 / 100 / 0(model-free)

We agree with the reviewers on the main point. On PAI-Diag's templated questions, deterministic answering is better. The rule parser is perfect by construction, because PAI-Diag's gold answers are computed the same way. Asking the model for a query also beats asking it for prose.

What the paper overstated was tying the evidence contract to free-text generation. The contract does not depend on who produces the answer. A typed query is also an unverified model output: LM-to-query silently delivered 11.6–13.8 wrong answers per 100 questions, mostly by answering a different question (e.g., the value of one node when the question asks for the maximum). The same checker removes them, and the best system overall is the hybrid behind the checker.

On the held-out question types below, neither route transfers cleanly:

  • Rule parser. It answers 70.3% correctly and declines the rest. This is optimistic, because the same authors wrote the parser and the new types.
  • LM-to-query, ungated. Qwen's queries answer 73.8 correctly but deliver 13.8 errors per 100. Llama's answer 29.7 correctly with 58.4 errors.
  • LM-to-query with v3. v3 removes those errors (≤0.3 per 100), but it also withholds most correct query answers, because it checks their rendered sentence and cannot parse several formatter phrasings.

With Qwen2.5-7B, queries answer 87.5 held-out-type questions per 100 correctly, more than the rule parser's 71.9 on the same 160 questions. Behind v3 they answer 48.1, at 100% delivered accuracy. A stronger model writes good queries for unseen types, and the rendering-based gate is the bottleneck. Checking the typed query itself rather than its rendering is the obvious next step; we list it as future work rather than fit another rule post hoc. No rule parser can answer explanation or recommendation questions.

The revised paper presents ClusterMind as a verification layer for any answerer (generation, query or agent), with query-first answering as the recommended configuration.

(continued in part 2/3)

AuthorsOpenReview ↗

General response to all reviewers (part 2/3)

E2. Generalisation beyond the eight types (SXXF W2/W3, DvmA W2/Q2). We wrote 10 new question types after the paper's checker was frozen: 8 scored (argmax memory and running instances, count busy/idle, threshold filter, pairwise comparison, cluster total, a busy-and-failed conjunction) and 2 open (explain, recommend). Each has two paraphrases, on the same 40 snapshots: 320 scored and 80 open questions. Findings:

  1. The paper's checker does not generalise to new types. On the new types, frozen v2 delivers answers that are only 45.2% (Qwen) and 21.9% (Llama) correct. Counts, totals, thresholds and comparisons are outside its claim vocabulary, so wrong answers about them pass. This confirms the reviewers' concern.

  2. The cost of extending the rules. We extended the claim language with counts, threshold and conjunction sets, comparisons, totals, and an "extremum questions need a superlative claim" rule (v3, 221 lines; drafted with an AI coding assistant, so we report lines rather than hours). We developed v3 only on Qwen's answers to the new types and froze it (SHA-256 recorded in rebuttal/V3_FROZEN.txt) before looking at any held-out output. It changes no PAI-Diag result by more than 0.9 points of coverage, with no new delivered error.

    • On the held-out model (Llama, new types), v3 raises delivered accuracy from 21.9% to 77.8% (coverage 8.4%). All six delivered "errors" are true answers to yes/no comparison questions that state both values ("Node-6 uses 0.0 GB, while Node-3 uses 10.2 GB") without saying yes or no. On a second held-out model, Qwen2.5-7B, it goes from 56.6% to 97.1% (coverage 42.8%). Only 1 of its 4 delivered "errors" is genuine ("same" is not a comparative claim).
    • On LM-to-query answers it raises Llama's delivered accuracy from 88.4% to 100.0% (above).
  3. Independently written operator questions. Three colleagues who are not authors wrote 84 questions in their own words about three snapshots shown as a table, typos included ("How many node are idle?"). Before any system was run, we wrote a gold answer for each and froze it (hashes in rebuttal/COLLEAGUE_FROZEN.txt). v3 and v3s had been frozen before the questions were collected.

    Composition. 44 are answerable from the table (37 in our query language; 7 not, e.g. second-highest). 11 ask for data the telemetry lacks (history, owners, causes, GPU count). 29 are open (15 explanations, 12 recommendations, 2 predictions).

    Results per 100 answerable questions (correct & delivered / delivered accuracy / delivered errors):

    AnswererQwen2.5-1.5BLlama-3.2-1BQwen2.5-7B
    Keyword rule parser (no model)61.4 / 75.0 / 20.5
    Free text, ungated31.8 / 31.8 / 68.222.7 / 22.7 / 77.365.9 / 65.9 / 34.1
    Free text + v2 (paper)18.2 / 61.5 / 11.415.9 / 58.3 / 11.459.1 / 86.7 / 9.1
    Free text + v318.2 / 88.9 / 2.315.9 / 77.8 / 4.550.0 / 95.7 / 2.3
    Free text + v3s15.9 / 100.0 / 0.015.9 / 87.5 / 2.345.5 / 95.2 / 2.3
    LM-to-query, ungated63.6 / 68.3 / 29.540.9 / 41.9 / 56.893.2 / 97.6 / 2.3
    LM-to-query + v325.0 / 84.6 / 4.511.4 / 71.4 / 4.531.8 / 93.3 / 2.3
    Hybrid + v336.4 / 84.2 / 6.827.3 / 75.0 / 9.156.8 / 96.2 / 2.3

    Two scoring corrections after the first run are disclosed: a scorer bug (answers to "which nodes are idle" were treated as negated) was fixed, and an answer that lists exactly the qualifying nodes for a count question is counted correct.

    What transfers.

    • The extended claim language keeps free-text delivered accuracy at 78–96% on questions nobody on our side wrote.
    • The keyword rule parser, which fit our templates perfectly, answers only 61.4% correctly and delivers 20.5 wrong answers per 100. Real phrasing breaks keyword rules much more than it breaks the checker.
    • Qwen2.5-7B writing queries is the best single answerer on answerable questions (93.2% correct).

    What does not.

    • Typos switch off question-aware rules. "How many node have…" does not trigger the count rule.
    • Threshold masking can hide a false claim. v3 masks threshold phrases copied from the question, which once hid a false threshold claim about a node.
    • The checker cannot recognise questions about data the telemetry lacks. It verifies answers, not questions. So it delivers true but irrelevant answers ("Node-2 is busy with 47 running instances" to "Who owns the jobs on Node-2?") or fabrications ("Node-1 was idle an hour ago") to 9–55% of these 11 questions, depending on the model. It also withholds correct refusals, because its abstention rule covers only absent nodes.

    The revised Limitations state that a deployment needs an explicit out-of-scope check on the question.

(continued in part 3/3)

AuthorsOpenReview ↗

General response to all reviewers (part 3/3)

  1. Evidence formats. We rendered the same records as JSON and as a Markdown table instead of sentences (PAI-Diag, seed 0).
    • Checker. Its delivered errors stay at most 2.8 per 100 by the strict scorer. By hand, at most 0.6 per 100 are genuine: 4 of 16 flagged answers. The other 12 are correct answers in unaccepted forms.
    • The 4 genuine misses show lexicon gaps: "is not currently running any instances", "did not fail recently".
    • Models. Rendering changes the models more than the checker. Llama's ungated accuracy rises from 32.8% to 44.7% with a table, while Qwen's falls from 56.6% to 45.6%.
  2. Unsupported claims that pass verification (DvmA). We audited every open-ended answer that v3 delivered: 44 from Qwen and 29 from Llama. 2/44 and 9/29 contained a false factual statement from which v3 extracted no claim. The causes were pronoun subjects ("It has the highest GPU utilization"), comparisons with "the other nodes", number words ("two instances"), and one negation-scope error. 22/73 contained a qualitative judgement the evidence cannot settle ("relatively high", "nearing the end of its life cycle").
    • We then added a sentence-coverage rule (v3s, 76 lines): every sentence with a factual cue must yield a claim of the matching kind (after resolving a leading pronoun), otherwise the answer is UNVERIFIABLE. It withholds all 11 such answers and none we labelled fully correct, at ≤1.5 coverage points on PAI-Diag. It is post hoc; its held-out test is the colleague set, collected after it was frozen.
    • Held-out test of v3s (colleague set).
      • Answerable questions. v3s raises free-text delivered accuracy from 88.9% to 100% (Qwen) and from 77.8% to 87.5% (Llama).
      • Open questions. Of the 20 answers v3 delivered, 2 contain an unchecked false sentence. v3s withholds one of them. It misses the other, whose false statement is phrased like a refusal ("The telemetry does not mention any running instances on Node-1"), and it withholds one true answer.
      • Shut-down recommendations. Two delivered answers recommend shutting down a node running 24 instances; the checker cannot judge recommendations, but the action guard would block the shutdown.

E3. Model size (SXXF Q1). Qwen2.5 at 0.5B, 1.5B, 3B and 7B, on PAI-Diag with the same CPU (columns: ungated accuracy / v2 coverage / delivered accuracy / median latency):

  • 0.5B: 40.9 / 36.6 / 95.7 / 1.6 s
  • 1.5B: 56.6 / 55.6 / 100.0 / 3.3 s
  • 3B: 85.9 / 82.8 / 97.0 / 5.9 s
  • 7B: 92.5 / 89.1 / 100.0 / 13.0 s

Coverage rises steeply between 1.5B and 3B. Most delivered "errors" at 0.5B and 3B are scorer false negatives (2 of 5 and 1 of 8 are genuine).

E4. Agent harness (Wjoo W1/Q1). We built a model-agnostic, ReAct-style JSON tool-calling harness for the same small models: tools list_nodes, get_node, top_nodes, find_nodes and final; schema-constrained steps; at most 5 steps. The checker uses E = the records the agent's tools returned.

At 1–1.5B the harness hurts. Qwen finishes only 56% of questions and Llama 13%. They answer 32.8 and 10.3 questions per 100 correctly, against 56.6 and 32.8 for a single pass, at 6–7× the latency. The contract carries over unchanged: v3 cuts the Qwen agent's delivered errors from 23.4 to 2.5 per 100. By hand, 7 of those 8 are correct answers without a number ("did not fail recently").

Changes to the manuscript (summary; section and appendix letters refer to the revised PDF)

  • Table 1 is restored and extended with correct-and-delivered, correct-withheld and decision-accuracy columns (E0).
  • New §6.4, "Is generation needed, and does the checker transfer?", with Table 3. It covers query answering and the hybrid, model size, the agent, the held-out question types, the colleague questions, evidence formats, and the audit.
  • New appendices: I (query language, rule parser, hybrid), J (held-out types), K (v3 and its cost), L (audit, v3s), M (sizes, formats, agent), N (colleague questions); Table 15 lists all configurations.
  • Framing. Abstract, introduction and conclusion present the contract as a verification layer for any answerer, recommend query-first answering inside the schema, and state the measured cost of extending the claim language. The introduction adds the API-agent trade-offs (Wjoo Q3).
  • Related work adds semantic parsing and program execution (e.g., Spider [1] and symbolic languages [2]).

[1] Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., et al. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://aclanthology.org/D18-1425/

[2] Cheng, Z., Xie, T., Shi, P., Li, C., Nadkarni, R., Hu, Y., et al. (2023). Binding language models in symbolic languages. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=lH1PV42cbF

Dear Reviewers,

Thank you once again for your time and the feedback on our submission. We posted individual responses addressing your comments on 13 Nov 2026. Today is the last day of the author-reviewer discussion period. We would be happy to answer any remaining questions on our rebuttal before it closes.

Best regards,

The authors of the submission

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.