- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ClusterMind, an assistant that uses small local language models to answer GPU-cluster questions and propose operational actions. A frozen telemetry snapshot is shared by the model, operator, and deterministic checker. The checker validates numerical claims, node states, comparisons, and question targets, while disruptive actions require confirmation and a fresh state check. Evaluation on a benchmark derived from the Alibaba PAI trace shows higher accuracy among accepted answers, with reduced coverage. The approach offers useful lessons about verification, but the necessity of generative answering and the generality of its predefined checking rules remain insufficiently demonstrated in the evaluation.
Critical strengths
-
Clearly demonstrates that individually correct facts can still produce an incorrect answer to the question.
-
Sharing one evidence snapshot makes verification consistent and auditable, with low checking overhead.
-
Reports accuracy alongside coverage and openly examines important failure cases.
Critical weaknesses
-
Establish the model’s additional value over simpler answering methods. The evaluated questions mainly require lookups, comparisons, filtering, and state checks. The checker already implements several operations needed to compute their answers. It is therefore unclear why generating an answer and checking it afterward is preferable to executing a structured query and formatting the result. A model could still provide a natural-language interface by translating questions into validated query parameters. Please compare against both deterministic answering and this language-to-query alternative, measuring accuracy, coverage, latency, and operator effort. Such comparisons would establish whether free-text generation provides a practical benefit.
-
Evaluate generalization beyond the predefined checking rules. Testing held-out Llama outputs provides useful evidence across models, but the questions still use the same task types, templates, and telemetry format. It does not demonstrate robustness to new metrics, unfamiliar phrasing, or questions combining several operations. Furthermore, qualitative statements outside the recognized vocabulary can remain unchecked even when an answer receives a supported verdict. Please evaluate independently collected operator questions and hold out question types, paraphrases, and evidence formats during checker development. Reporting unsupported claims that pass verification and the effort needed to extend the rules would clarify the approach’s practical scope.
Questions for authors
-
What benefit does free-text answer generation provide over deterministic query execution, optionally using a model only to interpret the user’s intent?
-
How accurately does the checker handle unseen questions and evidence formats, and how much manual rule development is required to support them?
2 comments
Response to Reviewer DvmA (part 2/2)
Evidence formats. We rendered the same records as JSON and as a Markdown table instead of sentences (PAI-Diag, seed 0).
- Checker. Its delivered errors stay at most 2.8 per 100 by the strict scorer. By hand, at most 0.6 per 100 are genuine: 4 of 16 flagged answers. The other 12 are correct answers in unaccepted forms.
- The 4 genuine misses show lexicon gaps: "is not currently running any instances", "did not fail recently".
- Models. Rendering changes the models more than the checker. Llama's ungated accuracy rises from 32.8% to 44.7% with a table, while Qwen's falls from 56.6% to 45.6%.
Independently collected operator questions. Three colleagues who are not authors wrote 84 questions about three snapshots. We froze the gold answers before running any system; the full table is in the general response. Of the 84 questions, 44 are answerable from the table, 11 ask for data it lacks, and 29 are open.
On the answerable questions:
- v3 keeps free-text delivered accuracy at 88.9% (Qwen), 77.8% (Llama) and 95.7% (7B).
- The keyword rule parser answers only 61.4% correctly and delivers 20.5 wrong answers per 100.
- Qwen2.5-7B writing queries answers 93.2% correctly.
Two weaknesses appear that our templates had hidden:
- Typos switch off question-aware rules. "How many node have…" does not trigger the count rule.
- The checker cannot recognise questions about missing data. It verifies answers, not questions, so it delivers true but irrelevant or fabricated answers to them, and it withholds correct refusals.
The revised Limitations call for an explicit out-of-scope check on the question.
Unsupported claims that pass verification. We audited every open-ended answer that v3 delivered (73 answers):
- 2/44 (Qwen) and 9/29 (Llama) contained a false factual sentence from which no claim was extracted. Causes: pronoun subjects ("It has the highest GPU utilization (113.2%)"), comparisons with "the other nodes", number words ("two instances running" for 22), and one negation-scope error.
- 22/73 contained a judgement that the evidence cannot settle ("relatively high", "nearing the end of its life cycle").
- On "Should I shut down X?", Qwen never gave an explicit recommendation.
We added a sentence-coverage rule (v3s, 76 lines): every sentence with a factual cue must yield a supported claim of the matching kind. It withholds all 11 answers with false sentences and none of those we labelled fully true, at a cost of ≤1.5 coverage points on PAI-Diag. Because v3s was written after this audit, its held-out test is the colleague set.
Held-out test of v3s (colleague set).
- Answerable questions. v3s raises free-text delivered accuracy from 88.9% to 100% (Qwen) and from 77.8% to 87.5% (Llama).
- Open questions. Of the 20 answers v3 delivered, 2 contain an unchecked false sentence. v3s withholds one of them. It misses the other, whose false statement is phrased like a refusal ("The telemetry does not mention any running instances on Node-1"), and it withholds one true answer.
- Shut-down recommendations. Two delivered recommendations propose shutting down a node that runs 24 instances. The checker cannot judge recommendations, but the action guard would block the shutdown.
Recommendations and judgements remain unchecked by design, and the revised Limitations say this explicitly.
Thank you again. These experiments changed how we present the contribution. ClusterMind is now presented as a verification layer that also gates query-based and agent answers, together with an honest measurement of what extending it costs.
We would be happy to address any further questions or concerns the reviewer may have.
Response to Reviewer DvmA (part 1/2)
Thank you for a precise review. Your two requests define the two new subsections of the revised paper (§6.4 and Appendices I–N). Below we answer each, including results that go against our original framing.
W1 / Q1. What does free-text generation add over deterministic query execution, optionally with a model only to interpret intent?
We ran the comparison you asked for: deterministic answering, a language-to-query answerer, free text, and their combination. The table shows PAI-Diag, 320 questions, seed 0: correct & delivered per 100 / delivered accuracy / delivered errors per 100 / median latency.
Our conclusions:
Operator effort. We use two measures: the share of questions on which the operator must look up the answer themselves (100 − correct & delivered), and how many wrong answers reach them silently. The hybrid behind the checker minimises both among the model-based options.
W2 / Q2. Generalisation beyond the predefined rules; unsupported claims that pass; effort to extend.
Held-out question types. We wrote 10 new types after the paper's checker was frozen:
Each type has two paraphrases.
The frozen v2 does not generalise. Its delivered accuracy on the new types is 45.2% (Qwen) and 21.9% (Llama), because counts, totals, thresholds and comparisons are outside its claim language. You anticipated this.
Effort to extend. v3 adds six claim types in 221 lines. It was developed only on Qwen's answers to the new types and frozen (hash recorded) before the held-out data were inspected.
The code was drafted with an AI coding assistant, so we report lines of code and claim types rather than hours as the effort measure.
(continued in part 2/2)