- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DualMind maintains a calibrated set of candidate interpretations for a text-to-SQL request, then filters the set using trusted evidence and clarification answers. It emits SQL only when the surviving set is nonempty and all candidates have equivalent query semantics; otherwise it asks another question or abstains. The paper presents an error bound for adaptive clarification, a condition for distinguishing semantic classes, and a questioning policy under a finite budget. Evaluation uses a small Transformer on a synthetic task with a fixed database schema, 16 candidate interpretations and simulated truthful answers.
Critical strengths
-
The paper is careful about the scope of its guarantees. Keeping one initial calibrated set and filtering it with evidence that preserves the true interpretation gives a clear correctness argument for adaptive clarification. The discussion of noisy evidence, additional pruning and the different error denominators helps avoid misinterpreting the result.
-
Semantic agreement is a reasonable stopping condition. Different groundings can produce equivalent SQL, so requiring a unique interpretation can lead to unnecessary questions. The restricted SQL fragment provides a concrete example, and Appendix B proves the signature's exactness by constructing distinguishing database instances. Execution checks are clearly presented as implementation tests.
Critical weaknesses
-
The technical advances relative to prior work remain limited. The adaptive error bound largely follows from the property that the true interpretation stays within the candidate set, combined with a union bound. The finite‑budget questioning policy builds on Bellman recursion. Prior work such as KnowNo already combines conformal uncertainty with help‑seeking mechanisms, and equivalence‑class active learning has studied question selection for discriminating different classes. While the authors acknowledge these connections, they have not fully articulated what new key problems are solved by porting these ideas into the SQL setting. The contribution would be more compelling if the paper explicitly identified novel guarantees or decision behaviours that cannot be achieved by existing combinations of prior techniques.
-
Existing experiments do not fully resolve the above concerns about contribution. The model only predicts one out of sixteen plan tokens under a fixed schema, and external evidence together with clarification replies directly supply ground‑truth attributes. This setup is suitable for verifying whether the framework behaves as intended, yet its value for more realistic grounding problems remains unclear. The reported 96.5 % coverage figure also benefits from injected ground‑truth information. More direct comparisons should give interactive unique‑interpretation baselines and evidence‑filtering baselines access to identical information and query budgets. Across questioning‑policy comparisons, performance gaps are minimal: under a budget of two answers, optimal, edge‑based, entropy‑driven and random policies achieve 96.5 %, 96.5 %, 96.4 % and 96.4 % coverage respectively (Appendix D, Table 2).
Questions for authors
- Given a fixed candidate set, truth-preserving evidence and a known semantic equivalence relation, which result provides a substantial advance over existing conformal help-seeking and equivalence-class identification methods? Can you give a concrete example of what their combination lacks and what DualMind adds?
- Are there interactive unique-grounding or evidence-filtered baselines with the same evidence, available questions and answer budget? At comparable error rates among answered queries, does DualMind retain a clear advantage in coverage or question cost?
- Have any experiments used candidates that are not fully enumerated in advance, or evidence constraints obtained from natural-language input? If not, which result within the current finite setting establishes value beyond a correctly implemented controlled example?