- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how clarification can repair incorrect defaults or regress correct ones. It formalizes clarification utility through a repair–regression trade-off and proposes Default-Preserving Clarification (DPC), combining context- and cost-aware question selection with retain-or-revise prompting. Paired code experiments show that DPC achieves 49.0% pass@1, compared with 47.3% for draft-aware rewriting. Policy comparisons match SAGE’s task success with fewer questions, while ranking and transfer results reveal limitations in generalization.
Critical strengths
-
The paper goes beyond question informativeness to examine whether clarification improves end-to-end task outcomes. By separating question selection from answer incorporation, it distinguishes the value of asking from the risk of incorporating a reply, while paired experiments isolate how incorporation affects repair and regression.
-
The evaluation provides a nuanced account of clarification utility, jointly considering task success, interaction frequency, revision cost, and generalization. Its explicit discussion of source overlap, limited replication, partial cost accounting, and mixed transfer results helps clarify both the benefits and the boundaries of the proposed approach.
Critical weaknesses
-
The incorporation gains warrant cautious interpretation. Preserve improves pass@1 from 47.3% to 49.0%, but the gain is concentrated in a small number of source problems. (Table 2.)
-
The selector relies on heuristic scoring. Its uncalibrated model judgments and hand-designed cost terms are not direct estimates of the repair–regression utility. A clearer explanation of how the scoring rule reflects the proposed framework would strengthen the method.
-
The policy comparison has limited scope. DPC matches SAGE-adapted on task success while asking fewer questions, but the shared-question setup does not fully evaluate each method’s native clarification process. (Table 3.)
Questions for authors
Could the unresolved disagreements in the matcher audit affect the reported repair–regression gains, given that reply matching determines whether a clarification answer is incorporated or the default is retained?