- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RisCoSet, a method for uncertainty quantification in LLM-based code generation. It constructs prediction sets as partial programs by selectively removing uncertain AST nodes. Unlike previous PAC-based approaches, RisCoSet uses the Learn Then Test (LTT) framework to relax monotonicity constraints and account for multiple valid solutions. The authors also introduce a selective execution strategy to reduce calibration costs. Experiments across three LLMs and five benchmarks show improved code retention while maintaining the target empirical coverage.
Critical strengths
-
Well-motivated approach: The paper addresses an important problem in code generation. Combining AST-based prediction sets with LTT is an interesting way to handle multiple valid programs without imposing strict monotonicity constraints.
-
Promising experimental results: The method consistently removes fewer AST nodes than the baselines while maintaining comparable coverage. The experiments across different models and benchmarks, along with the sampling analysis, support the usefulness of the approach.
Critical weaknesses
-
Concerns about the theoretical guarantees: The selective execution analysis appears to have some gaps. The confidence bound in Eq. (10) is derived for a fixed threshold, but Algorithm 3 selects the threshold adaptively. Additionally, the proof of Proposition 3.4 relies on an assumption about false-positive labels that is not clearly justified. It is also unclear how the global labeling error bound translates into the per-task guarantee used in the proof.
-
Limited practical evaluation: The experiments show improved AST-node retention, but this does not necessarily translate into more useful code. A detailed runtime comparison would also be helpful, particularly given the reliance on Z3 optimization. Additionally, the selective execution results in Table 1 use relaxed coverage targets, making the trade-off between code retention and statistical guarantees important to clarify.
Questions for authors
-
How is the confidence bound in Eq. (10) justified for the adaptively selected threshold in Algorithm 3?
-
Can the authors clarify the false-positive assumption and the transition from the global labeling error rate to the per-task bound in Proposition 3.4?
-
What justifies the combined confidence level of (1−γ)(1−δ) in Proposition 3.4?
-
Could the authors provide an end-to-end runtime comparison, including the cost of calibration and Z3 optimization?