- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes NeuroFORM to repair generated legal formulas that omit conditions or exceptions. At inference time, pretrained models generate and repair Z3 constraints using legal clauses, verified annotations, and scenario feedback, with a separate model checking consistency with the source. The proposed benchmark covers 238 provisions from four privacy laws, with separate repair and evaluation scenarios. The 7B and 32B configurations reach 75.07% and 78.43% held-out accuracy, respectively, but the main comparisons use unequal computation budgets.
Critical strengths
- The paper's use of solver feedback to improve legal formulas is well motivated, especially for missing conditions and exceptions.
- The proposed benchmark covers 238 provisions across four privacy laws and includes exception and boundary cases.
- The ablations are informative in showing that removing the combined structural warning and repair lowers accuracy, and the separate model check improves accuracy.
Critical weaknesses
- The main comparisons use unequal inference budgets (Section 4.1). It is unclear how much of the improvement comes from the proposed design versus extra computation.
- In the matched study, diagnostic feedback improves accuracy by 3.78% and 1.40%, but both 95% confidence intervals include zero (Table 9). Its added value over generic feedback remains unclear.
- Existing legal systems also repair constraints after solver failures (Section 5). B7/B9 often fail to execute, but the manuscript gives little detail about the failures or which original components these adaptations preserve.
Questions for authors
- Could the authors compare NeuroFORM with CoT-Retry and generic solver-guided repair using identical inputs and repair labels, the same generator, and comparable total computation, including auxiliary model calls?
- Does Table 9 retain the full pipeline, including structural repair, the semantic gate, candidate search, and repair budgets, with only the feedback changed?
- Why do B7/B9 often fail to execute, and which original components do these adaptations preserve?
- Does B9 implement Hsia et al.'s constraint-refinement procedure? If not, what prevents a comparison with that procedure?