- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether test-gated updates to automated policy judges provide meaningful safety assurance. It introduces PolicyPatch, which uses an LLM to propose typed rule updates and accepts them only after schema validation and regression testing, alongside a constructed benchmark of policy amendments. Its central finding is that splitting by textual cases permits underlying evidence states to overlap between gating and evaluation, inflating apparent safety: the previously reported 100% error catch falls to approximately 21% under a state-disjoint protocol. The corrected experiments reveal limited test coverage and errors concentrated on criteria with no available gating states. Validation includes budget sweeps, baseline comparisons, and noise and expressivity stress tests. The paper also formalizes update-specific metrics and outlines conditional certification through exhaustive checking, although the proposed coverage-generation mechanisms remain empirically unvalidated.
Critical strengths
The useful contribution is the concrete demonstration that evidence-state reuse can substantially inflate the measured effectiveness of an update gate. The reversal after correcting the evaluation protocol provides a practical warning for systems evaluated using paraphrases or templated cases. The proposed state-disjoint protocol offers an actionable way to avoid this specific source of misleading assurance. The paper also provides useful reporting practices for policy updates. Separating success on affected cases from harmful changes on protected cases exposes failures that aggregate accuracy can conceal, while reporting unique-state capacity makes testing limitations explicit.
Critical weaknesses
The scientific novelty is limited. The central principles, i.e., that passing a finite test suite does not establish correctness on untested inputs, that semantically overlapping evaluation examples inflate performance, and that updates require both edit success and locality, are established ideas from software testing, evaluation methodology, and model editing. Propositions 1 and 2 largely formalize direct consequences of the testing setup and the assumed correctness of the old rule and amendment scope. Applying these principles to policy-judge updates is relevant, but the paper does not sufficiently establish a distinct conceptual or technical advance. Its empirical findings therefore need to carry most of the contribution. The empirical evidence is too narrow to make that contribution sufficiently compelling. The main held-out splits contain only three and six criterion IDs, and the amendments and cases are constructed within a heavily assisted, closed-world pipeline. The budget saturation result is closely tied to an inventory containing very few unique states, with zero-capacity criteria accounting for much of the observed failure. Some ablations consequently yield outcomes that are structurally expected, while the unmatched model comparisons do not isolate the effect of scale. The external contract pool has no reported outcomes, and the proposed mechanisms for overcoming the coverage ceiling are unvalidated. These limitations leave the broader significance of the findings insufficiently established.
Questions for authors
-
For the zero-capacity criteria, how many states are forced into evaluation by the singleton-stratum constraint? Is zero gating capacity unavoidable under the stated constraints, or could another valid state-disjoint partition provide coverage? This would help distinguish an inherent inventory limitation from an effect of the partitioning design.
-
Table 2 states that aggregation conventions differ between the legacy and corrected results. Can the existing outputs be summarized using a common aggregation convention, with legacy performance separated by whether evaluation states overlap gating states? This would clarify how much of the reported reversal is attributable specifically to state reuse rather than changes in evaluation composition or weighting.