- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies whether the benefit of training on a synthetic source can be bounded before running the training. It combines a bound on a finite learner’s policy change with target-transition constraints and the probability of completing training within a budget. The authors derive source-selection thresholds and construct examples where rankings change with the budget or cannot be determined from the supplied information. Validation consists of proofs and exact finite-state calculations.
Critical strengths
- The one strength of EnvAudit that I want to point out is that it compares both policies in the same unknown target environment. This preserves information that is lost when their utilities are bounded separately. The example where this changes an unresolved comparison into a definite ranking makes the benefit clear.
- The counterexample is also meaningful for understanding the limitations of the audit. Two targets satisfy the same supplied constraints but give opposite rankings. This helps distinguish a loose certificate from a situation where the available information cannot determine the answer.
Critical weaknesses
-
My main concern is how restrictive the source-to-target construction is. The learner has one scalar parameter, and the target assumes a uniformly helpful action, a constant positive policy shift, and a reward proportional to progress. These assumptions already determine the direction of improvement, while the policy interface explicitly connects source training to that helpful change. The authors acknowledge this limitation, but it is still hard to understand how the audit would apply when the effect of synthetic training is actually unknown.
-
The contribution over stronger verification alternatives needs clearer support. The main comparison uses independent intervals, which discard the shared environmental information. However, the paper builds on established relational verification such as eRHL, and Section 6 gives an example where parameter-preserving evaluation obtains the same gain. I would want a clearer explanation of when the proposed certificate provides better bounds or lower computation than these alternatives.
Reference: Avanzini, M., Barthe, G., Davoli, D., & Grégoire, B. (2024). A quantitative probabilistic relational Hoare logic. arXiv:2407.17127.
Questions for authors
- Can the authors identify a concrete class of independently specified environments where the target constraints and source-to-target policy interface can be justified? What information would be needed to check them?
- Beyond independent interval subtraction, when does the proposed certificate improve on parameter-preserving verification in tightness or computational cost?
- Section 6 notes that the reference policy can be deployed directly in the scalar example. What advantage remains for auditing and training once this alternative, and the cost of the audit itself, are included?