- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces CedarInstruct, a dataset that translates between natural language-specified access control policies into ones written in Cedar, which was validated with boundary plans generated by AutoCedar. It demonstrates that supervised finetuning (SFT) over this dataset improves a small open-source model's capability on this task, while RL with an existing method, Off-context GRPO, drives some further improvement.
Critical strengths
- The problem this paper studies is of practical value.
Critical weaknesses
- The paper lacks clarity in some crucial details central to its contribution. It's unclear what domains were selected and how, how the data is split between the various splits in the experiments, how it was quality-controlled, and whether any human inspection was ever performed. It is also unclear to me what policy correctness means on L115, and how it differs from the floor and ceiling definitions seen in the previous paragraph.
- Aside from the data, most of the experiments are applying existing techniques to this problem without any modification. Much of the analysis also focus on contrasting the behavior of SFT, vanilla GRPO and OC-GRPO (not contributions of this work) in properties that are not mainly pertaining to those of the proposed task.
- Given the small evaluation dataset size, the gains obtained are very moderate -- it would be useful to validate whether they are statistically significant in the first place.
- Overall, the paper is very difficult to follow. The motivating example is not very clear, and a large part of it is in the appendix.
Questions for authors
- How is "correctness" different from floor=true and ceiling=true?
3 comments
Thank you for the clarification. I can acknowledge that W1 is largely addressed by Appendix C.5, though it does seem like a missed opportunity for generalization evaluation that none of the domains are held out for dev and test splits.
W2: I think the proposed approach makes sense, but is nevertheless very specific to a particular problem.
W3: thank you for adding this, this is helpful!
W4: I don't seem to see the updated revision.
Regardless, I think my assessment of the paper's clarity remains. However, I'll raise my score to 2 thanks to the authors' clarification.
Thank you for following up. The revised PDF is the current version on this submission page. The changes that address your concerns are:
Statistical significance (W3). The paired tests are now reported in Appendix F.3 (p. 29) and Table 12 (p. 32).
Motivating example (W4). The running example is revised in the Introduction (p. 1, lines 42–49), and Figure 1 (p. 4) uses the same requirement. Appendix C.7 (p. 26) shows the incorrect and the corrected policy in Cedar and explains why the verifier rejects the former.
Thank you for the review and for the suggestions on presentation.
W1. Section 3 gives the splits (4,386 SFT / 1,039 RL / 375 test) and states that no test requirement, schema, policy, or plan appears in training. Assignment uses a fixed seed, and the test split covers all areas and domains (Appendix C.5). As we stated in the main paper, some details are given in the corresponding Appendix due to page limitations. This is not uncommon for ML papers. We again summarize them here: scenarios are sampled from 44 hand-written domain skeletons covering 19 application areas. Each scenario is assigned to exactly one split with a fixed seed (4,386 SFT / 1,039 RL / 375 test), and the test split covers all areas and domains. Every scenario must pass an executable filter: all policies validate against the schema, and the target policy satisfies every check under cedar symcc. For the faithfulness review, the results are in our reply to Reviewer m34K (Q4).
W2. The off-context update follows OC-GRPO, as Section 4.2 states. The guidance does not: it is built from formal verification of the model's own failures, pooled across candidates, and explored one check per context, with no reference solution. Table 9 shows each choice matters (46.93% vs. 42.13% without pooling). See point 1 of the general response. On task-specific analysis, Section 6.1 separates validity from authorization reasoning (untrained Qwen3.8-27B is 36.1% correct among valid policies, comparable to both frontier models), Table 10 studies repair from symbolic counterexamples, and Figure 5 separates over-permissive from over-restrictive failures.
W3. Please see point 2 of the general response.
W4. We have revised the motivating example to make it clearer. Because of the page limit, its full details remain in the appendix.
Q. Each scenario has a verification plan with 7.65 checks on average. A check is a ceiling (the policy must not over-permit), a floor (it must not over-restrict), or a liveness check (at least one request is permitted), and each has its own reference policy. As defined in Section 2, a policy is correct only if it parses, validates against the schema, and passes every check in the plan. This is much stricter than passing individual checks: the 9B SFT model passes 78% of checks but only 43% of full plans.