acceptodds
Under review as a conference paper at ICLR 2027

RAISE: Reinforcing Access Control Policy Synthesis in LLMs via Symbolic Evaluation

Abstract

Translating natural-language access-control requirements into policies requires careful reasoning about permissions, constraints, and exceptions, and even frontier LLMs often produce policies that violate the intended authorization semantics. We construct CedarInstruct, to our knowledge the first dataset that supports both training and semantic evaluation for formally verifiable Cedar policy synthesis. It contains 5,800 scenarios across 44 domains and 1,408 representing a single synthetic organization, each with a verified target policy and an executable verification plan. On this data we introduce RAISE, which trains policy synthesizers from formal verification in two stages, verified supervised fine-tuning (SFT) followed by a reinforcement learning (RL) stage that learns from verifier signal. We find that SFT succeeds largely by letting models express authorization logic they already have, since untrained models rarely write valid Cedar but often reason correctly when they do. After SFT, how the verifier's information is used matters more than how much of it is used. Of six RL instantiations that consume progressively richer verifier signal, only RAISE-OC improves meaningfully on SFT; it turns failed checks and symbolic counterexamples into guided exploration and learns from the result with off-context GRPO. With about 5.4K verified scenarios and LoRA fine-tuning, RAISE-OC trains Qwen3.5-9B to surpass zero-shot GPT-6 Astra and Claude Opus 5 by 13.33 and 16.26 percentage points in semantic success on held-out scenarios, and training transfers to the independently constructed CedarBench. The implementation is available at https://anonymous.4open.science/r/raise-424A/.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Access Control, Formal Verification, Reinforcement Learning with Verifiable Rewards, Feedback-Guided Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D5tCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CedarInstruct, a dataset that translates between natural language-specified access control policies into ones written in Cedar, which was validated with boundary plans generated by AutoCedar. It demonstrates that supervised finetuning (SFT) over this dataset improves a small open-source model's capability on this task, while RL with an existing method, Off-context GRPO, drives some further improvement.

Critical strengths
  • The problem this paper studies is of practical value.
Critical weaknesses
  • The paper lacks clarity in some crucial details central to its contribution. It's unclear what domains were selected and how, how the data is split between the various splits in the experiments, how it was quality-controlled, and whether any human inspection was ever performed. It is also unclear to me what policy correctness means on L115, and how it differs from the floor and ceiling definitions seen in the previous paragraph.
  • Aside from the data, most of the experiments are applying existing techniques to this problem without any modification. Much of the analysis also focus on contrasting the behavior of SFT, vanilla GRPO and OC-GRPO (not contributions of this work) in properties that are not mainly pertaining to those of the proposed task.
  • Given the small evaluation dataset size, the gains obtained are very moderate -- it would be useful to validate whether they are statistically significant in the first place.
  • Overall, the paper is very difficult to follow. The motivating example is not very clear, and a large part of it is in the appendix.
Questions for authors
  • How is "correctness" different from floor=true and ceiling=true?
3 comments

Thank you for the review and for the suggestions on presentation.

W1. Section 3 gives the splits (4,386 SFT / 1,039 RL / 375 test) and states that no test requirement, schema, policy, or plan appears in training. Assignment uses a fixed seed, and the test split covers all areas and domains (Appendix C.5). As we stated in the main paper, some details are given in the corresponding Appendix due to page limitations. This is not uncommon for ML papers. We again summarize them here: scenarios are sampled from 44 hand-written domain skeletons covering 19 application areas. Each scenario is assigned to exactly one split with a fixed seed (4,386 SFT / 1,039 RL / 375 test), and the test split covers all areas and domains. Every scenario must pass an executable filter: all policies validate against the schema, and the target policy satisfies every check under cedar symcc. For the faithfulness review, the results are in our reply to Reviewer m34K (Q4).

W2. The off-context update follows OC-GRPO, as Section 4.2 states. The guidance does not: it is built from formal verification of the model's own failures, pooled across candidates, and explored one check per context, with no reference solution. Table 9 shows each choice matters (46.93% vs. 42.13% without pooling). See point 1 of the general response. On task-specific analysis, Section 6.1 separates validity from authorization reasoning (untrained Qwen3.8-27B is 36.1% correct among valid policies, comparable to both frontier models), Table 10 studies repair from symbolic counterexamples, and Figure 5 separates over-permissive from over-restrictive failures.

W3. Please see point 2 of the general response.

W4. We have revised the motivating example to make it clearer. Because of the page limit, its full details remain in the appendix.

Q. Each scenario has a verification plan with 7.65 checks on average. A check is a ceiling (the policy must not over-permit), a floor (it must not over-restrict), or a liveness check (at least one request is permitted), and each has its own reference policy. As defined in Section 2, a policy is correct only if it parses, validates against the schema, and passes every check in the plan. This is much stricter than passing individual checks: the 9B SFT model passes 78% of checks but only 43% of full plans.

Thank you for the clarification. I can acknowledge that W1 is largely addressed by Appendix C.5, though it does seem like a missed opportunity for generalization evaluation that none of the domains are held out for dev and test splits.

W2: I think the proposed approach makes sense, but is nevertheless very specific to a particular problem.

W3: thank you for adding this, this is helpful!

W4: I don't seem to see the updated revision.

Regardless, I think my assessment of the paper's clarity remains. However, I'll raise my score to 2 thanks to the authors' clarification.

Thank you for following up. The revised PDF is the current version on this submission page. The changes that address your concerns are:

Statistical significance (W3). The paired tests are now reported in Appendix F.3 (p. 29) and Table 12 (p. 32).

Motivating example (W4). The running example is revised in the Introduction (p. 1, lines 42–49), and Figure 1 (p. 4) uses the same requirement. Appendix C.7 (p. 26) shows the incorrect and the corrected policy in Cedar and explains why the verifier rejects the former.

Reviewer_m34KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to turn natural-language access-control requirements into Cedar policies. It makes two contributions. First, CedarInstruct: a dataset of 5,800 synthetic scenarios from 44 hand-written domain skeletons, plus 1,408 scenarios for a single synthetic organization. Specifications are sampled from the skeletons and GPT-5.5 writes each one up as a natural-language requirement. AutoCedar then produces the schema, a target policy, and a verification plan of ceiling, floor and liveness checks. A scenario is kept only if its target policy passes every check under Cedar's SMT-based symbolic compiler. The split is 4,386 SFT, 1,039 RL and 375 test scenarios, all drawn from the same skeletons. Second, RAISE: SFT on the verified target policies, followed by RL from verifier feedback. Six RL variants are compared, ordered by how much verifier output each one uses: binary GRPO, a check-coverage reward, set-dominance preferences, Critique-GRPO, SDPO and RAISE-OC. RAISE-OC applies OC-GRPO to groups where every candidate fails. It takes the M=4 checks failed most often across the K=8 candidates and samples fresh candidates with each check's description and counterexample appended, one context per check. The model is then updated on the original prompt using an importance-weighted objective.

Experiments use Qwen3.5-9B and Qwen3.8-27B, trained with LoRA and a single training seed. SFT gives most of the gain, raising semantic success from 2.67% to 42.93% on the 9B model and from 28.80% to 46.40% on the 27B model. RAISE-OC adds a further 4.00 and 1.60 points, reaching 46.93% and 48.00%. None of the other five variants improves on SFT by more than 0.80 points, and on the 27B model some fall below it by up to 2.93 points. Both trained models beat zero-shot GPT-6 Astra (33.60%) and Claude Opus 5 (30.67%). On CedarBench (221 scenarios rewritten into natural language), RAISE-OC adds 4.07 and 2.71 points over SFT. In the single-organization setting, an SDPO variant whose teacher sees the reference policy solves 94.92% of unseen rule combinations. The paper also includes guidance ablations, a guided update without the importance correction, and repair at inference time.

Critical strengths
  1. Clear writing and a careful setup. All six RL variants start from the same SFT checkpoint and see only the requirement and schema at inference. The paper says exactly what it borrows, and its limitations section names its own weak points: one seed, extra samples, and grading by the same pipeline. The claims are easy to check, and the paper is a very good read.
  2. A trainable, formally checkable resource. Each CedarInstruct requirement comes with an SMT-checked verification plan, so the reward is sound rather than based on tests or an LLM judge. A LoRA-tuned 9B model beats zero-shot GPT-6 Astra and Claude Opus 5 by 13–16 points at a small fraction of the cost (Table 11). This matters for a security-critical task.
Critical weaknesses
  1. Limited novelty. The recipe is the standard SFT+RLVR pipeline, and the method is OC-GRPO applied to Cedar.

    • By its own account (Sec 4.2), RAISE-OC adds a frequency count over failed checks and one guided context per check.
    • Feedback-guided RL for all-failure groups already exists: Scaf-GRPO, Critique-GRPO, SDPO, and adaptive OC-GRPO with tutor hints.
    • RL from compiler or test feedback for code is also established: RLTF, StepCoder.
    • The dataset's schemas, plans and target policies come from AutoCedar, which is prior work. The new part is the sampler over hand-written skeletons.

    The paper is a careful application of existing methods to a new language, but I do not find it to be a new method or a new finding about RL.

  2. The empirical support is weaker than the headline numbers suggest.

    • The RL gain is small. Over SFT, RAISE-OC solves 15 more of the 375 test scenarios on the 9B model and 6 more on the 27B model. It beats the next-best variant by 12 and 4 scenarios.
    • No measure of variance. Every result is a single run with one output per scenario: greedy decoding for Qwen, one sample for each frontier model, and one training seed for every method. No standard deviations or confidence intervals are given. F.3 promises McNemar tests and bootstrap intervals but reports no results, and only pass@1 is reported. With margins of 4 to 15 scenarios, another seed could reorder the RL variants.
    • The ablations vary as much as the gain itself. On the 9B model, M=2 scores 41.87%, below SFT (42.93%), while M=1 scores 44.53% and M=4 scores 46.93%. On the 27B model, every ablation lies between 45.87% and 48.00%. On CedarBench, plain binary GRPO gains 3.17 points on the 9B model, against 4.07 for RAISE-OC.
    • It is unclear how M=4 was chosen. The paper states that no hyperparameter was chosen on the test split, but there is no multi-domain dev split, and Table 9 is computed on the test split.
    • The single-organization result doesn't support RAISE-OC. The 94.92% comes from an SDPO variant, not RAISE-OC, and there is no SFT-only baseline.
    • The frontier comparison favours the trained models. The test split is in-distribution, and correctness is defined by plans that the AutoCedar/GPT-5.5 pipeline wrote; the trained models saw 4,386 examples from it. GPT-5.5, which built the dataset, is correct on 46% of its valid policies, against 34% for GPT-6 Astra and 31% for Claude Opus 5, and close to the trained models' 47–48%. Part of the frontier gap may therefore reflect how the pipeline reads ambiguous requirements rather than weaker reasoning. The frontier models are also not evaluated on CedarBench, so the +13–16 point claim holds only in-distribution.
    • The single-shot setting may not reflect how such pipelines are used. It is an interesting finding that a fine-tuned 9B model beats GPT-6 Astra and Claude Opus 5 on semantic correctness. In practice, though, NL-to-spec pipelines usually run as a multi-turn or agentic loop with a verifier. The authors acknowledge this in the introduction but mention that such loops are costly, and they focus on single-shot performance. That is a fair choice, but I would be interested to see if the gap would persist if both the frontier models and the fine-tuned model ran in an agentic loop with access to the verifier. The paper's own Table 10 hints at this; it runs verifier-guided repair only for the Qwen models. On the 9B model, RAISE-OC's lead over SFT shrinks from 4.00 to 2.67 points after two rounds of repair. On the 27B model, binary GRPO overtakes it (80.80% vs 78.93%). It would be interesting to see whether the fine-tuned model keeps its lead once the frontier models get the same loop.
Questions for authors

See weaknesses. Also:

  1. Can you report the mean and standard deviation over at least three training seeds for SFT, binary GRPO and RAISE-OC, along with the McNemar and bootstrap results promised in F.3? Can you also report pass@k with sampling, both for each method and for the frontier models?
  2. Were the frontier models' accuracy runs done through Codex CLI and Claude Code, and could they call tools such as a Cedar validator? How do they score on CedarBench, and on CedarInstruct with a few CedarInstruct training examples in the prompt?
  3. How do GPT-6 Astra and Claude Opus 5 score with the same two rounds of verifier-guided repair as in Table 10, or inside AutoCedar's loop? Does the fine-tuned model keep its lead there, and at what cost per scenario?
  4. How many scenarios did the LLM and human faithfulness reviews cover, and what fraction of the plans did each find faithful?
4 comments

Thank you for the detailed review. We follow your numbering.

W1.1. We agree that RAISE-OC builds on OC-GRPO, and the method section says so. The additions are small, but each one contributes. In Table 9 (9B): frequency-based selection vs. random (46.93 vs. 45.60), separate contexts vs. one shared prompt (46.93 vs. 44.80), and pooled failures vs. per-rollout failures (46.93 vs. 42.13).

W1.2. Critique-GRPO and SDPO are included as baselines and receive the same failure descriptions. Neither gains more than 0.80 points over SFT, and RAISE-OC is significantly better than Critique-GRPO on 9B. Scaf-GRPO and OC-GRPO rely on reference solutions or LLM-written hints, which RAISE-OC does not need. We did not run a head-to-head comparison with them, and the paper lists this as a limitation.

W1.3. Coverage GRPO and Formal Dominance GRPO are our counterparts of the finer-grained rewards in RLTF and StepCoder. Neither gains more than 0.80 points over SFT.

W1.4. Correct, the schemas, plans and targets are produced with AutoCedar, as the paper states. Our additions are the specification sampler, the filtering and splits, the single-organization set, and the use of these artifacts for training rather than inference-time repair.

W2.1, W2.2. Please see point 2 of the general response, which reports the tests promised in the appendix.

W2.3. The ablations in Table 9 point the same way: on 9B the full RAISE-OC configuration is the best setting in every comparison (selection rule, separate contexts, pooled failures, and M). On CedarBench, RAISE-OC also has the highest semantic success among the trained models on both 9B and 27B.

W2.4. M = 4 gave the highest score in Table 9 on 9B, and we used it. You are right that this comparison was made on the test split, since we have no development split, so the choice of M is not independent of the test data. We have corrected the protocol statements in Sec. 5.1 and Appendix C.5.

W2.5. This study asks a different question from the main experiments: how far a broadly trained model can be adapted to one organization's fixed schema, not which training method is best. We used an SDPO variant because the setting is narrow, and a teacher that sees the verified reference gives a dense signal (Sec. 6.2).

W2.6. The test split is in-distribution, and the paper acknowledges that trained models "may partly learn its reading of ambiguous requirements". CedarBench keeps its original verification plans and gives partial evidence beyond our pipeline, and following your comment we have evaluated the frontier models there as well (revised Table 4 and Section 6.2).

W2.7, Q3. We disagree that this is a critical weakness of our approach. Verifier-guided repair at inference is outside the scope of our claim, which is limited to single-shot generation without verifier feedback at inference. We have not run frontier models with verifier-guided repair, so we do not claim the advantage persists in that setting. On Table 10, your reading is correct. RAISE-OC uses verifier feedback only during training, and only to guide exploration; its off-context update credits the policy conditioned on the requirement and schema alone. The model is therefore never trained to condition on feedback, so we would not expect its single-shot gain to carry over fully to repair.

Q1. For seeds and significance tests, please see point 2 of the general response. Since our setting is single-shot generation, we report sampled pass@1: the mean success rate over 8 samples per scenario at temperature 1, from the Table 2 checkpoints.

ModelGreedySampled pass@1
9B SFT42.9338.07
9B Binary GRPO43.4742.77
9B RAISE-OC46.9343.77
27B SFT46.4037.33
27B Binary GRPO44.8045.47
27B RAISE-OC48.0043.93

Under sampling, RAISE-OC improves over SFT by 5.70 points on 9B and 6.60 on 27B. Binary GRPO improves similarly, and the two RL methods are not separable here. RAISE-OC's advantage over Binary GRPO is in greedy decoding, the paper's primary metric. We have not run this evaluation for the other instantiations or the frontier models.

Q2. The frontier models were run through Codex CLI and Claude Code with the zero-shot prompt in the appendix. They had no access to the Cedar validator or the symbolic checker, and each produced one policy per scenario with no verifier feedback. Their CedarBench results are in the revised Table 4.

Q4. For the faithfulness review, we audited 775 policy synthesis instances across the test, SFT, and RL splits, conducting 1,550 independent judgments over two stages: specification-to-requirement fidelity and requirement-to-verification-plan alignment. The automated audit achieved pass rates of 94.2% and 89.0% for the two stages, respectively, with 84.0% of instances passing both. In a separate blind human evaluation of 100 sampled instances, the automated judgments achieved 88.0% agreement with human annotations.

Dear Reviewer m34K,

Thank you again for your detailed review. As the discussion period closes today, we wanted to check whether our response and the revised PDF address your concerns. We would be happy to clarify anything that remains open before the deadline.

Reviewer_m34KOpenReview ↗

Official comment by Reviewer m34K

Thank you for the detailed response and for conducting the additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the additional CedarBench evaluation, sampled results, and faithfulness audit.

However, several important concerns remain. First, while the ablations help justify the design choices in RAISE-OC, they do not sufficiently establish its novelty over existing feedback-guided RL approaches. Second, selecting M=4 based on the test split introduces a degree of test-set tuning, making the reported performance less conclusive. Third, although RAISE-OC performs best under greedy decoding, the sampled results show that binary GRPO performs comparably on the 9B model and better on the 27B model. This makes it difficult to conclude that RAISE-OC offers a consistent advantage over the simpler baseline. Finally, although the CedarBench evaluation is a useful addition, the main evaluation still relies on verification plans generated by the same AutoCedar pipeline used to construct the training data.

I appreciate the authors' effort in addressing my concerns and providing these additional results. The response has helped clarify some aspects of the work. However, given the remaining concerns, I feel that maintaining my original score is appropriate. Thank you again for the thoughtful response.

Thank you for the follow-up and for acknowledging the additional results. We briefly respond to the four remaining points.

Novelty over feedback-guided RL. Our experiments include a head-to-head comparison with existing feedback-guided RL methods: Critique-GRPO and SDPO receive the same failed-check descriptions as RAISE-OC. Neither gains more than 0.80 points over SFT, and RAISE-OC is significantly better than Critique-GRPO on 9B (p = 0.011, Table 12). Since all three methods receive the same verifier information, the difference lies in how it is used, which is the contribution we claim.

Choice of M. The conclusion does not depend on M = 4. In Table 9, every configuration on 27B except per-rollout failures, and five of seven on 9B, score above every other RL instantiation in Table 2. CedarBench played no role in choosing M, and RAISE-OC is also the best trained model there on both Qwen models (Table 4).

Sampled pass@1. The two metrics measure different things. Sampled pass@1 is the chance that a randomly sampled policy is correct, so it rewards any probability placed on correct policies. Greedy decoding, which the paper reports and a deployed synthesizer would use, produces a single deterministic policy by choosing the most likely token at each step, so it requires the model's preferred output to be correct. The difference is concentrated on hard scenarios. On the 97 scenarios that the 27B SFT model solves in fewer than half of its samples, Binary GRPO's samples succeed about as often as RAISE-OC's, but its greedy output is correct on 25 of them, fewer than SFT's 31, while RAISE-OC's is correct on 34. These are the inputs most likely to produce all-failure groups during training, where binary reward has no gradient and RAISE-OC adds guided signal (Section 6.1). On CedarBench, Binary GRPO gains nothing over SFT on 27B (49.77 vs. 49.77), while RAISE-OC gains 2.71 points.

Shared pipeline. All training methods in Table 2 are trained on data verified against, and graded by, the same verification plans, so the comparison between them is controlled for this. CedarBench was constructed independently of CedarInstruct, with its own verification plans, and RAISE-OC remains the best trained model there.

We thank you again for the careful and constructive review.

Reviewer_ZfwYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper teaches a model to output access-control policies in Cedar by including off-policy rollouts sampled with privileged information from the target model. They also create the dataset called CedarInstruct necessary for the training. Their procedure trains the model through an SFT stage and then an off-policy mixed RL stage which they call RAISE-OC and show that SFT is sufficient to learn most of the syntax but not the semantics required to be correct. Naive RL also cannot increase semantic performance but RAISE-OC can.

Critical strengths
  1. They create a new dataset called CedarInstruct.
Critical weaknesses
  1. Why does normal RL decrease performance over the SFTed model, especially 27B? Even if about 30% of the groups do not succeed 70% of the prompts should be enough signal to improve upon the base model.
  2. The gains from RAISE-OC seem minimal. How come using privileged information did not help that much? Is the nature of this task that hard?
  3. I do not understand the novelty of this work. This work applies SFT and then their own stage of off-policy mixed RL to the model on access-control policies in Cedar. The training pipeline itself is not novel so the contribution would be that they applied mid-training and RL to access-control policies. However, applying training to a task is not novel. Furthermore, the analysis isolates the effects of syntax learning and semantics learning and state that syntax can be increased by SFT but semantics are increased by RL. This is also not surprising. The analogy to math tasks would be that mid-training gives the base model math capabilities for the RL to improve upon and RL can help the base model find answers on hard tasks. The base math capabilities are like the syntax and further enhancing that to harder math problems is like the semantics. These intuitions that translate to other domains seem to be already known.
Questions for authors

See above.

2 comments

Thank you for the thoughtful review.

W1. The 70% of groups that are not all-failure do not all carry signals. A GRPO group also yields no gradient when all its candidates succeed, and on 27B 32% of the groups are all-success, on top of the 33% that are all-failure (the learning cliff of Figure 2). Only 35% of the groups therefore carry any signal, and that signal is used for prompts the model can already solve. The hard prompts are the ones that fall into all-failure groups, so binary reward gives them almost no signal, and this is where Binary GRPO falls below SFT on 27B, which is the drop you noticed. RAISE-OC is built to learn from exactly these groups, and it solves more of the hard scenarios than both SFT and Binary GRPO on both 9b and 27b.

W2. Three remarks. First, the guidance is used only during training, and at inference the model sees just the requirement and the schema. When the same feedback is available at inference, one repair round takes 9B SFT from 42.93% to 64.00% (Table 10), so the information is valuable and the difficulty lies in internalizing it into single-shot generation. Second, the 9B gain is statistically significant (+4.00, p = 0.014, see the general response). Third, the task is indeed hard. A policy must pass every check in the corresponding verification plan, so the 9B SFT model passes 78% of individual checks but only 43% of full plans, and GPT-6 Astra and Claude Opus 5 reach only 33.60% and 30.67% on this test split.

W3. In brief, our claim is not that "SFT gives syntax and RL gives semantics." Our claim is that standard RL largely fails to improve semantics in this setting, and we trace this to reward sparsity. Unlike test-based rewards for code, correctness here requires passing every symbolic check in a plan (7.65 on average), each quantified over all requests. A policy that gets most of a requirement right therefore earns the same zero as one that gets none of it: the 9B SFT model passes 78% of checks but only 43% of plans, and 33 to 39% of sampled groups contain no correct policy and yield no gradient. Denser check-level rewards do not fix this, because passing more checks does not make a policy correct (Table 2). The remaining errors lie in how rules compose, for example an exception overriding a restriction it was not meant to lift, which syntax supervision does not teach. Verifier-derived guidance recovers part of the missing signal without any reference solution or tutor model.

Thank you for your response. You have provided sufficient detailed explanation, especially regarding the novelty of this work, and my concerns have been addressed. I raise my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their time and constructive feedback. Two concerns are shared across the reviews, novelty and the statistical reliability of the gains, so we address them here and refer to this response in the individual replies.

  1. Novelty. We agree that SFT followed by RL is a standard recipe and that the RL components we use have precedents, and the paper does not claim otherwise. Our finding is that this recipe does not work out of the box for policy synthesis, and the paper’s contribution is investigating why this is the case and how to address it. With SFT taking care of syntax, one would expect RL with a verifier reward to fix the semantics. Table 2 shows that it does not: five of the six RL instantiations gain at most 0.80 points over SFT, including Critique-GRPO and SDPO, which are given the same failed-check descriptions as RAISE-OC. The cause is the reward signal. When we went back to the Binary GRPO training logs for this rebuttal, we found that in addition to the roughly one third of groups in which every candidate fails (Figure 2), another third contain only correct candidates, so only 29% (9B) and 35% (27B) of the groups give any gradient. RAISE-OC is our answer to this. Prior guided-exploration methods take their guidance from reference solutions or tutor-written hints, whereas RAISE-OC builds it from the verifier's failed checks and counterexamples on the model's own candidates, and it is the only instantiation that gains more than 0.80 points on either backbone. None of this could be trained or measured without CedarInstruct, which to our knowledge is the first dataset that supports both training and semantic evaluation of Cedar policy synthesis: each of its 5,800 scenarios has a verified target policy and an SMT-checkable verification plan.

  2. Statistical significance. The appendix describes paired tests but did not include the numbers due to an oversight; we thank the reviewers for pointing this out. The revised version reports them in Table 12 (Appendix F.3), computed on the Table 2 outputs (375 scenarios, two-sided exact McNemar test, 95% paired bootstrap CI). RAISE-OC is ahead in all twelve comparisons. On 9B the gain is significant against SFT (+4.00, 95% CI [+1.07, +7.20], p = 0.014), Binary GRPO (+3.47, p = 0.024), Coverage GRPO (+4.80, p = 0.003) and Critique-GRPO (+4.27, p = 0.011), and falls just short of significance against Formal Dominance and SDPO (both +3.20, p ≈ 0.1). On 27B it is significant against Formal Dominance (+4.53, p = 0.027), while the +1.60 over SFT is not significant, which is consistent with the paper’s own caveat that margins of a few scenarios may reflect run-to-run variation. All results use a single training seed, as stated in the paper. With twelve RL runs in Table 2 and roughly 27 GPU-hours for one RAISE-OC run on 9B, we were unable to add seeds during the rebuttal period. We note that this is not uncommon, as other related works (SCoRe, Scaf-GRPO, Critique-Coder) also report point estimates, without results over multiple training seeds or significance tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.