acceptodds
Under review as a conference paper at ICLR 2027

When Is One Reading Enough? Decision-Relevant Ambiguity in Constrained Reinforcement Learning

44 views

Abstract

Safety rules for AI agents, online services, and industrial equipment are usually written in natural language, yet a learned controller can obey a rule only once it is a precise cost that training can measure. One sentence, such as "drain a server when load stays high," often admits several plausible executable readings, each a different constraint. We cast this as constrained reinforcement learning (RL) with candidate cost functions and ask whether the controller can train under one reading or must obey them all. Obeying all is safe but can give up performance, while picking one can yield a controller another reading forbids. We propose ARROW (**A**mbiguity **R**elevance for **R**einforcement Learning **O**ver **W**ritten Rules), a test that runs before training and searches for a controller nearly optimal under one reading yet violating another. If none exists, training under that reading is safe for every candidate reading. Otherwise ARROW keeps the competing readings or asks the operator targeted questions, re-testing after each answer. The test is exact when the dynamics are known, gives a high-confidence certificate from a fixed log, and works with any constrained learner. On agent-service tasks, training under the reading ARROW certifies raises return by a per-learner median of 15.0 to 37.3 percentage points of the best return that obeys every reading, across six learners. On service-monitoring rules, four questions settle what to enforce for 99% of ambiguous rules, usually before the intended reading is known. On 182 real ventilator, robotics, and cyber-physical requirements, 87.4% keep several distinct readings after redundant ones are removed, so the question is routine.

Primary area:
reinforcement learning
Keywords:
Decision-relevant ambiguity, Ambiguous safety rules, Natural-language constraints, Constrained reinforcement learning, Safe reinforcement learning, Offline certification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7MwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In constraint-based RL, users may express their constraints and requirements in Natural language (NL). NL can be ambiguous and we may get multiple readings from it. This paper introduces ARROW a method that decides whether one reading can replace the whole candidate set or not. If this reading does not exist, it asks question to finally find one. The method tested on different aspect to show improvement and effectiveness in constraint-based RL.

Critical strengths

1- The like and fully agree that NL is vague and we need a more rigor approach to guide RL agents, so I like the main motivation of the paper.

2- In sec.3, the process of how we decide one reading is sufficient or not seems effective and smart.

Critical weaknesses

I thank the authors for their hard work, however I believe there are multiple issues regarding their submission which I split them into two main categories Formalism and Writing and Experiments.

Formalism and Writing

1- What are readings, are they logical formulas, and how we get the set of all readings from the NL input?

2- An important concern is, in line 70, paper mentions that ARROW take place before training, so how we get set of policies Π\Pi? because there is no training take place that we have access to the policies.

3- If the dynamics are known this will limit the usage of this work only to model-based RL. And if we know the dynamics, we can easily generate different formulas using NL-to-LTL and then start from the most expressive one, by producing the composition of MDP and buchi automata and then check the feasible paths. If no feasible paths found we can try the less expressive ones, and continue till we find one formula that has a feasible path.

Experiments

4- What are the benchmarks used for the experiments, there is no context of what exactly used? For an example, what is the context of agent-service benchmark? What are the objectives? Why 9 out of 36?

5- The baseline methods used in Tab.3 translate a NL to LTL formula, ARROW’s objective is how one reading can replace all others, the output of ARROW is a reading (which I am not sure what is that) but these methods are LTL formulas, so what exactly was compared with these methods since they have different outputs.

Minor Issues

1- In line 36 Written safety rules are ambiguous is too strong, we may express rules in NL in a formal way that we get only one reading from it.

2- The way the MDP is defined, ARROW cant be applied in to continuous environments, if this is true, authors should mention that in the limitations.

3- What is B\mathcal{B}?

4- Π\Pi sometimes gets a reading ψ\psi or a set UU, this is inconsistent. Write it as {ψ}\lbrace\psi\rbrace.

5- What is DEPLOY ?

6- What is going to be predicted in lines 230-231?

7- kl in 308 should be KL.

8- The why paragraphs dont really explain why.

Questions for authors

I like the idea that NL is vague and we may get different understandings from a single NL constraint. However, imagine we get a NL constraint from the user and we extract different readings (for example LTL formulas), why not from the beginning we ask the users (we know we have access to the users since in ARROW we will ask questions from them if we dont find a good reading) that please clarify your request since we are getting multiple reading from this. I think the motivation of the paper lacks something HUGE that I am asking this (maybe basic) question. I would like the authors to explain that what was their intention of solving this problem and what is missing that I am proposing this trivial approach to solve this problem?

11 comments
AuthorsOpenReview ↗

Clarifying ARROW's Motivation, New Experiments, and Formalism

Thank you for the detailed review. We address your questions below.

Why not ask the operator to clarify immediately?

Asking the operator is valid, and ARROW does so when needed (§3.2). Identifying the intended interpretation seeks semantic identification. ARROW checks whether one training constraint already protects every feasible near-optimal policy under all remaining readings. If so, training proceeds under that constraint and the learned policy is verified or certified before deployment (§3.3). Otherwise, ARROW asks targeted questions about distinctions preventing certification.

This distinction matters empirically. §5.2 shows that 26/87 ambiguous monitoring sets need no question. Among the other 61, 56 reach sufficiency before the governing reading is identified. After four questions, ARROW reaches 99% sufficiency while balanced splitting achieves 37% full identification. Because those measure different outcomes, we ran a new paired comparison on one metric, the ambiguity price recovered. It covers 82 positive-price sets with bootstrap resampling over 77 alert families. Five zero-price sets were excluded because they had no recoverable price.

QuestionsDecision-focusedBalanced splittingImprovement95% CI
164.5%48.0%+16.6 pp[+10.2, +23.6]
282.4%77.0%+5.4 pp[+2.4, +9.0]
495.2%85.9%+9.4 pp[+6.7, +12.2]

Differences use unrounded values. All intervals exclude zero, supported by paired sign and family-level permutation tests.

Formalism and experiments

  1. What is a reading? A plausible executable interpretation of a written safety rule, represented by a cost cψc_\psi. The plain-language definition appears in §1 (lines 42–43) and the mathematical one in §2.1 (lines 129–133). Generation appears in §3.2 (lines 218–221).
  2. How can ARROW act before training? Π\Pi specifies the learner's policy class (§2.1, lines 119–121). ARROW examines its near-optimal policy set through occupancy LPs before training (§3.1, lines 197–204).
  3. How does this relate to NL-to-LTL and automaton planning? NL-to-LTL produces formal specifications, and automaton planning finds feasible policies. ARROW certifies whether every near-optimal policy under one reading satisfies the others (§2.2, lines 153–159). Table 1 shows why equal optimal returns are insufficient. Theorem 4.2 addresses unknown dynamics with fixed logs.
  4. What is the agent-service benchmark and why 9/36? A tool-using agent calls external services under an ambiguously scoped limit (§5.3, lines 405–410). Across 36 demand, service-count and budget settings, 9 admit a sufficient reading. ARROW correctly retains full protection in the other 27. This demonstrates its intended behavior of simplifying when certified and preserving all constraints when needed (Appendix B.2, lines 1050–1061 and Appendix D.3.1, lines 2346–2353).
  5. What does Table 3 compare? On 182 real requirements, every candidate is an executable temporal-logic formula assessed against expert-accepted interpretations. Translator rows report whether the first or any sampled formula is plausible. SCORE-KEEP filters pooled formulas and measures whether at least one plausible reading survives (§5.4, lines 432–442 and Table 3).

Minor points

  1. Agreed. We will change “Written safety rules are ambiguous” to “can be ambiguous” (§1, line 36).
  2. Certification uses finite or compiled MDPs (§2.1, lines 113–116). Table 19 evaluates portability across five released offline safe-RL learners on DSRL/OSRL. We will clarify the certification scope in Limitations.
  3. B(π^,UΞ)\mathcal B(\hat\pi,U_\Xi) is CHECK's upper cost-confidence bound (§3.3, lines 241–247). BqB_q is the clarification-question budget (§3.2, line 229). We will distinguish these symbols.
  4. Πψ(d)=Π{ψ}(d)\Pi_\psi(d)=\Pi_{\{\psi\}}(d) denotes policies satisfying reading ψ\psi (§2.1, lines 140–149). We will state the equality explicitly.
  5. DEPLOY is the event that the pipeline deploys π^\hat\pi, with CHECK passing on the checked route and the certified planner returning a policy on the other (§4.4, lines 324–326). We will introduce it earlier.
  6. qξ(ϕ)q_\xi(\phi) is the answer predicted by reading ϕ\phi to question qξq_\xi. The operator's answer determines which readings remain (§3.2, lines 229–233). We will state this.
  7. kl\mathrm{kl} denotes binary relative entropy and KL\mathrm{KL} denotes divergence between general distributions (§4.2, lines 298–309). We will clarify the distinction.
  8. The experimental “Why” paragraphs explain why each study matters and the practical motivation behind it. We will make this motivation and its importance clearer.

We hope these answers address your concerns.

Revision plan. We will sharpen the motivation, add the new paired comparison results and implement the presentation changes above.

Thank the authors for their rebuttal.

  1. Your rebuttal make me more confused in 1 you are saying ψ\psi is plausible executable interpretation of a written safety rule in 5 you are saying it is executable temporal-logic formula, these are contradicting. First of all what is a executable temporal-logic formula. Formula is a logical expression it is not program that we can execute that.

  2. In ICLR, authors can revise their paper for rebuttal, so why not revising the issues right now, and then the reviewer can really see the revised versions.

Thank you for the follow-up.

1. What Table 3 compares. Your Question 5 asked how ARROW's readings compare with the LTL formulas the baselines produce. All candidate readings in that study are represented as LTL formulas, so the comparison uses one representation and one criterion. Each translator row is scored on its first sample, with the parenthesized column on any sampled translation. Score-Keep filters the pooled candidates and measures whether a plausible interpretation survives. The pooled rows draw on the union of all archives at 190 samples per requirement, with no new model calls. Matching uses exact LTL equivalence (Appendix B.3, lines 1271–1292).

How readings are represented. A reading is one interpretation of a written safety rule (§1, lines 42–43). In constrained RL it determines a safety cost cψc_\psi (§2.1, lines 129–133). Each decision model charges its own quantity, and Table 6 lists the four. In the monitoring model a threshold-duration reading compiles to a duration counter in the state, and the cost is evaluated at each timestep (Appendix B.2, lines 1020–1044). Table 3 evaluates candidate interpretations, while the decision-model studies evaluate certification.

What executable means. Executable temporal logic is established terminology in formal methods, surveyed under that name by Fisher (The Knowledge Engineering Review, 1996). The formula is the specification and the executed object is produced from it. One established mechanism transforms a specification into rules of the form "if the past satisfies A then do B", and those rules are what execute. This is consistent with §1 (line 37), where we explain that a written rule can be made executable in several ways. Since Table 3 scores formulas by equivalence, candidate temporal-logic formula is the more precise term there and we will use it.

2. The revised manuscript

We posted the responses first so that the new evidence was available as early as possible. We are now incorporating all the reviewers' feedback into the manuscript and will upload a revised PDF soon.

Thank you for your time.

Reviewer_UMH2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies ambiguity in natural language safety specifications for constrained reinforcement learning. Given several candidate cost functions representing possible interpretations of a rule, it asks whether training under one interpretation is sufficient to ensure that every feasible, sufficiently near-optimal policy satisfies the other interpretations. The proposed framework, ARROW, tests this property using occupancy-measure linear programs when the dynamics are known and a robust-dual certificate when they are estimated from offline data. When no interpretation is sufficient, the framework retains the competing interpretations or asks clarification questions. Experiments examine specification ambiguity, clarification effort, offline certification, and downstream constrained learning.

Critical strengths
  1. The problem is well motivated. Different executable interpretations of a safety instruction can change which policies are acceptable. Distinguishing ambiguity that affects decisions from ambiguity that need not be resolved is a useful objective.
  2. The formulation makes an important distinction. Equal optimal returns under two interpretations do not imply that their near-optimal policies satisfy the same constraints. Requiring safety across all policies within an optimality tolerance accounts for multiple optima as well.
  3. The paper provides extensive supporting analyses. Different policy classes, budgets, multiple downstream learners, real requirements, and comparisons of offline certificates.
Critical weaknesses
  1. In the known finite-MDP setting, ARROW computes an optimal value and solves additional occupancy-measure LPs to determine whether one constraint suffices. However, the same representation can directly compute a policy satisfying all candidate constraints. It is therefore unclear when certification followed by RL training is preferable to solving multi-constraint planning problem itself.
  2. The headline results are based on avoiding a conservative single-cost surrogate. The pointwise maximum cost used for learners accepting one cost signal is generally stricter than enforcing each expected-cost constraint separately. Improvements over this surrogate do not establish comparable benefits over native multi-constraint learning. Appendix D.3 acknowledges this, and Table 18 reports identical policies for vector Lagrangian FQI, vector PID-Lagrangian with sufficient data, and the occupancy planner. These results narrow the performance claim. The contribution appears to be strongest for learners restricted to a single cost interface rather than a general reduction in the conservatism of multi-constraint RL.
  3. Several mathematical results directly follow from the definitions. Marking several straightforward consequences of the definitions as a theorem dilutes the major contribution, which appears to be the robust dual-test (Thm 4.2)..
Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Certification, Native Baselines, and Theoretical Contributions

Thank you for recognizing the distinction between optimal-value agreement and safety across all near-optimal policies. We address the practical role of certification, native multi-constraint baselines and the central theorem.

W1. Why certify instead of directly planning with every constraint?

The two approaches answer different questions. A full-set occupancy LP constructs one safe policy when the finite model is known. ARROW certifies whether every feasible ε\varepsilon-near-optimal policy trained under one reading obeys the entire set. This lets existing single-cost learners use a certified constraint. Theorem 4.2 extends this certification to unknown dynamics with a fixed offline log.

Following your feedback, we conducted two additional analyses.

  1. Nontrivial certification. Among 17 real compiled rules, 16 admit a sufficient reading although none has a pointwise-dominant reading.
  2. Exact-optimum preservation. We verified zero optimal-value gap between singleton and full-set optimization on all 9 certified agent-service instances and 22 feasible real-rule instances.

Table 1 already illustrates why value agreement alone is insufficient. At d=0.005d=0.005, 22/28 rules pass value screening while none passes the full sufficiency test.

For the 22 easier sufficient readings at d=0.05d=0.05, Table 14 reports the smallest reliably certifying tested grid sizes. They are 3.2×1033.2\times10^3 transitions for ARROW's robust dual, 3.2×1063.2\times10^6 for occupancy-weighted sampling and 3.2×1073.2\times10^7 for uniform sampling. These results demonstrate the fixed-log advantage across certification methods. They do not compare direct-planning runtime.

W2. Do gains extend beyond the conservative scalar surrogate?

Yes. Beyond the 15.0–37.3-point gains over the scalar surrogate (Table 2), ARROW improves native vector CPQ by 3.5–5.0 points on all 9 instances. The other native methods show comparable performance (Table 18).

These results demonstrate that ARROW simplifies the training constraint while preserving the exact optimum, as guaranteed by Theorem 4.1(c). It can also improve finite-data performance for native vector-constrained learners.

W3. What is the central theoretical contribution?

Theorem 4.2 is our main theoretical contribution. It establishes a sound robust-dual certificate using fixed logged data, together with a decision-information lower bound. Theorem 4.1 provides the near-optimal-set criterion underlying this certificate. The 22/28 versus 0/28 result illustrates why this criterion is stronger than matching optimal values alone. We will emphasize this connection in the revision.

We hope these explanations and additional experiments address your main concerns. We would appreciate reconsideration of the overall assessment.

Revision plan. We will emphasize the existing native-vector comparisons (Table 18) and Theorem 4.2, incorporate the new certification and exact-optimum analyses and clarify the fixed-log experiments.

Thank you for the clarification and extra experiments. The practical advantage of certifying every near-optimal policy, rather than directly obtaining a safe policy, is still unclear to me.

  1. Where is certification preferable to direct planning? Could you provide a concrete setting where a full occupancy LP is feasible, but deploying its policy is unsuitable and certifying a constraint for another learner offers a meaningful advantage?
  2. What benefit remains for native multi-constraint learners? Beyond vector CPQ, is there evidence of reduced training time, sample requirements, or instability when the final policy quality is unchanged?

These questions are about the practical significance of the stronger guarantee, I understand that certification and policy construction answer different mathematical questions.

Thank you for your time and reply.

1. When certification is preferable to direct planning. It helps when the dynamics are unknown and when an existing learner works through a single-cost interface. We show both.

Unknown dynamics. A full-set occupancy program needs the true transition kernel, which a fixed log does not give us (p. 19, line 1002). Theorem 4.2 enables high-confidence sufficiency certification from the log. On the compiled rules the robust dual certifies 22 readings from 3,200 transitions against 3.2 million for the occupancy-weighted certificate (Table 14, p. 45). Both of those are certification methods, so that thousandfold gap is in how cheaply sufficiency is established.

Existing learners. A single-cost learner can protect every candidate through their pointwise maximum, which costs return. ARROW certifies when one candidate cost can replace that surrogate. We compared the options on 9 agent-service instances, where full-set planning is feasible in 0.8 to 35.8 ms.

  • the surrogate protects every candidate and gives up return
  • the certified reading earns 17.3 to 21.5 points more than the surrogate, on at least 357 of 360 paired runs per learner
  • an uncertified reading incurs a competing cost of up to 2.06 times the budget

Learned policies are checked before deployment, since an approximate learner can fall outside the certified set (p. 5, line 248). Where the kernel is known and the planned policy is deployable, direct planning is the right tool.

2. Is there benefit beyond vector CPQ and what benefit remains? Yes. We varied the constraint count over K∈{2,4,8,16,32}K\in\{2,4,8,16,32\}, keeping the state and action spaces fixed. All 35 instances (7 per KK) have one reading certified as sufficient, with the same exact optimum as the full constraint set. No candidate reading dominates another.

LearnerFull KK constraints versus one certified reading
occupancy LP on P^\hat P1.23x to 1.59x the simplex iterations and up to 1.36x the wall-clock, with identical policies on 140/140 pairs at every KK
vector CPQ3.0x to 42.1x the wall-clock, and the certified reading achieves higher return
both vector Lagrangian learnersidentical policies at the main log size, with no material runtime difference

The occupancy planner directly addresses the unchanged-policy-quality question. Using one certified reading reduces optimization work while producing the same policy, and the computational benefit increases with KK. Although individual occupancy solves take only 0.6 to 1.0 ms on these small instances, vector CPQ achieves a substantial absolute saving of 6.3 seconds per run at K=32K=32.

Three practical benefits are now demonstrated. ARROW enables substantially more data-efficient offline certification, less conservative training for single-cost learners, and reduced optimization costs for native multi-constraint planning.

Thank you again.

Reviewer_K7dnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates ambiguity in natural-language safety constraints for constrained reinforcement learning. It proposes ARROW, a framework for determining whether every feasible, near-optimal policy under one interpretation also satisfies all interpretations in a specified candidate set. The framework combines occupancy-measure and robust-dual certification, targeted clarification, and deployment checks. The decision-relevant treatment of ambiguity is informative.

Critical strengths
  1. The sufficiency criterion distinguishes complete semantic identification from the information needed to support constrained decision-making. Requiring constraint satisfaction across the entire near-optimal policy set appropriately accounts for ties and approximate optimization. The sufficiency radius provides a precise characterization of the associated tolerance.

  2. The paper develops exact certification for finite MDPs and extends the analysis to transition uncertainty through robust-dual bounds. The distinction between certification, counterexamples, and inconclusive outcomes is useful for understanding what can be established from available information.

  3. Applying decision-region and hyperedge-cutting ideas allows clarification to stop when the remaining ambiguity permits a sufficient interpretation. This provides a principled alternative to requiring complete identification of the governing interpretation.

Critical weaknesses
  1. The generation and semantic-retention terms in Theorem 4.4 concern coverage of the actual governing interpretation. However, the ARTEMIS evaluation measures whether the generated or retained pool contains at least one expert-plausible interpretation. These events are not equivalent: retaining one plausible interpretation does not establish coverage of another interpretation that governs the task. Consequently, the reported coverage statistics cannot directly be interpreted as bounds on the corresponding theoretical failure probabilities without an additional argument. This leaves an important gap between the semantic evaluation and the empirical support for the deployment guarantee.

  2. The main reported improvements of approximately 15–37 percentage points are obtained against the pointwise-maximum single-cost surrogate. Table 18 shows smaller benefits against native multi-constraint methods, including zero gain for vector Lagrangian FQI and approximately 3.5–5.0 percentage points for vector CPQ. The distinction matters because the surrogate can impose stronger constraints than enforcing the interpretations separately. Moreover, Theorem 4.1 establishes equality of the optimal values when an interpretation is sufficient; practical improvements must therefore be attributed to avoiding surrogate conservatism, improving optimization, or reducing other costs. The logging distribution also draws 70% of the data from the primary interpretation’s optimal policy. Although this is a legitimate experimental choice, its alignment with the selected interpretation leaves the robustness of the learning gains insufficiently established.

  3. Table 19 demonstrates compatibility with existing deep RL learners, but does not establish the computational scalability of a nontrivial LP or robust-dual sufficiency test in high-dimensional continuous state spaces. The reductions achieved through constraint pruning are useful, but do not establish scalability when many mutually incomparable interpretations remain. The finite-MDP setting is a legitimate scope; the concern is that learner-interface portability provides limited evidence for broader applicability of the certification contribution.

  4. Transition uncertainty within a specified model class differs from errors introduced by state abstraction, rule compilation, or unmodeled execution effects. Such errors can affect both the sufficiency decision and the correspondence between modeled costs and the governing specification. Their implications also differ across deployment routes. The certified-offline route depends on its model and uncertainty assumptions, whereas CHECK can provide protection through valid fresh policy-cost evaluation under its sampling assumptions. Evaluation within the same compiled model does not establish protection against errors in that model’s representation of the governing specification. The boundary between these guarantees and broader deployment claims should be stated more explicitly.

Questions for authors
  1. Relationship between semantic coverage and the deployment guarantee. Can the authors establish how the ARTEMIS generation and retention statistics bound the specific (deltagen(delta_{\mathrm{gen}}) and (δsem\delta_{\mathrm{sem}}) failure events in Theorem 4.4? If the evaluation supports only coverage of at least one expert-plausible interpretation, how should the associated empirical safety claims be qualified

  2. Incremental benefit over native multi-constraint methods. Under matched data and tuning budgets, do the benefits persist with alternative logging policies, including mixtures balanced across interpretations? Can the authors separate improvements attributable to avoiding the pointwise-maximum surrogate from improvements in learning or computational efficiency? An assessment incorporating CHECK rejection and fallback behavior would further clarify the realized deployment benefit.

  3. Computational scope of certification. How do certification runtime and memory requirements scale with the state space and the number of interpretations remaining after pruning? Which parts of the continuous-control evaluation exercise the sufficiency test itself? If abstraction or function approximation is intended as a route to broader applicability, which approximation errors can be incorporated into a sound certificate?

  4. Model mismatch and deployment evaluation. Which abstraction, transition, and cost-specification errors are covered by the certified-offline and CHECK-based routes, respectively? What assumptions on the source and validity of fresh evaluation data are needed for CHECK to retain its guarantee when the model used during training or certification is inaccurate?

3 comments
AuthorsOpenReview ↗

Clarifications, Additional Experiments, and Scalability Results

Thank you for the detailed review. We address all four concerns.

W1/Q1 — Governing-reading coverage

ARTEMIS's 73.1% is retention of some expert-plausible reading (Table 21), while Theorem 4.4 concerns the governing reading ψ†\psi^\dagger. We measured retention conditional on generation against two known references.

Reference populationRetention
Monitoring rules, gold parse (n=111n=111)94.6%, 95% CI [91.2, 97.6]
ARTEMIS, unique expert formalization (n=28n=28)96.4% (27/28)

The monitoring interval is a cluster bootstrap, and a stricter audit gives 91.9%. The 90% guarantee also assumes exchangeability (Appendix C.1.4). Generation of the governing reading on grammar-reachable rules is 35.2% [27.0, 43.5].

We do not multiply these terms into an unconditional deployment probability, because they come from different campaigns. Where no reading is certifiable, ARROW retains the full constraint set, with deployment separately verified (§3.3).

W2/Q2 — Baselines, logging, and deployed benefit

Baselines. Table 2's 15.0–37.3-point gains are measured against the pointwise-maximum single-cost surrogate, not against native vector methods. Table 18 reports zero median differences for several native methods and +3.5–5.0 points for vector CPQ on 9/9 instances. Theorem 4.1(c) gives equal exact optimal values, verified on 9/9 agent-service and 22/22 real-rule instances. Table 17 has the structural-versus-learned split.

Balanced logging. We reran all six learners under 0.7/K0.7/K per reading-optimal policy plus 0.3 uniform, with matched instances, seeds and budgets. Realized costs changed in 78.3% of 3,600 paired cells. Median gains at n=20,000n=20{,}000 follow.

LearnerGain
BCQ-FQI / O3SRL15.0 → 15.0 (both)
PID / CPQ / CAPS22.0 → 21.7 / 20.7 → 20.7 / 20.2 → 20.2
Neural CQL37.3 → 2.3

All five tabular learners stay positive on 9/9 (max median shift 0.36 points). CQL stays positive on 5/9, and its selected-policy safe rate falls 98.9% → 60.0%. This is finite-data sensitivity to the logging distribution, leaving the sufficiency certificate unchanged.

CHECK/fallback under both logs. On 540 cells per log we checked the selected policy, then a combined-cost fallback, else abstained. The budget δev=0.05\delta_{\mathrm{ev}}=0.05 is split across both policies and their readings, extending the submitted CHECK-abstain route (§3.3).

OutcomeOriginalBalanced
Selected accepted / fallback80.9% / 19.1%79.3% / 20.7%
Abstentions / unsafe deployments observed0 / 00 / 0
Deployed return / VUV_U0.86410.8479
Combined-cost baseline / VUV_U0.68150.6822
Return gain+18.26 pp+16.57 pp

Across 1,080 cells neither event was observed. CQL alone gains 11.1 points under balanced logging after CHECK/fallback. The formal guarantee stays conditional.

W3/Q3 — Scalability

Exact finite or compiled-MDP testing uses one value LP and up to K−1K-1 face LPs per candidate, so the worst case is O(K2)O(K^2) face programs. Constructed positive-control timings follow.

StatesKKFace LPsTimeMemory
191,0009994.5 s0.07 GB
2441,00099910.9 s0.08 GB
2,9171009979 s0.54 GB
32,8061091,231 s48.3 GB

Those controls plant a dominant reading to bound cost. On 17 real compiled rules none has a pointwise-dominant reading, yet 16/17 certify (median 1.6 ms). Sufficiency there follows from the geometry of the near-optimal policy set, so a dominance check would certify none of them. Product-state growth is a bottleneck (Appendix B.3). Memory usage reflects dense LP storage, which sparse implementations could reduce. Table 19 further demonstrates compatibility with five released offline safe-RL learners.

W4/Q4 — Mismatch and guarantees

  • Certified-offline. Covers transition uncertainty inside the stated confidence set, relying on the compiled model being faithful.
  • CHECK. Evaluates retained costs on fresh deployment-system episodes without relying on estimated dynamics, capturing mismatch reflected in those measured costs.

Neither route repairs a mismatch between an executable cost and the intended instruction. Theorem 4.4 separately accounts for generation and retention of the governing reading, assuming its cost faithfully represents that instruction.

Revision plan. We will add the balanced-logging and checked-deployment results, the scaling study, and clearer deployment assumptions.

We respectfully note that all three aspect ratings are 2, while the overall score is 1. Given your assessment that the soundness concerns are minor or fixable, we hope this additional evidence warrants reconsideration of the overall score.

Thanks for your response. The new data confirm my primary concerns regarding the paper's practical inapplicability (e.g., requiring 48.3 GB of memory for merely 32806 states) and fragile empirical gains. Moreover, it leaves me confused about the fundamental definition of a "reading" and how it maps to the proposed framework, as argued by Reviewer 7MwQ. I also find this paper hard to follow in its key ideas.

Thank you for the follow-up. We respectfully disagree that the evidence supports practical inapplicability or fragile gains.

The 48.25 GB is an implementation figure. The caption attributes it to dense storage (p. 52, line 2754), where two arrays of ∣S∣2∣A∣|S|^2|A| entries cost 25.8 GB each at 32,806 states. Holding each transition as one successor index, the same ten programs at the same 32,806 states and K=10K=10 peak at 0.227 GB in 190 s. That is over 200 times less memory. Both formulations return identical margins at three smaller sizes. The 17 real compiled rules run the whole test at a median of 1.6 ms and 58 MB.

The alternatives carry their own costs. These differ in computation, return, and safety. The following results come from the separate agent-service and KK-scaling studies.

AlternativeIts cost
Protect every reading3.18 s against 0.11 s per vector-CPQ run at K=32K=32
Merge the readings into one worst-case costUp to 6.91% of achievable return (p. 46)
Deploy an uncertified readingUnsafe on 1,435 of 1,440 runs
Certify0.44 s once, yielding a 5.8x end-to-end speedup on the first run

Fragility. All five tabular learners retain positive gains on all 9 certified instances under balanced logging. Neural CQL is sensitive, which we explicitly report. After CHECK and fallback, deployed return improves by 18.26 percentage points under original logging and 16.57 under balanced logging, with no unsafe deployments observed across 1,080 cells.

A reading. We have already defined this explicitly, both informally (p. 1, lines 42–43) and mathematically (p. 3, lines 134–155). We also clarified the mapping from written rules to executable costs in our responses to you and Reviewer 7MwQ. Please specify which part of the definition or its use in the framework remains unclear.

Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Summary of Changes

Dear Area Chair and Reviewers,

We have uploaded a revised PDF of our manuscript. The changes are summarized by reviewer below, with line numbers referring to the revised PDF. The mathematical formulation, the method, the theoretical guarantees and the previously reported numerical results remain unchanged. All section, theorem, table and figure numbers are preserved.

The revision also clarifies the mathematical representation of a reading and its relationship to temporal-logic methods. Additional experiments appear in Appendix E (pp. 51–54, Tables 22–28). Theorem 4.2 includes a notation clarification, and Theorem 4.4 now cross-references the earlier definition of DEPLOY. Line numbers in our earlier replies may have shifted. The cited sections and tables remain unchanged.

Reviewer K7dn

PointMain changes
W1/Q1Theoretical bound distinguished from empirical rates (§4.4, line 355). Governing-reading generation and retention evaluated on two reference populations (Table 25).
W2/Q2Balanced logging evaluated across six learners (Table 22). CHECK with a combined-cost fallback evaluated under both logging policies (Table 23).
W3/Q3Certification scaling with state and candidate counts (Table 24). Sufficiency certified on 16 of 17 non-dominant compiled rules (Table 26).
W4/Q4Coverage and limitations of each deployment route clarified (§6, line 529). Fresh-data assumptions for CHECK explained (§C.1.4, line 2052).

Reviewer UMH2

PointMain changes
W1Certification distinguished from planning under every constraint (Appendix A, line 997). Sufficiency certified on 16 of 17 non-dominant compiled rules (Table 26).
W2Native multi-constraint comparisons added to the main discussion (§5.3, line 445; Table 18). Zero optimum gap verified on every certified instance (Table 27).
W3Theorem 4.2 identified as the main result and Theorem 4.1 as the criterion it certifies (§4, line 278; §1, line 96).

Reviewer 7MwQ

PointMain changes
MainClarification motivated by the possibility of stopping before identifying the intended reading (§1, line 57). Among 87 candidate sets, 26 need no question, and clarification strategies are evaluated with paired tests (§5.2, line 407; Table 28).
Formalism 1–3The mathematical representation is now explicit. A reading ψ\psi identifies a candidate interpretation, and ARROW takes its associated nonnegative state-action cost cψc_\psi as input (§2.1, lines 149–153). Costs may be specified directly or compiled from parsed rules or suitable temporal-logic specifications.
LTL follow-upThe existing discussion of automaton composition has been expanded to cite Li et al. (IROS 2017) and explain the distinction between temporal-logic rewards and ARROW's safety costs (Appendix A, lines 1051–1060). Our ARTEMIS study already uses LTL formulas as candidate readings (§5.4, line 480). Only 12.6% of the 182 requirements reduce to one reading after redundancy removal, with a median of four retained (§5.4, lines 487–488).
Experiments 4–5Table 3 explicitly identifies candidates as temporal-logic formulas in its caption and column header (§5.4, line 491; Table 3).
Minor 1–4Ambiguity qualified (§1, line 36), applicability scope stated (§6, line 526), two budgets distinguished (§3.3, line 260), and subscript convention explained (§2.1, line 161).
Minor 5–8DEPLOY defined earlier (§3.3, line 268), predicted clarification answers explained (§3.2, line 243), kl\mathrm{kl} distinguished from KL\mathrm{KL} (§4.2, line 323), and all four experimental motivation paragraphs revised (§5.1–§5.4).

Thank you for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.