- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In constraint-based RL, users may express their constraints and requirements in Natural language (NL). NL can be ambiguous and we may get multiple readings from it. This paper introduces ARROW a method that decides whether one reading can replace the whole candidate set or not. If this reading does not exist, it asks question to finally find one. The method tested on different aspect to show improvement and effectiveness in constraint-based RL.
Critical strengths
1- The like and fully agree that NL is vague and we need a more rigor approach to guide RL agents, so I like the main motivation of the paper.
2- In sec.3, the process of how we decide one reading is sufficient or not seems effective and smart.
Critical weaknesses
I thank the authors for their hard work, however I believe there are multiple issues regarding their submission which I split them into two main categories Formalism and Writing and Experiments.
Formalism and Writing
1- What are readings, are they logical formulas, and how we get the set of all readings from the NL input?
2- An important concern is, in line 70, paper mentions that ARROW take place before training, so how we get set of policies ? because there is no training take place that we have access to the policies.
3- If the dynamics are known this will limit the usage of this work only to model-based RL. And if we know the dynamics, we can easily generate different formulas using NL-to-LTL and then start from the most expressive one, by producing the composition of MDP and buchi automata and then check the feasible paths. If no feasible paths found we can try the less expressive ones, and continue till we find one formula that has a feasible path.
Experiments
4- What are the benchmarks used for the experiments, there is no context of what exactly used? For an example, what is the context of agent-service benchmark? What are the objectives? Why 9 out of 36?
5- The baseline methods used in Tab.3 translate a NL to LTL formula, ARROW’s objective is how one reading can replace all others, the output of ARROW is a reading (which I am not sure what is that) but these methods are LTL formulas, so what exactly was compared with these methods since they have different outputs.
Minor Issues
1- In line 36 Written safety rules are ambiguous is too strong, we may express rules in NL in a formal way that we get only one reading from it.
2- The way the MDP is defined, ARROW cant be applied in to continuous environments, if this is true, authors should mention that in the limitations.
3- What is ?
4- sometimes gets a reading or a set , this is inconsistent. Write it as .
5- What is DEPLOY ?
6- What is going to be predicted in lines 230-231?
7- kl in 308 should be KL.
8- The why paragraphs dont really explain why.
Questions for authors
I like the idea that NL is vague and we may get different understandings from a single NL constraint. However, imagine we get a NL constraint from the user and we extract different readings (for example LTL formulas), why not from the beginning we ask the users (we know we have access to the users since in ARROW we will ask questions from them if we dont find a good reading) that please clarify your request since we are getting multiple reading from this. I think the motivation of the paper lacks something HUGE that I am asking this (maybe basic) question. I would like the authors to explain that what was their intention of solving this problem and what is missing that I am proposing this trivial approach to solve this problem?
11 comments
Thank the authors for their rebuttal.
-
Your rebuttal make me more confused in 1 you are saying is plausible executable interpretation of a written safety rule in 5 you are saying it is executable temporal-logic formula, these are contradicting. First of all what is a executable temporal-logic formula. Formula is a logical expression it is not program that we can execute that.
-
In ICLR, authors can revise their paper for rebuttal, so why not revising the issues right now, and then the reviewer can really see the revised versions.
Thank you for the follow-up.
1. What Table 3 compares. Your Question 5 asked how ARROW's readings compare with the LTL formulas the baselines produce. All candidate readings in that study are represented as LTL formulas, so the comparison uses one representation and one criterion. Each translator row is scored on its first sample, with the parenthesized column on any sampled translation. Score-Keep filters the pooled candidates and measures whether a plausible interpretation survives. The pooled rows draw on the union of all archives at 190 samples per requirement, with no new model calls. Matching uses exact LTL equivalence (Appendix B.3, lines 1271–1292).
How readings are represented. A reading is one interpretation of a written safety rule (§1, lines 42–43). In constrained RL it determines a safety cost (§2.1, lines 129–133). Each decision model charges its own quantity, and Table 6 lists the four. In the monitoring model a threshold-duration reading compiles to a duration counter in the state, and the cost is evaluated at each timestep (Appendix B.2, lines 1020–1044). Table 3 evaluates candidate interpretations, while the decision-model studies evaluate certification.
What executable means. Executable temporal logic is established terminology in formal methods, surveyed under that name by Fisher (The Knowledge Engineering Review, 1996). The formula is the specification and the executed object is produced from it. One established mechanism transforms a specification into rules of the form "if the past satisfies A then do B", and those rules are what execute. This is consistent with §1 (line 37), where we explain that a written rule can be made executable in several ways. Since Table 3 scores formulas by equivalence, candidate temporal-logic formula is the more precise term there and we will use it.
2. The revised manuscript
We posted the responses first so that the new evidence was available as early as possible. We are now incorporating all the reviewers' feedback into the manuscript and will upload a revised PDF soon.
Thank you for your time.
Clarifying ARROW's Motivation, New Experiments, and Formalism
Thank you for the detailed review. We address your questions below.
Why not ask the operator to clarify immediately?
Asking the operator is valid, and ARROW does so when needed (§3.2). Identifying the intended interpretation seeks semantic identification. ARROW checks whether one training constraint already protects every feasible near-optimal policy under all remaining readings. If so, training proceeds under that constraint and the learned policy is verified or certified before deployment (§3.3). Otherwise, ARROW asks targeted questions about distinctions preventing certification.
This distinction matters empirically. §5.2 shows that 26/87 ambiguous monitoring sets need no question. Among the other 61, 56 reach sufficiency before the governing reading is identified. After four questions, ARROW reaches 99% sufficiency while balanced splitting achieves 37% full identification. Because those measure different outcomes, we ran a new paired comparison on one metric, the ambiguity price recovered. It covers 82 positive-price sets with bootstrap resampling over 77 alert families. Five zero-price sets were excluded because they had no recoverable price.
Differences use unrounded values. All intervals exclude zero, supported by paired sign and family-level permutation tests.
Formalism and experiments
Minor points
We hope these answers address your concerns.
Revision plan. We will sharpen the motivation, add the new paired comparison results and implement the presentation changes above.