acceptodds
Under review as a conference paper at ICLR 2027

Feedback Makes Perfect: A Closed-Loop Framework for NL-to-STL Translation

Abstract

Signal Temporal Logic (STL) enables rigorous verification and control of cyber-physical systems, but writing correct specifications requires expertise that most requirement holders lack. Large language models can translate natural-language (NL) requirements into STL, yet stronger translators alone approach an accuracy ceiling. We argue that this ceiling stems from how the task is posed: one-shot, open-loop translation is somewhat ill-defined. Natural language is ambiguous, and, more fundamentally, what a person writes may not always be what they intend, so the target specification is not fully contained in the input text. We therefore reformulate NL-to-STL translation as a closed-loop feedback process. Each generated formula is translated back into natural language for the user to check, and natural-language corrections drive revision until the user accepts the specification. Users never read or write formal syntax. This framework rests on an asymmetry familiar from feedback control theory. The forward path, from ambiguous language to formal logic, is hard and error-prone. The feedback path, from structured STL back to language, can be made highly precise, and a precise feedback path lets an imprecise forward path achieve precise closed-loop behavior. Experiments on 500 expert-authored requirements and seven LLMs support this view. Back-translated explanations agree with expert judgments in 99.5% of cases. Closed-loop refinement raises strong models from about 89% open-loop accuracy to 98.0–99.2%, and yields gains of over 30 percentage points for weaker models (e.g., 17.6%48.0%). Ablations show these gains come from the semantic content of the feedback rather than from repeated attempts. An expert audit and a 280-session user study further confirm the reliability of the loop. We also identify a capability threshold above which feedback no longer helps.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Human-in-the-Loop Learning, Formal Methods, Natural Language Processing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z8ezOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a closed-loop framework for natural language to Signal Temporal Logic (NL-to-STL) conversion. The authors argue that natural language may be ambiguous or incomplete, so one-shot translation alone — even with a very capable model — cannot guarantee that the generated STL matches the user's true intent. The framework paraphrases the generated STL back into natural language, which the user then checks, accepts, or requests modifications to, and a new formula is generated accordingly. The user does not need to read or write STL. The authors evaluate the method on 500 expert-written requirements and seven models: the back-translation achieves a 99.5% agreement rate with expert judgment; the closed-loop procedure improves the conversion accuracy of multiple models, and the reliability of the feedback loop is evaluated through expert review and 280 user–model interactions.

Critical strengths
  • It re-examines the task formulation. The paper points out that requirement texts may be insufficient to uniquely determine the target specification, so some errors cannot be resolved by improving the translation model's capability alone. Reframing the task as a closed-loop confirmation involving the user addresses a practical problem in formal requirements engineering.
  • The interaction mode is friendly to non-experts. Users check the meaning of the STL and propose modifications in natural language, without needing to master the formal grammar. This design has the potential to lower the barrier between engineers and formal tools.
  • The results show that closed-loop feedback is effective across multiple models. The authors report accuracy improvements for most models, with substantial improvements for some weaker models. Informative feedback outperforms the control of simply asking the model to retry, which also supports the claim that the feedback content itself contributes to the correction.
  • Feedback reliability is validated from multiple angles. The paper evaluates not only the models' translation results, but also reports the agreement rate between back-translation and expert judgment, expert review of automatically accepted results, and real user interaction experiments. This is closer to the actual usage workflow than reporting accuracy only on static test sets.
Critical weaknesses
  • User acceptance is still not a proof of formal correctness. What users see is a natural language paraphrase of the STL; if the paraphrase omits or obscures certain logical details, users may accept a formula that does not actually match their intent. The paper also acknowledges that the accepted result depends on the accuracy of the back-translation.
  • The interaction cost varies greatly with model capability. For weaker models, the closed-loop accuracy improvement is limited but requires multiple rounds of feedback; for example, the paper reports that Qwen2.5-0.5B achieves a closed-loop accuracy of only 7.8% with an average of 8.34 feedback rounds. The practicality of the closed-loop approach may therefore depend on the model used and on how many rounds of interaction users are willing to invest.
  • The coverage of the real-user evaluation is limited. The user study includes 40 requirements and 280 model–requirement interactions. This provides valuable preliminary evidence, but it is not yet sufficient to indicate how the method performs across more domains, different user groups, and more complex requirements.
  • The success criteria of the closed-loop evaluation still deserve further clarification. The paper reports accuracy, acceptance rate, and the number of user feedback rounds, but these metrics respectively reflect the consistency between the formula and the reference, whether users accept it, and interaction cost; they may not fully measure whether the requirement was accurately understood and whether the final specification is suitable for downstream verification or control.
Questions for authors
  • Regarding the reliability of user acceptance: How do the authors ensure that the STL-to-natural-language back-translation completely preserves the logical relations, temporal ranges, thresholds, and units in the formula? Is there a more systematic analysis of cases where users accepted formulas that still did not match the requirement?
  • Regarding interaction cost: Can the authors report the distribution of feedback rounds required by different models to reach a target accuracy, and discuss how requirements that remain unresolved after many rounds would be handled in real deployment? For weaker models with many feedback rounds, is there a mechanism for interaction budgets or escalation to experts?
  • Regarding the representativeness of the user study: Do the 40 requirements and the participants cover multiple application domains and users with different levels of formal methods experience? Can the results on acceptance rate and number of feedback rounds be reproduced across more requirements, users, and task scenarios?
  • Regarding the evaluation criteria: Besides closed-loop accuracy and user acceptance rate, do the authors evaluate the performance of the final STL in downstream verification or control tasks? Can these evaluations distinguish between "matching the reference formula" and "genuinely matching user intent in the target application"?
3 comments
AuthorsOpenReview ↗

Response to Reviewer z8ez: Acceptance Reliability, Interaction Cost, and Real-User Evaluation

Thank you for your careful review and detailed feedback.

W1 and Q1. Reliability of user acceptance

To assess whether users made reliable acceptance decisions, we independently evaluated two links: whether the backward description faithfully represented the underlying STL formula, and whether the accepted STL formula itself was semantically correct.

i) Formula-to-language fidelity. Each candidate is represented as a JSON-encoded STL syntax tree. A deterministic renderer recursively converts every predicate, comparison direction, threshold, unit, Boolean relation, temporal operator, interval, and nesting relation into controlled language. It therefore does not ask an LLM to infer the formula’s meaning. GPT-5.6-Sol is used only to improve readability, and the system returns the deterministic description whenever the polished output fails the consistency check. In our expert audit, the deterministic descriptions preserved the complete source semantics in 1,000/1,000 cases, while the polished descriptions were correct in 996/1,000 cases. The deterministic renderer also preserved the relevant semantic distinction in all 402/402 minimal contrastive pairs.

ii) Correctness of accepted formulas. Participants made all acceptance decisions and provided all corrective feedback themselves. After the study, two blinded STL experts examined the underlying terminal formulas from all 280 sessions. For each session, they evaluated the complete formula semantics against a separate intent record derived from the participant’s initial requirement and subsequent corrections, without seeing the model identity or the participant’s acceptance decision. All 202 participant-accepted formulas were judged semantically correct (202/202, 100.0%, 95% CI: [98.1%, 100.0%]), with Cohen’s κ=1.00\kappa=1.00. Thus, we observed no case in which a participant accepted an incorrect underlying formula. Separately, experts confirmed 629/634 (99.2%) outputs accepted by the automated protocol; the remaining five are explicitly reported as false acceptances.

W2 and Q2. Interaction cost and unresolved cases

Figure 3(b) already reports the cumulative closed-loop accuracy of all seven translators after 1, 2, 3, 5, and 10 forward calls. The first call produces the open-loop candidate, so kk forward calls correspond to at most k−1k-1 feedback events. Equivalently, each curve is the cumulative distribution of the number of calls required for successful refinement: the increase between two points is the fraction of cases resolved during that interval, and the fraction remaining below 100% after ten calls is unresolved under the fixed interaction budget. The results show that additional interaction is useful only when the translator can apply corrective feedback. Qwen2.5-0.5B improves only from 6.8% to 7.8% despite requiring 8.34 feedback events per sample, whereas the locally deployable Qwen3.8-27B reaches 98.0% closed-loop accuracy with only 0.41 feedback events per sample. Thus, our results do not suggest that users should continue interacting indefinitely with an under-capable translator; selecting a sufficiently capable local model is substantially more effective. The evaluated protocol already enforces a fixed maximum budget of 10 forward calls and returns unresolved if no candidate is accepted within that budget, rather than delivering an unconfirmed formula. The current prototype does not automatically switch models or escalate cases to experts. In deployment, an unresolved result can serve as a handoff point for routing the requirement to a stronger translator or a formal-methods expert, but this routing policy is outside the evaluated protocol.

AuthorsOpenReview ↗

Response to Reviewer z8ez (Part 2)

W3 and Q3. Scope and representativeness of the real-user evaluation

The real-user study involved four participants with different professional backgrounds and levels of formal-methods experience. The group included students as well as researchers and engineers working in autonomous driving and robotics. Two participants had prior exposure to formal methods, while the other two had no formal-methods background. Each participant independently contributed 10 natural-language requirements and interacted with all seven translators on the same requirements, yielding 40 distinct requirements and 4×10×7=2804\times10\times7=\mathbf{280} model–requirement sessions. The requirements were authored freely rather than sampled under a fixed per-domain quota, and included autonomous-driving, robotic-system, and other CPS scenarios. Using the same requirement with all seven models controls for differences in requirement difficulty when comparing acceptance rates and interaction costs. The study produced 1,010 human-written NL inputs, including 730 corrective feedback messages. On the same 40-requirement subset, the real-user protocol produced 202/280 (72.1%) accepted and independently confirmed outputs, while the automated protocol produced 204/280 (72.9%) expert-confirmed correct outputs, supporting the representativeness of the automated evaluation on this subset. However, we do not claim that four participants establish reproducibility across arbitrary users or application domains. Broader task coverage is evaluated separately using the 500-case benchmark, while replication with more users, requirements, and deployed task scenarios remains an important direction for a larger-scale study. The revised manuscript now reports the participant composition, experience levels, per-participant workload, scope of the participant-authored requirements, and same-subset comparison.

W4 and Q4. Success criteria and downstream use

i) What counts as success. These metrics answer different questions: formula accuracy measures agreement with the benchmark reference, acceptance or convergence measures whether the interaction terminates within the prescribed budget, and the number of feedback rounds measures interaction cost. Operationally, a run terminates when the user or automated panel accepts a candidate; we report this as acceptance or convergence, not as ground-truth correctness. Correctness is evaluated separately through benchmark references, post-study expert review of all 280 human-study sessions, and the stratified audit of 634 automated acceptances. In the real-user study, two blinded STL experts confirmed all 202 participant-accepted formulas as semantically correct (202/202, 100.0%, 95% CI: [98.1%, 100.0%]). Separately, experts confirmed 629/634 (99.2%) outputs in the stratified audit of automated acceptances. Thus, the ACCEPT decision is an operational stopping decision rather than a correctness label.

ii) Downstream use. The closed-loop protocol does not change the artifact delivered to downstream systems. Verification, runtime-monitoring, and controller-synthesis tools take an STL formula over declared signals as input, and our framework returns exactly the same schema-valid JSON STL syntax tree as the underlying open-loop translator. A deterministic serializer converts this tree into the documented STL dialect. For verification or monitoring, the resulting formula can be evaluated over system traces; for controller synthesis, it can be supplied as a temporal constraint or objective. Therefore, any parser or adapter already used with the open-loop translator can be reused without modification. The role of our closed loop is to improve and confirm the formula’s alignment with user intent before this downstream step. It does not replace environment-specific predicate grounding, system modeling, feasibility analysis, or tool-specific validation. The revised manuscript now clarifies both the semantic success criterion and this downstream system boundary.

Changes Made in the Revised Manuscript

Revisions made specifically in response to this review are highlighted in teal, while revisions shared across reviewers are highlighted in orange. The revised manuscript distinguishes operational acceptance from semantic correctness, reports the expert audits of human and automated acceptances, clarifies interaction cost and unresolved cases, expands the description of the user study and same-subset comparison, and explains how the refined STL output connects to downstream tools.

AuthorsOpenReview ↗

Looking forward to the reviewer's reply!

Dear Reviewer z8ez,

Thank you again for your thoughtful review and valuable suggestions. We have revised the manuscript and provided further evidence concerning acceptance reliability, interaction cost, unresolved cases, and the scope and representativeness of the real-user evaluation.

As the discussion period draws to a close, we would be most grateful if you could let us know whether our response and revisions have adequately addressed your concerns. Should any point remain unclear, we would greatly appreciate the opportunity to provide further clarification.

Sincerely,

The Authors of Submission 25081

Reviewer_RsaPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Formal specifications in Signal Temporal Logic (STL) play an important role in the verification, runtime monitoring, and controller synthesis for safety-critical cyber-physical systems. However, domain engineers and end users rarely possess the capabilities to correctly formalize the requirements they hold for such a system as a precise logical formula. While Large Language Models (LLMs) have shown great success in translating requirements given in natural language (NL) to STL, "traditional" one-shot translation approaches suffer from the problem that non-experts are unable to judge whether a produced formula accurately captures their intent. The paper aims to close this gap by reformulating NL-to-STL as a closed-loop feedback process rather than a one-shot task. After a user provides an initial description of their desired requirement, a forward translator produces an STL formula ϕ\phi that gets translated back to natural language by an independent backward translator. This backwards-translated description is then presented to the user, who can either confirm that it matches their intent (in which case ϕ\phi becomes the output of the process) or trigger a revision of the formula by describing the source of mismatch. The authors demonstrate the effectiveness of their framework in a fully automated benchmark study using a set of 500 expert-authored requirements, seven Qwen models of varying size, and an LLM-as-Judge for making the acceptance decision and providing the source of discrepancy. Furthermore, they conducted a feedback ablation study to demonstrate the effect of feedback compared to simple retrying, an expert audit to confirm the reliability of the automated semantic acceptance, and a human study in which participants interacted with the tool.

Critical strengths
  • The main idea of the presented framework - translating STL formulas back to NL in order to support users in writing formal specifications - is easy to understand, well motivated, and the authors differentiate their work clearly from existing literature.
  • The paper tackles a core challenge that still remains in the translation from NL to formal logics like STL: Ensuring that the produced formula actually matches the user's intent. The paper presents a first step towards solving this challenge and provides an extensive evaluation of the presented approach, including an expert audit and a human user study. Frameworks like this can have a large impact on the wider adaptation of formal methods and help to guarantee the safety and correctness of cyber-physical systems.
Critical weaknesses
  • To the best of my knowledge, the concept of translating a formal specification back to natural language was not yet used in NL-to-STL as a mechanism to support a feedback loop that requires no formal education and uses exclusively natural language. However, the idea of backwards translation is not entirely new and can only be a first step towards ensuring that a specification produced by an LLM matches a user's intent. Natural language is highly ambiguous, which is one of the reasons why NL-to-STL is such a hard task. When a precise, formal specification is translated back to natural language, it is not obvious that the resulting description is easy to understand and free of any ambiguity. This is demonstrated by one example from the appendix of the paper: the reach and maintain example (in line 1090) states the requirement that "The drone must reach the cruise altitude band within 40 seconds and keep the altitude between 78 and 82m for the following 5 minutes." This requirement is highly ambiguous and could be interpreted in at least three different ways (already assuming that the cruise altitude is 78-82m): (i) a drone must enter the band within the first 40s and upon entering must maintain there for 5 min; (ii) some time within the first 40s the drone enters the band and maintains altitude for 5min (this allows the drone to exit and re-enter within the first 40s before staying there for 5 min); (iii) at some point within the first 40s the drone is inside the band and it maintains inside the band for the 5min following the initial 40s. A formula fulfilling any of the three interpretations could easily be translated back to the original requirement. This raises questions about the soundness of the framework in real-world use-cases.

  • While the main idea of the framework is easy to understand and well motivated, many technical details are missing in the paper. Some are stated in the appendix, but others remain unclear. Examples include "how is the formula translated back to natural language" (the appendix shows a prompt in this direction; thus I assume this is also done by an LLM, but it is unclear which one), "which model is used as an LLM-as-Judge in the closed-loop protocol" (as far as I understood, this is actually a majority vote of three distinct LLMs), or "how is the localized discrepancy determined in the closed-loop protocol" (similar to the first question, a prompt hints at using the LLM-as-Judge mentioned in the protocol). These details should be made clearer and already be addressed in the main paper. However, this problem can be addressed in a revision for the camera-ready version.

  • Assuming the back translation and the determination of the local discrepancy are done by an LLM-based system, one core problem with this approach is that it can not guarantee the correctness of the translation. While the authors claim that the back-translation is "far easier" and "can be made highly precise", they do not provide clear evidence supporting this claim. Since NL and logical formalisms like STL describe properties from two completely different abstraction levels, many of the reasons why NL-to-STL is such a hard taskalso applies in the backwards direction. First, relying on an LLM for the translation can not guarantee correctness. Second, translating to NL again suffers from the problem of ambiguity as two semantically different STL formulas may be translated to the same NL description. Finally, for a non-expert user to effectively use the framework, the translation must be easy to understand, which can not be ensured when translating a concise and highly-nested logical formula to NL. As a consequence, a bad or even incorrect back translation can cause the whole framework to produce outputs that still miss a user's intent. To fully resolve this issue, the framework would need a mechanism that can ensure the correctness of the back translation (in which case it may still struggle with ambiguity of NL) or additional mechanisms helping the user to confirm that the produced formula matches their intent (including the ones the authors already mention in the future work part of the appendix). This poses concerns on the soundness and usefulness of the presented framework.

  • The authors provide an extensive evaluation of their approach, including a fully automated accuracy study, an expert audit, and a human user study. The numbers suggest that the approach works well in practice and that some of the problems mentioned above may not influence the framework too much. However, the evaluation raises two main questions: First, large parts of the evaluation rely on the requirement-specification pairs constructed by a set of experts and their judgement of the framework's performance. However, the paper doesn't mention any qualifications these experts possess (besides stating in the appendix that they passed an undisclosed qualification test). Providing the qualification test (as part of the appendix or supplementary material) or expanding on the experts' qualifications would help improve the validity of the results. Second, the human user study was conducted on only a subset of the full benchmark (which is understandable due to the size of the benchmark), and only human acceptance rates are communicated. With these participants presumably not being experts, the more interesting statistics to report would be how often the human users were able to come up with the correct formal specification while interacting with the tool (not how often they assumed they had the correct one). It would also help to see the comparison of their results and the results of the fully automated benchmark test on the same subset in order to get a better understanding of how representative the fully automated test is. Both of these issues may be addressed in the author rebuttal phase. If the experts possess sufficient expertise to accurately evaluate the performance of the framework and the results from the user study show that the framework indeed helps non-experts in writing correct formal specifications, I may be willing to improve my overall assessment.

  • Two of the paper's claims are not or only partially supported: The authors claim that "a precise feedback path substantially reduces dependence on translator capability". To achieve good closed-loop accuracy (80%+), one of the models is required that already achieves 80%+ open-loop accuracy. Weakening this claim to "a precise feedback path can substantially improve the performance of a naive NL-to-STL translator" is more in line with the paper's results. Furthermore, the claim that the paper "characterizes the capability threshold above which feedback no longer helps" seems to be unsupported. In contrast, the authors state in Appendix E (Line 1181) that "The current results [...], they do not establish a model-independent capability threshold"

To conclude, I evaluate the paper as a "weak reject".

The paper presents a first step towards solving the important problem and facilitating a wider application of formal methods in practice. Translating a formal specification back to natural language removes a user's burden to be fluent in logical formalisms but can't solve the problem entirely due to the high ambiguity of NL; this will require future work. The main idea is presented clearly, but some technical details are missing in the main paper and should not just be hinted at in the appendix. The authors present an extensive evaluation, and the numbers in the evaluation seem to show that the approach works very well in practice, aside from the open questions at the technical level.

Minor Comments that did not impact the score:

  • Page 4, line 185: this should go into the definition of Until in Eq. 3
  • Overloading of symbols: While it is usually clear from the context, introducing different symbols helps improve understanding
    • The symbol II: In the definition of STL, II is defined as an interval I=[a,b]I = [a,b] (line 173). In line 195, it is also used to denote the underlying human intent.
    • The variable xx: In the definition of STL, xix_i denotes a real-valued variable. In the context of NL-to-STL, it denotes the natural language expression of a certain iteration.
  • Line 196: It may make sense to denote the generated STL formula by Φx\Phi_x to emphasize that Φ\Phi was generated from xx
  • Line 204: Maybe, "Checking exact matching ..."
  • Line 269: Eq.5 or Equation 5
  • Line 1070: I assume there is a typo and a G\{[0,\infty]} is missing. Otherwise, I think the result is not the correct formalization of the requirement, as the "Whenever" component is missing.
  • The "Calibration" part in Table 2 is misleading. The appendix shows that D1/2D_{1/2} and B3B_3 are not directly comparable (different benchmark sets). This should be separated, and one should also not draw conclusions from this (e.g., line 453)
Questions for authors
  • What is the correct formula in the reach and maintain example (in line 1090)? It would also be interesting to see the back-translated description of ϕ0\phi_0; does the formula matches one of the three alternative views described above?
  • The experts audit 634 specifications accepted by the framework. The accuracy of the methods indicates that there should be a much larger number of specifications (just taking the four best models, there should be 4∗0.98∗500≈20004 * 0.98 * 500 \approx 2000). Are the missing ones duplicates, or how did you choose this subset of 634?
  • What qualifications do the experts have, and can you share the qualification test?
  • Does the informative feedback study use the same benchmark set as the closed-loop accuracy study? If yes, then there is a typo in Table 1 or 3, as they report different accuracies.
6 comments
AuthorsOpenReview ↗

Response to Reviewer RsaP: Backward-Translation Fidelity, Expert Qualifications, and Real-User Correctness

Thank you for your careful reading and constructive feedback.

W1(a) and Q1. Natural-language ambiguity and the reach-and-maintain example

i) Why ambiguity motivates the closed loop. We agree that one natural-language requirement may have several reasonable interpretations. However, the user may still know exactly which behavior they intend. Our framework does not assume that the first description determines a unique STL formula. Instead, it presents the semantics of the current candidate in natural language, allowing the user to confirm or correct the interpretation. The goal is not to make the initial requirement unambiguous, but to use feedback to reach an STL formula that reflects the user’s intended behavior. This requires the backward description to faithfully expose the meaning of the current formula, as evaluated in our response to W1(b), W2, and W3.

ii) The reach-and-maintain trace. Let b:=(78≤alt≤82)b := (78 \leq alt \leq 82). The Round-1 formula is F[0,40](b∧G[0,300]b)F_{[0,40]}(b \land G_{[0,300]}b). It states that there is a time within the first 40 seconds from which the drone remains inside the band for 300 seconds. The inner G[0,300]G_{[0,300]} is evaluated from the time selected by F[0,40]F_{[0,40]}, not from time 0. This formula matches the benchmark reference and corresponds to the reviewer’s interpretation (ii). Its backward description is: “At some time within the first 40 seconds, the drone is inside the altitude band and, from that same time, remains inside it for 300 seconds.” The final formula is ¬b U[0,40](b∧G[0,300]b)\neg b\ U_{[0,40]}(b \land G_{[0,300]}b). It adds the stronger condition that the selected time must be the first entry into the band and corresponds to interpretation (i). For completeness, interpretation (iii), which anchors the five-minute maintenance period at t=40t=40, corresponds to F[0,40]b∧G[40,340]bF_{[0,40]}b \land G_{[40,340]}b. The appropriate formula depends on the user’s intent; the benchmark reference adopts interpretation (ii), while the appendix terminal formula represents interpretation (i). This automated trace is not part of the real-user study, and ACCEPT records the decision of the automated acceptance mechanism rather than an independent correctness label. Here, the simulated feedback rejects the correct Round-1 formula and moves the system toward the stronger first-entry interpretation. The trace is therefore a false rejection followed by semantic drift, not evidence that the terminal formula is the unique interpretation.

W1(b), W2, and W3. Backward translation and reliability

Appendix B describes the overall backward interface and the three-model acceptance panel; we clarify the complete information flow here.

i) Backward translation. The forward translator directly outputs each candidate as a JSON-encoded STL syntax tree. The backward interface applies fixed recursive rules rather than asking an LLM to infer its meaning. For example, GG, FF, and UU are rendered as “always,” “eventually,” and “until”; predicates, comparison directions, thresholds, interval bounds, Boolean structure, and temporal nesting are preserved directly. Within the supported STL grammar, this first stage is lossless with respect to the candidate tree. It receives only that tree and cannot access the original requirement or reference STL. GPT-5.6-Sol is used only to improve readability, followed by a numerical consistency check; if that check fails, the deterministic description is used.

ii) Acceptance and localized feedback. In the automated experiments, DeepSeek-V4-Flash, Kimi-K3, and GLM-5.3 independently compare the backward description with the intent-side NL: the original requirement for well-specified cases and the expert-corrected NL for underspecified cases. The reference STL remains hidden. Each model checks thresholds, polarity, temporal bounds, operators, logical scope, structure, and coverage, and majority vote determines acceptance. When a majority finds a mismatch, the checklist localizes the discrepancy, and GPT-5.6-Sol converts a validated discrepancy into short corrective feedback. In the real-user study, participants make the acceptance decision and write the correction themselves.

iii) Fidelity evaluation. Two STL experts independently audited 1,000 formula-description pairs, including all 500 reference formulas and 500 distinct candidates from automated runs. The deterministic renderer preserved the complete source semantics in 1,000/1,000 cases, while the optionally polished descriptions were correct in 996/1,000 cases. Expert agreement was 99.7%, with Cohen’s κ=0.97\kappa=0.97. On all 402 minimal contrastive pairs, the corresponding results were 402/402 and 399/402. The page-22 change from “greater than” to “not equal to” was caused by optional polishing; the deterministic description was correct. This audit measures whether a description faithfully represents its source formula, while non-expert usability is evaluated separately in the real-user study.

AuthorsOpenReview ↗

Response to Reviewer RsaP (Part 2)

W4 and Q3. Expert qualifications and real-user correctness

We provide the requested details on the annotators and an independent correctness audit of the real-user sessions.

i) Expert qualifications and qualification exercise. The benchmark was developed by five contributors: one author and four external annotators. This explains why Appendix A counts five contributors, while the Acknowledgments thank four external experts. The annotators were master’s or doctoral researchers in Control Science and Engineering, with research experience in control and formal methods, and each had authored at least one peer-reviewed paper at an international venue. Before annotation, they reviewed STL syntax and semantics and completed a 50-item bidirectional multiple-choice qualification exercise designed by one author and a professor specializing in formal methods. The exercise covered both NL-to-STL and STL-to-NL mappings, with alternatives differing in Boolean structure, temporal operators, interval bounds, implication direction, or nesting. All four annotators answered all 50 questions correctly.

ii) Independent correctness audit of real-user outcomes. Participants made every accept-or-revise decision and provided all corrective feedback; experts did not participate in the interaction. After the study, two independent STL experts compared every participant-accepted formula with a separate record of the participant’s intended requirement. They were blinded to model identity, interaction round, and participant decisions, and agreed on all cases, with Cohen’s κ=1.00\kappa=1.00. All 202 participant-accepted formulas were semantically correct (202/202, 100.0%, 95% CI: [98.1%, 100.0%]), yielding a false-acceptance rate of 0/202. On the same 40 requirements and seven models, the automated protocol produced 204/280 (72.9%) expert-verified correct final formulas, closely matching the 202/280 (72.1%) accepted-and-confirmed outcomes with real users.

W5. Translator capability and saturation

i) An intermediate-capability model. The original seven-model suite covered a broad range of capabilities rather than densely sampling every open-loop accuracy interval, leaving a gap between Qwen2.5-7B-Instruct at 45.2% and Qwen3.6-Flash at 81.4%. We therefore evaluated Qwen3.5-35B-A3B on the same 500 requirements. Its open-loop accuracy is 312/500 (62.4%), while closed-loop refinement raises it to 471/500 (94.2%), a gain of 31.8 points, with 2.16 feedback events per sample. Thus, open-loop accuracy above 80% is not necessary for high closed-loop accuracy, although the translator must still be capable of applying corrective feedback.

ii) Task-dependent saturation. The original “capability threshold” wording referred to the saturation pattern in Figure 3(b), not a universal parameter-count or accuracy cutoff. The smallest model receives many corrections but improves little, intermediate models obtain the largest gains, and stronger models approach 98.0–99.2% within the first few rounds. Once performance is near the ceiling, additional rounds provide little benefit. This ceiling is task-dependent: an easier distribution may saturate at 100%, while a harder one may saturate lower. We therefore describe the result as capability-dependent gains and task-dependent saturation.

W6. Minor comments

Thank you for the careful reading and detailed suggestions. The revised manuscript incorporates the complete Until semantics into Eq. (3), resolves the overloaded notation, applies the suggested wording and typographical corrections, restores the missing global operator in the execution trace, and separates the non-comparable calibration results in Table 2 while removing the associated comparative claim.

Q2. Construction of the 634-item expert audit

The 634 cases form a stratified audit sample rather than the full set of accepted outputs. We sampled clean terminal acceptances separately for each of the seven translators, with at most 100 outputs per model and proportional coverage of the five domains. Models with fewer than 100 eligible outputs contributed all available cases, resulting in 634 cases rather than 700. Because the audit unit is a model–requirement output, the same requirement may appear for multiple translators, potentially with different terminal formulas. These are separate system outputs rather than accidental duplicates.

AuthorsOpenReview ↗

Response to Reviewer RsaP (Part 3)

Q4. Evaluation set for the informative-feedback ablation

We apologize that the manuscript did not make this distinction clear. Table 1 reports the main results on the full 500-case evaluation set, whereas the informative-versus-uninformative feedback ablation uses a fixed 200-case subset. We selected this subset to keep the cost of the paired, multi-round ablation manageable while preserving benchmark coverage. It contains 40 requirements from each of the five domains and was sampled using seed 42. Within the ablation, both feedback conditions use exactly the same requirements, models, acceptance protocol, and interaction budget; only the feedback content differs. For Qwen3.8-Max, two requests failed at the API layer and produced no valid candidate, so the submitted Max results were computed on the same 198 completed cases: 197/198 (99.5%) with informative feedback and 178/198 (89.9%) with generic retry. We report these denominators and the two infrastructure failures explicitly here. The different accuracies are therefore not typos. The revised manuscript now states the subset size, composition, sampling seed, and reason for using it.

Changes Made in the Revised Manuscript

Revisions made specifically in response to this review are highlighted in blue, while revisions shared across reviewers are highlighted in orange. The revised manuscript clarifies the backward-translation and acceptance pipelines, adds the fidelity and expert-validation results, explains annotator qualifications and the qualification exercise, corrects the reach-and-maintain and polishing-failure discussions, adds the intermediate-capability model, and addresses the technical and presentation issues noted in W6. Sincerely thank you again for your detailed review.

Reviewer_M1SzOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper separates a user's intention, its natural-language description, and the temporal-logic formula generated from that description. The generated formula may not express what the user meant. The proposed loop makes the formula's meaning visible in natural language so the user can correct it without editing formal syntax. The model uses those corrections in its next attempt. Most experiments simulate the user's checks and replies. Specific corrections help more than generic retries. I currently lean towards weak acceptance. My main concern is the missing comparison with a simple loop that generates, explains, critiques and revises formulas. I also want to know whether explanations help the checker find mistakes. The same models should compare each requirement with the formula directly and with its explanation; the paper currently changes both the models and the test cases. Experts should also check whether formulas accepted by participants match what they intended. A participant could accept an explanation even if the formula expresses something different. The existing expert audit covers the automated system, rather than the user study.

Critical strengths
  1. The models produce substantially more correct formulas when feedback explains the mistake than when it simply asks them to try again. The largest gains are for the 3B and 7B models.

  2. Experts checked a sample of formulas accepted by the system and found few errors. This gives an independent check on the results, since the translator and checker could otherwise agree on a wrong formula. The authors also report the errors and an uncertainty interval.

  3. Users can check and correct a formula through its explanation without editing STL. The system builds that explanation from the candidate formula alone and retains corrections for later attempts. The workflow handles both missing details in a requirement and mistakes in the translation.

Critical weaknesses
  1. The paper does not compare with a simple loop that generates a formula, explains it in ordinary language, checks for mistakes and tries again. The retry baseline does not ask the model to identify mistakes. I cannot tell what the proposed workflow adds without testing this alternative using the same models, access to clarification and computation budget.

  2. Take the same requirement and formula. Have the same checking models compare the requirement with the formula directly, and then with its explanation. This would show whether the explanation helps them find mistakes. The paper currently changes both the checking models and the test cases, so either difference could affect the result. The authors acknowledge this limitation, but the main text should make it clear.

  3. A participant could accept an explanation even if the underlying formula expresses something different. Experts should compare those formulas with what participants intended. The existing expert audit covers formulas accepted by the automated system, rather than those accepted in the user study. The conclusion should not say that feedback "ensures" a match with human intent.

Questions for authors
  1. How does the method compare with a simple loop instructed to generate a formula, explain it, check its meaning against the requirement and revise, using the same models, access to user clarification and computation budget?
  2. Does the explanation help the automated checker find mistakes? Please use the same requirements and candidate formulas, and have the same models compare each requirement with either the formula directly or its explanation. Keep the voting rule and computation budget the same.
  3. Do the formulas accepted by participants match what they intended? Please have independent experts check the initial and final formulas against a separate record of each participant's intended requirement. How much experience did participants have with logical formulas?
  4. Is STL essential to the method, or is it the application chosen for evaluation? Could the same approach support first-order fragments with mixed integer and real arithmetic or other combinations of theories? Please clarify which parts of the formula-to-language conversion and checking would need to change.
  5. What quantitative evidence supports the claim that STL-to-language translation is easier than language-to-STL translation? The 99.5% figure measures the complete acceptance system, while p.22 reports a back-translation that changes "greater than" to "not equal to". Is the backward translator deterministic, LLM-based or a combination? Please report separately how often its descriptions preserve the source formula's meaning.
9 comments
AuthorsOpenReview ↗

Response to Reviewer M1Sz: Self-Refinement Comparison, Checker Ablation, and Human-Intent Validation

Thank you for your thoughtful comments and valuable suggestions.

W1 and Q1. Comparison with self-refinement

Thank you for suggesting this comparison. We understand the requested baseline as a loop in which the same translator generates an STL formula, explains its meaning, checks the explanation against the requirement and available user clarification, and then revises the formula. If the translator identifies an ambiguity or missing intent information, it may ask the user a clarification question and incorporate the answer. However, unlike our protocol, the user is not shown the backward description of every candidate. The user therefore cannot correct an unintended interpretation unless the translator itself recognizes that clarification is needed.

i) Experimental design. We evaluate four settings on the same 200 requirements used in the feedback ablation, with 40 requirements from each domain and seed 42. Open loop performs a single translation without refinement. Generic retry uses the same backward interface and three-model majority verifier only to determine whether another attempt is required. When the verifier returns MISMATCH, all judge notes and localized discrepancies are discarded. The translator receives only the fixed instruction, “The previous specification is incorrect. Please try again,” together with the original requirement and previous candidate. It is not shown the backward description, is not asked to diagnose the error, and does not request additional user clarification. Self-refinement implements the reviewer-proposed generate–explain–critique–revise loop using the same translator. The translator determines by itself whether the candidate matches the requirement, what may be wrong, and whether user clarification is needed. Our protocol exposes the backward description of every candidate formula to the user, allowing the user to identify any mismatch with their intended behavior and provide an explicit localized correction.

All settings use the same translator models, decoding configuration, initial requirements, and intent-side information. Self-refinement and our protocol use the same user simulator when clarification is requested. The iterative settings use the same maximum of ten candidate-generation calls, the same per-call decoding limit, and the same maximum number of user interactions. We also record the actual end-to-end inference cost of each completed run, including all provider-reported input, output, and reasoning tokens used by the complete protocol.

ii) Independent correctness evaluation. The submitted Figure 3(a) reports outcomes produced under automated acceptance. For this comparison, we separately measure the semantic correctness of the delivered formulas. Two STL experts independently and blindly evaluate every terminal formula from all four settings against the intended requirement. They do not see the model, method, automated decision, or one another’s labels. Invalid outputs, unresolved runs, and accepted-but-incorrect formulas are counted as failures, and disagreements are adjudicated. The table below reports expert-verified accuracy, defined as the proportion of NL–STL pairs judged semantically consistent.

Expert-verified semantic accuracy on the shared 200-case subset

TranslatorOpen loopGeneric retrySelf-refinementOurs
Qwen2.5-3B-Instruct17.0%16.5%14.0%43.5%
Qwen2.5-7B-Instruct44.5%49.0%40.0%75.0%
Qwen3.8-27B85.5%87.0%82.0%97.0%
Qwen3.8-Max87.5%88.5%87.0%98.5%

The experts agreed on 98.9% of the judgments, with Cohen’s κ=0.96\kappa=0.96.

End-to-end inference cost for Qwen3.8-Max on the shared 200-case subset

MethodQwen3.8-Max avg. total tokens/case
Open loop4,300
Generic retry21,600
Self-refinement14,700
Ours23,900

Generic retry consumes a similar amount of end-to-end inference as our protocol, yet reaches only 88.5% expert-verified accuracy, compared with 98.5% for our protocol. This indicates that the gain is not explained by additional inference alone, but by the semantic content of the localized user feedback.

AuthorsOpenReview ↗

Response to Reviewer M1Sz (Part 2)

Following the former W1 and Q1. Comparison with self-refinement

iii) Results. Generic retry provides another generation opportunity but no direction for correction, producing only limited gains. Self-refinement does not consistently outperform generic retry. When the same translator must both generate a candidate and diagnose its own error, its critique often reproduces the interpretation that caused the original mistake. Moreover, it asks the user for clarification only when it recognizes an ambiguity; if it considers an unintended interpretation valid, the user is never given an opportunity to correct it.

Our protocol exceeds self-refinement by 29.5, 35.0, 15.0, and 11.5 percentage points across the four translators. The difference is not simply another retry or critique step. Every candidate interpretation is made visible to the user, who can identify a mismatch that the translator failed to recognize and provide information about the intended behavior. These results show that self-reflection cannot replace an external, content-bearing feedback channel when the intended semantics are absent from, or ambiguous in, the original requirement.

iv) Translation cost. We count every translator call that attempts to produce an STL candidate, including calls that return an invalid or unparseable output. Explanations, acceptance decisions, and user feedback are not counted as translations unless they lead to another candidate-generation call.

TranslatorOpen loopGeneric retrySelf-refinementOurs
Qwen2.5-3B-Instruct1.09.02.56.4
Qwen2.5-7B-Instruct1.06.52.33.8
Qwen3.8-27B1.02.11.61.4
Qwen3.8-Max1.01.81.41.2

Generic retry requires substantially more attempts, particularly for weaker translators, because it provides no direction for correction. Self-refinement appears inexpensive in terms of translation count, but this is partly caused by premature self-acceptance: the translator often concludes that its own interpretation is correct and stops without exposing the candidate to the user. Our protocol performs additional translations only when the user identifies a mismatch, while localized corrections reduce repeated unsuccessful attempts. For the 27B and Max translators, it achieves the highest expert-verified accuracy with only 1.4 and 1.2 translation attempts per sample, respectively.

W2 and Q2. Isolating the effect of backward explanation

Thank you for pointing out that the original comparison changed both the checker and the evaluation set. We agree that this makes it difficult to isolate the contribution of the backward explanation. We therefore conduct a paired evaluation on the same 400 intent–formula pairs. The only difference is the checker input: in the direct condition, the checker compares the requirement with the STL formula; in the backward-explanation condition, it compares the same requirement with the controlled-language explanation of the same formula.

i) Controlled setup. Both conditions use the same complete acceptance interface. The deterministic rule layer is retained unchanged and resolves the same 92 mechanically decidable cases without an API call. All remaining cases are evaluated by DeepSeek-V4-Flash, Kimi-K3, and GLM-5.3 under the same prompts, decoding settings, and majority-vote rule. The single-model rows below use the corresponding model for the unresolved cases, while the final row uses the full three-model majority vote. Both conditions use the same number of checker calls, the same maximum output-token budget, and the same deterministic rule layer; therefore, the only changed variable is whether the candidate formula is presented as STL or as its backward explanation. Thus, the requirement, candidate formula, checker models, rule layer, test cases, and computation budget are held fixed.

Paired checker accuracy on the shared 400-case calibration set

CheckerDirect NL–STLNL–ExplanationGain
DeepSeek-V4-Flash372/400 (93.0%)393/400 (98.3%)+5.3
Kimi-K3370/400 (92.5%)392/400 (98.0%)+5.5
GLM-5.3368/400 (92.0%)390/400 (97.5%)+5.5
Complete interface, majority vote376/400 (94.0%)398/400 (99.5%)+5.5
AuthorsOpenReview ↗

Response to Reviewer M1Sz (Part 3)

Following the former W2 and Q2. Isolating the effect of backward explanation

ii) Error breakdown. The paired results show improvements for both matches and mismatches, with the largest benefit on nested formulas.

Input to the complete acceptance interfaceOverall accuracyMatch recallMismatch recallNested recall
Direct NL–STL376/400 (94.0%)234/249 (94.0%)142/151 (94.0%)70/81 (86.4%)
NL–Explanation398/400 (99.5%)248/249 (99.6%)150/151 (99.3%)81/81 (100.0%)

The primary purpose of backward translation is to let users inspect and correct a candidate at the natural-language level. We also expected it to help automated checking because comparing two natural-language descriptions is easier for general-purpose models than directly interpreting nested STL syntax. The paired results support this expectation, while the three-model voting panel provides an additional reliability improvement in both conditions. The revised manuscript replaces the original cross-dataset comparison with this controlled result and clearly distinguishes it from the earlier results obtained on different evaluation sets.

W3 and Q3. User acceptance and alignment with human intent

We evaluate this concern at two separate links in the interaction. i) Does the description shown to the participant faithfully represent the underlying formula? Each candidate is a JSON-encoded STL syntax tree. Fixed recursive rules convert this tree into controlled language while preserving its predicates, thresholds, comparison directions, Boolean scope, temporal operators, intervals, and nesting. In a blind source-formula audit, the deterministic descriptions preserved the complete semantics in 1,000/1,000 cases and distinguished all 402/402 minimal contrastive pairs. The optional readability-polished descriptions were correct in 996/1,000 cases, with the deterministic description used whenever the consistency check failed.

ii) Does the underlying formula match the participant’s intent? After the study, two blinded STL experts independently inspected both the initial and terminal formulas from all 280 sessions, yielding 560 formula–intent comparisons. Both formulas were evaluated against the same participant-specific intent record, consisting of the initial requirement and all subsequent corrections. The experts inspected the STL formulas directly and did not see the backward descriptions, translator identities, acceptance decisions, or one another’s labels. Expert-verified accuracy increased from 166/280 (59.3%) for the initial formulas to 202/280 (72.1%) for the terminal formulas. All 202 participant-accepted formulas were judged semantically consistent with the corresponding participant intent (202/202, 100.0%, 95% CI: [98.1%, 100.0%]), with Cohen’s κ=1.00\kappa=1.00. The remaining sessions were not counted as successful deliveries.

iii) Participant and expert backgrounds. The study involved four participants: students as well as researchers and engineers working in autonomous driving and robotics. Two had prior exposure to formal methods or temporal logic, while the other two had no formal-methods background. None participated in constructing the benchmark or evaluating the system. The two post-study auditors were graduate researchers in Control Science and Engineering, with research experience in control and formal methods; each had authored at least one peer-reviewed international publication. They performed the audit independently, and any disagreement was resolved through adjudication.

The backward-description audit checks that the interface does not conceal a difference in the underlying formula, while the post-study audit directly checks whether the initial and final formulas match the participant’s recorded intent. We observed no false user acceptance in this study. However, this remains empirical evidence rather than a formal guarantee. The revised manuscript therefore replaces “ensures alignment with human intent” with the more precise statement that the protocol improves and empirically supports alignment with human intent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Review Feedback and Rebuttal Discussion

Dear Area Chair and Reviewers,

Thank you for your careful reading and constructive discussion. Following the rebuttal, Reviewer M1Sz raised their rating from 3 to 4 (Clear acceptance) and wrote that the additional checks “strengthen my support for acceptance.” Reviewer RsaP raised their rating from 2 to 3 (Weak acceptance) after finding that the revisions addressed most of their concerns.

The reviewers highlighted three strengths of the paper.

  • Practical value for formal methods (RsaP). RsaP found the approach well motivated and described the revised paper as “a valuable first step toward facilitating a wider application of formal methods in practice.”
  • An accessible way to clarify intent (z8ez). In their original review, z8ez recognized that the initial requirement can leave the intended behavior underspecified. Users can inspect and correct the system’s interpretation in their own words. As the reviewer noted, “This design has the potential to lower the barrier between engineers and formal tools.”
  • Feedback that helps locate an error (M1Sz). M1Sz observed that models produced more correct formulas “when feedback explains the mistake than when it simply asks them to try again.” They highlighted the gains for the 3B and 7B models and the value of an independent expert check.

The revised manuscript develops these points through three additional evaluations.

i) Localized feedback improves on self-refinement. At M1Sz’s suggestion, we evaluated a self-refinement baseline on the same 200 requirements under matched interaction limits. Our protocol achieved 11.5 to 35.0 percentage points higher expert-verified accuracy across four translators. For the 7B model, accuracy was 40.0% with self-refinement, 49.0% with generic retry, and 75.0% with localized feedback. Showing users every candidate interpretation gives them an opportunity to correct a mistake even when the translator sees no reason to revise. We also report translation calls and measured token costs so readers can assess the cost of these gains.

ii) The backward descriptions preserve distinctions that matter for checking. In a blind expert audit, the deterministic renderer preserved the source formula’s meaning in 1,000/1,000 cases. After optional language polishing, 996/1,000 descriptions remained correct. In a separate comparison using the same three checkers and the same 400 labeled pairs, providing the explanation raised majority-vote accuracy from 94.0% to 99.5%. RsaP also raised a point we share: a faithful description can still be difficult for a non-expert to interpret. We discussed how a concrete execution trace could help users see which behaviors a formula allows.

iii) Independent experts checked what participants actually received. Four participants contributed 40 requirements and evaluated them with seven translators. Two independent STL experts checked all 202 participant-accepted formulas against the participants’ recorded intent and confirmed every one. RsaP highlighted this finding as evidence that “the framework could provide a substantial benefit in practice.” To address z8ez’s questions about the scope of the study, we also compared the two protocols on the same requirements. Across 280 sessions per protocol, the human study yielded 202 accepted and expert-confirmed formulas, and the automated protocol yielded 204 expert-confirmed formulas. We clarified the participants’ backgrounds, the ten-call stopping rule, and how downstream tools can use the resulting formulas. We followed up with z8ez to ask whether any point needed further clarification.

Together, these results show where user feedback helps: users can identify an unintended interpretation, express the correction in natural language, and obtain a formula that independent experts can check against their intent. The revised paper documents the comparisons, audit procedures, and interaction costs so readers can assess both the gains and the effort required.

We sincerely thank the Area Chair and all three reviewers for their time and constructive suggestions.

Sincerely,
The Authors of Submission 25081

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.