- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a closed-loop framework for natural language to Signal Temporal Logic (NL-to-STL) conversion. The authors argue that natural language may be ambiguous or incomplete, so one-shot translation alone — even with a very capable model — cannot guarantee that the generated STL matches the user's true intent. The framework paraphrases the generated STL back into natural language, which the user then checks, accepts, or requests modifications to, and a new formula is generated accordingly. The user does not need to read or write STL. The authors evaluate the method on 500 expert-written requirements and seven models: the back-translation achieves a 99.5% agreement rate with expert judgment; the closed-loop procedure improves the conversion accuracy of multiple models, and the reliability of the feedback loop is evaluated through expert review and 280 user–model interactions.
Critical strengths
- It re-examines the task formulation. The paper points out that requirement texts may be insufficient to uniquely determine the target specification, so some errors cannot be resolved by improving the translation model's capability alone. Reframing the task as a closed-loop confirmation involving the user addresses a practical problem in formal requirements engineering.
- The interaction mode is friendly to non-experts. Users check the meaning of the STL and propose modifications in natural language, without needing to master the formal grammar. This design has the potential to lower the barrier between engineers and formal tools.
- The results show that closed-loop feedback is effective across multiple models. The authors report accuracy improvements for most models, with substantial improvements for some weaker models. Informative feedback outperforms the control of simply asking the model to retry, which also supports the claim that the feedback content itself contributes to the correction.
- Feedback reliability is validated from multiple angles. The paper evaluates not only the models' translation results, but also reports the agreement rate between back-translation and expert judgment, expert review of automatically accepted results, and real user interaction experiments. This is closer to the actual usage workflow than reporting accuracy only on static test sets.
Critical weaknesses
- User acceptance is still not a proof of formal correctness. What users see is a natural language paraphrase of the STL; if the paraphrase omits or obscures certain logical details, users may accept a formula that does not actually match their intent. The paper also acknowledges that the accepted result depends on the accuracy of the back-translation.
- The interaction cost varies greatly with model capability. For weaker models, the closed-loop accuracy improvement is limited but requires multiple rounds of feedback; for example, the paper reports that Qwen2.5-0.5B achieves a closed-loop accuracy of only 7.8% with an average of 8.34 feedback rounds. The practicality of the closed-loop approach may therefore depend on the model used and on how many rounds of interaction users are willing to invest.
- The coverage of the real-user evaluation is limited. The user study includes 40 requirements and 280 model–requirement interactions. This provides valuable preliminary evidence, but it is not yet sufficient to indicate how the method performs across more domains, different user groups, and more complex requirements.
- The success criteria of the closed-loop evaluation still deserve further clarification. The paper reports accuracy, acceptance rate, and the number of user feedback rounds, but these metrics respectively reflect the consistency between the formula and the reference, whether users accept it, and interaction cost; they may not fully measure whether the requirement was accurately understood and whether the final specification is suitable for downstream verification or control.
Questions for authors
- Regarding the reliability of user acceptance: How do the authors ensure that the STL-to-natural-language back-translation completely preserves the logical relations, temporal ranges, thresholds, and units in the formula? Is there a more systematic analysis of cases where users accepted formulas that still did not match the requirement?
- Regarding interaction cost: Can the authors report the distribution of feedback rounds required by different models to reach a target accuracy, and discuss how requirements that remain unresolved after many rounds would be handled in real deployment? For weaker models with many feedback rounds, is there a mechanism for interaction budgets or escalation to experts?
- Regarding the representativeness of the user study: Do the 40 requirements and the participants cover multiple application domains and users with different levels of formal methods experience? Can the results on acceptance rate and number of feedback rounds be reproduced across more requirements, users, and task scenarios?
- Regarding the evaluation criteria: Besides closed-loop accuracy and user acceptance rate, do the authors evaluate the performance of the final STL in downstream verification or control tasks? Can these evaluations distinguish between "matching the reference formula" and "genuinely matching user intent in the target application"?
3 comments
Response to Reviewer z8ez (Part 2)
W3 and Q3. Scope and representativeness of the real-user evaluation
The real-user study involved four participants with different professional backgrounds and levels of formal-methods experience. The group included students as well as researchers and engineers working in autonomous driving and robotics. Two participants had prior exposure to formal methods, while the other two had no formal-methods background. Each participant independently contributed 10 natural-language requirements and interacted with all seven translators on the same requirements, yielding 40 distinct requirements and model–requirement sessions. The requirements were authored freely rather than sampled under a fixed per-domain quota, and included autonomous-driving, robotic-system, and other CPS scenarios. Using the same requirement with all seven models controls for differences in requirement difficulty when comparing acceptance rates and interaction costs. The study produced 1,010 human-written NL inputs, including 730 corrective feedback messages. On the same 40-requirement subset, the real-user protocol produced 202/280 (72.1%) accepted and independently confirmed outputs, while the automated protocol produced 204/280 (72.9%) expert-confirmed correct outputs, supporting the representativeness of the automated evaluation on this subset. However, we do not claim that four participants establish reproducibility across arbitrary users or application domains. Broader task coverage is evaluated separately using the 500-case benchmark, while replication with more users, requirements, and deployed task scenarios remains an important direction for a larger-scale study. The revised manuscript now reports the participant composition, experience levels, per-participant workload, scope of the participant-authored requirements, and same-subset comparison.
W4 and Q4. Success criteria and downstream use
i) What counts as success. These metrics answer different questions: formula accuracy measures agreement with the benchmark reference, acceptance or convergence measures whether the interaction terminates within the prescribed budget, and the number of feedback rounds measures interaction cost. Operationally, a run terminates when the user or automated panel accepts a candidate; we report this as acceptance or convergence, not as ground-truth correctness. Correctness is evaluated separately through benchmark references, post-study expert review of all 280 human-study sessions, and the stratified audit of 634 automated acceptances. In the real-user study, two blinded STL experts confirmed all 202 participant-accepted formulas as semantically correct (202/202, 100.0%, 95% CI: [98.1%, 100.0%]). Separately, experts confirmed 629/634 (99.2%) outputs in the stratified audit of automated acceptances. Thus, the ACCEPT decision is an operational stopping decision rather than a correctness label.
ii) Downstream use. The closed-loop protocol does not change the artifact delivered to downstream systems. Verification, runtime-monitoring, and controller-synthesis tools take an STL formula over declared signals as input, and our framework returns exactly the same schema-valid JSON STL syntax tree as the underlying open-loop translator. A deterministic serializer converts this tree into the documented STL dialect. For verification or monitoring, the resulting formula can be evaluated over system traces; for controller synthesis, it can be supplied as a temporal constraint or objective. Therefore, any parser or adapter already used with the open-loop translator can be reused without modification. The role of our closed loop is to improve and confirm the formula’s alignment with user intent before this downstream step. It does not replace environment-specific predicate grounding, system modeling, feasibility analysis, or tool-specific validation. The revised manuscript now clarifies both the semantic success criterion and this downstream system boundary.
Changes Made in the Revised Manuscript
Revisions made specifically in response to this review are highlighted in teal, while revisions shared across reviewers are highlighted in orange. The revised manuscript distinguishes operational acceptance from semantic correctness, reports the expert audits of human and automated acceptances, clarifies interaction cost and unresolved cases, expands the description of the user study and same-subset comparison, and explains how the refined STL output connects to downstream tools.
Looking forward to the reviewer's reply!
Dear Reviewer z8ez,
Thank you again for your thoughtful review and valuable suggestions. We have revised the manuscript and provided further evidence concerning acceptance reliability, interaction cost, unresolved cases, and the scope and representativeness of the real-user evaluation.
As the discussion period draws to a close, we would be most grateful if you could let us know whether our response and revisions have adequately addressed your concerns. Should any point remain unclear, we would greatly appreciate the opportunity to provide further clarification.
Sincerely,
The Authors of Submission 25081
Response to Reviewer z8ez: Acceptance Reliability, Interaction Cost, and Real-User Evaluation
Thank you for your careful review and detailed feedback.
W1 and Q1. Reliability of user acceptance
To assess whether users made reliable acceptance decisions, we independently evaluated two links: whether the backward description faithfully represented the underlying STL formula, and whether the accepted STL formula itself was semantically correct.
i) Formula-to-language fidelity. Each candidate is represented as a JSON-encoded STL syntax tree. A deterministic renderer recursively converts every predicate, comparison direction, threshold, unit, Boolean relation, temporal operator, interval, and nesting relation into controlled language. It therefore does not ask an LLM to infer the formula’s meaning. GPT-5.6-Sol is used only to improve readability, and the system returns the deterministic description whenever the polished output fails the consistency check. In our expert audit, the deterministic descriptions preserved the complete source semantics in 1,000/1,000 cases, while the polished descriptions were correct in 996/1,000 cases. The deterministic renderer also preserved the relevant semantic distinction in all 402/402 minimal contrastive pairs.
ii) Correctness of accepted formulas. Participants made all acceptance decisions and provided all corrective feedback themselves. After the study, two blinded STL experts examined the underlying terminal formulas from all 280 sessions. For each session, they evaluated the complete formula semantics against a separate intent record derived from the participant’s initial requirement and subsequent corrections, without seeing the model identity or the participant’s acceptance decision. All 202 participant-accepted formulas were judged semantically correct (202/202, 100.0%, 95% CI: [98.1%, 100.0%]), with Cohen’s κ=1.00. Thus, we observed no case in which a participant accepted an incorrect underlying formula. Separately, experts confirmed 629/634 (99.2%) outputs accepted by the automated protocol; the remaining five are explicitly reported as false acceptances.
W2 and Q2. Interaction cost and unresolved cases
Figure 3(b) already reports the cumulative closed-loop accuracy of all seven translators after 1, 2, 3, 5, and 10 forward calls. The first call produces the open-loop candidate, so k forward calls correspond to at most k−1 feedback events. Equivalently, each curve is the cumulative distribution of the number of calls required for successful refinement: the increase between two points is the fraction of cases resolved during that interval, and the fraction remaining below 100% after ten calls is unresolved under the fixed interaction budget. The results show that additional interaction is useful only when the translator can apply corrective feedback. Qwen2.5-0.5B improves only from 6.8% to 7.8% despite requiring 8.34 feedback events per sample, whereas the locally deployable Qwen3.8-27B reaches 98.0% closed-loop accuracy with only 0.41 feedback events per sample. Thus, our results do not suggest that users should continue interacting indefinitely with an under-capable translator; selecting a sufficiently capable local model is substantially more effective. The evaluated protocol already enforces a fixed maximum budget of 10 forward calls and returns
unresolvedif no candidate is accepted within that budget, rather than delivering an unconfirmed formula. The current prototype does not automatically switch models or escalate cases to experts. In deployment, anunresolvedresult can serve as a handoff point for routing the requirement to a stronger translator or a formal-methods expert, but this routing policy is outside the evaluated protocol.