acceptodds
Desk rejected at ICLR 2027

ARGOS:Automated Functional Safety Requirement Synthesis for EmbodiedAI via Attribute-Guided Combinatorial Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Ensuring functional safety is essential for deploying Embodied AI in complex open-world environments. Traditional Hazard Analysis and Risk Assessment (HARA) relies on enumerating risks for finite, pre-defined function lists, yet Embodied AI operates on open-ended natural language instructions whose combinatorial interactions give rise to unbounded hazard scenarios. Large Language Models (LLMs) offer a promising path to scalability, but without physical grounding they fall back on statistical correlations and overlook long-tail coupled risks. To address these limitations, we present ARGOS (Attribute-Guided Combinatorial Reasoning), a two-stage pipeline that bridges open-ended instructions and physical constraints. Stage I decomposes entities into fine-grained physical attributes and performs bounded combinatorial reasoning to discover multi-factor hazard scenarios. Stage II integrates these scenarios with robot capabilities and regulatory standards to synthesize auditable Functional Safety Requirements (FSRs). Experiments across two backbone models show that ARGOS significantly outperforms intrinsic-knowledge and chain-of-thought baselines in hazard scenario quality (Cohen's d=0.62–1.00). ARGOS also achieves the highest Overall score against full-rulebook long-context and STPA/FMEA-structured baselines (8.57 vs. 7.78 and 7.57). Human evaluation further reveals a Verbosity Trap: chain-of-thought reasoning generates redundant scenarios that inflate automated metrics yet are penalized by human experts. Stage-II ablations show that physical grounding consistently improves requirement synthesis and expose a Regulatory Mismatch on the primary backbone when regulatory knowledge is applied without sufficient physical context. Together, these findings establish explicit attribute grounding as a critical prerequisite for reliable safety reasoning in Embodied AI. The source code is available at https://anonymous.4open.science/r/ARGOS-274E.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Embodied AI, LLM Safety, Functional Safety, Hazard Analysis, Physical Grounding, Combinatorial Reasoning, Requirements Engineering, Safety-Critical AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9uP9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ARGOS, a two-stage LLM-based framework for automatically generating Functional Safety Requirements (FSRs) from open-ended instructions for embodied agents. In the first stage, the framework extracts task-relevant entities and retrieves physical attributes from an expert-curated knowledge base containing 82 typed profiles. It then uses structured prompting to combine these attributes and generate potentially hazardous interaction scenarios. In the second stage, the generated hazards are combined with robot capability specifications and regulatory information to synthesize FSRs.

The authors evaluate ARGOS on 48 seed scenarios expanded into 365 hazard descriptions, using DeepSeek-V3.2 and GPT-4o. The evaluation includes comparisons with vanilla prompting, physics-aware chain-of-thought, full-rulebook long-context prompting, and an STPA/FMEA-structured baseline. Generated outputs are assessed using LLM-based scoring and human evaluations, with additional ablations examining the contributions of physical attribute grounding and regulatory retrieval.

Critical strengths
  1. Practically relevant problem formulation.

Automating hazard analysis and functional safety requirement generation for open-ended embodied tasks is a meaningful problem. The attempt to identify multi-factor hazards through reusable physical attributes, rather than relying exclusively on predefined task-level hazard descriptions, provides a potentially useful direction for safety engineering.

  1. Inclusion of stronger baselines and human evaluation.

The paper goes beyond comparisons with basic prompting methods by including full-rulebook long-context and STPA/FMEA-structured baselines. The additional blinded human evaluation and paired statistical analysis in Appendix H provide useful evidence for assessing the quality of generated outputs. These efforts improve the empirical presentation, although they do not resolve my main concerns about methodological novelty and the validity of the central scientific claims.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for a distinct combinatorial reasoning mechanism.

The proposed framework appears to rely primarily on a combination of established techniques, including expert-curated knowledge retrieval, embedding-based matching, structured prompting, and LLM-based text generation. In particular, the central combinatorial reasoning operation in Section 3.1 is implemented through an LLM prompted with combinations of retrieved physical attributes. Appendix A.1 further identifies the Risk Injection Block, essentially a structured prompting template, as the core methodological contribution.

While such a pipeline may be practically useful, it is unclear what new generalizable reasoning or learning mechanism is introduced beyond existing retrieval-augmented generation and structured safety-analysis prompting.

More importantly, the experiments do not sufficiently isolate the contribution of the proposed combinatorial reasoning procedure. Although the authors include full-rulebook long-context and STPA/FMEA baselines, these comparisons do not directly establish whether explicit attribute combination provides benefits beyond supplying the same relevant physical knowledge through a simpler prompt. A controlled comparison using identical retrieved attributes, backbone models, and generation budgets, while varying only the proposed combination mechanism, is needed to support the claimed methodological contribution.

Consequently, I find the methodological novelty insufficiently established for ICLR.

  1. Insufficient validation of open-world hazard discovery, knowledge-base robustness, and safety requirement correctness.

A fundamental limitation is the framework's dependence on an expert-curated knowledge base containing only 82 typed physical profiles. Although the authors argue that combining reusable attributes avoids explicitly enumerating every hazard, successful discovery still depends on the relevant physical factors being represented, correctly retrieved, and appropriately combined.

The paper does not systematically evaluate what happens when essential physical attributes are absent from the knowledge base, when retrieval produces incorrect or incomplete results, or when hazards involve previously unseen physical mechanisms. The retrieval audit in Appendix H.4 evaluates matching against the existing expert-defined profiles, rather than whether those profiles provide sufficient coverage of hazards in genuinely novel scenarios. Therefore, the reported retrieval accuracy does not establish the claimed open-world scalability.

Furthermore, the evaluation primarily measures the perceived quality of generated hazard descriptions and FSRs rather than their correctness and completeness against independently established safety references. Although expert evaluations are included, these assessments do not establish hazard discovery recall, the physical validity of the proposed causal interactions, or the correctness and verifiability of the resulting safety requirements.

The evaluation rubric in Appendix E.2 also explicitly rewards multi-factor coupling, which is closely aligned with the structure enforced by the proposed generation prompts. This raises concerns about whether the reported improvements reflect genuinely better hazard discovery or greater conformity to the evaluation criteria.

The evidence against the strongest long-context baseline is also less conclusive than the main results suggest. In the additional 12-seed blinded audit (Appendix H, Tables 9–10), the overall improvement is only 0.153 points, with a paired sign-flip test p-value of 0.0898. The long-context baseline also performs better on the Long-tail Risk dimension in this audit.

Taken together, these limitations make it difficult to conclude that ARGOS provides a reliable, scalable, or substantially novel approach to discovering hazards in open-ended embodied environments. The current experiments primarily demonstrate improvements in the quality of structured safety-analysis text, rather than establishing the broader open-world hazard discovery capabilities claimed by the paper.

Questions for authors

Q1. What is the independent contribution of the proposed combinatorial reasoning mechanism?

Section 3.1 and Appendix A.1 describe the central combinatorial operation primarily as structured LLM prompting over retrieved physical attributes. Can the authors clarify what distinguishes this mechanism from conventional retrieval-augmented prompting? More importantly, can they provide a controlled comparison in which the same retrieved attributes, backbone model, and generation budget are used, with and without the proposed combination-specific prompting? This would help determine whether the improvement comes from the proposed reasoning mechanism or simply from supplying relevant physical knowledge.

Q2. How does ARGOS handle incomplete knowledge and retrieval failures?

The framework relies on 82 expert-curated physical profiles. What happens when a task involves a critical physical property or hazard mechanism that is not represented in this knowledge base? Similarly, how does the system respond when retrieval fails to return a relevant attribute or introduces an incorrect one? Do the authors have quantitative evidence from controlled knowledge-removal, retrieval-error, or unseen-hazard experiments? Such evidence is important for evaluating the claimed applicability to open-ended environments.

Q3. What evidence establishes the actual correctness and coverage of generated hazards and FSRs?

The reported evaluations focus primarily on rubric-based assessments of generated descriptions. Can the authors provide independent evidence that the identified multi-factor interactions are physically valid, that important hazards are not systematically omitted, and that the synthesized FSRs are consistent with the applicable regulatory and robot capability constraints? In particular, are there expert-validated reference hazards or safety requirements against which discovery recall, false hazard generation, and requirement correctness can be evaluated?

Answers to these questions would materially affect my assessment of both the methodological novelty and the strength of the experimental validation.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 9uP9

We appreciate the reviewer's recognition of the practical relevance of automated hazard analysis for embodied AI, as well as the acknowledgment of the stronger baselines and blinded human evaluation.

The central issue is how the methodological contribution and experimental evidence of ARGOS should be interpreted. We clarify these points below.

Q1: Independent contribution of the combinatorial reasoning mechanism

ARGOS does not claim to introduce a new neural architecture or a new general-purpose learning mechanism. Its contribution is a structured physical reasoning methodology for hazard discovery and FSR synthesis.

The distinction from conventional retrieval-augmented prompting lies in how retrieved knowledge is represented, constrained, and used.

In conventional retrieval-based prompting, retrieved information is generally supplied as additional context for text generation. ARGOS instead treats typed physical profiles as explicit reasoning premises and constructs bounded interaction hypotheses from them.

The proposed procedure has four connected components:

  1. Typed physical-attribute representation: Semantic entities are mapped to reusable physical profiles containing explicit properties and physical mechanisms.
  2. Explicit interaction composition: Retrieved profiles are combined into joint reasoning contexts, with the generation process instructed to identify the physical mechanism connecting the factors rather than simply describe them independently.
  3. Causal and counterfactual constraints: The Risk Injection Block requires both specified factors to be present for the interaction hazard and requires the hazard to disappear when either factor is removed. It also enforces seed-bound semantics and physical consistency.
  4. Scenario-anchored FSR synthesis: The resulting hazard mechanisms are propagated into a separate synthesis stage conditioned on robot hardware capabilities and retrieved regulatory information.

This is more specific than simply retrieving related text and asking an LLM to generate hazards.

For example, in the Rose-and-Balloon case, the relevant physical properties are not merely supplied as descriptions. The generation procedure explicitly requires reasoning about their interaction: a rigid thorn and a low-puncture-resistance membrane jointly produce a puncture mechanism. This leads to a force-level safety constraint rather than an unrelated generic speed limit.

The contribution is therefore the operational formulation of attribute-grounded, bounded combinatorial hazard reasoning and its integration into downstream FSR synthesis. We do not rely on a claim that the Risk Injection Block alone constitutes a new learning architecture.

Regarding the experimental isolation of combinatorial reasoning:

The full-rulebook baseline receives all 82 physical profiles in one prompt without truncation. Both ARGOS and this baseline use the same backbone, seed inputs, robot capability profiles, target counts, and evaluation rubric.

This comparison tests whether presenting the complete physical knowledge base jointly is equivalent to explicitly selecting and composing relevant physical premises. The results show that it is not: ARGOS obtains higher Physical Reliability and FSR Derivation, while the full-rulebook baseline remains competitive on Long-tail Risk.

The comparison therefore provides evidence for the value of structured physical knowledge organization and interaction composition.

We also distinguish this evidence from a stricter causal ablation. The current experiments do not include a fully matched comparison that holds the exact retrieved attributes, generation budget, and backbone constant while removing only the combination-specific interaction mechanism. We do not claim that the existing full-rulebook comparison independently isolates the contribution of that single component.

We will revise the manuscript to distinguish the methodological contribution of the complete ARGOS procedure from the causal attribution of individual components. We will also clarify that the current baseline comparisons establish the advantage of the complete structured workflow, rather than claiming that they independently isolate the effect of combinatorial reasoning under identical token and generation budgets.

Q2: Incomplete knowledge bases and retrieval failures

The 82-profile knowledge base should be understood as a collection of reusable physical primitives rather than a catalog of predefined hazards.

Its purpose is to provide physical premises that can be composed across different tasks and scenarios. The 48 seed instructions are expanded into 365 context-specific hazard scenarios through attribute retrieval and bounded interaction reasoning.

Importantly, we distinguish retrieval accuracy from knowledge-base completeness.

Retrieval reliability.

ARGOS uses BGE-based semantic retrieval with a configurable similarity threshold. We evaluate thresholds from 0.5 to 0.9.

At τattr=0.7\tau_{\mathrm{attr}}=0.7, a case-by-case audit found no hallucinated or missed profiles within the evaluated household-service subset. Lowering the threshold to 0.6 introduces approximately 25% hallucinated retrievals, whereas thresholds of 0.8 or above begin to miss valid semantic matches.

The framework also provides a configurable Hybrid Hard-Match Fallback, under which exact lexical matches override embedding disagreements for known entries. This combines semantic matching with deterministic handling of explicitly represented profiles.

Knowledge-base coverage.

The reviewer correctly distinguishes matching an existing profile from establishing that the knowledge base contains every relevant physical mechanism. These are different questions.

ARGOS does not claim that successful retrieval from the existing knowledge base proves exhaustive physical coverage. If a critical physical property or mechanism is absent from the knowledge base, the current pipeline does not provide a formal guarantee that it will independently recover that missing information.

Likewise, the current experiments do not include controlled knowledge-removal, retrieval-error injection, or systematically held-out unseen-mechanism evaluations. We do not interpret the retrieval audit as evidence of universal open-world coverage.

The contribution of the current system is compositional hazard discovery using reusable physical attributes, within a defined household-service domain and bounded interaction space.

We will revise the manuscript to explicitly distinguish retrieval reliability from knowledge-base coverage, clarify the scope of the 82 profiles, and add a limitation statement concerning missing physical attributes and previously unseen mechanisms. We will also narrow the open-world generalization claims to avoid implying guaranteed discovery beyond the represented physical knowledge.

Q3: Correctness and coverage of generated hazards and FSRs

The evaluation of ARGOS involves distinct criteria at different stages, and these should not be conflated.

1. Physical validity and seed consistency

The Stage-I evaluation measures Physical Reliability, Long-tail Risk, and FSR Derivation Capability.

The evaluation is not based merely on whether a generated scenario contains multiple factors. Physical Reliability explicitly assesses real-world physical plausibility and adherence to the seed's closed-world constraints. The generation procedure also requires the hazard to be causally attributable to the interaction of the specified factors.

In addition, three robot-safety researchers independently evaluated 108 anonymized and shuffled scenarios. ARGOS obtained the highest human mean scores across the three dimensions:

MetricARGOSCoTVanilla
Physical Reliability7.947.346.97
Long-tail Risk7.265.296.43
FSR Derivation7.345.466.49

These results provide independent human assessments of the physical quality and requirement relevance of the generated hazard descriptions.

2. Interpretation of the multi-factor evaluation criterion

The reviewer notes that the Long-tail Risk rubric explicitly rewards multi-factor coupling.

This criterion is intentional: discovering coupled physical risks is the central objective of ARGOS. However, the metric does not simply count the number of mentioned factors. It evaluates whether the generated scenario identifies a credible physical mechanism, remains within the seed context, and reveals a meaningful safety boundary or secondary consequence.

Physical Reliability and FSR Derivation are also evaluated separately, preventing the Long-tail Risk score from being interpreted as a complete measure of overall scenario correctness.

Furthermore, the three human evaluators assessed anonymized outputs using the same predefined rubric, providing an assessment beyond the automatic generation model's own output.

We will revise the manuscript to explain more explicitly why multi-factor coupling is part of the target construct, distinguish it from mere factor enumeration, and clarify the complementary roles of the three evaluation metrics.

3. FSR correctness and hardware compatibility

Stage II uses a dedicated audit protocol that evaluates hardware compatibility, physical risk coverage, and logical robustness.

The audit explicitly checks whether each FSR assumes unsupported sensor capabilities, violates hardware limitations, misses secondary physical consequences, or lacks coherent triggering and recovery conditions.

The ablation results show that ARGOS achieves the highest Overall FSR quality on the primary backbone (7.97) and maintains its advantage in the controlled GPT-4o study (7.59).

We emphasize that these results support requirement-level quality and hardware consistency under the declared evaluation protocol. They are not equivalent to independent certification of the applicable standards clauses or physical validation of an implemented controller.

We will revise the manuscript to make the validation scope explicit and distinguish the current rubric-based requirement audit from downstream expert certification, TSR verification, and physical robot testing. We will also clarify the role of regulatory retrieval in connecting the hazard descriptions to the available standards knowledge base.

4. Interpretation of the strongest baseline comparison

We agree that the 12-seed blinded audit should be interpreted according to its paired statistics rather than relying solely on the Overall mean.

ARGOS obtains an Overall score of 8.45, compared with 8.30 for full-rulebook long-context prompting. The Overall difference is 0.153 (p=0.0898p=0.0898), which is not statistically significant.

However, the dimension-level results are informative:

  • Physical Reliability: +0.283 (p=0.0195p=0.0195).
  • Long-tail Risk: −0.192 (p=0.0547p=0.0547).
  • FSR Derivation: +0.367 (p=0.0020p=0.0020).

Long-context performs better on Long-tail Risk in this subset, while ARGOS demonstrates significant improvements in Physical Reliability and FSR Derivation.

We therefore do not claim statistically significant Overall superiority over long-context prompting in this 12-seed audit. The evidence supports a more specific conclusion: explicit physical grounding and bounded interaction composition improve physical reliability and the derivability of safety requirements.

We will revise the manuscript's Abstract, Results, and Discussion to make this distinction explicit and avoid overstating the strength of the Overall comparison.

Overall methodological positioning

ARGOS addresses a specific problem in embodied-AI safety engineering: deriving physically grounded hazard hypotheses and functional safety requirements from open-ended natural-language task instructions.

Its contribution is not unrestricted physical-world discovery or a new general-purpose LLM learning mechanism. It is a structured two-stage procedure that connects:

  1. Reusable typed physical attributes.
  2. Bounded, causally constrained interaction reasoning.
  3. Scenario-specific hazard descriptions.
  4. Regulatory- and hardware-grounded FSR synthesis.

The experiments demonstrate the benefits of this complete procedure across multiple backbones, stronger baselines, ablation studies, and human evaluation.

We will revise the Abstract, Introduction, Methodology, and Discussion to make the scope, methodological contribution, evaluation boundaries, and limitations more precise. In particular, the revised manuscript will consistently describe ARGOS as an offline design-time framework for open-ended task instructions with bounded physical-attribute composition, rather than suggesting unrestricted open-world coverage or runtime safety control.

Reviewer_5DofOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to address the problem that LLMs might miss risks arising from combinations of factors grounded in the physical world. To do so, they decompose the entities into several attributes and then perform combinatorial reasoning over these attributes. Then, they map it back to the necessary standards and capabilities of the robot.

Critical strengths

S1: The paper is well motivated. The problem is interesting. I think the direction of this research is promising, even if it is not yet fully explored.

Critical weaknesses

W1: The paper's motivation talks about solving the problem of online safety in a general setting, but the evaluation seems to be focused on offline (line 270), and the combinatorial argument is bound to a set of fixed attributes. This seems to be a large discrepancy. I also do not see how to access the required standards efficiently when a new prompt is given. W2: While the idea of the running examples is nice, there are two for an unknown reason, which I found quite confusing when reading.

Questions for authors
  • Could you please explain in more detail what the advantage over the baseline is when this method is not applicable at runtime?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 5Dof

We appreciate the reviewer’s recognition of the importance of physical-attribute-based hazard analysis. We clarify ARGOS’s intended operating stage and the roles of the two running examples.

W1: Offline evaluation, open-ended instructions, and runtime applicability

ARGOS targets offline, design-time safety analysis, not runtime collision avoidance or real-time safety monitoring. Here, open-ended describes the input space: ARGOS accepts natural-language task instructions rather than requiring a predefined list of robot functions and manually enumerated hazards. It derives task-specific hazards from the entities and physical interactions implied by each instruction.

The distinction is:

  • Open-ended task inputs: Different natural-language instructions can be analyzed without manually defining a separate hazard list for each task.
  • Offline analysis: ARGOS performs safety analysis during design or task preparation, before FSRs are translated into executable mechanisms.
  • Runtime operation: Downstream technical safety requirements (TSRs) specify the concrete hardware and software mechanisms implemented by the robot.

ARGOS therefore addresses the scalability of design-stage hazard analysis; it is not intended to replace runtime safety systems. Its 82 profiles are reusable physical primitives, not predefined task-level hazard scenarios. ARGOS composes retrieved profiles to generate context-specific hazards, with interaction order bounded at three for tractable analysis. The demonstrated scope is household-service safety analysis within the coverage of the current profile base; we do not claim guaranteed discovery of arbitrary physical mechanisms absent from that base.

Regulatory information is retrieved semantically rather than by loading the entire standards library for every instruction. In Stage II, ARGOS embeds each generated hazard description, retrieves relevant clauses from the regulatory knowledge base using a similarity threshold, and combines them with the robot’s hardware specifications during FSR synthesis. Thus, task-specific hazards guide the selection of relevant regulatory context.

We will revise the Abstract, Introduction, and Discussion to distinguish open-ended task inputs from online runtime operation, describe ARGOS more precisely as an offline framework for open-ended task instructions with bounded physical-attribute composition, and state the scope of the 82-profile knowledge base. This will make clear that ARGOS supports design-time analysis without claiming exhaustive coverage of unseen physical mechanisms.

W2: Purpose of the two running examples

The examples illustrate complementary stages and physical reasoning mechanisms.

Rose and Balloon—attribute-level material interaction. This Stage-I example shows how the rose’s rigid thorn interacts with the balloon’s thin, low-puncture-resistance latex membrane to produce a puncture hazard. A generic velocity limit does not address this material-interaction mechanism; the analysis instead motivates a contact-force-level constraint. The example demonstrates how explicit physical attributes determine the relevant safety constraint.

Hot Soup Delivery—coupled dynamics and FSR synthesis. This example illustrates the transition from hazard discovery to requirement synthesis. When a robot carrying hot soup brakes to avoid a running child, inertial forces may destabilize the liquid payload, creating secondary burn-and-slip hazards. A generic emergency-stop command is therefore insufficient. Stage II derives an admissible deceleration constraint based on the robot’s braking capability and payload stability.

The examples are thus not redundant: Rose and Balloon illustrates material-interaction reasoning in Stage I, while Hot Soup illustrates coupled dynamic hazards and scenario-anchored FSR synthesis. We will revise the transitions and descriptions to make these distinct roles explicit.

Thank you for the response. I see. Given the examples of delivering the soup, I expected ARGOS to run on the robot itself, which I would classify as online. I also appreciate the clarifications in the paper.

Reviewer_VvBMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ARGOS extracts entities from household-robot instructions, retrieves expert-written physical attributes, prompts a language model to analyze hazards that arise from interactions among multiple factors, and then generates functional safety requirements by combining those hazards with robot capabilities and standards clauses. The experiments are built around 48 seed scenarios and compare direct generation, chain-of-thought prompting, a complete rule library, and structured safety analysis. Human evaluation is reported for the hazard descriptions.

Critical strengths

The work is fairly complete as an end-to-end workflow, and the purpose of each of the two stages is clear. Writing physical attributes down as explicit premises is helpful, because it lets a reader inspect how a hazard is supposed to follow from those premises rather than treating the hazard as an opaque model output. The rose-and-balloon example is especially intuitive: whether the balloon is punctured depends on the contact mechanism, so a speed limit by itself does not resolve the risk. The human evaluation of hazards, together with comparisons against the complete rule library and STPA/FMEA-style prompting, gives the workflow a reasonable amount of empirical support.

Critical weaknesses

My main concern is whether the final generated safety requirements are actually correct. The strongest human evaluation in the main text focuses on the physical soundness of the hazard descriptions, on long-tail risks, and on whether those hazards could in principle inform requirements. The second stage, which produces the functional safety requirements themselves, relies much more on automatic quality scores. As a result, it remains difficult to judge whether the generated requirements apply standards clauses correctly, whether the physical parameters have a defensible basis, and whether executing the requirements would actually mitigate the stated hazards. Table 1 already identifies logical auditing as the validation method for FSRs. An independent check of complete derivation chains would therefore support the paper’s central output much more directly than further improvements in text-quality scores.

The distinctive benefit of combinatorial reasoning is also not separated from other changes that happen at the same time. ARGOS makes an average of 7.62 generation calls per seed, whereas the complete-rule-library baseline supplies knowledge in a single prompt. Context selection and reasoning effort therefore change together, so it is hard to tell how much of the gain comes from combinatorial reasoning itself. In the human audit of stronger baselines in Section 4.2, the total scores are 8.45 versus 8.30, which also suggests caution about the advantage over the strongest alternative. The results support the complete workflow, but they do not establish that its gains are independent of the extra calls or of the way context is organized.

Overall assessment: I appreciate the clarity of the workflow and the amount of effort behind it. The paper’s goal, however, is to deliver reliable functional safety requirements, while the current evidence much more strongly supports the quality of the hazard descriptions. Together with the still unresolved attribution of the method’s gains, this leads me to a rating of 2.

Questions for authors

Have independent experts checked complete physical-attribute-to-hazard-to-standards-clause-to-FSR derivation chains? Please provide a representative complete result, particularly the basis for numerical constraints and the applicability of the clauses.

What are the generation-call counts and token budgets for the complete rule library, structured safety analysis, and ARGOS? Are there existing comparisons that use the same attribute information and similar budgets, and that therefore identify the contribution of combinatorial reasoning more directly?

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer VvBM

We appreciate the reviewer's positive assessment of ARGOS, particularly the end-to-end workflow, explicit physical-attribute representation, and the Rose-and-Balloon example. We address the two concerns regarding FSR validation and the contribution of combinatorial reasoning.

W1: Correctness and validation of synthesized FSRs

The evaluation of ARGOS should be considered in the context of its intended position in the functional safety lifecycle.

FSRs and physical implementation are two different engineering stages. ARGOS operates at the FSR synthesis stage, where the objective is to define what safety constraints must hold, rather than to implement or execute the corresponding safety controller. These FSRs subsequently serve as input constraints for downstream Technical Safety Requirements (TSRs), which specify concrete hardware and software mechanisms.

Therefore, requiring physical execution to establish the validity of an FSR conflates requirement-level synthesis with implementation-level verification. The absence of physical robot deployment does not invalidate the assessment of whether a generated requirement is physically grounded, consistent with the robot's declared capabilities, and logically traceable to its originating hazard.

Our Stage-II evaluation directly examines these properties through three dedicated metrics:

  • Capability Compliance (CC): Whether requirements respect hardware specifications and physical limitations.
  • Physical Risk Coverage (PRC): Whether requirements address primary hazards, secondary consequences, and scenario-specific risks.
  • Logic Robustness (LRC): Whether requirements define consistent triggering conditions, safety-state transitions, and recovery conditions.

The evaluation uses a dedicated FSR technical-audit prompt that explicitly requires hardware cross-referencing, scenario-specific hazard enumeration, and trigger-action-recovery analysis. It also requires the evaluator to identify individual FSRs that violate the declared robot capabilities.

The results demonstrate a consistent advantage for physically grounded requirement synthesis:

MethodCCPRCLRCOverall
Vanilla7.916.967.287.38
ISO-Only7.967.006.907.28
w/o ISO8.567.717.297.85
ARGOS8.567.817.547.97

The results show that physical grounding provides the principal improvement in capability-aware FSR synthesis, while regulatory retrieval provides additional improvements in risk coverage and logical robustness when combined with physical context.

This finding also persists across generation backbones. In the controlled 10-seed GPT-4o Stage-II study, ARGOS achieves an Overall score of 7.59, compared with 7.38 for w/o ISO, 7.10 for ISO-Only, and 6.85 for Vanilla.

Regarding the request for independent validation of complete derivation chains, we distinguish the two existing evaluation protocols. The three human reviewers evaluated 108 Stage-I hazard scenarios, whereas Stage-II FSR quality was assessed using the dedicated technical-audit evaluator. We do not characterize the latter as independent human certification of the complete physical-attribute-to-standard-to-FSR chain.

We will revise the manuscript to make this distinction explicit, clarify the scope of the existing validation, and provide a more transparent explanation of the traceability from physical attributes to hazards and subsequent FSR synthesis. We will also clarify that parameters such as asafea_{\mathrm{safe}} are conditioned on robot braking capabilities and payload stability constraints, rather than universal experimentally calibrated constants. The manuscript will explicitly distinguish requirement-level evaluation from downstream implementation and certification.

W2: Contribution of combinatorial reasoning and generation overhead

The reviewer raises a question about whether the improvement comes from combinatorial reasoning or the additional generation calls. The relevant comparison is not simply the number of calls, but how physical knowledge is organized and used during hazard discovery.

ARGOS does not perform unrestricted repeated sampling. It uses a fixed candidate-generation policy, with interaction order capped at three. Across the 48 evaluation seeds, the average is 7.62 generation calls per seed, with a maximum of 14.

The allocation is explicitly determined by the number of retrieved profiles:

Retrieved profilesCalls per seedNumber of seeds
2612
3730
4146

Each generation call corresponds to a predefined single-, pairwise-, or applicable triple-profile interaction hypothesis. This maintains explicit provenance between physical premises and generated hazards.

More importantly, the full-rulebook baseline receives the complete 82-profile knowledge base in a single prompt without truncation. Both methods use the same DeepSeek-V3.2 backbone, seed inputs, robot capability profiles, target counts, and evaluation rubric. The comparison therefore examines the difference between presenting all physical profiles jointly and explicitly selecting and composing relevant physical attributes.

The primary results show that ARGOS achieves higher Physical Reliability and FSR Derivation than the full-rulebook baseline:

  • Physical Reliability: 8.97 vs. 7.55.
  • Long-tail Risk: 8.21 vs. 8.15.
  • FSR Derivation: 8.55 vs. 7.65.

These results indicate that access to the complete physical knowledge base alone does not reproduce the performance of explicit attribute selection and interaction composition.

We also clarify the 12-seed blinded audit. ARGOS achieves 8.45 Overall compared with 8.30 for long-context prompting. However, the Overall difference is not statistically significant (p=0.0898p=0.0898). The stronger evidence is dimension-specific: ARGOS significantly improves Physical Reliability (+0.283, p=0.0195p=0.0195) and FSR Derivation (+0.367, p=0.0020p=0.0020), whereas long-context performs better on Long-tail Risk.

We do not interpret these results as evidence of statistically significant Overall superiority over long-context prompting in the 12-seed subset. Rather, they demonstrate a specific advantage in physical grounding and requirement derivability.

The current manuscript reports ARGOS's generation-call budget and the single-prompt setup of the full-rulebook baseline, but does not provide a matched token-budget comparison across every baseline.

We will revise the manuscript to make the generation budget and baseline settings more explicit, clarify that the experiments evaluate the complete workflow rather than isolate a compute-normalized causal contribution of combinatorial reasoning, and distinguish the statistically supported dimension-level improvements from the non-significant Overall difference in the blinded audit.

Thank you for uploading the revision. I have read the revised PDF, and I appreciate that the scope, statistical qualifications, and limitations are now stated clearly and candidly. However, my main concern remains: the revision does not provide evidence that isolates the benefit of the combinatorial reasoning itself, and the advantage over the strongest baseline remains limited. I will therefore keep my rating at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.