- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OntoAct, a neuro-symbolic agent framework that extracts reusable knowledge from historical experience through ontological abstraction. The framework converts operational records into typed assertions and conditional axioms. During interaction, an LLM proposes hypotheses and actions, while a symbolic verifier checks their consistency with the learned knowledge and current observations. Hypotheses are updated as new evidence and action outcomes become available.
The authors also introduce Tacit-FLE, a benchmark built on the Factorio Learning Environment for diagnosing and repairing hidden faults using historical maintenance records. Experiments are conducted on Tacit-FLE, Mars, and ALFWorld, with comparisons against ReAct, Reflexion, IfR, NeSyC, and WALL-E. The reported results show improvements in task success, reduced unsuccessful interventions, and lower inference costs in several settings.
Critical strengths
- Meaningful problem formulation and execution-based evaluation.
The paper addresses the important problem of transferring knowledge from historical experience to new situations while verifying whether previously learned conditions remain applicable. Tacit-FLE provides an interesting evaluation setting in which an executed repair does not necessarily resolve the underlying fault. Measuring both successful repairs and unsuccessful interventions offers more meaningful evidence than relying solely on textual assessments.
- Detailed diagnostic analyses.
The paper provides extensive supplementary analyses, including hypothesis tracking, candidate fault coverage, resource usage, and action-level verification diagnostics. These analyses are useful for understanding the behavior of the proposed system and provide some evidence that explicit knowledge verification can reduce unnecessary interventions.
Critical weaknesses
- Serious concerns about the validity and fidelity of a key neuro-symbolic baseline.
My primary concern is the evaluation of NeSyC, a closely related neuro-symbolic learning framework.
In Table 1, NeSyC achieves 0% repair success across all 104 Tacit-FLE evaluation episodes. More importantly, Appendix D.1 reveals that this result is not simply due to poor fault diagnosis or unsuccessful repair attempts. Across the 3,000 evaluation turns on the small and medium suites, every executed action is a waiting action. Among the 64 initial planner outputs, 47 contain no plan and the remaining 17 contain only a waiting action.
This is a substantial concern. The reported behavior suggests that the evaluated NeSyC implementation was effectively unable to perform the intended repair task. Such a result does not provide meaningful evidence that OntoAct outperforms NeSyC in knowledge abstraction, hypothesis evaluation, or fault diagnosis.
Appendix C.2 states that the authors reconstructed NeSyC's induction and rule-application procedures using the released implementation as a reference and introduced additional adaptations to connect its symbolic planning mechanism to the target environments. However, the paper does not establish that the reconstructed baseline was independently validated as a functioning implementation before comparing its performance with OntoAct.
A similar issue appears in ALFWorld. Appendix D.5 reports that the reconstructed NeSyC completes 0/134 tasks under standard observations, with all 6,566 executed steps being idle actions. Providing additional initial-state information raises its performance to 35/134, but this does not resolve the fundamental concern about the validity of the baseline adaptation.
I appreciate that the authors disclose these failures rather than concealing them. Nevertheless, including a baseline that consistently fails to produce meaningful actions substantially weakens the interpretation of the comparison, particularly because NeSyC is one of the most directly relevant prior neuro-symbolic approaches.
The other baseline comparisons are not automatically invalidated by this issue. However, the evidence presented is insufficient to establish a reliable advantage over this important line of prior work. A functional and validated baseline implementation is necessary before the reported comparative conclusions can be considered convincing.
- Insufficient isolation of the proposed methodological contributions.
The main conceptual components of OntoAct—LLM-based knowledge extraction, symbolic rule representations, hypothesis evaluation, and action verification—are closely related to existing neuro-symbolic agent frameworks, including NeSyC and WALL-E.
The proposed distinction is the use of ontological abstraction to generalize historical experiences into reusable relational and conditional knowledge, together with an explicit hypothesis-management mechanism.
However, the paper does not provide sufficiently direct component-level ablations to establish the independent contributions of these mechanisms.
In particular, it remains unclear how much of the improvement comes from ontological abstraction itself, rather than the predefined domain-specific schemas, the quality of extracted rules, the hypothesis ledger, or the action verifier. Appendix B further shows that the supported knowledge representations and verification procedures differ substantially across Tacit-FLE, Mars, and ALFWorld.
Without controlled experiments that remove or replace these components while preserving the remaining system, the results primarily demonstrate the effectiveness of the complete engineered pipeline. They do not convincingly establish the necessity or generalizable value of the proposed ontological abstraction.
This concern is particularly important given the substantial overlap between OntoAct and prior neuro-symbolic approaches. The absence of direct mechanism-isolation experiments, combined with the questionable validity of a key baseline, makes the claimed methodological advancement difficult to assess.
Overall, although the proposed system achieves promising results in the reported environments, I do not find the current experimental evidence sufficiently reliable or the independent methodological contribution sufficiently established to support acceptance.
Questions for authors
Q1. Can the authors demonstrate that their NeSyC reconstruction is a valid and functional implementation of the original approach?
Appendix D.1 reports that NeSyC executes only waiting actions across 3,000 evaluation turns on the small and medium Tacit-FLE suites. What specifically causes its planning mechanism to return no executable plans or only waiting actions?
Can the authors provide implementation validation results, such as successful execution on the original benchmark or controlled diagnostic cases demonstrating that the reconstructed planner can generate meaningful actions under its intended assumptions?
Q2. How do the authors distinguish a baseline implementation or environment-adaptation failure from a genuine methodological limitation of NeSyC?
The reconstructed NeSyC also fails on all ALFWorld tasks under standard observations and improves only after additional initial-state information is provided. Does this indicate that the baseline requires different observation or planning interfaces?
Can the authors clarify whether the evaluated configuration preserves the essential operating assumptions of the original method and provide evidence from a functional adaptation under comparable conditions?
Q3. What is the independent contribution of ontological abstraction and explicit hypothesis management?
Can the authors provide controlled ablations that independently remove ontological abstraction, the hypothesis ledger, and action verification while maintaining the same backbone, environment access, and relevant knowledge?
In particular, would the framework retain similar performance if its learned relational knowledge were represented as ordinary structured rules without the proposed ontology?
These results are necessary to determine whether the improvements arise from the claimed methodological contributions or from the overall system configuration.
4 comments
Clarifications on the ASP Control and Remaining Interpretation
Thank you for these concrete examples and the repair-activity analysis. The successful reuse of a variable-containing Chain rule substantially addresses my concern that the ASP condition might be restricted to instance-specific knowledge. The Conjunction traces also make the distinction between proposing a correct target and providing admissible diagnostic support much clearer.
I appreciate the explicit clarification that this comparison evaluates representation together with structured induction, rather than representation alone. This is a reasonable scope for the experiment. Please retain this qualification in the revised claims: the observed target-specific rule heads characterize the evaluated induction pipeline, not an inherent limitation of ASP.
The repair counts also show that ASP is not simply inactive in most episodes. However, fewer repairs per episode can contribute to lower SI, and the current counts do not establish how much of the SI difference reflects repair volume versus intervention accuracy. Please present success, SI, and executed repairs together and avoid attributing the SI difference to a single explanation. A descriptive spurious-repair fraction from the existing logs would further help, provided its event definition is consistent.
Please incorporate the induction prompts, representative rules, and execution traces into the revised manuscript or supplement, as proposed. These additions substantially reduce my concerns about the ASP control. Together with the previously reported results, they make me inclined toward 3 (weak accept), with the remaining reservation concerning the attribution of gains to representation versus structured induction.
Final Assessment of the Revised ASP Comparison
Thank you for incorporating the additional analyses into the revised manuscript.
I have reviewed the updated Section 5.3, Table 8, and Appendix E.5. The revision satisfactorily addresses my main outstanding concerns about the ASP comparison.
In particular, reporting success, SI, executed repairs, and the spurious-repair fraction together makes the interpretation substantially clearer. The results appropriately distinguish reduced intervention volume from intervention accuracy. The representative rules and execution traces also demonstrate that the ASP condition can generalize across targets, while identifying specific limitations of the evaluated induction pipeline.
I also appreciate the explicit qualification that the comparison evaluates representation together with structured induction, rather than establishing an inherent limitation of ASP.
The remaining distinction between representation effects and induction-procedure effects is a methodological limitation, but it is now appropriately acknowledged and does not invalidate the overall system-level contribution.
Given these substantive additions and clarifications, I am updating my rating to 3 (Weak Accept).
Thank you for the careful and constructive discussion throughout the review process. Your questions helped us strengthen the experimental evidence and clarify the scope of our contribution. We appreciate the time you devoted to reviewing the additional analyses and revised manuscript.
Follow-up on the ASP Representation Control and Potential Rating Revision
Thank you for conducting the additional ASP-based experiment. I appreciate the authors' continued efforts to address my concerns through targeted experiments.
This comparison directly addresses my previous request for an alternative structured representation retaining symbolic verification and hypothesis management. The three-run results and task-level breakdowns are valuable.
The reported overall improvements are substantial: +23.8, +39.9, and +52.5 percentage points across Small, Medium, and Large settings. If the comparison is fair and the ASP baseline functions correctly, these results would provide stronger evidence for OntoAct's methodological contribution.
However, I would appreciate clarification of two remaining issues.
1. Equivalence of structural priors and available information
OntoAct uses a predefined ontology schema specifying entity types, relations, observable attributes, and admissible axiom templates.
Does the ASP control receive equivalent information, including entity types, relation constraints, diagnostic templates, and action semantics?
Does its induction prompt explicitly encourage variable-based generalization over entity roles and types?
This matters because ASP supports reusable variable-containing rules. If the ASP baseline receives weaker structural guidance, the performance gap may partly reflect differences in domain knowledge or inductive bias rather than representation alone.
I do not require identical induction algorithms, since structured induction is part of the proposed contribution. However, the comparison should clarify which advantages arise from the representation and induction mechanisms versus differences in supplied prior knowledge.
2. Functional validation and failure analysis of ASP
The task-level results reveal an interesting pattern: ASP outperforms OntoAct on Small Chain (69.2% versus 56.4%) but performs substantially worse on Conjunction, declining from 33.3% to 15.2% across scales, while OntoAct achieves 100%.
The explanation that induced ASP rules predominantly reference specific units is plausible, but requires supporting evidence.
Could the authors provide representative rules from successful Chain and failed Conjunction cases and clarify:
Representative execution traces or brief stage-level diagnostics would help establish that the ASP baseline functions as intended.
Additionally, ASP reports fewer overall spurious interventions. Could the authors clarify whether this reflects fewer repair attempts or more conservative behavior? Reporting the proportion of episodes without repair attempts would help interpret this result.
Updated assessment and potential rating revision
The rebuttal has substantially strengthened the paper. My earlier concerns about missing component ablations and unexplained NeSyC behavior have been meaningfully addressed. The ASP comparison also directly targets my remaining concern about the independent contribution of ontological abstraction.
I am now seriously considering increasing my rating from 2 (Weak Rejection) to 3 (Weak Acceptance).
My decision depends primarily on whether the ASP baseline receives comparable domain information and whether its induction, verification, and hypothesis-management components function correctly.
If these points are satisfactorily clarified, I would regard the new results as substantial evidence for OntoAct's contribution and would be inclined to update my recommendation.
Thank you again for the constructive discussion and additional experimental effort.