- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how to improve the traceability and auditability of LLM-based software agents. In typical agent workflows, logs record the actions taken by an agent, but it can be difficult to determine which original user intent motivated a specific action, how the decision was made, and who was responsible. The authors propose SPINE, a framework that organizes policies into a hierarchy of declarative, definitive, and imperative policies, with explicit parent-child relationships linking the original intent to executable actions.
The main contribution is an explicit lineage structure connecting user intent, intermediate policies, and runtime actions. Each policy records its parent, definer, and enforcer, while an orchestrator attaches the active policy ID to execution records at write time. This allows actions to be traced back to the original intent without having to reconstruct these relationships from logs afterward.
The authors evaluate SPINE on three SWE-bench Lite software bug-fixing tasks using two language models, comparing it with raw prompts, free-form plans, and trajectory-based representations. The results show that SPINE substantially improves traceability and accountability while achieving task resolution rates comparable to those of other methods.
Critical strengths
A key strength of this work is that policy attribution is handled by the orchestrator rather than left to the LLM itself. This makes execution records more reliable, as the model cannot simply omit or incorrectly report the policy associated with an action.
Critical weaknesses
The main weakness is the limited experimental evaluation. SPINE is tested on only three software bug-fixing tasks, making it difficult to determine whether the results generalize to more complex tasks or other application domains.
Questions for authors
Does the construction of the policy tree rely entirely on the LLM generating valid structured outputs? If so, for more complex tasks, does SPINE provide mechanisms to ensure policy ID uniqueness and the validity of parent references to ensure that every referenced parent node actually exists?
Response to Reviewer 1sQi
We sincerely thank the reviewer for their encouraging assessment and for recognizing the core strength of SPINE. Your constructive feedback directly guided our expanded evaluation, and we are glad to share that our revised manuscript fully resolves your concern about scale and clarifies our structural guarantees.
Limited experimental evaluation: We have addressed this concern. We have expanded the evaluation to all 300 SWE-bench Lite instances for refinement and 225 to 285 per representation and model for execution, with five representations, including the newly added Tree of Thought (ToT) planning, and two models (Table 1).
Policy ID uniqueness and the validity of parent references: SPINE does not rely on the LLM alone to maintain tree integrity. Instead, SPINE cleanly separates policy proposal (by the LLM) from deterministic admission validation (enforced by the orchestrator before any policy enters the tree):
Parent Reference Exists (parent_id): The referenced parent_id is already committed in the tree, guaranteeing a single valid parent and a cycle-free path back to root intent P0.
Tier Ordering Holds: The parent-to-child refinement strictly follows Declarative -> Definitive -> Imperative, and only Imperative leaf nodes can execute tools.
Required Fields Are Present: Definer, Enforcer, and (R,S,T,CT) scope fields are properly populated.
We have added this deterministic admission mechanism in Section 5.2 of the revised paper, and we hope these full-scale evaluation and clarifications further strengthen your confidence in our work.