- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates cross-runtime transfer of agent harnesses, emphasizing that producing a plausible final answer is not equivalent to preserving the source harness's execution behavior. The proposed HarnessIR uses a fixed harness ontology to represent states, actions, decisions, constraints, and related entities. The method combines ontology-guided static semantic extraction from AST-partitioned source programs with policy inference from source-native execution traces. The recovered semantic graph specifies state- and decision-dependent action rules with associated constraints. A separate target-runtime semantic map, obtained from target framework documentation, guides the realization of the graph as a native target harness.
The authors also introduce HarnessIR-Bench, comprising 18 source-harness entries, with six held-out evaluation scenarios per test source and transfers to LangGraph and Microsoft Agent Framework. Across deepseek-v4-flash and GLM-5.2, the reported mean Behavior Success is 45.00%, versus 34.03% for the strongest static ontology-based baseline, SOT. The paper also reports task-level and joint end-to-end metrics, dimension-level analyses, and component ablations.
Critical strengths
1. Practical and well-motivated research problem.
The paper addresses a practical and increasingly relevant problem: transferring agent harnesses across heterogeneous runtimes while preserving their original behavior. As agent frameworks continue to evolve, the ability to reuse existing harnesses without losing important behavioral constraints is valuable and well motivated.
2. An appropriate application of the compiler IR paradigm.
The paper draws a natural and compelling analogy to intermediate representations in compiler-based code translation. Introducing an ontology-guided IR to decouple source harness semantics from target-runtime implementations is a well-motivated design choice that provides a clear and systematic approach to cross-runtime harness transfer.
Critical weaknesses
While the paper evaluates behavioral preservation after cross-runtime transfer, it does not directly assess the accuracy and completeness of the recovered HarnessIR. In particular, it remains unclear whether the extracted execution policies faithfully capture the source harness's guards, state dependencies, and behavioral constraints, especially for branches not observed in the available execution traces.
Questions for authors
Why is the evaluation limited to general-purpose agent frameworks? Have the authors considered evaluating HarnessIR on widely used coding-agent systems such as Codex and Claude Code? Such an evaluation could provide stronger evidence of the practical applicability and generalizability of HarnessIR.
3 comments
Reply to above section
Thank you for the detailed response and for conducting additional experiments on Codex and Claude Code. I appreciate the clarification regarding the complementary roles of static analysis and execution traces in semantic recovery. The additional experiments strengthen the practical relevance of the work and help address my concerns about its broader applicability. However, the accuracy and completeness of the recovered IR, particularly for branches not observed in the available execution traces, remain insufficiently validated. Therefore, I have decided to maintain my original rating (Weak Accept).
Response to the Follow-up Comments
Thank you very much for your thoughtful feedback and positive assessment of our work.
We acknowledge that our response may not have fully addressed your concern regarding the accuracy and completeness of IR recovery, particularly for branches not covered by the available execution traces. We sincerely appreciate this observation and will continue to refine the semantic recovery process and explore more effective approaches to validating the recovery of unobserved branches in our future work.
Thank you again for your time and constructive suggestions.
Response to Weakness 1 and Question 1
It is encouraging that you recognize the significance of our research and the motivation behind our IR-based approach. We highly value your thoughtful feedback, which helps us further improve this work. We respond to each point below.
For W1: Semantic recovery, traceability, and IR evaluation limitations.
First, we would like to clarify that HarnessIR does not recover execution policies solely from execution traces. Instead, it combines static semantic analysis of the source code with dynamic execution evidence, leveraging their complementary information to recover the Harness's semantic structure and execution policies.
For branches not observed in the execution traces, static code analysis provides a basis for recovering the explicit control structures defined in the source code, without relying entirely on dynamic observations. Meanwhile, during the recovery process, we preserve the corresponding source code snippets and execution traces as evidence for all recovered semantic entities, relations, and execution rules, enabling their traceability and verification with respect to the recovered Guard Conditions, State Dependencies, and Behavioral Constraints. When the evidence is insufficient or ambiguous, we preserve the uncertainty rather than infer unverified runtime behavior. This mechanism aims to mitigate information loss caused by limited execution traces and avoid generating unsupported execution policies.
We also acknowledge that our current experiments primarily validate HarnessIR indirectly through behavioral preservation after migration, without independently evaluating the accuracy and completeness of the recovered IR. Therefore, directly evaluating IR-level recovery quality, particularly the recovery of unobserved branches, remains a limitation of this work and an important direction for future research.
For Q1: Coding agent evaluation.
We initially selected general-purpose agent frameworks such as LangGraph and Microsoft Agent Framework because they provide explicit programming interfaces that facilitate the direct implementation of target Harness components, including state management, control flow, and action execution logic. This choice was primarily motivated by the experimental setup and the implementation interfaces provided by the target runtimes, rather than a limitation of HarnessIR to general-purpose agent frameworks.
To further evaluate the applicability of HarnessIR to specialized coding-agent systems, we followed your suggestion and added Codex and Claude Code as target systems in supplementary experiments. It is worth noting that although both Codex and Claude Code provide extension mechanisms for external integration, unlike LangGraph, they do not directly expose programming interfaces for constructing an entire Harness. Therefore, in our experiments, we use their respective tool interfaces, MCP, and Hooks to accommodate and implement the migrated execution policies, rather than directly controlling or replacing their entire internal execution mechanisms. The results are as follows:
The results demonstrate that HarnessIR can transfer source Harness behaviors to these two specialized coding-agent systems, achieving non-trivial behavioral preservation and end-to-end task execution. Meanwhile, because the native execution mechanisms and extension interfaces of these systems differ from those of general-purpose agent frameworks, some source-Harness state dependencies and execution constraints may be difficult to fully realize through the available extension mechanisms, which may also affect the resulting behavioral preservation. Nevertheless, these supplementary experiments extend the target-runtime coverage of HarnessIR beyond general-purpose agent frameworks and provide additional empirical evidence of its practical applicability and generalizability across different types of agent systems. We have added the coding-agent transfer experiments to Section 5.6 of the revised manuscript.
We sincerely appreciate your time and thoughtful consideration of our responses. We hope our clarifications have addressed your concerns, and would be grateful for any further questions or suggestions you may have.